From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by sourceware.org (Postfix) with ESMTPS id 26EB43858D35 for ; Fri, 7 Aug 2020 08:07:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 26EB43858D35 Received: by mail-ed1-x544.google.com with SMTP id c2so629897edx.8 for ; Fri, 07 Aug 2020 01:07:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=dcGwRTf/QrfUgDpt6mvopZaWNE667gD2SD3zAkEotS0=; b=G0Yvp4NM/kJZxegF6OvfyJMxniLcog/KgxwVHC6PbLsvb337BxrCPbXYrpViJq8xNG bqwF2eqPIwBGGxD8K12F/o+rJ5MUwSm7RUCPIa7eX9L83cipxy/k8tilr1XsaW7WbtEr segauW6/e17S7dE/n3BxxVwYrVEkxrW2n/HBvxVOhFYShRgx0WqOKZcKDNNHTG5uOK0u d7nRuCfXCgf8QOhHSiILvz26DXjjtcCOdISJfc1TU2i8hZurLQObCHbnmutyAo41MDae iIIFJdRysX+mPXqTj9ksk2SohDyMZSr1LoIHdetViPwe8zOIBh40V+2wMRalRcN+lvCv z7uw== X-Gm-Message-State: AOAM5332d8o2RadagLW9xG4hnvJ8KicKrPMFaWcuOUcjFb4Qrs58E1U0 AJQMHi7LAgPt47SljtJ406pxN3Lh5JDvXFZS4zjVhdB18vw= X-Google-Smtp-Source: ABdhPJxo0GtS5OH3k1xnbLcNXzE8BWdkPCzNqVGR6njf9lEAOoH6OaQsEskYJeBmDPaI7WSzPc65AtFtUIwISbYK/CA= X-Received: by 2002:a05:6402:1a26:: with SMTP id be6mr7642947edb.162.1596787625881; Fri, 07 Aug 2020 01:07:05 -0700 (PDT) MIME-Version: 1.0 References: <323895c2-0df4-f8da-791e-03579f2a8caf@cornell.edu> In-Reply-To: <323895c2-0df4-f8da-791e-03579f2a8caf@cornell.edu> From: =?UTF-8?Q?Morten_Kj=C3=A6rulff?= Date: Fri, 7 Aug 2020 10:06:54 +0200 Message-ID: Subject: Re: name pipe problem: 1 writer, multiple concurrent readers To: cygwin@cygwin.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Aug 2020 08:07:09 -0000 On Thu, Jul 16, 2020 at 9:26 PM Ken Brown wrote: > > On 7/3/2020 7:09 AM, Ken Brown via Cygwin wrote: > > On 7/2/2020 1:50 PM, Morten Kj=C3=A6rulff via Cygwin wrote: > >> I think we got a new release around the beginning of June, right? > >> You said that there were still issues (I can confirm). > >> If it can help, here is the output I see today of above scripts: > >> > >> $ ./tp.sh > > [...] > >> 0 [fifo_reader] diff 1806 C:\cygwin\bin\diff.exe: *** fatal > >> error - Can't update my handlers, Win32 error 87 > > > > Thanks for the report and the simple test case. Obviously I still have= more > > work to do on this. > > Hi Morten, > > I've attempted to fix the bugs (see > https://cygwin.com/pipermail/cygwin-patches/2020q3/010380.html). With th= ese > patches installed, I no longer see a fatal error or hanging diff processe= s. But > your script still doesn't work as you expect. On a typical run of the pa= rallel > part, 6 or 7 of the 10 diff processes see the FIFO t.pip as empty. > > Here's a sample run under strace, so that I could see what was going on. = 7 of > the 10 diff processes saw t.pip as empty on this run. > > $ strace -o tpip.sh.strace sh -c ./tpip.sh > PID PPID PGID WINPID TTY UID STIME COMMAND > 1307 1306 1307 10932 pty1 197609 17:50:12 /usr/bin= /bash > 18426 1 18426 9360 cons0 197609 06:47:31 /usr/bin= /sh > 18429 18426 18426 13900 cons0 197609 06:47:32 /usr/bin= /ps > 1306 1 1306 3768 ? 197609 17:50:11 /usr/bin= /mintty > 18424 1307 18424 21840 pty1 197609 06:47:31 /usr/bin= /strace > result1 start > 10 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > result1 end > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > 0a1,2 > > line1 > > line1 > > line2 > > line2 > result2 start > 10 > 0 > 1 > 1 > 0 > 1 > 1 > 1 > 1 > 1 > 0 > result2 end > PID PPID PGID WINPID TTY UID STIME COMMAND > 18480 18430 18426 15580 cons0 197609 06:47:33 /usr/bin= /cp > 1307 1306 1307 10932 pty1 197609 17:50:12 /usr/bin= /bash > 18484 18426 18426 21264 cons0 197609 06:47:44 /usr/bin= /ps > 18430 18426 18426 23472 cons0 197609 06:47:32 /usr/bin= /sh > 18426 1 18426 9360 cons0 197609 06:47:31 /usr/bin= /sh > 1306 1 1306 3768 ? 197609 17:50:11 /usr/bin= /mintty > 18424 1307 18424 21840 pty1 197609 06:47:31 /usr/bin= /strace > > I'm attaching your script for ease of reference, and I'm attaching an exc= erpt > from the strace output, to which I've added a few comments. The excerpt = shows > all open, close, read, and write system calls involving t.pip. > > Here's a summary of what you can see from those system calls in the paral= lel > part of the script. In what follows, I've called the diff processes diff= -1, > diff-2,..., diff-10, and similarly for the cp processes (although there a= re only > four of them). > > 1. cp-1 tries to open t.pip for writing and blocks. It unblocks when dif= f-1 > opens t.pip for reading, and both processes run to completion as expected= . > > 2. diff-2, diff-3, diff-4, and diff-5 try to open t.pip for reading, and = they > block until cp-2 opens it for writing. Then cp-2 writes 12 bytes to t.pi= p and > closes it, and the four diff processes all try to read. diff-4 gets ther= e first > and reads the 12 bytes; it reads once more and sees EOF because there is = no data > available in the pipe and there are no writers open[1], so it considers t= hose 12 > bytes to constitute the file t.pip. It later exits with success. > > diff-2, diff-3, and diff-5 all complete their reads before cp-3 opens t.p= ip. > They see EOF for the same reason as above, so t.pip appears empty and the= y exit > with failure. > > 3. diff-6, diff-7, diff-8, diff-9, and diff-10 try to open t.pip for read= ing, > and they block until cp-3 opens it for writing. Then cp-3 writes 12 byte= s to > t.pip and closes it, and the five diff processes all try to read. diff-1= 0 gets > there first and reads 12 bytes followed by EOF; it later exits with succe= ss. > > diff-6, diff-7, diff-8, and diff-9 all complete their reads before cp-4 o= pens > t.pip. They see EOF, so t.pip appears empty and they exit with failure. > > 4. cp-4 tries to open t.pip and blocks because there are no more diff pro= cesses. > > I've run your script on Linux a few times, and it usually[2] behaves as y= ou > expect, with all diff processes succeeding. For reasons I don't understa= nd, the > diff and cp processes apparently alternate most of the time on Linux, rat= her > than having 4 or 5 diff processes lumped together between the cp processe= s as on > Cygwin. > > If someone can figure out the reason for the difference, and if it turns = out to > be related to the FIFO code, I could try to modify the code to make Cygwi= n > behave more like Linux. > > Ken > > [1] From https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.= html: > > When attempting to read from an empty pipe or FIFO: > > * If no process has the pipe open for writing, read() shall return 0= to > indicate end-of-file. > > [2] But I did have one Linux run in which one of the ten diff processes s= aw an > empty t.pip and failed as on Cygwin. Hi, Also from https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.h= tml: "The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified." Would that be the reason why it also fails on Linux? If I change the script to: #diff t.pip t.txt flock t.pip.lock diff t.pip t.txt It seems to work as I want it to, both on cygwin and Linux. My goal is to convert my ~/.netrc to a pipe. The writer would decrypt an `gpg2 -e --default-recipient-self` encrypted version. As a workaround, I tried wrappers for all commands accessing ~/.netrc, like this for curl: #!/bin/sh flock ~/.netrc.rlock /usr/bin/curl "$@" It seems to work, but not really a fine "solution". Is there a more general way of serializing access to a file/pipe? /Morten