Broken process substitution

Broken process substitution

[Daniel Colascione]

Try "echo hello > >(cat)" -- that's supposed to output "hello".

On Cygwin, we get "bash: echo: write error: Bad file descriptor"


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken process substitution

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On 08/13/2010 02:04 PM, Daniel Colascione wrote:
> Try "echo hello > >(cat)" -- that's supposed to output "hello".

What makes you think it's supposed to echo hello?  That's system
specific on what will happen.  According to the bash manual,

>(cat)

is evaluated first, and will result in a /dev/fd reference, or a named
pipe (it so happens that it is a /dev/fd reference in cygwin).  But this
pipe is tied to the subprocess, so it only exists as long as the
subprocess exists.

Then bash does the redirection.  If >(cat) has already gone away, then
the resulting string /dev/fd/63 no longer exists.  So redirecting to a
non-existent file fails.  But if cat hasn't finished executing yet, then
the redirection will succeed.

Classic data race.  Your expectations are wrong, and this is not a bug
in cygwin, per se.

On the other hand, a named fifo might be more persistent; bash might
keep the fifo around until the entire statement is complete, rather than
keeping the pipe and /dev/fd reference for the life of the subprocess.
But I'd have to recompile bash to force it to use a named fifo, and we
already know that named fifos have bugs in cygwin.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Broken process substitution

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

On 08/13/2010 02:17 PM, Eric Blake wrote:
> On 08/13/2010 02:04 PM, Daniel Colascione wrote:
>> Try "echo hello > >(cat)" -- that's supposed to output "hello".
> 
> What makes you think it's supposed to echo hello?  That's system
> specific on what will happen.  According to the bash manual,
> 
>> (cat)
> 
> is evaluated first, and will result in a /dev/fd reference, or a named
> pipe (it so happens that it is a /dev/fd reference in cygwin).  But this
> pipe is tied to the subprocess, so it only exists as long as the
> subprocess exists.

Then again, cat should exist until something causes the input side of
its pipe to declare EOF; so I guess there's no race in this example
after all.  Rather, it looks like a limitation in cygwin1.dll.  I don't
know why bash is unable to duplicate the output end of the pipe to the
echo process, unless cygwin's /dev/fd handling doesn't work on pipes.
But that's highly likely that you are dealing with yet another one of
cygwin's pipe handling shortfalls.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Broken process substitution

[Daniel Colascione]

On Fri, Aug 13, 2010 at 1:25 PM, Eric Blake <eblake@redhat.com> wrote:
> Then again, cat should exist until something causes the input side of
> its pipe to declare EOF; so I guess there's no race in this example
> after all.  Rather, it looks like a limitation in cygwin1.dll.  I don't
> know why bash is unable to duplicate the output end of the pipe to the
> echo process, unless cygwin's /dev/fd handling doesn't work on pipes.
> But that's highly likely that you are dealing with yet another one of
> cygwin's pipe handling shortfalls.

Would these shortfalls also explain why this script doesn't do what
I'd expect (that is, output "hello" and exit)? It just hangs right now
--- this is the ps output:

I    8580    7740    7740       6340    3 4412345 13:41:41 /usr/bin/cygpath
I    7724    7740    7740       4796    3 4412345 13:41:41 /usr/bin/cygpath
O    1736    7740    7740       8796    3 4412345 13:41:41 /usr/bin/echo

So, err, echo is waiting to output, and cygpath is waiting to receive
input? I don't see why the script shouldn't be making forward
progress.

#!/bin/bash

tmpdir=$(mktemp -dt cygfilter-XXXXXX)
stdout_pid=
stderr_pid=

function cleanup() {
    [[ -n $stdout_pid ]] && /bin/kill $stdout_pid
    [[ -n $stderr_pid ]] && /bin/kill $stderr_pid
    rm -rf "$tmpdir"
}

trap cleanup 0

mkfifo "$tmpdir/f-out"
mkfifo "$tmpdir/f-err"

cygpath -u -f "$tmpdir/f-out"&
stdout_pid=$!
disown %%

cygpath -u -f "$tmpdir/f-err" >&2 &
stderr_pid=$!
disown %%

"$@" >"$tmpdir/f-out" 2>"$tmpdir/f-err"


# Run as cygfilter /bin/echo hello
# -----------

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken process substitution

[Daniel Colascione]

On Fri, Aug 13, 2010 at 1:44 PM, Daniel Colascione
<dan.colascione@gmail.com> wrote:
> On Fri, Aug 13, 2010 at 1:25 PM, Eric Blake <eblake@redhat.com> wrote:
>> Then again, cat should exist until something causes the input side of
>> its pipe to declare EOF; so I guess there's no race in this example
>> after all.  Rather, it looks like a limitation in cygwin1.dll.  I don't
>> know why bash is unable to duplicate the output end of the pipe to the
>> echo process, unless cygwin's /dev/fd handling doesn't work on pipes.
>> But that's highly likely that you are dealing with yet another one of
>> cygwin's pipe handling shortfalls.
>
> Would these shortfalls also explain why this script doesn't do what
> I'd expect (that is, output "hello" and exit)? It just hangs right now
> --- this is the ps output:

Actually, the seemingly-equivalent version below works fine. Maybe
there was a race between the cygpath's starting to read the fifo and
the last line starting to write to it.

The *real* bug seems to be triggered by the following commands:

#!/bin/sh
cd /tmp
mkfifo blah
( echo hello > blah )&
cat blah

On other systems (OS X and Linux), that just outputs "hello", then
both processes exit. On Cygwin, the writer is blocked indefinitely and
has to be SIGKILLed --- even if a reader then starts. And the reader
acts as if there were no writer at all.

--------

#!/bin/bash
set -e

tmpdir=$(mktemp -dt cygfilter-XXXXXX)

function cleanup() {
    rm -rf "$tmpdir"
}

trap cleanup 0

mkfifo "$tmpdir/f-out"
mkfifo "$tmpdir/f-err"

cygpath -u -f- < "$tmpdir/f-out"&
cygpath -u -f- < "$tmpdir/f-err" >&2 &

"$@" >"$tmpdir/f-out" 2>"$tmpdir/f-err"

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken process substitution

[Corinna Vinschen]

On Aug 13 14:25, Eric Blake wrote:
> On 08/13/2010 02:17 PM, Eric Blake wrote:
> > On 08/13/2010 02:04 PM, Daniel Colascione wrote:
> >> Try "echo hello > >(cat)" -- that's supposed to output "hello".
> > 
> > What makes you think it's supposed to echo hello?  That's system
> > specific on what will happen.  According to the bash manual,
> > 
> >> (cat)
> > 
> > is evaluated first, and will result in a /dev/fd reference, or a named
> > pipe (it so happens that it is a /dev/fd reference in cygwin).  But this
> > pipe is tied to the subprocess, so it only exists as long as the
> > subprocess exists.
> 
> Then again, cat should exist until something causes the input side of
> its pipe to declare EOF; so I guess there's no race in this example
> after all.  Rather, it looks like a limitation in cygwin1.dll.  I don't
> know why bash is unable to duplicate the output end of the pipe to the
> echo process, unless cygwin's /dev/fd handling doesn't work on pipes.

I think I found the culprit in Cygwin.  The problem occurs in writev.
At the start of this function is a test which checks for the original
open mode of the file descriptor.  If it's O_RDONLY, the writev function
errno set to EBADF.

However, this doesn't make much sense, afaics.  For one thing, if the
underlying handle is really only readable, the underlying NtWriteFile
function will fail anyway.  And, for this pipe problem it seems to
be in the way.  If I remove this test, I get

  bash-3.2$  echo hallo > >(cat)
  bash-3.2$ hallo
  [...hangs here until any key is pressed...]
  bash-3.2$

That's the exact same behaviour as on Linux, afaics.

However, I'm not sure just removing the test is really the right
solution.  I'll have to take a deeper look first.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken process substitution

[Corinna Vinschen]

On Aug 14 12:32, Corinna Vinschen wrote:
> On Aug 13 14:25, Eric Blake wrote:
> > On 08/13/2010 02:17 PM, Eric Blake wrote:
> > > On 08/13/2010 02:04 PM, Daniel Colascione wrote:
> > >> Try "echo hello > >(cat)" -- that's supposed to output "hello".
> > > 
> > > What makes you think it's supposed to echo hello?  That's system
> > > specific on what will happen.  According to the bash manual,
> > > 
> > >> (cat)
> > > 
> > > is evaluated first, and will result in a /dev/fd reference, or a named
> > > pipe (it so happens that it is a /dev/fd reference in cygwin).  But this
> > > pipe is tied to the subprocess, so it only exists as long as the
> > > subprocess exists.
> > 
> > Then again, cat should exist until something causes the input side of
> > its pipe to declare EOF; so I guess there's no race in this example
> > after all.  Rather, it looks like a limitation in cygwin1.dll.  I don't
> > know why bash is unable to duplicate the output end of the pipe to the
> > echo process, unless cygwin's /dev/fd handling doesn't work on pipes.
> 
> I think I found the culprit in Cygwin.  The problem occurs in writev.
> At the start of this function is a test which checks for the original
> open mode of the file descriptor.  If it's O_RDONLY, the writev function
> errno set to EBADF.
> 
> However, this doesn't make much sense, afaics.  For one thing, if the
> underlying handle is really only readable, the underlying NtWriteFile
> function will fail anyway.  And, for this pipe problem it seems to
> be in the way.  If I remove this test, I get
> 
>   bash-3.2$  echo hallo > >(cat)
>   bash-3.2$ hallo
>   [...hangs here until any key is pressed...]
>   bash-3.2$
> 
> That's the exact same behaviour as on Linux, afaics.
> 
> However, I'm not sure just removing the test is really the right
> solution.  I'll have to take a deeper look first.

Yep, it was another problem.  At one point the code missed to copy
over information about a file descriptor.  I applied a fix to CVS.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Broken process substitution

[Daniel Colascione]

On Sat, Aug 14, 2010 at 4:17 AM, Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
> Yep, it was another problem.  At one point the code missed to copy
> over information about a file descriptor.  I applied a fix to CVS.

I'm running a DLL built from the latest CVS, and all is well. Thanks
for the fix!

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple