Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

[James Johnston]

Hi,

I am calling a few different .NET Framework 4.0 command line tools from
Cygwin and have noted that .NET Framework 4.0 introduced a change that
causes an incompatibility between it and Cygwin.  I have carefully examined
.NET Framework 4.0 source code and also checked things out with an API
monitor utility, and have arrived at a simple test case in straight Win32 C
code.

// File ReaderCPP.cpp
// The following code was compiled in Visual C++ 2008.  However,
// I expect any standard Windows compiler to suffice.

#include <windows.h>

int main()
{
    HANDLE hIn = GetStdHandle(STD_INPUT_HANDLE);

    // NOTE:  NETFX 4.0 introduced these calls to PeekNamedPipe
    // and WaitForMultipleObjectsEx.  NETFX 2.0 would just go
    // straight to ReadFile.  The reasoning in the NETFX 4.0 source
    // source code for the change was to support aborting the
    // thread.

    // Program will not hang if this code is removed:
    DWORD read, avail, left, written;
    PeekNamedPipe(hIn, NULL, 0, &read, &avail, &left);

    // This function never unblocks if PeekNamedPipe is called
    // and we are in Cygwin.  It works fine from msysGit's bash
    // and also works fine from cmd.exe.
    WaitForMultipleObjectsEx(1, &hIn, TRUE, INFINITE, TRUE);

    // Echo output (for debug purpose)
    char buf[1000];
    ReadFile(hIn, buf, 1000, &read, NULL);
    WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), buf, read, &written, NULL);
    return 0;
}

Then run this from bash shell:

( sleep 3 ; echo test ) | ./ReaderCPP

This issue will affect every .NET Framework 4.0 program that attempts to
read from standard input.  I suspect there is a race condition as sometimes
some NETFX 4.0 programs don't hang, and sometimes they do.  This test case
seems to hang 100% reliably on my system, however. 

I tested under these configurations:

 * ReaderCPP always compiled with Visual C++ 2008.  (Other compilers will
probably yield identical results.)
 * Cygwin with setup updated and all pending packages installed today, and I
used the latest Cygwin1.dll snapshot as of today:  fails; ReaderCPP hangs.
 * Released version of Cygwin from a few months ago:  fails.
 * Same command run from msysGit's bash:  works fine; ReaderCPP exits after
3 seconds and echoes input, as expected.
 * Running "echo test | ReaderCPP" from Windows command prompt: works fine.

Seems like Cygwin is doing something different with the standard input pipe
compared to other non-Cygwin shells that causes this synchronization issue
and causes WaitForMultipleObjectsEx to hang, even though there is pending
input after 3 seconds.  Does anyone else reproduce this?  Any ideas why or
what the fix might look like?  Thank you for any feedback.

Best regards,

James Johnston



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

MSDN does not list pipe handles as waitable with WaitForMultipleObjectsEx
(only a user comment at the end does).  If you trace execution without
PeekNamedPipe(), you'll see that WaitForMultipleObjectsEx() returns
immediately, and that it is ReadFile() that blocks and waits for
the input (which will be supplied after 3 seconds).

--quote--
The WaitForMultipleObjectsEx function can specify handles of any of the following object types in the lpHandles array:

    Change notification
    Console input
    Event
    Memory resource notification
    Mutex
    Process
    Semaphore
    Thread
    Waitable timer
--quote--

Also from here:

http://stackoverflow.com/questions/14203618/windows-api-wait-for-data-to-be-available-on-non-gui-console-input-pipe-based

--quote--
Is it possible to wait for data to be available on a non-GUI-based console input? The issue is that WaitFor* methods do not support PIPE handles and therefore STDIN is not supported if the process input is fed from another process, e.g. using the pipe | functionality of cmd.

The only way I know to query a pipe for data without actually reading the data is to use PeekNamedPipe(), but then you lose the ability to wait on a timeout effectively, because it has no timeout so you would have to implement a manual loop instead.
--quote--

HTH,

Anton Lavrentiev
Contractor NIH/NLM/NCBI


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> Check line 192 for a call to the WaitForAvailableConsoleInput function.

I guess they expect hFile to be a console handle (which IS waitable).

And my guess that they added the Peek intentionally to prevent
the WaitFor call from slipping through (as we've seen it does, otherwise)
on things that weren't supposed to be there (such as pipes).
They probably know of a side effect of peeking a pipe, which makes
the WaitFor API block on them (thus, still create a good cancellation
point for the thread) -- again, as they would not expect a pipe there (but
a good console handle;  and if it isn't -- well, it's by design).

But I'm only speculating here.  On the other hand, I would not be surprised
to learn that one team in MS would not know how to deal with the pipes and
introduced a bug, easily.  MS is a large company, and I witnessed such
things to happen in teams of a much smaller scale, from my own experience.

In our apps we use the Peek everywhere for pipe polling, and do not
rely on anything else.

And from Cygwin point of view, I was trying to deal with mixing native
Windows things and UNIX, and it seems to be a big no-no, from the entire
design point -- Cygwin is not a tool for be-friending Windows and Unix,
rather a way to run your beloved GNU tools on a Windows box.

Sorry you ran into this trouble...

Anton Lavrentiev
Contractor NIH/NLM/NCBI


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

[James Johnston]

(apologies to the list; I accidentally sent this reply directly to Anton and
so his reply that he just sent will appear out of order on the list
archive).

Hi Anton,

Thanks for the response.  Comments below...

> MSDN does not list pipe handles as waitable with WaitForMultipleObjectsEx
> (only a user comment at the end does).

That is true.  One would rightfully assume that it's not supported.  I have
assumed that myself in the past when working on asynchronous code for pipes
for a completely unrelated application.

However, their own .NET Framework 4.0 does this, so it must be at least
supported on some undocumented level.  This is the stock version of the
framework available on Windows 8; the older version that does not have the
wait, NETFX 2.0, is not installed by default.  And of course, the older
version doesn't have any of the new features available in NETFX 4.x - of
which there are many.  Sadly, "fixing" the code in the .NET Framework to use
a "supported" asynchronous mechanism is not realistic.  Nobody except
Microsoft can patch the .NET Framework, unfortunately... (it is not
(L)GPL!!)  The reference code is just for debugging and learning; you can't
use it to modify anything.

Here is the reference source code:

http://www.dotnetframework.org/default.aspx/4@0/4@0/untmp/DEVDIV_TFS/Dev10/R
eleases/RTMRel/ndp/clr/src/BCL/System/IO/__ConsoleStream@cs/1305376/__Consol
eStream@cs

Check line 192 for a call to the WaitForAvailableConsoleInput function.  I
was unable to find the source code for that function, which is an internal
C/C++ function (this is a mixed-mode assembly).  However, I was able to
determine that function calls PeekNamedPipe followed by
WaitForMultipleObjectsEx using the very handy freeware API Monitor from
rohitab.com.  Using this tool I also learned the exact parameters that they
pass to each of those functions, and then distill it into a C program that I
sent in my original message that reproduces the issue on my machine.
	
This WaitForAvailableConsoleInput function is new to .NET Framework 4.0, and
according to the comment, was added in order to prevent the ReadFile call
from blocking in order to support aborting a thread.  I think both of us can
probably suggest better ways of handling this that uses a supported
mechanism.  But this is how they did it. :(

As mentioned in the original message, this pattern of PeekNamedPipe &
WaitForMultipleObjectsEx doesn't cause any problems except with Cygwin for
reasons that are currently unknown to me.

> If you trace execution without
> PeekNamedPipe(), you'll see that WaitForMultipleObjectsEx() returns
> immediately, and that it is ReadFile() that blocks and waits for the input
> (which will be supplied after 3 seconds).

Right; this is very strange and contrary to what one would naturally expect
if WaitForMultipleObjectsEx was supported on named pipes (which it
apparently is at least on some undocumented level, if their own .NET
Framework 4.0 is doing it).  Intuitively, peeking something shouldn't affect
any synchronization.  Yet it does...  (Perhaps waiting on a pipe is not
"fully" supported and Cygwin found an edge case!)

> --quote--
> Is it possible to wait for data to be available on a non-GUI-based console
> input? The issue is that WaitFor* methods do not support PIPE handles and
> therefore STDIN is not supported if the process input is fed from another
> process, e.g. using the pipe | functionality of cmd.
> 
> The only way I know to query a pipe for data without actually reading the
> data is to use PeekNamedPipe(), but then you lose the ability to wait on a
> timeout effectively, because it has no timeout so you would have to
> implement a manual loop instead.
> --quote--

Indeed; there is no clean solution.  All the ideas that come to mind have
problems:

 * WaitForMultipleObjectsEx: not officially supported, as previously noted.
 * Overlapped I/O: perfect supported solution except requires the pipe to be
initialized to support that, which probably wasn't done.
 * PeekNamedPipe in a busy loop as suggested in the quote: (1) high CPU
utilization, (2) and/or high latency on the pipe, due to unnecessary
sleeping, (3) your code stays active in memory and can't be paged out to
disk, (4) all the above kills power efficiency...

I've used overlapped I/O myself in the past on an unrelated application, but
I had full control over the creation of the pipes and full control over the
reading/writing of the pipes - not the case here.

So I can see why they went for WaitForMultipleObjectsEx method... the other
methods have significant performance and/or functional limitations, and they
probably found out from a group within Microsoft that it's ok to wait on a
pipe anyway.

Best regards,

James Johnston


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

[James Johnston]

> > Check line 192 for a call to the WaitForAvailableConsoleInput function.
> 
> I guess they expect hFile to be a console handle (which IS waitable).
> 
> And my guess that they added the Peek intentionally to prevent the WaitFor
> call from slipping through (as we've seen it does, otherwise) on things
that
> weren't supposed to be there (such as pipes).
> They probably know of a side effect of peeking a pipe, which makes the
> WaitFor API block on them (thus, still create a good cancellation point
for the
> thread) -- again, as they would not expect a pipe there (but a good
console
> handle;  and if it isn't -- well, it's by design).
> 
> But I'm only speculating here.  On the other hand, I would not be
surprised to
> learn that one team in MS would not know how to deal with the pipes and
> introduced a bug, easily.  MS is a large company, and I witnessed such
things
> to happen in teams of a much smaller scale, from my own experience.

Perhaps, but how daft would you have to be to not realize that the handle
might be a pipe?  Did they never use any command line interface, like, ever?

I suspect the authors were aware it might be a pipe.  Immediately after
calling WaitForAvailableConsoleInput, they call ReadFile and check the
return value & error code.  They check for ERROR_BROKEN_PIPE, with a comment
"A pipe into stdin was closed.  Not an error, but EOF".  I think they
knew...  There are similar comments/checks when calling WriteFile to write
to stdout/stderr.

Although perhaps, as you might be alluding to, they did not realize that a
pipe isn't waitable (i.e. didn't RTFM).  At any rate, frustrating change.

> In our apps we use the Peek everywhere for pipe polling, and do not rely
on
> anything else.

Nice & convenient if you can do it.  Problem that Microsoft had was they
probably felt they had to have a solution that performs better than that if
it's going into a generic, widely-used framework...

> And from Cygwin point of view, I was trying to deal with mixing native
> Windows things and UNIX, and it seems to be a big no-no, from the entire
> design point -- Cygwin is not a tool for be-friending Windows and Unix,
rather
> a way to run your beloved GNU tools on a Windows box.

So it might seem.  I did not learn this until after I wrote a fair number of
bash scripts that use some Cygwin-specific stuff though.  But then one can
make a valid argument that it's useful to allow GNU tools to interface with
Windows on some level.  Also, I did not learn until later that Cygwin is
required to grovel into a number of undocumented / unsupported behaviors in
Windows, with no supported solution possible.  See: the problems with
forking.  No easy answers, apparently, but the Cygwin team seems to do a
good job working around them as best they can.

And in this case, they might be playing by the documented rules, for all I
know.  I haven't yet had time to delve into what Cygwin does differently
from the other shells I tried.  Maybe there is some uncommon pipe setting or
configuration that causes this issue...

Best regards,

James Johnston



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0

[James Johnston]

> I am calling a few different .NET Framework 4.0 command line tools from
> Cygwin and have noted that .NET Framework 4.0 introduced a change that
> causes an incompatibility between it and Cygwin.  I have carefully
examined
> .NET Framework 4.0 source code and also checked things out with an API
> monitor utility, and have arrived at a simple test case in straight Win32
C
> code.
> 
> <snip test case that calls WaitForSingleObject with unsupported pipe
handle>
> 
> Then run this from bash shell:
> 
> ( sleep 3 ; echo test ) | ./ReaderCPP
> 
> This issue will affect every .NET Framework 4.0 program that attempts to
> read from standard input.

I narrowed down the problem.  The stdin file handle inherited by the
non-Cygwin program ReaderCPP is an overlapped file handle.  I wrote a simple
Win32 C test case that uses CreateNamedPipe & CreateFile like Cygwin does in
fhandler_pipe::create in pipes.cc file.  The test case opens both handles
with FILE_FLAG_OVERLAPPED.  It then calls CreateProcess to start WriterCPP
(containing a Sleep & a WriteFile to stdout) and the existing ReaderCPP
program.  The pipe is used to connect WriterCPP's stdout to ReaderCPP's
stdin.  This causes ReaderCPP to hang, just like it was when called from
Cygwin.  If I remove the problematic WaitForMultipleObjectsEx call from
ReaderCPP, then everything works normally.  Alternatively, if I modify the
program to NOT open pipes in overlapped mode, the WaitForMultipleObjectsEx
is no longer a problem - pointing to the solution.  I can send sample code
if it would be helpful to anyone.

As far as I'm aware, this practice of passing overlapped standard file
handles to Win32 child processes is probably unique to Cygwin.  The standard
CMD.EXE doesn't do it, for example.  One reason why this probably MUST be
the case is given in the MSDN docs for ReadFile:

"A pointer to an OVERLAPPED structure is required if the hFile parameter was
opened with FILE_FLAG_OVERLAPPED, otherwise it can be NULL."

This is further confirmed in the remarks ("The lpOverlapped parameter must
not be NULL").  Because probably every Win32 program in existence isn't
normally using overlapped I/O with their standard file handles, this causes
all of them to exhibit undefined behavior.

The bottom line is that .NET Framework 4.0 programs will hang when they try
to read from standard input unless Cygwin can give them pipe handles that
were NOT opened with overlapped I/O.  Also, any standard Win32 program will
exhibit undefined behavior because most (all?) Win32 C runtime libraries do
not use overlapped I/O with the standard file handles.  This will require
modifications to Cygwin.  It seems that fhandler_pipe::create needs to NOT
create overlapped pipes - at least in the case of pipes destined for
standard Win32 programs' stdin.  That might be easier said than done,
however.  On the other hand, a blanket change of removing
FILE_FLAG_OVERLAPPED carries the risk of breaking existing code in Cygwin -
I don't know how catastrophic such a change might be.

Any ideas?  If removing this flag causes breakage only in rare edge cases or
results in non-POSIX behavior (or no breakage at all), perhaps this behavior
can be customized via a flag in the CYGWIN environment variable.  On the
other hand, if removing the flag causes mass breakage, I am not sure what
the solution should be.

Best regards,

James Johnston


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple