Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[James Johnston]

I have "run into an issue with Cygwin".  This is arguably not a bug in
Cygwin, but in other platform runtime libraries.  Never-the-less, the
symptoms occur only with Cygwin and not the Windows command prompt.  So,
from a practical standpoint, Cygwin is "broken."  This is almost certainly
the same issue mentioned in the "1.7.10->1.7.13 : output from .NET programs
does not get through pipeline to a visual c++ program" thread started by
cygwin AT alanhowells.e4ward.com last week.  It's also related to the issue
I raised over a month ago, titled "Can't reliably redirect standard output
from C# program in recent Cygwin".

To summarize: when piping output from one program to another (i.e. running
"A | B" command to pipe standard output from A to B's standard input),
Cygwin passes through ALL writes performed by a program to standard output
(and probably error), including null writes.  Cygwin passing through these
null writes is a big problem, because multiple runtimes - I tested .NET
Framework 3.5 and the Visual C++ 2008 runtime - cannot properly handle null
writes that they receive on standard input.  (Recall that a null write would
be when a program calls WriteFile Win32 API to write 0 bytes to standard
output or standard error.)  While a null write appears nonsensical, every
single .NET program that uses the Console class to write to standard
output/error will do a null write, as .NET does this to verify the stream is
OK.  Other software could easily decide to write zero bytes to standard
output as well (e.g. if outputting an empty string).

I think these are bugs in the runtimes that handle standard input, because
the documentation for ReadFile clearly states that if the handle is a pipe,
the call succeeds, and zero bytes are returned - then it is a null write on
the pipe and does NOT signify end-of-file.  Instead, end-of-file is
signified by ERROR_BROKEN_PIPE.  Yet, these runtimes erroneously handle the
null write as an end-of-file on standard input anyway, causing the software
using the runtime to malfunction.  For example, I used Reflector to
decompile the Read method in the Stream class that handles standard
input/output in the .NET Framework.  It's very obvious that it calls
ReadFile, and a successful ReadFile that returns zero bytes is treated as
end-of-file.

An alternative explanation might be that all this behavior is by design, and
Cygwin is buggy.  But I haven't found anything in MSDN that would justify
that.  (For example, documentation stating that a redirected standard input
may never have a null write would mean that runtimes could safely assume
that standard input won't have null writes, and Cygwin in error for doing a
null write.  However, as an example, the STARTUPINFO structure documentation
imposes no such requirements when describing the hStdInput handle.)

These runtimes (.NET Framework and Visual C++ runtime) are in wide use.  I
don't think it is realistic to expect any fixes for them any time soon:
Microsoft would need to fix both of these runtimes, and then application
vendors would need to use them.  That will take years.  Therefore,
realistically I hope Cygwin can work around them.

I think a workable fix would be for Cygwin not to pass through null writes
it receives on an output/error pipe.  For example, somewhere in Cygwin I
assume there is a loop that calls ReadFile to read the redirected standard
output from the first program, and then calls WriteFile to send this output
to the second program's standard input.  If the call to WriteFile was
skipped if it would write zero bytes (i.e. so Cygwin doesn't do null writes
itself), I think it would fix the problem and work around all these buggy
runtimes.

I am providing several sample programs that can be compiled in Visual C#
2008 / .NET Framework 3.5, or Visual C++ 2008.  They should demonstrate the
problem 100% of the time on any system, because appropriate Sleep() calls
are made.  This was reproduced on a Cygwin 1.7.14-2 system that I updated
this morning.  Sample programs are divided as follows:

 * A "Sender" program performs a null write on standard output, and then
writes a normal line of text.
 * A "Receiver" program attempts to write all received lines of text.

Source code is below; directions for testing and test results at end of
e-mail following code:

========== SenderCS.cs: Sender program in Visual C# 2008 ==========

class SenderCS {
	static void Main(string[] args) {
		/* Notice how this is such a basic program */
		System.Threading.Thread.Sleep(1000); /* wait for pipes to
set up and for receiving app to block on first ReadFile call */
		/* This will do a null write; it could be left out and the
problem would still occur because any WriteLine() call will do it. */
		/* I include it so that the null write can be placed between
two Sleep() calls. */
		System.Console.Write("");
		System.Threading.Thread.Sleep(1000); /* wait for receiving
end to get the null write */
		System.Console.WriteLine("Hello world!"); /* normal line of
text */
	}
}

========== SenderC.c: Sender program in Visual C++ 2008 ==========

#include <windows.h>
int main() {
	char * test = "Hello world!\n";
	DWORD written;
	HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE); /* Get standard output
file handle */
	Sleep(1000); /* wait for pipes to set up and for receiving app to
block on first ReadFile call */
	WriteFile(h, test, 0, &written, NULL); /* do null write */
	Sleep(1000); /* wait for receiving app to get the null write */
	WriteFile(h, test, lstrlenA(test), &written, NULL); /* print hello
message */
	return 0;
}

========== ReceiverCS.cs: Receiver program in Visual C# 2008 that
demonstrates bug in .NET Framework 3.5 ==========

class Program {
	static void Main(string[] args) {
		/* use a retry loop, because we can't distinguish between a
null write
		and end-of file, due to bugs in Console stream's Read
method. */
		for (int i = 0; i < 10; i++) {
			/* BUG:  docs for ReadLine() say a null signifies
end-of-file, but if it
			encounters a null write on standard input pipe, it
incorrectly thinks it
			is end-of-file and will return null. */
			string nl = System.Console.ReadLine();
			if (nl == null) {
				System.Console.WriteLine("Got end-of-file");
			} else {
				System.Console.WriteLine("Got line {0}",
nl);
			}
		}
	}
}

========== ReceiverCPP.cpp: Receiver program in Visual C++ 2008 that
demonstrates bug in VC++ 2008 runtime / STL ==========

#include <iostream>
#include <string>
using namespace std;
int main() {
	/* you have to use a retry loop, for exact same reasons given for C#
receiver program:
	there is no way to tell difference between end-of-file and null
write. */
	for (int i = 0; i < 10; i++) {
		string str;
		/* BUG: cin will indicate end-of-file on a null write. */
		getline(cin, str);
		if (cin.eof()) {
			cout << "Got end-of-file" << endl;
		} else {
			cout << "Got line " << str << endl;
		}
		cin.clear(); /* future getline calls will always immediately
fail without attempting another read unless we clear EOF/fail flags */
	}
	return 0;
}

========== Test results ==========

The test programs are designed so that they can be run in any combination
from the command prompt.  The output from a sender is piped to the input of
a receiver.  Each combination delivers identical output to the other
combinations:

 * ./SenderCS | ./ReceiverCS
 * ./SenderCS | ./ReceiverCPP
 * ./SenderC | ./ReceiverCS
 * ./SenderC | ./ReceiverCPP

Output from Cygwin will always be:

Got end-of-file
Got line Hello world!
Got end-of-file
Got end-of-file
<snip>

This is wrong, because the program received end-of-file before it was
actually at the end of the input stream, due to the bug in its runtime's
handling of return values from ReadFile API.  I did not do any tests using
standard error, but I assume Cygwin redirects standard error in the same way
it redirects standard output, in which case it would have the same problem.

Note that we can do the same test from the Windows command prompt:

Got line Hello world!
Got end-of-file
Got end-of-file
<snip>

The Windows command prompt apparently strips the null writes before passing
them to the receiver's standard input, so that the results are now correct.
The fact that Windows command prompt delivers correct results is probably a
reason why these bugs exist in the runtimes in the first place: probably
nobody really tested to make sure the runtime was compatible with other
redirected standard inputs that might have null writes.

Best regards,

James Johnston



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Christopher Faylor]

On Thu, Apr 26, 2012 at 09:18:27PM -0000, James Johnston wrote:
>========== SenderC.c: Sender program in Visual C++ 2008 ==========
>
>#include <windows.h>
>int main() {
>	char * test = "Hello world!\n";
>	DWORD written;
>	HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE); /* Get standard output
>file handle */
>	Sleep(1000); /* wait for pipes to set up and for receiving app to
>block on first ReadFile call */
>	WriteFile(h, test, 0, &written, NULL); /* do null write */
>	Sleep(1000); /* wait for receiving app to get the null write */
>	WriteFile(h, test, lstrlenA(test), &written, NULL); /* print hello
>message */
>	return 0;
>}
>
>========== ReceiverCPP.cpp: Receiver program in Visual C++ 2008 that
>demonstrates bug in VC++ 2008 runtime / STL ==========
>
>#include <iostream>
>#include <string>
>using namespace std;
>int main() {
>	/* you have to use a retry loop, for exact same reasons given for C#
>receiver program:
>	there is no way to tell difference between end-of-file and null
>write. */
>	for (int i = 0; i < 10; i++) {
>		string str;
>		/* BUG: cin will indicate end-of-file on a null write. */
>		getline(cin, str);
>		if (cin.eof()) {
>			cout << "Got end-of-file" << endl;
>		} else {
>			cout << "Got line " << str << endl;
>		}
>		cin.clear(); /* future getline calls will always immediately
>fail without attempting another read unless we clear EOF/fail flags */
>	}
>	return 0;
>}
>
>========== Test results ==========
>
>The test programs are designed so that they can be run in any combination
>from the command prompt.  The output from a sender is piped to the input of
>a receiver.  Each combination delivers identical output to the other
>combinations:
>
> * ./SenderCS | ./ReceiverCS
> * ./SenderCS | ./ReceiverCPP
> * ./SenderC | ./ReceiverCS
> * ./SenderC | ./ReceiverCPP
>
>Output from Cygwin will always be:
>
>Got end-of-file
>Got line Hello world!
>Got end-of-file
>Got end-of-file
><snip>

Nope, it won't always be that because I get what's expected.  I built
the C++ files using mingw g++.  Although I actually expected the reader
to honor the null byte, it did not.  Perhaps you are using a different
version of Windows than I am or a different runtime.

What you are seeing may be because Cygwin was changed to use
message-type pipes a couple of revisions ago.  This is not going to
change.  The change was adopted to fix a problem with Cygwin programs
and those are obviously our #1 priority.

>This is wrong, because the program received end-of-file before it was
>actually at the end of the input stream, due to the bug in its runtime's
>handling of return values from ReadFile API.  I did not do any tests using
>standard error, but I assume Cygwin redirects standard error in the same way
>it redirects standard output, in which case it would have the same problem.
>
>[moved]
>
>I think a workable fix would be for Cygwin not to pass through null writes
>it receives on an output/error pipe.  For example, somewhere in Cygwin I
>assume there is a loop that calls ReadFile to read the redirected standard
>output from the first program, and then calls WriteFile to send this output
>to the second program's standard input.  If the call to WriteFile was
>skipped if it would write zero bytes (i.e. so Cygwin doesn't do null writes
>itself), I think it would fix the problem and work around all these buggy
>runtimes.

There's no way that Cygwin could know to "skip" a call to WriteFile().
Cygwin doesn't interpose itself in the middle of a pipe.  That would be
truly disastrous.  If it somehow looked at every pipe read/write rather
than just allowing I/O to flow from one end to the other, the mailing
list would be even more filled with people complaining that Cygwin is
slow.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[cygwin]

On 26 April 2012 22:18, James Johnston <JamesJ@motionview3d.com> wrote:
> I have "run into an issue with Cygwin".  This is arguably not a bug in
> Cygwin, but in other platform runtime libraries.  Never-the-less, the
> symptoms occur only with Cygwin and not the Windows command prompt.  So,
> from a practical standpoint, Cygwin is "broken."  This is almost certainly
> the same issue mentioned in the "1.7.10->1.7.13 : output from .NET programs
> does not get through pipeline to a visual c++ program" thread started by
> cygwin AT alanhowells.e4ward.com last week.  It's also related to the issue
> I raised over a month ago, titled "Can't reliably redirect standard output
> from C# program in recent Cygwin".

Thank you! It is good to see someone else reproduce this!

<snip>

> I am providing several sample programs that can be compiled in Visual C#
> 2008 / .NET Framework 3.5, or Visual C++ 2008.  They should demonstrate the
> problem 100% of the time on any system, because appropriate Sleep() calls
> are made.

Of course, why didn't I think of using Sleep??

When adding sleep to my test programs, the test where it failed on the
first attempt reliably fails.

I think I hinted at an issue with the VC++ runtime as I proved that
using the Win32 API calls succeeded, but unfortunately didn't follow
that through.

I tested with VC++ 2010 runtime and that still has the issue.

<snip>

Christopher Faylor says
> Nope, it won't always be that because I get what's expected.  I built
> the C++ files using mingw g++.  Although I actually expected the reader
> to honor the null byte, it did not.  Perhaps you are using a different
> version of Windows than I am or a different runtime.

But the point by James is that it is the VC++ runtimes (and .NET
runtimes) that think that a null write is EOF. Other runtimes may not
(e.g. mingw).

I agree with James that the runtimes are probably buggy BUT I also
agree that cygwin should be able to have a handle these scenarios.

I have also found a workaround, by piping through a cygwin command,
you lose the null writes, a good one to use is cat so:

$ ./SenderC | cat | ./ReceiverCPP
$ Got line Hello world!
$ Got end-of-file
<snip>

Unfortunately, this isn't a real workaround unless you remember to use
it all the time.


Alan

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[James Johnston]

> -----Original Message-----
> Sent: Friday, April 27, 2012 00:17
> Subject: Re: Cygwin passes through null writes to other software when
> redirecting standard input/output (i.e. piping)
> 
> Nope, it won't always be that because I get what's expected.  I built the
C++
> files using mingw g++.  Although I actually expected the reader to honor
the
> null byte, it did not.  Perhaps you are using a different version of
Windows
> than I am or a different runtime.

As I stated, I did two demo programs using Visual C++ 2008 and .NET
Framework 3.5.  I did not test mingw.  This is on Windows 7 64-bit, fully
up-to-date.  Perhaps your mingw runtime doesn't have this bug, in which
case, good for them!  But that doesn't change the fact that not many people
use mingw in comparison to Visual C++ or .NET Framework 3.5.  And I'm not
sure how you can complain that you can't reproduce it if you don't use
similar tools to what was used to originally reproduce the issue.  (Did I
really need to preface my phrase of "will always be" with conditions that
you use Visual C++ / .NET Framework?  I thought it would be obvious,
especially since I was referring to bugs in their runtimes...) 

If you don't want to install a full Visual Studio environment, you can use
the C# compiler on any computer with the .NET Framework installed.  For
example, framework 3.5's compiler is at
C:\Windows\Microsoft.NET\Framework\v3.5\csc.exe.  The free Windows SDK
contains a Visual C++ compiler in it.  And there are free express versions
of Visual Studio available if you want an IDE.  You can test these sorts of
things without forking over big bucks for paid Visual Studio licenses.

> What you are seeing may be because Cygwin was changed to use message-
> type pipes a couple of revisions ago.  This is not going to change.  The
change
> was adopted to fix a problem with Cygwin programs and those are obviously
> our #1 priority.

In which case, it would be a shame for Cygwin to break compatibility with
the vast majority of Windows programs out there, which are probably using
the two most common development platforms on Windows: Visual C++ and the
.NET Framework.  That sounds insane; I can't believe the developers would
make that call.  It seems crazy that they would not even regularly test with
software compiled using non-free compilers.  (May as well rip out the
cygpath tool and friends; why make pretenses about supporting
interoperability with Windows programs?)

I liked Cygwin since I thought it offered compatibility with software
designed for both Linux and Windows.  Apparently the fine print is that it's
now only supporting compatibility with stuff linked to cygwin1.dll, and
standard Windows software need not apply - since compatibility with Windows
software is not a real goal?  Unfortunately, now I can't think of any good
solutions that support both Linux and Windows software, whose developers
place a priority on compatibility with both.  If Cygwin doesn't support
Windows software compiled using very common platforms like Visual C++/.NET,
then nobody really does that I can think of.  (Sure, there is Windows
Services for UNIX, but we all know the mess that brings - we're using Cygwin
for a reason...  And MSYS isn't really a replacement for Cygwin either; they
don't pretend to be "UNIX on Windows".)

More to the point, I didn't have these problems with Cygwin 1.7.9.  What
"problem with Cygwin programs" are you referring to?  Because I never had
any - 1.7.9 worked great.  All I know, is I upgraded to a more recent
version and boom - everything stopped working.  Why were message pipes used?
And what source code files exactly are you talking about?  Cygwin has a lot
of source code, but if I knew exactly where to look or better understood
what it is doing and why, maybe I could offer some more constructive ideas -
having done some low-level programming with pipes on Windows myself.  (At
first glance, I don't really see how message pipes vs byte pipes would make
a difference.)

And from a practical standpoint - again, while ideally it would be nice for
Microsoft to go back and fix every single version of Visual C++ and .NET
Framework that have this bug (which probably dates back to when these
products were first created), and for Windows app developers to then
redeploy their apps with these fixed runtimes - I think the chances of that
happening in the next few years are zero.  I'm a practical/pragmatic person,
which is why I suggested working around their bugs.  (As a regular reader of
Raymond Chen's blog, apparently Windows works around app bugs, too...)
	
> There's no way that Cygwin could know to "skip" a call to WriteFile().
> Cygwin doesn't interpose itself in the middle of a pipe.  That would be
truly
> disastrous.  If it somehow looked at every pipe read/write rather than
just
> allowing I/O to flow from one end to the other, the mailing list would be
> even more filled with people complaining that Cygwin is slow.

If there is no other way, then maybe it should.  Better to be functional and
sacrifice a couple milliseconds, rather than be fast and broken.  Given that
I've never seen pipes in the Windows command prompt or Cygwin 1.7.9 run
"disastrously slow", I'm sure there is a solution.

James


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Corinna Vinschen]

On Apr 26 21:18, James Johnston wrote:
> I have "run into an issue with Cygwin".  This is arguably not a bug in
> Cygwin, but in other platform runtime libraries.  Never-the-less, the
> symptoms occur only with Cygwin and not the Windows command prompt.  So,
> from a practical standpoint, Cygwin is "broken."  This is almost certainly
> the same issue mentioned in the "1.7.10->1.7.13 : output from .NET programs
> does not get through pipeline to a visual c++ program" thread started by
> cygwin AT alanhowells.e4ward.com last week.  It's also related to the issue
> I raised over a month ago, titled "Can't reliably redirect standard output
> from C# program in recent Cygwin".
> 
> To summarize: when piping output from one program to another (i.e. running
> "A | B" command to pipe standard output from A to B's standard input),
> Cygwin passes through ALL writes performed by a program to standard output
> (and probably error), including null writes.

What I don't grok is this:

  In your example, A and B are both native (== non-Cygwin) applications.

As I already mentioned in another mail related to the .Net problem,
while the Cygwin shell running at the time creates the pipe, it doesn't
handle read and write calls on the pipe.  It only creates the two native
child processes and passes on the pipe handles.

So in the end, there is no Cygwin process involved while the native processes
are doing their reads and writes.  It's all in the hands of these processes.
There's no third process in the middle which handles the passing of information
from A to B.

So, what do you expect Cygwin to do?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Christopher Faylor]

On Fri, Apr 27, 2012 at 10:08:23AM +0100, cygwin-alanhowells.e4ward.com wrote:
>Christopher Faylor says
>> Nope, it won't always be that because I get what's expected.  I built
>> the C++ files using mingw g++.  Although I actually expected the reader
>> to honor the null byte, it did not.  Perhaps you are using a different
>> version of Windows than I am or a different runtime.
>
>But the point by James is that it is the VC++ runtimes (and .NET
>runtimes) that think that a null write is EOF. Other runtimes may not
>(e.g. mingw).

I was reporting what I found with the tools I have at hand.  My
experience refutes the use of the word "always".  The whole point
of my mentioning "a different runtime" was to show that I understood
that others' mileage varies.  I wasn't saying that I don't believe
what's being reported.

>I agree with James that the runtimes are probably buggy BUT I also
>agree that cygwin should be able to have a handle these scenarios.

Your complete agreement with each other is not going to have much
effect.  Cygwin source code is not changed by voting.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[jojelino]

On 2012-04-27 AM 9:17, Christopher Faylor wrote:
> There's no way that Cygwin could know to "skip" a call to WriteFile().
> Cygwin doesn't interpose itself in the middle of a pipe.  That would be
> truly disastrous.  If it somehow looked at every pipe read/write rather
> than just allowing I/O to flow from one end to the other, the mailing
> list would be even more filled with people complaining that Cygwin is
> slow.
>
Maybe i can measure how much it slowed down after applying the 
workaround of unworkable runtime of some non-free software.
-- 
Regards.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Christopher Faylor]

On Sat, Apr 28, 2012 at 12:11:35AM +0900, jojelino wrote:
>On 2012-04-27 AM 9:17, Christopher Faylor wrote:
>> There's no way that Cygwin could know to "skip" a call to WriteFile().
>> Cygwin doesn't interpose itself in the middle of a pipe.  That would be
>> truly disastrous.  If it somehow looked at every pipe read/write rather
>> than just allowing I/O to flow from one end to the other, the mailing
>> list would be even more filled with people complaining that Cygwin is
>> slow.
>>
>Maybe i can measure how much it slowed down after applying the 
>workaround of unworkable runtime of some non-free software.

Thanks for the offer but that presupposes that I'd actually be willing
to go to the effort of coding something up.  I'm really not.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Christopher Faylor]

On Fri, Apr 27, 2012 at 02:26:13PM -0000, James Johnston wrote:
>>On Thu, Apr 26, 2012 at 08:17:18PM -0400, Christopher Faylor wrote:
>>Nope, it won't always be that because I get what's expected.  I built
>>the C++ files using mingw g++.  Although I actually expected the reader
>>to honor the null byte, it did not.  Perhaps you are using a different
>>version of Windows than I am or a different runtime.
>
>As I stated, I did two demo programs using Visual C++ 2008 and .NET
>Framework 3.5.  I did not test mingw.

I did not say that you tested MinGW.  I was telling you what I tested.
However, as it turns out, I created the executable on Linux and my
environment still ended up creating a Cygwin binary.  Correctly
rebuilding with MinGW shows the same behavior you reported.  Sorry
for getting that wrong.

Btw, MinGW does not have its own runtime.

>This is on Windows 7 64-bit, fully up-to-date.  Perhaps your mingw
>runtime doesn't have this bug, in which case, good for them! But that
>doesn't change the fact that not many people use mingw in comparison to
>Visual C++ or .NET Framework 3.5.  And I'm not sure how you can
>complain that you can't reproduce it if you don't use similar tools to
>what was used to originally reproduce the issue.  (Did I really need to
>preface my phrase of "will always be" with conditions that you use
>Visual C++ / .NET Framework?  I thought it would be obvious, especially
>since I was referring to bugs in their runtimes...)

I wasn't "complaining" that I could not reproduce the problem.  I was
stating a fact.  And, again, to be clear, I was also not doubting that
you could demonstrate a behavior.  I can see it now too and I'm actually
kind of relieved to see that it matches my expectations.

Regardless of how many people use Visual C++ or .NET they are really not
our target audience.  It's nice when things work for people who don't
want to use UNIX tools but they are not our primary focus.  Fixing
problems for people who want to use non-Cygwin stuff is not something
that I find a high priority.

If you want to a fix for this problem then you'd be best off if you
supplied one yourself:

http://cygwin.com/acronyms/#PTC

Maybe I'm in the minority but I find long-winded impassioned pleas
rather boring and off-putting.  So, continuing in this vein is not going
to be productive for you, at least as far as I'm concerned.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[James Johnston]

> -----Original Message-----
> Sent: Friday, April 27, 2012 14:38
> Subject: Re: Cygwin passes through null writes to other software when
> redirecting standard input/output (i.e. piping)
> 
> What I don't grok is this:
> 
>   In your example, A and B are both native (== non-Cygwin) applications.
> 
> As I already mentioned in another mail related to the .Net problem, while
> the Cygwin shell running at the time creates the pipe, it doesn't handle read
> and write calls on the pipe.  It only creates the two native child processes and
> passes on the pipe handles.
> 
> So in the end, there is no Cygwin process involved while the native processes
> are doing their reads and writes.  It's all in the hands of these processes.
> There's no third process in the middle which handles the passing of
> information from A to B.
> 
> So, what do you expect Cygwin to do?

Thanks for responding.  I apologize for not fully understanding the problem earlier.  In .NET, if you wanted to pipe the output of one process to another, you'd have to be exactly this third process as far as I can tell - the framework creates a pipe/stream for each process, so you'd need a loop to copy information from one pipe to another.

To help understand how this is all working, I used API Monitor to spy and see how the Windows command prompt sets up its pipe.  It's very simple: it can be simplified down to a call to CreatePipe to make an anonymous pipe.  The two child processes are then spawned, with the pipe reader/writer handles being passed to CreateProcess.  Unused pipe handles are closed in both the parent process and the child processes, so that the processes terminate correctly.  I wrote a straight Win32 C program to spawn my child processes and direct output from one to another's input, using CreatePipe.  The results were correct.

> What you are seeing may be because Cygwin was changed to use
> message-type pipes a couple of revisions ago.  This is not going to
> change.  The change was adopted to fix a problem with Cygwin
> programs and those are obviously our #1 priority.

I will certainly not disagree with you that Cygwin programs need to work 100%!  But surely there has got to be some solution that can achieve this while not widely breaking non-Cygwin programs..!?  I'm actually surprised more people wouldn't  be using Windows programs from Cygwin - to me, that is part of the point of using a system like Cygwin rather than just installing the Linux distro of choice - so I can mix/match both Windows and UNIX software.  For me, both Windows and Cygwin software need to work - not just one or the other.  While you have near-zero interest in running Windows programs from Cygwin - I would bet that does not describe a significant number of users.

So to confirm your hypothesis, I modified my previously-described C program.  It uses CreateNamedPipe followed by CreateFile to get reader/writer handles instead of making an anonymous pipe using CreatePipe.  If I use byte mode, the results are still correct - apparently byte mode pipes won't pass through null writes.  HOWEVER, if I switch to message mode - the results are incorrect - just like the bad results from Cygwin.  Message mode pipes pass through null writes.

So I think for sure, Cygwin's use of message pipes is breaking a lot of Windows software, because of the null writes.  And ALSO additionally perhaps because of this: while reading MSDN today, I came across an interesting snippet that probably indicates a second source of problems I have not explicitly tested for, but would probably find:

"If the specified number of bytes to read is less than the size of the next message, the function reads as much of the message as possible before returning zero (the GetLastError function returns ERROR_MORE_DATA). The remainder of the message can be read using another read operation."

While I did not test for this, I think this would also break a lot of Windows software.  It's conceivable that the sender could write out a lot of data, while the receiver isn't reading anything.  Then, if the reader tries to read but with a small buffer, according to the docs it will get ERROR_MORE_DATA.  Most runtimes don’t expect ReadFile to fail with ERROR_MORE_DATA.  For example, the decompiled .NET Framework code I read would throw an exception, as it only gracefully handles ERROR_BROKEN_PIPE.  Probably others like the Visual C++ runtime will treat it as EOF or set the failbit.  If there's interest I expect I could write a test case. (e.g. run "cat" to pipe a gigantic text file to a Windows program that takes a "long" time to process each line).

So I have some questions...

1.  Why on earth did Cygwin start using message pipes?  "adopted to fix a problem" doesn't help... Googling site:Cygwin.com "message pipe" - didn't yield anything either.  I do know older Cygwin versions worked fine, and I had no problem with piping data to/from both Cygwin and non-Cygwin programs.  I am sure it was changed for a good reason, but would someone be willing to offer an explanation for why message pipes are now used?  Do they have problems reading from a byte pipe?  Writing to a byte pipe?  Both?  Under what conditions?

2.  Is it possible that byte pipes could be used to communicate with non-Cygwin programs, while message pipes used to communicate between Cygwin programs?  Since I don't know the answer to the previous question, I have no idea how this question could be answered without causing / moving the breakage somewhere else. 

Here's sample code that can demonstrate both byte and message pipes.  Edit the EXE names in the calls to StartProc() accordingly.

#include <windows.h>
#include <iostream>
using namespace std;
/* error handling left out for brevity */

/* start process with specified input/output handles and return process handle */
HANDLE StartProc(wchar_t *proc, HANDLE in, HANDLE out)
{
	STARTUPINFO si;
	PROCESS_INFORMATION pi;
	memset(&si, 0, sizeof(si));
	memset(&pi, 0, sizeof(pi));

	si.cb = sizeof(si);
	si.dwFlags = STARTF_USESTDHANDLES;
	si.hStdInput = in;
	si.hStdOutput = out;
	si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
	CreateProcess(proc, NULL, NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi);
	CloseHandle(pi.hThread);
	return pi.hProcess;
}

int main()
{
	HANDLE hPipeRead, hPipeWrite;
	SECURITY_ATTRIBUTES secat;
	memset(&secat, 0, sizeof(secat));
	secat.nLength = sizeof(secat);
	secat.bInheritHandle = TRUE;
	/* note: changing to PIPE_TYPE_BYTE will fix the problems */
	hPipeWrite = CreateNamedPipe(L"\\\\.\\pipe\\CoordTestPipe", PIPE_ACCESS_OUTBOUND, PIPE_TYPE_MESSAGE, 1, 0, 0, 0, &secat);
	/* FILE_WRITE_ATTRIBUTES required if you play with SetNamedPipeHandleState */
	hPipeRead = CreateFile(L"\\\\.\\pipe\\CoordTestPipe", GENERIC_READ | FILE_WRITE_ATTRIBUTES, 0, &secat, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
	DWORD mode = PIPE_READMODE_BYTE;
	/*SetNamedPipeHandleState(reader, &mode, NULL, NULL); // this won't fix anything for the poor reader program */

	HANDLE hSender, hReceiver;
	hSender = StartProc(L"Sender.exe", GetStdHandle(STD_INPUT_HANDLE), hPipeWrite);
	hReceiver = StartProc(L"Receiver.exe", hPipeRead, GetStdHandle(STD_OUTPUT_HANDLE));
	/* close pipe handles that aren't needed; processes won't terminate otherwise */
	DuplicateHandle(hSender, hPipeRead, NULL, NULL, 0, FALSE, DUPLICATE_CLOSE_SOURCE);
	DuplicateHandle(hReceiver, hPipeWrite, NULL, NULL, 0, FALSE, DUPLICATE_CLOSE_SOURCE);
	CloseHandle(hPipeRead);
	CloseHandle(hPipeWrite);
	/* wait for child processes to exit */
	WaitForSingleObject(hSender, INFINITE);
	WaitForSingleObject(hReceiver, INFINITE);
	CloseHandle(hSender);
	CloseHandle(hReceiver);
	return 0;
}

> Your complete agreement with each other is not going to have much
> effect.  Cygwin source code is not changed by voting.

OK, fine.  What would you have us do then?  Not provide any seemingly-bothersome input / opinions at all, if the maintainers have no interest in what the users might run into?  I realize you are all very busy people, and that the Cygwin project doesn't have the major backing that larger projects do.  Fixing lower priority bugs might not be possible on your own.   But do you even want to hear feedback and bug reports if they are not accompanied by a perfectly-written patch, or are they too bothersome?  Are the maintainers interested in bug reports, or are they not?  I have already spent a fair bit of time researching this issue, and it's not the first Cygwin issue I've encountered.  I'm going to this trouble because I was hoping that in some way, Cygwin can ultimately be improved - if not by my hands then perhaps by the hands of someone else who might be reading this list and have more ideas or skills!

Write my own patch?  OK, but... I get the strong feeling that: (1) I won't have anyone to ask regarding where in the code I might need to even start looking and would be willing to answer any questions, and any considerations I might need to take into account - I have never actually worked on the Cygwin code before - and there is a lot of it - and I don't want to "cowboy" my way through it and make more problems than I fix! (2) I'm afraid you'd summarily ignore any patch I might come up with, without providing real, helpful feedback on what would need to changed and improved in the patch for it to be accepted.  Reading over the Cygwin contribution guidelines indicates there are an awful lot of things I have to get perfectly right, and frankly I am not sure I am going to get it perfectly right.  I don't want to waste more hours trying to patch this thing if there's no real chance of acceptance.

> Maybe I'm in the minority but I find long-winded impassioned pleas
> rather boring and off-putting.  So, continuing in this vein is not going
> to be productive for you, at least as far as I'm concerned.

Oh good grief... Most of the length in my e-mails have been due to me attempting to do more detailed research than the average person might do.  Would I be better off writing short e-mails that say "Cygwin doesn’t work, why?"- without providing more research?

James 


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Daniel Colascione]

On 4/27/2012 3:38 PM, James Johnston wrote:
> So I think for sure, Cygwin's use of message pipes is breaking a lot
> of Windows software, because of the null writes. And ALSO additionally
> perhaps because of this: while reading MSDN today, I came across an
> interesting snippet that probably indicates a second source of problems
> I have not explicitly tested for, but would probably find:
> 
> "If the specified number of bytes to read is less than the size of
> the
> next message, the function reads as much of the message as possible
> before returning zero (the GetLastError function returns
> ERROR_MORE_DATA). The remainder of the message can be read using another
> read operation."

Confusingly, a PIPE_TYPE_MESSAGE pipe's read end can be in either
PIPE_READMODE_MESSAGE or PIPE_READMODE_BYTE mode. You're describing the
behavior of PIPE_READMODE_MESSAGE; Cygwin's pipes are in
PIPE_READMODE_BYTE, so they don't have this particular problem ---
although the null-write problem remains.


> 1. Why on earth did Cygwin start using message pipes? "adopted to
> fix
> a problem" doesn't help... Googling site:Cygwin.com "message pipe" -
> didn't yield anything either. I do know older Cygwin versions worked
> fine, and I had no problem with piping data to/from both Cygwin and
> non-Cygwin programs. I am sure it was changed for a good reason, but
> would someone be willing to offer an explanation for why message pipes
> are now used? Do they have problems reading from a byte pipe? Writing to
> a byte pipe? Both? Under what conditions?

IIRC, it had something to do with preserving message boundaries at the
TTY later. Again IIRC (and I only faintly recall this part) the PTY
layer has to deliver one line at a time to readers. If a client calls
read(2) with a giant buffer, he's still supposed to get back only a
single line at a time. A message-mode pipe seems like a good way to
provide these semantics because it preserves write boundaries --- so if
you write only a line at a time, the reader reads only a line at a time.
It's a shame that this approach causes problems.

Why can't only pseudoterminal pipes be created as PIPE_TYPE_MESSAGE?
Cygwin already has special logic to tell whether a handle it's opening
is a pty and to treat it specially in that case.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Christopher Faylor]

On Fri, Apr 27, 2012 at 05:25:31PM -0700, Daniel Colascione wrote:
>IIRC, it had something to do with preserving message boundaries at the
>TTY later. Again IIRC (and I only faintly recall this part) the PTY
>layer has to deliver one line at a time to readers. If a client calls
>read(2) with a giant buffer, he's still supposed to get back only a
>single line at a time. A message-mode pipe seems like a good way to
>provide these semantics because it preserves write boundaries --- so if
>you write only a line at a time, the reader reads only a line at a time.
>It's a shame that this approach causes problems.
>
>Why can't only pseudoterminal pipes be created as PIPE_TYPE_MESSAGE?
>Cygwin already has special logic to tell whether a handle it's opening
>is a pty and to treat it specially in that case.

Use the source code, Jake.  From pipe.cc:

 Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
 This *seems* to more closely mimic Linux pipe behavior and is
 definitely required for pty handling since fhandler_pty_master
 writes to the pipe in chunks, terminated by newline when CANON mode
 is specified.  

The above comment shows an "and" relationship here.  Message type pipes
more closely mimic Linux (UNIX) pipe behavior AND are definitely
required for ptys.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

On 4/27/12 10:27 PM, Christopher Faylor wrote:
> The above comment shows an "and" relationship here.  Message type pipes
> more closely mimic Linux (UNIX) pipe behavior AND are definitely
> required for ptys.

Yeah, but because message pipes break other programs. Cygwin has only
been using message pipes since 23-Oct-11 (1.134), and it got along
fine before that, ICANON aside. If Cygwin programs don't actually
_need_ message pipes in the general case, Cygwin should use byte pipes
instead.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 235 bytes --]

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Linda Walsh]

Daniel Colascione wrote:

> On 4/27/12 10:27 PM, Christopher Faylor wrote:
>> The above comment shows an "and" relationship here.  Message type pipes
>> more closely mimic Linux (UNIX) pipe behavior AND are definitely
>> required for ptys.
> 
> Yeah, but because message pipes break other programs. Cygwin has only
> been using message pipes since 23-Oct-11 (1.134), and it got along
> fine before that, ICANON aside. If Cygwin programs don't actually
> _need_ message pipes in the general case, Cygwin should use byte pipes
> instead.

----
But Daniel -- if the point is to make cygwin behave like *nix, wouldn't it be 
proper to use the method that most closely emulates it's behavior?


I can't say with 100% certainty, but I would bet with 90+% confidence that this 
is a bug in MS's libraries -- they "cheat" and use a null/0 byte read as a 
signal for end of file rather than sending out-of-band information that *nix
supports.

A suggestion for a workaround:

Write a simple C program as a pipe for stdin -- that buffers
the I/O by 'line' (or xx bytes).  Then you can have it NOT do writes
of 0 bytes to MS progs.

You could even do it as a wrapper

avoidmsbug "normal progline with it's arguments"

avoid would open the appropriate pipes and do the buffering to
any buggy MS lib...

Essentially you need a 'shim' layer between a POSIX compliant subsystem
to NON-POSIX compliant programs.

I'm sure that in the case that my assumptions are true, you wouldn't want to 
deliberate put something in cygwin that would make it less POSIX compliant when 
it is only to support external-to-cygwin, NON-POSIX compliant programs...

Note -- I use programs between cygwin and Windows 'alot', so I want these things 
to work as well.

Another approach in using the wrapper...  Just have your
real program be named "xxx".bin.exe, and let "xxx" be a link to 
"avoidmsbug.exe".  That prog could look at it's name, automatically append the 
".bin" to it's name to execute it...

Then you could have any number of cygwin programs with transparent wrappers
around them designed to deal with calling programs that mishandle pipe I/O...?


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[James Johnston]

> -----Original Message-----
> Sent: Wednesday, May 09, 2012 19:00
> Subject: Re: Cygwin passes through null writes to other software when
> redirecting standard input/output (i.e. piping)
>
> I can't say with 100% certainty, but I would bet with 90+% confidence that
> this
> is a bug in MS's libraries -- they "cheat" and use a null/0 byte read as a
> signal for end of file rather than sending out-of-band information that *nix
> supports.

From examining .NET Framework source, it seems clear to me they did not plan on message pipe inputs.  Win32 ReadFile API normally uses a zero byte read to indicate end-of-file; I think it's not cheating.  Pipes present a special case.  I'm not sure how MS's runtime libraries could send out-of-band information; the operating system does not really support this as far as I know?

Of course, these libraries are very widely used... 

> Essentially you need a 'shim' layer between a POSIX compliant subsystem
> to NON-POSIX compliant programs.
> 
> I'm sure that in the case that my assumptions are true, you wouldn't want to
> deliberate put something in cygwin that would make it less POSIX compliant
> when
> it is only to support external-to-cygwin, NON-POSIX compliant programs...
> 
> Note -- I use programs between cygwin and Windows 'alot', so I want these
> things
> to work as well.

Maybe I'm naïve and this is harder than it looks, but couldn't Cygwin determine if the program being piped to links with CYGWIN1.DLL (or similar detection technique), and then use byte or message pipes accordingly?  Example:

# Byte pipe used because Win32Program.exe does not link with CYGWIN1.DLLL
cat Testfile.txt | ./Win32Program.exe
# Message pipe used because grep links with CYGWIN1.DLL
cat Testfile.txt | grep Hello
# Message pipe still used because the program we are calling links with CYGWIN1.DLL
./Win32Program.exe | grep Hello

If that is still not POSIX compliant enough for the sending program (e.g. "cat" in this example), then I suppose Cygwin could automatically insert a message-to-byte pipe shim like the author proposes.  That seems pretty kludgy though.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Christopher Faylor]

On Wed, May 09, 2012 at 07:21:34PM -0000, James Johnston wrote:
>On Wed, May 09, 2012 at 11:59:58AM -0700, Linda Walsh, the dead discussion resurrector wrote:
>>I can't say with 100% certainty, but I would bet with 90+% confidence
>>that this is a bug in MS's libraries -- they "cheat" and use a null/0
>>byte read as a signal for end of file rather than sending out-of-band
>>information that *nix supports.
>
>Maybe I'm na??ve and this is harder than it looks, but couldn't Cygwin
>determine if the program being piped to links with CYGWIN1.DLL (or
>similar detection technique), and then use byte or message pipes
>accordingly?  Example:

You're being naive.  Of course this is harder than it looks.  Or, if it
really is so easy, why haven't you submitted a patch or at least a
proof of concept?

># Byte pipe used because Win32Program.exe does not link with CYGWIN1.DLLL
>cat Testfile.txt | ./Win32Program.exe

In the above, the series of events are likely something like:

1) Shell creates a pipe

2) Shell starts cat with its stdout set to the output side of the pipe, first
closing the input side.

3) Shell starts ./Win32Program.exe with its input set to the input side of
the pipe, first closing the output side.

4) Shell closes pipe.

5) Processes execute.

The cygwin DLL can't intercede here.  It's not some superior process
controlling I/O.  It's just a DLL used by two programs.  cat is writing
to the stdout that it inherited from the shell.  It doesn't know (and
shouldn't know) what's on the other end of the pipe and that it should
be changing the mode of the pipe.

Please stop thinking that there is a simple solution that you can
intuit without fully understanding how Cygwin works.

1.7.15 will contain a new CYGWIN option "pipe_byte" which will force
non-tty, non-fifo pipes to use byte mode.  That's about as far as we're
willing to modify Cygwin for this apocalyptic problem.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[James Johnston]

> The cygwin DLL can't intercede here.  It's not some superior process
> controlling I/O.  It's just a DLL used by two programs.  cat is writing to
the
> stdout that it inherited from the shell.

Good point, I had not thought this through enough.  In this case, bash.exe
is asking CYGWIN1.DLL for a pipe, and then setting up the pipe between the
two processes exactly as you outline.  Again maybe I'm naive, but to do what
I mention:

1.  Cygwin would need an additional function with a parameter to specify
byte pipes (maybe you already added this, I have not checked the recent
source code very much).
2.  Bash and other shells would need a platform-specific patch to do the
checks I described: if piping to a non-Cygwin program, then use the special
Cygwin function for making byte pipes.

But that would require patching upstream shell with a platform-specific
change.  Any other code in packages that might pipe data to a non-Cygwin
process would need patching too (although, I cannot at the moment think of
anything besides a shell).  The bright side is that it's a local solution to
a local problem - no global variable that might adversely affect other code.

> [cat] doesn't know (and shouldn't know)
> what's on the other end of the pipe and that it should be changing the
mode
> of the pipe.

Agree 100%; the previous message wasn't meant to imply otherwise.

> 1.7.15 will contain a new CYGWIN option "pipe_byte" which will force non-
> tty, non-fifo pipes to use byte mode.  That's about as far as we're
willing to
> modify Cygwin for this apocalyptic problem. 

Thanks - I guess that will address the immediate issue in practice, although
it's a global solution to a local problem - though I don't see a way around
it without patching the shell itself.

Forgive me for asking perhaps a dumb question, but I was reading the page at
http://cygwin.com/cygwin-ug-net/setup-env.html and wasn't too sure what was
meant by the sentence "All Windows environment variables are imported when
Cygwin starts" - which I assume includes parsing and using CYGWIN option?
Is it: (1) the variables are processed the first time CYGWIN1.DLL is loaded
into a process, and then never used again thereafter, or (2) every time a
process linking with CYGWIN1.DLL loads - even if CYGWIN1.DLL is concurrently
loaded by someone else?  I would assume it is #2....

Or to phrase another way: does Cygwin "start" once when the first process
linking to it is loaded, or does it "start" every time a process loads?  (I
realize Cygwin has extensive IPC and shared memory mechanisms, some of which
is probably done only on first load of first process to initialize shared
memory, which is what prompted this question.)


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Otto Meta]

Hi James,

> 1.  Cygwin would need an additional function with a parameter to specify
> byte pipes (maybe you already added this, I have not checked the recent
> source code very much).

You just missed the announcement:

On 2012-05-10 08:28, Corinna Vinschen wrote:
[snip]
> I just released 1.7.15.  This is a bugfix release.  Only one new feature
> has been added.
[snip]
> - Add CYGWIN=pipe_byte option to force opening of pipes in byte mode
>   rather than message mode.

That's probably just for you.

Cheers,
Otto

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Earnie Boyd]

On Thu, May 10, 2012 at 12:28 PM, James Johnston wrote:
> Forgive me for asking perhaps a dumb question, but I was reading the page at
> http://cygwin.com/cygwin-ug-net/setup-env.html and wasn't too sure what was
> meant by the sentence "All Windows environment variables are imported when
> Cygwin starts" - which I assume includes parsing and using CYGWIN option?
> Is it: (1) the variables are processed the first time CYGWIN1.DLL is loaded
> into a process, and then never used again thereafter, or (2) every time a
> process linking with CYGWIN1.DLL loads - even if CYGWIN1.DLL is concurrently
> loaded by someone else?  I would assume it is #2....
>
> Or to phrase another way: does Cygwin "start" once when the first process
> linking to it is loaded, or does it "start" every time a process loads?  (I
> realize Cygwin has extensive IPC and shared memory mechanisms, some of which
> is probably done only on first load of first process to initialize shared
> memory, which is what prompted this question.)

IIRC Cygwin processes the Windows environment upon initial parent
process load and stores it internally for each parent process.  Child
processes inherit the environment of the parent.  So if I start to
shell command windows both look the same on start-up but if I modify
one of the environments in one command window the other is unaffected.
 If I start a process in the changed environment then the subsequent
process will have the changed environment.

HTH,
-- 
Earnie
-- https://sites.google.com/site/earnieboyd

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Corinna Vinschen]

On May 10 13:57, Earnie Boyd wrote:
> On Thu, May 10, 2012 at 12:28 PM, James Johnston wrote:
> > Forgive me for asking perhaps a dumb question, but I was reading the page at
> > http://cygwin.com/cygwin-ug-net/setup-env.html and wasn't too sure what was
> > meant by the sentence "All Windows environment variables are imported when
> > Cygwin starts" - which I assume includes parsing and using CYGWIN option?
> > Is it: (1) the variables are processed the first time CYGWIN1.DLL is loaded
> > into a process, and then never used again thereafter, or (2) every time a
> > process linking with CYGWIN1.DLL loads - even if CYGWIN1.DLL is concurrently
> > loaded by someone else?  I would assume it is #2....
> >
> > Or to phrase another way: does Cygwin "start" once when the first process
> > linking to it is loaded, or does it "start" every time a process loads?  (I
> > realize Cygwin has extensive IPC and shared memory mechanisms, some of which
> > is probably done only on first load of first process to initialize shared
> > memory, which is what prompted this question.)
> 
> IIRC Cygwin processes the Windows environment upon initial parent
> process load and stores it internally for each parent process.  Child
> processes inherit the environment of the parent.  So if I start to
> shell command windows both look the same on start-up but if I modify
> one of the environments in one command window the other is unaffected.
>  If I start a process in the changed environment then the subsequent
> process will have the changed environment.

That has nothing to do with Cygwin.  This is just how the environment is
handled in general.  Child processes inherit it from their parent
processes, regardless whether on Windows or Unix.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

On 5/9/12 1:11 PM, Christopher Faylor wrote:
> Please stop thinking that there is a simple solution that you can
> intuit without fully understanding how Cygwin works.

Indeed. _Everything_ is more complicated than it looks.

> 
> 1.7.15 will contain a new CYGWIN option "pipe_byte" which will force
> non-tty, non-fifo pipes to use byte mode.  That's about as far as we're
> willing to modify Cygwin for this apocalyptic problem.

What's the downside to using pipe_byte?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 235 bytes --]

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Linda Walsh]

James Johnston wrote:


>> I can't say with 100% certainty, but I would bet with 90+% confidence that
>> this
>> is a bug in MS's libraries -- they "cheat" and use a null/0 byte read as a
>> signal for end of file rather than sending out-of-band information that *nix
>> supports.
> 
> From examining .NET Framework source, it seems clear to me they did not plan on message 

pipe inputs.  Win32 ReadFile API normally uses a zero byte read to indicate end-of-file; 

I think it's not cheating.  Pipes present a special case.  I'm not sure how MS's runtime 

libraries could send out-of-band information; the operating system does not really support

this as far as I know?

----
	It is cheating because they didn't follow the standard.  It is perfectly legal 
to write 0 bytes.


> 
> Of course, these libraries are very widely used... 

---
	MSWindows is widely used and is known to be one of the buggiest OS's on the 
planet.  So what's your point?


> 
>> Essentially you need a 'shim' layer between a POSIX compliant subsystem
>> to NON-POSIX compliant programs.
>>
>> I'm sure that in the case that my assumptions are true, you wouldn't want to
>> deliberate put something in cygwin that would make it less POSIX compliant
>> when
>> it is only to support external-to-cygwin, NON-POSIX compliant programs...
>>
>> Note -- I use programs between cygwin and Windows 'alot', so I want these
>> things
>> to work as well.
> 
> Maybe I'm naïve and this is harder than it looks, but couldn't Cygwin determine if 

the program being piped to links with CYGWIN1.DLL (or similar detection technique),

 and then use byte or message pipes accordingly?  Example:

---
	Because (1) it wouldn't be reliable.  2, cygwin could be delay-loaded so  again 
back to 1... and have you contacted Microsoft and asked them to fix the problem? 
   Who did you contact and what did they say?


> 
> # Byte pipe used because Win32Program.exe does not link with CYGWIN1.DLLL
> cat Testfile.txt | ./Win32Program.exe
> # Message pipe used because grep links with CYGWIN1.DLL
> cat Testfile.txt | grep Hello
> # Message pipe still used because the program we are calling links with CYGWIN1.DLL
> ./Win32Program.exe | grep Hello
> 
> If that is still not POSIX compliant enough for the sending program 

(e.g. "cat" in this example), then I suppose Cygwin could automatically insert a 

message-to-byte pipe shim like the author proposes.  That seems pretty kludgy though.

====
	Cygwin wouldn't insert a special shim for an app using a buggy library.  Or 
rather it wouldn't be the place of cygwin to do so.  It would be the 
responsibility of whoever needs to run some sort of adaptation with **THEIR** 
program... (and perhaps whoever they pay to do

I don't speak for the Cygwin project - the above is purely my point of view.

I was suggesting a way for you to create a workaround for your program


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

[Andrey Repin]

Greetings, Corinna Vinschen!

> That has nothing to do with Cygwin.  This is just how the environment is
> handled in general.  Child processes inherit it from their parent
> processes, regardless whether on Windows or Unix.

Windows have means of notifying running processes about changes in global
environment.
It isn't really reliable, though. Only works, when you're editing environment
from the GUI, AFAIK.


--
WBR,
Andrey Repin (anrdaemon@freemail.ru) 11.05.2012, <12:55>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple