From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 25835 invoked by alias); 27 Apr 2012 22:38:28 -0000 Received: (qmail 25522 invoked by uid 22791); 27 Apr 2012 22:38:25 -0000 X-SWARE-Spam-Status: No, hits=-1.0 required=5.0 tests=AWL,BAYES_00,KHOP_THREADED,RCVD_IN_DNSWL_NONE,RCVD_IN_HOSTKARMA_YE,RCVD_IN_SORBS_WEB,SPF_HELO_PASS X-Spam-Check-By: sourceware.org Received: from mout.perfora.net (HELO mout.perfora.net) (74.208.4.194) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Fri, 27 Apr 2012 22:38:11 +0000 Received: from JamesJPC (75-146-80-177-Chattanooga.hfc.comcastbusiness.net [75.146.80.177]) by mrelay.perfora.net (node=mrus3) with ESMTP (Nemesis) id 0Lx8qn-1S8RDt12zp-0177rP; Fri, 27 Apr 2012 18:38:09 -0400 From: "James Johnston" To: References: <020501cd23f2$20f07620$62d16260$@motionview3d.com> <20120427143753.GO25385@calimero.vinschen.de> In-Reply-To: <20120427143753.GO25385@calimero.vinschen.de> Subject: RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping) Date: Fri, 27 Apr 2012 22:38:00 -0000 Message-ID: <025501cd24c6$6a353990$3e9facb0$@motionview3d.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com X-SW-Source: 2012-04/txt/msg00656.txt.bz2 > -----Original Message----- > Sent: Friday, April 27, 2012 14:38 > Subject: Re: Cygwin passes through null writes to other software when > redirecting standard input/output (i.e. piping) >=20 > What I don't grok is this: >=20 > In your example, A and B are both native (=3D=3D non-Cygwin) applicatio= ns. >=20 > As I already mentioned in another mail related to the .Net problem, while > the Cygwin shell running at the time creates the pipe, it doesn't handle = read > and write calls on the pipe. It only creates the two native child proces= ses and > passes on the pipe handles. >=20 > So in the end, there is no Cygwin process involved while the native proce= sses > are doing their reads and writes. It's all in the hands of these process= es. > There's no third process in the middle which handles the passing of > information from A to B. >=20 > So, what do you expect Cygwin to do? Thanks for responding. I apologize for not fully understanding the problem= earlier. In .NET, if you wanted to pipe the output of one process to anot= her, you'd have to be exactly this third process as far as I can tell - the= framework creates a pipe/stream for each process, so you'd need a loop to = copy information from one pipe to another. To help understand how this is all working, I used API Monitor to spy and s= ee how the Windows command prompt sets up its pipe. It's very simple: it c= an be simplified down to a call to CreatePipe to make an anonymous pipe. T= he two child processes are then spawned, with the pipe reader/writer handle= s being passed to CreateProcess. Unused pipe handles are closed in both th= e parent process and the child processes, so that the processes terminate c= orrectly. I wrote a straight Win32 C program to spawn my child processes a= nd direct output from one to another's input, using CreatePipe. The result= s were correct. > What you are seeing may be because Cygwin was changed to use > message-type pipes a couple of revisions ago. This is not going to > change. The change was adopted to fix a problem with Cygwin > programs and those are obviously our #1 priority. I will certainly not disagree with you that Cygwin programs need to work 10= 0%! But surely there has got to be some solution that can achieve this whi= le not widely breaking non-Cygwin programs..!? I'm actually surprised more= people wouldn't be using Windows programs from Cygwin - to me, that is pa= rt of the point of using a system like Cygwin rather than just installing t= he Linux distro of choice - so I can mix/match both Windows and UNIX softwa= re. For me, both Windows and Cygwin software need to work - not just one o= r the other. While you have near-zero interest in running Windows programs= from Cygwin - I would bet that does not describe a significant number of u= sers. So to confirm your hypothesis, I modified my previously-described C program= . It uses CreateNamedPipe followed by CreateFile to get reader/writer hand= les instead of making an anonymous pipe using CreatePipe. If I use byte mo= de, the results are still correct - apparently byte mode pipes won't pass t= hrough null writes. HOWEVER, if I switch to message mode - the results are= incorrect - just like the bad results from Cygwin. Message mode pipes pas= s through null writes. So I think for sure, Cygwin's use of message pipes is breaking a lot of Win= dows software, because of the null writes. And ALSO additionally perhaps b= ecause of this: while reading MSDN today, I came across an interesting snip= pet that probably indicates a second source of problems I have not explicit= ly tested for, but would probably find: "If the specified number of bytes to read is less than the size of the next= message, the function reads as much of the message as possible before retu= rning zero (the GetLastError function returns ERROR_MORE_DATA). The remaind= er of the message can be read using another read operation." While I did not test for this, I think this would also break a lot of Windo= ws software. It's conceivable that the sender could write out a lot of dat= a, while the receiver isn't reading anything. Then, if the reader tries to= read but with a small buffer, according to the docs it will get ERROR_MORE= _DATA. Most runtimes don=E2=80=99t expect ReadFile to fail with ERROR_MORE= _DATA. For example, the decompiled .NET Framework code I read would throw = an exception, as it only gracefully handles ERROR_BROKEN_PIPE. Probably ot= hers like the Visual C++ runtime will treat it as EOF or set the failbit. = If there's interest I expect I could write a test case. (e.g. run "cat" to = pipe a gigantic text file to a Windows program that takes a "long" time to = process each line). So I have some questions... 1. Why on earth did Cygwin start using message pipes? "adopted to fix a p= roblem" doesn't help... Googling site:Cygwin.com "message pipe" - didn't yi= eld anything either. I do know older Cygwin versions worked fine, and I ha= d no problem with piping data to/from both Cygwin and non-Cygwin programs. = I am sure it was changed for a good reason, but would someone be willing t= o offer an explanation for why message pipes are now used? Do they have pr= oblems reading from a byte pipe? Writing to a byte pipe? Both? Under wha= t conditions? 2. Is it possible that byte pipes could be used to communicate with non-Cy= gwin programs, while message pipes used to communicate between Cygwin progr= ams? Since I don't know the answer to the previous question, I have no ide= a how this question could be answered without causing / moving the breakage= somewhere else.=20 Here's sample code that can demonstrate both byte and message pipes. Edit = the EXE names in the calls to StartProc() accordingly. #include #include using namespace std; /* error handling left out for brevity */ /* start process with specified input/output handles and return process han= dle */ HANDLE StartProc(wchar_t *proc, HANDLE in, HANDLE out) { STARTUPINFO si; PROCESS_INFORMATION pi; memset(&si, 0, sizeof(si)); memset(&pi, 0, sizeof(pi)); si.cb =3D sizeof(si); si.dwFlags =3D STARTF_USESTDHANDLES; si.hStdInput =3D in; si.hStdOutput =3D out; si.hStdError =3D GetStdHandle(STD_ERROR_HANDLE); CreateProcess(proc, NULL, NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi); CloseHandle(pi.hThread); return pi.hProcess; } int main() { HANDLE hPipeRead, hPipeWrite; SECURITY_ATTRIBUTES secat; memset(&secat, 0, sizeof(secat)); secat.nLength =3D sizeof(secat); secat.bInheritHandle =3D TRUE; /* note: changing to PIPE_TYPE_BYTE will fix the problems */ hPipeWrite =3D CreateNamedPipe(L"\\\\.\\pipe\\CoordTestPipe", PIPE_ACCESS_= OUTBOUND, PIPE_TYPE_MESSAGE, 1, 0, 0, 0, &secat); /* FILE_WRITE_ATTRIBUTES required if you play with SetNamedPipeHandleState= */ hPipeRead =3D CreateFile(L"\\\\.\\pipe\\CoordTestPipe", GENERIC_READ | FIL= E_WRITE_ATTRIBUTES, 0, &secat, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); DWORD mode =3D PIPE_READMODE_BYTE; /*SetNamedPipeHandleState(reader, &mode, NULL, NULL); // this won't fix an= ything for the poor reader program */ HANDLE hSender, hReceiver; hSender =3D StartProc(L"Sender.exe", GetStdHandle(STD_INPUT_HANDLE), hPipe= Write); hReceiver =3D StartProc(L"Receiver.exe", hPipeRead, GetStdHandle(STD_OUTPU= T_HANDLE)); /* close pipe handles that aren't needed; processes won't terminate otherw= ise */ DuplicateHandle(hSender, hPipeRead, NULL, NULL, 0, FALSE, DUPLICATE_CLOSE_= SOURCE); DuplicateHandle(hReceiver, hPipeWrite, NULL, NULL, 0, FALSE, DUPLICATE_CLO= SE_SOURCE); CloseHandle(hPipeRead); CloseHandle(hPipeWrite); /* wait for child processes to exit */ WaitForSingleObject(hSender, INFINITE); WaitForSingleObject(hReceiver, INFINITE); CloseHandle(hSender); CloseHandle(hReceiver); return 0; } > Your complete agreement with each other is not going to have much > effect. Cygwin source code is not changed by voting. OK, fine. What would you have us do then? Not provide any seemingly-bothe= rsome input / opinions at all, if the maintainers have no interest in what = the users might run into? I realize you are all very busy people, and that= the Cygwin project doesn't have the major backing that larger projects do.= Fixing lower priority bugs might not be possible on your own. But do yo= u even want to hear feedback and bug reports if they are not accompanied by= a perfectly-written patch, or are they too bothersome? Are the maintainer= s interested in bug reports, or are they not? I have already spent a fair = bit of time researching this issue, and it's not the first Cygwin issue I'v= e encountered. I'm going to this trouble because I was hoping that in some= way, Cygwin can ultimately be improved - if not by my hands then perhaps b= y the hands of someone else who might be reading this list and have more id= eas or skills! Write my own patch? OK, but... I get the strong feeling that: (1) I won't = have anyone to ask regarding where in the code I might need to even start l= ooking and would be willing to answer any questions, and any considerations= I might need to take into account - I have never actually worked on the Cy= gwin code before - and there is a lot of it - and I don't want to "cowboy" = my way through it and make more problems than I fix! (2) I'm afraid you'd s= ummarily ignore any patch I might come up with, without providing real, hel= pful feedback on what would need to changed and improved in the patch for i= t to be accepted. Reading over the Cygwin contribution guidelines indicate= s there are an awful lot of things I have to get perfectly right, and frank= ly I am not sure I am going to get it perfectly right. I don't want to was= te more hours trying to patch this thing if there's no real chance of accep= tance. > Maybe I'm in the minority but I find long-winded impassioned pleas > rather boring and off-putting. So, continuing in this vein is not going > to be productive for you, at least as far as I'm concerned. Oh good grief... Most of the length in my e-mails have been due to me attem= pting to do more detailed research than the average person might do. Would= I be better off writing short e-mails that say "Cygwin doesn=E2=80=99t wor= k, why?"- without providing more research? James=20 -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple