fcntl with O_APPEND fails to force append mode on stderr for native Windows programs

fcntl with O_APPEND fails to force append mode on stderr for native Windows programs

[Tom Honermann]

This is a follow up to a bug originally reported as https://cygwin.com/ml/cygwin/2015-05/msg00140.html.  We continue to face this issue and recently spent some time trying to further isolate it.  We now have a better reproducer (below) that does not require GNU make.

From the original bug report, we bisected changes to GNU make and eventually identified that the reported problem was introduced by this change:

  *   http://git.savannah.gnu.org/cgit/make.git/commit/?id=9453a7451d66ee1ea5762a75161bf8a818c01a1f

The following program demonstrates the problem.

$ cat test.c
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>

void set_append_mode(int fd) {
  int flags = fcntl(fd, F_GETFL, 0);
  if (flags >= 0)
    fcntl(fd, F_SETFL, flags | O_APPEND);
}

int main() {
  set_append_mode(fileno(stdout));
  set_append_mode(fileno(stderr));
  system("echo first > out");
  system("echo second >> out");
  system("net 2>> out");
}

The expected output of this program is that file 'out' have the content "first", followed by "second", followed by the error message emitted by the Windows 'net' utility (any native Windows program that writes to stderr should suffice to reproduce this issue).  And indeed, that is what is seen when the program is compiled and run with stdout and stderr set to the console.

$ gcc test.c -o test
$ ./test
$ cat out
first
second
The syntax of this command is:

NET
    [ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
      HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START |
      STATISTICS | STOP | TIME | USE | USER | VIEW ]

Here is where things go bananas.  If the program is run with stdout initially redirected to a pipe, then the stderr output from the native Windows program ends up overwriting the stdout output from the two calls to echo:

$ ./test | cat
$ cat out
The syntax of this command is:

NET
    [ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
      HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START |
      STATISTICS | STOP | TIME | USE | USER | VIEW ]

Note that the lines containing "first" and "second" are now absent in the file.

If the calls to set_append_mode are commented out, then the output is consistent in both cases.

The problem only occurs with native Windows programs that write to stderr.  If the above test is changed to append the stdout output from a native Windows program, then the output is consistent.

I've reproduced this problem on ancient versions of Cygwin such as 1.7.23 and recent versions such as 2.11.2.

Tom.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fcntl with O_APPEND fails to force append mode on stderr for native Windows programs

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Feb  8 22:25, Tom Honermann wrote:
> The following program demonstrates the problem.
> [...]
> Here is where things go bananas.  If the program is run with stdout
> initially redirected to a pipe, then the stderr output from the native
> Windows program ends up overwriting the stdout output from the two
> calls to echo:
> 
> $ ./test | cat
> $ cat out
> The syntax of this command is:
> 
> NET
>     [ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
>       HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START |
>       STATISTICS | STOP | TIME | USE | USER | VIEW ]
> 
> Note that the lines containing "first" and "second" are now absent in
> the file.

Thanks for the testcase.  I tracked this down to a bug when calling
native (non-Cygwin) apps.  If the descriptor is in O_APPEND mode, we
have to set the file offset to EOF before calling native apps.  However,
this only worked reliably for the first descriptor in O_APPEND mode, but
only by chance for subsequent descriptors in O_APPEND mode.  Your
example code with the extra pipe to `cat' triggered the misbehaviour.

I fixed that in git master:
https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0be0b8f0335e

This will be in the 3.0.0 release.

I just uploaded new developer snapshots to https://cygwin.com/snapshots/
You only need to replace the Cygwin DLL for testing.  Please give it a try.


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: fcntl with O_APPEND fails to force append mode on stderr for native Windows programs

[Tom Honermann]

On 2/9/2019 9:51 AM, Corinna Vinschen wrote:
> On Feb  8 22:25, Tom Honermann wrote:
>> The following program demonstrates the problem.
>> [...]
>> Here is where things go bananas.  If the program is run with stdout
>> initially redirected to a pipe, then the stderr output from the native
>> Windows program ends up overwriting the stdout output from the two
>> calls to echo:
>>
>> $ ./test | cat
>> $ cat out
>> The syntax of this command is:
>>
>> NET
>>     [ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
>>       HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START |
>>       STATISTICS | STOP | TIME | USE | USER | VIEW ]
>>
>> Note that the lines containing "first" and "second" are now absent in
>> the file.
> Thanks for the testcase.  I tracked this down to a bug when calling
> native (non-Cygwin) apps.  If the descriptor is in O_APPEND mode, we
> have to set the file offset to EOF before calling native apps.  However,
> this only worked reliably for the first descriptor in O_APPEND mode, but
> only by chance for subsequent descriptors in O_APPEND mode.  Your
> example code with the extra pipe to `cat' triggered the misbehaviour.
>
> I fixed that in git master:
> https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0be0b8f0335e
>
> This will be in the 3.0.0 release.
>
> I just uploaded new developer snapshots to https://cygwin.com/snapshots/
> You only need to replace the Cygwin DLL for testing.  Please give it a try.

Thank you, Corinna!  This does appear to fix the problem.  I verified
the snapshot with the original test case and the one above and both work
as expected now.  We'll do some more extensive testing and report back
if we find any further issues.  You are awesome!

Tom.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fcntl with O_APPEND fails to force append mode on stderr for native Windows programs

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

On Feb  9 15:33, Tom Honermann wrote:
> On 2/9/2019 9:51 AM, Corinna Vinschen wrote:
> > On Feb  8 22:25, Tom Honermann wrote:
> >> The following program demonstrates the problem.
> >> [...]
> >> Here is where things go bananas.  If the program is run with stdout
> >> initially redirected to a pipe, then the stderr output from the native
> >> Windows program ends up overwriting the stdout output from the two
> >> calls to echo:
> >>
> >> $ ./test | cat
> >> $ cat out
> >> The syntax of this command is:
> >>
> >> NET
> >>     [ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
> >>       HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START |
> >>       STATISTICS | STOP | TIME | USE | USER | VIEW ]
> >>
> >> Note that the lines containing "first" and "second" are now absent in
> >> the file.
> > Thanks for the testcase.  I tracked this down to a bug when calling
> > native (non-Cygwin) apps.  If the descriptor is in O_APPEND mode, we
> > have to set the file offset to EOF before calling native apps.  However,
> > this only worked reliably for the first descriptor in O_APPEND mode, but
> > only by chance for subsequent descriptors in O_APPEND mode.  Your
> > example code with the extra pipe to `cat' triggered the misbehaviour.
> >
> > I fixed that in git master:
> > https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=0be0b8f0335e
> >
> > This will be in the 3.0.0 release.
> >
> > I just uploaded new developer snapshots to https://cygwin.com/snapshots/
> > You only need to replace the Cygwin DLL for testing.  Please give it a try.
> 
> Thank you, Corinna!  This does appear to fix the problem.  I verified
> the snapshot with the original test case and the one above and both work
> as expected now.  We'll do some more extensive testing and report back
> if we find any further issues.  You are awesome!

Thanks for testing.  I'll upload another 3.0.0 test release early next week
and then I hope the release can be done a few days later.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]