1.5.25-6: Win32 programs don't get correct >> redirection

1.5.25-6: Win32 programs don't get correct >> redirection

[Jack Brennen]

Running Cygwin version 1.5.25-6 with version 3.2.25-17 of bash with Win 
XP Pro...


Start up bash.

Enter these commands in order:


sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
sh-3.2$ echo UVWXYZ >> foo.txt
sh-3.2$ cat foo.txt
ABCDEFGHIJKLMNOPQRST
UVWXYZ
sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
sh-3.2$ cmd /c echo UVWXYZ >> foo.txt
sh-3.2$ cat foo.txt
UVWXYZ
IJKLMNOPQRST
sh-3.2$


The first time that we use the ">>" redirection operator, it works 
correctly.

The second time that we use the ">>" redirection operator, it overwrites 
the beginning of the file rather than appending to the end.
The only difference is that the Win32 command interpreter CMD.EXE is 
echoing the text rather than bash.



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.25-6: Win32 programs don't get correct >> redirection

[René Berber]

Jack Brennen wrote:

> Running Cygwin version 1.5.25-6 with version 3.2.25-17 of bash with Win
> XP Pro...
> 
> Start up bash.
> 
> Enter these commands in order:
> 
> sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
> sh-3.2$ echo UVWXYZ >> foo.txt
> sh-3.2$ cat foo.txt
> ABCDEFGHIJKLMNOPQRST
> UVWXYZ
> sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
> sh-3.2$ cmd /c echo UVWXYZ >> foo.txt
> sh-3.2$ cat foo.txt
> UVWXYZ
> IJKLMNOPQRST
[snip]

What you don't see is that Windows' echo added a CR so you get to see
the second line followed by CR-LF and the rest of the (overwritten)
first line.

You can delete the CR and get the correct output, for instance using:

cmd /c echo UVWXYZ | tr -d "\r" >> foo.txt

Or directly with something like:

cmd /c echo UVWXYZ | tr -d "\r" | od -t x1z

take out the translate part to see the CR-LF.
-- 
René Berber


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.25-6: Win32 programs don't get correct >> redirection

[Erich Dollansky]

Hi,

René Berber wrote:
> Jack Brennen wrote:
> 
> 
> What you don't see is that Windows' echo added a CR so you get to see
> the second line followed by CR-LF and the rest of the (overwritten)
> first line.
> 
the CR will be added after the appended characters and so not interfere 
with the example's data.

Erich


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.25-6: Win32 programs don't get correct >> redirection

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Jack Brennen on 12/13/2007 6:34 PM:
> sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
> sh-3.2$ cmd /c echo UVWXYZ >> foo.txt
> sh-3.2$ cat foo.txt
> UVWXYZ
> IJKLMNOPQRST

This issue was discussed on the lists in May:

http://thread.gmane.org/gmane.os.cygwin/89327/focus=89411

Cygwin 1.5.24 was incorrectly positioning fd's opened in append mode to
the last byte just before exec'ing a child process, rather than leaving
the position untouched (note that POSIX requires
open(O_RDWR|O_APPEND)/lseek to always be at position 0, but
fopen("a+")/ftell can be either 0 [as on Linux] or the end of the file [as
on BSD] - cygwin currently uses the Linux approach).

But it looks like this means that Windows programs are too stupid to
realize that they are being handed a file opened in append mode, and
blindly write to the current position, which happens to be the beginning
of the file instead of the end.

Maybe it is worth teaching cygwin's exec code that fd's in append mode
must be repositioned to the end if the child about to be spawned is a
non-cygwin program?  But how reliable is cygwin's detection of whether the
child is a cygwin process, so that we aren't violating POSIX by mistakenly
repositioning the fd when the child is a cygwin process?  Or do we make it
even more complicated, by having the pre-exec code always reposition, but
remember where it was beforehand, then cygwin children use that knowledge
to position back to the correct location?

- --
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHYexH84KuGfSFAYARAnQ9AKDTupShtU7xMcku4I3k6AVBcn8S/ACfchqI
r/mbrPmHdgxE7TuOpxuCnvA=
=O+VW
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.25-6: Win32 programs don't get correct >> redirection

[Corinna Vinschen]

On Dec 13 19:36, Eric Blake wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> According to Jack Brennen on 12/13/2007 6:34 PM:
> > sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
> > sh-3.2$ cmd /c echo UVWXYZ >> foo.txt
> > sh-3.2$ cat foo.txt
> > UVWXYZ
> > IJKLMNOPQRST
> 
> This issue was discussed on the lists in May:
> 
> http://thread.gmane.org/gmane.os.cygwin/89327/focus=89411
> 
> Cygwin 1.5.24 was incorrectly positioning fd's opened in append mode to
> the last byte just before exec'ing a child process, rather than leaving
> the position untouched (note that POSIX requires
> open(O_RDWR|O_APPEND)/lseek to always be at position 0, but
> fopen("a+")/ftell can be either 0 [as on Linux] or the end of the file [as
> on BSD] - cygwin currently uses the Linux approach).
> 
> But it looks like this means that Windows programs are too stupid to
> realize that they are being handed a file opened in append mode [...]

Even though it sounds nice to blame Windows, the blame is on Cygwin
entirely here.  The problem with O_APPEND mode is that you can switch
a file descriptor to and from append mode at will using fcntl.  This
functionality doesn't exist in the Win32 API.  You either open the
file handle in append mode or not, but you can't switch it back and 
forth.

For that reason, the file is always opened in generic write mode in
Cygwin.  Append mode is emulated in the write() call by seeking to
the end of the file before writing (in 1.5.x) or by using a feature
of the native NT ZwWriteFile call (in 1.7.x).

At this point we have three choices:

1. We revert to the old behaviour and accept a (long-standing) POSIX
   misbehaviour in terms of the file position in Cygwin child
   processes.

2. We keep the current behaviour and accept a (new) Win32 misbehaviour
   in terms of the file position in native Win32 child processes.

3. We (probably I) rework the append mode handling using a native
   NT feature to re-open a new handle from an existing handle while
   changing the append mode and keeping everything else as it is,
   thus risking that I break something else.

Solution 3 sounds like something which should only go into 1.7.x.  While
I usually tend to prefer correct POSIX behaviour over correct native
behaviour, it looks like a rather surprising change which might break a
lot of existing installations.  So it appears as if 1 would be the way
to go for now.  Oh well...


Corinna


-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: 1.5.25-6: Win32 programs don't get correct >> redirection

[Corinna Vinschen]

On Dec 14 11:50, Corinna Vinschen wrote:
> On Dec 13 19:36, Eric Blake wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > According to Jack Brennen on 12/13/2007 6:34 PM:
> > > sh-3.2$ echo ABCDEFGHIJKLMNOPQRST > foo.txt
> > > sh-3.2$ cmd /c echo UVWXYZ >> foo.txt
> > > sh-3.2$ cat foo.txt
> > > UVWXYZ
> > > IJKLMNOPQRST
> > 
> > This issue was discussed on the lists in May:
> > 
> > http://thread.gmane.org/gmane.os.cygwin/89327/focus=89411
> > 
> > Cygwin 1.5.24 was incorrectly positioning fd's opened in append mode to
> > the last byte just before exec'ing a child process, rather than leaving
> > the position untouched (note that POSIX requires
> > open(O_RDWR|O_APPEND)/lseek to always be at position 0, but
> > fopen("a+")/ftell can be either 0 [as on Linux] or the end of the file [as
> > on BSD] - cygwin currently uses the Linux approach).
> > 
> > But it looks like this means that Windows programs are too stupid to
> > realize that they are being handed a file opened in append mode [...]
> 
> Even though it sounds nice to blame Windows, the blame is on Cygwin
> entirely here.  The problem with O_APPEND mode is that you can switch
> a file descriptor to and from append mode at will using fcntl.  This
> functionality doesn't exist in the Win32 API.  You either open the
> file handle in append mode or not, but you can't switch it back and 
> forth.
> 
> For that reason, the file is always opened in generic write mode in
> Cygwin.  Append mode is emulated in the write() call by seeking to
> the end of the file before writing (in 1.5.x) or by using a feature
> of the native NT ZwWriteFile call (in 1.7.x).
> 
> At this point we have three choices:
> [...]

Scratch that, I implemented the forth choice.  Eric's point

> Maybe it is worth teaching cygwin's exec code that fd's in append mode
> must be repositioned to the end if the child about to be spawned is a
> non-cygwin program?

was right.  I implemented this now, for 1.5.25 as well as for the main
trunk.  I will look into implementing this in a more transparent way for
1.7.x, though.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/