GNU coreutils does not work on Cygwin because of freopen() ?

GNU coreutils does not work on Cygwin because of freopen() ?

[dazjorz]

Hey all,

I'm not very great with the inner workings of GNU or Cygwin libc, so 
please bear with me if I make any mistakes or misunderstandings.

I was trying to get GNU coreutils to work on Cygwin. Maybe this seems 
weird to you guys, because Cygwin has coreutils, but there are a few 
reasons:
- Cygwin didn't have md5sum, GNU coreutils did (and I need md5sum)
- It should work, and "Because I Can" is always a valid reason in UNIX world
- I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is 
my first try; getting coreutils from Cygwin may be done later.

However, I noticed there are some very weird bugs happening. See for 
example, this bug in cat (don't look at the title of the page, I blamed 
bash first)
http://paster.dazjorz.com/?p=3845

A friend of mine and me have been trying to figure out what causes this. 
First, we tested if it wasn't a bug in open() + fork(). This seemed not 
to be the case. So we started looking further, and we're down to GNU 
`cat` calling freopen(). This is not POSIX-compliant and can cause 
problems like this.

`cat` and other coreutils tools only call freopen() if O_BINARY is 
defined on this platform. It's defined on Cygwin, because Cygwin makes a 
difference between O_TEXT and O_BINARY.

1. Why does Cygwin distinguish between O_TEXT and O_BINARY? Shouldn't 
this difference be removed from the Linux API side of Cygwin?
2. I don't know why coreutils is calling freopen() at all. Should I 
patch it out completely, or add something like an #ifndef __CYGWIN__ 
around the pieces of code in question?

I'm hesitant to patch coreutils, so I'm asking for your opinion first.

Thanks for any replies,
Sjors

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Dave Korn]

dazjorz wrote:

> - Cygwin didn't have md5sum,

  Cygwin does have md5sum in coreutils.  You can see this at the cygwin
package list:

   http://cygwin.com/packages/

by entering 'md5sum.exe' in the search box.

> - It should work, and "Because I Can" is always a valid reason in UNIX
> world

  However, you may have to recreate all the cygwin-specific patches that the
cygwin bash maintainer hasn't managed to send upstream yet...

> - I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is
> my first try; getting coreutils from Cygwin may be done later.

  Dunno if your project makes sense.  You'll have to end up porting
everything where the differences in the underlying OS show through... which is
basically what Cygwin package maintainers have already done for most of this
stuff.  Do you mean you're trying to cross-compile Debian on Cygwin, or do you
actually want to build the whole Debian distro under Cygwin to run on Windows
rather than natively under a linux kernel?

> However, I noticed there are some very weird bugs happening. See for
> example, this bug in cat (don't look at the title of the page, I blamed
> bash first)
> http://paster.dazjorz.com/?p=3845
>
> A friend of mine and me have been trying to figure out what causes this.
> First, we tested if it wasn't a bug in open() + fork(). This seemed not
> to be the case. So we started looking further, and we're down to GNU
> `cat` calling freopen(). This is not POSIX-compliant and can cause
> problems like this.

?? Are you using an old version??  It should be fixed by now.

http://www.mail-archive.com/bug-coreutils@gnu.org/msg10188.html

> `cat` and other coreutils tools only call freopen() if O_BINARY is
> defined on this platform. It's defined on Cygwin, because Cygwin makes a
> difference between O_TEXT and O_BINARY.
>
> 1. Why does Cygwin distinguish between O_TEXT and O_BINARY? Shouldn't
> this difference be removed from the Linux API side of Cygwin?

  Nope.  Cygwin runs on windows, which uses CR-LF for line endings, but
Cygwin is POSIX-compliant and uses LF internally.  This is exactly what O_TEXT
was invented for: it translates LF->CRLF on write and CRLF->LF on read.  On
Linux platforms, the native format is LF line-ends, so O_TEXT and O_BINARY are
the same, but not here.

> 2. I don't know why coreutils is calling freopen() at all. Should I
> patch it out completely, or add something like an #ifndef __CYGWIN__
> around the pieces of code in question?

  Well, if you really want to build coreutils under cygwin, why not use
setup.exe to install the corresponding source package, which comes with all
these problems pre-solved for you?

    cheers,
      DaveK

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Dave Korn on 1/18/2009 2:42 PM:
>   However, you may have to recreate all the cygwin-specific patches that the
> cygwin bash maintainer hasn't managed to send upstream yet...

Not only that, but some of those patches will never be sent upstream,
because they are too platform-specific.  As for this particular issue of
freopen, I'm dragging my feet - the cygwin patch works, and no other
mainstream platform is affected, so even though I have _already_ pointed
out upstream that the current freopen use is buggy, I haven't bothered
patching things upstream yet (I only have so much free time).

http://lists.gnu.org/archive/html/bug-coreutils/2008-10/msg00199.html

>> However, I noticed there are some very weird bugs happening. See for
>> example, this bug in cat (don't look at the title of the page, I blamed
>> bash first)
>> http://paster.dazjorz.com/?p=3845

Yes, this is a known upstream bug in coreutils, but which only affects
cygwin, and which is already fixed on cygwin, so there is very little need
or desire to spend time fixing it upstream.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklzrW4ACgkQ84KuGfSFAYBiMwCcCmf54S2kJPf8e/cO79NVVkFi
Xz0AoKBlI79e4GNO/wLOHK7fv9PhOYec
=PekT
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

GNU coreutils does not work on Cygwin because of freopen() ?

[dazjorz]

Hey all,

I'm not very great with the inner workings of GNU or Cygwin libc, so 
please bear with me if I make any mistakes or misunderstandings.

I was trying to get GNU coreutils to work on Cygwin. Maybe this seems 
weird to you guys, because Cygwin has coreutils, but there are a few 
reasons:
- Cygwin didn't have md5sum, GNU coreutils did (and I need md5sum)
- It should work, and "Because I Can" is always a valid reason in UNIX world
- I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is 
my first try; getting coreutils from Cygwin may be done later.

However, I noticed there are some very weird bugs happening. See for 
example, this bug in cat (don't look at the title of the page, I blamed 
bash first)
http://paster.dazjorz.com/?p=3845

A friend of mine and me have been trying to figure out what causes this. 
First, we tested if it wasn't a bug in open() + fork(). This seemed not 
to be the case. So we started looking further, and we're down to GNU 
`cat` calling freopen(). This is not POSIX-compliant and can cause 
problems like this.

`cat` and other coreutils tools only call freopen() if O_BINARY is 
defined on this platform. It's defined on Cygwin, because Cygwin makes a 
difference between O_TEXT and O_BINARY.

1. Why does Cygwin distinguish between O_TEXT and O_BINARY? Shouldn't 
this difference be removed from the Linux API side of Cygwin?
2. I don't know why coreutils is calling freopen() at all. Should I 
patch it out completely, or add something like an #ifndef __CYGWIN__ 
around the pieces of code in question?

I'm hesitant to patch coreutils, so I'm asking for your opinion first.

Thanks for any replies,
Sjors

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Larry Hall (Cygwin)]

dazjorz wrote:
> Hey all,
> 
> I'm not very great with the inner workings of GNU or Cygwin libc, so 
> please bear with me if I make any mistakes or misunderstandings.
> 
> I was trying to get GNU coreutils to work on Cygwin. Maybe this seems 
> weird to you guys, because Cygwin has coreutils, but there are a few 
> reasons:
> - Cygwin didn't have md5sum, GNU coreutils did (and I need md5sum)

<http://cygwin.com/cgi-bin2/package-grep.cgi?grep=bin%2Fmd5sum>

Did you miss this?

> - It should work, and "Because I Can" is always a valid reason in UNIX 
> world
> - I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is 
> my first try; getting coreutils from Cygwin may be done later.
> 
> However, I noticed there are some very weird bugs happening. See for 
> example, this bug in cat (don't look at the title of the page, I blamed 
> bash first)
> http://paster.dazjorz.com/?p=3845

Your example works fine for me with the latest coreutils.  Perhaps you're
working with a buggy locally built version?

-- 
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      (508) 893-9779 - RFK Office
216 Dalton Rd.                          (508) 893-9889 - FAX
Holliston, MA 01746

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Sjors Gielen]

Larry Hall (Cygwin) wrote:
> dazjorz wrote:
>> Hey all,
>>
>> I'm not very great with the inner workings of GNU or Cygwin libc, so 
>> please bear with me if I make any mistakes or misunderstandings.
>>
>> I was trying to get GNU coreutils to work on Cygwin. Maybe this seems 
>> weird to you guys, because Cygwin has coreutils, but there are a few 
>> reasons:
>> - Cygwin didn't have md5sum, GNU coreutils did (and I need md5sum)
> 
> <http://cygwin.com/cgi-bin2/package-grep.cgi?grep=bin%2Fmd5sum>
> 
> Did you miss this?

Hmm. I guess. Oh well, I didn't have md5sum before, I think :/

>> - It should work, and "Because I Can" is always a valid reason in UNIX 
>> world
>> - I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian 
>> is my first try; getting coreutils from Cygwin may be done later.
>>
>> However, I noticed there are some very weird bugs happening. See for 
>> example, this bug in cat (don't look at the title of the page, I 
>> blamed bash first)
>> http://paster.dazjorz.com/?p=3845
> 
> Your example works fine for me with the latest coreutils.  Perhaps you're
> working with a buggy locally built version?
> 

Did you try with Cygwin coreutils or GNU coreutils? :)

By the way, list, I've been looking for some other things. This problem 
is caused by:
- cat calling freopen to make the handle binary (shouldn't it be binary 
in the first place? shouldn't it check if it's already opened in binary 
mode?)
- freopen() not checking if O_APPEND is set, etc. (is it possible to 
check this?)

I'll write a test case for freopen(), that shows O_APPEND is being 
missed, causing this bug.

Sjors

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Yaakov (Cygwin/X)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Sjors Gielen wrote:
> Did you try with Cygwin coreutils or GNU coreutils? :)

Cygwin's coreutils *is* GNU coreutils, with some patches which you will
find necessary.


Yaakov
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREIAAYFAklzorEACgkQpiWmPGlmQSOl0ACg5+4mOgMwaTkMMkQTirY2Xlwz
xJ4An3OO1gl/7fl+GBzay8urXpDaJJDS
=1fhI
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Sjors Gielen]

Yaakov (Cygwin/X) wrote:
> Sjors Gielen wrote:
>> Did you try with Cygwin coreutils or GNU coreutils? :)
> 
> Cygwin's coreutils *is* GNU coreutils, with some patches which you will
> find necessary.
> 
[snip]
Dave Korn wrote:
 >   However, you may have to recreate all the cygwin-specific patches
 > that the
 > cygwin bash maintainer hasn't managed to send upstream yet...
[snip]
 > ?? Are you using an old version??  It should be fixed by now.
 >
 > http://www.mail-archive.com/bug-coreutils@gnu.org/msg10188.html

Oooh. Can I get a list of patches the Cygwin project has already applied 
to packages? :)
So this bug was fixed in Cygwin's version of GNU coreutils, but not in 
Debian's version of GNU coreutils. Has a patch for that been sent 
upstream? I took GNU coreutils from Debian sid, which should be almost 
as up to date as it gets...

 >   Dunno if your project makes sense.  You'll have to end up porting
 > everything where the differences in the underlying OS show through...
 > which is
 > basically what Cygwin package maintainers have already done for most
 > of this
 > stuff.  Do you mean you're trying to cross-compile Debian on Cygwin,
 > or do you
 > actually want to build the whole Debian distro under Cygwin to run on
 > Windows
 > rather than natively under a linux kernel?

I'm trying to create a full Cygwin port for Debian. This means the base 
system packages, et cetera. Let's just keep it at I love apt and dpkg 
for package managers.

 >   Nope.  Cygwin runs on windows, which uses CR-LF for line endings,
 > but
 > Cygwin is POSIX-compliant and uses LF internally.  This is exactly
 > what O_TEXT
 > was invented for: it translates LF->CRLF on write and CRLF->LF on
 > read.  On
 > Linux platforms, the native format is LF line-ends, so O_TEXT and
 > O_BINARY are
 > the same, but not here.

But if Cygwin uses LF internally, why does it translate to CRLF on 
write? I remember this installation option, asking if it should use CRLF 
or LF for writing files. I think I chose LF only, because that's the 
Unix way of doing it.

Do I misunderstand here? Currently, when I write a file in vim and save 
it, it has LF line endings anyway. If it is saved with LF line endings, 
there's no difference between O_TEXT and O_BINARY, right? So why not 
skip this at all and make *every* fd O_BINARY by default?

 >> 2. I don't know why coreutils is calling freopen() at all. Should I
 >> patch it out completely, or add something like an #ifndef __CYGWIN__
 >> around the pieces of code in question?
 >
 >   Well, if you really want to build coreutils under cygwin, why not
 > use
 > setup.exe to install the corresponding source package, which comes
 > with all
 > these problems pre-solved for you?

That would, of course, defeat the idea of bootstrapping Debian on Cygwin. ;)

Sjors


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Matt Wozniski]

On Sun, Jan 18, 2009 at 5:11 PM, Sjors Gielen wrote:
> I'm trying to create a full Cygwin port for Debian. This means the base
> system packages, et cetera. Let's just keep it at I love apt and dpkg for
> package managers.

Keep in mind that apt and dpkg will never be able to work completely
on Windows, since they were written with some fundamental assumptions
of how the system will behave that don't hold on Windows.  dpkg
expects to be able to update itself, or the shell it's running in, by
just writing a new file with the same name to the same directory, and
this won't work on windows; you can't overwrite an executable that is
currently being executed.

~Matt

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Ed Schouten]

[-- Attachment #1: Type: text/plain, Size: 1971 bytes --]

Hello all,

* Sjors Gielen <mailinglist@dazjorz.com> wrote:
> Yaakov (Cygwin/X) wrote:
>> Sjors Gielen wrote:
>>> Did you try with Cygwin coreutils or GNU coreutils? :)
>>
>> Cygwin's coreutils *is* GNU coreutils, with some patches which you will
>> find necessary.
>>
> [snip]
> Dave Korn wrote:
> >   However, you may have to recreate all the cygwin-specific patches
> > that the
> > cygwin bash maintainer hasn't managed to send upstream yet...
> [snip]
> > ?? Are you using an old version??  It should be fixed by now.
> >
> > http://www.mail-archive.com/bug-coreutils@gnu.org/msg10188.html
>
> Oooh. Can I get a list of patches the Cygwin project has already applied  
> to packages? :)
> So this bug was fixed in Cygwin's version of GNU coreutils, but not in  
> Debian's version of GNU coreutils. Has a patch for that been sent  
> upstream? I took GNU coreutils from Debian sid, which should be almost  
> as up to date as it gets...

But that's not a proper fix anyway. cat can't just decide to reopen the
file descriptor of the standard output. That even breaks the first
sentence of POSIX:

	The cat utility shall read files in sequence and shall write
	their contents to the standard output in the same sequence.

If it says "the standard output" doesn't mean you can reopen the output
descriptor. This means you end up with two different file descriptors.
They can point to the same file, but they are not the same file
descriptor.

They fixed it by using SET_BINARY(), but not on Cygwin, because I assume
SET_BINARY() is a Windows-thing, not Cygwin.

I can't really understand why Cygwin just doesn't force O_BINARY to be
set. File descriptors on UNIX are supposed to display raw file contents.
There shouldn't be any (newline) processing. It makes no sense to
cripple Cygwin only because Notepad doesn't support newlines without the
carriage return.

-- 
 Ed Schouten <ed@80386.nl>
 WWW: http://80386.nl/

[-- Attachment #2: Type: application/pgp-signature, Size: 195 bytes --]

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Ed Schouten on 1/18/2009 3:19 PM:
> But that's not a proper fix anyway. cat can't just decide to reopen the
> file descriptor of the standard output. That even breaks the first
> sentence of POSIX:
> 
> 	The cat utility shall read files in sequence and shall write
> 	their contents to the standard output in the same sequence.
> 
> If it says "the standard output" doesn't mean you can reopen the output
> descriptor. This means you end up with two different file descriptors.
> They can point to the same file, but they are not the same file
> descriptor.

That's where you are wrong.  freopen can be passed a NULL pointer instead
of a filename in order to reuse the same file descriptor, but with an
implementation-defined different mode.  In cygwin's case, this means you
can use freopen to swap a file between text and binary mode.

> 
> They fixed it by using SET_BINARY(), but not on Cygwin, because I assume
> SET_BINARY() is a Windows-thing, not Cygwin.

SET_BINARY is an upstream coreutils thing designed for cygwin and other
platforms that have O_BINARY in the first place (Debian doesn't).  It is a
macro that expands to either the non-standard setmode(), or the
standardized (but implementation-dependent) freopen().

> 
> I can't really understand why Cygwin just doesn't force O_BINARY to be
> set. File descriptors on UNIX are supposed to display raw file contents.

That's exactly what that freopen is doing - it is forcing O_BINARY to be
set, regardless of how stdin was inherited.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklzrKgACgkQ84KuGfSFAYCZzACeJcwTKhDejFWZLP8LkhKRkv/4
bj0AniqGtXHep5FcUIavGtl2YOWeSRwQ
=aiau
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: GNU coreutils does not work on Cygwin because of freopen() ?

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Sjors Gielen on 1/18/2009 3:11 PM:
>> cygwin bash maintainer hasn't managed to send upstream yet...
> [snip]
>> ?? Are you using an old version??  It should be fixed by now.
>>
>> http://www.mail-archive.com/bug-coreutils@gnu.org/msg10188.html
> 
> Oooh. Can I get a list of patches the Cygwin project has already applied
> to packages? :)

Yes.  Use setup.exe, and select the "src" checkbox next to coreutils (or
any other package where you want to know what went into the cygwin
version).  For coreutils, that will get you all of my cygwin-specific
patches to coreutils.

> So this bug was fixed in Cygwin's version of GNU coreutils, but not in
> Debian's version of GNU coreutils. Has a patch for that been sent
> upstream? I took GNU coreutils from Debian sid, which should be almost
> as up to date as it gets...

Debian coreutils does not care about cygwin-specific patches, and the
freopen in question, while it is indeed buggy upstream, only affects
cygwin and other platforms that define O_BINARY.  Since that excludes
debian, then debian's fork of coreutils has no reason to patch it.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklzq6cACgkQ84KuGfSFAYC0ogCfSvjhCtwLoUbTlLV7S4GR2HFV
wpkAoLdjblp/mz+7OhrIcIk3jZTDv01E
=xvRw
-----END PGP SIGNATURE-----

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/