compiling coreutils with cygport

compiling coreutils with cygport

[Jeff Janes]

If I use setup.exe to download the src for coreutils (just by checking
the "Src?" checkbox in the gui) , and then use cygport to build it, it
fails.  The reason it fails  seems to be that ginstall.exe does not
have the proper manifest file so windows refuses to execute it.  This
is the same error you get if you try to build coreutils downloaded
from upstream gnu sources.

cygport coreutils-8.15-1.cygport prep compile
....
make[2]: Entering directory `/usr/src/coreutils-8.15-1/build/man'
  GEN    chcon.1
  GEN    chgrp.1
  GEN    chmod.1
  GEN    cksum.1
  GEN    chown.1
  GEN    dd.1
  GEN    dir.1
  GEN    dircolors.1
  GEN    install.1
help2man: can't get `--help' info from install.td/install
Makefile:1833: recipe for target `install.1' failed
make[2]: *** [install.1] Error 126
make[2]: *** Waiting for unfinished jobs....
  GEN    ls.1
make[2]: Leaving directory `/usr/src/coreutils-8.15-1/build/man'
Makefile:1594: recipe for target `all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/src/coreutils-8.15-1/build'
Makefile:1549: recipe for target `all' failed
make: *** [all] Error 2
*** ERROR: make failed

This error is present when using the latest cygport, but it has not
new to that version, it has been around for months, and on multiple
different systems.

If I delete the two patch files which setup.exe automatically
downloads into /usr/src (coreutils-8.15-1.src.patch and
coreutils-8.15-1.cygwin.patch), and then run cygport as before, it
compiles fine.  Also, it works if I delete all the files which
setup.exe downloads to /usr/src except for the
"coreutils-8.15-1.cygport" one, and then run:

cygport coreutils-8.15-1.cygport download prep compile

So it looks like the setup.exe is giving me a coreutils tar ball which
has already been patched to work with cygwin, but is also giving me
some patch files which undo that and cause it to be broken again.  I
can work around that by deleting the patch files, but it doesn't seem
like that should be necessary.  Is the problem in cygport, or
setup.exe, or in my understanding of how these things are supposed to
work together?


As an aside, the reason I wanted to build coreutils from source is
that I was trying to figure out why cut.exe is so slow, even under
LANG=C.  It is slow both compared to a functionally equivalent
invocation of cygwin's awk.exe, and compared to running cut on a linux
VM.   What I found was that cut.exe is slow because the "putchar"
function is slow under cygwin.   cut.exe uses "putchar" to generate
its output, while awk.exe does not.   I'm kind of at a dead-end there,
as I can't figure out where putchar itself is implemented.  So I've
just learned to use awk.exe rather than cut.exe when I care about
performance.

Cheers,

Jeff

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: compiling coreutils with cygport

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4439 bytes --]

On 07/13/2012 03:25 PM, Jeff Janes wrote:
> If I use setup.exe to download the src for coreutils (just by checking
> the "Src?" checkbox in the gui) , and then use cygport to build it, it
> fails.  The reason it fails  seems to be that ginstall.exe does not
> have the proper manifest file so windows refuses to execute it.  This
> is the same error you get if you try to build coreutils downloaded
> from upstream gnu sources.

>   GEN    install.1
> help2man: can't get `--help' info from install.td/install

Wow.  I'm glad I build the coreutils package on my Windows XP box, where
I'm not impacted by newer Window's stupidity.

> 
> If I delete the two patch files which setup.exe automatically
> downloads into /usr/src (coreutils-8.15-1.src.patch and
> coreutils-8.15-1.cygwin.patch), and then run cygport as before, it
> compiles fine.

But then it doesn't have any cygwin-specific patches, so you aren't
getting the full coreutils experience that I have intended to package.
(Hmm, that reminds me, I'm due for a coreutils upgrade for cygwin soon)

>  Also, it works if I delete all the files which
> setup.exe downloads to /usr/src except for the
> "coreutils-8.15-1.cygport" one, and then run:
> 
> cygport coreutils-8.15-1.cygport download prep compile
> 
> So it looks like the setup.exe is giving me a coreutils tar ball which
> has already been patched to work with cygwin, but is also giving me
> some patch files which undo that and cause it to be broken again.  I
> can work around that by deleting the patch files, but it doesn't seem
> like that should be necessary.  Is the problem in cygport, or
> setup.exe, or in my understanding of how these things are supposed to
> work together?

setup.exe gives you the working tarball (which I compiled on a windows
box that doesn't need a manifest file to be built), as well as the
patches that are included in that tarball.  If you don't apply the
patches, then you aren't building the cygwin version of coreutils, at
which point, if things work, great, but it's not quite relevant here.
So the real trick is to figure out how to make the upstream make process
work with the need for manifest files during the build (rather than my
current solution of only creating manifest files after the build), and
unfortunately, since I don't have access to anything newer than XP, I
can't really test that.  I'll be relying on you (or anyone else really
bothered by this) to provide patches.

> As an aside, the reason I wanted to build coreutils from source is
> that I was trying to figure out why cut.exe is so slow, even under
> LANG=C.  It is slow both compared to a functionally equivalent
> invocation of cygwin's awk.exe, and compared to running cut on a linux
> VM.   What I found was that cut.exe is slow because the "putchar"
> function is slow under cygwin.   cut.exe uses "putchar" to generate
> its output, while awk.exe does not. 

Actually, coreutils tries to use all the *_unlocked variants of stdio,
since those have better speed in single-threaded programs (all but sort
are single-threaded, and sort skips stdio).  I do know that cygwin has
putchar_unlocked, but does not have quite as many *_unlocked variants as
Linux, so it may be a case where the configure logic to detect _unlocked
functions went wrong and cygwin ends up using putchar instead of
putchar_unlocked as a result.  I'll have to look into it more.

It may also be an efficiency thing - calling one putchar per byte is
slower than calling operations to put a line at a time, just due to the
sheer overhead of a function call.  But looking at cut.c, it really is
calling fwrite() for the bulk of a line, and saving putchar() for just
the line terminator, which means the bulk of the processing isn't
suffering from the putchar() slowdowns.

>  I'm kind of at a dead-end there,
> as I can't figure out where putchar itself is implemented.

In newlib (cygwin's libc).  http://sourceware.org/newlib/

http://sourceware.org/cgi-bin/cvsweb.cgi/src/newlib/libc/stdio/putchar_u.c?annotate=1.3&cvsroot=src

>  So I've
> just learned to use awk.exe rather than cut.exe when I care about
> performance.

Interesting observation, and may be one worth making upstream that cut.c
isn't being as efficient as possible.

-- 
Eric Blake   eblake@redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: compiling coreutils with cygport

[Corinna Vinschen]

On Jul 13 16:03, Eric Blake wrote:
> unfortunately, since I don't have access to anything newer than XP, I
> can't really test that.  I'll be relying on you (or anyone else really
> bothered by this) to provide patches.

http://msdn.microsoft.com/en-us/evalcenter/ee175713.aspx

Allows you to run 2008R2 for 180 days.  Should be enough for testing
purposes...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: compiling coreutils with cygport

[Corinna Vinschen]

On Jul 14 09:49, Corinna Vinschen wrote:
> On Jul 13 16:03, Eric Blake wrote:
> > unfortunately, since I don't have access to anything newer than XP, I
> > can't really test that.  I'll be relying on you (or anyone else really
> > bothered by this) to provide patches.
> 
> http://msdn.microsoft.com/en-us/evalcenter/ee175713.aspx
> 
> Allows you to run 2008R2 for 180 days.  Should be enough for testing
> purposes...

Alternatively, Windows 7 for 90 days:

http://technet.microsoft.com/en-US/evalcenter/cc442495.aspx


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: compiling coreutils with cygport

[Yaakov (Cygwin/X)]

On 2012-07-13 17:03, Eric Blake wrote:
> Actually, coreutils tries to use all the *_unlocked variants of stdio,
> since those have better speed in single-threaded programs (all but sort
> are single-threaded, and sort skips stdio).  I do know that cygwin has
> putchar_unlocked, but does not have quite as many *_unlocked variants as
> Linux, so it may be a case where the configure logic to detect _unlocked
> functions went wrong and cygwin ends up using putchar instead of
> putchar_unlocked as a result.  I'll have to look into it more.

Currently we have the four required by POSIX: getc_unlocked, 
getchar_unlocked, putc_unlocked, and putchar_unlocked.  A little while 
ago I was working on the others which are available in Linux, as gcc 
will use them if available, but they didn't seem to make much difference 
to gcc's performance.  I still have that work locally, so if these would 
be useful, I could try to finish them up.


Yaakov

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: compiling coreutils with cygport

[Yaakov Selkowitz]

On 2012-07-13 16:25, Jeff Janes wrote:
> If I use setup.exe to download the src for coreutils (just by checking
> the "Src?" checkbox in the gui) , and then use cygport to build it, it
> fails.  The reason it fails  seems to be that ginstall.exe does not
> have the proper manifest file so windows refuses to execute it.  This
> is the same error you get if you try to build coreutils downloaded
> from upstream gnu sources.

Right, cygport really isn't the problem here.

> cygport coreutils-8.15-1.cygport prep compile
> ....
> make[2]: Entering directory `/usr/src/coreutils-8.15-1/build/man'
>    GEN    install.1
> help2man: can't get `--help' info from install.td/install
> Makefile:1833: recipe for target `install.1' failed
> make[2]: *** [install.1] Error 126

This occurs because help2man runs ginstall to get its --help and 
--version output, but UAC doesn't allow it because of its name (but 
let's not go *there* again).  This also would occur during make install, 
where the in-tree ginstall is used as $(INSTALL).

cygport has for some time created .manifest files during postinstall, 
which has taken care of UAC issues for end users.  However, there is no 
way for cygport to know what executables are being created during the 
*build*, and then that they actually need to be run in the build tree 
(most do not).  That being said, adding this as a manual command for 
cases like this is pretty easy.

Please try cygport git master together with adding the following line to 
coreutils-*.cygport immediately *before* the cygmake command:

   manifestize src/ginstall.exe

That WFM on W7.

(Eric: BTW, while I'm at it, there's an easier way to handle DIR_COLORS:

   insinto /etc
   doins $B/src/DIR_COLORS
   make_etc_defaults /etc/DIR_COLORS

No custom postinstall/preremove necessary. :-)


Yaakov

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple