Re: Build spends a long time in "mkimport".

Re: Build spends a long time in "mkimport".

[Mark Geisert]

Kaz Kylheku (Cygwin) via Cygwin wrote:
> Hi All,
> 
> When building the Cygwin DLL, this single step takes almost ten minutes:
> 
>    ../../.././winsup/cygwin/mkimport --cpu=i686 --ar=ar --as=as --nm=nm 
> --objcopy=objcopy \
>    --replace=atexit= --replace=timezone= --replace=uname=uname_x 
> --replace=__xdrrec_getrec=
> 
>    [ .. SNIP ... ]
> 
>    --replace=truncate=_truncate64 libcygwin.a cygdll.a _cygwin_crt0_common.o \
>    atexit.o cygwin_attach_dll.o cygwin_crt0.o dll_entry.o dll_main.o dso_handle.o \
>    libcmain.o premain0.o premain1.o premain2.o premain3.o pseudo-reloc-dummy.o
> 
> What's puzzling is that there is very CPU activity during this time. It's launching
> some objcopy commands.
> 
> Is there some documentation that provides an overview of what exactly this does,
> other than studying its perl source code?
> 
> Maybe it can be sped up?
> 
> I'm going to have to cycle quite a few times on some changes, so this is frustrating.

Hi Kaz,
I'm redirecting this to the cygwin-developers list as it's a Cygwin build issue. 
Please follow up there.

I've looked at .../winsup/cygwin/mkimport for the same reason as you -- it takes 
forever on my Windows machines.  But I don't know enough perl to make any changes.

Near the end of mkimport there's a loop over all the "--replace" args, 
essentially.  For each one there are two system() calls launching two objcopy 
processes to do something.  This is much slower on Cygwin than it would be 
cross-building from Linux, for example.

One could parallelize the operations on a multicore machine.  But probably better 
would be figuring out, if possible, how to do the objcopy operations in one pass, 
over two calls if necessary.

PTC, as we've always said :-)

..mark

Re: Build spends a long time in "mkimport".

[Ken Brown]

On 10/11/2020 3:24 AM, Mark Geisert wrote:
> Kaz Kylheku (Cygwin) via Cygwin wrote:
>> Hi All,
>>
>> When building the Cygwin DLL, this single step takes almost ten minutes:
>>
>>    ../../.././winsup/cygwin/mkimport --cpu=i686 --ar=ar --as=as --nm=nm 
>> --objcopy=objcopy \
>>    --replace=atexit= --replace=timezone= --replace=uname=uname_x 
>> --replace=__xdrrec_getrec=
>>
>>    [ .. SNIP ... ]
>>
>>    --replace=truncate=_truncate64 libcygwin.a cygdll.a _cygwin_crt0_common.o \
>>    atexit.o cygwin_attach_dll.o cygwin_crt0.o dll_entry.o dll_main.o 
>> dso_handle.o \
>>    libcmain.o premain0.o premain1.o premain2.o premain3.o pseudo-reloc-dummy.o
>>
>> What's puzzling is that there is very CPU activity during this time. It's 
>> launching
>> some objcopy commands.
>>
>> Is there some documentation that provides an overview of what exactly this does,
>> other than studying its perl source code?
>>
>> Maybe it can be sped up?
>>
>> I'm going to have to cycle quite a few times on some changes, so this is 
>> frustrating.
> 
> Hi Kaz,
> I'm redirecting this to the cygwin-developers list as it's a Cygwin build issue. 
> Please follow up there.

I don't have any insight into why mkimport takes so long.  But the good news is 
that it doesn't run very often.  Most changes to winsup/cygwin files don't 
trigger the mkimport command.

Ken

Re: Build spends a long time in "mkimport".

[Jon Turney]

On 11/10/2020 08:24, Mark Geisert wrote:
> Kaz Kylheku (Cygwin) via Cygwin wrote:
>> Hi All,
>>
>> When building the Cygwin DLL, this single step takes almost ten minutes:
>>
>>    ../../.././winsup/cygwin/mkimport --cpu=i686 --ar=ar --as=as 
>> --nm=nm --objcopy=objcopy \
>>    --replace=atexit= --replace=timezone= --replace=uname=uname_x 
>> --replace=__xdrrec_getrec=
>>
>>    [ .. SNIP ... ]
>>
>>    --replace=truncate=_truncate64 libcygwin.a cygdll.a 
>> _cygwin_crt0_common.o \
>>    atexit.o cygwin_attach_dll.o cygwin_crt0.o dll_entry.o dll_main.o 
>> dso_handle.o \
>>    libcmain.o premain0.o premain1.o premain2.o premain3.o 
>> pseudo-reloc-dummy.o
>>
>> What's puzzling is that there is very CPU activity during this time. 
>> It's launching
>> some objcopy commands.
>>
>> Is there some documentation that provides an overview of what exactly 
>> this does,
>> other than studying its perl source code?

I'm afraid not.

>> Maybe it can be sped up?

Almost certainly and yes please :)

>> I'm going to have to cycle quite a few times on some changes, so this 
>> is frustrating.
> 
> Hi Kaz,
> I'm redirecting this to the cygwin-developers list as it's a Cygwin 
> build issue. Please follow up there.
> 
> I've looked at .../winsup/cygwin/mkimport for the same reason as you -- 
> it takes forever on my Windows machines.  But I don't know enough perl 
> to make any changes.
> 
> Near the end of mkimport there's a loop over all the "--replace" args, 
> essentially.  For each one there are two system() calls launching two 

I think this perhaps is looping over all the exported symbols.

> objcopy processes to do something.  This is much slower on Cygwin than 
> it would be cross-building from Linux, for example.

What I think mkimport is doing:  This produces the import library for 
the Cygwin DLL, which is actually a 'hybrid' library.  Mainly it 
consists of import stubs for functions in the DLL (some of which are 
renamed by --replace options), but it also contains some specific object 
files directly (this is stuff called by crt0.o, but idk why it's can't 
be in the DLL).

If it's actually the objcopy which is slow, then it would be nice to 
understand why, although they could also be parallelized.

Re: Build spends a long time in "mkimport".

[Corinna Vinschen]

On Oct 11 18:14, Jon Turney wrote:
> What I think mkimport is doing:  This produces the import library for the
> Cygwin DLL, which is actually a 'hybrid' library.  Mainly it consists of
> import stubs for functions in the DLL (some of which are renamed by
> --replace options), but it also contains some specific object files directly
> (this is stuff called by crt0.o, but idk why it's can't be in the DLL).

Some stuff must be statically linked into the executable to work.
For instance, see the stuff lib/_cygwin_crt0_common.cc sets up so
it will be available when calling into the DLL the first time.
Another heck of an example is lib/atexit.c with really detailed
comments in lib/atexit.c and dcrt0.cc.


Corinna