RE: ordinal linking for cygwin ld

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de] 
> Sent: Saturday, April 27, 2002 10:55 AM

> > > Or ld has a switch to explicit use ordinals (see other mails from 
> > > me)
> >
> > I don't see what such a switch gains. The hint ordinal 
> should provide 
> > the same performance as an ordinal-only-link.
> 
> By default ld does not use ordinals or linking. Only when an 
> import lib does not contain names in the _nm_vector, ld takes 
> the ordinal for linking (identified by flag_noname in 
> pe-dll.c/ei386pe.c). Thats what I have used. I have patched 
> ld to write the import library only with ordinals and ld does 
> the rest. Only an additional patch for auto-imported date was 
> necessary. (see the patch)

Well then, this is only half the puzzle. I can see what you gain from
such a patch, but as Chuck as indicated, it will cause -major-
difficulties in management. 

A patch to use hint ordinals when linking by name would be _very_ useful
though, as that would
a) give the performance benefit you are looking for
b) allow backward compatible library versioning as link-by-name does.

We'd probably also need to ensure that strip leaves the names in the IAT
(I wasn't clear from your other email whether it does that or not).

Rob

RE: ordinal linking for cygwin ld

[Ralf Habacker]

>A patch to use hint ordinals when linking by name would be _very_ useful
>though, as that would
>a) give the performance benefit you are looking for
>b) allow backward compatible library versioning as link-by-name does.

But as I stated already, this needs an ld switch to enable/disable ordinal
linking and allow limiting of used libs or path

for example

--enable-ordinal-link[=path1[,path2]]

	enable ordinal linking for given path/files or unlimited ordinal linking if no
path is given.

--disable-ordinal-link[=path1[,path2]]

	disable ordinal linking for the following path/files

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> Well then, this is only half the puzzle. I can see what you gain from
> such a patch, but as Chuck as indicated, it will cause -major-
> difficulties in management.

Do you have read the rules I have stated for kde ?

> A patch to use hint ordinals when linking by name would be _very_ useful
> though, as that would
> a) give the performance benefit you are looking for
> b) allow backward compatible library versioning as link-by-name does.
>
> We'd probably also need to ensure that strip leaves the names in the IAT
> (I wasn't clear from your other email whether it does that or not).
>
Yes it does. See some parts about my auto-import documentation.

$ objdump -x client.exe

<snip>
The Import Tables (interpreted .idata section contents)


  vma:            Hint    Time      Forward  DLL       First
                  Table   Stamp     Chain    Name      Thunk
(1) 00004000     00004068 00000000 00000000 000041a8 000040b4
                   |(3)                                   |(2)
                   |            _nm_....  (4)        IAT  |  _imp_...
                +------+        +------------+        +------+
                |  (6)----+---->| Hint 16bit |(7)<------     |
                +------+  |     +------------+        +------+
                |      |  |     | membername |(8)     +------+
                +------+  |     +------------+        +------+
                          |                           +------+
                          |      _nm_...  (5)         +------+
                          |     +-------------+       +------+
                          +---->|Ordinal 16bit|(7)    +------+
                                +-------------+
                                | 0x00 0x80   |
                                +-------------+

(1) is the IMAGE_IMPORT_DESCRIPTOR.

FirstThunk (2) points to the import address table (IAT), which entries are
identifed by corresponding _imp_... symbols.

HintTab (3) point to a rva (relative virtual address = offset from image start)
vector pointing to IMAGE_IMPORT_BY_NAME unions.

(4) shows an IMAGE_IMPORT_BY_NAME union for a named link. It is identifed by the
_nm_<smbol> name .
In this structure the symbol name string is stored independed from the regular
symbol name, which can be only stripped.

(5) shows a IMAGE_IMPORT_BY_NAME union for a ordinal link.

Now see the relating objdump part. The numbers 6,7,8 identify the related
structure element.

        DLL Name: dll.dll
        vma:  Hint/Ord Member-Name
         (6)       (7)     (8)
        40f4        0  printfunc0000__1Av
        410c        1  var0000


00404290 <__nm__var0000>:
  404290:	01 00
  404292:	76 61
  404294:	72 30
  404296:	30 30
  404298:	30 00


No see this for an auto-imported data

  vma:            Hint    Time      Forward  DLL       First
                  Table   Stamp     Chain    Name      Thunk
 00004000       00004068 00000000 00000000 000041a8 00001064
                   |(3)                                 |(2)
              nm_thnk_...       _nm_...  (4)            |
                +------+        +------------+        __fu0_...
                |  (6)----+---->| Hint 16bit |(7)     points directly into the
text segement
                +------+  |     +------------+        to the relating
instructions address
                |      |  |     | Membername |(8)
                +------+  |     +------------+
                          |
                          |     _nm_...  (5)
                          |     +-------------+
                          +---->|Ordinal 16bit|(7)
                                +-------------+
                                | 0x00 0x80   |
                                +-------------+

It's the same as above except

(2) points directly into the text segement to the relating instructions address.
    The target is identified by the __fu0_<symbol> name. See a sample below.

    0xa1 is the instruction at offset 0x401063, the address starts at 0x0401063.

  401063:	a1 b8 40 40 00       	mov    0x4040b8,%eax
               ^^^^^^^^^^^
00401064 <__fu0__var0000>:
      ^^

(3) HintTab is identifed by the _nm_thnk_<symbol>.


004040a8 <__nm_thnk__var0000>:
  4040a8:       02 00
  4040aa:       00 80 00 00 00 00


Hope that helps
Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

>
>  00004014       00004070 ffffffff ffffffff 00004230 000040a8
>
>         DLL Name: cygwin1.dll
>         vma:  Hint/Ord Member-Name
>         4154       26  __main   61004308
>         4160      581  calloc   0000
>         416c      634  cygwin_internal  0000
>         4180      652  dll_crt0__FP11per_process        0000
>         419c      722  free     0000
>         41a4      876  malloc   0000
>         41b0     1006  realloc  0000
> ----------------------------------------------------------------------
> ----------
> ------
>                                 ^^^^^^
> Do you see this ? The first is set correct, the other set to zero.
>
The problem is fixed. It was caused by the Callbackfunction of BindImageEx.
Using BindImage works.

If anyone is interessed to see rebind by the cygwin utils, let me know, so I can
prepare a full cygwin application.

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> a self
> > written rebind app and got a problem.
> >
> > rebind does only patch the first IAT entry, if the dll is created with
> ld. The
> > others are set to zero ????
> >
> > Rebinding for example the cygwin1.dll to some natvie windows apps
> works, so this
> > seems to be an ld incompatiblity.
> >
> > Does anyone have got experience in such binding stuff ?
>
> No, but perhaps it's the auto-import duplicates that cause the problem?

No. I have checked this by a dll without vars. See below:

The Import Tables (interpreted .idata section contents)
 vma:            Hint    Time      Forward  DLL       First
                 Table   Stamp     Chain    Name      Thunk
 00004000       00004054 00000000 00000000 0000420c 00004074

        DLL Name: dll.dll
        vma:  Hint/Ord Member-Name
        40dc        0  printfunc0000__1Av
        40f4        1  printfunc0001__1Av
        410c        2  printfunc0002__1Av
        4124        3  printfunc0003__1Av
        413c        4  printfunc0004__1Av

 00004014       00004070 00000000 00000000 00004230 000040a8

        DLL Name: cygwin1.dll
        vma:  Hint/Ord Member-Name
        4154       26  __main
        4160      581  calloc
        416c      634  cygwin_internal
        4180      652  dll_crt0__FP11per_process
        419c      722  free
        41a4      876  malloc
        41b0     1006  realloc
        The Import Address Table is identical

 00004028       00004094 00000000 00000000 00004248 000040cc

        DLL Name: KERNEL32.dll
        vma:  Hint/Ord Member-Name
        41bc      248  GetCurrentProcess
        41d0      296  GetModuleHandleA
        41e4      661  TerminateProcess
        The Import Address Table is identical

--------------------------------------------------------------------------------
------
rebinding client.exe
--------------------------------------------------------------------------------
------
rebind client.exe ./
rebind - message: client.exe dll.dll Module import is starting.
rebind - message: client.exe dll.dll Procedure import is starting.
printfunc0000__1Av
rebind - message: client.exe dll.dll Procedure import failed. printfunc0000__1Av

He has patched, which you can see below, but break after this.

rebind - message: client.exe cygwin1.dll Module import failed. -> not found.
okay
rebind - message: client.exe KERNEL32.dll Module import failed. -> not found.
okay
rebind - message: client.exe (null) Image modified.
BIND: Details of binding client.exe
    Import from dll.dll [6]
                    ^^^^^^^^^
    Import from cygwin1.dll [0]
    Import from KERNEL32.dll [0]
rebind 2
--------------------------------------------------------------------------------
------
The Import Tables (interpreted .idata section contents)

 vma:            Hint    Time      Forward  DLL       First
                 Table   Stamp     Chain    Name      Thunk
 00004000       00004054 ffffffff ffffffff 0000420c 0004074

        DLL Name: dll.dll
        vma:  Hint/Ord Member-Name
        40dc        0  printfunc0000__1Av       0001
        40f4        1  printfunc0001__1Av       0004
        410c        2  printfunc0002__1Av       0000
        4124        3  printfunc0003__1Av       8000
        413c        4  printfunc0004__1Av       0400
--------------------------------------------------------------------------------
------
rebinding cygwin1.dll
--------------------------------------------------------------------------------
------
$ rebind ./client.exe /bin
rebind client.exe c:\programme\cygwin\bin
rebind - message: client.exe dll.dll Module import is starting.
rebind - message: client.exe dll.dll Procedure import is starting.
printfunc0000__1Av
rebind - message: client.exe dll.dll Procedure import failed. printfunc0000__1Av
rebind - message: client.exe cygwin1.dll Module import is starting.
rebind - message: client.exe c:\programme\cygwin\bin\cygwin1.dll Procedure
import is starting. __main
rebind - message: client.exe c:\programme\cygwin\bin\cygwin1.dll Procedure
import failed. __main
rebind - message: client.exe KERNEL32.dll Module import failed.
rebind - message: client.exe (null) Image modified.
BIND: Details of binding client.exe
    Import from dll.dll [6]
    Import from cygwin1.dll [6]
    Import from KERNEL32.dll [0]
--------------------------------------------------------------------------------
------

 00004014       00004070 ffffffff ffffffff 00004230 000040a8

        DLL Name: cygwin1.dll
        vma:  Hint/Ord Member-Name
        4154       26  __main   61004308
        4160      581  calloc   0000
        416c      634  cygwin_internal  0000
        4180      652  dll_crt0__FP11per_process        0000
        419c      722  free     0000
        41a4      876  malloc   0000
        41b0     1006  realloc  0000
--------------------------------------------------------------------------------
------
                                ^^^^^^
Do you see this ? The first is set correct, the other set to zero.

The masin difference I see currently are differences in the data directory
relating entry c, which is defined on native dll's and not for ld builded dll#s

ChkRes2K.dll
The Data Directory
Entry 0 00000000 00000000 Export Directory [.edata (or where ever we found it)]
Entry 1 00013fe0 0000008c Import Directory [parts of .idata]
Entry 2 0001b000 00003a00 Resource Directory [.rsrc]
Entry 3 00000000 00000000 Exception Directory [.pdata]
Entry 4 00000000 00000000 Security Directory
Entry 5 00000000 00000000 Base Relocation Directory [.reloc]
Entry 6 00000000 00000000 Debug Directory
Entry 7 00000000 00000000 Description Directory
Entry 8 00000000 00000000 Special Directory
Entry 9 00000000 00000000 Thread Storage Directory [.tls]
Entry a 00000000 00000000 Load Configuration Directory
Entry b 00000000 00000000 Bound Import Directory
Entry c 00011000 00000394 Import Address Table Directory
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Entry d 00000000 00000000 Delay Import Directory
Entry e 00000000 00000000 Reserved
Entry f 00000000 00000000 Reserved


my client.exe

The Data Directory
Entry 0 00000000 00000000 Export Directory [.edata (or where ever we found it)]
Entry 1 00004000 00000258 Import Directory [parts of .idata]
Entry 2 00000000 00000000 Resource Directory [.rsrc]
Entry 3 00000000 00000000 Exception Directory [.pdata]
Entry 4 00000000 00000000 Security Directory
Entry 5 00000000 00000000 Base Relocation Directory [.reloc]
Entry 6 00000000 00000000 Debug Directory
Entry 7 00000000 00000000 Description Directory
Entry 8 00000000 00000000 Special Directory
Entry 9 00000000 00000000 Thread Storage Directory [.tls]
Entry a 00000000 00000000 Load Configuration Directory
Entry b 00000000 00000000 Bound Import Directory
Entry c 00000000 00000000 Import Address Table Directory
         ^^^^^^^^^^^^^^
Entry d 00000000 00000000 Delay Import Directory
Entry e 00000000 00000000 Reserved
Entry f 00000000 00000000 Reserved


after rebinding client.exe

The Data Directory
Entry 0 00000000 00000000 Export Directory [.edata (or where ever we found it)]
Entry 1 00004000 00000258 Import Directory [parts of .idata]
Entry 2 00000000 00000000 Resource Directory [.rsrc]
Entry 3 00000000 00000000 Exception Directory [.pdata]
Entry 4 00000000 00000000 Security Directory
Entry 5 00000000 00000000 Base Relocation Directory [.reloc]
Entry 6 00000000 00000000 Debug Directory
Entry 7 00000000 00000000 Description Directory
Entry 8 00000000 00000000 Special Directory
Entry 9 00000000 00000000 Thread Storage Directory [.tls]
Entry a 00000000 00000000 Load Configuration Directory
Entry b 00000268 00000044 Bound Import Directory
           ^^^^^^^^^^^^^^^^
Entry c 00000000 00000000 Import Address Table Directory
Entry d 00000000 00000000 Delay Import Directory
Entry e 00000000 00000000 Reserved
Entry f 00000000 00000000 Reserved

Ralf

Re: ordinal linking for cygwin ld

[Robert Collins]

===
----- Original Message -----
From: "Ralf Habacker" <Ralf.Habacker@freenet.de>
To: "Robert Collins" <robert.collins@itdomain.com.au>
Cc: <kde-cygwin@mail.kde.org>; "Binutils" <binutils@sources.redhat.com>;
"Cygwin-Apps" <cygwin-apps@cygwin.com>
Sent: Tuesday, April 30, 2002 12:00 AM
Subject: RE: ordinal linking for cygwin ld


> > One could be found at
> > http://www.geocities.com/shewitt_au/speedload_files/speedload.html
> >
>
> I have tried to rebind ld created apps and applications with this and
a self
> written rebind app and got a problem.
>
> rebind does only patch the first IAT entry, if the dll is created with
ld. The
> others are set to zero ????
>
> Rebinding for example the cygwin1.dll to some natvie windows apps
works, so this
> seems to be an ld incompatiblity.
>
> Does anyone have got experience in such binding stuff ?

No, but perhaps it's the auto-import duplicates that cause the problem?

Rob

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> One could be found at
> http://www.geocities.com/shewitt_au/speedload_files/speedload.html
>

I have tried to rebind ld created apps and applications with this and a self
written rebind app and got a problem.

rebind does only patch the first IAT entry, if the dll is created with ld. The
others are set to zero ????

Rebinding for example the cygwin1.dll to some natvie windows apps works, so this
seems to be an ld incompatiblity.

Does anyone have got experience in such binding stuff ?

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

>
> > So linking by ordinal only will help you a little. rebinding and
> > rebasing your .dll's will help much much more.
>
> .. which has to be analysed. Has anyone a working bind app ?


One could be found at
http://www.geocities.com/shewitt_au/speedload_files/speedload.html

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> Ok, I see what it does. Doesn't it have to do that for each reference to
> the auto-imported variable?

For each auto-imported variable an IMAGE_IMPORT_DESCRIPTOR and an
IMPORT_THUNK_DATA element is necessary.

> If so, then heavy use of an imported variable could make the
> INT and IAT quite large - and lots of fixups
> needed.

Yes, these are the costs.

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> > Mostly. I'm a bit rusty - it's been a while since I grokked the
> > auto-import stuff. I'm getting back into it at the moment. The thing
> > that I don't follow at the moment is the how the linker fixup places the
> > exported data -variable- at a fixed rva at dll load time. The IAT is
> > -meant- to point straight to functions after the fixup occurs, so
> > pointing at an address in the .dll .text segment is fine.
                                  ^^^^
>
> Not into the dll, into the client code.

I was wrong. ... points to a data variable address located in the dll, which may
be in the .text or .data or .bss segment of the dll, but the patched address
lies in the clients .text segment.

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de] 
> Sent: Sunday, April 28, 2002 10:18 PM

> Firstthunk points to 0x401064, the symbol <__fu0__var0000>, 
> which is the address part of the mov instruction. After run 
> time linking the loader has relocated this address to the 
> propper value.

Ok, I see what it does. Doesn't it have to do that for each reference to
the auto-imported variable? If so, then heavy use of an imported
variable could make the INT and IAT quite large - and lots of fixups
needed. However, binding could still work, as we effectively have a
pointer to each address reference in the binary.

Rob

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> Mostly. I'm a bit rusty - it's been a while since I grokked the
> auto-import stuff. I'm getting back into it at the moment. The thing
> that I don't follow at the moment is the how the linker fixup places the
> exported data -variable- at a fixed rva at dll load time. The IAT is
> -meant- to point straight to functions after the fixup occurs, so
> pointing at an address in the .dll .text segment is fine.
                                 ^^^^

Not into the dll, into the client code.

objdump -D client.exe
<snip>

401063:	a1 b8 40 40 00       	mov    0x4040b8,%eax
             | ^^^^^^^^^^
          opcode     address
00401064 :

Firstthunk points to 0x401064, the symbol <__fu0__var0000>, which is the address
part of the mov instruction. After run time linking the loader has relocated
this address to the propper value.

401063:	a1 ab cd ef 00       	mov    0x00efcdab,%eax

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> > 
> > > Well then, this is only half the puzzle. I can see what you 
> > gain from 
> > > such a patch, but as Chuck as indicated, it will cause -major- 
> > > difficulties in management.
> > 
> > Do you have read the rules I have stated for kde ?
> 
> Yes.
>  
> > > We'd probably also need to ensure that strip leaves the 
> > names in the 
> > > IAT (I wasn't clear from your other email whether it does that or 
> > > not).
> > >
> > Yes it does. See some parts about my auto-import documentation.
> 
> Cool.
>  
> > $ objdump -x client.exe
> > FirstThunk (2) points to the import address table (IAT), 
> > which entries are identifed by corresponding _imp_... symbols.
> 
> The IAT is an array of IMAGE_THUNK_DATA, that gets overwritten by the
> physical PE header of the linked image by the win32 .exe load process.
>  
yes
> > HintTab (3) point to a rva (relative virtual address = offset 
> > from image start) vector pointing to IMAGE_IMPORT_BY_NAME unions.
> 
> Huh? The Hinttab->hintname array is an array of IMAGE_THUNK_DATA. 

yes. The entries poing to the IMAGE_IMPORT_BY_NAME. 

> > (4) shows an IMAGE_IMPORT_BY_NAME union for a named link. It 
> > is identifed by the _nm_<smbol> name . In this structure the 
> > symbol name string is stored independed from the regular 
> > symbol name, which can be only stripped.
> 
> Hmm. the diagram above is a little misleading. The IMAGE_IMPORT_BY_NAME
> struct is not a union. It's IMAGE_THUNK_DATA that is a union, and for
> named imports contains that RVA of the IMAGE_IMPORT_BY_NAME struct,
> which has both the ordinal and the member name. Lastly there is _no_
> requirement for the INT and IAT to share IMAGE_IMPORT_BY_NAME structs,
> and they definitely won't share IMAGE_THUNK_DATA elements.

Thank for this hint. In this case I was wrong. 
 

Ralf 

RE: ordinal linking for cygwin ld

[Ralf Habacker]

Ralf Habacker


> -----Original Message-----
> From: kde-cygwin-admin@mail.kde.org
> [mailto:kde-cygwin-admin@mail.kde.org]On Behalf Of Robert Collins
> Sent: Saturday, April 27, 2002 8:44 PM
> To: Ralf Habacker; Charles Wilson
> Cc: Kde-Cygwin; Binutils; Cygwin-Apps
> Subject: RE: ordinal linking for cygwin ld
>
>
>
>
> > -----Original Message-----
> > From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de]
> > Sent: Sunday, April 28, 2002 4:28 AM
> > >
> > > Hangon, lets go back a bit. Why do you want ordinal-only
> > linking? For

> > > runtime or link-time performance?

> Or for on-disk import > > library size?
Besides the faster load time associated with importing and exporting by ordinal,
there's another more subtle benefit. When exporting an API by ordinal, you can
tell the linker not to store the exported API name in the exporting DLL. This
means a smaller export section, and potentially a smaller binary with less data
to demand page in. To eliminate the API name, use the NONAME modifier when
exporting by ordinal.
....
If you look at MFC42.DLL, you'll see that it exports almost all of its 6000+
APIs by ordinal, and with the NONAME modifier. Imagine the added bulk if
MFC42.DLL had to store all 6000 mangled C++ names in its exports!

> > > Or....
> > runtime performance
>
> Ok, well then check this :
> http://msdn.microsoft.com/msdnmag/issues/0500/hood/hood0500.asp article
> out.
>


> Linking by ordinal only gives a 2%-4% improvement. rebinding and
> rebasing gives up to 18%.
>

from this article:
" Before getting to the actual numbers, let me first say that I was amazed at
how fast programs can load. I intentionally created LoadTimeTest.EXE to make a
lot of work for the Win32 loader. It has a fair number of DLLs and lots of
exported functions and relocations. Even under the slowest scenario, my machine
still loaded the program under Windows 2000 in less than 1/50th of a second.

If your program takes a long time to load, don't blame the loader. The problem
is almost certainly that somebody's initialization code is taking too long.

He does testing with short named c functions. C++ function names are much
longer. Additional the problem is the first loading, calling again works fast in
any way. (Except of the binary size) See below.

<snip>
make shortnames   -> 13 character function names
$ time ./client.exe
real    0m1.392s
user    0m0.010s
sys     0m0.000s

$ time ./client.exe
real    0m0.032s
user    0m0.020s
sys     0m0.010s

make longnames    -> 403 character function names
$ time ./client.exe

real    0m6.990s
user    0m0.020s
sys     0m0.000s

habacker@BRAMSCHE ~/src/ld-contrib/ld-bugfi
$ time ./client.exe

real    0m0.065s
user    0m0.020s
sys     0m0.000s


5000 functions with symbol name linking (symbols not stripped)

$ time ./client.exe
real    0m15.182s
user    0m0.060s
sys     0m0.040s

5000 functions with ordinal linking (symbols not stripped)

$ time ./client.exe
real    0m0.572s
user    0m0.020s
sys     0m0.030s

habacker@BRAMSCHE ~/src/ld-contrib/ld-bugfix-test
$ ls *.exe *.dll -l
-rwxrwxrwx    1 1002     Kein      2188288 Apr 28 14:00 client.exe
-rwxrwxrwx    1 1002     Kein       128512 Apr 28 14:00 clientordinal.exe
-rwxrwxrwx    1 1002     Kein      2273280 Apr 28 14:00 dll.dll
-rwxrwxrwx    1 1002     Kein      2273280 Apr 28 14:00 dllordinal.dll

In the dll avoiding the IMAGE_IMPORT_BY_NAME structure would save 2MB in the
dll.

BTW: Do you know why the exports in the dll are necessary ? Does the implib not
contain this stuff already, so they could be removed ?

> So linking by ordinal only will help you a little. rebinding and
> rebasing your .dll's will help much much more.

.. which has to be analysed. Has anyone a working bind app ?

Ralf

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de] 
> Sent: Sunday, April 28, 2002 4:28 AM
> > 
> > Hangon, lets go back a bit. Why do you want ordinal-only 
> linking? For 
> > runtime or link-time performance? Or for on-disk import 
> library size? 
> > Or....
> runtime performance 

Ok, well then check this :
http://msdn.microsoft.com/msdnmag/issues/0500/hood/hood0500.asp article
out.

Linking by ordinal only gives a 2%-4% improvement. rebinding and
rebasing gives up to 18%.

So linking by ordinal only will help you a little. rebinding and
rebasing your .dll's will help much much more.

And infact, rebasing and rebinding makes the INT obsolete - because the
IAT is precalculated, so you can't get 22% by linking by ordinal and
rebinding and rebasing.

Rob

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> >A patch to use hint ordinals when linking by name would be _very_ 
> > >useful though, as that would
> > >a) give the performance benefit you are looking for
> > >b) allow backward compatible library versioning as link-by-name does.
> > 
> > But as I stated already, this needs an ld switch to 
> > enable/disable ordinal linking and allow limiting of used libs or path
> 
> Hangon, lets go back a bit. Why do you want ordinal-only linking? For
> runtime or link-time performance? Or for on-disk import library size?
> Or....
runtime performance 
> Rob
> 

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de] 
> Sent: Saturday, April 27, 2002 10:47 PM
> To: kde-cygwin@mail.kde.org; Charles Wilson; Robert Collins
> Cc: Binutils; Cygwin-Apps
> Subject: RE: ordinal linking for cygwin ld
> 
> 
> > Well then, this is only half the puzzle. I can see what you 
> gain from 
> > such a patch, but as Chuck as indicated, it will cause -major- 
> > difficulties in management.
> 
> Do you have read the rules I have stated for kde ?

Yes.
 
> > We'd probably also need to ensure that strip leaves the 
> names in the 
> > IAT (I wasn't clear from your other email whether it does that or 
> > not).
> >
> Yes it does. See some parts about my auto-import documentation.

Cool.
 
> $ objdump -x client.exe
> FirstThunk (2) points to the import address table (IAT), 
> which entries are identifed by corresponding _imp_... symbols.

The IAT is an array of IMAGE_THUNK_DATA, that gets overwritten by the
physical PE header of the linked image by the win32 .exe load process.
 
> HintTab (3) point to a rva (relative virtual address = offset 
> from image start) vector pointing to IMAGE_IMPORT_BY_NAME unions.

Huh? The Hinttab->hintname array is an array of IMAGE_THUNK_DATA. 
 
> (4) shows an IMAGE_IMPORT_BY_NAME union for a named link. It 
> is identifed by the _nm_<smbol> name . In this structure the 
> symbol name string is stored independed from the regular 
> symbol name, which can be only stripped.

Hmm. the diagram above is a little misleading. The IMAGE_IMPORT_BY_NAME
struct is not a union. It's IMAGE_THUNK_DATA that is a union, and for
named imports contains that RVA of the IMAGE_IMPORT_BY_NAME struct,
which has both the ordinal and the member name. Lastly there is _no_
requirement for the INT and IAT to share IMAGE_IMPORT_BY_NAME structs,
and they definitely won't share IMAGE_THUNK_DATA elements.

> (5) shows a IMAGE_IMPORT_BY_NAME union for a ordinal link.

Again, this is wrong, it stores a IMAGE_THUNK_DATA union.
 
> No see this for an auto-imported data
> 
...

> Hope that helps

Mostly. I'm a bit rusty - it's been a while since I grokked the
auto-import stuff. I'm getting back into it at the moment. The thing
that I don't follow at the moment is the how the linker fixup places the
exported data -variable- at a fixed rva at dll load time. The IAT is
-meant- to point straight to functions after the fixup occurs, so
pointing at an address in the .dll .text segment is fine. I'll re-read
Paul's explanation when I get to looking at your patch anyway.

Cheers,
Rob

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de] 
> Sent: Saturday, April 27, 2002 10:17 PM
> To: Robert Collins; Charles Wilson
> Cc: Kde-Cygwin; Binutils; Cygwin-Apps
> Subject: RE: ordinal linking for cygwin ld
> 
> 
> >A patch to use hint ordinals when linking by name would be _very_ 
> >useful though, as that would
> >a) give the performance benefit you are looking for
> >b) allow backward compatible library versioning as link-by-name does.
> 
> But as I stated already, this needs an ld switch to 
> enable/disable ordinal linking and allow limiting of used libs or path

Hangon, lets go back a bit. Why do you want ordinal-only linking? For
runtime or link-time performance? Or for on-disk import library size?
Or....

Rob

RE: ordinal linking for cygwin ld

[Ralf Habacker]

Ralf Habacker


> -----Original Message-----
> From: binutils-owner@sources.redhat.com
> [mailto:binutils-owner@sources.redhat.com]On Behalf Of Robert Collins
> Sent: Saturday, April 27, 2002 2:26 AM
> To: Ralf Habacker; Charles Wilson
> Cc: Kde-Cygwin; Binutils; Cygwin-Apps
> Subject: RE: ordinal linking for cygwin ld
>
>
>
>
> > -----Original Message-----
> > From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de]
> > Sent: Saturday, April 27, 2002 10:23 AM
> > To: Robert Collins; Charles Wilson
> > Cc: Kde-Cygwin; Binutils; Cygwin-Apps
> > Subject: RE: ordinal linking for cygwin ld
> >
> >
> > > The PE spec (as I read it) indicates that as long as a name is
> > > included (ie it's not link-only-by-ordinal) then ordinals
> > can change
> > > and nothing will break.
> > >
> > > It's only when the only link information is the ordinal
> > that problems
> > > will appear.
> >
> > Or ld has a switch to explicit use ordinals (see other mails from me)
>
> I don't see what such a switch gains. The hint ordinal should provide
> the same performance as an ordinal-only-link.

By default ld does not use ordinals or linking. Only when an import lib does not
contain names in the _nm_vector, ld takes the ordinal for linking (identified by
flag_noname in pe-dll.c/ei386pe.c).
Thats what I have used. I have patched ld to write the import library only with
ordinals and ld does the rest.
Only an additional patch for auto-imported date was necessary. (see the patch)

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

Ld rules: 
> 1. By default ordinal linking is disabled 
> 
> 2. Add an ld option to enable ordinal linking. 
>    As ordinal the hint number is used. (Could this have any unknown 
> side effect ??) 
>    ordinal = hint number + 1. 
>    How should such an option be named ? 

--enable-ordinal-link ? 

Now, I'm going to bed, it is 2:43am in germany. God night. 

Ralf 

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> > OTOH, if you've linked by ordinal, and then you strip symbols 
> > -- are the 
> > names of the imports still retained?
> 
The symbols are removed, but in the _nm_vector the names are still retained. 

Ralf 

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Ralf Habacker [mailto:Ralf.Habacker@freenet.de] 
> Sent: Saturday, April 27, 2002 10:23 AM
> To: Robert Collins; Charles Wilson
> Cc: Kde-Cygwin; Binutils; Cygwin-Apps
> Subject: RE: ordinal linking for cygwin ld
> 
> 
> > The PE spec (as I read it) indicates that as long as a name is 
> > included (ie it's not link-only-by-ordinal) then ordinals 
> can change 
> > and nothing will break.
> > 
> > It's only when the only link information is the ordinal 
> that problems 
> > will appear.
> 
> Or ld has a switch to explicit use ordinals (see other mails from me)

I don't see what such a switch gains. The hint ordinal should provide
the same performance as an ordinal-only-link.
Rob

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> The PE spec (as I read it) indicates that as long as a name is included
> (ie it's not link-only-by-ordinal) then ordinals can change and nothing
> will break.
> 
> It's only when the only link information is the ordinal that problems
> will appear.

Or ld has a switch to explicit use ordinals (see other mails from me)

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> 
> If *you* release new "compatible" libs with the ordinals different from 
> the current libs, *my* application breaks.  Or, you might get ripple 
> effects: what if I distribute a dll that depends on KDE's libs, and Bob 
> has an app that depends on my dll?  Bob's app breaks because my dll is 
> broken because you released new "compatible" libs.  Not good -- and it'd 
> take a while to track the problem down too: (a) I need to relink my dll, 
> and give it to Bob, and then (and only then) (b) Bob can relink his app.
> 
> Blech.

Wait for the rules in my next mail. 
> 
> >>I shudder to think of maintaining all of that for the 200-odd DLLs that
> >>come with KDE.
> >>
> > 
> > Using def files is unusable.
> 
> 
> Unless it's the only way to avoid the problems above.   Then, you ignore 
> the pain and just do it -- or write code to do it for you, like the 
> openoffice guys have done.
> 
> 
> >>Then, there's one smaller problem:
> >>
> >>If you use a .def file to generate your DLL, will the auto-import stuff
> >>get added to it?  Can auto-import even work, if you're linking by
> >>ordinal -- I thought the _nm_ hints were based on the symbol name,
> >>
> > 
> > no, the symbol name was build from the undef section, if a corresponding
> > _imp_symbol was found, that means from the client side. There was no access
> > (except of the dll name) to the implibs data.
> 
> 
> Okay...
> 
> 
> >>not the symbol ordinal...
> >>
> > 
> > I have patched auto-import stuff which got the ordinal from the implib.
> > It works with the appended testcase. Currently I've build a new 
> working qt with
> > this.
> 
> 
> > 
> > In the second solution I have told about, using the regular implibs 
> and decide
> > on a per case base
> > if ordinal linking or not.
> > 
> > I'm thinking about creating to areas, an internal and an external.
> > New releases of kdelibs and perhaps kdebase for example are build 
> together. So
> > ordinal linking is not problem. -> internal area. Any other app may 
> be linked by
> > name -> external area.
> 
> 
> Ah, so there'd be a special switch that KDE's internal makefiles use to 
> turn on 'link by ordinal' behavior; other applications would not use it, 
> and would thus link by name?  (or, they'd use it and link-by-ordinal at 
> their own risk...)
> 
> That seems workable to me.
> 
>  
> > Second some dll's are only for specific apps for example konqueror 
> or kcontrol,
> > which holds there code in some dll's. When rebuilding konqueror  
> the dll's could
> > be linked ordinal without any problems, because there were all rebuild.
> 
> 
> Errr...not so fast.  Code reuse -- konqueror dlls can be linked into 
> other apps, can't they? -- or is all code reuse of that nature in KDE 
> done via KParts/Bonobo/CORBA/whatever, and not actual linking?
> 
We will have to look at this a little deeper. 

Ralf 

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> In the second solution I have told about, using the regular implibs and decide
> on a per case base
> if ordinal linking or not.
>
> I'm thinking about creating to areas, an internal and an external.
> New releases of kdelibs and perhaps kdebase for example are build together. So
> ordinal linking is not problem. -> internal area. Any other app may
> be linked by name -> external area.

If there are some rules, I think it is possible to manage this stuff.

1.  Each KDE major release 2, 3 is to see as a single
    release with a fixed ordinal linking schema.
    This is archived by the kde-team, who said,
    that the kdelibs api is fixed for each major release
    except some enhancements, which are added.

2.  kde packages are arranged into two groups, identified by
    the linking schema (ordinal or named)

3.  packages, which belongs to the ordinal group needs special
    attention on basic libs updates.

4.  packages, which belongs to the named group are less affected
    by updated basic libs. Relinking is only need by a major release
    change.

5.  For each KDE major release the used ld release is fixed,
    except of updates not affecting the bfd/ld binary layout.
    Otherwise all packages of the ordinal group has to be relinked
    and released.

6. The packages of the ordinal group (and some of the named group)
   are maintained by the kde-cygwin team.

7. If an external one likes to maintain a package of the ordinal group
   he is responsible to follow topic 5.

8. Currently the following package belongs to the ordinal group:
     kdelibs, kdebase, kdevelop

9. basic library which should not be referenced ordinal, must not
   have import libs with ordinals.
   (This speeks against using ordinal in importlibs by default)


Ld rules:
1. By default ordinal linking is disabled

2. Add an ld option to enable ordinal linking.
   As ordinal the hint number is used. (Could this have any unknown side effect
??)
   ordinal = hint number + 1.
   How should such an option be named ?

Any comments ?

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

> > If you use a .def file to generate your DLL, will the auto-import stuff
> > get added to it?  Can auto-import even work, if you're linking by
> > ordinal -- I thought the _nm_ hints were based on the symbol name,
> 
> no, the symbol name was build from the undef section, if a corresponding
> _imp_symbol was found, that means from the client side. There was no access
> (except of the dll name) to the implibs data.
> 
Of course are the sections of the implib imported in the resulting binary, 
but currently I have an idea, how to save additional space in the implib. 
(for example the qt2.3.1 implib is about 8MB and contains 12079 symbols) 
At least each symbol name exist twice. Perhaps this can be put together 

d012073.o:
00000000 i .idata$4
00000000 i .idata$5
00000000 i .idata$6
00000000 i .idata$7
00000000 t .text
         U __head_qt_2_3_dll
(a) 00000000 I __imp__y__C13QIconViewItem
(b) 00000000 T _y__C13QIconViewItem

->(a) __imp__y__C13QIconViewItem
            ^ 
  (b)  points to (a) + offset 

If this is possible there is no space need to build a import 
lib without names, because it will be a little about the half of the current 
size. I will see. 

Regards 
Ralf 

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Charles Wilson [mailto:cwilson@ece.gatech.edu] 
> Sent: Saturday, April 27, 2002 9:04 AM
> To: Robert Collins
> Cc: Ralf Habacker; Kde-Cygwin; Binutils; Cygwin-Apps
> Subject: Re: ordinal linking for cygwin ld
> 
> 
> Robert Collins wrote:
> 
> > 
> > The PE spec (as I read it) indicates that as long as a name is 
> > included (ie it's not link-only-by-ordinal) then ordinals 
> can change 
> > and nothing will break.
> > 
> > It's only when the only link information is the ordinal 
> that problems 
> > will appear.
> 
> 
> That's what I thought, too, until I got bit.  Check the 
> cygwin(-apps?) 
> ml archives during spring/summer 2000.  At the time, the 
> symptoms seemed 
> like a smoking gun: link by ordinal == link ONLY by ordinal.  OTOH, 
> binutils on cygwin was royally broken at the time, DJ was 
> transitioning 
> away from cygwin, Mumit had been gone for half a year or 
> more, and bit 
> rot was setting in.

Yup. I was part of that conversation too :}.
 
> Perhaps the problems that burned me then were due to broken 
> binutils/maintainerless cygwin-gcc, and it wasn't the smoking gun I 
> thought it was.

IIRC it was never conclusively identified.
 
> OTOH, if you've linked by ordinal, and then you strip symbols 
> -- are the 
> names of the imports still retained?

Search me. 

Rob (cygwin sortof-binutils
maintainer-it-would-help-if-I-knew-these-things).

Re: ordinal linking for cygwin ld

[Charles Wilson]

Robert Collins wrote:

> 
> The PE spec (as I read it) indicates that as long as a name is included
> (ie it's not link-only-by-ordinal) then ordinals can change and nothing
> will break.
> 
> It's only when the only link information is the ordinal that problems
> will appear.


That's what I thought, too, until I got bit.  Check the cygwin(-apps?) 
ml archives during spring/summer 2000.  At the time, the symptoms seemed 
like a smoking gun: link by ordinal == link ONLY by ordinal.  OTOH, 
binutils on cygwin was royally broken at the time, DJ was transitioning 
away from cygwin, Mumit had been gone for half a year or more, and bit 
rot was setting in.

Perhaps the problems that burned me then were due to broken 
binutils/maintainerless cygwin-gcc, and it wasn't the smoking gun I 
thought it was.

OTOH, if you've linked by ordinal, and then you strip symbols -- are the 
names of the imports still retained?

--Chuck

Re: ordinal linking for cygwin ld

[Charles Wilson]

Ralf Habacker wrote:

> Ralf Habacker wrote:
> 
>>I'm thinking about creating to areas, an internal and an external.
>>New releases of kdelibs and perhaps kdebase for example are build together. So
>>ordinal linking is not problem. -> internal area. Any other app may
>>be linked by name -> external area.
>>
>>
> That would cause the second solution I have told, normal import library and
> using a special linker switch for using ordinal linking.


Yeah, I think I like this -- but it should be used carefully in 
foundation libraries like KDE (and Gnome), and only where you're 
guaranteed that the ordinal-linked apps/dlls will always be upgraded 
together.  The potential for muck-up is high...

--Chuck

RE: ordinal linking for cygwin ld

[Robert Collins]

> -----Original Message-----
> From: Charles Wilson [mailto:cwilson@ece.gatech.edu] 
> Sent: Saturday, April 27, 2002 8:55 AM
> To: Ralf Habacker
> Cc: Kde-Cygwin; Binutils; Cygwin-Apps
> Subject: Re: ordinal linking for cygwin ld
> 
> 
> Ralf Habacker wrote:
> 
> >>
> >>Since your app linked by ordinal, it will break if you try 
> to run it 
> >>with the new DLL, without re-linking.
> >>
> > 
> > accepted
> > 
> > 
> >>So how does the vendor ensure that he doesn't unnecessarily break 
> >>backwards compatibility, and keep the ordinals the same?  
> By using a 
> >>def file.
> >>
> > 
> > Or be relinking all the libs.
> 
> 
> If *you* release new "compatible" libs with the ordinals 
> different from 
> the current libs, *my* application breaks.  Or, you might get ripple 
> effects: what if I distribute a dll that depends on KDE's 
> libs, and Bob 
> has an app that depends on my dll?  Bob's app breaks because 
> my dll is 
> broken because you released new "compatible" libs.  Not good 
> -- and it'd 
> take a while to track the problem down too: (a) I need to 
> relink my dll, 
> and give it to Bob, and then (and only then) (b) Bob can 
> relink his app.
> 
> Blech.

The PE spec (as I read it) indicates that as long as a name is included
(ie it's not link-only-by-ordinal) then ordinals can change and nothing
will break.

It's only when the only link information is the ordinal that problems
will appear.

Rob

Re: ordinal linking for cygwin ld

[Charles Wilson]

Ralf Habacker wrote:

>>
>>Since your app linked by ordinal, it will break if you try to run it
>>with the new DLL, without re-linking.
>>
> 
> accepted
> 
> 
>>So how does the vendor ensure that he doesn't unnecessarily break
>>backwards compatibility, and keep the ordinals the same?  By using a def
>>file.
>>
> 
> Or be relinking all the libs.


If *you* release new "compatible" libs with the ordinals different from 
the current libs, *my* application breaks.  Or, you might get ripple 
effects: what if I distribute a dll that depends on KDE's libs, and Bob 
has an app that depends on my dll?  Bob's app breaks because my dll is 
broken because you released new "compatible" libs.  Not good -- and it'd 
take a while to track the problem down too: (a) I need to relink my dll, 
and give it to Bob, and then (and only then) (b) Bob can relink his app.

Blech.


>>I shudder to think of maintaining all of that for the 200-odd DLLs that
>>come with KDE.
>>
> 
> Using def files is unusable.


Unless it's the only way to avoid the problems above.   Then, you ignore 
the pain and just do it -- or write code to do it for you, like the 
openoffice guys have done.


>>Then, there's one smaller problem:
>>
>>If you use a .def file to generate your DLL, will the auto-import stuff
>>get added to it?  Can auto-import even work, if you're linking by
>>ordinal -- I thought the _nm_ hints were based on the symbol name,
>>
> 
> no, the symbol name was build from the undef section, if a corresponding
> _imp_symbol was found, that means from the client side. There was no access
> (except of the dll name) to the implibs data.


Okay...


>>not the symbol ordinal...
>>
> 
> I have patched auto-import stuff which got the ordinal from the implib.
> It works with the appended testcase. Currently I've build a new working qt with
> this.


> 
> In the second solution I have told about, using the regular implibs and decide
> on a per case base
> if ordinal linking or not.
> 
> I'm thinking about creating to areas, an internal and an external.
> New releases of kdelibs and perhaps kdebase for example are build together. So
> ordinal linking is not problem. -> internal area. Any other app may be linked by
> name -> external area.


Ah, so there'd be a special switch that KDE's internal makefiles use to 
turn on 'link by ordinal' behavior; other applications would not use it, 
and would thus link by name?  (or, they'd use it and link-by-ordinal at 
their own risk...)

That seems workable to me.

 
> Second some dll's are only for specific apps for example konqueror or kcontrol,
> which holds there code in some dll's. When rebuilding konqueror  the dll's could
> be linked ordinal without any problems, because there were all rebuild.


Errr...not so fast.  Code reuse -- konqueror dlls can be linked into 
other apps, can't they? -- or is all code reuse of that nature in KDE 
done via KParts/Bonobo/CORBA/whatever, and not actual linking?

--Chuck

RE: ordinal linking for cygwin ld

[Ralf Habacker]

Ralf Habacker wrote:
> I'm thinking about creating to areas, an internal and an external.
> New releases of kdelibs and perhaps kdebase for example are build together. So
> ordinal linking is not problem. -> internal area. Any other app may
> be linked by name -> external area.
>
That would cause the second solution I have told, normal import library and
using a special linker switch for using ordinal linking.

Ralf

RE: ordinal linking for cygwin ld

[Ralf Habacker]

Charles Wilson writes:
> > Any comments ?

> Yes: compatibility.  The problem with ordinal linking is, suppose
> version 1 of a DLL has the following exports:
>
> func_one  @1
> func_two  @2
> var_one DATA @3
>
> and you link your executable to that dll by number.
>
> Now, the vendor releases a new version of the DLL with an additional
> data export.  Since no interface elements have been changed or removed,
> only added, your app SHOULD be compatible with this DLL.  But, if the
> vendor isn't careful, the new DLL could end up like this:
>
> func_one @1
> var_two DATA @2
> func_two @3
> var_one DATA @4
>
> Since your app linked by ordinal, it will break if you try to run it
> with the new DLL, without re-linking.

accepted

> So how does the vendor ensure that he doesn't unnecessarily break
> backwards compatibility, and keep the ordinals the same?  By using a def
> file.

Or be relinking all the libs.

> There's one big problem with that:
>
> def file are a @!#@^%@ to maintain.  You *always* forget to add
> something -- upstream maintainers are absolutely horrible about NOT
> documenting all of the new functions/data exports that they add.  So you
> end up building the DLL with auto-export, use objdump to list the
> exports, sort 'em, eliminate against your old .def file, create a new
> .def file....
>
> It's just a PITA.  And one time, back when I was using .def files, I
> actually managed, even with all of that effort, to release a DLL that
> dropped one of the exports the OLD dll had, and broke everybody's app.
> And I only deal with about 20 packages and DLLs.
>
> I shudder to think of maintaining all of that for the 200-odd DLLs that
> come with KDE.

Using def files is unusable.

> Then, there's one smaller problem:
>
> If you use a .def file to generate your DLL, will the auto-import stuff
> get added to it?  Can auto-import even work, if you're linking by
> ordinal -- I thought the _nm_ hints were based on the symbol name,

no, the symbol name was build from the undef section, if a corresponding
_imp_symbol was found, that means from the client side. There was no access
(except of the dll name) to the implibs data.

> not the symbol ordinal...

I have patched auto-import stuff which got the ordinal from the implib.
It works with the appended testcase. Currently I've build a new working qt with
this.

> IMO, the better "solution" is to provide a pre-load app, to load all of
> the core KDE dlls into memory at bootup.  This is what Netscape/Mozilla
> now do, and what MS has been doing with IE for years: hide the load time
> during the bootup sequence; at least that way, KDE/Moz/Ntsp/IE doesn't
> get the blame...

In the second solution I have told about, using the regular implibs and decide
on a per case base
if ordinal linking or not.

I'm thinking about creating to areas, an internal and an external.
New releases of kdelibs and perhaps kdebase for example are build together. So
ordinal linking is not problem. -> internal area. Any other app may be linked by
name -> external area.

Second some dll's are only for specific apps for example konqueror or kcontrol,
which holds there code in some dll's. When rebuilding konqueror  the dll's could
be linked ordinal without any problems, because there were all rebuild.

Ralf





> --Chuck
>
>

Re: ordinal linking for cygwin ld

[Charles Wilson]

Martin Hollmichel wrote:


> Maybe you may look to openoffice.org how do the ensure that ordinals 
> keep the same over some time. There's a tool named ldump (located in 
> project tools, modules soltools) which keep's a database to keep the 
> ordinals in track. Maybe this helps.


> at openoffice.org we're able to keep track of about 100 dll, maybe it's 
> usable for 1000 also.


Well, if maintaining the .def files and ordinals is supported *from the 
top* -- as seems to be the case in openoffice -- then it wouldn't be so 
bad.  I was trying to do it, for the cygwin project, "downstream" of the 
actual maintainers of certain unnamed packages 
(<cough>readline<cough>ncurses<cough>gettext).

However, gcc only supports a subset of the .def file language that MSVC 
does, so one would either need to maintain two different .def files, or 
munge the cygwin/mingw one from the msvc one at build time (assuming the 
msvc .def file was maintained by the "real" KDE folks using some tool 
like yours).  OTOH, I've never heard of a mingw or MSVC port of kde, so 
maybe KDE just needs cygwin .def files....

--Chuck

Re: ordinal linking for cygwin ld

[Charles Wilson]

Ralf Habacker wrote:


> Any comments ?


Yes: compatibility.  The problem with ordinal linking is, suppose 
version 1 of a DLL has the following exports:

func_one  @1
func_two  @2
var_one DATA @3

and you link your executable to that dll by number.

Now, the vendor releases a new version of the DLL with an additional 
data export.  Since no interface elements have been changed or removed, 
only added, your app SHOULD be compatible with this DLL.  But, if the 
vendor isn't careful, the new DLL could end up like this:

func_one @1
var_two DATA @2
func_two @3
var_one DATA @4

Since your app linked by ordinal, it will break if you try to run it 
with the new DLL, without re-linking.

So how does the vendor ensure that he doesn't unnecessarily break 
backwards compatibility, and keep the ordinals the same?  By using a def 
file.

There's one big problem with that:

def file are a @!#@^%@ to maintain.  You *always* forget to add 
something -- upstream maintainers are absolutely horrible about NOT 
documenting all of the new functions/data exports that they add.  So you 
end up building the DLL with auto-export, use objdump to list the 
exports, sort 'em, eliminate against your old .def file, create a new 
.def file....

It's just a PITA.  And one time, back when I was using .def files, I 
actually managed, even with all of that effort, to release a DLL that 
dropped one of the exports the OLD dll had, and broke everybody's app. 
And I only deal with about 20 packages and DLLs.

I shudder to think of maintaining all of that for the 200-odd DLLs that 
come with KDE.

Then, there's one smaller problem:

If you use a .def file to generate your DLL, will the auto-import stuff 
get added to it?  Can auto-import even work, if you're linking by 
ordinal -- I thought the _nm_ hints were based on the symbol name, not 
the symbol ordinal...

IMO, the better "solution" is to provide a pre-load app, to load all of 
the core KDE dlls into memory at bootup.  This is what Netscape/Mozilla 
now do, and what MS has been doing with IE for years: hide the load time 
during the bootup sequence; at least that way, KDE/Moz/Ntsp/IE doesn't 
get the blame...

--Chuck

RE: ordinal linking for cygwin ld

[Ralf Habacker]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

> 1. Currently I' have a working solution for binutils 20011002 using a
> specific import library create with an -out-implib-ordinal option,
> which contains only ordinals and no symbols in the
> IMPORT_DESCRIPTOR_BY_NAME structure. (The patches and testcase are appended)

The previous patch contains one undefinded var im pe.em, which is fixed with the
current patch. Sorry

Regards
Ralf

[-- Attachment #2: ld_ordinal_linking_3.patch --]
[-- Type: application/octet-stream, Size: 5593 bytes --]

--- pe-dll.c.orig	Fri Apr 26 17:23:34 2002
+++ pe-dll.c	Fri Apr 26 21:55:42 2002
@@ -120,6 +120,8 @@
 
     See also: ld/emultempl/pe.em.  */
 
+extern int pe_implib_ordinal;
+
 static void
 add_bfd_to_link PARAMS ((bfd *, const char *, struct bfd_link_info *));
 
@@ -1895,6 +1897,8 @@ make_one (exp, parent)
   return abfd;
 }
 
+extern int pe_data_import_ordinal;
+
 static bfd *
 make_singleton_name_thunk (import, parent)
      const char *import;
@@ -1927,6 +1931,14 @@ make_singleton_name_thunk (import, paren
   d4 = (unsigned char *) xmalloc (4);
   id4->contents = d4;
   memset (d4, 0, 8);
+  
+  /* handle ordinal linking */ 
+  if (pe_data_import_ordinal) {
+    d4[0]= pe_data_import_ordinal;
+    d4[1]= pe_data_import_ordinal>>8;
+    d4[3]= 0x80;
+  }
+  else 
   quick_reloc (abfd, 0, BFD_RELOC_RVA, 2);
   save_relocs (id4);
 
@@ -2121,6 +2133,10 @@ pe_dll_generate_implib (def, impfilename
       bfd *n;
 
       def->exports[i].internal_name = def->exports[i].name;
+
+      if (pe_implib_ordinal) {
+         def->exports[i].flag_noname = 1;
+      }
       n = make_one (def->exports + i, outarch);
       n->next = head;
       head = n;
--- emultempl/pe.em.orig	Tue Apr 23 13:47:55 2002
+++ emultempl/pe.em	Fri Apr 26 22:37:59 2002
@@ -153,6 +153,8 @@ static char *pe_out_def_filename = NULL;
 static char *pe_implib_filename = NULL;
 static int pe_enable_auto_image_base = 0;
 static char *pe_dll_search_prefix = NULL;
+int pe_implib_ordinal = 0;
+int pe_data_import_ordinal = 0; 
 #endif
 
 extern const char *output_filename;
@@ -221,6 +223,7 @@ gld_${EMULATION_NAME}_before_parse()
 #define OPTION_DLL_ENABLE_AUTO_IMPORT	(OPTION_NO_DEFAULT_EXCLUDES + 1)
 #define OPTION_DLL_DISABLE_AUTO_IMPORT	(OPTION_DLL_ENABLE_AUTO_IMPORT + 1)
 #define OPTION_ENABLE_EXTRA_PE_DEBUG	(OPTION_DLL_DISABLE_AUTO_IMPORT + 1)
+#define OPTION_IMPLIB_ORDINAL		(OPTION_ENABLE_EXTRA_PE_DEBUG + 1)
 
 static struct option longopts[] = {
   /* PE options */
@@ -252,6 +255,7 @@ static struct option longopts[] = {
   {"enable-stdcall-fixup", no_argument, NULL, OPTION_ENABLE_STDCALL_FIXUP},
   {"disable-stdcall-fixup", no_argument, NULL, OPTION_DISABLE_STDCALL_FIXUP},
   {"out-implib", required_argument, NULL, OPTION_IMPLIB_FILENAME},
+  {"out-implib-ordinal", required_argument, NULL, OPTION_IMPLIB_ORDINAL},
   {"warn-duplicate-exports", no_argument, NULL, OPTION_WARN_DUPLICATE_EXPORTS},
   {"compat-implib", no_argument, NULL, OPTION_IMP_COMPAT},
   {"enable-auto-image-base", no_argument, NULL, OPTION_ENABLE_AUTO_IMAGE_BASE},
@@ -336,6 +340,8 @@ gld_${EMULATION_NAME}_list_options (file
   fprintf (file, _("  --export-all-symbols               Automatically export all globals to DLL\n"));
   fprintf (file, _("  --kill-at                          Remove @nn from exported symbols\n"));
   fprintf (file, _("  --out-implib <file>                Generate import library\n"));
+  fprintf (file, _("  --out-implib-ordinal <file>        Generate import library using ordinals \n"));
+  fprintf (file, _("                                       instead of names\n"));
   fprintf (file, _("  --output-def <file>                Generate a .DEF file for the built DLL\n"));
   fprintf (file, _("  --warn-duplicate-exports           Warn about duplicate exports.\n"));
   fprintf (file, _("  --compat-implib                    Create backward compatible import libs;\n"));
@@ -630,6 +636,10 @@ gld_${EMULATION_NAME}_parse_args(argc, a
     case OPTION_ENABLE_EXTRA_PE_DEBUG:
       pe_dll_extra_pe_debug = 1;
       break;
+    case OPTION_IMPLIB_ORDINAL:
+      pe_implib_filename = xstrdup (optarg);
+      pe_implib_ordinal = 1;
+      break;
 #endif
     }
   return 1;
@@ -885,6 +895,57 @@ make_import_fixup (rel, s)
 
   return 1;
 }
+void get_data_import_ordinal(bfd *b)
+{
+	struct sec *s = bfd_get_section_by_name (b, ".idata$5");
+	if (s)
+	{
+	  int size = bfd_get_section_size_before_reloc (s);
+	  char *buf = xmalloc (size);
+	
+	  bfd_get_section_contents (b, s, buf, 0, size);
+	  pe_data_import_ordinal = *(unsigned short *)buf;
+	  if (pe_dll_extra_pe_debug)
+	    printf("#%04x\n",pe_data_import_ordinal);
+	  free (buf);
+	}
+	else 
+	  pe_data_import_ordinal = 0;
+}
+
+
+/* ordinal linking 
+
+ordinal import library object file 
+
+d001500.o:     file format pe-i386
+
+Contents of section .idata$7:
+ 0000 00000000                             ....
+Contents of section .idata$5:
+ 0000 dc050080                             ....
+      ^^^^^
+  This value is used 
+      
+Contents of section .idata$4:
+ 0000 dc050080                             ....
+
+
+regular import library object file 
+
+d001500.o:     file format pe-i386
+
+Contents of section .idata$7:
+ 0000 00000000                             ....
+Contents of section .idata$5:
+ 0000 00000000                             ....
+Contents of section .idata$4:
+ 0000 00000000                             ....
+Contents of section .idata$6:
+ 0000 db057661 72303439 39000000           ..var0499...
+      ^^^^
+      This value seems also be usable (not implemented yet)
+*/
 
 static void
 pe_find_data_imports ()
@@ -916,6 +977,8 @@ pe_find_data_imports ()
                 symbols = (asymbol **) xmalloc (symsize);
                 nsyms = bfd_canonicalize_symtab (b, symbols);
 
+		get_data_import_ordinal(b);
+
                 for (i = 0; i < nsyms; i++)
                   {
                     if (memcmp(symbols[i]->name, "__head_",

ordinal linking for cygwin ld

[Ralf Habacker]

[-- Attachment #1: Type: text/plain, Size: 3123 bytes --]

Hi all,

one of the biggest problems with kde 2.2.x currently is the very bad loading
time of applications and dll's, I've investigated some time to analyse this. On
this way I recognized, that runtime linking using symbol names is one of the
major time eater.

At first I have tried to estimate the influence of symbol name to the runtime
linking time. So I have build a dll 5000 functions and a client app, which uses
this functions.

This dll/apps are compiled in two variants, 1. with a symbolname lenght of a 13
characters and 2. with a symbolname lenght of 403 characters (should simulate
long c++ names).

cd shortname
time ./client.exe
real    0m0.772s
user    0m0.030s
sys     0m0.030s

$ cd longname
$ time ./client.exe
real    0m15.182s
user    0m0.060s
sys     0m0.040s

The result shows that this has a great effect of app/dll loading time (about 20
times more).

After thinking a while about this, I remembered, that the windows pe format
allows two runtime linking types. The first, usually used, is linking by the
symbol name, the second is linking by symbols ordinal.

A look in the last official cygwin ld (20011002) source shows, that linking by
ordinal seems to be prepared with one limitation.
There is no command line switch to create an import library for ordinal linking
on dll building, so I've added a switch --out-implib-ordinal <filename>.

(BTW: Another possibibily was automatic ordinal linking. See later)

Now I've tried a second test. 5000 functions with symbolname lenght of 403
characters linked 1. with names and 2. with ordinals.

5000 functions with symbol name linking (symbols not stripped)

$ time ./client.exe
real    0m15.182s
user    0m0.060s
sys     0m0.040s

5000 functions with ordinal linking (symbols not stripped)

$ time ./client.exe
real    0m0.572s
user    0m0.020s
sys     0m0.030s

Wow, thats great.

1. Currently I' have a working solution for binutils 20011002 using a specific
import library create with an -out-implib-ordinal option, which contains only
ordinals and no symbols in the IMPORT_DESCRIPTOR_BY_NAME structure. (The patches
and testcase are appended)

2. Another possibility is using the regular import libraries hint number for
ordinal linking (hint number = Ordinal number-1), but currently I don't know if
this is always true.

Solution 1 has an advantage on building big c++ libs, because the import library
are smaller (about 20-50%). See the following example. The amount depends on the
symbols count and length.

old
-rwxr-xr-x    1 1002     Kein      1682944 Apr 22 09:42
/opt/kde2/bin/cygkdecore-3.dll
-rwxr-xr-x    1 1002     Kein      3190016 Apr 20 19:12
/opt/kde2/lib/libkdecore.dll.a

new
-rwxrwxrwx    1 1002     Kein      1956812 Apr 26 22:03 libkdecore.dll.a
-rwxrwxrwx    1 1002     Kein      1608192 Apr 26 22:28 cygkdecore-3.dll

Solution 2 is full compatible to dll's builded with previous ld, because the dll
and import lib are still the same. Compiling an app with new ld will
automatically use ordinal linking. The only necessary point seems to me a
command line switch to switch of this behavior.

Any comments ?

Regards
Ralf


[-- Attachment #2: ld_ordinal_linking_2.patch --]
[-- Type: application/octet-stream, Size: 5559 bytes --]

--- pe-dll.c.orig	Fri Apr 26 17:23:34 2002
+++ pe-dll.c	Fri Apr 26 21:55:42 2002
@@ -120,6 +120,8 @@
 
     See also: ld/emultempl/pe.em.  */
 
+extern int pe_implib_ordinal;
+
 static void
 add_bfd_to_link PARAMS ((bfd *, const char *, struct bfd_link_info *));
 
@@ -1895,6 +1897,8 @@ make_one (exp, parent)
   return abfd;
 }
 
+extern long pe_data_import_ordinal;
+
 static bfd *
 make_singleton_name_thunk (import, parent)
      const char *import;
@@ -1927,6 +1931,14 @@ make_singleton_name_thunk (import, paren
   d4 = (unsigned char *) xmalloc (4);
   id4->contents = d4;
   memset (d4, 0, 8);
+  
+  /* handle ordinal linking */ 
+  if (pe_data_import_ordinal) {
+    d4[0]= pe_data_import_ordinal;
+    d4[1]= pe_data_import_ordinal>>8;
+    d4[3]= 0x80;
+  }
+  else 
   quick_reloc (abfd, 0, BFD_RELOC_RVA, 2);
   save_relocs (id4);
 
@@ -2121,6 +2133,10 @@ pe_dll_generate_implib (def, impfilename
       bfd *n;
 
       def->exports[i].internal_name = def->exports[i].name;
+
+      if (pe_implib_ordinal) {
+         def->exports[i].flag_noname = 1;
+      }
       n = make_one (def->exports + i, outarch);
       n->next = head;
       head = n;
--- emultempl/pe.em.orig	Tue Apr 23 13:47:55 2002
+++ emultempl/pe.em	Fri Apr 26 22:20:13 2002
@@ -153,6 +153,7 @@ static char *pe_out_def_filename = NULL;
 static char *pe_implib_filename = NULL;
 static int pe_enable_auto_image_base = 0;
 static char *pe_dll_search_prefix = NULL;
+int pe_implib_ordinal = 0;
 #endif
 
 extern const char *output_filename;
@@ -221,6 +222,7 @@ gld_${EMULATION_NAME}_before_parse()
 #define OPTION_DLL_ENABLE_AUTO_IMPORT	(OPTION_NO_DEFAULT_EXCLUDES + 1)
 #define OPTION_DLL_DISABLE_AUTO_IMPORT	(OPTION_DLL_ENABLE_AUTO_IMPORT + 1)
 #define OPTION_ENABLE_EXTRA_PE_DEBUG	(OPTION_DLL_DISABLE_AUTO_IMPORT + 1)
+#define OPTION_IMPLIB_ORDINAL		(OPTION_ENABLE_EXTRA_PE_DEBUG + 1)
 
 static struct option longopts[] = {
   /* PE options */
@@ -252,6 +254,7 @@ static struct option longopts[] = {
   {"enable-stdcall-fixup", no_argument, NULL, OPTION_ENABLE_STDCALL_FIXUP},
   {"disable-stdcall-fixup", no_argument, NULL, OPTION_DISABLE_STDCALL_FIXUP},
   {"out-implib", required_argument, NULL, OPTION_IMPLIB_FILENAME},
+  {"out-implib-ordinal", required_argument, NULL, OPTION_IMPLIB_ORDINAL},
   {"warn-duplicate-exports", no_argument, NULL, OPTION_WARN_DUPLICATE_EXPORTS},
   {"compat-implib", no_argument, NULL, OPTION_IMP_COMPAT},
   {"enable-auto-image-base", no_argument, NULL, OPTION_ENABLE_AUTO_IMAGE_BASE},
@@ -336,6 +339,8 @@ gld_${EMULATION_NAME}_list_options (file
   fprintf (file, _("  --export-all-symbols               Automatically export all globals to DLL\n"));
   fprintf (file, _("  --kill-at                          Remove @nn from exported symbols\n"));
   fprintf (file, _("  --out-implib <file>                Generate import library\n"));
+  fprintf (file, _("  --out-implib-ordinal <file>        Generate import library using ordinals \n"));
+  fprintf (file, _("                                       instead of names\n"));
   fprintf (file, _("  --output-def <file>                Generate a .DEF file for the built DLL\n"));
   fprintf (file, _("  --warn-duplicate-exports           Warn about duplicate exports.\n"));
   fprintf (file, _("  --compat-implib                    Create backward compatible import libs;\n"));
@@ -630,6 +635,10 @@ gld_${EMULATION_NAME}_parse_args(argc, a
     case OPTION_ENABLE_EXTRA_PE_DEBUG:
       pe_dll_extra_pe_debug = 1;
       break;
+    case OPTION_IMPLIB_ORDINAL:
+      pe_implib_filename = xstrdup (optarg);
+      pe_implib_ordinal = 1;
+      break;
 #endif
     }
   return 1;
@@ -885,6 +894,57 @@ make_import_fixup (rel, s)
 
   return 1;
 }
+void get_data_import_ordinal(bfd *b)
+{
+	struct sec *s = bfd_get_section_by_name (b, ".idata$5");
+	if (s)
+	{
+	  int size = bfd_get_section_size_before_reloc (s);
+	  char *buf = xmalloc (size);
+	
+	  bfd_get_section_contents (b, s, buf, 0, size);
+	  pe_data_import_ordinal = *(unsigned short *)buf;
+	  if (pe_dll_extra_pe_debug)
+	    printf("#%04x\n",pe_data_import_ordinal);
+	  free (buf);
+	}
+	else 
+	  pe_data_import_ordinal = 0;
+}
+
+
+/* ordinal linking 
+
+ordinal import library object file 
+
+d001500.o:     file format pe-i386
+
+Contents of section .idata$7:
+ 0000 00000000                             ....
+Contents of section .idata$5:
+ 0000 dc050080                             ....
+      ^^^^^
+  This value is used 
+      
+Contents of section .idata$4:
+ 0000 dc050080                             ....
+
+
+regular import library object file 
+
+d001500.o:     file format pe-i386
+
+Contents of section .idata$7:
+ 0000 00000000                             ....
+Contents of section .idata$5:
+ 0000 00000000                             ....
+Contents of section .idata$4:
+ 0000 00000000                             ....
+Contents of section .idata$6:
+ 0000 db057661 72303439 39000000           ..var0499...
+      ^^^^
+      This value seems also be usable (not implemented yet)
+*/
 
 static void
 pe_find_data_imports ()
@@ -916,6 +976,8 @@ pe_find_data_imports ()
                 symbols = (asymbol **) xmalloc (symsize);
                 nsyms = bfd_canonicalize_symtab (b, symbols);
 
+		get_data_import_ordinal(b);
+
                 for (i = 0; i < nsyms; i++)
                   {
                     if (memcmp(symbols[i]->name, "__head_",

[-- Attachment #3: ld_ordinal_linking_testcase.tar.bz2 --]
[-- Type: application/octet-stream, Size: 5272 bytes --]