Question about readelf output from shared library built with lld, gold, and bfd linkers

Question about readelf output from shared library built with lld, gold, and bfd linkers

[Tom Kacvinsky]

[-- Attachment #1: Type: text/plain, Size: 7945 bytes --]

Isee the following output from readelf for the lld, gold and bfd linkers
(binutils 2.39, lld 14.0.6)
Output below.  Notice how the ordering is different in each case.  The
interesting thing about this is, and why I am looking at various
differences between object code linked by these three linkers, is that I am
trying to track down why startup times are slower due to relocations (based
on the perf tool output).  Would any of these differences make, well, a
difference in startup time?

Thanks,

Tom

Output from readelf for the different linkers.  The particular function is
irrelevant, it happens with other symbols as well.

lld linker:

readelf -Wa libVcAppLib.so | grep ada__strings__search__index
000000000162e508  000055c800000007 R_X86_64_JUMP_SLOT     0000000001571220
ada__strings__search__index + 0
000000000162e510  0000587700000007 R_X86_64_JUMP_SLOT     0000000001570a30
ada__strings__search__index__2 + 0
000000000162e518  0000587d00000007 R_X86_64_JUMP_SLOT     0000000001570c40
ada__strings__search__index__3 + 0
000000000162e520  0000588400000007 R_X86_64_JUMP_SLOT     00000000015715d0
ada__strings__search__index__4 + 0
000000000162e528  0000588a00000007 R_X86_64_JUMP_SLOT     0000000001570d00
ada__strings__search__index__5 + 0
000000000162e530  0000588d00000007 R_X86_64_JUMP_SLOT     0000000001570dd0
ada__strings__search__index__6 + 0
000000000162e540  000003fe00000007 R_X86_64_JUMP_SLOT     0000000001570ea0
ada__strings__search__index_non_blank + 0
000000000162e548  0000278900000007 R_X86_64_JUMP_SLOT     0000000001570f00
ada__strings__search__index_non_blank__2 + 0
  1022: 0000000001570ea0    91 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index_non_blank
 10121: 0000000001570f00   167 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index_non_blank__2
 21960: 0000000001571220   930 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index
 22647: 0000000001570a30   523 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__2
 22653: 0000000001570c40   187 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__3
 22660: 00000000015715d0   207 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__4
 22666: 0000000001570d00   207 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__5
 22669: 0000000001570dd0   199 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__6
 36851: 0000000001571220   930 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index
 36853: 0000000001570a30   523 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__2
 36854: 0000000001570c40   187 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__3
 36856: 00000000015715d0   207 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__4
 36858: 0000000001570d00   207 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__5
 36860: 0000000001570dd0   199 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index__6
 36862: 0000000001570ea0    91 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index_non_blank
 36864: 0000000001570f00   167 FUNC    GLOBAL DEFAULT   14
ada__strings__search__index_non_blank__2

gold linker:

readelf -Wa libVcAppLib.so | grep ada__strings__search__index
0000000001610048  00000f0600000007 R_X86_64_JUMP_SLOT     000000000115a220
ada__strings__search__index + 0
0000000001610050  00003fa300000007 R_X86_64_JUMP_SLOT     0000000001159a30
ada__strings__search__index__2 + 0
0000000001610058  00003fa600000007 R_X86_64_JUMP_SLOT     0000000001159c40
ada__strings__search__index__3 + 0
0000000001610060  00003fa800000007 R_X86_64_JUMP_SLOT     000000000115a5d0
ada__strings__search__index__4 + 0
0000000001610068  00003fa900000007 R_X86_64_JUMP_SLOT     0000000001159d00
ada__strings__search__index__5 + 0
0000000001610070  00003fad00000007 R_X86_64_JUMP_SLOT     0000000001159dd0
ada__strings__search__index__6 + 0
0000000001610078  0000075000000007 R_X86_64_JUMP_SLOT     0000000001159ea0
ada__strings__search__index_non_blank + 0
0000000001610080  000050cb00000007 R_X86_64_JUMP_SLOT     0000000001159f00
ada__strings__search__index_non_blank__2 + 0
  1872: 0000000001159ea0    91 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank
  3846: 000000000115a220   930 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index
 16291: 0000000001159a30   523 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__2
 16294: 0000000001159c40   187 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__3
 16296: 000000000115a5d0   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__4
 16297: 0000000001159d00   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__5
 16301: 0000000001159dd0   199 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__6
 20683: 0000000001159f00   167 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank__2
 16810: 0000000001159f00   167 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank__2
 16812: 0000000001159ea0    91 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank
 16814: 0000000001159dd0   199 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__6
 16816: 0000000001159d00   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__5
 16818: 000000000115a5d0   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__4
 16820: 0000000001159c40   187 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__3
 16821: 0000000001159a30   523 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__2
 16823: 000000000115a220   930 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index

bfd linker:

readelf -Wa libVcAppLib.so | grep ada__strings__search__index
00000000015db948  00003fa600000007 R_X86_64_JUMP_SLOT     000000000114fa20
ada__strings__search__index__2 + 0
00000000015de700  00003fa800000007 R_X86_64_JUMP_SLOT     000000000114fc30
ada__strings__search__index__3 + 0
00000000015e1908  0000075200000007 R_X86_64_JUMP_SLOT     000000000114fe90
ada__strings__search__index_non_blank + 0
00000000015e55c8  00000f0a00000007 R_X86_64_JUMP_SLOT     0000000001150210
ada__strings__search__index + 0
00000000015e6e40  00003fae00000007 R_X86_64_JUMP_SLOT     000000000114fdc0
ada__strings__search__index__6 + 0
00000000015ec3a0  00003faa00000007 R_X86_64_JUMP_SLOT     00000000011505c0
ada__strings__search__index__4 + 0
00000000015edb08  000050cd00000007 R_X86_64_JUMP_SLOT     000000000114fef0
ada__strings__search__index_non_blank__2 + 0
00000000015ef168  00003fad00000007 R_X86_64_JUMP_SLOT     000000000114fcf0
ada__strings__search__index__5 + 0
  1874: 000000000114fe90    91 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank
  3850: 0000000001150210   930 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index
 16294: 000000000114fa20   523 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__2
 16296: 000000000114fc30   187 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__3
 16298: 00000000011505c0   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__4
 16301: 000000000114fcf0   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__5
 16302: 000000000114fdc0   199 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__6
 20685: 000000000114fef0   167 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank__2
 16406: 000000000114fa20   523 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__2
 19426: 000000000114fc30   187 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__3
 22716: 000000000114fe90    91 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank
 26768: 0000000001150210   930 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index
 28382: 000000000114fdc0   199 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__6
 34154: 00000000011505c0   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__4
 35651: 000000000114fef0   167 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index_non_blank__2
 37103: 000000000114fcf0   207 FUNC    GLOBAL DEFAULT   12
ada__strings__search__index__5

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Nick Clifton]

Hi Tom,

> Isee the following output from readelf for the lld, gold and bfd linkers
> (binutils 2.39, lld 14.0.6)

To be clear - this is not a readelf problem.  It is showing you the correct
results.  It is the fact that the three linkers are not producing identical
output and instead showing slight variations in their layout of the linked
binary that is bothering you, yes ?

In general variations in the layout like this should not make any difference
to the program#s startup time.  There might - possibly - be variations in
performance due to affects like cache misses and the like, but this is hard
to quantify in isolation.


> Output below.  Notice how the ordering is different in each case.  The
> interesting thing about this is, and why I am looking at various
> differences between object code linked by these three linkers, is that I am
> trying to track down why startup times are slower due to relocations (based
> on the perf tool output).  Would any of these differences make, well, a
> difference in startup time?

I don't think so.  The number of relocations is the same, so the amount
of start up time spent resolving them should effectively be the same as
well.

I assume that you have compared started up times when linking with "-z now"
vs "-z lazy" ?

Cheers
   Nick

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Tom Kacvinsky]

[-- Attachment #1: Type: text/plain, Size: 2195 bytes --]

On Mon, Nov 21, 2022 at 9:14 AM Nick Clifton <nickc@redhat.com> wrote:

> Hi Tom,
>
> > Isee the following output from readelf for the lld, gold and bfd linkers
> > (binutils 2.39, lld 14.0.6)
>
> To be clear - this is not a readelf problem.  It is showing you the correct
> results.  It is the fact that the three linkers are not producing identical
> output and instead showing slight variations in their layout of the linked
> binary that is bothering you, yes ?
>

It's not bothering me so much as I was wondering if this would point to an
issue with slow start up times.  Reading your reply in totality, I now see
this
should not make a difference.


> In general variations in the layout like this should not make any
> difference
> to the program#s startup time.  There might - possibly - be variations in
> performance due to affects like cache misses and the like, but this is hard
> to quantify in isolation.
>

I didn't stop to think about caching and the like for relocations at start
up.  I was
testing with the perf tool and LD_DEBUG=statistics to see how much time was
spent in start up by symbols/relocations.  Results varied from run to run,
so I can
now see how caching may play into this.

> Output below.  Notice how the ordering is different in each case.  The
> > interesting thing about this is, and why I am looking at various
> > differences between object code linked by these three linkers, is that I
> am
> > trying to track down why startup times are slower due to relocations
> (based
> > on the perf tool output).  Would any of these differences make, well, a
> > difference in startup time?
>
> I don't think so.  The number of relocations is the same, so the amount
> of start up time spent resolving them should effectively be the same as
> well.
>
> I assume that you have compared started up times when linking with "-z now"
> vs "-z lazy" ?
>

I thought -z lazy was the default, and that if you wanted the equivalent of
LD_BIND_NOW=1
on the command line, then one would use -z now.  We don't want the latter,
we already spawn
enough processes that -z now would slow things down even more.  I can try
with -z lazy and
report back.

Thanks for the input,

Tom

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Nick Clifton]

Hi Tom,

>> I assume that you have compared started up times when linking with "-z now"
>> vs "-z lazy" ?
>>
> 
> I thought -z lazy was the default, and that if you wanted the equivalent of
> LD_BIND_NOW=1 on the command line, then one would use -z now.

It depends upon the environment.  In Fedora (and RHEL) "-z now" is the
default.  The reason is program security - lazy binding means that an
attacker might be able to alter the relocations before they are evaluated,
allowing all kinds of unexpected things to happen.

Cheers
   Nick

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Tom Kacvinsky]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On Mon, Nov 21, 2022 at 10:17 AM Nick Clifton <nickc@redhat.com> wrote:

> Hi Tom,
>
> >> I assume that you have compared started up times when linking with "-z
> now"
> >> vs "-z lazy" ?
> >>
> >
> > I thought -z lazy was the default, and that if you wanted the equivalent
> of
> > LD_BIND_NOW=1 on the command line, then one would use -z now.
>
> It depends upon the environment.  In Fedora (and RHEL) "-z now" is the
> default.  The reason is program security - lazy binding means that an
> attacker might be able to alter the relocations before they are evaluated,
> allowing all kinds of unexpected things to happen.
>

Good to know.  I am using CentOS 7 with a built from source binutils (2.39)
as the system linker is too old to handle the DWARF5 debug symbols made
by GCC 12.1 (also built from source).  So the modification made by Fedora
and RHEL is custom to their distros, right?  If so, then I think that I am
using
-z lazy.

Also, a question: would -z combreloc make a difference with relocations at
startup?  I was just perusing the ld manual and ran across that.

Tom

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Nick Clifton]

Hi Tom,

> Also, a question: would -z combreloc make a difference with relocations at
> startup?  I was just perusing the ld manual and ran across that.

It may well do.  It is certainly worth a try.

Cheers
   Nick

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Tom Kacvinsky]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

On Mon, Nov 21, 2022 at 11:48 AM Nick Clifton <nickc@redhat.com> wrote:

> Hi Tom,
>
> > Also, a question: would -z combreloc make a difference with relocations
> at
> > startup?  I was just perusing the ld manual and ran across that.
>
> It may well do.  It is certainly worth a try.
>

Alas, no joy.  These are the types of fun problems I face at work - change
one of the tools
we use to build our product and one of our performance tests has a
regression.  Then it
is on me to figure out why.  But, to be honest, this is black magic to me.
I do not know
where to start looking to figure out why lld generates the final ELF file
that runs quicker
than what is generated by the gold linker.

So, I guess the next question is this - what should I look for in the files
generated by lld
and gold, respectively, using  readelf?

Thanks,

Tom

Re: Question about readelf output from shared library built with lld, gold, and bfd linkers

[Nick Clifton]

Hi Tom,

> I do not know
> where to start looking to figure out why lld generates the final ELF file
> that runs quicker than what is generated by the gold linker.

Are we talking about application start-up times or application run-time
performance ?  The two are different and likely to be affected by different
changes in the linker.

[As an aside have you also compared performance when linking with the BFD
based linker, and with mold ?)


> So, I guess the next question is this - what should I look for in the files
> generated by lld and gold, respectively, using  readelf?

You don't.  At least that is not where I would start.  Your best bet is to
use one or more profiling tools and figure out why one version of the
application is faster than the other.  If it is a case that the profiles of
the two versions are basically the same - ie they spend the same percentages
of their time in the same functions - then the cause is likely to be the
layout of the code.  The faster performing code is getting better performance
from the cache.  If they have different profiles, then maybe one linker is
able to discard more unneeded code, or optimize the linked code more effectively.

Another place to look is in the changelogs/reflogs/announcements for the
two linkers.  Likely lld has recently had some new features/improvements added
to it, and these are causing the performance improvements.

Cheers
   Nick

PS.  Waiving my devil's advocate flag here: why bother with this search ?
   Isn't it enough that you have found a linker that provides better performance ?