ld: document static libraries when performing static linking

ld: document static libraries when performing static linking

[Christian Tramnitz]

Hello,

For statically linked binaries one wants to keep track of external
libraries that were pulled in at link time. I haven't found any
initiatives or standards trying to achieve this yet, so I would like
to make a proposal.

ld already supports the `--depedency-file=` option to create a
depfile, to some degree, documenting dependencies in Makefile format.
While it's a good starting point, it's not sufficient.

Would it be feasible to create another option, that
a) just records dependencies to external, static-libs (within the
depfile this could be achieved by looking for path names outside the
build dir and having an .a file extension - but this is not
neccessarily exhaustive). This doesn't need to be unique,
post-processing is expected to take place anyway.
b) documents the actual objects (enhanced: down to function level if
information exists from compiling with -ffunction-sections and
-fdata-sections) that were pulled in from the library archive, similar
to the output from `--verbose`
"(/usr/lib/gcc/x86_64-pc-linux
-gnu/13/../../../../lib64/libc.a)stpcpy-avx2.o"
c) outputs this in some sort of structured format for later
processing, e.g. as simple as
<object-requiring-the-library>,</some/library.a>,<requested-object-in-library>
my-obj,/usr/lib64/libc.a,stpcpy-avx2.o
my-obj,/usr/lib64/libc.a,stpcpy-avx2-rtm.o
my-obj,/usr/lib64/libc.a,getsrvbynm.o
my-obj,/usr/lib64/libc.a,stpcpy-evex.o
d) - if ld doesn't support this already (couldn't find looking through
man) - ld had some sort of no-op/pretend option that doesn't actually
produce a linked output but only pretends it would?


Purpose:
The consumer of this information might be a distributor, keeping track
of things in the distribution specific packaging database format. Or
it could even be embedded into the resulting binary itself.


Background:
This output can be used to
I) validate license compliance (it would make it possible to discover
conflicting licenses of derived work)
II) allow vulnerability tracking, when vulnerabilities existed in
included static libraries (at build/link time)

The latter would be further supported by b) as a vulnerability may not
even exist in the resulting object if the vulnerable function wasn't
included by linking.



Best regards,
Christian

Re: ld: document static libraries when performing static linking

[Howard Chu]

Christian Tramnitz wrote:
> Hello,

Hi -
   a linker plugin to handle static library dependencies already exists, it's
been around nearly 4 years. https://sourceware.org/pipermail/binutils/2020-September/113188.html

There was a bit of a push to make it an integral feature of ld instead of a plugin, but that
has stalled. https://sourceware.org/pipermail/binutils/2023-February/126155.html

I guess it's worth asking again, what help is needed to move integration forward?

> 
> For statically linked binaries one wants to keep track of external
> libraries that were pulled in at link time. I haven't found any
> initiatives or standards trying to achieve this yet, so I would like
> to make a proposal.
> 
> ld already supports the `--depedency-file=` option to create a
> depfile, to some degree, documenting dependencies in Makefile format.
> While it's a good starting point, it's not sufficient.
> 
> Would it be feasible to create another option, that
> a) just records dependencies to external, static-libs (within the
> depfile this could be achieved by looking for path names outside the
> build dir and having an .a file extension - but this is not
> neccessarily exhaustive). This doesn't need to be unique,
> post-processing is expected to take place anyway.
> b) documents the actual objects (enhanced: down to function level if
> information exists from compiling with -ffunction-sections and
> -fdata-sections) that were pulled in from the library archive, similar
> to the output from `--verbose`
> "(/usr/lib/gcc/x86_64-pc-linux
> -gnu/13/../../../../lib64/libc.a)stpcpy-avx2.o"
> c) outputs this in some sort of structured format for later
> processing, e.g. as simple as
> <object-requiring-the-library>,</some/library.a>,<requested-object-in-library>
> my-obj,/usr/lib64/libc.a,stpcpy-avx2.o
> my-obj,/usr/lib64/libc.a,stpcpy-avx2-rtm.o
> my-obj,/usr/lib64/libc.a,getsrvbynm.o
> my-obj,/usr/lib64/libc.a,stpcpy-evex.o
> d) - if ld doesn't support this already (couldn't find looking through
> man) - ld had some sort of no-op/pretend option that doesn't actually
> produce a linked output but only pretends it would?
> 
> 
> Purpose:
> The consumer of this information might be a distributor, keeping track
> of things in the distribution specific packaging database format. Or
> it could even be embedded into the resulting binary itself.
> 
> 
> Background:
> This output can be used to
> I) validate license compliance (it would make it possible to discover
> conflicting licenses of derived work)
> II) allow vulnerability tracking, when vulnerabilities existed in
> included static libraries (at build/link time)
> 
> The latter would be further supported by b) as a vulnerability may not
> even exist in the resulting object if the vulnerable function wasn't
> included by linking.
> 
> 
> 
> Best regards,
> Christian
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: ld: document static libraries when performing static linking

[Howard Chu]

Howard Chu wrote:
> Christian Tramnitz wrote:
>> Hello,
> 
> Hi -
>    a linker plugin to handle static library dependencies already exists, it's
> been around nearly 4 years. https://sourceware.org/pipermail/binutils/2020-September/113188.html

Ah no, I hadn't read the entire message before replying. My bad.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: ld: document static libraries when performing static linking

[Nick Clifton]

Hi Christian,

> For statically linked binaries one wants to keep track of external
> libraries that were pulled in at link time. I haven't found any
> initiatives or standards trying to achieve this yet, so I would like
> to make a proposal.
> 
> ld already supports the `--depedency-file=` option to create a
> depfile, to some degree, documenting dependencies in Makefile format.
> While it's a good starting point, it's not sufficient.
> 
> Would it be feasible to create another option, that
> a) just records dependencies to external, static-libs (within the
> depfile this could be achieved by looking for path names outside the
> build dir and having an .a file extension - but this is not
> neccessarily exhaustive). This doesn't need to be unique,
> post-processing is expected to take place anyway.

If post-processing is an option, then how about using the linker map
file and parsing that ?  Ie link with -Map=foo.map and then post-link
scan foo.map.  That would give you full details of which libraries were
used, and which components from inside those libraries if they are
archives.

It might take a bit of work to get a reasonably robust scanner, but
it would mean that it would work with any version of the linker, and
you would not have the hassle of maintaining a linker plugin.


> b) documents the actual objects (enhanced: down to function level if
> information exists from compiling with -ffunction-sections and
> -fdata-sections) that were pulled in from the library archive, similar
> to the output from `--verbose`

The linker map idea above should let you do this.


> c) outputs this in some sort of structured format for later
> processing, e.g. as simple as
> <object-requiring-the-library>,</some/library.a>,<requested-object-in-library>
> my-obj,/usr/lib64/libc.a,stpcpy-avx2.o
> my-obj,/usr/lib64/libc.a,stpcpy-avx2-rtm.o
> my-obj,/usr/lib64/libc.a,getsrvbynm.o
> my-obj,/usr/lib64/libc.a,stpcpy-evex.o

If you do not like the map idea then maybe adding some code to the linker
at the places where it generates its map output, but this time in a format
you like, would work ?


> d) - if ld doesn't support this already (couldn't find looking through
> man) - ld had some sort of no-op/pretend option that doesn't actually
> produce a linked output but only pretends it would?

It doesn't.  Although of course you can always throw away any files it
creates and start again.


> Purpose:
> The consumer of this information might be a distributor, keeping track
> of things in the distribution specific packaging database format. Or
> it could even be embedded into the resulting binary itself.
There is another possibility...

Check out the annobin project: https://sourceware.org/annobin/

The compiler plugins that are part of this project record notes that
include the filename associated with each compilation unit.  These
notes are aggregated together during a link and end up in the resulting
binary.  So you could scan the binary for these notes, extract the
filenames and you would end up with a list of (main) sources for the
program.  If your static libraries are also compiled with annobin enabled
then they will contain notes of their own that will also contribute to
the final binary.

Cheers
   Nick

Re: ld: document static libraries when performing static linking

[Christian Tramnitz]

Hi Nick,

thanks for your detailed response!

> If post-processing is an option, then how about using the linker map
> file and parsing that ?  Ie link with -Map=foo.map and then post-link
> scan foo.map.  That would give you full details of which libraries were
> used, and which components from inside those libraries if they are
> archives.

-Map does indeed provide the required information - thanks a lot for
the hint, that is already much better than parsing through verbose
build logs or the dependency-file.
However, like --depedency-file, it has one major drawback: when
multiple targets exist, the "last" action overwrites the output file.
I haven't seen an option to use a dynamic file names (foo-$$.map
doesn't seem to work).

> There is another possibility...
>
> Check out the annobin project: https://sourceware.org/annobin/
>
> The compiler plugins that are part of this project record notes that
> include the filename associated with each compilation unit.  These
> notes are aggregated together during a link and end up in the resulting
> binary.  So you could scan the binary for these notes, extract the
> filenames and you would end up with a list of (main) sources for the
> program.  If your static libraries are also compiled with annobin enabled
> then they will contain notes of their own that will also contribute to
> the final binary.

I indeed looked into annobin initially, but did not get that the notes
also contain references to the sources/archive/library (only compile
options as per https://sourceware.org/annobin/annobin.html/Plugins.html).
But since you're the author, you know better for sure. Will have a
look again as it may cover large parts of what I'm trying to achieve
already. Except for the lookup which package owns that file of course.
But that could be a post-processing step again.


Best regards,
   Christian

Re: ld: document static libraries when performing static linking

[Nick Clifton]

Hi Christian,

> -Map does indeed provide the required information - thanks a lot for
> the hint, that is already much better than parsing through verbose
> build logs or the dependency-file.
> However, like --depedency-file, it has one major drawback: when
> multiple targets exist, the "last" action overwrites the output file.
> I haven't seen an option to use a dynamic file names (foo-$$.map
> doesn't seem to work).

Ah - but directory names do. :-)

Check out the linker manual's description of the -Map option.  In
particular:

   Specifying a directory as <mapfile> causes the linker map to be
   written as a file inside the directory.  Normally name of the file
   inside the directory is computed as the basename of the <output>
   file with .map appended.   If however the special character
   % is used then this will be replaced by the full path of the
   output file.  Additionally if there are any characters after the
   % symbol then .map will no longer be appended.

Cheers
   Nick