* ld: document static libraries when performing static linking [not found] <CADddEsCm7OpQ51GE+hfix2OntdnGmcWTppK8-g_2O175qS5VjA@mail.gmail.com> @ 2024-02-21 18:01 ` Christian Tramnitz 2024-02-21 20:55 ` Howard Chu 2024-02-28 14:02 ` Nick Clifton 0 siblings, 2 replies; 6+ messages in thread From: Christian Tramnitz @ 2024-02-21 18:01 UTC (permalink / raw) To: binutils Hello, For statically linked binaries one wants to keep track of external libraries that were pulled in at link time. I haven't found any initiatives or standards trying to achieve this yet, so I would like to make a proposal. ld already supports the `--depedency-file=` option to create a depfile, to some degree, documenting dependencies in Makefile format. While it's a good starting point, it's not sufficient. Would it be feasible to create another option, that a) just records dependencies to external, static-libs (within the depfile this could be achieved by looking for path names outside the build dir and having an .a file extension - but this is not neccessarily exhaustive). This doesn't need to be unique, post-processing is expected to take place anyway. b) documents the actual objects (enhanced: down to function level if information exists from compiling with -ffunction-sections and -fdata-sections) that were pulled in from the library archive, similar to the output from `--verbose` "(/usr/lib/gcc/x86_64-pc-linux -gnu/13/../../../../lib64/libc.a)stpcpy-avx2.o" c) outputs this in some sort of structured format for later processing, e.g. as simple as <object-requiring-the-library>,</some/library.a>,<requested-object-in-library> my-obj,/usr/lib64/libc.a,stpcpy-avx2.o my-obj,/usr/lib64/libc.a,stpcpy-avx2-rtm.o my-obj,/usr/lib64/libc.a,getsrvbynm.o my-obj,/usr/lib64/libc.a,stpcpy-evex.o d) - if ld doesn't support this already (couldn't find looking through man) - ld had some sort of no-op/pretend option that doesn't actually produce a linked output but only pretends it would? Purpose: The consumer of this information might be a distributor, keeping track of things in the distribution specific packaging database format. Or it could even be embedded into the resulting binary itself. Background: This output can be used to I) validate license compliance (it would make it possible to discover conflicting licenses of derived work) II) allow vulnerability tracking, when vulnerabilities existed in included static libraries (at build/link time) The latter would be further supported by b) as a vulnerability may not even exist in the resulting object if the vulnerable function wasn't included by linking. Best regards, Christian ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ld: document static libraries when performing static linking 2024-02-21 18:01 ` ld: document static libraries when performing static linking Christian Tramnitz @ 2024-02-21 20:55 ` Howard Chu 2024-02-22 3:49 ` Howard Chu 2024-02-28 14:02 ` Nick Clifton 1 sibling, 1 reply; 6+ messages in thread From: Howard Chu @ 2024-02-21 20:55 UTC (permalink / raw) To: Christian Tramnitz, binutils Christian Tramnitz wrote: > Hello, Hi - a linker plugin to handle static library dependencies already exists, it's been around nearly 4 years. https://sourceware.org/pipermail/binutils/2020-September/113188.html There was a bit of a push to make it an integral feature of ld instead of a plugin, but that has stalled. https://sourceware.org/pipermail/binutils/2023-February/126155.html I guess it's worth asking again, what help is needed to move integration forward? > > For statically linked binaries one wants to keep track of external > libraries that were pulled in at link time. I haven't found any > initiatives or standards trying to achieve this yet, so I would like > to make a proposal. > > ld already supports the `--depedency-file=` option to create a > depfile, to some degree, documenting dependencies in Makefile format. > While it's a good starting point, it's not sufficient. > > Would it be feasible to create another option, that > a) just records dependencies to external, static-libs (within the > depfile this could be achieved by looking for path names outside the > build dir and having an .a file extension - but this is not > neccessarily exhaustive). This doesn't need to be unique, > post-processing is expected to take place anyway. > b) documents the actual objects (enhanced: down to function level if > information exists from compiling with -ffunction-sections and > -fdata-sections) that were pulled in from the library archive, similar > to the output from `--verbose` > "(/usr/lib/gcc/x86_64-pc-linux > -gnu/13/../../../../lib64/libc.a)stpcpy-avx2.o" > c) outputs this in some sort of structured format for later > processing, e.g. as simple as > <object-requiring-the-library>,</some/library.a>,<requested-object-in-library> > my-obj,/usr/lib64/libc.a,stpcpy-avx2.o > my-obj,/usr/lib64/libc.a,stpcpy-avx2-rtm.o > my-obj,/usr/lib64/libc.a,getsrvbynm.o > my-obj,/usr/lib64/libc.a,stpcpy-evex.o > d) - if ld doesn't support this already (couldn't find looking through > man) - ld had some sort of no-op/pretend option that doesn't actually > produce a linked output but only pretends it would? > > > Purpose: > The consumer of this information might be a distributor, keeping track > of things in the distribution specific packaging database format. Or > it could even be embedded into the resulting binary itself. > > > Background: > This output can be used to > I) validate license compliance (it would make it possible to discover > conflicting licenses of derived work) > II) allow vulnerability tracking, when vulnerabilities existed in > included static libraries (at build/link time) > > The latter would be further supported by b) as a vulnerability may not > even exist in the resulting object if the vulnerable function wasn't > included by linking. > > > > Best regards, > Christian > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ld: document static libraries when performing static linking 2024-02-21 20:55 ` Howard Chu @ 2024-02-22 3:49 ` Howard Chu 0 siblings, 0 replies; 6+ messages in thread From: Howard Chu @ 2024-02-22 3:49 UTC (permalink / raw) To: Christian Tramnitz, binutils Howard Chu wrote: > Christian Tramnitz wrote: >> Hello, > > Hi - > a linker plugin to handle static library dependencies already exists, it's > been around nearly 4 years. https://sourceware.org/pipermail/binutils/2020-September/113188.html Ah no, I hadn't read the entire message before replying. My bad. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ld: document static libraries when performing static linking 2024-02-21 18:01 ` ld: document static libraries when performing static linking Christian Tramnitz 2024-02-21 20:55 ` Howard Chu @ 2024-02-28 14:02 ` Nick Clifton 2024-02-28 23:50 ` Christian Tramnitz 1 sibling, 1 reply; 6+ messages in thread From: Nick Clifton @ 2024-02-28 14:02 UTC (permalink / raw) To: Christian Tramnitz, binutils Hi Christian, > For statically linked binaries one wants to keep track of external > libraries that were pulled in at link time. I haven't found any > initiatives or standards trying to achieve this yet, so I would like > to make a proposal. > > ld already supports the `--depedency-file=` option to create a > depfile, to some degree, documenting dependencies in Makefile format. > While it's a good starting point, it's not sufficient. > > Would it be feasible to create another option, that > a) just records dependencies to external, static-libs (within the > depfile this could be achieved by looking for path names outside the > build dir and having an .a file extension - but this is not > neccessarily exhaustive). This doesn't need to be unique, > post-processing is expected to take place anyway. If post-processing is an option, then how about using the linker map file and parsing that ? Ie link with -Map=foo.map and then post-link scan foo.map. That would give you full details of which libraries were used, and which components from inside those libraries if they are archives. It might take a bit of work to get a reasonably robust scanner, but it would mean that it would work with any version of the linker, and you would not have the hassle of maintaining a linker plugin. > b) documents the actual objects (enhanced: down to function level if > information exists from compiling with -ffunction-sections and > -fdata-sections) that were pulled in from the library archive, similar > to the output from `--verbose` The linker map idea above should let you do this. > c) outputs this in some sort of structured format for later > processing, e.g. as simple as > <object-requiring-the-library>,</some/library.a>,<requested-object-in-library> > my-obj,/usr/lib64/libc.a,stpcpy-avx2.o > my-obj,/usr/lib64/libc.a,stpcpy-avx2-rtm.o > my-obj,/usr/lib64/libc.a,getsrvbynm.o > my-obj,/usr/lib64/libc.a,stpcpy-evex.o If you do not like the map idea then maybe adding some code to the linker at the places where it generates its map output, but this time in a format you like, would work ? > d) - if ld doesn't support this already (couldn't find looking through > man) - ld had some sort of no-op/pretend option that doesn't actually > produce a linked output but only pretends it would? It doesn't. Although of course you can always throw away any files it creates and start again. > Purpose: > The consumer of this information might be a distributor, keeping track > of things in the distribution specific packaging database format. Or > it could even be embedded into the resulting binary itself. There is another possibility... Check out the annobin project: https://sourceware.org/annobin/ The compiler plugins that are part of this project record notes that include the filename associated with each compilation unit. These notes are aggregated together during a link and end up in the resulting binary. So you could scan the binary for these notes, extract the filenames and you would end up with a list of (main) sources for the program. If your static libraries are also compiled with annobin enabled then they will contain notes of their own that will also contribute to the final binary. Cheers Nick ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ld: document static libraries when performing static linking 2024-02-28 14:02 ` Nick Clifton @ 2024-02-28 23:50 ` Christian Tramnitz 2024-02-29 10:58 ` Nick Clifton 0 siblings, 1 reply; 6+ messages in thread From: Christian Tramnitz @ 2024-02-28 23:50 UTC (permalink / raw) To: Nick Clifton, binutils Hi Nick, thanks for your detailed response! > If post-processing is an option, then how about using the linker map > file and parsing that ? Ie link with -Map=foo.map and then post-link > scan foo.map. That would give you full details of which libraries were > used, and which components from inside those libraries if they are > archives. -Map does indeed provide the required information - thanks a lot for the hint, that is already much better than parsing through verbose build logs or the dependency-file. However, like --depedency-file, it has one major drawback: when multiple targets exist, the "last" action overwrites the output file. I haven't seen an option to use a dynamic file names (foo-$$.map doesn't seem to work). > There is another possibility... > > Check out the annobin project: https://sourceware.org/annobin/ > > The compiler plugins that are part of this project record notes that > include the filename associated with each compilation unit. These > notes are aggregated together during a link and end up in the resulting > binary. So you could scan the binary for these notes, extract the > filenames and you would end up with a list of (main) sources for the > program. If your static libraries are also compiled with annobin enabled > then they will contain notes of their own that will also contribute to > the final binary. I indeed looked into annobin initially, but did not get that the notes also contain references to the sources/archive/library (only compile options as per https://sourceware.org/annobin/annobin.html/Plugins.html). But since you're the author, you know better for sure. Will have a look again as it may cover large parts of what I'm trying to achieve already. Except for the lookup which package owns that file of course. But that could be a post-processing step again. Best regards, Christian ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ld: document static libraries when performing static linking 2024-02-28 23:50 ` Christian Tramnitz @ 2024-02-29 10:58 ` Nick Clifton 0 siblings, 0 replies; 6+ messages in thread From: Nick Clifton @ 2024-02-29 10:58 UTC (permalink / raw) To: Christian Tramnitz, binutils Hi Christian, > -Map does indeed provide the required information - thanks a lot for > the hint, that is already much better than parsing through verbose > build logs or the dependency-file. > However, like --depedency-file, it has one major drawback: when > multiple targets exist, the "last" action overwrites the output file. > I haven't seen an option to use a dynamic file names (foo-$$.map > doesn't seem to work). Ah - but directory names do. :-) Check out the linker manual's description of the -Map option. In particular: Specifying a directory as <mapfile> causes the linker map to be written as a file inside the directory. Normally name of the file inside the directory is computed as the basename of the <output> file with .map appended. If however the special character % is used then this will be replaced by the full path of the output file. Additionally if there are any characters after the % symbol then .map will no longer be appended. Cheers Nick ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-02-29 10:58 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CADddEsCm7OpQ51GE+hfix2OntdnGmcWTppK8-g_2O175qS5VjA@mail.gmail.com> 2024-02-21 18:01 ` ld: document static libraries when performing static linking Christian Tramnitz 2024-02-21 20:55 ` Howard Chu 2024-02-22 3:49 ` Howard Chu 2024-02-28 14:02 ` Nick Clifton 2024-02-28 23:50 ` Christian Tramnitz 2024-02-29 10:58 ` Nick Clifton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).