Re: [PATCH] dependency list for static libraries

[Howard Chu <hyc@symas.com>]

Fangrui Song wrote:
> Thanks to Nick for mentioning me :)
> 
> On 2020-09-23, Howard Chu wrote:
>> Nick Clifton wrote:

>> I can send an updated patch after the major issue is addressed.

>>> The major issue is that I would really like for this extension to be supported
>>> by the LLVM community as well.  It would be a shame to add it to the binutils
>>> only to have a different solution implemented there.  I might have misread the
>>> emails but I believe that Fangrui still has some misgivings about this approach.
>>> Is that correct ?  If so, then I would very much like to see them resolved
>>> before we commit the patch.
>>
>> It still seems pretty obvious that handling dependencies once at the library
>> level is more efficient than sticking a bunch of free-form notes in every single
>> object file. And it should also be obvious that choice of library is a buildtime
>> decision, not a decision made solely at the time the source code is written.
>> (And for shared libraries, it's a runtime decision - even further removed from
>> the time of code being written.)
> 
> From my experience (I have investigated hundreds of link issues),
> archive link errors are usually caused by the following two problems:
> 
> * There is a backward reference between two archives. This is related to a.out/ELF style linker behaviors
> (https://sourceware.org/pipermail/binutils/2020-September/113194.html ).
>   + A dependency exists but is incorrectly omitted. The linker actually captures a layering problem.
>   + A dependency is intentionally omitted. If the linker allows backward references, this will not be a problem.
> 
>   I have recently thought a lot on the topic and written http://lld.llvm.org/ELF/warn_backrefs.html

That is not a problem I encounter frequently, and is not in the scope of what
I'm concerned with.

> * Some archives are incorrectly omitted: the executable uses A, but A's
>   dependency B is not on the command line.  Now I hope Howard can clarify on this
>   topic:) What do you want to achieve with the dependency recorded in the archive?
>   I assume that you want the linker to smartly link B.

Yes.

>   This is indeed the #pragma
>   comment(lib, ...) and .deplibs feature clang/LLD support.

Perhaps, but I remain skeptical.

>   The proposal does not mention the ld part, which seems important. How
>   does ld handle __.LIBDEP ?

ld effectively inserts the flags from __.LIBDEP into its command line immediately
following its archive.

>   + The executable does not references B. I am still unsure why you want to move the dependency information from the
>     object file to the archive.

The most common case is we reference a library we know about, and it has dependencies
we don't know about. An example I currently have in mind is libzmq, which may or may
not be built against libsodium, and may or may not have other library dependencies
as well. The "traditional" solution to this for the past couple decades has been to
use pkg-config or libtool .la files to propagate this information but those are both
ugly hacks that require excessive tooling to support. The information clearly belongs
to the archive file itself.

>     - First, I want to mention that in some cases archives are not needed.

Irrelevant. I'm talking about software that is shipped in library form.

>     - Moving the dependency to the archive can actually increases the work for
>       the linker.  Say the two members of A are a0.o and a1.o. If a0.o depends on B
>       but a0.o is not selected, then B does not need to be linked. Having the
>       dependency in the archive will cause B to be linked.
>     - How do you ensure the dependency information does not become stale? i.e.
>       How do you maintain the dependency information when members are added to/removed from
>       the archive? The information is more difficult to maintain than the index, which only contains
>       the definitions. You may argue that the archive is finalized after it is created - then
>       we go back to square one, --start-lib may be more suitable.

How do you maintain the dependency information of a shared library? This is a
problem developers already deal with, not a new problem.
> 
> ---
> 
> Last,
> 
>>  fprintf (s, _("  [L LIBDEPS]  - specify dependencies of this library\n"));
> 
> In llvm-ar, 'L' is an extension used with 'q' to add all members of an archive to the current archive.
> It is similar to ADDLIB (https://sourceware.org/binutils/docs/binutils/ar-scripts.html )
> Hope the two tools don't have conflicting operations.

We could just use 'l' instead, which has historically been a no-op. The specific choice
of letter option here is not significant.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/