Re: [PATCH] ld: Add more tests for --as-needed

["H.J. Lu" <hjl.tools@gmail.com>]

On Wed, Sep 9, 2020 at 8:30 PM Alan Modra <amodra@gmail.com> wrote:
>
> On Wed, Sep 09, 2020 at 09:16:40AM -0700, H.J. Lu via Binutils wrote:
> > commit fb7331ae2b551247a33e8da5387ebc910c4a7308
> > Author: Alan Modra <amodra@gmail.com>
> > Date:   Wed Sep 9 12:16:23 2020 +0930
> >
> >     lto-18 test
> >
> > added tests to show the reason to use IR symbols when deciding an
> > --as-needed library should be loaded.
>
> Exactly.  The testcase compiles and links fine without LTO, and
> compiles and links with LTO if references from IR symbols are used to
> decide whether a library is needed.  Reverting 1e3b96fd6cf0 results in
>
> FAIL: lto-18 (1)
> FAIL: lto-18 (2)
>
> You kept asking for a testcase in the thread where I committed
> 1e3b96fd6cf0, and now you have one.  I think the testcase is valid,
> but like many testcases, is nasty.
>
> That testcase is not the only reason why I think we should be using IR
> symbol refs to decide that an as-needed library is needed.
>
> Here are my reasons:
>
> 1) --as-needed is supposed to load a shared library if there is a
> non-weak undefined symbol remaining from previously loaded objects,
> defined by the library.  By previously loaded objects we mean those
> loaded on the command line or in scripts prior to the library, similar
> to the logic used by the linker to extract objects from archives.
> Archive searches must extract objects to satisfy IR references,
> because the object in the archive might be an LTO object.  Since
> --as-needed was intended to be somewhat modelled on the way ld
> extracts objects from archives, it makes sense to do the same for
> --as-needed libraries.
>
> 2) Trying to only emit DT_NEEDED for --as-needed libraries based on
> the final LTO set of symbols is complicated, and has led to a number
> of bug reports in the past.  Those bug reports have meant that we keep
> some symbol state from an as-needed library that isn't marked as
> needed at that point in linking, to feed to the LTO recompilation.
> While doing that for references generally won't break anything if
> the as-needed library is also found to be not needed after LTO, I
> worry that doing the same for symbol definitions will lead to
> breakage.  If a symbol being defined might affect LTO output, but the
> LTO output not have a reference to the symbol in question, then that
> could result in an as-needed library not being marked as needed on the
> second pass through libraries after LTO.  The result would be that ld
> tells LTO a symbol is defined but it ends up undefined.
>
> 3) Further patches complicating the linker like the one you posted at
> https://sourceware.org/pipermail/binutils/2020-September/113227.html
> are likely to be necessary, if we go back to IR symbols not marking
> as-needed libraries as needed.
>
> >  But the requirement of the
> > --as-needed library doesn't come from the IR inputs since the IR symbol
> > reference of the --as-needed library is removed by LTO.  The requirement
> > comes from another --as-needed library which is referenced by the LTO
> > output as shown by PR ld/26590.
>
> PR26590 merely shows --as-needed working as designed.
>
> >  Not use IR symbols when deciding an
> > --as-needed library should be loaded exposes the same issue with input
> > --as-needed libraries as non-LTO inputs.  The current ld behavior is
> > desirable since it sures that both lazy and non-lazy bindings work the
> > same way.
> >
> > Add more tests for --as-needed and add tmpdir/liblto-18b.so DT_NEEDED
> > to liblto-18c.so.
>
> No, you may not commit the lto.exp patch.  Shared libraries are not
> required to have DT_NEEDED entries.  Imagine for moment that
> liblto-18b.so is a library from author B that has some undefined
> functions that normally would be provided in the executable.  Author B
> doesn't even know about liblto-18c.so, thus no DT_NEEDED for it.
> Author C has a similar liblto-18c.so library with some undefined
> functions normally provided in the executables.  Author A writing the
> app wants to use both liblto-18b.so and liblto-18c.so and realizes
> that they satisfy each other's undefined symbols.  So he doesn't need
> to write those functions in the executable.  He gets everything
> working without LTO, then turns on LTO and it fails mysteriously.
>

The problem is with liblto-18c.so and liblto-18b.so.  The same problem
can show up without LTO.  liblto-18c.so and liblto-18b.so should be fixed
if they should work in all cases, with and without LTO.

One advantage of LTO is to remove unused IR symbol references, including
unnecessary DT_NEEDED.  The current ld no longer does that.

-- 
H.J.