--as-needed by default

--as-needed by default

[Mike Hearn]

Hi,

I'm wondering why the --as-needed ld option is not used by default.
Presumably it could break software, and I'm wondering if this is actually
the case or if there is some other reason.

If it breaks backwards compatibility, is it known how widespread the
problems are?

I understand that changing ld itself may not be what is wanted. So in that
case the question would become, "Why is --as-needed not in the gcc
specfiles so it's used automatically?".

thanks -mike

Re: --as-needed by default

[Alan Modra]

On Thu, Mar 03, 2005 at 02:30:11PM +0000, Mike Hearn wrote:
> I'm wondering why the --as-needed ld option is not used by default.

Code supporting --as-needed is relatively new and quite complicated.
It is likely to have bugs.  Also, some projects might rely on the
existing ld behaviour of always adding DT_NEEDED tags for shared
libraries.

-- 
Alan Modra
IBM OzLabs - Linux Technology Centre

Re: --as-needed by default

[Zack Weinberg]

Alan Modra <amodra@bigpond.net.au> writes:

> On Thu, Mar 03, 2005 at 02:30:11PM +0000, Mike Hearn wrote:
>> I'm wondering why the --as-needed ld option is not used by default.
>
> Code supporting --as-needed is relatively new and quite complicated.
> It is likely to have bugs.  Also, some projects might rely on the
> existing ld behaviour of always adding DT_NEEDED tags for shared
> libraries.

I've argued in the past for --as-needed to be the default, because it
makes shared library semantics match static library semantics, and
because it takes a quite contrived application to depend on DT_NEEDEDs
for libraries that satisfy no symbols. (You would need a library that
did something useful, e.g. via a global constructor, just by being
linked into your application, even if you referenced no symbols from it.)

However, Alan's concerns are valid, especially the "it's new and
complicated and might be buggy" ones.  Is this apropos the thread on
ubuntu-devel?  If so, why don't y'all change the default, rebuild the
whole distro, and see what breaks?  If you find bugs, then they can be
fixed, and if you find that nothing breaks, that'll be solid evidence
that making --as-needed the default is safe.

zw

Re: --as-needed by default

[Mike Hearn]

On Thu, 03 Mar 2005 15:40:53 -0800, Zack Weinberg wrote:
> However, Alan's concerns are valid, especially the "it's new and
> complicated and might be buggy" ones.  Is this apropos the thread on
> ubuntu-devel?  If so, why don't y'all change the default, rebuild the
> whole distro, and see what breaks?  If you find bugs, then they can be
> fixed, and if you find that nothing breaks, that'll be solid evidence
> that making --as-needed the default is safe.

Yep, that's what I just suggested. I want to use it in autopackage too. So
let's see how it goes ...

thanks -mike

Re: --as-needed by default

[Alexandre Oliva]

On Mar  3, 2005, Zack Weinberg <zack@codesourcery.com> wrote:

> I've argued in the past for --as-needed to be the default, because it
> makes shared library semantics match static library semantics,

In what sense?  I can come up with numerous examples in which it still
wouldn't match.  In fact, in *most* cases it still wouldn't match.  I
can't however figure out what particular example would make them
match.

> and because it takes a quite contrived application to depend on
> DT_NEEDEDs for libraries that satisfy no symbols.

Not really.  Consider a library libfoo that is linked with libbar not
because it uses symbols from libbar, but because libbar used to be
part of its API before the code was split into a separate library.

Dropping libbar from libfoo's DT_NEEDED would break programs linked
with libfoo, and even force libfoo to change its SONAME.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: --as-needed by default

[Mike Hearn]

Alexandre Oliva wrote:
> Dropping libbar from libfoo's DT_NEEDED would break programs linked
> with libfoo, and even force libfoo to change its SONAME.

Good point, but can you think of any libraries that have actually done 
this? It seems splitting things out into separate libraries and relying 
on this quirk of ELF to work is a rather bad idea but I'm sure somebody 
has done it somewhere :)

I guess this needs to be enabled/disabled on a case by case basis then. 
Shame.

thanks -mike

Re: --as-needed by default

[Zack Weinberg]

Alexandre Oliva <aoliva@redhat.com> writes:

> On Mar  3, 2005, Zack Weinberg <zack@codesourcery.com> wrote:
>
>> I've argued in the past for --as-needed to be the default, because it
>> makes shared library semantics match static library semantics,
>
> In what sense?  I can come up with numerous examples in which it still
> wouldn't match.  In fact, in *most* cases it still wouldn't match.  I
> can't however figure out what particular example would make them
> match.

In the sense that, if you write -lfoo on the link command line, and
libfoo.a is a static library which satisfies no undefined symbols from
the link up to this point (assume -( -) are not in use) then no
objects from libfoo.a will be included in the final linked image, even
if there might have been some other reason for them to be included
(such as C++ global constructors).

Autoconf's AC_CHECK_LIB macro assumes this semantic.  Yes, there is
now AC_SEARCH_LIBS that doesn't, but AC_CHECK_LIB is not deprecated
and is still widely used.  Because AC_CHECK_LIB is used in a
widely-copied recipe for getting at the socket API on Solaris, there
are 238 binaries on my Linux box that have an unnecessary DT_NEEDED
for libnsl.so.1.¹

>> and because it takes a quite contrived application to depend on
>> DT_NEEDEDs for libraries that satisfy no symbols.
>
> Not really.  Consider a library libfoo that is linked with libbar not
> because it uses symbols from libbar, but because libbar used to be
> part of its API before the code was split into a separate library.
>
> Dropping libbar from libfoo's DT_NEEDED would break programs linked
> with libfoo, and even force libfoo to change its SONAME.

I doubt this case has ever arisen in practice, because it is
impossible to do a similar backward compatibility hack with static
libraries, and most people still support static libraries.  Also,
wouldn't a GROUP() script be a better choice (as e.g. used in glibc)?

zw

¹ On a glibc system, the only thing in libnsl is NIS helper routines,
so the only things that should be linked with libnsl are the NIS
servers and utilities, and a few NSS modules.

Re: --as-needed by default

[Alexandre Oliva]

On Mar  8, 2005, Zack Weinberg <zack@codesourcery.com> wrote:

> In the sense that, if you write -lfoo on the link command line, and
> libfoo.a is a static library which satisfies no undefined symbols from
> the link up to this point (assume -( -) are not in use) then no
> objects from libfoo.a will be included in the final linked image


> Autoconf's AC_CHECK_LIB macro assumes this semantic.

AC_CHECK_LIB isn't a test for whether a library is needed to satisfy a
symbol.  It's a test for whether the library exists and can be linked
with and, while at that, making sure a symbol is defined when the
library is linked in (even if it would also be defined without it).

So changing the linker in a non-straightforward way to work around a
bug in gazillions of programs doesn't sound like a good plan to me.

Also, arguing that it would make the link-time semantics of shared
libs closer to that of static libs isn't necessarily a strong
argument.  Consider the argument to stop opening DT_NEEDED libraries
of dynamic libraries listed in the command line, so that people have
to specify dependencies explicitly, just like they have to do with
static libs.

>> Consider a library libfoo that is linked with libbar not because it
>> uses symbols from libbar, but because libbar used to be part of its
>> API before the code was split into a separate library.

>> Dropping libbar from libfoo's DT_NEEDED would break programs linked
>> with libfoo, and even force libfoo to change its SONAME.

> I doubt this case has ever arisen in practice, because it is
> impossible to do a similar backward compatibility hack with static
> libraries, and most people still support static libraries.  Also,
> wouldn't a GROUP() script be a better choice (as e.g. used in glibc)?

For link-time, yes.  But the issue at hand is the construction of
libfoo, that would, after your proposed change, require a
--no-as-needed to make sure the dep on the libbar dynamic lib is not
dropped.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: --as-needed by default

[Zack Weinberg]

Alexandre Oliva <aoliva@redhat.com> writes:

> AC_CHECK_LIB isn't a test for whether a library is needed to satisfy
> a symbol.  It's a test for whether the library exists and can be
> linked with and, while at that, making sure a symbol is defined when
> the library is linked in (even if it would also be defined without it).

You are making a distinction that is not made by the Autoconf manual,
nor is it evident to the vast majority of autoconf users.

> So changing the linker in a non-straightforward way to work around a
> bug in gazillions of programs doesn't sound like a good plan to me.

I don't agree that it is a non-straightforward change, and I would far
rather do that than attempt to fix each of the gazillions of programs
individually.

>>> Consider a library libfoo that is linked with libbar not because it
>>> uses symbols from libbar, but because libbar used to be part of its
>>> API before the code was split into a separate library.
>>>
>>> Dropping libbar from libfoo's DT_NEEDED would break programs linked
>>> with libfoo, and even force libfoo to change its SONAME.
>>
>> I doubt this case has ever arisen in practice, because it is
>> impossible to do a similar backward compatibility hack with static
>> libraries, and most people still support static libraries.  Also,
>> wouldn't a GROUP() script be a better choice (as e.g. used in glibc)?
>
> For link-time, yes.  But the issue at hand is the construction of
> libfoo, that would, after your proposed change, require a
> --no-as-needed to make sure the dep on the libbar dynamic lib is not
> dropped.

What I was saying is, I don't think anyone has ever attempted, or is
likely ever to attempt, to construct such a libfoo.so, because it
wouldn't help with the corresponding libfoo.a.  People instead bite
the bullet and increment libfoo's soname.  Therefore I do not think
this is a case to be worried about.

zw

Re: --as-needed by default

[Alexandre Oliva]

On Mar  8, 2005, Zack Weinberg <zack@codesourcery.com> wrote:

> What I was saying is, I don't think anyone has ever attempted, or is
> likely ever to attempt, to construct such a libfoo.so, because it
> wouldn't help with the corresponding libfoo.a.

libfoo.a could be the linker script you suggest.

> People instead bite the bullet and increment libfoo's soname.

Yuck.  I'd hope not.  Breaking binary compatibility shouldn't be done
lightly.

> Therefore I do not think this is a case to be worried about.

We'll have to agree to disagree on this one.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: --as-needed by default

[Zack Weinberg]

Alexandre Oliva <aoliva@redhat.com> writes:

> On Mar  8, 2005, Zack Weinberg <zack@codesourcery.com> wrote:
>> People instead bite the bullet and increment libfoo's soname.
>
> Yuck.  I'd hope not.  Breaking binary compatibility shouldn't be done
> lightly.

Um, prescriptivism is all very well, but I am telling you what people
actually do, from my recollecton of times when this sort of thing has
happened to libraries packaged by Debian (so I see the upgrade notice
go by, and I see the soname get bumped).

zw

Re: --as-needed by default

[Thorsten Glaser]

Alexandre Oliva dixit:

>Dropping libbar from libfoo's DT_NEEDED would break programs linked
>with libfoo, and even force libfoo to change its SONAME.

So, really? Wow.

Architectures with static linkage have had that behaviour for ages.
Even the programme pulling libfoo will have to change its Makefile.

And there are systems out there, such as OpenBSD, to name just one,
which do *not* support DT_NEEDED, in order to have the same behavi-
our on static and shared arches, to reduce the amount of breakage &
workarounds required.

//mirabile
-- 
> [...] Echtzeit hat weniger mit "Speed"[...] zu tun, sondern damit, daß der
> richtige Prozeß voraussagbar rechtzeitig sein Zeitscheibchen bekommt.
Wir haben uns[...] geeinigt, dass das verwendete Echtzeit-Betriebssystem[...]
weil selbst einfachste Operationen *echt* *Zeit* brauchen.	(aus d.a.s.r)