Re: Binutils linker bug?

Re: Binutils linker bug?

[Mark Kettenis]

[ For the people on binutils@sourceware: this is a followup on a
  binutils bug report that was sent to bug-utils@gnu.org.  I added
  binutils@sourceware to give it a bit more exposure. ]

jongk@cs.utwente.nl (Kars de Jong) writes:

> Note that this testcase doesn't entirely represent the situation: in the
> cases I ran into the bug (JDK 1.2.1 and ToolTalk version 1.3) the main
> program that used the lib didn't call any pthread functions itself!
> But I couldn't get the testcase to pull in pthread symbols without adding the
> pthread_self() call to the main program.

I think you get the same result if you use write() instead of
pthread_self() in your main program.  libpthread overrides several
system calls that can block for a long time to make them cancellation
points.  In this context the fact that `write' is weak in libc.so but
an ordinary symbol in libpthread.so.

To me, this indicates that Kars's problems might be more complicated
than they would seem at first sight.

I think that everybody agrees that the testcase Kars gave (with main
explicitly calling pthread_self()) is broken in the sense that he
should link his program explicitly against libpthread.so.  Would it be
a good idea to let `ld' issue a warning if he doesn't link against
libpthread.so?  Should `ld' avoid recording the version dependencies
in this case?

However, if the program references `write' instead of `pthread_self'
things are a little different.  I think the desired behaviour here is
to record that the program needs the appropriate version of `write'
from libc.so instead of libpthread.so (which gets sucked in via an other
library we link against).  Is this failing because `write' is weak in
libc.so and isn't in libpthread.so?  Or would it be a problem anyhow,
that is, even when `write' is weak in libpthread.so too?

Mark

Re: Binutils linker bug?

[Ian Lance Taylor]

   From: Mark Kettenis <kettenis@wins.uva.nl>
   Date: 08 Nov 1999 22:23:22 +0100

   > Note that this testcase doesn't entirely represent the situation: in the
   > cases I ran into the bug (JDK 1.2.1 and ToolTalk version 1.3) the main
   > program that used the lib didn't call any pthread functions itself!
   > But I couldn't get the testcase to pull in pthread symbols without adding the
   > pthread_self() call to the main program.

   I think you get the same result if you use write() instead of
   pthread_self() in your main program.  libpthread overrides several
   system calls that can block for a long time to make them cancellation
   points.  In this context the fact that `write' is weak in libc.so but
   an ordinary symbol in libpthread.so.

   To me, this indicates that Kars's problems might be more complicated
   than they would seem at first sight.

   I think that everybody agrees that the testcase Kars gave (with main
   explicitly calling pthread_self()) is broken in the sense that he
   should link his program explicitly against libpthread.so.  Would it be
   a good idea to let `ld' issue a warning if he doesn't link against
   libpthread.so?

I don't think so.  We added recursive lookup of libraries which are
NEEDED by other libraries included in the link specifically to avoid
issuing warnings.

When considering whether ld should issue a warning, we should ask what
the dynamic linker will do.  ld should not issue a warning if the
dynamic linker will handle things correctly.  ld should issue a
warning if the dynamic linker will not handle things correctly.

To put it another way, I don't agree that the testcase is broken.
There is no requirement that he explicitly link against libpthread.so.
I agree that it would be more normal to do so.

   Should `ld' avoid recording the version dependencies
   in this case?

I don't think this would be correct.  After all, if version
dependencies mean anything at all, you have to record them when a
program calls a versioned symbol in a shared library.  That is what is
happening here: a program is calling the versioned symbol
pthread_self.  If ld doesn't store the version dependencies, what are
we saying?  That the versions don't matter?  But then why should we
bother with them at all?

   However, if the program references `write' instead of `pthread_self'
   things are a little different.  I think the desired behaviour here is
   to record that the program needs the appropriate version of `write'
   from libc.so instead of libpthread.so (which gets sucked in via an other
   library we link against).  Is this failing because `write' is weak in
   libc.so and isn't in libpthread.so?  Or would it be a problem anyhow,
   that is, even when `write' is weak in libpthread.so too?

Again, the question to ask is what the dynamic linker will do.  The
behaviour of ld should be guided by that.  If the dynamic linker will
use `write' from libpthread.so, then I think ld should as well.

Ian

Re: Binutils linker bug?

[Kars de Jong]

On Mon, Nov 08, 1999 at 10:23:22PM +0100, Mark Kettenis wrote:
> > But I couldn't get the testcase to pull in pthread symbols without adding the
> > pthread_self() call to the main program.
> 
> I think you get the same result if you use write() instead of
> pthread_self() in your main program.  libpthread overrides several
> system calls that can block for a long time to make them cancellation
> points.  In this context the fact that `write' is weak in libc.so but
> an ordinary symbol in libpthread.so.

Well, I tested this (using write()) and that got the testcase I really meant
to send :) No calls to pthread specific functions and still a libpthread
version requirement in the main program.

Wether this is to be bugfixed by glibc or binutils, or both,
I leave to you :)

Kars.
-- 
------------------------------------------------------------------------------
Kars de Jong             Signaalkamp rules the waves!       Turrican@Discworld
--------======]**-----|      jongk@cs.utwente.nl      |-----**[======---------

Re: Binutils linker bug?

[Geoff Keating]

> Date: 8 Nov 1999 16:30:47 -0500
> From: Ian Lance Taylor <ian@zembu.com>
> CC: jongk@cs.utwente.nl, binutils@sourceware.cygnus.com

> When considering whether ld should issue a warning, we should ask what
> the dynamic linker will do.  ld should not issue a warning if the
> dynamic linker will handle things correctly.  ld should issue a
> warning if the dynamic linker will not handle things correctly.
> 
> To put it another way, I don't agree that the testcase is broken.
> There is no requirement that he explicitly link against libpthread.so.
> I agree that it would be more normal to do so.

I'd disagree with this.  Consider the situation when we have the
following:

a shared library, libsomething.so, linked against libpthread.so.0.9.0
a system with libpthread.so.0.9.1 in /usr/lib, which might be
symlinked as version 0.9.0 or might be different.

and we are building a main program, which uses libsomething, and which
calls routines that are in libpthread but does not link against it.

Which version of the routines should we be using?

IMHO, this should be a hard error, "symbol not found".

>    However, if the program references `write' instead of `pthread_self'
>    things are a little different.  I think the desired behaviour here is
>    to record that the program needs the appropriate version of `write'
>    from libc.so instead of libpthread.so (which gets sucked in via an other
>    library we link against).  Is this failing because `write' is weak in
>    libc.so and isn't in libpthread.so?  Or would it be a problem anyhow,
>    that is, even when `write' is weak in libpthread.so too?
> 
> Again, the question to ask is what the dynamic linker will do.  The
> behaviour of ld should be guided by that.  If the dynamic linker will
> use `write' from libpthread.so, then I think ld should as well.

Consider a shared library that uses libpthread.so as part of its
internal processing, and a main program that is not threaded and does
not use libpthread, but is linked against this shared library.

Now consider what happens when the shared library is modified, so that
it no longer uses libpthread.

Shouldn't the main program still work?

-- 
Geoffrey Keating <geoffk@cygnus.com>

Re: Binutils linker bug?

[Ian Lance Taylor]

   Date: Tue, 9 Nov 1999 12:29:00 +1100
   From: Geoff Keating <geoffk@ozemail.com.au>

   > When considering whether ld should issue a warning, we should ask what
   > the dynamic linker will do.  ld should not issue a warning if the
   > dynamic linker will handle things correctly.  ld should issue a
   > warning if the dynamic linker will not handle things correctly.
   > 
   > To put it another way, I don't agree that the testcase is broken.
   > There is no requirement that he explicitly link against libpthread.so.
   > I agree that it would be more normal to do so.

   I'd disagree with this.  Consider the situation when we have the
   following:

   a shared library, libsomething.so, linked against libpthread.so.0.9.0
   a system with libpthread.so.0.9.1 in /usr/lib, which might be
   symlinked as version 0.9.0 or might be different.

   and we are building a main program, which uses libsomething, and which
   calls routines that are in libpthread but does not link against it.

   Which version of the routines should we be using?

   IMHO, this should be a hard error, "symbol not found".

A hard error from which program?  The program linker or the dynamic
linker?

Are you suggesting that the program linker should reject an attempt to
use a symbol from a shared library which is only included indirectly?
That would break existing systems.  For example, on Solaris, the
shared X library includes other libraries.  If we rejected references
to shared libraries included indirectly, people would no longer be
able to simply link against -lX11.  This would also be incompatible
with the Solaris linker.

   >    However, if the program references `write' instead of `pthread_self'
   >    things are a little different.  I think the desired behaviour here is
   >    to record that the program needs the appropriate version of `write'
   >    from libc.so instead of libpthread.so (which gets sucked in via an other
   >    library we link against).  Is this failing because `write' is weak in
   >    libc.so and isn't in libpthread.so?  Or would it be a problem anyhow,
   >    that is, even when `write' is weak in libpthread.so too?
   > 
   > Again, the question to ask is what the dynamic linker will do.  The
   > behaviour of ld should be guided by that.  If the dynamic linker will
   > use `write' from libpthread.so, then I think ld should as well.

   Consider a shared library that uses libpthread.so as part of its
   internal processing, and a main program that is not threaded and does
   not use libpthread, but is linked against this shared library.

   Now consider what happens when the shared library is modified, so that
   it no longer uses libpthread.

   Shouldn't the main program still work?

Yes, clearly.  I hope I haven't said anything to suggest otherwise.

Ian

Re: Binutils linker bug?

[Geoff Keating]

> Date: 8 Nov 1999 22:30:04 -0500
> From: Ian Lance Taylor <ian@zembu.com>

> Are you suggesting that the program linker should reject an attempt to
> use a symbol from a shared library which is only included indirectly?
> That would break existing systems.  For example, on Solaris, the
> shared X library includes other libraries.  If we rejected references
> to shared libraries included indirectly, people would no longer be
> able to simply link against -lX11.  This would also be incompatible
> with the Solaris linker.

I am suggesting exactly this.  I just checked, and the solaris 2.5.1
linker does what I am suggesting:

geoffk@sloth 2:38pm>cat > t.c
#include <stdio.h>
extern int XSolarisOvlGetPaintType();
int main(void)
{ printf("XSolarisOvlGetPaintType is at %p\n", XSolarisOvlGetPaintType);  
  return 0;
}

geoffk@sloth 2:39pm>gcc t.c -o t -L/usr/openwin/lib -lX11
Undefined                       first referenced
 symbol                             in file
XSolarisOvlGetPaintType             /tmp/ccTuJchj.o  (symbol belongs to implicit dependency /usr/openwin/lib/libXext.so.0)
ld: fatal: Symbol referencing errors. No output written to t
collect2: ld returned 1 exit status

but if you instead write

geoffk@sloth 2:39pm>gcc t.c -o t -L/usr/openwin/lib -lX11 -lXext

it works.  They even print the library name that you need to add to
the link line, which we should try to do too :-).

Here, libX11 has:

geoffk@sloth 2:40pm>objdump -p /usr/openwin/lib/libX11.so.4 | grep libXext
  NEEDED      libXext.so.0

and libXext has:

geoffk@sloth 2:41pm>objdump -T /usr/openwin/lib/libXext.so.0 | grep XSolarisOvlGetPaintType
0000a978 g    DF .text  000000ec XSolarisOvlGetPaintType

>    >    However, if the program references `write' instead of `pthread_self'
>    >    things are a little different.  I think the desired behaviour here is
>    >    to record that the program needs the appropriate version of `write'
>    >    from libc.so instead of libpthread.so (which gets sucked in via an other
>    >    library we link against).  Is this failing because `write' is weak in
>    >    libc.so and isn't in libpthread.so?  Or would it be a problem anyhow,
>    >    that is, even when `write' is weak in libpthread.so too?
...

I think this is superceded by the above comment.  If you're not
allowing indirect dynamic symbol referencing, the linker should assume
that the symbol that applies is the one in libc if that is the one it
can see.

-- 
Geoffrey Keating <geoffk@cygnus.com>

Re: Binutils linker bug?

[Ian Lance Taylor]

   Date: Tue, 9 Nov 1999 14:46:57 +1100
   From: Geoff Keating <geoffk@ozemail.com.au>

   > Are you suggesting that the program linker should reject an attempt to
   > use a symbol from a shared library which is only included indirectly?
   > That would break existing systems.  For example, on Solaris, the
   > shared X library includes other libraries.  If we rejected references
   > to shared libraries included indirectly, people would no longer be
   > able to simply link against -lX11.  This would also be incompatible
   > with the Solaris linker.

   I am suggesting exactly this.  I just checked, and the solaris 2.5.1
   linker does what I am suggesting:

Hmmm, I guess I misremembered about the Solaris linker.

As I recall, I put in the indirect referencing stuff at the request of
Roland McGrath for glibc.  It's been in there since 1995, and I'm a
bit reluctant to take it out now.

Let's go back to why it's a bad idea.  You're right that it can permit
a link to succeed which can then break if a shared library is changed.
But, after all, that is always true: any link can break if a shared
library is changed.  Why is this different?

Ian

Re: Binutils linker bug?

[Roland McGrath]

> As I recall, I put in the indirect referencing stuff at the request of
> Roland McGrath for glibc.  It's been in there since 1995, and I'm a
> bit reluctant to take it out now.

I can believe I requested this at that time.  But now I am inclined to
think the behavior Geoff wants does perhaps make the most sense.  It's also
a bit unpleasant to be incompatible with other canonical ELF linkers on
something like this.

This affects the Hurd, where libc depends on some other shared objects to
provide symbols called by libc itself.  What's affected are programs that
use some of these symbols from the secondary libraries directly.  Currently
a program linking against just libc can refer to the symbols defined in
libc's dependency libraries; it links ok, and gets a DT_NEEDED only for
libc and not for the indirect dependency's soname.  A change to the Solaris
behavior would require these programs to add the appropriate libraries to
their link lines.  Given the exact details of what these libraries are in
the Hurd, I don't think it would be a problem for us to adapt to this by
changing the affected programs' makefiles.

> Let's go back to why it's a bad idea.  You're right that it can permit
> a link to succeed which can then break if a shared library is changed.
> But, after all, that is always true: any link can break if a shared
> library is changed.  Why is this different?

The rationale I see is that one might reasonably consider a given shared
object's set of dependency shared objects and the symbols they provide to
be an implementation detail of that shared object, rather than contributing
to that shared object's ABI.  It would then be reasonable to want to
replace that shared object with a new one that has a compatible ABI
(defined as just the symbols defined in that shared object itself), but has
a different implementation that includes a different set of dependency
shared objects.

The benefit of the Solaris behavior is that it allows a shared object to
use other shared objects as an incidental implementation detail, and later
be replaced with a shared object using a different set of other objects
(with different symbols) for its implementation, without breaking the ABI
compatibility that could have reasonably been expected by anyone who
succeeded in linking against the old object.

Conversely, the benefit of the current GNU behavior is that it permits a
shared object to use other shared objects as an incidental implementation
detail, including letting those other objects define some of the symbols of
"its" ABI.  This is something that I would like to be able to exploit as
well.  However, it now occurs to me that DT_AUXILIARY does exactly this
just fine and I don't see why I wouldn't be happy using that for whatever
new arrangements come up in the future.  It may well be that my original
thinking on this and my request to you for this behavior in GNU ld predate
Sun's invention of the DT_AUXILIARY extension (or at least my knowledge of it).

So in conclusion I think that GNU ld should indeed change to match the
Solaris behavior.  But it is an incompatible change that at least Hurd code
will need to adapt to, so it should be well noted and perhaps wait a while.
(I can probably just change Hurd's libc to use DT_AUXILIARY for now and not
require any other immediate adaptations to work with the new ld behavior.)