Ensuring symbol resolution order at runtime

Ensuring symbol resolution order at runtime

[Kevin P. Fleming]

(please keep me in the CC list, as I'm not subscribed. thanks)

I'll ask this question in the context of 'modern' Linux (Ubuntu 11.10, 
for example) using ELF binaries and current binutils, but I'm interested 
in knowing whether the behavior I desire, if it is in fact how the 
static/dynamic linkers operate, is specified by any standards, or is 
just 'common practice'.

Scenario 1: An executable is being linked, and the linker is provided 
the names of two static libraries. Both of those libraries export a 
symbol named "foo", and one of the objects being linked into the 
executable refers to that symbol.

In this situation, it is understood that the linker will choose a symbol 
"foo" to satisfy the reference from the *first* library specified in the 
linker's arguments that provides a symbol "foo".

Scenario 2: An executable is being linked, and the static linker is 
provided the names of two shared libraries. Both of those libraries 
export a symbol named "foo", and one of the objects being linked into 
the executable refers to that symbol.

In this situation, it is understood that the static linker will locate a 
symbol "foo" to satisfy the reference from the *first* library specified 
in the linker's arguments that provides a symbol "foo". This is fine. 
However, at runtime, when the dynamic linker has the responsibility for 
loading these shared libraries into the process and resolving symbols, 
is there any guarantee that the the *same* "foo" will be chosen by the 
dynamic linker?

Scenario 3: An executable is being linked, and the static linker is 
provided the name of a shared library. That shared library contains a 
reference to another shared library. Both of those libraries export a 
symbol named "foo", and one of the objects being linked into the 
executable refers to that symbol.

In this situation, the static linker only has the symbol "foo" from the 
first library available to satisfy the reference, because it was not 
made explicitly aware of the second library that also provides "foo". At 
runtime, it seems reasonable to assume that the dynamic linker will load 
the first shared library into the process before the second shared 
library, and *should* resolve symbol references from the executable into 
that library before loading the second shared library into the process.

I'm trying to work on a problem related to multiple shared libraries 
linking to libssl.so (from OpenSSL), and need to come up with a way to 
ensure that only *one* library calls the initialization functions from 
libssl, without requiring any cooperating behavior from the other 
libraries that would normally do it as well. My approach so far has been 
to 'mask' the real functions from libssl.so in another shared library 
that is loaded *first* into the process, thereby ensuring that no matter 
what libraries are loaded into the process later, they will not be able 
to gain access to the underlying functions from libssl.so (other than 
via exploiting dlsym() to do so explicitly).

So, is this approach reasonable, and if so, is it backed up by standards 
or other documents that prescribe the linker's behavior for these 
scenarios? I'm really not keen on relying on any assumptions about the 
dynamic linker's behavior (other than that it will follow the relevant 
standards if it claims to do so).

Thanks in advance for any light you can shed on this :-)

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
Jabber: kfleming@digium.com | SIP: kpfleming@digium.com | Skype: kpfleming
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at www.digium.com & www.asterisk.org

Re: Ensuring symbol resolution order at runtime

[Cary Coutant]

> Scenario 1: An executable is being linked, and the linker is provided the
> names of two static libraries. Both of those libraries export a symbol named
> "foo", and one of the objects being linked into the executable refers to
> that symbol.
>
> In this situation, it is understood that the linker will choose a symbol
> "foo" to satisfy the reference from the *first* library specified in the
> linker's arguments that provides a symbol "foo".

Yes, as long as the reference to "foo" comes from an object that
precedes both libraries in the link order. It's possible to do a link
like this:

  ld does-not-call-foo.o lib1.a calls-foo.o lib2.a

and the "foo" from lib2.a will be chosen.

> Scenario 2: An executable is being linked, and the static linker is provided
> the names of two shared libraries. Both of those libraries export a symbol
> named "foo", and one of the objects being linked into the executable refers
> to that symbol.
>
> In this situation, it is understood that the static linker will locate a
> symbol "foo" to satisfy the reference from the *first* library specified in
> the linker's arguments that provides a symbol "foo". This is fine. However,
> at runtime, when the dynamic linker has the responsibility for loading these
> shared libraries into the process and resolving symbols, is there any
> guarantee that the the *same* "foo" will be chosen by the dynamic linker?

As long as the runtime loader sees the same libraries that the linker
saw, yes, there is a guarantee. However, if lib1.so is replaced with a
library that does not define "foo", then the dynamic loader will bind
"foo" to the definition in lib2.so without any complaint.

This is standard Unix behavior, and is unlike what you would get on
Windows, for example, where the static linker records the binding of
each definition.

> Scenario 3: An executable is being linked, and the static linker is provided
> the name of a shared library. That shared library contains a reference to
> another shared library. Both of those libraries export a symbol named "foo",
> and one of the objects being linked into the executable refers to that
> symbol.
>
> In this situation, the static linker only has the symbol "foo" from the
> first library available to satisfy the reference, because it was not made
> explicitly aware of the second library that also provides "foo". At runtime,
> it seems reasonable to assume that the dynamic linker will load the first
> shared library into the process before the second shared library, and
> *should* resolve symbol references from the executable into that library
> before loading the second shared library into the process.

Yes, subject to the conditions I mentioned above.

> I'm trying to work on a problem related to multiple shared libraries linking
> to libssl.so (from OpenSSL), and need to come up with a way to ensure that
> only *one* library calls the initialization functions from libssl, without
> requiring any cooperating behavior from the other libraries that would
> normally do it as well. My approach so far has been to 'mask' the real
> functions from libssl.so in another shared library that is loaded *first*
> into the process, thereby ensuring that no matter what libraries are loaded
> into the process later, they will not be able to gain access to the
> underlying functions from libssl.so (other than via exploiting dlsym() to do
> so explicitly).
>
> So, is this approach reasonable, and if so, is it backed up by standards or
> other documents that prescribe the linker's behavior for these scenarios?
> I'm really not keen on relying on any assumptions about the dynamic linker's
> behavior (other than that it will follow the relevant standards if it claims
> to do so).
>
> Thanks in advance for any light you can shed on this :-)

I'd think a better approach would be to use a static variable in the
initialization function to protect it from being called twice. Even
better would be to have libssl's initialization functions actually
marked as initialization functions so that they run automatically when
the library is loaded.

-cary

Re: Ensuring symbol resolution order at runtime

[Kevin P. Fleming]

On 01/20/2012 03:52 PM, Cary Coutant wrote:

> As long as the runtime loader sees the same libraries that the linker
> saw, yes, there is a guarantee. However, if lib1.so is replaced with a
> library that does not define "foo", then the dynamic loader will bind
> "foo" to the definition in lib2.so without any complaint.

So, to be as specific as I can, not only will the runtime loader see 
those same libraries, but it will look at them in the *same order* that 
the static linker did (presumably because the static linker will list 
them in the linked executable in the order it processed them, and the 
dynamic linker follows that order).

> I'd think a better approach would be to use a static variable in the
> initialization function to protect it from being called twice. Even
> better would be to have libssl's initialization functions actually
> marked as initialization functions so that they run automatically when
> the library is loaded.

No argument here; unfortunately the maintainers of OpenSSL have not seen 
fit to do that, even though this problem has been known about for quite 
some time. We are not the first project to have to work around the 
problem in our own code, and even other projects that use OpenSSL and 
produce their own libraries for consumption (PostgreSQL, for example) 
have made accommodations in their libraries to allow the eventual 
application using the combined libraries to be able to avoid this 
complication. Unfortunately not every library that uses OpenSSL has made 
these accommodations, so we need a 'belt and suspenders' approach that 
we can rely on.

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
Jabber: kfleming@digium.com | SIP: kpfleming@digium.com | Skype: kpfleming
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at www.digium.com & www.asterisk.org

Re: Ensuring symbol resolution order at runtime

[Cary Coutant]

>> As long as the runtime loader sees the same libraries that the linker
>> saw, yes, there is a guarantee. However, if lib1.so is replaced with a
>> library that does not define "foo", then the dynamic loader will bind
>> "foo" to the definition in lib2.so without any complaint.
>
> So, to be as specific as I can, not only will the runtime loader see those
> same libraries, but it will look at them in the *same order* that the static
> linker did (presumably because the static linker will list them in the
> linked executable in the order it processed them, and the dynamic linker
> follows that order).

Yes. The ABI requires that:

"When the dynamic linker creates the memory segments for an object
file, the dependencies (recorded in DT_NEEDED entries of the dynamic
structure) tell what shared objects are needed to supply the program's
services. By repeatedly connecting referenced shared objects and their
dependencies, the dynamic linker builds a complete process image. When
resolving symbolic references, the dynamic linker examines the symbol
tables with a breadth-first search. That is, it first looks at the
symbol table of the executable program itself, then at the symbol
tables of the DT_NEEDED entries (in order), and then at the second
level DT_NEEDED entries, and so on. Shared object files must be
readable by the process; other permissions are not required." [from
http://www.sco.com/developers/gabi/latest/ch5.dynamic.html#shobj_dependencies
]

-cary

Re: Ensuring symbol resolution order at runtime

[Kevin P. Fleming]

On 01/20/2012 04:57 PM, Cary Coutant wrote:
>>> As long as the runtime loader sees the same libraries that the linker
>>> saw, yes, there is a guarantee. However, if lib1.so is replaced with a
>>> library that does not define "foo", then the dynamic loader will bind
>>> "foo" to the definition in lib2.so without any complaint.
>>
>> So, to be as specific as I can, not only will the runtime loader see those
>> same libraries, but it will look at them in the *same order* that the static
>> linker did (presumably because the static linker will list them in the
>> linked executable in the order it processed them, and the dynamic linker
>> follows that order).
>
> Yes. The ABI requires that:
>
> "When the dynamic linker creates the memory segments for an object
> file, the dependencies (recorded in DT_NEEDED entries of the dynamic
> structure) tell what shared objects are needed to supply the program's
> services. By repeatedly connecting referenced shared objects and their
> dependencies, the dynamic linker builds a complete process image. When
> resolving symbolic references, the dynamic linker examines the symbol
> tables with a breadth-first search. That is, it first looks at the
> symbol table of the executable program itself, then at the symbol
> tables of the DT_NEEDED entries (in order), and then at the second
> level DT_NEEDED entries, and so on. Shared object files must be
> readable by the process; other permissions are not required." [from
> http://www.sco.com/developers/gabi/latest/ch5.dynamic.html#shobj_dependencies
> ]

Thank you very much for the help; the ABI documentation reference is 
especially helpful.

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
Jabber: kfleming@digium.com | SIP: kpfleming@digium.com | Skype: kpfleming
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at www.digium.com & www.asterisk.org