Auto-rpath revisited...

Auto-rpath revisited...

[Lee Iverson]

OK.  I have an implementation of a system for automatically generating
shared lib rpaths in collect2.  It carefully walks through
'-L'-specified paths and searches for shared libraries which will
actually resolve the link search.  This should deal with the
previously raised objections to automatic techniques which just map -L
to rpath, and definitely solves the problem of needing LD_LIBRARY_PATH
in order to even run a minimal C++ program on systems where
libstdc++.so is not installed in a standard place.

So, I have a sample implementation that I could submit today (modulo
assignment paperwork).  At this point we come to policy issues, what
exactly should it do?

Proposals:

1) Do an exhaustive walkthrough of -L/-l pairs, and add to the end of
   any user-supplied rpath an entry for any directory which resolves a
   shared lib link (this is what the implementation now does).

2) Restrict the auto-rpath generation to only checking against
   system-supplied -L directories (i.e. -L$(exec_prefix)/lib).

3) something else...

Suggestions?  Discussion?

-------------------------------------------------------------------------------
Lee Iverson     		SRI International
leei@ai.sri.com			333 Ravenswood Ave., Menlo Park CA 94025
http://www.ai.sri.com/~leei/	(650) 859-3307

Re: Auto-rpath revisited...

[Ian Lance Taylor]

   Date: Fri, 20 Feb 1998 11:25:15 -0800
   From: Lee Iverson <leei@ai.sri.com>

   OK.  I have an implementation of a system for automatically generating
   shared lib rpaths in collect2.  It carefully walks through
   '-L'-specified paths and searches for shared libraries which will
   actually resolve the link search.  This should deal with the
   previously raised objections to automatic techniques which just map -L
   to rpath, and definitely solves the problem of needing LD_LIBRARY_PATH
   in order to even run a minimal C++ program on systems where
   libstdc++.so is not installed in a standard place.

Thanks for doing this.

   1) Do an exhaustive walkthrough of -L/-l pairs, and add to the end of
      any user-supplied rpath an entry for any directory which resolves a
      shared lib link (this is what the implementation now does).

Personally, I think that if the user explicitly specifies -R or
-rpath, you should trust them, and not change its value.  I'm not sure
if that is what you are saying here.  You should only set up the rpath
if the user did not specify one; that is the case where the naive user
can get caught.

(A tool like libtool will cleverly set up the rpath for the installed
directory, even though it is linking against a library in the build
directory, and you do not want to foil libtool by putting the build
directory into the rpath.)

   2) Restrict the auto-rpath generation to only checking against
      system-supplied -L directories (i.e. -L$(exec_prefix)/lib).

I see no need for this restriction.

Ian

Re: Auto-rpath revisited...

[Alexandre Oliva]

Ian Lance Taylor writes:

> Personally, I think that if the user explicitly specifies -R or
> -rpath, you should trust them, and not change its value.

I'd suggest that the library path for libraries that gcc/g++ chooses
to link in automatically should be inserted automatically too.  This
would cover a quite common case of linking against several shared
libraries whose run-time directories you happen to know about with
libstdc++: you don't have to know where it lives, and it is
transparently linked in, so it should be transparently found at
run-time.

Just my 2 cents.

-- 
Alexandre Oliva
mailto:oliva@dcc.unicamp.br mailto:aoliva@acm.org
http://www.dcc.unicamp.br/~oliva
Universidade Estadual de Campinas, SP, Brasil

Re: Auto-rpath revisited...

[Ian Lance Taylor]

   From: Alexandre Oliva <oliva@dcc.unicamp.br>
   Date: 21 Feb 1998 17:05:59 -0200

   I was just worried about C++ programs that rely on libtool libraries:
   libtool would specify -rpath, but it wouldn't know where libstdc++
   was, so that specification would be missing.  But then, if g++ did
   implicitly append libstdc++ to the library path, it should implicitly
   append its runtime directory too.

I don't see libtool as an issue.  We can fix libtool to do the right
thing, either to set rpath correctly, or to use the new option.

It's uses without libtool that are a potential issue.

I still hold my position.  Not that it's my decision to make or
anything.

Ian

Re: Auto-rpath revisited...

[Ian Lance Taylor]

   Date: Sat, 21 Feb 1998 12:13:34 -0800
   From: Lee Iverson <leei@ai.sri.com>

   But I don't want you to shut up, I want to get this right.  What do
   you want to be saved from?  As I said before, any rpath entries the
   user specifies are retained and have highest priority.  I don't really
   think in terms of *changing* the user-supplied rpath, just *augmenting*
   it if and only if necessary.

The case I want to be saved from is specifically having an unexpected
directory in my rpath.  That's because many dynamic linkers will
search every directory in rpath.  If one of those directories happens
to be mounted over NFS, my program has an unexpected dependency on an
NFS server.  There is nothing wrong with an expected dependency on an
NFS server, but an unexpected one can be quite bad when the server
goes down.

We've already assumed that I'm explicitly using -rpath, so I
presumably understand how the dynamic linker works.  We're no longer
talking about the naive user baffled by shared library searching,
which I agree is a real problem which I think will be solved by your
proposed patch.

As a non-naive user, I may have made special arrangements to provide a
copy of libstdc++.so in a special location on local disk.  When gcc
automatically adds a directory to rpath, my careful plans will be
silently foiled, causing an unpleasant future surprise.

Basically, you are proposing a silent change to an existing option,
with potential bad consequences, and that makes me uncomfortable.

Ian

Re: Auto-rpath revisited...

[Alexandre Oliva]

Ian Lance Taylor writes:

> It's not unusual, because shared libraries are often found in the
> build directory.

But not libstdc++, and that's exactly what we're arguing about:
libraries that are implicitly added by the compiler itself.  When
you're linking against a libstdc++ that's in a build directory, you'll
usually add it explicitly, so its path won't be added to rpath, just
as you want.

-- 
Alexandre Oliva
mailto:oliva@dcc.unicamp.br mailto:aoliva@acm.org
http://www.dcc.unicamp.br/~oliva
Universidade Estadual de Campinas, SP, Brasil

Re: Auto-rpath revisited...

[Alexandre Oliva]

Ian Lance Taylor writes:

>    I'd suggest that the library path for libraries that gcc/g++ chooses
>    to link in automatically should be inserted automatically too.  This
>    would cover a quite common case of linking against several shared
>    libraries whose run-time directories you happen to know about with
>    libstdc++: you don't have to know where it lives, and it is
>    transparently linked in, so it should be transparently found at
>    run-time.

> I continue to feel that if the user explicitly specifies a -R or
> -rpath option, you should trust them.

> I would find it rather frustrating to want to set the rpath, and have
> gcc override me because it thought it knew better.

This is ok when you know what you're doing.  But most users don't know
g++ links -lstdc++ in, they just expect g++ to do the Right Thing
(TM).  *You* know what you're doing and what g++ will do, so you'll
just run gcc ... -R... -lstdc++, or g++ -nostdlib ... -lstdc++, which
will work.

I was just worried about C++ programs that rely on libtool libraries:
libtool would specify -rpath, but it wouldn't know where libstdc++
was, so that specification would be missing.  But then, if g++ did
implicitly append libstdc++ to the library path, it should implicitly
append its runtime directory too.

-- 
Alexandre Oliva
mailto:oliva@dcc.unicamp.br mailto:aoliva@acm.org
http://www.dcc.unicamp.br/~oliva
Universidade Estadual de Campinas, SP, Brasil

Re: Auto-rpath revisited...

[Lee Iverson]

In message < ork9aohj6w.fsf@iguacu.dcc.unicamp.br > you write:
> Ian Lance Taylor writes:
> 
> >    I'd suggest that the library path for libraries that gcc/g++ chooses
> >    to link in automatically should be inserted automatically too.  This
> >    would cover a quite common case of linking against several shared
> >    libraries whose run-time directories you happen to know about with
> >    libstdc++: you don't have to know where it lives, and it is
> >    transparently linked in, so it should be transparently found at
> >    run-time.
> 
> > I continue to feel that if the user explicitly specifies a -R or
> > -rpath option, you should trust them.
> 
> > I would find it rather frustrating to want to set the rpath, and have
> > gcc override me because it thought it knew better.
> 
> This is ok when you know what you're doing.  But most users don't know
> g++ links -lstdc++ in, they just expect g++ to do the Right Thing
> (TM).  *You* know what you're doing and what g++ will do, so you'll
> just run gcc ... -R... -lstdc++, or g++ -nostdlib ... -lstdc++, which
> will work.
> 
> I was just worried about C++ programs that rely on libtool libraries:
> libtool would specify -rpath, but it wouldn't know where libstdc++
> was, so that specification would be missing.  But then, if g++ did
> implicitly append libstdc++ to the library path, it should implicitly
> append its runtime directory too.

Fruitful discussion.  Two points:

1) I forgot to mention that collect2 shouldn't and doesn't add
   relative paths to the auto-rpath.  The only difficulty with this is
   testing before install, but the LD_LIBRARY_PATH management by
   recent releases of dejagnu seems to be the right way to manage
   this, I believe.

2) I agree with Alexandre, and respond to Ian by suggesting that I add
   an option that turns off the auto-rpath behaviour for the
   sophisticated user who is actually managing this properly.  As it
   currently stands, the implementation searches user-supplied rpath
   entries *before* searching the -L paths and adds needed directories
   onto the *end* of the user-supplied rpath.
   
   If you really are an Ian Taylor, specifying a complete rpath will
   thus obviate the need to add anything to the rpath.  Beyond that, I
   would expect it to be exceedingly unusual for a user to
   deliberately want to leave a shared library unresolved by the
   rpath.

-------------------------------------------------------------------------------
Lee Iverson     		SRI International
leei@ai.sri.com			333 Ravenswood Ave., Menlo Park CA 94025
http://www.ai.sri.com/~leei/	(650) 859-3307

Re: Auto-rpath revisited...

[Ian Lance Taylor]

   Date: Sat, 21 Feb 1998 11:36:00 -0800
   From: Lee Iverson <leei@ai.sri.com>

   2) I agree with Alexandre, and respond to Ian by suggesting that I add
      an option that turns off the auto-rpath behaviour for the
      sophisticated user who is actually managing this properly.  As it
      currently stands, the implementation searches user-supplied rpath
      entries *before* searching the -L paths and adds needed directories
      onto the *end* of the user-supplied rpath.

      If you really are an Ian Taylor, specifying a complete rpath will
      thus obviate the need to add anything to the rpath.  Beyond that, I
      would expect it to be exceedingly unusual for a user to
      deliberately want to leave a shared library unresolved by the
      rpath.

It's not unusual, because shared libraries are often found in the
build directory.  Skipping relative paths doesn't save you, because in
a complex build paths are often canonicalized so that you don't have
to remember how many `../'s to add.

As previously noted, I feel that changing a user specified rpath
option is surprising.  Users who specify rpath already knows what they
are doing.  The purpose of this change is to help naive users, and
users who specify rpath are not naive.

In any case, I think I've made my points, such as they are, and I'll
try to shut up now.

Ian

Re: Auto-rpath revisited...

[Lee Iverson]

In message < 199802211959.OAA06562@subrogation.cygnus.com > you write:
>    Date: Sat, 21 Feb 1998 11:36:00 -0800
>    From: Lee Iverson <leei@ai.sri.com>
> 
>    2) I agree with Alexandre, and respond to Ian by suggesting that I add
>       an option that turns off the auto-rpath behaviour for the
>       sophisticated user who is actually managing this properly.  As it
>       currently stands, the implementation searches user-supplied rpath
>       entries *before* searching the -L paths and adds needed directories
>       onto the *end* of the user-supplied rpath.
> 
>       If you really are an Ian Taylor, specifying a complete rpath will
>       thus obviate the need to add anything to the rpath.  Beyond that, I
>       would expect it to be exceedingly unusual for a user to
>       deliberately want to leave a shared library unresolved by the
>       rpath.
> 
> It's not unusual, because shared libraries are often found in the
> build directory.  Skipping relative paths doesn't save you, because in
> a complex build paths are often canonicalized so that you don't have
> to remember how many `../'s to add.
> 
> As previously noted, I feel that changing a user specified rpath
> option is surprising.  Users who specify rpath already knows what they
> are doing.  The purpose of this change is to help naive users, and
> users who specify rpath are not naive.
> 
> In any case, I think I've made my points, such as they are, and I'll
> try to shut up now.

But I don't want you to shut up, I want to get this right.  What do
you want to be saved from?  As I said before, any rpath entries the
user specifies are retained and have highest priority.  I don't really
think in terms of *changing* the user-supplied rpath, just *augmenting*
it if and only if necessary.

-------------------------------------------------------------------------------
Lee Iverson     		SRI International
leei@ai.sri.com			333 Ravenswood Ave., Menlo Park CA 94025
http://www.ai.sri.com/~leei/	(650) 859-3307

Re: Auto-rpath revisited...

[Ian Lance Taylor]

   From: Alexandre Oliva <oliva@dcc.unicamp.br>
   Date: 21 Feb 1998 01:52:48 -0200

   Ian Lance Taylor writes:

   > Personally, I think that if the user explicitly specifies -R or
   > -rpath, you should trust them, and not change its value.

   I'd suggest that the library path for libraries that gcc/g++ chooses
   to link in automatically should be inserted automatically too.  This
   would cover a quite common case of linking against several shared
   libraries whose run-time directories you happen to know about with
   libstdc++: you don't have to know where it lives, and it is
   transparently linked in, so it should be transparently found at
   run-time.

I continue to feel that if the user explicitly specifies a -R or
-rpath option, you should trust them.

I would find it rather frustrating to want to set the rpath, and have
gcc override me because it thought it knew better.

To put it another way: perhaps we should support your option.  But we
need to permit the user to override it.  By the principle of least
surprise, specifying -rpath should set the rpath as the user
specifies.  So the way to override your option should just be -rpath,
and your option should have a different name.

Ian

Re: Auto-rpath revisited...

[H.J. Lu]

> 
> Ian Lance Taylor writes:
> 
> > It's not unusual, because shared libraries are often found in the
> > build directory.
> 
> But not libstdc++, and that's exactly what we're arguing about:
> libraries that are implicitly added by the compiler itself.  When
> you're linking against a libstdc++ that's in a build directory, you'll
> usually add it explicitly, so its path won't be added to rpath, just
> as you want.
> 

Personally, I don't like auto-rpath, especially when I build the
shared libraries and the binaries which use them. It happpens with
many packages, from binutils to X11. I think auto-rpath should only
be enabled if you are 100% sure the directory you find is the
directory used at the runtime.

-- 
H.J. Lu (hjl@gnu.org)

Re: Auto-rpath revisited...

[H.J. Lu]

> 
> This is ok when you know what you're doing.  But most users don't know
> g++ links -lstdc++ in, they just expect g++ to do the Right Thing
> (TM).  *You* know what you're doing and what g++ will do, so you'll
> just run gcc ... -R... -lstdc++, or g++ -nostdlib ... -lstdc++, which
> will work.
> 
> I was just worried about C++ programs that rely on libtool libraries:
> libtool would specify -rpath, but it wouldn't know where libstdc++
> was, so that specification would be missing.  But then, if g++ did
> implicitly append libstdc++ to the library path, it should implicitly
> append its runtime directory too.
> 

I'd like to see it as a configuration time option. I don't need it
on my machine. It doesn't do me any good.

BTW, it would be better for g++/c++ to pass -rpath to collect2 since we
can tell g++ where libstdc++ will be installed. If people want to
use

# gcc .... -lstc++

They'd better add -rpath themselves. I can write a patch if necesssary.


-- 
H.J. Lu (hjl@gnu.org)

Re: Auto-rpath revisited...

[Richard Henderson]

On Sat, Feb 21, 1998 at 12:13:34PM -0800, Lee Iverson wrote:
> But I don't want you to shut up, I want to get this right.  What do
> you want to be saved from?  As I said before, any rpath entries the
> user specifies are retained and have highest priority.

But this looses when the library isn't installed yet.

In that case the library doesn't exist in the rpath I gave you, so
you're going to add directories I don't want.  Filtering relative
paths doesn't help, as Ian mentioned, because many packages canonize
path names during build to keep from getting confused.


r~

Re: Auto-rpath revisited...

[Alexandre Oliva]

Richard Henderson writes:

> On Sat, Feb 21, 1998 at 12:13:34PM -0800, Lee Iverson wrote:
>> But I don't want you to shut up, I want to get this right.  What do
>> you want to be saved from?  As I said before, any rpath entries the
>> user specifies are retained and have highest priority.

> But this looses when the library isn't installed yet.

If the library isn't installed yet, you'll *have* to specify its
runtime path, so the right thing to do is to not guess library path if
the user specifies it himself.  However, for libraries that are
implicitly linked in by the compiler, it is reasonable that the
compiler provides the shared library runtime path itself.

-- 
Alexandre Oliva
mailto:oliva@dcc.unicamp.br mailto:aoliva@acm.org
http://www.dcc.unicamp.br/~oliva
Universidade Estadual de Campinas, SP, Brasil

Re: Auto-rpath revisited...

[Todd Vierling]

On 21 Feb 1998, Alexandre Oliva wrote:

: >> But I don't want you to shut up, I want to get this right.  What do
: >> you want to be saved from?  As I said before, any rpath entries the
: >> user specifies are retained and have highest priority.
: 
: > But this looses when the library isn't installed yet.
: 
: If the library isn't installed yet, you'll *have* to specify its
: runtime path, so the right thing to do is to not guess library path if
: the user specifies it himself.  However, for libraries that are
: implicitly linked in by the compiler, it is reasonable that the
: compiler provides the shared library runtime path itself.

What do you do in a cross-compile environment?

I have a, say, NetBSD/i386 system which cross compiles for
sparc-sun-solaris2.5.1.  What use would I have to have
/usr/local/sparc-sun-solaris2.5.1/gcc-lib in my rpath--_particularly_ if my
/usr/local is NFS?  Bah, I'll take my nfs server and throw it out the window
first. 

I had this problem with the SunOS incantation of gld and discussed it on the
list.  I got so fed up with auto-rpath that I completely zapped the whole
slew of code out of my loccal copy of gld and I am glad that it doesn't ever
create rpaths without my explicit permission.  This should be a
configure-time option, or at the very least a command line option.  (For a
while, adding '-rpath=' to the gcc specs link line was sufficient, but it
still left some to be desired.)

Remember, ELF and SunOS ld.so dynamic linkers know the system directories
(typically /usr/lib) implicitly and they don't _need_ an rpath unless shlibs
are installed in a "funny" place.  Specifying one for the sake of runtime
happiness is nice for the naive user, but the experienced user should know
how to install libstdc++ in a system directory and disable a spurious rpath. 

-- 
-- Todd Vierling (Personal tv@pobox.com; Bus. todd_vierling@xn.xerox.com)

Re: Auto-rpath revisited...

[Lee Iverson]

In message < Pine.NEB.3.96.980223094925.12722D-100000@like.duh.org > you write:
> On 21 Feb 1998, Alexandre Oliva wrote:
> 
> : >> But I don't want you to shut up, I want to get this right.  What do
> : >> you want to be saved from?  As I said before, any rpath entries the
> : >> user specifies are retained and have highest priority.
> : 
> : > But this looses when the library isn't installed yet.
> : 
> : If the library isn't installed yet, you'll *have* to specify its
> : runtime path, so the right thing to do is to not guess library path if
> : the user specifies it himself.  However, for libraries that are
> : implicitly linked in by the compiler, it is reasonable that the
> : compiler provides the shared library runtime path itself.
> 
> What do you do in a cross-compile environment?

As H.J. Lu already pointed out, this functionality is disabled in
cross-sompile environment.

-------------------------------------------------------------------------------
Lee Iverson     		SRI International
leei@ai.sri.com			333 Ravenswood Ave., Menlo Park CA 94025
http://www.ai.sri.com/~leei/	(650) 859-3307

Re: Auto-rpath revisited...

[H.J. Lu]

> > What do you do in a cross-compile environment?
> 
> As H.J. Lu already pointed out, this functionality is disabled in
> cross-sompile environment.
> 

I still don't like collect2 adds rpath for me. However, I don't
mind g++/c++ does it since it knows where libstdc++ is installed
during configuration. I think the best solution is

1. g++/c++ passes "--try-rpath installed_libdir" to collect2.
2. collect adds -rpath installed_libdir to ld if installed_libdir if
   it is used by ld for libstdc++.so.


-- 
H.J. Lu (hjl@gnu.org)

Re: Auto-rpath revisited...

[Richard Henderson]

On Mon, Feb 23, 1998 at 04:50:10PM -0800, H.J. Lu wrote:
> 1. g++/c++ passes "--try-rpath installed_libdir" to collect2.
> 2. collect adds -rpath installed_libdir to ld if installed_libdir if
>    it is used by ld for libstdc++.so.

This doesn't sound quite as bad, if you expand it to maybe

	--try-rpath libstdc++.so=/some/path/here
	--try-rpath libf2c.so=/some/other/path/here

ie very directed searching.  You should still be able to
turn it off from the g++ command line, and -nostdlibs should
remember to turn it off as well.


r~

Re: Auto-rpath revisited...

[Marc Espie]

In article < Pine.NEB.3.96.980223094925.12722D-100000@like.duh.org >,
Todd Vierling <tv@pobox.com> wrote:

>I had this problem with the SunOS incantation of gld and discussed it on the
>list.  I got so fed up with auto-rpath that I completely zapped the whole
>slew of code out of my loccal copy of gld and I am glad that it doesn't ever
>create rpaths without my explicit permission.  This should be a
>configure-time option, or at the very least a command line option.  (For a
>while, adding '-rpath=' to the gcc specs link line was sufficient, but it
>still left some to be desired.)

>Remember, ELF and SunOS ld.so dynamic linkers know the system directories
>(typically /usr/lib) implicitly and they don't _need_ an rpath unless shlibs
>are installed in a "funny" place.  Specifying one for the sake of runtime
>happiness is nice for the naive user, but the experienced user should know
>how to install libstdc++ in a system directory and disable a spurious rpath. 

Thanks to diplomacy problems, the `experienced user' doesn't always have
the rights to install libstdc++ in a system directory, in which case having
to hack g++ / rpaths manually is no fun.  Especially when said experienced
user is more interested in using a better C++ compiler than tracking this
sort of thing all over the place.  

If you prefer, having this as a configure option looks quite reasonable to
me, especially IF it's documented.  What should at least be possible is to
install egcs as joe-average-user, WITHOUT having to be root and adding 
libstdc++.so in a system directory, or adding a new directory to ldconfig,
so that it can be used by several people as a drop-in replacement for the
old g++, without having to go back AND coach people in the use of
-Xlinker -rpath.

Another point:   trying out egcs may also mean trying to compile as many
packages as possible with it. Some of them use autoconfig extensively, know
already a lot about shared libraries and rpath options, and will try to
impose their view on the matter.  IF libstdc++.so does NOT appear as a
`silent' drop-in, even when it is not in the system directory, this means
lots of hackings with the configure scripts/Makefile whatever if you don't
happen to live in an enlightened place where sysadmins trust you.

The only reasonable solution then would be to replace g++ with a smart enough
script, or to hack its specs (or collect2) so that use of libstdc++ will be
transparent. In that case, it's simpler to get it once and for all inside
egcs as a configure option, isn't it ?

Re: Auto-rpath revisited...

[H.J. Lu]

> The only reasonable solution then would be to replace g++ with a smart enough
> script, or to hack its specs (or collect2) so that use of libstdc++ will be
> transparent. In that case, it's simpler to get it once and for all inside
> egcs as a configure option, isn't it ?
> 

May I propose we change LIB_SPEC to this?

#define C_LIB_SPEC	The current LIB_SPEC
#if defined __STC__ && we are g++/c++.
#define STDCXX_RPATH_SPEC
  "{!shared: {!static: %{!rpath: -rpath " INSTALL_LIBSTCCXXDIR "}}}"
#define LIB_SPEC	C_LIB_SPEC STDCXX_RPATH_SPEC
#else
#define LIB_SPEC 	C_LIB_SPEC
#endif

Will it work?


-- 
H.J. Lu (hjl@gnu.org)

Re: Auto-rpath revisited...

[Jim Wilson]

Making a shared libstdc++ work better with gcc is an important problem to
solve, but adding automatic -rpath options by default is probably not a good
way to solve it.

There has already been a lot of discussion pointing out that we shouldn't add
any automatic rpath options if the user specified one.  This is because this
will break some features of shared libraries for expert users.  However, I
don't think anyone has explored just what this involves.  Clearly, we need to
check for command line options -Wl,-rpath,foo.  However, we also need to
check for environment variables.  Just checking the Solaris 2.5.1 ld man
page, I see the following environment variables that appear to be relevant:
LD_LIBRARY_PATH, LD_OPTIONS, LD_PRELOAD, LD_RUN_PATH.  If any of these
environment variables are defined, then we have to check them to see if it
is still safe to add auto-rpath options.  We also need to handle the Solaris 2
-R linker option, and any other options that may be relevant such as -i,
-Bstatic, and -r.  If we don't do all of this, then gcc will not be a drop in
replacement for the Sun C compiler.  And this is just one operating systems.
There will be a different set of linker options and linker environment
variables for every systems.  Once we start encoding this knowledge of linkers
into collect, it will become less portable, harder to maintain, and more
likely to break with a minor OS revision.

A better solution is to add the directory we need to the run-time linkers
default search path.  On some systems, like Linux and SunOS4, this means
running ldconfig.  Yes, this requires root access, but that isn't an
unreasonable requirement for someone who really wants to solve this problem
correctly.  Other major packages (like X11) have the same requirement if
shared libraries are being used.  If someone wants a shared libstdc++ and
can't get root, then we should tell them to set an appropriate environment
variable.

Nowadays, many people are just going to install gcc from a binary package
anyways,  so it is only the package maintainers who need to worry about making
sure that the package sets the dynamic linker's default search path correctly.
If this gets to be a problem, we could perhaps start making our own packages
available in addition to the sources.

I can see some use for having an option that enables auto-rpath.  That would
make it easier for gcc developers and testers who might have multiple or
temporary gcc install trees where ldconfig would be inconvenient or might
not work.  But I don't think this should be the default, because it breaks
the way that shared libraries were designed to work.

Jim

Re: Auto-rpath revisited...

[Alexandre Oliva]

Jim Wilson writes:

> Making a shared libstdc++ work better with gcc is an important problem to
> solve, but adding automatic -rpath options by default is probably not a good
> way to solve it.

Seems like there are people who disagree with this.

The problem, as I see it, is that expert users know exactly how to
`fix' library path problems, but novice users, who wouldn't know that,
are not in this list to defend their point.  I agree that modifying
the behavior of g++ at this point might break several scripts, test
programs, whatever else these expert users have produced to make
things work.  However, it would save me the time I and some other
experts who care for novices spend telling them they have to link with
such and such command line option in order for their `Hello World' C++
program to work.

My point is that expert users may find minor problems as soon as we
install this change, but, since they're expert users, they'll be able
to fix them quickly.  OTOH, if we don't install the change, new users
who might benefit from our project, and help us in the future, may
simply abandon g++ because a program that works with Sun C++ won't
work with g++.

Of course, novice users aren't installing gcc/g++ themselves; system
administrators, that are expected to be expert users, will install
gcc/g++, and they have the following alternatives to fix the shared
library problem:

1) not building shared libraries at all.  Then user N.O.Vice user will
post a message to gnu.g++.bug asking why his hello world program is so
large.

2) building shared libraries and telling users to link with
-Wl,-rpath,/opt/egcs/lib.  Then user N.O.Vice will post a message to
the Lilypond (GNU music) mailing list of Lilypond reporting that
running lilypond produces a strange error message like about not
finding the C++ library, and asking why it didn't report that at
compile time!?!

3) adding /opt/egcs/lib to the standard library path.  Then, as soon
as egcs is upgraded, user N.O.Vice will ask the system administrator
to recover a lot of files from the last month's backup, because his
C++ applications that worked then would no longer work

4) adding -Wl,-rpath,/opt/egcs/lib to the egcs specs C++ link command
line, so that any programs linked with egcs++ will find the library.
N.O.Vice is happy, but X.Pert complains his own modified copy of
libstdc++ no longer works.

5) modifying the libstdc++ Makefile so that it creates the shared
library with an soname that includes the full pathname where the
library is going to get installed.  Both N.O.Vice and X.Pert are
happy, until egcs is updated, and the old libraries are moved
to somewhere else

6) modifying the standard .login/.profile so that it includes
/opt/egcs/lib in LD_LIBRARY_PATH.  This might be good, but then,
N.O.Vice will complain to egcs-bugs that his own installation of egcs
will use the outdated libstdc++ installed by the system manager
instead of the one he installed himself

As you can see, no solution is perfect.  But, IMO, we should try to
make egcs as `plug&play' as possible, and this really means either
embedding the full pathname of the shared library in the library
itself (#5), or adding -rpath switches at link-time (#4).  I
particularly prefer #5, as long as, at configure time, it is possible
to specify a shared library runtime path that is different from the
one libraries will be installed on (so that I can have a shared
library repository, but still install applications in separate
directories instead of a huge /usr/local).  However, I know this
doesn't work on all systems, so I'd be happy with #4.

> Clearly, we need to
> check for command line options -Wl,-rpath,foo.  However, we also need to
> check for environment variables.  Just checking the Solaris 2.5.1 ld man
> page, I see the following environment variables that appear to be relevant:
> LD_LIBRARY_PATH, LD_OPTIONS, LD_PRELOAD, LD_RUN_PATH.  If any of these
> environment variables are defined, then we have to check them to see if it
> is still safe to add auto-rpath options.

AFAIK, only LD_OPTIONS and LD_RUN_PATH affect link-time behavior.  We
should really take care of them, but we don't need a finished solution
to start testing it.  We should fix this problem just like most other
problems: provide a simple yet generic fix, and try to solve specific
problems as they appear.  I don't think anyone would be able to do a
complete fix by himself, since no one knows all possible environment
variables that may be used by ld.

> A better solution is to add the directory we need to the run-time linkers
> default search path.  On some systems, like Linux and SunOS4, this means
> running ldconfig.

On other systems, there's no such thing.  This is *less* portable than
the solution above, and requires novice user intervention.

> Other major packages (like X11) have the same requirement if
> shared libraries are being used.

Whenever I installed X11, it did embed full pathnames in the shared
libraries, so we didn't have to specify runtime paths or such.  I have
always thought this is the right approach, but some people don't agree
with that, and I understand their reasons for that.

> If someone wants a shared libstdc++ and
> can't get root, then we should tell them to set an appropriate environment
> variable.

Why, if we can make it work without that?  What about multiple
installations of the same package, in different directories?  ldconfig
won't allow that!

-- 
Alexandre Oliva
mailto:oliva@dcc.unicamp.br mailto:aoliva@acm.org
http://www.dcc.unicamp.br/~oliva
Universidade Estadual de Campinas, SP, Brasil

Re: Auto-rpath revisited...

[H.J. Lu]

> 
> Jim Wilson writes:
> 
> > Making a shared libstdc++ work better with gcc is an important problem to
> > solve, but adding automatic -rpath options by default is probably not a good
> > way to solve it.
> 
> Seems like there are people who disagree with this.
> 

What's wrong with my proposal to modify LIB_SPECS? It is
easy and should be harmless. We only need to modify LIB_SPECS
on a platform by platform basis.


H.J.

Re: Auto-rpath revisited...

[Jim Wilson]

There is no disagreement over the importance of the problem or the need
to fix it.  There is some disagreement I think about what we are discussing.

The original message talked about modifying collect to automatically add
multiple rpath options depending on the -L and -l options passed into collect.
I believe this is unwise, and this is what I am arguing against.  This runs
the risk of breaking build environments for existing projects that already
know how shared libraries are supposed to work.  If you helpfully add -rpath
options to an X11 build, then you are likely to break that X11 build in some
way.

Your message talks about adding a single -rpath option for libstdc++, perhaps
generated by the g++ driver.  I will offer no objection to this if the ldconfig
solution can't work, and there is someway to disable it.  Adding a -rpath
option for libraries that come with egcs isn't going to break other programs,
and will make egcs work better.  This gets us to H.J. Lu's suggestions which
look reasonable to me.  He made two suggestions, modify LIB_SPEC to pass
-rpath for libstdc++, modify g++ driver to pass down --try-rpath option for
libstdc++.  I like the g++ driver change a little better than the LIB_SPEC
change.  There might be portability problems with the g++ driver change though,
since -rpath isn't a universal linker option.  Richard Henderson's extension
of the --try-rpath idea also looked useful.

Jim

Re: Auto-rpath revisited...

[H.J. Lu]

> option for libraries that come with egcs isn't going to break other programs,
> and will make egcs work better.  This gets us to H.J. Lu's suggestions which
> look reasonable to me.  He made two suggestions, modify LIB_SPEC to pass
> -rpath for libstdc++, modify g++ driver to pass down --try-rpath option for
> libstdc++.  I like the g++ driver change a little better than the LIB_SPEC
> change.  There might be portability problems with the g++ driver change though,
> since -rpath isn't a universal linker option.  Richard Henderson's extension
> of the --try-rpath idea also looked useful.
> 
That is why I prefer the LIB_SPEC solution. We can fine tune it for each
configuration. Besides, it needs very few changes to other places.


-- 
H.J. Lu (hjl@gnu.org)

Re: Auto-rpath revisited...

[Swen Thuemmler]

On Sat, 28 Feb 1998, H.J. Lu wrote:

> What's wrong with my proposal to modify LIB_SPECS? It is
> easy and should be harmless. We only need to modify LIB_SPECS
> on a platform by platform basis.

This unfortunately conflicts with LD_RUN_PATH, at least on solaris. If you
add -rpath (or -R) to the ld options, they will happily override the
setting in LD_RUN_PATH, which may lead to strange effects. When you for
example generate a Makefile via xmkmf, the linking stage becomes something
like
   LD_RUN_PATH=/whereever/X11R6/lib g++ -o ...
Just changing LIB_SPECS will let the program find the libg++, but all of a
sudden, the X11-libraries cannot be found.

Greetings, Swen

Re: Auto-rpath revisited...

[Lee Iverson]

In message <199802212208.RAA06708@subrogation.cygnus.com> you write:
>    Date: Sat, 21 Feb 1998 13:42:12 -0800
>    From: Lee Iverson <leei@ai.sri.com>
> 
> I think we're somehow talking at cross purposes.  I don't quite
> understand what you are saying.  I'll try to respond as best, and as
> completely, as I can.  I'm sure you already understand some of what I
> will say.
>
> [excellent explanation of tradeoffs deleted].

I think I understand now (mostly), although I do believe that the case
you're describing is really the unusual one.  If it really is the case
that there exist runtime loaders that will search directories in the
rpath *after* all shared libraries have been resolved, then I would
have to call that a definite bug, which should be reported as such to
the OS vendor.

That said, it would clearly be useful to have a way of avoiding this
new behaviour.  Ian suggests that a user-supplied rpath should be
sufficient to turn it off.  For the naive user, specifying a single
rpath directory shouldn't turn off the automatic resolution of
libstdc++.so, for example.

I'd say that if you're assuming a sophisticated enough user to be able
to generate proper rpaths that will work properly at run-time but
don't look right at link time then that user should be able to add
another option that disables the auto-rpath augmentation.

-------------------------------------------------------------------------------
Lee Iverson     		SRI International
leei@ai.sri.com			333 Ravenswood Ave., Menlo Park CA 94025
http://www.ai.sri.com/~leei/	(650) 859-3307