Can I stop linking the same library twice?

Can I stop linking the same library twice?

[Dr. David Kirkby]

I have seen issues in the Sage mathematics software

http://www.sagemath.org/

where problems arise if a library included in Sage is installed on the system 
too. Sage included the source for all the libraries it needs (e.g. readline), in 
case the system does not have that library. That sometimes causes problems, if 
the option "-L $SAGE_HOME/local/lib" is added and the library is also on the 
system.

Is there a way to specify that only the first copy of the library is linked, and 
not any other copies gcc might find? Readline is not the only library where 
issues have arisen (OpenSSL libraries on OpenSolaris is another example).

I'd rather some way of doing this which is not GCC specific, and certainly not 
specific to the GNU linker, as on Solaris the Sun linker works better, and in 
any case we hope to support non-GNU compilers too. But in the short term, 
anything that works with gcc, even if not with other compilers, would be useful.

Dave

Re: Can I stop linking the same library twice?

[Philipp Thomas]

On Sat, 12 Dec 2009 11:08:11 +0000, you wrote:

>where problems arise if a library included in Sage is installed on the system 
>too. 

Of cause they do! It's simply wrong to use and install libraries that
the system already supplies. A sane software package detects the
libraries installed in the system and only compiles/installs those
that are missing. If distributed in compiled form, the only sane way
is to statically link in the libraries to avoid clashes with system
supplied ones.

>Is there a way to specify that only the first copy of the library is linked, and 
>not any other copies gcc might find? Readline is not the only library where 
>issues have arisen (OpenSSL libraries on OpenSolaris is another example).

AFAIK there is no way.

Philipp

Re: Can I stop linking the same library twice?

[Dr. David Kirkby]

Philipp Thomas wrote:
> On Sat, 12 Dec 2009 11:08:11 +0000, you wrote:
> 
>> where problems arise if a library included in Sage is installed on the system 
>> too. 
> 
> Of cause they do! It's simply wrong to use and install libraries that
> the system already supplies. A sane software package detects the
> libraries installed in the system and only compiles/installs those
> that are missing. If distributed in compiled form, the only sane way
> is to statically link in the libraries to avoid clashes with system
> supplied ones.


What could you do if the system supplies a library, but it's not the latest 
release, and you need the latest?

>> Is there a way to specify that only the first copy of the library is linked, and 
>> not any other copies gcc might find? Readline is not the only library where 
>> issues have arisen (OpenSSL libraries on OpenSolaris is another example).
> 
> AFAIK there is no way.
> 
> Philipp
> 

Re: Can I stop linking the same library twice?

[Philipp Thomas]

On Sun, 13 Dec 2009 23:36:43 +0000, you wrote:

>What could you do if the system supplies a library, but it's not the latest 
>release, and you need the latest?

Either explicitely link in the static version of said library or
install the dynamic library somewhere else and link it in by passing
the path to ld via -L and then adding said library to the users
LD_LIBRARY_PATH.

Philipp

Re: Can I stop linking the same library twice?

[Dr. David Kirkby]

Philipp Thomas wrote:
> On Sun, 13 Dec 2009 23:36:43 +0000, you wrote:
> 
>> What could you do if the system supplies a library, but it's not the latest 
>> release, and you need the latest?
> 
> Either explicitely link in the static version of said library or
> install the dynamic library somewhere else and link it in by passing
> the path to ld via -L and then adding said library to the users
> LD_LIBRARY_PATH.
> 
> Philipp
> 

We have taken the second approach, but it does not appear to work. The problem 
is at compile time, so the setting of LD_LIBRARY_PATH is ignored. This is one 
such error message, which I've been unable to resolve on Open Solaris, but it 
does not cause an issue on Solaris 10, or most linux distributions. But it does 
also show up on Some linux distributions. If you have any ideas, I'd appreciate it.

http://trac.sagemath.org/sage_trac/ticket/7387

This output is using the Sun linker, but others have similar issues with the GNU 
  linker.

Dave

Re: Can I stop linking the same library twice?

[Ralf Wildenhues]

Hello David,

* Dr. David Kirkby wrote on Sat, Dec 12, 2009 at 12:08:11PM CET:
> I have seen issues in the Sage mathematics software

> where problems arise if a library included in Sage is installed on
> the system too. Sage included the source for all the libraries it
> needs (e.g. readline), in case the system does not have that
> library. That sometimes causes problems, if the option "-L
> $SAGE_HOME/local/lib" is added and the library is also on the
> system.

Is libtool used to create this library?  Then this shouldn't happen
(famous last words), and if it does, it's typically the sign of a bug in
the makefiles or in libtool somewhere.

In case libtool is used, please send a message to the bug-libtool at
gnu.org mailing list, including a reference to this thread, and the
output of the `.../libtool --mode=link ...' command that fails, but with
--debug added as first argument to libtool, as well as the output of
  .../libtool --config

Thanks,
Ralf

Re: Can I stop linking the same library twice?

[Dr. David Kirkby]

Ralf Wildenhues wrote:
> Hello David,
> 
> * Dr. David Kirkby wrote on Sat, Dec 12, 2009 at 12:08:11PM CET:
>> I have seen issues in the Sage mathematics software
> 
>> where problems arise if a library included in Sage is installed on
>> the system too. Sage included the source for all the libraries it
>> needs (e.g. readline), in case the system does not have that
>> library. That sometimes causes problems, if the option "-L
>> $SAGE_HOME/local/lib" is added and the library is also on the
>> system.
> 
> Is libtool used to create this library?  Then this shouldn't happen
> (famous last words), and if it does, it's typically the sign of a bug in
> the makefiles or in libtool somewhere.
> 
> In case libtool is used, please send a message to the bug-libtool at
> gnu.org mailing list, including a reference to this thread, and the
> output of the `.../libtool --mode=link ...' command that fails, but with
> --debug added as first argument to libtool, as well as the output of
>   .../libtool --config
> 
> Thanks,
> Ralf
> 

Thank you Ralf,

No, libtool is not used. Nor is autoconf or automake. Sage consists of about 100 
open source packages. For each one, there is a small script which builds that 
package, which at the bare essentials might be:

#!/bin/sh
cd src
configure --prefix=$SAGE_LOCAL
make
make install

In other cases it might the script which does the building might be a python 
script. Each one is customised to build that particular bit of software.

Sometimes patches are applied. In each case the 'src' directory is the 
unmodified open-source software. Any patches in Sage are then added. So there 
might be something like

#!/bin/sh
cp patches/Makefile-new src/Makefile
configure --prefix=$SAGE_LOCAL
make
make install

That's resulting in a mixup of libraries, as nothing actually checks if the 
system has libraries of the type.

I believe a fundamentally different way might be needed, but I don't have the 
knowledge to implement it, nor do I have sufficient knowledge to argue why this 
is wrong. I just believe it is wrong.

Re: Can I stop linking the same library twice?

[Ralf Wildenhues]

* Dr. David Kirkby wrote on Thu, Dec 31, 2009 at 02:44:50AM CET:
> Ralf Wildenhues wrote:
> >* Dr. David Kirkby wrote on Sat, Dec 12, 2009 at 12:08:11PM CET:
> >>I have seen issues in the Sage mathematics software
> >
> >>where problems arise if a library included in Sage is installed on
> >>the system too. Sage included the source for all the libraries it
> >>needs (e.g. readline), in case the system does not have that
> >>library. That sometimes causes problems, if the option "-L
> >>$SAGE_HOME/local/lib" is added and the library is also on the
> >>system.
> >
> >Is libtool used to create this library?  Then this shouldn't happen
> >(famous last words), and if it does, it's typically the sign of a bug in
> >the makefiles or in libtool somewhere.

> No, libtool is not used. Nor is autoconf or automake. Sage consists
> of about 100 open source packages. For each one, there is a small
> script which builds that package, which at the bare essentials might
> be:

That doesn't answer the question though.  Your super-project Sage might
not use libtool, nor might some of the packages it is comprised of.  If
the sub package where the failing link happens uses libtool though,
above advice might still apply.

> In other cases it might the script which does the building might be
> a python script. Each one is customised to build that particular bit
> of software.

I don't think you can debug them without going into specifics and
looking at each one that is failing.

Cheers,
Ralf