[RFC] python-config.py --ldflags should return relocated path to libpython

[RFC] python-config.py --ldflags should return relocated path to libpython

[Joel Brobecker]

This is the fix that I alluded to in my earlier email:
http://www.sourceware.org/ml/gdb-patches/2010-07/msg00167.html.

There are currently 2 issues, I think, with python-config --ldflags:

  1. When python is configured with --enable-shared:
     -L<prefix>/lib is missing.  This is a problem if python was
     installed in a non-standard location.

  2. When Python is configured without --enable-shared:
     The path to the libpython archive is provided in the command output,
     but the path that gets returned is the path used at configure time.
     If the Python install was copied over to a different machine and
     then installed at a different location, then the path in -L<path>
     is incorrect (by comparison, python-config --cflags returns
     the location with the correct prefix).

I think that one might be able to work around the situation by adding
the missing -L option to LDFLAGS either during make or configure...
I will also admit that the situation is a little unorthodox, in the fact
that Python is installed at a different prefix than the one that was
used when configuring the build...

But the thing is it's been really useful on platforms where it's a pain
to build it (think Windows, for instance). 

This patch should fix both situations.  I will also send the patch
upstream to the Python developers, to see what they say.

gdb/ChangeLog:

        * python/python-config.py: When --ldflags is specified, make sure
        that the paths printed are relative to the run-time prefix.
        Always provide the path to libpython, even when the python
        install provides a shared version of libpython.

Tested on x86_64-linux, with 2.5 and 2.6. Briefly tested with 2.7,
but only by running python-config.py and visually verifying the output.

---
 gdb/python/python-config.py |   15 ++++++++++++++-
 1 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/gdb/python/python-config.py b/gdb/python/python-config.py
index 0230eb4..aa4aea3 100644
--- a/gdb/python/python-config.py
+++ b/gdb/python/python-config.py
@@ -50,8 +50,21 @@ for opt in opt_flags:
         # add the prefix/lib/pythonX.Y/config dir, but only if there is no
         # shared library in prefix/lib/.
         if opt == '--ldflags':
+            # Provide the location where the Python library is installed.
+            # We always provide it, because Python may have been installed
+            # at a non-standard location.
             if not getvar('Py_ENABLE_SHARED'):
-                libs.insert(0, '-L' + getvar('LIBPL'))
+                # There is no shared library in prefix/lib.  The static
+                # library is in prefix/lib/pythonX.Y/config.
+                #
+                # Note that we cannot use getvar('LIBPL') like we used to,
+                # because it provides the location at build time, which might
+                # be different from the actual location at runtime.
+                libdir = sysconfig.get_python_lib(standard_lib=True) + '/config'
+            else:
+                # The Python shared library is installed in prefix/lib.
+                libdir = sysconfig.PREFIX + '/lib'
+            libs.insert(0, '-L' + libdir)
             libs.extend(getvar('LINKFORSHARED').split())
         print ' '.join(libs)
 
-- 
1.7.1

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Joel Brobecker]

> This patch should fix both situations.  I will also send the patch
> upstream to the Python developers, to see what they say.

I found that one of the issues that this patch fixes was already opened:

    http://bugs.python.org/issue7352

I described my own issue, and posted a proposed patch.  Alea jacta est.

-- 
Joel

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Joel Brobecker]

> There are currently 2 issues, I think, with python-config --ldflags:
> 
>   1. When python is configured with --enable-shared:
>      -L<prefix>/lib is missing.  This is a problem if python was
>      installed in a non-standard location.
> 
>   2. When Python is configured without --enable-shared:
>      The path to the libpython archive is provided in the command output,
>      but the path that gets returned is the path used at configure time.
>      If the Python install was copied over to a different machine and
>      then installed at a different location, then the path in -L<path>
>      is incorrect (by comparison, python-config --cflags returns
>      the location with the correct prefix).
[...]
> This patch should fix both situations.  I will also send the patch
> upstream to the Python developers, to see what they say.

No comment either on the Python side or on the gdb-patches side.

> gdb/ChangeLog:
> 
>         * python/python-config.py: When --ldflags is specified, make sure
>         that the paths printed are relative to the run-time prefix.
>         Always provide the path to libpython, even when the python
>         install provides a shared version of libpython.

I think it's too late for 7.2, but I would like to apply this patch
on the HEAD. Any objections? If we ever create a 7.2.1, and this patch
was shown to not cause any problem, then perhaps I'll consider it for
backporting on the 7.2 branch - if approved by the GMs.

> Tested on x86_64-linux, with 2.5 and 2.6. Briefly tested with 2.7,
> but only by running python-config.py and visually verifying the output.
> 
> ---
>  gdb/python/python-config.py |   15 ++++++++++++++-
>  1 files changed, 14 insertions(+), 1 deletions(-)
> 
> diff --git a/gdb/python/python-config.py b/gdb/python/python-config.py
> index 0230eb4..aa4aea3 100644
> --- a/gdb/python/python-config.py
> +++ b/gdb/python/python-config.py
> @@ -50,8 +50,21 @@ for opt in opt_flags:
>          # add the prefix/lib/pythonX.Y/config dir, but only if there is no
>          # shared library in prefix/lib/.
>          if opt == '--ldflags':
> +            # Provide the location where the Python library is installed.
> +            # We always provide it, because Python may have been installed
> +            # at a non-standard location.
>              if not getvar('Py_ENABLE_SHARED'):
> -                libs.insert(0, '-L' + getvar('LIBPL'))
> +                # There is no shared library in prefix/lib.  The static
> +                # library is in prefix/lib/pythonX.Y/config.
> +                #
> +                # Note that we cannot use getvar('LIBPL') like we used to,
> +                # because it provides the location at build time, which might
> +                # be different from the actual location at runtime.
> +                libdir = sysconfig.get_python_lib(standard_lib=True) + '/config'
> +            else:
> +                # The Python shared library is installed in prefix/lib.
> +                libdir = sysconfig.PREFIX + '/lib'
> +            libs.insert(0, '-L' + libdir)
>              libs.extend(getvar('LINKFORSHARED').split())
>          print ' '.join(libs)
>  
> -- 
> 1.7.1

-- 
Joel

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Tom Tromey]

>>>>> "Joel" == Joel Brobecker <brobecker@adacore.com> writes:

Joel> There are currently 2 issues, I think, with python-config --ldflags:
Joel> 1. When python is configured with --enable-shared:
Joel> -L<prefix>/lib is missing.  This is a problem if python was
Joel> installed in a non-standard location.
[...]


Joel> No comment either on the Python side or on the gdb-patches side.

Joel> I think it's too late for 7.2, but I would like to apply this patch
Joel> on the HEAD. Any objections? If we ever create a 7.2.1, and this patch
Joel> was shown to not cause any problem, then perhaps I'll consider it for
Joel> backporting on the 7.2 branch - if approved by the GMs.

It sounds reasonable to me.
I don't know enough about all the Python configuration and install
options to fully understand the patch, but it also seemed ok.

Tom

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Jan Kratochvil]

On Fri, 09 Jul 2010 00:24:45 +0200, Joel Brobecker wrote:
> --- a/gdb/python/python-config.py
> +++ b/gdb/python/python-config.py
> @@ -50,8 +50,21 @@ for opt in opt_flags:
>          # add the prefix/lib/pythonX.Y/config dir, but only if there is no
>          # shared library in prefix/lib/.
>          if opt == '--ldflags':
> +            # Provide the location where the Python library is installed.
> +            # We always provide it, because Python may have been installed
> +            # at a non-standard location.
>              if not getvar('Py_ENABLE_SHARED'):
> -                libs.insert(0, '-L' + getvar('LIBPL'))
> +                # There is no shared library in prefix/lib.  The static
> +                # library is in prefix/lib/pythonX.Y/config.
> +                #
> +                # Note that we cannot use getvar('LIBPL') like we used to,
> +                # because it provides the location at build time, which might
> +                # be different from the actual location at runtime.
> +                libdir = sysconfig.get_python_lib(standard_lib=True) + '/config'
> +            else:
> +                # The Python shared library is installed in prefix/lib.
> +                libdir = sysconfig.PREFIX + '/lib'

On Fedora 14snapshot x86_64 it generates the following warnings:

/usr/bin/ld: skipping incompatible /usr/lib/libtinfo.so when searching for -ltinfo
/usr/bin/ld: skipping incompatible /usr/lib/libz.so when searching for -lz
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libutil.so when searching for -lutil
/usr/bin/ld: skipping incompatible /usr/lib/libutil.a when searching for -lutil
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpython2.7.so when searching for -lpython2.7
/usr/bin/ld: skipping incompatible /usr/lib/libexpat.so when searching for -lexpat
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc
/usr/bin/ld: skipping incompatible /usr/lib/libtinfo.so when searching for -ltinfo
/usr/bin/ld: skipping incompatible /usr/lib/libz.so when searching for -lz
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libutil.so when searching for -lutil
/usr/bin/ld: skipping incompatible /usr/lib/libutil.a when searching for -lutil
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpython2.7.so when searching for -lpython2.7
/usr/bin/ld: skipping incompatible /usr/lib/libexpat.so when searching for -lexpat
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc

Sure it still links correctly.  But I somehow find incorrect to supply
inappropriate libraries into the -L path, such as for occasional buggy linker
script contained in the .so files missing proper OUTPUT_FORMAT protection etc.

I do not mind much, though, correct system should copy with it.


Regards,
Jan

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Joel Brobecker]

> On Fedora 14snapshot x86_64 it generates the following warnings:
> 
> /usr/bin/ld: skipping incompatible /usr/lib/libtinfo.so when searching for -ltinfo

I presume that this is because python is installed in /usr/bin, but
/usr/lib contains the 32bit SOs? Where is the libpython.so installed
for fedora? On Ubuntu, for instance, /usr/lib/libpython2.6.so.1.0 is
a 64bit library....

-- 
Joel

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Jan Kratochvil]

On Wed, 11 Aug 2010 21:07:22 +0200, Joel Brobecker wrote:
> > On Fedora 14snapshot x86_64 it generates the following warnings:
> > 
> > /usr/bin/ld: skipping incompatible /usr/lib/libtinfo.so when searching for -ltinfo
> 
> I presume that this is because python is installed in /usr/bin, but
> /usr/lib contains the 32bit SOs? Where is the libpython.so installed
> for fedora? On Ubuntu, for instance, /usr/lib/libpython2.6.so.1.0 is
> a 64bit library....

On Fedora 64-bit libraries are in /usr/lib64 (and /lib64 etc.).

For 32-bit legacy compatibility libraries 64-bit Fedora uses the exact binary
build of packages being used for 32-bit distro.


Thanks,
Jan

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Joel Brobecker]

> On Fedora 64-bit libraries are in /usr/lib64 (and /lib64 etc.).

What does your Python return for the following:

        from distutils import sysconfig
        sysconfig.get_python_lib()

?

Can you also dig out the command use to configure Python when you build
it for Fedora?

-- 
Joel

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Jan Kratochvil]

On Wed, 11 Aug 2010 21:39:42 +0200, Joel Brobecker wrote:
> > On Fedora 64-bit libraries are in /usr/lib64 (and /lib64 etc.).
> 
> What does your Python return for the following:
> 
>         from distutils import sysconfig
>         sysconfig.get_python_lib()
> 
> ?

Python 2.7 (r27:82500, Jul 26 2010, 18:19:48) 
[GCC 4.5.0 20100716 (Red Hat 4.5.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils import sysconfig
>>> sysconfig.get_python_lib()
'/usr/lib/python2.7/site-packages'
>>> 


> Can you also dig out the command use to configure Python when you build
> it for Fedora?

Build log is here:
	http://kojipkgs.fedoraproject.org/packages/python/2.7/7.fc14/data/logs/x86_64/build.log
->
	/builddir/build/BUILD/Python-2.7/configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-ipv6 --enable-unicode=ucs4 --enable-shared --with-system-ffi --with-valgrind --with-dtrace --with-tapset-install-dir=/usr/share/systemtap/tapset --with-system-expat --with-pydebug --with-tsc --with-count-allocs --with-call-profile

General build info:
	https://koji.fedoraproject.org/koji/buildinfo?buildID=186555


Thanks,
Jan

Re: [RFC] python-config.py --ldflags should return relocated path to libpython

[Joel Brobecker]

> > /usr/bin/ld: skipping incompatible /usr/lib/libtinfo.so when searching for -ltinfo
> 
> I presume that this is because python is installed in /usr/bin, but
> /usr/lib contains the 32bit SOs? Where is the libpython.so installed
> for fedora? On Ubuntu, for instance, /usr/lib/libpython2.6.so.1.0 is
> a 64bit library....

I have fought hard trying to come up with a simple way to make things
work in the case where Python is configured with a different prefixlib,
but to no avail. For now, I have to withdraw that patch as being
incomplete (and maybe even incorrect for Windows/Darwin). I will notify
the Python developers as well.  Thanks for spotting and reporting
the problem.

-- 
Joel