Finding GDB's Python Libraries

Finding GDB's Python Libraries

[Andrew Burgess]

I have been thinking about how we tell GDB where to find the Python
libraries, and I'm currently going around in circles trying to figure
out what the right thing to do is.

Some background reading, some original work to tell GDB about where to
find Python libraries:

  https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html

This allowed GDB to figure out where the Python it was configured with
was located, and from that Python can figure out where its libraries are.

Then there was this:

  https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html

which built on the above, but was more flexible in some cross
compiling situations, not requiring Python to actually be installed
into the GDB install tree (just the libraries can be installed).

I recently ran into a situation where I needed to cross-build GDB for
a MinGW target, including Python support.

By writing a support script as described here:

  https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport

I was able to successfully build a MinGW GDB on Linux.  I configured
this GDB something like this:

  /project/gdb/src/configure --target=.....  --host=i686-w64-mingw32 \
                             --prefix=/project/gdb/install
                             --with-python=/project/gdb/x-compile-script.sh
                             --with-python-libdir=/project/gdb/install/lib/

After building and installing GDB I then copied the Python libraries
and the Python DLL from an unpacked i686-w64-mingw32 Python package
into the GDB install directory. With this done I was able to zip the
GDB install directory and distribute it as needed, it all worked fine,
except....

... I ran into a case where a user had PYTHONHOME set in their
environment, and the Python libraries that this path pointed to were
not compatible with the MinGW Python version that GDB was linking
against.

The motivation for originally setting PYTHONHOME in the environment
have been lost to the mists of time, but this started me thinking.

If we have configured GDB to use a specific set of Python libraries,
even going so far (in this case) to ship the libraries along with GDB,
should we allow PYTHONHOME to override that library selection?

My initial thought was sure, that makes sense, that's what PYTHONHOME
is for.  But, what if the user really needed PYTHONHOME to be set in
order to correctly use their installed (nothing to do with GDB) Python
interpreter?  In that case they now have a situation, through no fault
of their own, where GDB doesn't "just work".


A quick digression on --with-python and --with-python-libdir:
Currently, even when the --with-python-libdir flag is not passed at
configure time, if configure chooses to use Python, then a setting for
this flag is created within the configure script.  As far as GDB is
concerned it is impossible to tell if the user specified
--with-python-libdir, or if configure filled this in for us.  I think
this might be important shortly.


Currently within GDB we call Py_SetProgramName.  This call tells the
Python library where the Python executable is, and from this path the
Python library figures out where the libraries are.  Funnily the value
we pass to Py_SetProgramName is actually built within GDB based on the
location where the libraries are.  So,

  --with-python-libdir=/usr/lib

Causes us to call:

  Py_SetProgramName ("/usr/bin/python")

From which the Python library figures out that the libraries are
located at:

  /usr/lib

It doesn't actually matter to the Python library if there isn't a
python at /usr/bin/python, so long as the libraries are where it
expects them all is good.  This is why we can do:

  --with-python=/usr/bin/python
  --with-python-libdir=/project/gdb/install/lib

Even if there is never a /project/gdb/install/bin/python, our the
Python library will still find the libraries correctly.

Setting the program name in this way still allows for PYTHONHOME to
override the location of the libraries.  However, we could switch to
using Py_SetPythonHome, which also tells the Python library where to
finds its library files, but doesn't honour PYTHONHOME.

In some cases using Py_SetPythonHome seems like a better choice, but
in other cases not listening to PYTHONHOME feels like a mistake, and
Py_SetProgramName feels like a better choice.  The problem I'm
struggling with is what would be a good set of rules for choosing
between these two options?

I initially thought that maybe if the user has specified a
--with-python-libdir that is under the install prefix of GDB then we
should not honour PYTHONPATH (so use Py_SetPythonHome), however, that
doesn't feel strong enough.

What if the user does:

  --with-python=/opt/my-special-python/bin/python
  --with-python-libdir=/opt/my-special-python/lib

Should they not expect GDB to pick this up in preference to any other
Python?

So then I wondered if we could detect the case where configure has
found the "default" Python on the machine, so, if the user configures
like:

  --with-python=yes

Then they will get whatever python configure can find, maybe in this
case, and this case only we should listen to PYTHONHOME, but if the
user has specifically specified a particular version of python in any
location, then we should force GDB to use that python above all
others?

If we did go down this route (or make any use of Py_SetPythonHome)
then I would suggest we have GDB look for a new environment variable
GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB,
and would override the value from --with-python-libdir.

I'd be interested to hear if anyone has any thoughts on this issue.

Thanks,
Andrew

Re: Finding GDB's Python Libraries

[Joel Brobecker]

[-- Attachment #1: Type: text/plain, Size: 7052 bytes --]

Hi Andrew,

On Mon, Jul 06, 2020 at 06:33:46PM +0100, Andrew Burgess wrote:
> I have been thinking about how we tell GDB where to find the Python
> libraries, and I'm currently going around in circles trying to figure
> out what the right thing to do is.
> 
> Some background reading, some original work to tell GDB about where to
> find Python libraries:
> 
>   https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html
> 
> This allowed GDB to figure out where the Python it was configured with
> was located, and from that Python can figure out where its libraries are.
> 
> Then there was this:
> 
>   https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html
> 
> which built on the above, but was more flexible in some cross
> compiling situations, not requiring Python to actually be installed
> into the GDB install tree (just the libraries can be installed).
> 
> I recently ran into a situation where I needed to cross-build GDB for
> a MinGW target, including Python support.
> 
> By writing a support script as described here:
> 
>   https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport
> 
> I was able to successfully build a MinGW GDB on Linux.  I configured
> this GDB something like this:
> 
>   /project/gdb/src/configure --target=.....  --host=i686-w64-mingw32 \
>                              --prefix=/project/gdb/install
>                              --with-python=/project/gdb/x-compile-script.sh
>                              --with-python-libdir=/project/gdb/install/lib/
> 
> After building and installing GDB I then copied the Python libraries
> and the Python DLL from an unpacked i686-w64-mingw32 Python package
> into the GDB install directory. With this done I was able to zip the
> GDB install directory and distribute it as needed, it all worked fine,
> except....
> 
> ... I ran into a case where a user had PYTHONHOME set in their
> environment, and the Python libraries that this path pointed to were
> not compatible with the MinGW Python version that GDB was linking
> against.
> 
> The motivation for originally setting PYTHONHOME in the environment
> have been lost to the mists of time, but this started me thinking.
> 
> If we have configured GDB to use a specific set of Python libraries,
> even going so far (in this case) to ship the libraries along with GDB,
> should we allow PYTHONHOME to override that library selection?
> 
> My initial thought was sure, that makes sense, that's what PYTHONHOME
> is for.  But, what if the user really needed PYTHONHOME to be set in
> order to correctly use their installed (nothing to do with GDB) Python
> interpreter?  In that case they now have a situation, through no fault
> of their own, where GDB doesn't "just work".
> 
> 
> A quick digression on --with-python and --with-python-libdir:
> Currently, even when the --with-python-libdir flag is not passed at
> configure time, if configure chooses to use Python, then a setting for
> this flag is created within the configure script.  As far as GDB is
> concerned it is impossible to tell if the user specified
> --with-python-libdir, or if configure filled this in for us.  I think
> this might be important shortly.
> 
> 
> Currently within GDB we call Py_SetProgramName.  This call tells the
> Python library where the Python executable is, and from this path the
> Python library figures out where the libraries are.  Funnily the value
> we pass to Py_SetProgramName is actually built within GDB based on the
> location where the libraries are.  So,
> 
>   --with-python-libdir=/usr/lib
> 
> Causes us to call:
> 
>   Py_SetProgramName ("/usr/bin/python")
> 
> >From which the Python library figures out that the libraries are
> located at:
> 
>   /usr/lib
> 
> It doesn't actually matter to the Python library if there isn't a
> python at /usr/bin/python, so long as the libraries are where it
> expects them all is good.  This is why we can do:
> 
>   --with-python=/usr/bin/python
>   --with-python-libdir=/project/gdb/install/lib
> 
> Even if there is never a /project/gdb/install/bin/python, our the
> Python library will still find the libraries correctly.
> 
> Setting the program name in this way still allows for PYTHONHOME to
> override the location of the libraries.  However, we could switch to
> using Py_SetPythonHome, which also tells the Python library where to
> finds its library files, but doesn't honour PYTHONHOME.
> 
> In some cases using Py_SetPythonHome seems like a better choice, but
> in other cases not listening to PYTHONHOME feels like a mistake, and
> Py_SetProgramName feels like a better choice.  The problem I'm
> struggling with is what would be a good set of rules for choosing
> between these two options?
> 
> I initially thought that maybe if the user has specified a
> --with-python-libdir that is under the install prefix of GDB then we
> should not honour PYTHONPATH (so use Py_SetPythonHome), however, that
> doesn't feel strong enough.
> 
> What if the user does:
> 
>   --with-python=/opt/my-special-python/bin/python
>   --with-python-libdir=/opt/my-special-python/lib
> 
> Should they not expect GDB to pick this up in preference to any other
> Python?
> 
> So then I wondered if we could detect the case where configure has
> found the "default" Python on the machine, so, if the user configures
> like:
> 
>   --with-python=yes
> 
> Then they will get whatever python configure can find, maybe in this
> case, and this case only we should listen to PYTHONHOME, but if the
> user has specifically specified a particular version of python in any
> location, then we should force GDB to use that python above all
> others?
> 
> If we did go down this route (or make any use of Py_SetPythonHome)
> then I would suggest we have GDB look for a new environment variable
> GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB,
> and would override the value from --with-python-libdir.
> 
> I'd be interested to hear if anyone has any thoughts on this issue.

At AdaCore, we ship a version that overrides any value that PYTHONHOME
might have, for exactly the reason that you explained. You hit that issue
on Windows, but from our perspective, this isn't just OS-specific either.

I found the following reference to a discussion where we proposed
the approach:
https://sourceware.org/pipermail/gdb-patches/2010-November/078220.html
https://sourceware.org/pipermail/gdb-patches/2010-December/078618.html
https://sourceware.org/pipermail/gdb-patches/2011-January/079287.html

It looks like the discussed died because I gave up due to lack of
support, that Doug clarified by saying we should use honor
GDB_PYTHONHOME and disregard PYTHONHOME instead. My answer was
not very clear, but it sounds like I was thinking "someone else"
can take care of it (I'm wondering how I could have thought that,
but anyways).

Attached is a local patch that we have internally at AdaCore.
I think you could build on it to add support for GDB_PYTHONHOME
and submit it, if others agree that this is going in the right
direction.

-- 
Joel

[-- Attachment #2: python-home-adacore.diff --]
[-- Type: text/x-diff, Size: 2088 bytes --]

diff --git a/gdb/python/python.c b/gdb/python/python.c
index 67f362b852d..f5ace2ff978 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -1644,6 +1644,8 @@ do_start_initialization ()
      program_name, and respect the requirement of Py_SetProgramName
      for Python versions that do not duplicate program_name.  */
   static wchar_t *progname_copy;
+  /* Similar treatment for the Python home directory.  */
+  static wchar_t *home_copy;
 #endif
 
 #ifdef WITH_PYTHON_PATH
@@ -1667,6 +1669,18 @@ do_start_initialization ()
       fprintf (stderr, "Could not convert python path to string\n");
       return false;
     }
+
+  std::string homedir = ldirname (python_libdir.c_str ());
+  home_copy = XNEWVEC (wchar_t, homedir.size () + 1);
+  count = mbstowcs (home_copy, homedir.c_str (), homedir.size () + 1);
+  if (count == (size_t) -1)
+    {
+      fprintf (stderr, "Could not convert python home to wide string\n");
+      return false;
+    }
+  /* Note that, as with the program name, we don't free HOME_COPY.  */
+  Py_SetPythonHome (home_copy);
+
   setlocale (LC_ALL, oldloc.c_str ());
 
   /* Note that Py_SetProgramName expects the string it is passed to
@@ -1678,8 +1692,22 @@ do_start_initialization ()
   PyImport_AppendInittab ("_gdb", init__gdb_module);
 #else
   Py_SetProgramName (progname.release ());
-#endif
-#endif
+
+  /* We override any value that the PYTHONHOME might have, as we want
+     to make sure that we use the Python library that comes with GDB.
+
+     Note that the Python 2.7.17 documentation for Py_SetPythonHome
+     says that the "home" argument should be:
+
+         [...] a zero-terminated character string in static storage
+	 whose contents will not change for the duration of the program’s
+	 execution. No code in the Python interpreter will change
+	 the contents of this storage.
+
+     That's why we have to xstrdup the dirname.  */
+  Py_SetPythonHome (xstrdup (ldirname (python_libdir.c_str ()).c_str ()));
+#endif	/* IS_PY3K */
+#endif	/* WITH_PYTHON_PATH */
 
   Py_Initialize ();
 #if PY_VERSION_HEX < 0x03090000