* Re: Finding GDB's Python Libraries
2020-07-06 17:33 Finding GDB's Python Libraries Andrew Burgess
@ 2020-07-06 18:50 ` Joel Brobecker
0 siblings, 0 replies; 2+ messages in thread
From: Joel Brobecker @ 2020-07-06 18:50 UTC (permalink / raw)
To: Andrew Burgess; +Cc: gdb
[-- Attachment #1: Type: text/plain, Size: 7052 bytes --]
Hi Andrew,
On Mon, Jul 06, 2020 at 06:33:46PM +0100, Andrew Burgess wrote:
> I have been thinking about how we tell GDB where to find the Python
> libraries, and I'm currently going around in circles trying to figure
> out what the right thing to do is.
>
> Some background reading, some original work to tell GDB about where to
> find Python libraries:
>
> https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html
>
> This allowed GDB to figure out where the Python it was configured with
> was located, and from that Python can figure out where its libraries are.
>
> Then there was this:
>
> https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html
>
> which built on the above, but was more flexible in some cross
> compiling situations, not requiring Python to actually be installed
> into the GDB install tree (just the libraries can be installed).
>
> I recently ran into a situation where I needed to cross-build GDB for
> a MinGW target, including Python support.
>
> By writing a support script as described here:
>
> https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport
>
> I was able to successfully build a MinGW GDB on Linux. I configured
> this GDB something like this:
>
> /project/gdb/src/configure --target=..... --host=i686-w64-mingw32 \
> --prefix=/project/gdb/install
> --with-python=/project/gdb/x-compile-script.sh
> --with-python-libdir=/project/gdb/install/lib/
>
> After building and installing GDB I then copied the Python libraries
> and the Python DLL from an unpacked i686-w64-mingw32 Python package
> into the GDB install directory. With this done I was able to zip the
> GDB install directory and distribute it as needed, it all worked fine,
> except....
>
> ... I ran into a case where a user had PYTHONHOME set in their
> environment, and the Python libraries that this path pointed to were
> not compatible with the MinGW Python version that GDB was linking
> against.
>
> The motivation for originally setting PYTHONHOME in the environment
> have been lost to the mists of time, but this started me thinking.
>
> If we have configured GDB to use a specific set of Python libraries,
> even going so far (in this case) to ship the libraries along with GDB,
> should we allow PYTHONHOME to override that library selection?
>
> My initial thought was sure, that makes sense, that's what PYTHONHOME
> is for. But, what if the user really needed PYTHONHOME to be set in
> order to correctly use their installed (nothing to do with GDB) Python
> interpreter? In that case they now have a situation, through no fault
> of their own, where GDB doesn't "just work".
>
>
> A quick digression on --with-python and --with-python-libdir:
> Currently, even when the --with-python-libdir flag is not passed at
> configure time, if configure chooses to use Python, then a setting for
> this flag is created within the configure script. As far as GDB is
> concerned it is impossible to tell if the user specified
> --with-python-libdir, or if configure filled this in for us. I think
> this might be important shortly.
>
>
> Currently within GDB we call Py_SetProgramName. This call tells the
> Python library where the Python executable is, and from this path the
> Python library figures out where the libraries are. Funnily the value
> we pass to Py_SetProgramName is actually built within GDB based on the
> location where the libraries are. So,
>
> --with-python-libdir=/usr/lib
>
> Causes us to call:
>
> Py_SetProgramName ("/usr/bin/python")
>
> >From which the Python library figures out that the libraries are
> located at:
>
> /usr/lib
>
> It doesn't actually matter to the Python library if there isn't a
> python at /usr/bin/python, so long as the libraries are where it
> expects them all is good. This is why we can do:
>
> --with-python=/usr/bin/python
> --with-python-libdir=/project/gdb/install/lib
>
> Even if there is never a /project/gdb/install/bin/python, our the
> Python library will still find the libraries correctly.
>
> Setting the program name in this way still allows for PYTHONHOME to
> override the location of the libraries. However, we could switch to
> using Py_SetPythonHome, which also tells the Python library where to
> finds its library files, but doesn't honour PYTHONHOME.
>
> In some cases using Py_SetPythonHome seems like a better choice, but
> in other cases not listening to PYTHONHOME feels like a mistake, and
> Py_SetProgramName feels like a better choice. The problem I'm
> struggling with is what would be a good set of rules for choosing
> between these two options?
>
> I initially thought that maybe if the user has specified a
> --with-python-libdir that is under the install prefix of GDB then we
> should not honour PYTHONPATH (so use Py_SetPythonHome), however, that
> doesn't feel strong enough.
>
> What if the user does:
>
> --with-python=/opt/my-special-python/bin/python
> --with-python-libdir=/opt/my-special-python/lib
>
> Should they not expect GDB to pick this up in preference to any other
> Python?
>
> So then I wondered if we could detect the case where configure has
> found the "default" Python on the machine, so, if the user configures
> like:
>
> --with-python=yes
>
> Then they will get whatever python configure can find, maybe in this
> case, and this case only we should listen to PYTHONHOME, but if the
> user has specifically specified a particular version of python in any
> location, then we should force GDB to use that python above all
> others?
>
> If we did go down this route (or make any use of Py_SetPythonHome)
> then I would suggest we have GDB look for a new environment variable
> GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB,
> and would override the value from --with-python-libdir.
>
> I'd be interested to hear if anyone has any thoughts on this issue.
At AdaCore, we ship a version that overrides any value that PYTHONHOME
might have, for exactly the reason that you explained. You hit that issue
on Windows, but from our perspective, this isn't just OS-specific either.
I found the following reference to a discussion where we proposed
the approach:
https://sourceware.org/pipermail/gdb-patches/2010-November/078220.html
https://sourceware.org/pipermail/gdb-patches/2010-December/078618.html
https://sourceware.org/pipermail/gdb-patches/2011-January/079287.html
It looks like the discussed died because I gave up due to lack of
support, that Doug clarified by saying we should use honor
GDB_PYTHONHOME and disregard PYTHONHOME instead. My answer was
not very clear, but it sounds like I was thinking "someone else"
can take care of it (I'm wondering how I could have thought that,
but anyways).
Attached is a local patch that we have internally at AdaCore.
I think you could build on it to add support for GDB_PYTHONHOME
and submit it, if others agree that this is going in the right
direction.
--
Joel
[-- Attachment #2: python-home-adacore.diff --]
[-- Type: text/x-diff, Size: 2088 bytes --]
diff --git a/gdb/python/python.c b/gdb/python/python.c
index 67f362b852d..f5ace2ff978 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -1644,6 +1644,8 @@ do_start_initialization ()
program_name, and respect the requirement of Py_SetProgramName
for Python versions that do not duplicate program_name. */
static wchar_t *progname_copy;
+ /* Similar treatment for the Python home directory. */
+ static wchar_t *home_copy;
#endif
#ifdef WITH_PYTHON_PATH
@@ -1667,6 +1669,18 @@ do_start_initialization ()
fprintf (stderr, "Could not convert python path to string\n");
return false;
}
+
+ std::string homedir = ldirname (python_libdir.c_str ());
+ home_copy = XNEWVEC (wchar_t, homedir.size () + 1);
+ count = mbstowcs (home_copy, homedir.c_str (), homedir.size () + 1);
+ if (count == (size_t) -1)
+ {
+ fprintf (stderr, "Could not convert python home to wide string\n");
+ return false;
+ }
+ /* Note that, as with the program name, we don't free HOME_COPY. */
+ Py_SetPythonHome (home_copy);
+
setlocale (LC_ALL, oldloc.c_str ());
/* Note that Py_SetProgramName expects the string it is passed to
@@ -1678,8 +1692,22 @@ do_start_initialization ()
PyImport_AppendInittab ("_gdb", init__gdb_module);
#else
Py_SetProgramName (progname.release ());
-#endif
-#endif
+
+ /* We override any value that the PYTHONHOME might have, as we want
+ to make sure that we use the Python library that comes with GDB.
+
+ Note that the Python 2.7.17 documentation for Py_SetPythonHome
+ says that the "home" argument should be:
+
+ [...] a zero-terminated character string in static storage
+ whose contents will not change for the duration of the program’s
+ execution. No code in the Python interpreter will change
+ the contents of this storage.
+
+ That's why we have to xstrdup the dirname. */
+ Py_SetPythonHome (xstrdup (ldirname (python_libdir.c_str ()).c_str ()));
+#endif /* IS_PY3K */
+#endif /* WITH_PYTHON_PATH */
Py_Initialize ();
#if PY_VERSION_HEX < 0x03090000
^ permalink raw reply [flat|nested] 2+ messages in thread