Hi Andrew, On Mon, Jul 06, 2020 at 06:33:46PM +0100, Andrew Burgess wrote: > I have been thinking about how we tell GDB where to find the Python > libraries, and I'm currently going around in circles trying to figure > out what the right thing to do is. > > Some background reading, some original work to tell GDB about where to > find Python libraries: > > https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html > > This allowed GDB to figure out where the Python it was configured with > was located, and from that Python can figure out where its libraries are. > > Then there was this: > > https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html > > which built on the above, but was more flexible in some cross > compiling situations, not requiring Python to actually be installed > into the GDB install tree (just the libraries can be installed). > > I recently ran into a situation where I needed to cross-build GDB for > a MinGW target, including Python support. > > By writing a support script as described here: > > https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport > > I was able to successfully build a MinGW GDB on Linux. I configured > this GDB something like this: > > /project/gdb/src/configure --target=..... --host=i686-w64-mingw32 \ > --prefix=/project/gdb/install > --with-python=/project/gdb/x-compile-script.sh > --with-python-libdir=/project/gdb/install/lib/ > > After building and installing GDB I then copied the Python libraries > and the Python DLL from an unpacked i686-w64-mingw32 Python package > into the GDB install directory. With this done I was able to zip the > GDB install directory and distribute it as needed, it all worked fine, > except.... > > ... I ran into a case where a user had PYTHONHOME set in their > environment, and the Python libraries that this path pointed to were > not compatible with the MinGW Python version that GDB was linking > against. > > The motivation for originally setting PYTHONHOME in the environment > have been lost to the mists of time, but this started me thinking. > > If we have configured GDB to use a specific set of Python libraries, > even going so far (in this case) to ship the libraries along with GDB, > should we allow PYTHONHOME to override that library selection? > > My initial thought was sure, that makes sense, that's what PYTHONHOME > is for. But, what if the user really needed PYTHONHOME to be set in > order to correctly use their installed (nothing to do with GDB) Python > interpreter? In that case they now have a situation, through no fault > of their own, where GDB doesn't "just work". > > > A quick digression on --with-python and --with-python-libdir: > Currently, even when the --with-python-libdir flag is not passed at > configure time, if configure chooses to use Python, then a setting for > this flag is created within the configure script. As far as GDB is > concerned it is impossible to tell if the user specified > --with-python-libdir, or if configure filled this in for us. I think > this might be important shortly. > > > Currently within GDB we call Py_SetProgramName. This call tells the > Python library where the Python executable is, and from this path the > Python library figures out where the libraries are. Funnily the value > we pass to Py_SetProgramName is actually built within GDB based on the > location where the libraries are. So, > > --with-python-libdir=/usr/lib > > Causes us to call: > > Py_SetProgramName ("/usr/bin/python") > > >From which the Python library figures out that the libraries are > located at: > > /usr/lib > > It doesn't actually matter to the Python library if there isn't a > python at /usr/bin/python, so long as the libraries are where it > expects them all is good. This is why we can do: > > --with-python=/usr/bin/python > --with-python-libdir=/project/gdb/install/lib > > Even if there is never a /project/gdb/install/bin/python, our the > Python library will still find the libraries correctly. > > Setting the program name in this way still allows for PYTHONHOME to > override the location of the libraries. However, we could switch to > using Py_SetPythonHome, which also tells the Python library where to > finds its library files, but doesn't honour PYTHONHOME. > > In some cases using Py_SetPythonHome seems like a better choice, but > in other cases not listening to PYTHONHOME feels like a mistake, and > Py_SetProgramName feels like a better choice. The problem I'm > struggling with is what would be a good set of rules for choosing > between these two options? > > I initially thought that maybe if the user has specified a > --with-python-libdir that is under the install prefix of GDB then we > should not honour PYTHONPATH (so use Py_SetPythonHome), however, that > doesn't feel strong enough. > > What if the user does: > > --with-python=/opt/my-special-python/bin/python > --with-python-libdir=/opt/my-special-python/lib > > Should they not expect GDB to pick this up in preference to any other > Python? > > So then I wondered if we could detect the case where configure has > found the "default" Python on the machine, so, if the user configures > like: > > --with-python=yes > > Then they will get whatever python configure can find, maybe in this > case, and this case only we should listen to PYTHONHOME, but if the > user has specifically specified a particular version of python in any > location, then we should force GDB to use that python above all > others? > > If we did go down this route (or make any use of Py_SetPythonHome) > then I would suggest we have GDB look for a new environment variable > GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB, > and would override the value from --with-python-libdir. > > I'd be interested to hear if anyone has any thoughts on this issue. At AdaCore, we ship a version that overrides any value that PYTHONHOME might have, for exactly the reason that you explained. You hit that issue on Windows, but from our perspective, this isn't just OS-specific either. I found the following reference to a discussion where we proposed the approach: https://sourceware.org/pipermail/gdb-patches/2010-November/078220.html https://sourceware.org/pipermail/gdb-patches/2010-December/078618.html https://sourceware.org/pipermail/gdb-patches/2011-January/079287.html It looks like the discussed died because I gave up due to lack of support, that Doug clarified by saying we should use honor GDB_PYTHONHOME and disregard PYTHONHOME instead. My answer was not very clear, but it sounds like I was thinking "someone else" can take care of it (I'm wondering how I could have thought that, but anyways). Attached is a local patch that we have internally at AdaCore. I think you could build on it to add support for GDB_PYTHONHOME and submit it, if others agree that this is going in the right direction. -- Joel