From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rock.gnat.com (rock.gnat.com [205.232.38.15]) by sourceware.org (Postfix) with ESMTP id 18C22385702C for ; Mon, 6 Jul 2020 18:50:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 18C22385702C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=adacore.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=brobecker@adacore.com Received: from localhost (localhost.localdomain [127.0.0.1]) by filtered-rock.gnat.com (Postfix) with ESMTP id E458A1169AC; Mon, 6 Jul 2020 14:50:45 -0400 (EDT) X-Virus-Scanned: Debian amavisd-new at gnat.com Received: from rock.gnat.com ([127.0.0.1]) by localhost (rock.gnat.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id I43Hb48SZbi9; Mon, 6 Jul 2020 14:50:45 -0400 (EDT) Received: from joel.gnat.com (localhost.localdomain [127.0.0.1]) by rock.gnat.com (Postfix) with ESMTP id AD9751169A6; Mon, 6 Jul 2020 14:50:45 -0400 (EDT) Received: by joel.gnat.com (Postfix, from userid 1000) id D64DB864E6; Mon, 6 Jul 2020 11:50:43 -0700 (PDT) Date: Mon, 6 Jul 2020 11:50:43 -0700 From: Joel Brobecker To: Andrew Burgess Cc: gdb@sourceware.org Subject: Re: Finding GDB's Python Libraries Message-ID: <20200706185043.GF15114@adacore.com> References: <20200706173346.GG3463@embecosm.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8P1HSweYDcXXzwPJ" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200706173346.GG3463@embecosm.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Spam-Status: No, score=-8.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2020 18:50:47 -0000 --8P1HSweYDcXXzwPJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Andrew, On Mon, Jul 06, 2020 at 06:33:46PM +0100, Andrew Burgess wrote: > I have been thinking about how we tell GDB where to find the Python > libraries, and I'm currently going around in circles trying to figure > out what the right thing to do is. > > Some background reading, some original work to tell GDB about where to > find Python libraries: > > https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html > > This allowed GDB to figure out where the Python it was configured with > was located, and from that Python can figure out where its libraries are. > > Then there was this: > > https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html > > which built on the above, but was more flexible in some cross > compiling situations, not requiring Python to actually be installed > into the GDB install tree (just the libraries can be installed). > > I recently ran into a situation where I needed to cross-build GDB for > a MinGW target, including Python support. > > By writing a support script as described here: > > https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport > > I was able to successfully build a MinGW GDB on Linux. I configured > this GDB something like this: > > /project/gdb/src/configure --target=..... --host=i686-w64-mingw32 \ > --prefix=/project/gdb/install > --with-python=/project/gdb/x-compile-script.sh > --with-python-libdir=/project/gdb/install/lib/ > > After building and installing GDB I then copied the Python libraries > and the Python DLL from an unpacked i686-w64-mingw32 Python package > into the GDB install directory. With this done I was able to zip the > GDB install directory and distribute it as needed, it all worked fine, > except.... > > ... I ran into a case where a user had PYTHONHOME set in their > environment, and the Python libraries that this path pointed to were > not compatible with the MinGW Python version that GDB was linking > against. > > The motivation for originally setting PYTHONHOME in the environment > have been lost to the mists of time, but this started me thinking. > > If we have configured GDB to use a specific set of Python libraries, > even going so far (in this case) to ship the libraries along with GDB, > should we allow PYTHONHOME to override that library selection? > > My initial thought was sure, that makes sense, that's what PYTHONHOME > is for. But, what if the user really needed PYTHONHOME to be set in > order to correctly use their installed (nothing to do with GDB) Python > interpreter? In that case they now have a situation, through no fault > of their own, where GDB doesn't "just work". > > > A quick digression on --with-python and --with-python-libdir: > Currently, even when the --with-python-libdir flag is not passed at > configure time, if configure chooses to use Python, then a setting for > this flag is created within the configure script. As far as GDB is > concerned it is impossible to tell if the user specified > --with-python-libdir, or if configure filled this in for us. I think > this might be important shortly. > > > Currently within GDB we call Py_SetProgramName. This call tells the > Python library where the Python executable is, and from this path the > Python library figures out where the libraries are. Funnily the value > we pass to Py_SetProgramName is actually built within GDB based on the > location where the libraries are. So, > > --with-python-libdir=/usr/lib > > Causes us to call: > > Py_SetProgramName ("/usr/bin/python") > > >From which the Python library figures out that the libraries are > located at: > > /usr/lib > > It doesn't actually matter to the Python library if there isn't a > python at /usr/bin/python, so long as the libraries are where it > expects them all is good. This is why we can do: > > --with-python=/usr/bin/python > --with-python-libdir=/project/gdb/install/lib > > Even if there is never a /project/gdb/install/bin/python, our the > Python library will still find the libraries correctly. > > Setting the program name in this way still allows for PYTHONHOME to > override the location of the libraries. However, we could switch to > using Py_SetPythonHome, which also tells the Python library where to > finds its library files, but doesn't honour PYTHONHOME. > > In some cases using Py_SetPythonHome seems like a better choice, but > in other cases not listening to PYTHONHOME feels like a mistake, and > Py_SetProgramName feels like a better choice. The problem I'm > struggling with is what would be a good set of rules for choosing > between these two options? > > I initially thought that maybe if the user has specified a > --with-python-libdir that is under the install prefix of GDB then we > should not honour PYTHONPATH (so use Py_SetPythonHome), however, that > doesn't feel strong enough. > > What if the user does: > > --with-python=/opt/my-special-python/bin/python > --with-python-libdir=/opt/my-special-python/lib > > Should they not expect GDB to pick this up in preference to any other > Python? > > So then I wondered if we could detect the case where configure has > found the "default" Python on the machine, so, if the user configures > like: > > --with-python=yes > > Then they will get whatever python configure can find, maybe in this > case, and this case only we should listen to PYTHONHOME, but if the > user has specifically specified a particular version of python in any > location, then we should force GDB to use that python above all > others? > > If we did go down this route (or make any use of Py_SetPythonHome) > then I would suggest we have GDB look for a new environment variable > GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB, > and would override the value from --with-python-libdir. > > I'd be interested to hear if anyone has any thoughts on this issue. At AdaCore, we ship a version that overrides any value that PYTHONHOME might have, for exactly the reason that you explained. You hit that issue on Windows, but from our perspective, this isn't just OS-specific either. I found the following reference to a discussion where we proposed the approach: https://sourceware.org/pipermail/gdb-patches/2010-November/078220.html https://sourceware.org/pipermail/gdb-patches/2010-December/078618.html https://sourceware.org/pipermail/gdb-patches/2011-January/079287.html It looks like the discussed died because I gave up due to lack of support, that Doug clarified by saying we should use honor GDB_PYTHONHOME and disregard PYTHONHOME instead. My answer was not very clear, but it sounds like I was thinking "someone else" can take care of it (I'm wondering how I could have thought that, but anyways). Attached is a local patch that we have internally at AdaCore. I think you could build on it to add support for GDB_PYTHONHOME and submit it, if others agree that this is going in the right direction. -- Joel --8P1HSweYDcXXzwPJ Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="python-home-adacore.diff" Content-Transfer-Encoding: 8bit diff --git a/gdb/python/python.c b/gdb/python/python.c index 67f362b852d..f5ace2ff978 100644 --- a/gdb/python/python.c +++ b/gdb/python/python.c @@ -1644,6 +1644,8 @@ do_start_initialization () program_name, and respect the requirement of Py_SetProgramName for Python versions that do not duplicate program_name. */ static wchar_t *progname_copy; + /* Similar treatment for the Python home directory. */ + static wchar_t *home_copy; #endif #ifdef WITH_PYTHON_PATH @@ -1667,6 +1669,18 @@ do_start_initialization () fprintf (stderr, "Could not convert python path to string\n"); return false; } + + std::string homedir = ldirname (python_libdir.c_str ()); + home_copy = XNEWVEC (wchar_t, homedir.size () + 1); + count = mbstowcs (home_copy, homedir.c_str (), homedir.size () + 1); + if (count == (size_t) -1) + { + fprintf (stderr, "Could not convert python home to wide string\n"); + return false; + } + /* Note that, as with the program name, we don't free HOME_COPY. */ + Py_SetPythonHome (home_copy); + setlocale (LC_ALL, oldloc.c_str ()); /* Note that Py_SetProgramName expects the string it is passed to @@ -1678,8 +1692,22 @@ do_start_initialization () PyImport_AppendInittab ("_gdb", init__gdb_module); #else Py_SetProgramName (progname.release ()); -#endif -#endif + + /* We override any value that the PYTHONHOME might have, as we want + to make sure that we use the Python library that comes with GDB. + + Note that the Python 2.7.17 documentation for Py_SetPythonHome + says that the "home" argument should be: + + [...] a zero-terminated character string in static storage + whose contents will not change for the duration of the program’s + execution. No code in the Python interpreter will change + the contents of this storage. + + That's why we have to xstrdup the dirname. */ + Py_SetPythonHome (xstrdup (ldirname (python_libdir.c_str ()).c_str ())); +#endif /* IS_PY3K */ +#endif /* WITH_PYTHON_PATH */ Py_Initialize (); #if PY_VERSION_HEX < 0x03090000 --8P1HSweYDcXXzwPJ--