From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by sourceware.org (Postfix) with ESMTPS id 30BD4385E444 for ; Mon, 6 Jul 2020 17:33:49 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 30BD4385E444 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=embecosm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=andrew.burgess@embecosm.com Received: by mail-wr1-x442.google.com with SMTP id s10so41995602wrw.12 for ; Mon, 06 Jul 2020 10:33:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=embecosm.com; s=google; h=date:from:to:subject:message-id:mime-version:content-disposition; bh=1l4l/cu6arsqhvJU5R6xT7rgKyzq7GXXaEd0GJhZfwg=; b=M2sbmbuUrBoh+1nwN3FtVUkuhL5oQc/CUuqAFuNwZ51SecARcizMeYFhaQEcqB/es+ h8pc/jtGh86XdmgXpMp/XVahy8A/8azW9v+jTtr8FGKjnt07V0nsgu0v065EInhHGGvj g1TMUcgqGQiDNBqk6A90E21unDNpbqSkDlZ8sd/h4GuBaE9Z2uBHm4d2a/BeDrvzV1uV sW/n3iwPoMF10yqECtvQmfvtZnGXs7SaeLtw0zZuhuv4jQuO/UzAvm6+0qkkA/c/Rzlc 7TRqawMy6fopCnKSlGbEPMO+iFKBStvK6/3veFl+45w8MyiuzB0fWxLTmpIGTeFsKzm6 wC2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition; bh=1l4l/cu6arsqhvJU5R6xT7rgKyzq7GXXaEd0GJhZfwg=; b=O16pCkIYE99TwJ2H+aBvC/ZDEXTIeDVz4zWGavD5BYUtssXmLsHucmz/y71BWjBR+N +sLWMyaYmZYeVFd++Kj8q1lZGStTuWO9nBpxQIq/917ZNw0iF5/HzesbPGikgTIYqwlg 6u8WDrqD9utY69+Mijc9HSIDX1TnpaWgwcL/Qvbn7LxsNLuAPtF3WWo2KnHQQyHqFo3Y eA6tg1+yaPBUNNpQCevCy8anVGHJjrZ/+FM+kaWrSZ5pJiZgT5cSYYGr5qVMu3dZlw23 MNmnEawPnNwiCW2xSMZIRS5es3bZMopwlhqfkV7AeGLEmu7B6Vzce4fJh5/h+6LhC34a 7nWA== X-Gm-Message-State: AOAM531j+G+1jfLRSukJx2KOM3uBHnYslXlYM/YsTWeEPbKDTPTR/7C4 Ix81caVod1QPup5Fu5R6sQUEgdDWqA8= X-Google-Smtp-Source: ABdhPJyFMgAhVvjxBLm+s02wLGYv0Nr4Ymlo8YirGD4cHFtvdqeJGMwcdjbnE15U9vDde9JGlNEOXg== X-Received: by 2002:adf:edd0:: with SMTP id v16mr49100685wro.214.1594056827959; Mon, 06 Jul 2020 10:33:47 -0700 (PDT) Received: from localhost (host109-148-134-178.range109-148.btcentralplus.com. [109.148.134.178]) by smtp.gmail.com with ESMTPSA id j4sm20527717wrp.51.2020.07.06.10.33.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jul 2020 10:33:47 -0700 (PDT) Date: Mon, 6 Jul 2020 18:33:46 +0100 From: Andrew Burgess To: gdb@sourceware.org Subject: Finding GDB's Python Libraries Message-ID: <20200706173346.GG3463@embecosm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Operating-System: Linux/5.6.15-200.fc31.x86_64 (x86_64) X-Uptime: 18:12:44 up 28 days, 7:19, X-Editor: GNU Emacs [ http://www.gnu.org/software/emacs ] X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2020 17:33:50 -0000 I have been thinking about how we tell GDB where to find the Python libraries, and I'm currently going around in circles trying to figure out what the right thing to do is. Some background reading, some original work to tell GDB about where to find Python libraries: https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html This allowed GDB to figure out where the Python it was configured with was located, and from that Python can figure out where its libraries are. Then there was this: https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html which built on the above, but was more flexible in some cross compiling situations, not requiring Python to actually be installed into the GDB install tree (just the libraries can be installed). I recently ran into a situation where I needed to cross-build GDB for a MinGW target, including Python support. By writing a support script as described here: https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport I was able to successfully build a MinGW GDB on Linux. I configured this GDB something like this: /project/gdb/src/configure --target=..... --host=i686-w64-mingw32 \ --prefix=/project/gdb/install --with-python=/project/gdb/x-compile-script.sh --with-python-libdir=/project/gdb/install/lib/ After building and installing GDB I then copied the Python libraries and the Python DLL from an unpacked i686-w64-mingw32 Python package into the GDB install directory. With this done I was able to zip the GDB install directory and distribute it as needed, it all worked fine, except.... ... I ran into a case where a user had PYTHONHOME set in their environment, and the Python libraries that this path pointed to were not compatible with the MinGW Python version that GDB was linking against. The motivation for originally setting PYTHONHOME in the environment have been lost to the mists of time, but this started me thinking. If we have configured GDB to use a specific set of Python libraries, even going so far (in this case) to ship the libraries along with GDB, should we allow PYTHONHOME to override that library selection? My initial thought was sure, that makes sense, that's what PYTHONHOME is for. But, what if the user really needed PYTHONHOME to be set in order to correctly use their installed (nothing to do with GDB) Python interpreter? In that case they now have a situation, through no fault of their own, where GDB doesn't "just work". A quick digression on --with-python and --with-python-libdir: Currently, even when the --with-python-libdir flag is not passed at configure time, if configure chooses to use Python, then a setting for this flag is created within the configure script. As far as GDB is concerned it is impossible to tell if the user specified --with-python-libdir, or if configure filled this in for us. I think this might be important shortly. Currently within GDB we call Py_SetProgramName. This call tells the Python library where the Python executable is, and from this path the Python library figures out where the libraries are. Funnily the value we pass to Py_SetProgramName is actually built within GDB based on the location where the libraries are. So, --with-python-libdir=/usr/lib Causes us to call: Py_SetProgramName ("/usr/bin/python") >From which the Python library figures out that the libraries are located at: /usr/lib It doesn't actually matter to the Python library if there isn't a python at /usr/bin/python, so long as the libraries are where it expects them all is good. This is why we can do: --with-python=/usr/bin/python --with-python-libdir=/project/gdb/install/lib Even if there is never a /project/gdb/install/bin/python, our the Python library will still find the libraries correctly. Setting the program name in this way still allows for PYTHONHOME to override the location of the libraries. However, we could switch to using Py_SetPythonHome, which also tells the Python library where to finds its library files, but doesn't honour PYTHONHOME. In some cases using Py_SetPythonHome seems like a better choice, but in other cases not listening to PYTHONHOME feels like a mistake, and Py_SetProgramName feels like a better choice. The problem I'm struggling with is what would be a good set of rules for choosing between these two options? I initially thought that maybe if the user has specified a --with-python-libdir that is under the install prefix of GDB then we should not honour PYTHONPATH (so use Py_SetPythonHome), however, that doesn't feel strong enough. What if the user does: --with-python=/opt/my-special-python/bin/python --with-python-libdir=/opt/my-special-python/lib Should they not expect GDB to pick this up in preference to any other Python? So then I wondered if we could detect the case where configure has found the "default" Python on the machine, so, if the user configures like: --with-python=yes Then they will get whatever python configure can find, maybe in this case, and this case only we should listen to PYTHONHOME, but if the user has specifically specified a particular version of python in any location, then we should force GDB to use that python above all others? If we did go down this route (or make any use of Py_SetPythonHome) then I would suggest we have GDB look for a new environment variable GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB, and would override the value from --with-python-libdir. I'd be interested to hear if anyone has any thoughts on this issue. Thanks, Andrew