Re: Python enabled gdb on Windows and relocation

[Ruben Van Boxem <vanboxem.ruben@gmail.com>]

2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>:
> 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>:
>> 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>:
>>> 2011/5/14 Doug Evans <dje@google.com>:
>>>> On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem
>>>> <vanboxem.ruben@gmail.com> wrote:
>>>>> 2011/5/14 Doug Evans <dje@google.com>:
>>>>>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem
>>>>>> <vanboxem.ruben@gmail.com> wrote:
>>>>>>> (now in plain-text as required by gdb mailing list)
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am currently trying to integrate Python support into my toolchain
>>>>>>> build (including GDB of course). It is a sysrooted
>>>>>>> binutils+GCC+GDB+mingw-w64 toolchain.
>>>>>>>
>>>>>>> I currently have the basic setup working: I can link gdb with my
>>>>>>> manually generated import lib to the python dll from the official
>>>>>>> Windows install. If there is anything I am missing or a very easy
>>>>>>> solution to the problems decsribed below, please just say so. I am
>>>>>>> only suggesting what I would like to happen.
>>>>>>>
>>>>>>> Now on to the problems I'd like to discuss:
>>>>>>>
>>>>>>> 1. gdb.exe won't start without me having set PYTHONPATH manually.
>>>>>>
>>>>>> In a properly configured/built gdb on linux this isn't necessary, even
>>>>>> if python is installed in some random place.
>>>>>> I'm not sure about windows though.
>>>>>> Did you specify --with-python when you configured gdb, and if so did
>>>>>> you specify a value?
>>>>>> e.g., --with-python=SOME_VALUE
>>>>>
>>>>> I was cross-compiling a mingw toolchain+gdb from Linux, so I used
>>>>> --with-python without a value (because gdb configure tries to find the
>>>>> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS
>>>>> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it
>>>>> should. This is hacky though, and gdb configure should provide
>>>>> --with-python-libs and --with-python-include to make it more
>>>>> streamlined with any other build prerequisite (like
>>>>> gmp/mpfr/mpc/cloog/ppl in GCC for example).
>>>>
>>>> Ah.
>>>> Cross-compiling gdb with python is in need of improvement.
>>>> Alas python hasn't been designed with cross-compilation in mind (e.g.
>>>> build on linux, run on windows).
>>>> AIUI, the way to get the parameters required for compiling with
>>>> libpython is to get them from python's "distutils": kinda hard to do
>>>> in a cross-compile.  Done correctly there's no need to run python.
>>>>
>>>> I haven't done anything more to support python in gdb's configure.ac
>>>> because it's not clear to me what the right thing to do is: distutils
>>>> provides more than just --libs and --includes (btw, we don't use
>>>> --libs though, we use --ldflags which includes all of: the directory
>>>> in which to find libpython, the -l for libpython, and the -l's for all
>>>> the other libraries python needs). [Which isn't to say that someone
>>>> else isn't free to tackle this.]
>>>>
>>>> In the meantime, what I've been doing is a hack: write a script that
>>>> responds to:
>>>> --includes
>>>> --ldflags
>>>> --exec-prefix
>>>> and pass that as --with-python.
>>>>
>>>> E.g.
>>>> bash$ cat $HOME/my-python-for-config
>>>> #! /bin/sh
>>>>
>>>> if [ $# -ne 2 ]
>>>> then
>>>>        echo "Bad # args.  Blech!" >&2
>>>>        exit 1
>>>> fi
>>>>
>>>> # The first argument is the path to python-config.py, ignore it.
>>>>
>>>> case "$2" in
>>>> --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;;
>>>> --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm
>>>> -lpython2.6" ;;
>>>> --exec-prefix) echo "/usr" ;;
>>>> *) echo "Bad arg $2.  Blech!" >&2 ; exit 1 ;;
>>>> esac
>>>>
>>>> exit 0
>>>> bash$ ./configure --with-python=$HOME/my-python-for-config [...]
>>>> [...]
>>>>
>>>>
>>>> Note that --exec-prefix is the runtime location of python.
>>>> GCC uses this to tell libpython where to find its support files.
>>>> [grep for Py_SetProgramName in gdb/python/python.c]
>>>
>>> OK, I tried your script in a couple of variations. It gets rid of the
>>> traceback I had before, but still doesn't help the PYTHONPATH problem.
>>> My directory structure is as follows (this is not in root, "/" is just
>>> my main build directory, there are several levels below it):
>>>
>>> /gdb <-- gdb build dir
>>> /gdb/gdb <-- where the python configuration is done
>>> /python <-- temporary install dir for python files for build,
>>> extracted from the official Windows installer, also location of
>>> libpython2.7.a import library
>>> /python/include/python27 <-- python headers, found by gdb in both cases
>>> /mingw64 <-- toolchain sysroot prefix
>>> /mingw64/bin <-- install location of gdb and python27.dll
>>> /mingw64/lib/python27 <-- install location of all python scripts
>>>
>>> First variant:
>>> --includes) echo "-I../../python/include" ;;
>>> --ldflags) echo "-L../../python -lpython2.7" ;;
>>> --exec-prefix) echo "../../mingw64/lib/python27" ;;
>>>
>>> Here exec-prefix would be the relative path from where "configure"
>>> does its magic to the final location of the scripts on the build
>>> system.
>>>
>>> Second variant:
>>> --includes) echo "-I../../python/include" ;;
>>> --ldflags) echo "-L../../python -lpython2.7" ;;
>>> --exec-prefix) echo "../lib/python27" ;;
>>>
>>> I thought the second points gdb to the installed location of the
>>> python scripts, but it still needed PYTHONPATH for that. I used
>>> relative paths in an attempt to "do the right thing", hackwise...
>>> Remember that the whole "/mingw64" directory gets zipped and moved,
>>> then it gets extracted to a random location in the Windows filesystem.
>>>
>>> Thanks for the help so far, I think we might just get this worked out together.
>>>
>>> Ruben
>>
>> Wow, I think I have a partial solution. Delving into the Python docs,
>> for example here:
>> http://docs.python.org/using/windows.html#finding-modules, you can see
>> that PYTHONPATH is used first, then the Windows registry, then
>> PYTHONHOME, then some default relative paths. I placed the python
>> scripts all in the directory structure like so:
>>
>> /bin/gdb
>> /bin/Lib/<python scripts>
>> /bin/python27.dll
>>
>> This works, even without any manual PYTHONPATH intervention. Problem
>> is though, that as soon as someone has a PYTHONPATH environment
>> variable from a (incompatible) Python installation (think different
>> bitness or version 3.x instead of 2.7.1), I cannot predict what will
>> go wrong. This problem originates in Python's way of filling in the
>> search path (sys.path). A true solution in the GDB case to prevent
>> this collision of an incompatible PYTHONPATH would be that GDB sets an
>> internal PYTHONPATH as directed by configure, uses that to load its
>> Python internals, and allows the GDB child processes (apps being
>> debugged) to use the environment PYTHONPATH. For now, I have a
>> functional installation, but it will break as soon as someone installs
>> Python on their system.
>>
>> The part your script plays in this (if I haven't misunderstood the
>> intention/effects of the "--exec-prefix" part) is allowing a uniform
>> way of directing the configure script to the right directories for
>> includes/libs.
>>
>> Ruben
>
> I am sorry for the repeated messages that no one cares about, but I
> may have discovered GDB in its current form already allows what I
> want: I tried to figure out what exact paths the snake in gdb was
> using to search for its modules, and came up with this:
> (gdb) python import sys
> (gdb) python print sys.path
> ['m:\\development\\mingw64\\share\\gdb/python',
> 'M:\\Development\\mingw64\\bin\\python27.zip',
> 'M:\\Development\\mingw64\\bin\\DLLs',
> 'M:\\Development\\mingw64\\bin\\lib',
> 'M:\\Development\\mingw64\\bin\\lib\\plat-win',
> 'M:\\Development\\mingw64\\bin\\lib\\lib-tk',
> 'M:\\Development\\mingw64\\bin',
> 'M:\\Development\\mingw64\\bin\\lib\\site-packages']
>
> This means that every python command within gdb searches
> <sysroot>/share/gdb/python FIRST (even before an environment's
> PYTHONPATH), alleviating any concerns or problems I or anyone would
> have with another python installation, as this apparently built-in
> path comes up first. All I, or anyone interested in doing this kind of
> thing, have to do is copy all the python scripts from the Windows
> installation's Lib directory to the <sysroot>/share/gdb/python
> directory.
>
> I don't know where this path comes from, but it is quite handy, and
> makes this whole discussion  moot for Python people. Only "issue" that
> I'll have to work around is the --with-python-includes and
> --with-python-libs that are missing, using either manual
> CFLAGS/LDFLAGS or a variant of your script.
>
> Thanks for all the help and motivation to keep trying :)
>
> Ruben

Scratch that, I had  lingering install of Python laying around.
Without PYTHONPATH, no gdb (module site not found). Why not make gdb
search the ../share/gdb/python path first, and then complain?

Ruben

>
>>
>>>
>>>>
>>>>>>> I understand the need for this, but as gdb requires Python 2, and users
>>>>>>> of my toolchain may have installed Python 3 or a 32-bit version python
>>>>>>> they want to use from the same environment (without changing their own
>>>>>>> PYTHONPATH), there is no way to run python-enabled gdb.
>>>>>>> [...]
>>>>>>
>>>>>> Yeah.
>>>>>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have
>>>>>> gdb use that instead of PYTHONPATH if it exists, but there's been
>>>>>> resistance to it.
>>>>>> I think(!) what would happen is that gdb would set $PYTHONPATH to the
>>>>>> value of $GDB_PYTHONPATH.
>>>>>> [Inferiors started by gdb should still get the original value of
>>>>>> PYTHONPATH though.]
>>>>>
>>>>> That way would be almost ideal, but a hardcoded *relative* path to the
>>>>> python scripts (that is standardized within gdb) wouldn't hurt.
>>>>
>>>> See above re: --exec-prefix.
>>>>
>>>>> An
>>>>> extra environment variable would require a lot of explaining for
>>>>> Windows, and is not "plug-and-play", like the rest of a sysrooted
>>>>> toolchain is supposed to be like. I think this should work on all
>>>>> setups:
>>>>>
>>>>> 1. Check hardcoded path; my suggestion would be "<gdb
>>>>> executable>/../lib/python27"
>>>>> 2. If this fails to find the necessary files/scripts, find it like you
>>>>> described above in Linux, without PYTHONPATH set.
>>>>> 3. Check PYTHONPATH.
>>>>
>>>> The problem being solved by the proposed GDB_PYTHONPATH is "What if
>>>> the user has PYTHONPATH set and it points to an incompatible version
>>>> of python?".
>>>> Leaving such a value for PYTHONPATH set while gdb's python is running
>>>> feels wrong (and IIRC has caused some problems).
>>>>
>>>> The problem of telling python where to find itself is already solved
>>>> (or at least is intended to be solved) with gdb's calling
>>>> Py_SetProgramName with a value derived from the python-provided
>>>> --exec-prefix.
>>>>
>>>>> I would think only number one would change, and perhaps be only
>>>>> enabled with a special configure option. Nothing else would have to
>>>>> change, and Windows users would rejoice :)
>>>>> Again, this is only my suggestion, if there are problems with it in
>>>>> way I haven't thought of, please say so, and we can come up with
>>>>> another solution.
>>>>
>>>
>>
>