* Python enabled gdb on Windows and relocation [not found] <BANLkTikrK8DJOEpACA_uUVdZx25P5pc1tA@mail.gmail.com> @ 2011-05-12 16:20 ` Ruben Van Boxem 2011-05-14 6:39 ` Doug Evans 0 siblings, 1 reply; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-12 16:20 UTC (permalink / raw) To: gdb, python-list (now in plain-text as required by gdb mailing list) Hi, I am currently trying to integrate Python support into my toolchain build (including GDB of course). It is a sysrooted binutils+GCC+GDB+mingw-w64 toolchain. I currently have the basic setup working: I can link gdb with my manually generated import lib to the python dll from the official Windows install. If there is anything I am missing or a very easy solution to the problems decsribed below, please just say so. I am only suggesting what I would like to happen. Now on to the problems I'd like to discuss: 1. gdb.exe won't start without me having set PYTHONPATH manually. I understand the need for this, but as gdb requires Python 2, and users of my toolchain may have installed Python 3 or a 32-bit version python they want to use from the same environment (without changing their own PYTHONPATH), there is no way to run python-enabled gdb. What I suggest is perhaps a Windows only change: a kind of "sysroot"ed gdb+python, so that the python used by gdb is truly built in. I cannot require everyone using my toolchain to install the correct python version for obvious reasons (Windows != Linux in this regard). However way this is tackled, the gdb I want should look relative to it's current (relocatable) directory for Python modules/*;py files in a well-defined directory at build time. That would allow me to distribute gdb in a very clear way: <sysroot> <sysroot>/bin <sysroot>/bin/python27.dll <sysroot>/bin/gdb.exe <sysroot>/lib <sysroot>/lib/python27 <sysroot>/lib/python27/<all *.py files from the official windows install> Currently, this scheme only works if I manually set the PYTHONPATH environment variable to <root>/lib/python27. This global environment variable could conflict with existing Python installations, that may have another bitness (I have 64- and 32-bit toolchains). I think you can see there are problems enough with the current set up. I also read this old discussion: http://comments.gmane.org/gmane.comp.gdb.patches/62811 that might imply that the PYTHONPATH check is something built-in to the python dll. Can anyone shed some light on this? 2. With PYTHONPATH set as a temporary workaround, gdb starts, but spits out a traceback: Traceback (most recent call last): File "<string>", line 35, in <module> File "m:\development\mingw64\share\gdb/python/gdb/__init__.py", line 18, in <module> gdb.command.pretty_printers.register_pretty_printer_commands() File "m:\development\mingw64\share\gdb/python/gdb\command\pretty_printers.py", line 368, in register_pretty_printer_commands InfoPrettyPrinter() File "m:\development\mingw64\share\gdb/python/gdb\command\pretty_printers.py", line 100, in __init__ gdb.COMMAND_DATA) RuntimeError: Could not find command prefix info. This is a minor problem I think, as "python import time" "python print time.clock()" works as expected. What is wrong? Thanks very much! Ruben PS: I've sent this to both the gdb and python mailing lists, as these issues have a large overlap to both projects. PS2: Please reply-to-all as I do not want daily mails from both mailing lists on every issue brought up there. My apologies. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-12 16:20 ` Python enabled gdb on Windows and relocation Ruben Van Boxem @ 2011-05-14 6:39 ` Doug Evans 2011-05-14 9:09 ` Ruben Van Boxem 0 siblings, 1 reply; 16+ messages in thread From: Doug Evans @ 2011-05-14 6:39 UTC (permalink / raw) To: vanboxem.ruben; +Cc: gdb, python-list On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem <vanboxem.ruben@gmail.com> wrote: > (now in plain-text as required by gdb mailing list) > > Hi, > > I am currently trying to integrate Python support into my toolchain > build (including GDB of course). It is a sysrooted > binutils+GCC+GDB+mingw-w64 toolchain. > > I currently have the basic setup working: I can link gdb with my > manually generated import lib to the python dll from the official > Windows install. If there is anything I am missing or a very easy > solution to the problems decsribed below, please just say so. I am > only suggesting what I would like to happen. > > Now on to the problems I'd like to discuss: > > 1. gdb.exe won't start without me having set PYTHONPATH manually. In a properly configured/built gdb on linux this isn't necessary, even if python is installed in some random place. I'm not sure about windows though. Did you specify --with-python when you configured gdb, and if so did you specify a value? e.g., --with-python=SOME_VALUE > I understand the need for this, but as gdb requires Python 2, and users > of my toolchain may have installed Python 3 or a 32-bit version python > they want to use from the same environment (without changing their own > PYTHONPATH), there is no way to run python-enabled gdb. > [...] Yeah. There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have gdb use that instead of PYTHONPATH if it exists, but there's been resistance to it. I think(!) what would happen is that gdb would set $PYTHONPATH to the value of $GDB_PYTHONPATH. [Inferiors started by gdb should still get the original value of PYTHONPATH though.] > 2. With PYTHONPATH set as a temporary workaround, gdb starts, but > spits out a traceback: > Traceback (most recent call last): > File "<string>", line 35, in <module> > File "m:\development\mingw64\share\gdb/python/gdb/__init__.py", line > 18, in <module> > gdb.command.pretty_printers.register_pretty_printer_commands() > File "m:\development\mingw64\share\gdb/python/gdb\command\pretty_printers.py", > line 368, in register_pretty_printer_commands > InfoPrettyPrinter() > File "m:\development\mingw64\share\gdb/python/gdb\command\pretty_printers.py", > line 100, in __init__ > gdb.COMMAND_DATA) > RuntimeError: Could not find command prefix info. > > This is a minor problem I think, as "python import time" "python print > time.clock()" works as expected. What is wrong? I'm not sure. The error message is complaining that the "info" command prefix doesn't exist. I don't see how that can happen as python is initialized long after the info command is created. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 6:39 ` Doug Evans @ 2011-05-14 9:09 ` Ruben Van Boxem 2011-05-14 9:30 ` Eli Zaretskii 2011-05-14 18:31 ` Doug Evans 0 siblings, 2 replies; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-14 9:09 UTC (permalink / raw) To: Doug Evans; +Cc: gdb, python-list 2011/5/14 Doug Evans <dje@google.com>: > On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem > <vanboxem.ruben@gmail.com> wrote: >> (now in plain-text as required by gdb mailing list) >> >> Hi, >> >> I am currently trying to integrate Python support into my toolchain >> build (including GDB of course). It is a sysrooted >> binutils+GCC+GDB+mingw-w64 toolchain. >> >> I currently have the basic setup working: I can link gdb with my >> manually generated import lib to the python dll from the official >> Windows install. If there is anything I am missing or a very easy >> solution to the problems decsribed below, please just say so. I am >> only suggesting what I would like to happen. >> >> Now on to the problems I'd like to discuss: >> >> 1. gdb.exe won't start without me having set PYTHONPATH manually. > > In a properly configured/built gdb on linux this isn't necessary, even > if python is installed in some random place. > I'm not sure about windows though. > Did you specify --with-python when you configured gdb, and if so did > you specify a value? > e.g., --with-python=SOME_VALUE I was cross-compiling a mingw toolchain+gdb from Linux, so I used --with-python without a value (because gdb configure tries to find the Python executabe), and I added -I"/path/to/python/includes" to CFLAGS and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it should. This is hacky though, and gdb configure should provide --with-python-libs and --with-python-include to make it more streamlined with any other build prerequisite (like gmp/mpfr/mpc/cloog/ppl in GCC for example). > >> I understand the need for this, but as gdb requires Python 2, and users >> of my toolchain may have installed Python 3 or a 32-bit version python >> they want to use from the same environment (without changing their own >> PYTHONPATH), there is no way to run python-enabled gdb. >> [...] > > Yeah. > There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have > gdb use that instead of PYTHONPATH if it exists, but there's been > resistance to it. > I think(!) what would happen is that gdb would set $PYTHONPATH to the > value of $GDB_PYTHONPATH. > [Inferiors started by gdb should still get the original value of > PYTHONPATH though.] That way would be almost ideal, but a hardcoded *relative* path to the python scripts (that is standardized within gdb) wouldn't hurt. An extra environment variable would require a lot of explaining for Windows, and is not "plug-and-play", like the rest of a sysrooted toolchain is supposed to be like. I think this should work on all setups: 1. Check hardcoded path; my suggestion would be "<gdb executable>/../lib/python27" 2. If this fails to find the necessary files/scripts, find it like you described above in Linux, without PYTHONPATH set. 3. Check PYTHONPATH. I would think only number one would change, and perhaps be only enabled with a special configure option. Nothing else would have to change, and Windows users would rejoice :) Again, this is only my suggestion, if there are problems with it in way I haven't thought of, please say so, and we can come up with another solution. > >> 2. With PYTHONPATH set as a temporary workaround, gdb starts, but >> spits out a traceback: >> Traceback (most recent call last): >> File "<string>", line 35, in <module> >> File "m:\development\mingw64\share\gdb/python/gdb/__init__.py", line >> 18, in <module> >> gdb.command.pretty_printers.register_pretty_printer_commands() >> File "m:\development\mingw64\share\gdb/python/gdb\command\pretty_printers.py", >> line 368, in register_pretty_printer_commands >> InfoPrettyPrinter() >> File "m:\development\mingw64\share\gdb/python/gdb\command\pretty_printers.py", >> line 100, in __init__ >> gdb.COMMAND_DATA) >> RuntimeError: Could not find command prefix info. >> >> This is a minor problem I think, as "python import time" "python print >> time.clock()" works as expected. What is wrong? > > I'm not sure. > The error message is complaining that the "info" command prefix doesn't exist. > I don't see how that can happen as python is initialized long after > the info command is created. > Thanks for the prompt response. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 9:09 ` Ruben Van Boxem @ 2011-05-14 9:30 ` Eli Zaretskii 2011-05-14 10:10 ` Ruben Van Boxem 2011-05-14 18:38 ` Doug Evans 2011-05-14 18:31 ` Doug Evans 1 sibling, 2 replies; 16+ messages in thread From: Eli Zaretskii @ 2011-05-14 9:30 UTC (permalink / raw) To: vanboxem.ruben; +Cc: dje, gdb, python-list > Date: Sat, 14 May 2011 11:09:13 +0200 > From: Ruben Van Boxem <vanboxem.ruben@gmail.com> > Cc: gdb@sourceware.org, python-list@python.org > > 1. Check hardcoded path; my suggestion would be "<gdb executable>/../lib/python27" > 2. If this fails to find the necessary files/scripts, find it like you > described above in Linux, without PYTHONPATH set. > 3. Check PYTHONPATH. > > I would think only number one would change, and perhaps be only > enabled with a special configure option. Nothing else would have to > change, and Windows users would rejoice :) The problem, I think, is that it's not so easy on Unix to get the place where the GDB executable leaves. There isn't a system call to do that (similar to what Windows gives you). So I think on Posix platforms, number 2 would be used most of the time. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 9:30 ` Eli Zaretskii @ 2011-05-14 10:10 ` Ruben Van Boxem 2011-05-14 18:38 ` Doug Evans 1 sibling, 0 replies; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-14 10:10 UTC (permalink / raw) To: Eli Zaretskii; +Cc: dje, gdb, python-list 2011/5/14 Eli Zaretskii <eliz@gnu.org>: >> Date: Sat, 14 May 2011 11:09:13 +0200 >> From: Ruben Van Boxem <vanboxem.ruben@gmail.com> >> Cc: gdb@sourceware.org, python-list@python.org >> >> 1. Check hardcoded path; my suggestion would be "<gdb executable>/../lib/python27" >> 2. If this fails to find the necessary files/scripts, find it like you >> described above in Linux, without PYTHONPATH set. >> 3. Check PYTHONPATH. >> >> I would think only number one would change, and perhaps be only >> enabled with a special configure option. Nothing else would have to >> change, and Windows users would rejoice :) > > The problem, I think, is that it's not so easy on Unix to get the > place where the GDB executable leaves. There isn't a system call to > do that (similar to what Windows gives you). > > So I think on Posix platforms, number 2 would be used most of the > time. > I understand. So better/feasible would be: Windows: Check <gdb executable>/../lib/python27 or any configure-time specified relative path to gdb executable? If that fails, revert to current behavoir. POSIX (so pretty much everythin else): Keep current behavior. Which would come down to a configure option+ a small block of #if _WIN32 code in the Python part of the code. Thanks, Ruben ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 9:30 ` Eli Zaretskii 2011-05-14 10:10 ` Ruben Van Boxem @ 2011-05-14 18:38 ` Doug Evans 1 sibling, 0 replies; 16+ messages in thread From: Doug Evans @ 2011-05-14 18:38 UTC (permalink / raw) To: Eli Zaretskii; +Cc: vanboxem.ruben, gdb, python-list On Sat, May 14, 2011 at 2:29 AM, Eli Zaretskii <eliz@gnu.org> wrote: >> Date: Sat, 14 May 2011 11:09:13 +0200 >> From: Ruben Van Boxem <vanboxem.ruben@gmail.com> >> Cc: gdb@sourceware.org, python-list@python.org >> >> 1. Check hardcoded path; my suggestion would be "<gdb executable>/../lib/python27" >> 2. If this fails to find the necessary files/scripts, find it like you >> described above in Linux, without PYTHONPATH set. >> 3. Check PYTHONPATH. >> >> I would think only number one would change, and perhaps be only >> enabled with a special configure option. Nothing else would have to >> change, and Windows users would rejoice :) > > The problem, I think, is that it's not so easy on Unix to get the > place where the GDB executable leaves. There isn't a system call to > do that (similar to what Windows gives you). > > So I think on Posix platforms, number 2 would be used most of the > time. For reference sake, gdb is "relocatable". [meaning, if you take a gdb installation and move it, it should continue to work fine] And if gdb's python lives inside the gdb tree, that too should continue to work fine if moved with gdb (the value to pass to Py_SetProgramName is appropriately (re-)computed when gdb is run). [For completeness sake, IIRC the calculation of a path being "relocatable" isn't bulletproof, but it works in practice.] It's not impossible for gdb to find where it lives, but you're right it can be moderately difficult (basically, if argv[0] isn't an absolute path then scan $PATH for it). ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 9:09 ` Ruben Van Boxem 2011-05-14 9:30 ` Eli Zaretskii @ 2011-05-14 18:31 ` Doug Evans 2011-05-14 18:40 ` Doug Evans 2011-05-15 12:30 ` Ruben Van Boxem 1 sibling, 2 replies; 16+ messages in thread From: Doug Evans @ 2011-05-14 18:31 UTC (permalink / raw) To: vanboxem.ruben; +Cc: gdb, python-list On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem <vanboxem.ruben@gmail.com> wrote: > 2011/5/14 Doug Evans <dje@google.com>: >> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >> <vanboxem.ruben@gmail.com> wrote: >>> (now in plain-text as required by gdb mailing list) >>> >>> Hi, >>> >>> I am currently trying to integrate Python support into my toolchain >>> build (including GDB of course). It is a sysrooted >>> binutils+GCC+GDB+mingw-w64 toolchain. >>> >>> I currently have the basic setup working: I can link gdb with my >>> manually generated import lib to the python dll from the official >>> Windows install. If there is anything I am missing or a very easy >>> solution to the problems decsribed below, please just say so. I am >>> only suggesting what I would like to happen. >>> >>> Now on to the problems I'd like to discuss: >>> >>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >> >> In a properly configured/built gdb on linux this isn't necessary, even >> if python is installed in some random place. >> I'm not sure about windows though. >> Did you specify --with-python when you configured gdb, and if so did >> you specify a value? >> e.g., --with-python=SOME_VALUE > > I was cross-compiling a mingw toolchain+gdb from Linux, so I used > --with-python without a value (because gdb configure tries to find the > Python executabe), and I added -I"/path/to/python/includes" to CFLAGS > and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it > should. This is hacky though, and gdb configure should provide > --with-python-libs and --with-python-include to make it more > streamlined with any other build prerequisite (like > gmp/mpfr/mpc/cloog/ppl in GCC for example). Ah. Cross-compiling gdb with python is in need of improvement. Alas python hasn't been designed with cross-compilation in mind (e.g. build on linux, run on windows). AIUI, the way to get the parameters required for compiling with libpython is to get them from python's "distutils": kinda hard to do in a cross-compile. Done correctly there's no need to run python. I haven't done anything more to support python in gdb's configure.ac because it's not clear to me what the right thing to do is: distutils provides more than just --libs and --includes (btw, we don't use --libs though, we use --ldflags which includes all of: the directory in which to find libpython, the -l for libpython, and the -l's for all the other libraries python needs). [Which isn't to say that someone else isn't free to tackle this.] In the meantime, what I've been doing is a hack: write a script that responds to: --includes --ldflags --exec-prefix and pass that as --with-python. E.g. bash$ cat $HOME/my-python-for-config #! /bin/sh if [ $# -ne 2 ] then echo "Bad # args. Blech!" >&2 exit 1 fi # The first argument is the path to python-config.py, ignore it. case "$2" in --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm -lpython2.6" ;; --exec-prefix) echo "/usr" ;; *) echo "Bad arg $2. Blech!" >&2 ; exit 1 ;; esac exit 0 bash$ ./configure --with-python=$HOME/my-python-for-config [...] [...] Note that --exec-prefix is the runtime location of python. GCC uses this to tell libpython where to find its support files. [grep for Py_SetProgramName in gdb/python/python.c] >>> I understand the need for this, but as gdb requires Python 2, and users >>> of my toolchain may have installed Python 3 or a 32-bit version python >>> they want to use from the same environment (without changing their own >>> PYTHONPATH), there is no way to run python-enabled gdb. >>> [...] >> >> Yeah. >> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >> gdb use that instead of PYTHONPATH if it exists, but there's been >> resistance to it. >> I think(!) what would happen is that gdb would set $PYTHONPATH to the >> value of $GDB_PYTHONPATH. >> [Inferiors started by gdb should still get the original value of >> PYTHONPATH though.] > > That way would be almost ideal, but a hardcoded *relative* path to the > python scripts (that is standardized within gdb) wouldn't hurt. See above re: --exec-prefix. > An > extra environment variable would require a lot of explaining for > Windows, and is not "plug-and-play", like the rest of a sysrooted > toolchain is supposed to be like. I think this should work on all > setups: > > 1. Check hardcoded path; my suggestion would be "<gdb > executable>/../lib/python27" > 2. If this fails to find the necessary files/scripts, find it like you > described above in Linux, without PYTHONPATH set. > 3. Check PYTHONPATH. The problem being solved by the proposed GDB_PYTHONPATH is "What if the user has PYTHONPATH set and it points to an incompatible version of python?". Leaving such a value for PYTHONPATH set while gdb's python is running feels wrong (and IIRC has caused some problems). The problem of telling python where to find itself is already solved (or at least is intended to be solved) with gdb's calling Py_SetProgramName with a value derived from the python-provided --exec-prefix. > I would think only number one would change, and perhaps be only > enabled with a special configure option. Nothing else would have to > change, and Windows users would rejoice :) > Again, this is only my suggestion, if there are problems with it in > way I haven't thought of, please say so, and we can come up with > another solution. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 18:31 ` Doug Evans @ 2011-05-14 18:40 ` Doug Evans 2011-05-15 12:30 ` Ruben Van Boxem 1 sibling, 0 replies; 16+ messages in thread From: Doug Evans @ 2011-05-14 18:40 UTC (permalink / raw) To: vanboxem.ruben; +Cc: gdb, python-list On Sat, May 14, 2011 at 11:30 AM, Doug Evans <dje@google.com> wrote: > Note that --exec-prefix is the runtime location of python. > GCC uses this to tell libpython where to find its support files. > [grep for Py_SetProgramName in gdb/python/python.c] Oops. s/GCC/GDB/ ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-14 18:31 ` Doug Evans 2011-05-14 18:40 ` Doug Evans @ 2011-05-15 12:30 ` Ruben Van Boxem 2011-05-15 13:27 ` Ruben Van Boxem 1 sibling, 1 reply; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-15 12:30 UTC (permalink / raw) To: Doug Evans; +Cc: gdb, python-list 2011/5/14 Doug Evans <dje@google.com>: > On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem > <vanboxem.ruben@gmail.com> wrote: >> 2011/5/14 Doug Evans <dje@google.com>: >>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >>> <vanboxem.ruben@gmail.com> wrote: >>>> (now in plain-text as required by gdb mailing list) >>>> >>>> Hi, >>>> >>>> I am currently trying to integrate Python support into my toolchain >>>> build (including GDB of course). It is a sysrooted >>>> binutils+GCC+GDB+mingw-w64 toolchain. >>>> >>>> I currently have the basic setup working: I can link gdb with my >>>> manually generated import lib to the python dll from the official >>>> Windows install. If there is anything I am missing or a very easy >>>> solution to the problems decsribed below, please just say so. I am >>>> only suggesting what I would like to happen. >>>> >>>> Now on to the problems I'd like to discuss: >>>> >>>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >>> >>> In a properly configured/built gdb on linux this isn't necessary, even >>> if python is installed in some random place. >>> I'm not sure about windows though. >>> Did you specify --with-python when you configured gdb, and if so did >>> you specify a value? >>> e.g., --with-python=SOME_VALUE >> >> I was cross-compiling a mingw toolchain+gdb from Linux, so I used >> --with-python without a value (because gdb configure tries to find the >> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS >> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it >> should. This is hacky though, and gdb configure should provide >> --with-python-libs and --with-python-include to make it more >> streamlined with any other build prerequisite (like >> gmp/mpfr/mpc/cloog/ppl in GCC for example). > > Ah. > Cross-compiling gdb with python is in need of improvement. > Alas python hasn't been designed with cross-compilation in mind (e.g. > build on linux, run on windows). > AIUI, the way to get the parameters required for compiling with > libpython is to get them from python's "distutils": kinda hard to do > in a cross-compile. Done correctly there's no need to run python. > > I haven't done anything more to support python in gdb's configure.ac > because it's not clear to me what the right thing to do is: distutils > provides more than just --libs and --includes (btw, we don't use > --libs though, we use --ldflags which includes all of: the directory > in which to find libpython, the -l for libpython, and the -l's for all > the other libraries python needs). [Which isn't to say that someone > else isn't free to tackle this.] > > In the meantime, what I've been doing is a hack: write a script that > responds to: > --includes > --ldflags > --exec-prefix > and pass that as --with-python. > > E.g. > bash$ cat $HOME/my-python-for-config > #! /bin/sh > > if [ $# -ne 2 ] > then > echo "Bad # args. Blech!" >&2 > exit 1 > fi > > # The first argument is the path to python-config.py, ignore it. > > case "$2" in > --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; > --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm > -lpython2.6" ;; > --exec-prefix) echo "/usr" ;; > *) echo "Bad arg $2. Blech!" >&2 ; exit 1 ;; > esac > > exit 0 > bash$ ./configure --with-python=$HOME/my-python-for-config [...] > [...] > > > Note that --exec-prefix is the runtime location of python. > GCC uses this to tell libpython where to find its support files. > [grep for Py_SetProgramName in gdb/python/python.c] OK, I tried your script in a couple of variations. It gets rid of the traceback I had before, but still doesn't help the PYTHONPATH problem. My directory structure is as follows (this is not in root, "/" is just my main build directory, there are several levels below it): /gdb <-- gdb build dir /gdb/gdb <-- where the python configuration is done /python <-- temporary install dir for python files for build, extracted from the official Windows installer, also location of libpython2.7.a import library /python/include/python27 <-- python headers, found by gdb in both cases /mingw64 <-- toolchain sysroot prefix /mingw64/bin <-- install location of gdb and python27.dll /mingw64/lib/python27 <-- install location of all python scripts First variant: --includes) echo "-I../../python/include" ;; --ldflags) echo "-L../../python -lpython2.7" ;; --exec-prefix) echo "../../mingw64/lib/python27" ;; Here exec-prefix would be the relative path from where "configure" does its magic to the final location of the scripts on the build system. Second variant: --includes) echo "-I../../python/include" ;; --ldflags) echo "-L../../python -lpython2.7" ;; --exec-prefix) echo "../lib/python27" ;; I thought the second points gdb to the installed location of the python scripts, but it still needed PYTHONPATH for that. I used relative paths in an attempt to "do the right thing", hackwise... Remember that the whole "/mingw64" directory gets zipped and moved, then it gets extracted to a random location in the Windows filesystem. Thanks for the help so far, I think we might just get this worked out together. Ruben > >>>> I understand the need for this, but as gdb requires Python 2, and users >>>> of my toolchain may have installed Python 3 or a 32-bit version python >>>> they want to use from the same environment (without changing their own >>>> PYTHONPATH), there is no way to run python-enabled gdb. >>>> [...] >>> >>> Yeah. >>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >>> gdb use that instead of PYTHONPATH if it exists, but there's been >>> resistance to it. >>> I think(!) what would happen is that gdb would set $PYTHONPATH to the >>> value of $GDB_PYTHONPATH. >>> [Inferiors started by gdb should still get the original value of >>> PYTHONPATH though.] >> >> That way would be almost ideal, but a hardcoded *relative* path to the >> python scripts (that is standardized within gdb) wouldn't hurt. > > See above re: --exec-prefix. > >> An >> extra environment variable would require a lot of explaining for >> Windows, and is not "plug-and-play", like the rest of a sysrooted >> toolchain is supposed to be like. I think this should work on all >> setups: >> >> 1. Check hardcoded path; my suggestion would be "<gdb >> executable>/../lib/python27" >> 2. If this fails to find the necessary files/scripts, find it like you >> described above in Linux, without PYTHONPATH set. >> 3. Check PYTHONPATH. > > The problem being solved by the proposed GDB_PYTHONPATH is "What if > the user has PYTHONPATH set and it points to an incompatible version > of python?". > Leaving such a value for PYTHONPATH set while gdb's python is running > feels wrong (and IIRC has caused some problems). > > The problem of telling python where to find itself is already solved > (or at least is intended to be solved) with gdb's calling > Py_SetProgramName with a value derived from the python-provided > --exec-prefix. > >> I would think only number one would change, and perhaps be only >> enabled with a special configure option. Nothing else would have to >> change, and Windows users would rejoice :) >> Again, this is only my suggestion, if there are problems with it in >> way I haven't thought of, please say so, and we can come up with >> another solution. > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-15 12:30 ` Ruben Van Boxem @ 2011-05-15 13:27 ` Ruben Van Boxem 2011-05-15 16:12 ` Ruben Van Boxem 2011-05-17 20:36 ` Doug Evans 0 siblings, 2 replies; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-15 13:27 UTC (permalink / raw) To: Doug Evans; +Cc: gdb, python-list 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>: > 2011/5/14 Doug Evans <dje@google.com>: >> On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem >> <vanboxem.ruben@gmail.com> wrote: >>> 2011/5/14 Doug Evans <dje@google.com>: >>>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >>>> <vanboxem.ruben@gmail.com> wrote: >>>>> (now in plain-text as required by gdb mailing list) >>>>> >>>>> Hi, >>>>> >>>>> I am currently trying to integrate Python support into my toolchain >>>>> build (including GDB of course). It is a sysrooted >>>>> binutils+GCC+GDB+mingw-w64 toolchain. >>>>> >>>>> I currently have the basic setup working: I can link gdb with my >>>>> manually generated import lib to the python dll from the official >>>>> Windows install. If there is anything I am missing or a very easy >>>>> solution to the problems decsribed below, please just say so. I am >>>>> only suggesting what I would like to happen. >>>>> >>>>> Now on to the problems I'd like to discuss: >>>>> >>>>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >>>> >>>> In a properly configured/built gdb on linux this isn't necessary, even >>>> if python is installed in some random place. >>>> I'm not sure about windows though. >>>> Did you specify --with-python when you configured gdb, and if so did >>>> you specify a value? >>>> e.g., --with-python=SOME_VALUE >>> >>> I was cross-compiling a mingw toolchain+gdb from Linux, so I used >>> --with-python without a value (because gdb configure tries to find the >>> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS >>> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it >>> should. This is hacky though, and gdb configure should provide >>> --with-python-libs and --with-python-include to make it more >>> streamlined with any other build prerequisite (like >>> gmp/mpfr/mpc/cloog/ppl in GCC for example). >> >> Ah. >> Cross-compiling gdb with python is in need of improvement. >> Alas python hasn't been designed with cross-compilation in mind (e.g. >> build on linux, run on windows). >> AIUI, the way to get the parameters required for compiling with >> libpython is to get them from python's "distutils": kinda hard to do >> in a cross-compile. Done correctly there's no need to run python. >> >> I haven't done anything more to support python in gdb's configure.ac >> because it's not clear to me what the right thing to do is: distutils >> provides more than just --libs and --includes (btw, we don't use >> --libs though, we use --ldflags which includes all of: the directory >> in which to find libpython, the -l for libpython, and the -l's for all >> the other libraries python needs). [Which isn't to say that someone >> else isn't free to tackle this.] >> >> In the meantime, what I've been doing is a hack: write a script that >> responds to: >> --includes >> --ldflags >> --exec-prefix >> and pass that as --with-python. >> >> E.g. >> bash$ cat $HOME/my-python-for-config >> #! /bin/sh >> >> if [ $# -ne 2 ] >> then >> echo "Bad # args. Blech!" >&2 >> exit 1 >> fi >> >> # The first argument is the path to python-config.py, ignore it. >> >> case "$2" in >> --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; >> --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm >> -lpython2.6" ;; >> --exec-prefix) echo "/usr" ;; >> *) echo "Bad arg $2. Blech!" >&2 ; exit 1 ;; >> esac >> >> exit 0 >> bash$ ./configure --with-python=$HOME/my-python-for-config [...] >> [...] >> >> >> Note that --exec-prefix is the runtime location of python. >> GCC uses this to tell libpython where to find its support files. >> [grep for Py_SetProgramName in gdb/python/python.c] > > OK, I tried your script in a couple of variations. It gets rid of the > traceback I had before, but still doesn't help the PYTHONPATH problem. > My directory structure is as follows (this is not in root, "/" is just > my main build directory, there are several levels below it): > > /gdb <-- gdb build dir > /gdb/gdb <-- where the python configuration is done > /python <-- temporary install dir for python files for build, > extracted from the official Windows installer, also location of > libpython2.7.a import library > /python/include/python27 <-- python headers, found by gdb in both cases > /mingw64 <-- toolchain sysroot prefix > /mingw64/bin <-- install location of gdb and python27.dll > /mingw64/lib/python27 <-- install location of all python scripts > > First variant: > --includes) echo "-I../../python/include" ;; > --ldflags) echo "-L../../python -lpython2.7" ;; > --exec-prefix) echo "../../mingw64/lib/python27" ;; > > Here exec-prefix would be the relative path from where "configure" > does its magic to the final location of the scripts on the build > system. > > Second variant: > --includes) echo "-I../../python/include" ;; > --ldflags) echo "-L../../python -lpython2.7" ;; > --exec-prefix) echo "../lib/python27" ;; > > I thought the second points gdb to the installed location of the > python scripts, but it still needed PYTHONPATH for that. I used > relative paths in an attempt to "do the right thing", hackwise... > Remember that the whole "/mingw64" directory gets zipped and moved, > then it gets extracted to a random location in the Windows filesystem. > > Thanks for the help so far, I think we might just get this worked out together. > > Ruben Wow, I think I have a partial solution. Delving into the Python docs, for example here: http://docs.python.org/using/windows.html#finding-modules, you can see that PYTHONPATH is used first, then the Windows registry, then PYTHONHOME, then some default relative paths. I placed the python scripts all in the directory structure like so: /bin/gdb /bin/Lib/<python scripts> /bin/python27.dll This works, even without any manual PYTHONPATH intervention. Problem is though, that as soon as someone has a PYTHONPATH environment variable from a (incompatible) Python installation (think different bitness or version 3.x instead of 2.7.1), I cannot predict what will go wrong. This problem originates in Python's way of filling in the search path (sys.path). A true solution in the GDB case to prevent this collision of an incompatible PYTHONPATH would be that GDB sets an internal PYTHONPATH as directed by configure, uses that to load its Python internals, and allows the GDB child processes (apps being debugged) to use the environment PYTHONPATH. For now, I have a functional installation, but it will break as soon as someone installs Python on their system. The part your script plays in this (if I haven't misunderstood the intention/effects of the "--exec-prefix" part) is allowing a uniform way of directing the configure script to the right directories for includes/libs. Ruben > >> >>>>> I understand the need for this, but as gdb requires Python 2, and users >>>>> of my toolchain may have installed Python 3 or a 32-bit version python >>>>> they want to use from the same environment (without changing their own >>>>> PYTHONPATH), there is no way to run python-enabled gdb. >>>>> [...] >>>> >>>> Yeah. >>>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >>>> gdb use that instead of PYTHONPATH if it exists, but there's been >>>> resistance to it. >>>> I think(!) what would happen is that gdb would set $PYTHONPATH to the >>>> value of $GDB_PYTHONPATH. >>>> [Inferiors started by gdb should still get the original value of >>>> PYTHONPATH though.] >>> >>> That way would be almost ideal, but a hardcoded *relative* path to the >>> python scripts (that is standardized within gdb) wouldn't hurt. >> >> See above re: --exec-prefix. >> >>> An >>> extra environment variable would require a lot of explaining for >>> Windows, and is not "plug-and-play", like the rest of a sysrooted >>> toolchain is supposed to be like. I think this should work on all >>> setups: >>> >>> 1. Check hardcoded path; my suggestion would be "<gdb >>> executable>/../lib/python27" >>> 2. If this fails to find the necessary files/scripts, find it like you >>> described above in Linux, without PYTHONPATH set. >>> 3. Check PYTHONPATH. >> >> The problem being solved by the proposed GDB_PYTHONPATH is "What if >> the user has PYTHONPATH set and it points to an incompatible version >> of python?". >> Leaving such a value for PYTHONPATH set while gdb's python is running >> feels wrong (and IIRC has caused some problems). >> >> The problem of telling python where to find itself is already solved >> (or at least is intended to be solved) with gdb's calling >> Py_SetProgramName with a value derived from the python-provided >> --exec-prefix. >> >>> I would think only number one would change, and perhaps be only >>> enabled with a special configure option. Nothing else would have to >>> change, and Windows users would rejoice :) >>> Again, this is only my suggestion, if there are problems with it in >>> way I haven't thought of, please say so, and we can come up with >>> another solution. >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-15 13:27 ` Ruben Van Boxem @ 2011-05-15 16:12 ` Ruben Van Boxem 2011-05-17 16:32 ` Ruben Van Boxem ` (2 more replies) 2011-05-17 20:36 ` Doug Evans 1 sibling, 3 replies; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-15 16:12 UTC (permalink / raw) To: Doug Evans; +Cc: gdb, python-list 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>: > 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>: >> 2011/5/14 Doug Evans <dje@google.com>: >>> On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem >>> <vanboxem.ruben@gmail.com> wrote: >>>> 2011/5/14 Doug Evans <dje@google.com>: >>>>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >>>>> <vanboxem.ruben@gmail.com> wrote: >>>>>> (now in plain-text as required by gdb mailing list) >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am currently trying to integrate Python support into my toolchain >>>>>> build (including GDB of course). It is a sysrooted >>>>>> binutils+GCC+GDB+mingw-w64 toolchain. >>>>>> >>>>>> I currently have the basic setup working: I can link gdb with my >>>>>> manually generated import lib to the python dll from the official >>>>>> Windows install. If there is anything I am missing or a very easy >>>>>> solution to the problems decsribed below, please just say so. I am >>>>>> only suggesting what I would like to happen. >>>>>> >>>>>> Now on to the problems I'd like to discuss: >>>>>> >>>>>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >>>>> >>>>> In a properly configured/built gdb on linux this isn't necessary, even >>>>> if python is installed in some random place. >>>>> I'm not sure about windows though. >>>>> Did you specify --with-python when you configured gdb, and if so did >>>>> you specify a value? >>>>> e.g., --with-python=SOME_VALUE >>>> >>>> I was cross-compiling a mingw toolchain+gdb from Linux, so I used >>>> --with-python without a value (because gdb configure tries to find the >>>> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS >>>> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it >>>> should. This is hacky though, and gdb configure should provide >>>> --with-python-libs and --with-python-include to make it more >>>> streamlined with any other build prerequisite (like >>>> gmp/mpfr/mpc/cloog/ppl in GCC for example). >>> >>> Ah. >>> Cross-compiling gdb with python is in need of improvement. >>> Alas python hasn't been designed with cross-compilation in mind (e.g. >>> build on linux, run on windows). >>> AIUI, the way to get the parameters required for compiling with >>> libpython is to get them from python's "distutils": kinda hard to do >>> in a cross-compile. Done correctly there's no need to run python. >>> >>> I haven't done anything more to support python in gdb's configure.ac >>> because it's not clear to me what the right thing to do is: distutils >>> provides more than just --libs and --includes (btw, we don't use >>> --libs though, we use --ldflags which includes all of: the directory >>> in which to find libpython, the -l for libpython, and the -l's for all >>> the other libraries python needs). [Which isn't to say that someone >>> else isn't free to tackle this.] >>> >>> In the meantime, what I've been doing is a hack: write a script that >>> responds to: >>> --includes >>> --ldflags >>> --exec-prefix >>> and pass that as --with-python. >>> >>> E.g. >>> bash$ cat $HOME/my-python-for-config >>> #! /bin/sh >>> >>> if [ $# -ne 2 ] >>> then >>> echo "Bad # args. Blech!" >&2 >>> exit 1 >>> fi >>> >>> # The first argument is the path to python-config.py, ignore it. >>> >>> case "$2" in >>> --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; >>> --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm >>> -lpython2.6" ;; >>> --exec-prefix) echo "/usr" ;; >>> *) echo "Bad arg $2. Blech!" >&2 ; exit 1 ;; >>> esac >>> >>> exit 0 >>> bash$ ./configure --with-python=$HOME/my-python-for-config [...] >>> [...] >>> >>> >>> Note that --exec-prefix is the runtime location of python. >>> GCC uses this to tell libpython where to find its support files. >>> [grep for Py_SetProgramName in gdb/python/python.c] >> >> OK, I tried your script in a couple of variations. It gets rid of the >> traceback I had before, but still doesn't help the PYTHONPATH problem. >> My directory structure is as follows (this is not in root, "/" is just >> my main build directory, there are several levels below it): >> >> /gdb <-- gdb build dir >> /gdb/gdb <-- where the python configuration is done >> /python <-- temporary install dir for python files for build, >> extracted from the official Windows installer, also location of >> libpython2.7.a import library >> /python/include/python27 <-- python headers, found by gdb in both cases >> /mingw64 <-- toolchain sysroot prefix >> /mingw64/bin <-- install location of gdb and python27.dll >> /mingw64/lib/python27 <-- install location of all python scripts >> >> First variant: >> --includes) echo "-I../../python/include" ;; >> --ldflags) echo "-L../../python -lpython2.7" ;; >> --exec-prefix) echo "../../mingw64/lib/python27" ;; >> >> Here exec-prefix would be the relative path from where "configure" >> does its magic to the final location of the scripts on the build >> system. >> >> Second variant: >> --includes) echo "-I../../python/include" ;; >> --ldflags) echo "-L../../python -lpython2.7" ;; >> --exec-prefix) echo "../lib/python27" ;; >> >> I thought the second points gdb to the installed location of the >> python scripts, but it still needed PYTHONPATH for that. I used >> relative paths in an attempt to "do the right thing", hackwise... >> Remember that the whole "/mingw64" directory gets zipped and moved, >> then it gets extracted to a random location in the Windows filesystem. >> >> Thanks for the help so far, I think we might just get this worked out together. >> >> Ruben > > Wow, I think I have a partial solution. Delving into the Python docs, > for example here: > http://docs.python.org/using/windows.html#finding-modules, you can see > that PYTHONPATH is used first, then the Windows registry, then > PYTHONHOME, then some default relative paths. I placed the python > scripts all in the directory structure like so: > > /bin/gdb > /bin/Lib/<python scripts> > /bin/python27.dll > > This works, even without any manual PYTHONPATH intervention. Problem > is though, that as soon as someone has a PYTHONPATH environment > variable from a (incompatible) Python installation (think different > bitness or version 3.x instead of 2.7.1), I cannot predict what will > go wrong. This problem originates in Python's way of filling in the > search path (sys.path). A true solution in the GDB case to prevent > this collision of an incompatible PYTHONPATH would be that GDB sets an > internal PYTHONPATH as directed by configure, uses that to load its > Python internals, and allows the GDB child processes (apps being > debugged) to use the environment PYTHONPATH. For now, I have a > functional installation, but it will break as soon as someone installs > Python on their system. > > The part your script plays in this (if I haven't misunderstood the > intention/effects of the "--exec-prefix" part) is allowing a uniform > way of directing the configure script to the right directories for > includes/libs. > > Ruben I am sorry for the repeated messages that no one cares about, but I may have discovered GDB in its current form already allows what I want: I tried to figure out what exact paths the snake in gdb was using to search for its modules, and came up with this: (gdb) python import sys (gdb) python print sys.path ['m:\\development\\mingw64\\share\\gdb/python', 'M:\\Development\\mingw64\\bin\\python27.zip', 'M:\\Development\\mingw64\\bin\\DLLs', 'M:\\Development\\mingw64\\bin\\lib', 'M:\\Development\\mingw64\\bin\\lib\\plat-win', 'M:\\Development\\mingw64\\bin\\lib\\lib-tk', 'M:\\Development\\mingw64\\bin', 'M:\\Development\\mingw64\\bin\\lib\\site-packages'] This means that every python command within gdb searches <sysroot>/share/gdb/python FIRST (even before an environment's PYTHONPATH), alleviating any concerns or problems I or anyone would have with another python installation, as this apparently built-in path comes up first. All I, or anyone interested in doing this kind of thing, have to do is copy all the python scripts from the Windows installation's Lib directory to the <sysroot>/share/gdb/python directory. I don't know where this path comes from, but it is quite handy, and makes this whole discussion moot for Python people. Only "issue" that I'll have to work around is the --with-python-includes and --with-python-libs that are missing, using either manual CFLAGS/LDFLAGS or a variant of your script. Thanks for all the help and motivation to keep trying :) Ruben > >> >>> >>>>>> I understand the need for this, but as gdb requires Python 2, and users >>>>>> of my toolchain may have installed Python 3 or a 32-bit version python >>>>>> they want to use from the same environment (without changing their own >>>>>> PYTHONPATH), there is no way to run python-enabled gdb. >>>>>> [...] >>>>> >>>>> Yeah. >>>>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >>>>> gdb use that instead of PYTHONPATH if it exists, but there's been >>>>> resistance to it. >>>>> I think(!) what would happen is that gdb would set $PYTHONPATH to the >>>>> value of $GDB_PYTHONPATH. >>>>> [Inferiors started by gdb should still get the original value of >>>>> PYTHONPATH though.] >>>> >>>> That way would be almost ideal, but a hardcoded *relative* path to the >>>> python scripts (that is standardized within gdb) wouldn't hurt. >>> >>> See above re: --exec-prefix. >>> >>>> An >>>> extra environment variable would require a lot of explaining for >>>> Windows, and is not "plug-and-play", like the rest of a sysrooted >>>> toolchain is supposed to be like. I think this should work on all >>>> setups: >>>> >>>> 1. Check hardcoded path; my suggestion would be "<gdb >>>> executable>/../lib/python27" >>>> 2. If this fails to find the necessary files/scripts, find it like you >>>> described above in Linux, without PYTHONPATH set. >>>> 3. Check PYTHONPATH. >>> >>> The problem being solved by the proposed GDB_PYTHONPATH is "What if >>> the user has PYTHONPATH set and it points to an incompatible version >>> of python?". >>> Leaving such a value for PYTHONPATH set while gdb's python is running >>> feels wrong (and IIRC has caused some problems). >>> >>> The problem of telling python where to find itself is already solved >>> (or at least is intended to be solved) with gdb's calling >>> Py_SetProgramName with a value derived from the python-provided >>> --exec-prefix. >>> >>>> I would think only number one would change, and perhaps be only >>>> enabled with a special configure option. Nothing else would have to >>>> change, and Windows users would rejoice :) >>>> Again, this is only my suggestion, if there are problems with it in >>>> way I haven't thought of, please say so, and we can come up with >>>> another solution. >>> >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-15 16:12 ` Ruben Van Boxem @ 2011-05-17 16:32 ` Ruben Van Boxem 2011-05-17 20:52 ` Doug Evans 2011-06-09 2:04 ` Asm warrior 2 siblings, 0 replies; 16+ messages in thread From: Ruben Van Boxem @ 2011-05-17 16:32 UTC (permalink / raw) To: Doug Evans; +Cc: gdb, python-list 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>: > 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>: >> 2011/5/15 Ruben Van Boxem <vanboxem.ruben@gmail.com>: >>> 2011/5/14 Doug Evans <dje@google.com>: >>>> On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem >>>> <vanboxem.ruben@gmail.com> wrote: >>>>> 2011/5/14 Doug Evans <dje@google.com>: >>>>>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >>>>>> <vanboxem.ruben@gmail.com> wrote: >>>>>>> (now in plain-text as required by gdb mailing list) >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am currently trying to integrate Python support into my toolchain >>>>>>> build (including GDB of course). It is a sysrooted >>>>>>> binutils+GCC+GDB+mingw-w64 toolchain. >>>>>>> >>>>>>> I currently have the basic setup working: I can link gdb with my >>>>>>> manually generated import lib to the python dll from the official >>>>>>> Windows install. If there is anything I am missing or a very easy >>>>>>> solution to the problems decsribed below, please just say so. I am >>>>>>> only suggesting what I would like to happen. >>>>>>> >>>>>>> Now on to the problems I'd like to discuss: >>>>>>> >>>>>>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >>>>>> >>>>>> In a properly configured/built gdb on linux this isn't necessary, even >>>>>> if python is installed in some random place. >>>>>> I'm not sure about windows though. >>>>>> Did you specify --with-python when you configured gdb, and if so did >>>>>> you specify a value? >>>>>> e.g., --with-python=SOME_VALUE >>>>> >>>>> I was cross-compiling a mingw toolchain+gdb from Linux, so I used >>>>> --with-python without a value (because gdb configure tries to find the >>>>> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS >>>>> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it >>>>> should. This is hacky though, and gdb configure should provide >>>>> --with-python-libs and --with-python-include to make it more >>>>> streamlined with any other build prerequisite (like >>>>> gmp/mpfr/mpc/cloog/ppl in GCC for example). >>>> >>>> Ah. >>>> Cross-compiling gdb with python is in need of improvement. >>>> Alas python hasn't been designed with cross-compilation in mind (e.g. >>>> build on linux, run on windows). >>>> AIUI, the way to get the parameters required for compiling with >>>> libpython is to get them from python's "distutils": kinda hard to do >>>> in a cross-compile. Done correctly there's no need to run python. >>>> >>>> I haven't done anything more to support python in gdb's configure.ac >>>> because it's not clear to me what the right thing to do is: distutils >>>> provides more than just --libs and --includes (btw, we don't use >>>> --libs though, we use --ldflags which includes all of: the directory >>>> in which to find libpython, the -l for libpython, and the -l's for all >>>> the other libraries python needs). [Which isn't to say that someone >>>> else isn't free to tackle this.] >>>> >>>> In the meantime, what I've been doing is a hack: write a script that >>>> responds to: >>>> --includes >>>> --ldflags >>>> --exec-prefix >>>> and pass that as --with-python. >>>> >>>> E.g. >>>> bash$ cat $HOME/my-python-for-config >>>> #! /bin/sh >>>> >>>> if [ $# -ne 2 ] >>>> then >>>> echo "Bad # args. Blech!" >&2 >>>> exit 1 >>>> fi >>>> >>>> # The first argument is the path to python-config.py, ignore it. >>>> >>>> case "$2" in >>>> --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; >>>> --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm >>>> -lpython2.6" ;; >>>> --exec-prefix) echo "/usr" ;; >>>> *) echo "Bad arg $2. Blech!" >&2 ; exit 1 ;; >>>> esac >>>> >>>> exit 0 >>>> bash$ ./configure --with-python=$HOME/my-python-for-config [...] >>>> [...] >>>> >>>> >>>> Note that --exec-prefix is the runtime location of python. >>>> GCC uses this to tell libpython where to find its support files. >>>> [grep for Py_SetProgramName in gdb/python/python.c] >>> >>> OK, I tried your script in a couple of variations. It gets rid of the >>> traceback I had before, but still doesn't help the PYTHONPATH problem. >>> My directory structure is as follows (this is not in root, "/" is just >>> my main build directory, there are several levels below it): >>> >>> /gdb <-- gdb build dir >>> /gdb/gdb <-- where the python configuration is done >>> /python <-- temporary install dir for python files for build, >>> extracted from the official Windows installer, also location of >>> libpython2.7.a import library >>> /python/include/python27 <-- python headers, found by gdb in both cases >>> /mingw64 <-- toolchain sysroot prefix >>> /mingw64/bin <-- install location of gdb and python27.dll >>> /mingw64/lib/python27 <-- install location of all python scripts >>> >>> First variant: >>> --includes) echo "-I../../python/include" ;; >>> --ldflags) echo "-L../../python -lpython2.7" ;; >>> --exec-prefix) echo "../../mingw64/lib/python27" ;; >>> >>> Here exec-prefix would be the relative path from where "configure" >>> does its magic to the final location of the scripts on the build >>> system. >>> >>> Second variant: >>> --includes) echo "-I../../python/include" ;; >>> --ldflags) echo "-L../../python -lpython2.7" ;; >>> --exec-prefix) echo "../lib/python27" ;; >>> >>> I thought the second points gdb to the installed location of the >>> python scripts, but it still needed PYTHONPATH for that. I used >>> relative paths in an attempt to "do the right thing", hackwise... >>> Remember that the whole "/mingw64" directory gets zipped and moved, >>> then it gets extracted to a random location in the Windows filesystem. >>> >>> Thanks for the help so far, I think we might just get this worked out together. >>> >>> Ruben >> >> Wow, I think I have a partial solution. Delving into the Python docs, >> for example here: >> http://docs.python.org/using/windows.html#finding-modules, you can see >> that PYTHONPATH is used first, then the Windows registry, then >> PYTHONHOME, then some default relative paths. I placed the python >> scripts all in the directory structure like so: >> >> /bin/gdb >> /bin/Lib/<python scripts> >> /bin/python27.dll >> >> This works, even without any manual PYTHONPATH intervention. Problem >> is though, that as soon as someone has a PYTHONPATH environment >> variable from a (incompatible) Python installation (think different >> bitness or version 3.x instead of 2.7.1), I cannot predict what will >> go wrong. This problem originates in Python's way of filling in the >> search path (sys.path). A true solution in the GDB case to prevent >> this collision of an incompatible PYTHONPATH would be that GDB sets an >> internal PYTHONPATH as directed by configure, uses that to load its >> Python internals, and allows the GDB child processes (apps being >> debugged) to use the environment PYTHONPATH. For now, I have a >> functional installation, but it will break as soon as someone installs >> Python on their system. >> >> The part your script plays in this (if I haven't misunderstood the >> intention/effects of the "--exec-prefix" part) is allowing a uniform >> way of directing the configure script to the right directories for >> includes/libs. >> >> Ruben > > I am sorry for the repeated messages that no one cares about, but I > may have discovered GDB in its current form already allows what I > want: I tried to figure out what exact paths the snake in gdb was > using to search for its modules, and came up with this: > (gdb) python import sys > (gdb) python print sys.path > ['m:\\development\\mingw64\\share\\gdb/python', > 'M:\\Development\\mingw64\\bin\\python27.zip', > 'M:\\Development\\mingw64\\bin\\DLLs', > 'M:\\Development\\mingw64\\bin\\lib', > 'M:\\Development\\mingw64\\bin\\lib\\plat-win', > 'M:\\Development\\mingw64\\bin\\lib\\lib-tk', > 'M:\\Development\\mingw64\\bin', > 'M:\\Development\\mingw64\\bin\\lib\\site-packages'] > > This means that every python command within gdb searches > <sysroot>/share/gdb/python FIRST (even before an environment's > PYTHONPATH), alleviating any concerns or problems I or anyone would > have with another python installation, as this apparently built-in > path comes up first. All I, or anyone interested in doing this kind of > thing, have to do is copy all the python scripts from the Windows > installation's Lib directory to the <sysroot>/share/gdb/python > directory. > > I don't know where this path comes from, but it is quite handy, and > makes this whole discussion moot for Python people. Only "issue" that > I'll have to work around is the --with-python-includes and > --with-python-libs that are missing, using either manual > CFLAGS/LDFLAGS or a variant of your script. > > Thanks for all the help and motivation to keep trying :) > > Ruben Scratch that, I had lingering install of Python laying around. Without PYTHONPATH, no gdb (module site not found). Why not make gdb search the ../share/gdb/python path first, and then complain? Ruben > >> >>> >>>> >>>>>>> I understand the need for this, but as gdb requires Python 2, and users >>>>>>> of my toolchain may have installed Python 3 or a 32-bit version python >>>>>>> they want to use from the same environment (without changing their own >>>>>>> PYTHONPATH), there is no way to run python-enabled gdb. >>>>>>> [...] >>>>>> >>>>>> Yeah. >>>>>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >>>>>> gdb use that instead of PYTHONPATH if it exists, but there's been >>>>>> resistance to it. >>>>>> I think(!) what would happen is that gdb would set $PYTHONPATH to the >>>>>> value of $GDB_PYTHONPATH. >>>>>> [Inferiors started by gdb should still get the original value of >>>>>> PYTHONPATH though.] >>>>> >>>>> That way would be almost ideal, but a hardcoded *relative* path to the >>>>> python scripts (that is standardized within gdb) wouldn't hurt. >>>> >>>> See above re: --exec-prefix. >>>> >>>>> An >>>>> extra environment variable would require a lot of explaining for >>>>> Windows, and is not "plug-and-play", like the rest of a sysrooted >>>>> toolchain is supposed to be like. I think this should work on all >>>>> setups: >>>>> >>>>> 1. Check hardcoded path; my suggestion would be "<gdb >>>>> executable>/../lib/python27" >>>>> 2. If this fails to find the necessary files/scripts, find it like you >>>>> described above in Linux, without PYTHONPATH set. >>>>> 3. Check PYTHONPATH. >>>> >>>> The problem being solved by the proposed GDB_PYTHONPATH is "What if >>>> the user has PYTHONPATH set and it points to an incompatible version >>>> of python?". >>>> Leaving such a value for PYTHONPATH set while gdb's python is running >>>> feels wrong (and IIRC has caused some problems). >>>> >>>> The problem of telling python where to find itself is already solved >>>> (or at least is intended to be solved) with gdb's calling >>>> Py_SetProgramName with a value derived from the python-provided >>>> --exec-prefix. >>>> >>>>> I would think only number one would change, and perhaps be only >>>>> enabled with a special configure option. Nothing else would have to >>>>> change, and Windows users would rejoice :) >>>>> Again, this is only my suggestion, if there are problems with it in >>>>> way I haven't thought of, please say so, and we can come up with >>>>> another solution. >>>> >>> >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-15 16:12 ` Ruben Van Boxem 2011-05-17 16:32 ` Ruben Van Boxem @ 2011-05-17 20:52 ` Doug Evans 2011-06-09 2:04 ` Asm warrior 2 siblings, 0 replies; 16+ messages in thread From: Doug Evans @ 2011-05-17 20:52 UTC (permalink / raw) To: vanboxem.ruben; +Cc: gdb, python-list On Sun, May 15, 2011 at 9:11 AM, Ruben Van Boxem <vanboxem.ruben@gmail.com> wrote: > I am sorry for the repeated messages that no one cares about, but I > may have discovered GDB in its current form already allows what I > want: I tried to figure out what exact paths the snake in gdb was > using to search for its modules, and came up with this: > (gdb) python import sys > (gdb) python print sys.path > ['m:\\development\\mingw64\\share\\gdb/python', > 'M:\\Development\\mingw64\\bin\\python27.zip', > 'M:\\Development\\mingw64\\bin\\DLLs', > 'M:\\Development\\mingw64\\bin\\lib', > 'M:\\Development\\mingw64\\bin\\lib\\plat-win', > 'M:\\Development\\mingw64\\bin\\lib\\lib-tk', > 'M:\\Development\\mingw64\\bin', > 'M:\\Development\\mingw64\\bin\\lib\\site-packages'] > > This means that every python command within gdb searches > <sysroot>/share/gdb/python FIRST (even before an environment's > PYTHONPATH), alleviating any concerns or problems I or anyone would > have with another python installation, as this apparently built-in > path comes up first. All I, or anyone interested in doing this kind of > thing, have to do is copy all the python scripts from the Windows > installation's Lib directory to the <sysroot>/share/gdb/python > directory. > > I don't know where this path comes from, but it is quite handy, and > makes this whole discussion moot for Python people. Only "issue" that > I'll have to work around is the --with-python-includes and > --with-python-libs that are missing, using either manual > CFLAGS/LDFLAGS or a variant of your script. IMO *if* gdb wanted to support people adding files to *its* <sysroot>/share/gdb directory, more thought is needed - e.g. maybe document a "site" directory for such files. Dunno. IMO we certainly don't want to formally allow folks to willy-nilly put anything there - no guarantees a future release might add something that collides with something the user put there. If and until then, you're probably pretty safe if you put everything in a subdirectory with a unique enough name, if you really wanted to go this route. Btw, there is the system.gdbinit file which sites *are* free to customize. E.g., configure --with-system-gdbinit=<sysroot>/share/gdb/system.gdbinit. You can put whatever you want in that file. E.g., you could add a directory to python's sys.path. [Technically speaking, the path <sysroot>/share/gdb/system.gdbinit goes against what I just said, but IMO this name is safe.] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-15 16:12 ` Ruben Van Boxem 2011-05-17 16:32 ` Ruben Van Boxem 2011-05-17 20:52 ` Doug Evans @ 2011-06-09 2:04 ` Asm warrior [not found] ` <BANLkTimfNgaKWbZdN-YVz9Vs2UW6vKsfiQ@mail.gmail.com> 2 siblings, 1 reply; 16+ messages in thread From: Asm warrior @ 2011-06-09 2:04 UTC (permalink / raw) To: vanboxem.ruben; +Cc: gdb On 2011-5-16 0:11, Ruben Van Boxem wrote: > 2011/5/15 Ruben Van Boxem<vanboxem.ruben@gmail.com>: >> 2011/5/15 Ruben Van Boxem<vanboxem.ruben@gmail.com>: >>> 2011/5/14 Doug Evans<dje@google.com>: >>>> On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem >>>> <vanboxem.ruben@gmail.com> wrote: >>>>> 2011/5/14 Doug Evans<dje@google.com>: >>>>>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >>>>>> <vanboxem.ruben@gmail.com> wrote: >>>>>>> (now in plain-text as required by gdb mailing list) >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am currently trying to integrate Python support into my toolchain >>>>>>> build (including GDB of course). It is a sysrooted >>>>>>> binutils+GCC+GDB+mingw-w64 toolchain. >>>>>>> >>>>>>> I currently have the basic setup working: I can link gdb with my >>>>>>> manually generated import lib to the python dll from the official >>>>>>> Windows install. If there is anything I am missing or a very easy >>>>>>> solution to the problems decsribed below, please just say so. I am >>>>>>> only suggesting what I would like to happen. >>>>>>> >>>>>>> Now on to the problems I'd like to discuss: >>>>>>> >>>>>>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >>>>>> >>>>>> In a properly configured/built gdb on linux this isn't necessary, even >>>>>> if python is installed in some random place. >>>>>> I'm not sure about windows though. >>>>>> Did you specify --with-python when you configured gdb, and if so did >>>>>> you specify a value? >>>>>> e.g., --with-python=SOME_VALUE >>>>> >>>>> I was cross-compiling a mingw toolchain+gdb from Linux, so I used >>>>> --with-python without a value (because gdb configure tries to find the >>>>> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS >>>>> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it >>>>> should. This is hacky though, and gdb configure should provide >>>>> --with-python-libs and --with-python-include to make it more >>>>> streamlined with any other build prerequisite (like >>>>> gmp/mpfr/mpc/cloog/ppl in GCC for example). >>>> >>>> Ah. >>>> Cross-compiling gdb with python is in need of improvement. >>>> Alas python hasn't been designed with cross-compilation in mind (e.g. >>>> build on linux, run on windows). >>>> AIUI, the way to get the parameters required for compiling with >>>> libpython is to get them from python's "distutils": kinda hard to do >>>> in a cross-compile. Done correctly there's no need to run python. >>>> >>>> I haven't done anything more to support python in gdb's configure.ac >>>> because it's not clear to me what the right thing to do is: distutils >>>> provides more than just --libs and --includes (btw, we don't use >>>> --libs though, we use --ldflags which includes all of: the directory >>>> in which to find libpython, the -l for libpython, and the -l's for all >>>> the other libraries python needs). [Which isn't to say that someone >>>> else isn't free to tackle this.] >>>> >>>> In the meantime, what I've been doing is a hack: write a script that >>>> responds to: >>>> --includes >>>> --ldflags >>>> --exec-prefix >>>> and pass that as --with-python. >>>> >>>> E.g. >>>> bash$ cat $HOME/my-python-for-config >>>> #! /bin/sh >>>> >>>> if [ $# -ne 2 ] >>>> then >>>> echo "Bad # args. Blech!">&2 >>>> exit 1 >>>> fi >>>> >>>> # The first argument is the path to python-config.py, ignore it. >>>> >>>> case "$2" in >>>> --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; >>>> --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm >>>> -lpython2.6" ;; >>>> --exec-prefix) echo "/usr" ;; >>>> *) echo "Bad arg $2. Blech!">&2 ; exit 1 ;; >>>> esac >>>> >>>> exit 0 >>>> bash$ ./configure --with-python=$HOME/my-python-for-config [...] >>>> [...] >>>> >>>> >>>> Note that --exec-prefix is the runtime location of python. >>>> GCC uses this to tell libpython where to find its support files. >>>> [grep for Py_SetProgramName in gdb/python/python.c] >>> >>> OK, I tried your script in a couple of variations. It gets rid of the >>> traceback I had before, but still doesn't help the PYTHONPATH problem. >>> My directory structure is as follows (this is not in root, "/" is just >>> my main build directory, there are several levels below it): >>> >>> /gdb<-- gdb build dir >>> /gdb/gdb<-- where the python configuration is done >>> /python<-- temporary install dir for python files for build, >>> extracted from the official Windows installer, also location of >>> libpython2.7.a import library >>> /python/include/python27<-- python headers, found by gdb in both cases >>> /mingw64<-- toolchain sysroot prefix >>> /mingw64/bin<-- install location of gdb and python27.dll >>> /mingw64/lib/python27<-- install location of all python scripts >>> >>> First variant: >>> --includes) echo "-I../../python/include" ;; >>> --ldflags) echo "-L../../python -lpython2.7" ;; >>> --exec-prefix) echo "../../mingw64/lib/python27" ;; >>> >>> Here exec-prefix would be the relative path from where "configure" >>> does its magic to the final location of the scripts on the build >>> system. >>> >>> Second variant: >>> --includes) echo "-I../../python/include" ;; >>> --ldflags) echo "-L../../python -lpython2.7" ;; >>> --exec-prefix) echo "../lib/python27" ;; >>> >>> I thought the second points gdb to the installed location of the >>> python scripts, but it still needed PYTHONPATH for that. I used >>> relative paths in an attempt to "do the right thing", hackwise... >>> Remember that the whole "/mingw64" directory gets zipped and moved, >>> then it gets extracted to a random location in the Windows filesystem. >>> >>> Thanks for the help so far, I think we might just get this worked out together. >>> >>> Ruben >> >> Wow, I think I have a partial solution. Delving into the Python docs, >> for example here: >> http://docs.python.org/using/windows.html#finding-modules, you can see >> that PYTHONPATH is used first, then the Windows registry, then >> PYTHONHOME, then some default relative paths. I placed the python >> scripts all in the directory structure like so: >> >> /bin/gdb >> /bin/Lib/<python scripts> >> /bin/python27.dll >> >> This works, even without any manual PYTHONPATH intervention. Problem >> is though, that as soon as someone has a PYTHONPATH environment >> variable from a (incompatible) Python installation (think different >> bitness or version 3.x instead of 2.7.1), I cannot predict what will >> go wrong. This problem originates in Python's way of filling in the >> search path (sys.path). A true solution in the GDB case to prevent >> this collision of an incompatible PYTHONPATH would be that GDB sets an >> internal PYTHONPATH as directed by configure, uses that to load its >> Python internals, and allows the GDB child processes (apps being >> debugged) to use the environment PYTHONPATH. For now, I have a >> functional installation, but it will break as soon as someone installs >> Python on their system. >> >> The part your script plays in this (if I haven't misunderstood the >> intention/effects of the "--exec-prefix" part) is allowing a uniform >> way of directing the configure script to the right directories for >> includes/libs. >> >> Ruben > > I am sorry for the repeated messages that no one cares about, but I > may have discovered GDB in its current form already allows what I > want: I tried to figure out what exact paths the snake in gdb was > using to search for its modules, and came up with this: > (gdb) python import sys > (gdb) python print sys.path > ['m:\\development\\mingw64\\share\\gdb/python', > 'M:\\Development\\mingw64\\bin\\python27.zip', > 'M:\\Development\\mingw64\\bin\\DLLs', > 'M:\\Development\\mingw64\\bin\\lib', > 'M:\\Development\\mingw64\\bin\\lib\\plat-win', > 'M:\\Development\\mingw64\\bin\\lib\\lib-tk', > 'M:\\Development\\mingw64\\bin', > 'M:\\Development\\mingw64\\bin\\lib\\site-packages'] > > This means that every python command within gdb searches > <sysroot>/share/gdb/python FIRST (even before an environment's > PYTHONPATH), alleviating any concerns or problems I or anyone would > have with another python installation, as this apparently built-in > path comes up first. All I, or anyone interested in doing this kind of > thing, have to do is copy all the python scripts from the Windows > installation's Lib directory to the<sysroot>/share/gdb/python > directory. > > I don't know where this path comes from, but it is quite handy, and > makes this whole discussion moot for Python people. Only "issue" that > I'll have to work around is the --with-python-includes and > --with-python-libs that are missing, using either manual > CFLAGS/LDFLAGS or a variant of your script. > > Thanks for all the help and motivation to keep trying :) > > Ruben > >> >>> >>>> >>>>>>> I understand the need for this, but as gdb requires Python 2, and users >>>>>>> of my toolchain may have installed Python 3 or a 32-bit version python >>>>>>> they want to use from the same environment (without changing their own >>>>>>> PYTHONPATH), there is no way to run python-enabled gdb. >>>>>>> [...] >>>>>> >>>>>> Yeah. >>>>>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >>>>>> gdb use that instead of PYTHONPATH if it exists, but there's been >>>>>> resistance to it. >>>>>> I think(!) what would happen is that gdb would set $PYTHONPATH to the >>>>>> value of $GDB_PYTHONPATH. >>>>>> [Inferiors started by gdb should still get the original value of >>>>>> PYTHONPATH though.] >>>>> >>>>> That way would be almost ideal, but a hardcoded *relative* path to the >>>>> python scripts (that is standardized within gdb) wouldn't hurt. >>>> >>>> See above re: --exec-prefix. >>>> >>>>> An >>>>> extra environment variable would require a lot of explaining for >>>>> Windows, and is not "plug-and-play", like the rest of a sysrooted >>>>> toolchain is supposed to be like. I think this should work on all >>>>> setups: >>>>> >>>>> 1. Check hardcoded path; my suggestion would be "<gdb >>>>> executable>/../lib/python27" >>>>> 2. If this fails to find the necessary files/scripts, find it like you >>>>> described above in Linux, without PYTHONPATH set. >>>>> 3. Check PYTHONPATH. >>>> >>>> The problem being solved by the proposed GDB_PYTHONPATH is "What if >>>> the user has PYTHONPATH set and it points to an incompatible version >>>> of python?". >>>> Leaving such a value for PYTHONPATH set while gdb's python is running >>>> feels wrong (and IIRC has caused some problems). >>>> >>>> The problem of telling python where to find itself is already solved >>>> (or at least is intended to be solved) with gdb's calling >>>> Py_SetProgramName with a value derived from the python-provided >>>> --exec-prefix. >>>> >>>>> I would think only number one would change, and perhaps be only >>>>> enabled with a special configure option. Nothing else would have to >>>>> change, and Windows users would rejoice :) >>>>> Again, this is only my suggestion, if there are problems with it in >>>>> way I haven't thought of, please say so, and we can come up with >>>>> another solution. >>>> >>> >> > Hi, vanboxem. what's your finding is quite interesting. In fact, I just copy the Lib(python supporting scripts) under the MinGW/bin/ folder, then every thing works fine. I never know the details about the python search path. So, thank you very much. asmwarrior ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <BANLkTimfNgaKWbZdN-YVz9Vs2UW6vKsfiQ@mail.gmail.com>]
* Re: Python enabled gdb on Windows and relocation [not found] ` <BANLkTimfNgaKWbZdN-YVz9Vs2UW6vKsfiQ@mail.gmail.com> @ 2011-06-09 12:09 ` Ruben Van Boxem 0 siblings, 0 replies; 16+ messages in thread From: Ruben Van Boxem @ 2011-06-09 12:09 UTC (permalink / raw) To: gdb 2011/6/9 Asm warrior <asmwarrior@gmail.com> > > On 2011-5-16 0:11, Ruben Van Boxem wrote: >> >> 2011/5/15 Ruben Van Boxem<vanboxem.ruben@gmail.com>: >>> >>> 2011/5/15 Ruben Van Boxem<vanboxem.ruben@gmail.com>: >>>> >>>> 2011/5/14 Doug Evans<dje@google.com>: >>>>> >>>>> On Sat, May 14, 2011 at 2:09 AM, Ruben Van Boxem >>>>> <vanboxem.ruben@gmail.com> wrote: >>>>>> >>>>>> 2011/5/14 Doug Evans<dje@google.com>: >>>>>>> >>>>>>> On Thu, May 12, 2011 at 9:19 AM, Ruben Van Boxem >>>>>>> <vanboxem.ruben@gmail.com> wrote: >>>>>>>> >>>>>>>> (now in plain-text as required by gdb mailing list) >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am currently trying to integrate Python support into my toolchain >>>>>>>> build (including GDB of course). It is a sysrooted >>>>>>>> binutils+GCC+GDB+mingw-w64 toolchain. >>>>>>>> >>>>>>>> I currently have the basic setup working: I can link gdb with my >>>>>>>> manually generated import lib to the python dll from the official >>>>>>>> Windows install. If there is anything I am missing or a very easy >>>>>>>> solution to the problems decsribed below, please just say so. I am >>>>>>>> only suggesting what I would like to happen. >>>>>>>> >>>>>>>> Now on to the problems I'd like to discuss: >>>>>>>> >>>>>>>> 1. gdb.exe won't start without me having set PYTHONPATH manually. >>>>>>> >>>>>>> In a properly configured/built gdb on linux this isn't necessary, even >>>>>>> if python is installed in some random place. >>>>>>> I'm not sure about windows though. >>>>>>> Did you specify --with-python when you configured gdb, and if so did >>>>>>> you specify a value? >>>>>>> e.g., --with-python=SOME_VALUE >>>>>> >>>>>> I was cross-compiling a mingw toolchain+gdb from Linux, so I used >>>>>> --with-python without a value (because gdb configure tries to find the >>>>>> Python executabe), and I added -I"/path/to/python/includes" to CFLAGS >>>>>> and -L"/path/to/pythondll/importlib" to LDFLAGS, which built as it >>>>>> should. This is hacky though, and gdb configure should provide >>>>>> --with-python-libs and --with-python-include to make it more >>>>>> streamlined with any other build prerequisite (like >>>>>> gmp/mpfr/mpc/cloog/ppl in GCC for example). >>>>> >>>>> Ah. >>>>> Cross-compiling gdb with python is in need of improvement. >>>>> Alas python hasn't been designed with cross-compilation in mind (e.g. >>>>> build on linux, run on windows). >>>>> AIUI, the way to get the parameters required for compiling with >>>>> libpython is to get them from python's "distutils": kinda hard to do >>>>> in a cross-compile. Done correctly there's no need to run python. >>>>> >>>>> I haven't done anything more to support python in gdb's configure.ac >>>>> because it's not clear to me what the right thing to do is: distutils >>>>> provides more than just --libs and --includes (btw, we don't use >>>>> --libs though, we use --ldflags which includes all of: the directory >>>>> in which to find libpython, the -l for libpython, and the -l's for all >>>>> the other libraries python needs). [Which isn't to say that someone >>>>> else isn't free to tackle this.] >>>>> >>>>> In the meantime, what I've been doing is a hack: write a script that >>>>> responds to: >>>>> --includes >>>>> --ldflags >>>>> --exec-prefix >>>>> and pass that as --with-python. >>>>> >>>>> E.g. >>>>> bash$ cat $HOME/my-python-for-config >>>>> #! /bin/sh >>>>> >>>>> if [ $# -ne 2 ] >>>>> then >>>>> echo "Bad # args. Blech!">&2 >>>>> exit 1 >>>>> fi >>>>> >>>>> # The first argument is the path to python-config.py, ignore it. >>>>> >>>>> case "$2" in >>>>> --includes) echo "-I/usr/include/python2.6 -I/usr/include/python2.6" ;; >>>>> --ldflags) echo "-L/usr/lib/python2.6/config -lpthread -ldl -lutil -lm >>>>> -lpython2.6" ;; >>>>> --exec-prefix) echo "/usr" ;; >>>>> *) echo "Bad arg $2. Blech!">&2 ; exit 1 ;; >>>>> esac >>>>> >>>>> exit 0 >>>>> bash$ ./configure --with-python=$HOME/my-python-for-config [...] >>>>> [...] >>>>> >>>>> >>>>> Note that --exec-prefix is the runtime location of python. >>>>> GCC uses this to tell libpython where to find its support files. >>>>> [grep for Py_SetProgramName in gdb/python/python.c] >>>> >>>> OK, I tried your script in a couple of variations. It gets rid of the >>>> traceback I had before, but still doesn't help the PYTHONPATH problem. >>>> My directory structure is as follows (this is not in root, "/" is just >>>> my main build directory, there are several levels below it): >>>> >>>> /gdb<-- gdb build dir >>>> /gdb/gdb<-- where the python configuration is done >>>> /python<-- temporary install dir for python files for build, >>>> extracted from the official Windows installer, also location of >>>> libpython2.7.a import library >>>> /python/include/python27<-- python headers, found by gdb in both cases >>>> /mingw64<-- toolchain sysroot prefix >>>> /mingw64/bin<-- install location of gdb and python27.dll >>>> /mingw64/lib/python27<-- install location of all python scripts >>>> >>>> First variant: >>>> --includes) echo "-I../../python/include" ;; >>>> --ldflags) echo "-L../../python -lpython2.7" ;; >>>> --exec-prefix) echo "../../mingw64/lib/python27" ;; >>>> >>>> Here exec-prefix would be the relative path from where "configure" >>>> does its magic to the final location of the scripts on the build >>>> system. >>>> >>>> Second variant: >>>> --includes) echo "-I../../python/include" ;; >>>> --ldflags) echo "-L../../python -lpython2.7" ;; >>>> --exec-prefix) echo "../lib/python27" ;; >>>> >>>> I thought the second points gdb to the installed location of the >>>> python scripts, but it still needed PYTHONPATH for that. I used >>>> relative paths in an attempt to "do the right thing", hackwise... >>>> Remember that the whole "/mingw64" directory gets zipped and moved, >>>> then it gets extracted to a random location in the Windows filesystem. >>>> >>>> Thanks for the help so far, I think we might just get this worked out together. >>>> >>>> Ruben >>> >>> Wow, I think I have a partial solution. Delving into the Python docs, >>> for example here: >>> http://docs.python.org/using/windows.html#finding-modules, you can see >>> that PYTHONPATH is used first, then the Windows registry, then >>> PYTHONHOME, then some default relative paths. I placed the python >>> scripts all in the directory structure like so: >>> >>> /bin/gdb >>> /bin/Lib/<python scripts> >>> /bin/python27.dll >>> >>> This works, even without any manual PYTHONPATH intervention. Problem >>> is though, that as soon as someone has a PYTHONPATH environment >>> variable from a (incompatible) Python installation (think different >>> bitness or version 3.x instead of 2.7.1), I cannot predict what will >>> go wrong. This problem originates in Python's way of filling in the >>> search path (sys.path). A true solution in the GDB case to prevent >>> this collision of an incompatible PYTHONPATH would be that GDB sets an >>> internal PYTHONPATH as directed by configure, uses that to load its >>> Python internals, and allows the GDB child processes (apps being >>> debugged) to use the environment PYTHONPATH. For now, I have a >>> functional installation, but it will break as soon as someone installs >>> Python on their system. >>> >>> The part your script plays in this (if I haven't misunderstood the >>> intention/effects of the "--exec-prefix" part) is allowing a uniform >>> way of directing the configure script to the right directories for >>> includes/libs. >>> >>> Ruben >> >> I am sorry for the repeated messages that no one cares about, but I >> may have discovered GDB in its current form already allows what I >> want: I tried to figure out what exact paths the snake in gdb was >> using to search for its modules, and came up with this: >> (gdb) python import sys >> (gdb) python print sys.path >> ['m:\\development\\mingw64\\share\\gdb/python', >> 'M:\\Development\\mingw64\\bin\\python27.zip', >> 'M:\\Development\\mingw64\\bin\\DLLs', >> 'M:\\Development\\mingw64\\bin\\lib', >> 'M:\\Development\\mingw64\\bin\\lib\\plat-win', >> 'M:\\Development\\mingw64\\bin\\lib\\lib-tk', >> 'M:\\Development\\mingw64\\bin', >> 'M:\\Development\\mingw64\\bin\\lib\\site-packages'] >> >> This means that every python command within gdb searches >> <sysroot>/share/gdb/python FIRST (even before an environment's >> PYTHONPATH), alleviating any concerns or problems I or anyone would >> have with another python installation, as this apparently built-in >> path comes up first. All I, or anyone interested in doing this kind of >> thing, have to do is copy all the python scripts from the Windows >> installation's Lib directory to the<sysroot>/share/gdb/python >> directory. >> >> I don't know where this path comes from, but it is quite handy, and >> makes this whole discussion moot for Python people. Only "issue" that >> I'll have to work around is the --with-python-includes and >> --with-python-libs that are missing, using either manual >> CFLAGS/LDFLAGS or a variant of your script. >> >> Thanks for all the help and motivation to keep trying :) >> >> Ruben >> >>> >>>> >>>>> >>>>>>>> I understand the need for this, but as gdb requires Python 2, and users >>>>>>>> of my toolchain may have installed Python 3 or a 32-bit version python >>>>>>>> they want to use from the same environment (without changing their own >>>>>>>> PYTHONPATH), there is no way to run python-enabled gdb. >>>>>>>> [...] >>>>>>> >>>>>>> Yeah. >>>>>>> There is a proposal to add GDB_PYTHONPATH (or some such IIRC) and have >>>>>>> gdb use that instead of PYTHONPATH if it exists, but there's been >>>>>>> resistance to it. >>>>>>> I think(!) what would happen is that gdb would set $PYTHONPATH to the >>>>>>> value of $GDB_PYTHONPATH. >>>>>>> [Inferiors started by gdb should still get the original value of >>>>>>> PYTHONPATH though.] >>>>>> >>>>>> That way would be almost ideal, but a hardcoded *relative* path to the >>>>>> python scripts (that is standardized within gdb) wouldn't hurt. >>>>> >>>>> See above re: --exec-prefix. >>>>> >>>>>> An >>>>>> extra environment variable would require a lot of explaining for >>>>>> Windows, and is not "plug-and-play", like the rest of a sysrooted >>>>>> toolchain is supposed to be like. I think this should work on all >>>>>> setups: >>>>>> >>>>>> 1. Check hardcoded path; my suggestion would be "<gdb >>>>>> executable>/../lib/python27" >>>>>> 2. If this fails to find the necessary files/scripts, find it like you >>>>>> described above in Linux, without PYTHONPATH set. >>>>>> 3. Check PYTHONPATH. >>>>> >>>>> The problem being solved by the proposed GDB_PYTHONPATH is "What if >>>>> the user has PYTHONPATH set and it points to an incompatible version >>>>> of python?". >>>>> Leaving such a value for PYTHONPATH set while gdb's python is running >>>>> feels wrong (and IIRC has caused some problems). >>>>> >>>>> The problem of telling python where to find itself is already solved >>>>> (or at least is intended to be solved) with gdb's calling >>>>> Py_SetProgramName with a value derived from the python-provided >>>>> --exec-prefix. >>>>> >>>>>> I would think only number one would change, and perhaps be only >>>>>> enabled with a special configure option. Nothing else would have to >>>>>> change, and Windows users would rejoice :) >>>>>> Again, this is only my suggestion, if there are problems with it in >>>>>> way I haven't thought of, please say so, and we can come up with >>>>>> another solution. >>>>> >>>> >>> >> > Hi, vanboxem. > > what's your finding is quite interesting. In fact, I just copy the Lib(python supporting scripts) under the MinGW/bin/ folder, then every thing works fine. > > I never know the details about the python search path. Yes, it seems the Python library (the DLL in this case) is responsible for setting everything up, and that is documented, but you can't change it. Even though GDB prepends its own directory to Python's search path, the whole Python "blob" is already initialized at that point, so it needs to have found its init scripts. > > So, thank you very much. Well, I hope this report of things will help future generations :) > > asmwarrior > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Python enabled gdb on Windows and relocation 2011-05-15 13:27 ` Ruben Van Boxem 2011-05-15 16:12 ` Ruben Van Boxem @ 2011-05-17 20:36 ` Doug Evans 1 sibling, 0 replies; 16+ messages in thread From: Doug Evans @ 2011-05-17 20:36 UTC (permalink / raw) To: vanboxem.ruben; +Cc: gdb, python-list On Sun, May 15, 2011 at 6:26 AM, Ruben Van Boxem <vanboxem.ruben@gmail.com> wrote: > Wow, I think I have a partial solution. Delving into the Python docs, > for example here: > http://docs.python.org/using/windows.html#finding-modules, you can see > that PYTHONPATH is used first, then the Windows registry, then > PYTHONHOME, then some default relative paths. I placed the python > scripts all in the directory structure like so: > > /bin/gdb > /bin/Lib/<python scripts> > /bin/python27.dll > > This works, even without any manual PYTHONPATH intervention. Problem > is though, that as soon as someone has a PYTHONPATH environment > variable from a (incompatible) Python installation (think different > bitness or version 3.x instead of 2.7.1), I cannot predict what will > go wrong. This problem originates in Python's way of filling in the > search path (sys.path). A true solution in the GDB case to prevent > this collision of an incompatible PYTHONPATH would be that GDB sets an > internal PYTHONPATH as directed by configure, uses that to load its > Python internals, and allows the GDB child processes (apps being > debugged) to use the environment PYTHONPATH. For now, I have a > functional installation, but it will break as soon as someone installs > Python on their system. What if the user *wants* gdb's python to use $PYTHONPATH from his/her environment? To handle *this* case, *and* the case of an incompatible python installation using $PYTHONPATH, there is the $GDB_PYTHONPATH proposal (see earlier email for details). It feels problematic to decide at configure time whether there will or will not be an incompatible python at runtime. [I realize you have subsequent messages. Just replying in sequence.] ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2011-06-09 12:09 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <BANLkTikrK8DJOEpACA_uUVdZx25P5pc1tA@mail.gmail.com> 2011-05-12 16:20 ` Python enabled gdb on Windows and relocation Ruben Van Boxem 2011-05-14 6:39 ` Doug Evans 2011-05-14 9:09 ` Ruben Van Boxem 2011-05-14 9:30 ` Eli Zaretskii 2011-05-14 10:10 ` Ruben Van Boxem 2011-05-14 18:38 ` Doug Evans 2011-05-14 18:31 ` Doug Evans 2011-05-14 18:40 ` Doug Evans 2011-05-15 12:30 ` Ruben Van Boxem 2011-05-15 13:27 ` Ruben Van Boxem 2011-05-15 16:12 ` Ruben Van Boxem 2011-05-17 16:32 ` Ruben Van Boxem 2011-05-17 20:52 ` Doug Evans 2011-06-09 2:04 ` Asm warrior [not found] ` <BANLkTimfNgaKWbZdN-YVz9Vs2UW6vKsfiQ@mail.gmail.com> 2011-06-09 12:09 ` Ruben Van Boxem 2011-05-17 20:36 ` Doug Evans
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).