[python] Packacking - Python scripts directories

[python] Packacking - Python scripts directories

[Jan Kratochvil]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

Hi Tom,

while rpm-packaging [python] branch into Archer I found there is IMO incorrect
placement of the Python scripts.

https://fedoraproject.org/wiki/Packaging/Python
describes it should be placed in
  /usr/lib/python2.5/site-packages/gdb
Not /usr/lib64/python2.5 on 64-bit as these files are arch-independent.
I would also place it in /usr/share as arch-independent but this is the way
described both by the Fedora guidelines and seen at all the (tried) Fedora
existing packages providing 3rd party modules for Python (like libxml2-python).

--with-gdb-datadir should probably remain present as it is in use by
[archer-sergio-catch-syscall].

--with-gdb-pythondir and `maint set gdb_pythondir' should be new
(IMO the GDB style would be more like `maint set python-directory'.)

The configure.ac code trying 2.4/2.5/2.6 directories to find the include
directory can be IMO replaced by:
  python -c "from distutils.sysconfig import get_python_inc; print get_python_inc()"
  (which prints `/usr/include/python2.5')

The target directory for Python scripts should be:
  python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"
  (which prints `/usr/lib/python2.5/site-packages')

As gdb.pythonlibdir is now the same as the system Python modules directory it
can be IMO removed.  (I do not understand why `import' with some
error-catching is not used instead of locating the include file in the code
and using execfile() which IMO does not provide the modules namespaces.)

There are some remaining issue i do not have opinion on (relocatability of the
Python directories).

Checked-in
[archer-jankratochvil-python] ebd9eefcf0e56102c40cfd104eb1c290526b3f51 which
has been reduced and checked-in:
[archer] 181911192955f46beb536919ea096b83b8ec5d75

Feel free to modify it in [archer-jankratochvil-python] + merge in [archer],


Thanks,
Jan

[-- Attachment #2: archer-branch-python.patch --]
[-- Type: text/plain, Size: 5105 bytes --]

diff --git a/gdb/Makefile.in b/gdb/Makefile.in
index 716884d..951ade8 100644
--- a/gdb/Makefile.in
+++ b/gdb/Makefile.in
@@ -177,6 +177,8 @@ TARGET_SYSTEM_ROOT_DEFINE = @TARGET_SYSTEM_ROOT_DEFINE@
 # Did the user give us a --with-gdb-datadir option?
 GDB_DATADIR_PATH = @GDB_DATADIR_PATH@
 
+GDB_PYTHONDIR_PATH = @GDB_PYTHONDIR_PATH@
+
 # Helper code from gnulib.
 LIBGNU = gnulib/libgnu.a
 INCGNU = -I$(srcdir)/gnulib -Ignulib
@@ -1978,22 +1980,31 @@ PY_FILES = gdb/FrameIterator.py gdb/command/alias.py \
     gdb/command/pahole.py gdb/command/__init__.py \
     gdb/command/ignore_errors.py gdb/command/save_breakpoints.py \
     gdb/libstdcxx/v6/printers.py gdb/libstdcxx/v6/__init__.py \
-    gdb/libstdcxx/__init__.py gdb/function/caller_is.py	\
+    gdb/libstdcxx/__init__.py gdb/function/caller_is.py \
     gdb/function/in_scope.py gdb/function/__init__.py gdb/backtrace.py \
     gdb/__init__.py
 
 # Install the Python library.  Python library files go under
-# $(GDB_DATADIR_PATH)/python.
+# $(GDB_PYTHONDIR_PATH)/python.
 install-python:
 	files='$(PY_FILES)'; for file in $$files; do \
 	  dir=`echo "$$file" | sed 's,/[^/]*$$,,'`; \
-	  $(SHELL) $(srcdir)/../mkinstalldirs $(DESTDIR)$(GDB_DATADIR_PATH)/python/$$dir; \
-	  $(INSTALL_DATA) $(srcdir)/python/lib/$$file $(DESTDIR)$(GDB_DATADIR_PATH)/python/$$file; \
+	  $(SHELL) $(srcdir)/../mkinstalldirs $(DESTDIR)$(GDB_PYTHONDIR_PATH)/$$dir; \
+	  $(INSTALL_DATA) $(srcdir)/python/lib/$$file $(DESTDIR)$(GDB_PYTHONDIR_PATH)/$$file; \
 	done
 
-# Brute force.
+# Other packages may have their files installed in $(GDB_PYTHONDIR_PATH).
 uninstall-python:
-	rm -rf $(DESTDIR)/$(GDB_DATADIR_PATH)/python
+	rm -rf $(DESTDIR)/$(GDB_PYTHONDIR_PATH)/python
+	files='$(PY_FILES)'; for file in $$files; do \
+	  dir=`echo "$$file" | sed 's,/[^/]*$$,,'`; \
+	  rm -f $(DESTDIR)$(GDB_PYTHONDIR_PATH)/$$file; \
+	  while test "x$$file" != "x$$dir"; do \
+	    rmdir 2>/dev/null "$(DESTDIR)$(GDB_PYTHONDIR_PATH)/$$dir"; \
+	    file="$$dir"; \
+	    dir=`echo "$$file" | sed 's,/[^/]*$$,,'`; \
+	  done \
+	done
 
 #
 # Dependency tracking.  Most of this is conditional on GNU Make being
diff --git a/gdb/config.in b/gdb/config.in
index 3ef5bbd..1cfb12b 100644
--- a/gdb/config.in
+++ b/gdb/config.in
@@ -66,6 +66,9 @@
 /* Define to the default OS ABI for this configuration. */
 #undef GDB_OSABI_DEFAULT
 
+/* Base directory for GDB Python modules. */
+#undef GDB_PYTHONDIR_PATH
+
 /* Define to 1 if you have `alloca', as a function or macro. */
 #undef HAVE_ALLOCA
 
diff --git a/gdb/configure.ac b/gdb/configure.ac
index 5892c59..857a7cd 100644
--- a/gdb/configure.ac
+++ b/gdb/configure.ac
@@ -617,6 +617,7 @@ AC_ARG_WITH(python,
 AC_MSG_CHECKING([whether to use python])
 AC_MSG_RESULT([$with_python])
 
+GDB_PYTHONDIR_PATH=
 if test "${with_python}" = no; then
   AC_MSG_WARN([python support disabled; some features may be unavailable.])
   have_libpython=no
@@ -659,7 +660,11 @@ else
       AC_DEFINE(HAVE_LIBPYTHON2_4, 1, [Define if Python 2.4 is being used.])
     fi
   fi
-  if test ${have_libpython} = no; then
+  if test ${have_libpython} != no; then
+    GDB_PYTHONDIR_PATH="`python -c "from distutils.sysconfig import get_python_lib; print get_python_lib();"`"
+    AC_DEFINE_DIR(GDB_PYTHONDIR_PATH, GDB_PYTHONDIR_PATH, 
+		  [Base directory for GDB Python modules.])
+  else
     case "${with_python}" in
     yes)
       AC_MSG_ERROR([python is missing or unusable])
@@ -675,7 +680,9 @@ else
     LIBS=$save_LIBS
   fi
 fi
+AC_SUBST(GDB_PYTHONDIR_PATH)
 
+PYTHON_CFLAGS=
 if test "${have_libpython}" = yes; then
   AC_DEFINE(HAVE_PYTHON, 1, [Define if Python interpreter is being linked in.])
   CONFIG_OBS="$CONFIG_OBS \$(SUBDIR_PYTHON_OBS)"
diff --git a/gdb/python/python.c b/gdb/python/python.c
index a38ea92..18578f0 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -1768,8 +1768,6 @@ Enables or disables auto-loading of Python code when an object is opened."),
   PyModule_AddStringConstant (gdb_module, "VERSION", (char*) version);
   PyModule_AddStringConstant (gdb_module, "HOST_CONFIG", (char*) host_name);
   PyModule_AddStringConstant (gdb_module, "TARGET_CONFIG", (char*) target_name);
-  if (gdb_datadir)
-    PyModule_AddStringConstant (gdb_module, "datadir", gdb_datadir);
 
   gdbpy_initialize_values ();
   gdbpy_initialize_breakpoints ();
@@ -1821,14 +1819,13 @@ class GdbOutputFile:\n\
 \n\
 sys.stderr = GdbOutputFile()\n\
 sys.stdout = GdbOutputFile()\n\
-if hasattr (gdb, 'datadir'):\n\
-  gdb.pythonlibdir = gdb.datadir + '/python'\n\
-  sys.path.insert(0, gdb.pythonlibdir)\n\
-  gdb.__path__ = [gdb.pythonlibdir + '/gdb']\n\
-  from os.path import exists\n\
-  ipy = gdb.pythonlibdir + '/gdb/__init__.py'\n\
-  if exists (ipy):\n\
-    execfile (ipy)\n\
+# FIXME: gdb.pythonlibdir is deprecated as it is just the standard libdir.\n\
+gdb.pythonlibdir = '" GDB_PYTHONDIR_PATH "'\n\
+gdb.__path__ = [gdb.pythonlibdir + '/gdb']\n\
+from os.path import exists\n\
+ipy = gdb.pythonlibdir + '/gdb/__init__.py'\n\
+if exists (ipy):\n\
+  execfile (ipy)\n\
 ");
 
   /* Release the GIL while gdb runs.  */

Re: [python] Packacking - Python scripts directories

[Tom Tromey]

>>>>> "Jan" == Jan Kratochvil <jan.kratochvil@redhat.com> writes:

Jan> while rpm-packaging [python] branch into Archer I found there is
Jan> IMO incorrect placement of the Python scripts.

Jan> https://fedoraproject.org/wiki/Packaging/Python
Jan> describes it should be placed in
Jan>   /usr/lib/python2.5/site-packages/gdb

Yeah, this would be fine by me.

Jan> --with-gdb-pythondir and `maint set gdb_pythondir' should be new
Jan> (IMO the GDB style would be more like `maint set python-directory'.)

Sounds fine.

Jan> As gdb.pythonlibdir is now the same as the system Python modules
Jan> directory it can be IMO removed.  (I do not understand why
Jan> `import' with some error-catching is not used instead of locating
Jan> the include file in the code and using execfile() which IMO does
Jan> not provide the modules namespaces.)

I think the issue here is that we make the "gdb" module internally,
then play games to get python to understand that it can satisfy future
imports by reading from that directory.

If there's a better way, let's do it.

Jan> There are some remaining issue i do not have opinion on
Jan> (relocatability of the Python directories).

Yeah.  We have to preserve relocatability because various other
gdb-using groups want it.

I think the default setup should be to install the python scripts
somewhere under $prefix, then have a separate option to tell configure
to find the system python directory and use that instead.

Jan> Checked-in
Jan> [archer-jankratochvil-python] ebd9eefcf0e56102c40cfd104eb1c290526b3f51 which

We're going to have to clear up this distinction with multiple
branches.  I don't want us to lose changes due to branch confusion.

What if I merge master->archer-tromey-python again and we switch to that?

Tom

Re: [python] Packacking - Python scripts directories

[Jan Kratochvil]

On Mon, 02 Mar 2009 17:35:50 +0100, Tom Tromey wrote:
> I think the issue here is that we make the "gdb" module internally,
> then play games to get python to understand that it can satisfy future
> imports by reading from that directory.
> 
> If there's a better way, let's do it.

I expected something like:
-  from os.path import exists\n\
-  ipy = gdb.pythonlibdir + '/gdb/__init__.py'\n\
-  if exists (ipy):\n\
-    execfile (ipy)\n\
+  try:\n\
+    import gdb.__init__\n\
+  except ImportError:\n\
+    pass\n\

But the error checking and module names from variable gets more complicated so
I did not even prepare the patch.


> Yeah.  We have to preserve relocatability because various other
> gdb-using groups want it.

OK, understood now.

> I think the default setup should be to install the python scripts
> somewhere under $prefix, then have a separate option to tell configure
> to find the system python directory and use that instead.

As default --prefix is `/usr/local' and the default installation directory is
`/usr/lib/python2.5/site-packages' I see now it cannot be used as the upstream
GDB default.  This also makes the `sys.path.insert(0, gdb.pythonlibdir)'
requirement also still valid, OK.


So currently IMO should be changed:
* datadir and pythondir should be two separate directories (with possibly the
  same default as now).
* pythondir should be also relocatable if possible by default.
* /usr/include/pythonX.Y should be found by get_python_inc() in configure.ac.
* gdb.spec would use --with-pythondir=`python ... 'get_python_lib();''

So as a result just datadir should be separated from pythondir but otherwise
I agree with it all now, thanks for the explanation.


> Jan> Checked-in
> Jan> [archer-jankratochvil-python] ebd9eefcf0e56102c40cfd104eb1c290526b3f51 which
> 
> We're going to have to clear up this distinction with multiple
> branches.  I don't want us to lose changes due to branch confusion.
> 
> What if I merge master->archer-tromey-python again and we switch to that?

For the Fedora and [archer] branch there is a requirement of the merge with
[archer-jankratochvil-type-refcount] (for [archer-jankratochvil-vla]):
49f350efd1b7ea38798e77eb8951440160fd7c81
acd14fd258b498d2faa5dc34751c43da5c8e476f
7f4036e06538041e6bd97c5f57616517c003b3a5
- as the fixes show the [archer-tromey-python] branch can bring in a perfectly
  valid change which crashes on the merge with
  [archer-jankratochvil-type-refcount] due to different TYPE_OBJFILE.

There is also the problem of [archer-tromey-python] compatibility with
[archer-tromey-charset] due to the [archer-tromey-charset] change
f0fab6ed91c0d78127a52afc1965003efe5e64f5, fixed by:
807b1035c00186e947eab716ffd5d8993b9ca8a8
32697aed4550cbdc8ede26167bdca21f36ecb8f6
1d3229a480d990e3af9c4d53c385501fad3f7a42
64015dc38f646586318cbd09be01f93580a7aa96
- IMO [archer-tromey-python] should import
f0fab6ed91c0d78127a52afc1965003efe5e64f5 and redo the four fixes.

And the installation directories (sowere in:
ebd9eefcf0e56102c40cfd104eb1c290526b3f51

The branches compatibility fixes above (separate from the technical conflicts
resolution) have not been posted to <archer@sourceware.org> so far.


Regards,
Jan

Re: [python] Packacking - Python scripts directories

[Tom Tromey]

Jan> +  try:\n\
Jan> +    import gdb.__init__\n\
Jan> +  except ImportError:\n\
Jan> +    pass\n\

If it works, it is fine by me.  I don't recall if I tried this
approach.  In my view, this startup stuff has to hack around in Python
internals, so any working approach is probably ok.  If there's a
further reason to prefer one over the other, let's do it.

Jan> So currently IMO should be changed:
[...]

I'm working on a patch for this.

Tom