[PATCH v3] Handle Python 3.11 deprecation of PySys_SetPath and Py_SetProgramName

[PATCH v3] Handle Python 3.11 deprecation of PySys_SetPath and Py_SetProgramName

[Kevin Buettner]

Python 3.11 deprecates PySys_SetPath and Py_SetProgramName.  The
PyConfig API replaces these and other functions.  This commit uses the
PyConfig API to provide equivalent functionality while also preserving
support for older versions of Python, i.e. those before Python 3.8.

A beta version of Python 3.11 is available in Fedora Rawhide.  Both
Fedora 35 and Fedora 36 use Python 3.10, while Fedora 34 still used
Python 3.9.  I've tested these changes on Fedora 34, Fedora 36, and
rawhide, though complete testing was not possible on rawhide due to
a kernel bug.  That being the case, I decided to enable the newer
PyConfig API by testing PY_VERSION_HEX against 0x030a0000.  This
corresponds to Python 3.10.

We could try to use the PyConfig API for Python versions as early as 3.8,
but I'm reluctant to do this as there may have been PyConfig related
bugs in earlier versions which have since been fixed.  Recent linux
distributions should have support for Python 3.10.  This should be
more than adequate for testing the new Python initialization code in
GDB.

Information about the PyConfig API as well as the motivation behind
deprecating the old interface can be found at these links:

https://github.com/python/cpython/issues/88279
https://peps.python.org/pep-0587/
https://docs.python.org/3.11/c-api/init_config.html

The v2 commit also addresses several problems that Simon found in
the v1 version.

In v1, I had used Py_DontWriteBytecodeFlag in the new initialization
code, but Simon pointed out that this global configuration variable
will be deprecated in Python 3.12.  This version of the patch no longer
uses Py_DontWriteBytecodeFlag in the new initialization code.
Additionally, both Py_DontWriteBytecodeFlag and Py_IgnoreEnvironmentFlag
will no longer be used when building GDB against Python 3.10 or higher.
While it's true that both of these global configuration variables are
deprecated in Python 3.12, it makes sense to disable their use for
gdb builds against 3.10 and higher since those are the versions for
which the PyConfig API is now being used by GDB.  (The PyConfig API
includes different mechanisms for making the same settings afforded
by use of the soon-to-be deprecated global configuration variables.)

Simon also noted that PyConfig_Clear() would not have be called for
one of the failure paths.  I've fixed that problem and also made the
rest of the "bail out" code more direct.  In particular,
PyConfig_Clear() will always be called, both for success and failure.

The v3 patch addresses some rebase conflicts related to module
initialization .  Commit 3acd9a692dd ("Make 'import gdb.events' work")
uses PyImport_ExtendInittab instead of PyImport_AppendInittab.  That
commit also initializes a struct for each module to import.  Both the
initialization and the call to were moved ahead of the ifdefs to avoid
having to replicate (at least some of) the code three times in various
portions of the ifdefs.

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28668
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=29287
---
 gdb/python/python-internal.h |  5 ++
 gdb/python/python.c          | 99 +++++++++++++++++++++++++++++-------
 2 files changed, 86 insertions(+), 18 deletions(-)

diff --git a/gdb/python/python-internal.h b/gdb/python/python-internal.h
index 551438c4709..08749d14200 100644
--- a/gdb/python/python-internal.h
+++ b/gdb/python/python-internal.h
@@ -177,6 +177,10 @@ gdb_PySys_GetObject (const char *name)
 
 #define PySys_GetObject gdb_PySys_GetObject
 
+/* PySys_SetPath was deprecated in Python 3.11.  Disable the deprecated
+   code for Python 3.10 and newer.  */
+#if PY_VERSION_HEX < 0x030a0000
+
 /* PySys_SetPath's 'path' parameter was missing the 'const' qualifier
    before Python 3.6.  Hence, we wrap it in a function to avoid errors
    when compiled with -Werror.  */
@@ -190,6 +194,7 @@ gdb_PySys_SetPath (const GDB_PYSYS_SETPATH_CHAR *path)
 }
 
 #define PySys_SetPath gdb_PySys_SetPath
+#endif
 
 /* Wrap PyGetSetDef to allow convenient construction with string
    literals.  Unfortunately, PyGetSetDef's 'name' and 'doc' members
diff --git a/gdb/python/python.c b/gdb/python/python.c
index c75896dc324..c719e3dc90c 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -1820,8 +1820,14 @@ set_python_ignore_environment (const char *args, int from_tty,
 			       struct cmd_list_element *c)
 {
 #ifdef HAVE_PYTHON
+  /* Py_IgnoreEnvironmentFlag is deprecated in Python 3.12.  Disable
+     its usage in Python 3.10 and above since the PyConfig mechanism
+     is now (also) used in 3.10 and higher.  See do_start_initialization()
+     in this file.  */
+#if PY_VERSION_HEX < 0x030a0000
   Py_IgnoreEnvironmentFlag = python_ignore_environment ? 1 : 0;
 #endif
+#endif
 }
 
 /* When this is turned on before Python is initialised then Python will
@@ -1849,6 +1855,24 @@ show_python_dont_write_bytecode (struct ui_file *file, int from_tty,
 		value);
 }
 
+/* Return value to assign to PyConfig.write_bytecode or, when
+   negated (via !), Py_DontWriteBytecodeFlag.  Py_DontWriteBytecodeFlag
+   is deprecated in Python 3.12.  */
+
+static int
+python_write_bytecode ()
+{
+  int wbc = 0;
+
+  if (python_dont_write_bytecode == AUTO_BOOLEAN_AUTO)
+    wbc = (!python_ignore_environment
+	    && getenv ("PYTHONDONTWRITEBYTECODE") != nullptr) ? 0 : 1;
+  else
+    wbc = python_dont_write_bytecode == AUTO_BOOLEAN_TRUE ? 0 : 1;
+
+  return wbc;
+}
+
 /* Implement 'set python dont-write-bytecode'.  This sets Python's internal
    flag no matter when the command is issued, however, if this is used
    after Py_Initialize has been called then many modules could already
@@ -1859,13 +1883,13 @@ set_python_dont_write_bytecode (const char *args, int from_tty,
 				struct cmd_list_element *c)
 {
 #ifdef HAVE_PYTHON
-  if (python_dont_write_bytecode == AUTO_BOOLEAN_AUTO)
-    Py_DontWriteBytecodeFlag
-      = (!python_ignore_environment
-	 && getenv ("PYTHONDONTWRITEBYTECODE") != nullptr) ? 1 : 0;
-  else
-    Py_DontWriteBytecodeFlag
-      = python_dont_write_bytecode == AUTO_BOOLEAN_TRUE ? 1 : 0;
+  /* Py_DontWriteBytecodeFlag is deprecated in Python 3.12.  Disable
+     its usage in Python 3.10 and above since the PyConfig mechanism
+     is now (also) used in 3.10 and higher.  See do_start_initialization()
+     in this file.  */
+#if PY_VERSION_HEX < 0x030a0000
+  Py_DontWriteBytecodeFlag = !python_write_bytecode ();
+#endif
 #endif /* HAVE_PYTHON */
 }
 
@@ -1970,6 +1994,18 @@ gdbpy_gdb_exiting (int exit_code)
 static bool
 do_start_initialization ()
 {
+  /* Define all internal modules.  These are all imported (and thus
+     created) during initialization.  */
+  struct _inittab mods[] =
+  {
+    { "_gdb", init__gdb_module },
+    { "_gdbevents", gdbpy_events_mod_func },
+    { nullptr, nullptr }
+  };
+
+  if (PyImport_ExtendInittab (mods) < 0)
+    return false;
+
 #ifdef WITH_PYTHON_PATH
   /* Work around problem where python gets confused about where it is,
      and then can't find its libraries, etc.
@@ -2001,25 +2037,41 @@ do_start_initialization ()
     }
   setlocale (LC_ALL, oldloc.c_str ());
 
+  /* Py_SetProgramName was deprecated in Python 3.11.  Use PyConfig
+     mechanisms for Python 3.10 and newer.  */
+#if PY_VERSION_HEX < 0x030a0000
   /* Note that Py_SetProgramName expects the string it is passed to
      remain alive for the duration of the program's execution, so
      it is not freed after this call.  */
   Py_SetProgramName (progname_copy);
-#endif
+  Py_Initialize ();
+#else
+  PyConfig config;
 
-  /* Define all internal modules.  These are all imported (and thus
-     created) during initialization.  */
-  struct _inittab mods[3] =
-  {
-    { "_gdb", init__gdb_module },
-    { "_gdbevents", gdbpy_events_mod_func },
-    { nullptr, nullptr }
-  };
+  PyConfig_InitPythonConfig (&config);
+  PyStatus status = PyConfig_SetString (&config, &config.program_name,
+                                        progname_copy);
+  if (PyStatus_Exception (status))
+    goto init_done;
 
-  if (PyImport_ExtendInittab (mods) < 0)
-    return false;
+  config.write_bytecode = python_write_bytecode ();
+  config.use_environment = !python_ignore_environment;
 
+  status = PyConfig_Read (&config);
+  if (PyStatus_Exception (status))
+    goto init_done;
+
+  status = Py_InitializeFromConfig (&config);
+
+init_done:
+  PyConfig_Clear (&config);
+  if (PyStatus_Exception (status))
+    return false;
+#endif
+#else
   Py_Initialize ();
+#endif
+
 #if PY_VERSION_HEX < 0x03090000
   /* PyEval_InitThreads became deprecated in Python 3.9 and will
      be removed in Python 3.11.  Prior to Python 3.7, this call was
@@ -2325,12 +2377,23 @@ do_initialize (const struct extension_language_defn *extlang)
 
   sys_path = PySys_GetObject ("path");
 
+  /* PySys_SetPath was deprecated in Python 3.11.  Disable this
+     deprecated code for Python 3.10 and newer.  Also note that this
+     ifdef eliminates potential initialization of sys.path via
+     PySys_SetPath.  My (kevinb's) understanding of PEP 587 suggests
+     that it's not necessary due to module_search_paths being
+     initialized to an empty list following any of the PyConfig
+     initialization functions.  If it does turn out that some kind of
+     initialization is still needed, it should be added to the
+     PyConfig-based initialization in do_start_initialize().  */
+#if PY_VERSION_HEX < 0x030a0000
   /* If sys.path is not defined yet, define it first.  */
   if (!(sys_path && PyList_Check (sys_path)))
     {
       PySys_SetPath (L"");
       sys_path = PySys_GetObject ("path");
     }
+#endif
   if (sys_path && PyList_Check (sys_path))
     {
       gdbpy_ref<> pythondir (PyUnicode_FromString (gdb_pythondir.c_str ()));
-- 
2.36.1

Re: [PATCH v3] Handle Python 3.11 deprecation of PySys_SetPath and Py_SetProgramName

[Tom Tromey]

>>>>> "Kevin" == Kevin Buettner via Gdb-patches <gdb-patches@sourceware.org> writes:

Kevin> A beta version of Python 3.11 is available in Fedora Rawhide.  Both
Kevin> Fedora 35 and Fedora 36 use Python 3.10, while Fedora 34 still used
Kevin> Python 3.9.  I've tested these changes on Fedora 34, Fedora 36, and
Kevin> rawhide, though complete testing was not possible on rawhide due to
Kevin> a kernel bug.  That being the case, I decided to enable the newer
Kevin> PyConfig API by testing PY_VERSION_HEX against 0x030a0000.  This
Kevin> corresponds to Python 3.10.

This looks good to me.  Thank you.

Tom

Re: [PATCH v3] Handle Python 3.11 deprecation of PySys_SetPath and Py_SetProgramName

[Simon Marchi]

> @@ -1849,6 +1855,24 @@ show_python_dont_write_bytecode (struct ui_file *file, int from_tty,
>  		value);
>  }
>  
> +/* Return value to assign to PyConfig.write_bytecode or, when
> +   negated (via !), Py_DontWriteBytecodeFlag.  Py_DontWriteBytecodeFlag
> +   is deprecated in Python 3.12.  */
> +
> +static int
> +python_write_bytecode ()
> +{
> +  int wbc = 0;
> +
> +  if (python_dont_write_bytecode == AUTO_BOOLEAN_AUTO)
> +    wbc = (!python_ignore_environment
> +	    && getenv ("PYTHONDONTWRITEBYTECODE") != nullptr) ? 0 : 1;

I think it predates your patch (so, orthogonal), but the Python doc says:

  If this is set to a non-empty string, Python won’t try to write .pyc
  files on the import of source modules.

So I think we would need a "non-empty string" check here?

Simon

Re: [PATCH v3] Handle Python 3.11 deprecation of PySys_SetPath and Py_SetProgramName

[Kevin Buettner]

On Mon, 18 Jul 2022 11:49:48 -0400
Simon Marchi <simark@simark.ca> wrote:

> > @@ -1849,6 +1855,24 @@ show_python_dont_write_bytecode (struct ui_file *file, int from_tty,
> >  		value);
> >  }
> >  
> > +/* Return value to assign to PyConfig.write_bytecode or, when
> > +   negated (via !), Py_DontWriteBytecodeFlag.  Py_DontWriteBytecodeFlag
> > +   is deprecated in Python 3.12.  */
> > +
> > +static int
> > +python_write_bytecode ()
> > +{
> > +  int wbc = 0;
> > +
> > +  if (python_dont_write_bytecode == AUTO_BOOLEAN_AUTO)
> > +    wbc = (!python_ignore_environment
> > +	    && getenv ("PYTHONDONTWRITEBYTECODE") != nullptr) ? 0 : 1;  
> 
> I think it predates your patch (so, orthogonal), but the Python doc says:
> 
>   If this is set to a non-empty string, Python won't try to write .pyc
>   files on the import of source modules.
> 
> So I think we would need a "non-empty string" check here?

I agree.

I've pushed the v3 deprecation patch and have posted a new/separate
patch which fixes the problem you spotted.  It also fixes the set/show
python dont-write-bytecode command documentation.

See:

https://sourceware.org/pipermail/gdb-patches/2022-July/190912.html

Kevin