From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 6197738485B8 for ; Sat, 9 Jul 2022 15:52:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 6197738485B8 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-272-gWF1pPXhNUeiqHWOvbmoUg-1; Sat, 09 Jul 2022 11:51:52 -0400 X-MC-Unique: gWF1pPXhNUeiqHWOvbmoUg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E0ECC804184; Sat, 9 Jul 2022 15:51:51 +0000 (UTC) Received: from f36-1.lan (unknown [10.2.17.107]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5ADDC906B5; Sat, 9 Jul 2022 15:51:51 +0000 (UTC) From: Kevin Buettner To: gdb-patches@sourceware.org Cc: Kevin Buettner , Simon Marchi Subject: [PATCH v2] Handle Python 3.11 deprecation of PySys_SetPath and Py_SetProgramName Date: Sat, 9 Jul 2022 08:50:10 -0700 Message-Id: <20220709155009.176955-1-kevinb@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2022 15:52:08 -0000 Python 3.11 deprecates PySys_SetPath and Py_SetProgramName. The PyConfig API replaces these and other functions. This commit uses the PyConfig API to provide equivalent functionality while also preserving support for older versions of Python, i.e. those before Python 3.8. A beta version of Python 3.11 is available in Fedora Rawhide. Both Fedora 35 and Fedora 36 use Python 3.10, while Fedora 34 still used Python 3.9. I've tested these changes on Fedora 34, Fedora 36, and rawhide, though complete testing was not possible on rawhide due to a kernel bug. That being the case, I decided to enable the newer PyConfig API by testing PY_VERSION_HEX against 0x030a0000. This corresponds to Python 3.10. We could try to use the PyConfig API for Python versions as early as 3.8, but I'm reluctant to do this as there may have been PyConfig related bugs in earlier versions which have since been fixed. Recent linux distributions should have support for Python 3.10. This should be more than adequate for testing the new Python initialization code in GDB. Information about the PyConfig API as well as the motivation behind deprecating the old interface can be found at these links: https://github.com/python/cpython/issues/88279 https://peps.python.org/pep-0587/ https://docs.python.org/3.11/c-api/init_config.html This v2 commit also addresses several problems that Simon found in the v1 version. In v1, I had used Py_DontWriteBytecodeFlag in the new initialization code, but Simon pointed out that this global configuration variable will be deprecated in Python 3.12. This version of the patch no longer uses Py_DontWriteBytecodeFlag in the new initialization code. Additionally, both Py_DontWriteBytecodeFlag and Py_IgnoreEnvironmentFlag will no longer be used when building GDB against Python 3.10 or higher. While it's true that both of these global configuration variables are deprecated in Python 3.12, it makes sense to disable their use for gdb builds against 3.10 and higher since those are the versions for which the PyConfig API is now being used by GDB. (The PyConfig API includes different mechanisms for making the same settings afforded by use of the soon-to-be deprecated global configuration variables.) Simon also noted that PyConfig_Clear() would not have be called for one of the failure paths. I've fixed that problem and also made the rest of the "bail out" code more direct. In particular, PyConfig_Clear() will always be called, both for success and failure. --- gdb/python/python-internal.h | 5 ++ gdb/python/python.c | 88 ++++++++++++++++++++++++++++++++---- 2 files changed, 84 insertions(+), 9 deletions(-) diff --git a/gdb/python/python-internal.h b/gdb/python/python-internal.h index 5ff9989af83..3dae4e13a4c 100644 --- a/gdb/python/python-internal.h +++ b/gdb/python/python-internal.h @@ -177,6 +177,10 @@ gdb_PySys_GetObject (const char *name) #define PySys_GetObject gdb_PySys_GetObject +/* PySys_SetPath was deprecated in Python 3.11. Disable the deprecated + code for Python 3.10 and newer. */ +#if PY_VERSION_HEX < 0x030a0000 + /* PySys_SetPath's 'path' parameter was missing the 'const' qualifier before Python 3.6. Hence, we wrap it in a function to avoid errors when compiled with -Werror. */ @@ -190,6 +194,7 @@ gdb_PySys_SetPath (const GDB_PYSYS_SETPATH_CHAR *path) } #define PySys_SetPath gdb_PySys_SetPath +#endif /* Wrap PyGetSetDef to allow convenient construction with string literals. Unfortunately, PyGetSetDef's 'name' and 'doc' members diff --git a/gdb/python/python.c b/gdb/python/python.c index 8f526bba84e..0170e75d911 100644 --- a/gdb/python/python.c +++ b/gdb/python/python.c @@ -1820,8 +1820,14 @@ set_python_ignore_environment (const char *args, int from_tty, struct cmd_list_element *c) { #ifdef HAVE_PYTHON + /* Py_IgnoreEnvironmentFlag is deprecated in Python 3.12. Disable + its usage in Python 3.10 and above since the PyConfig mechanism + is now (also) used in 3.10 and higher. See do_start_initialization() + in this file. */ +#if PY_VERSION_HEX < 0x030a0000 Py_IgnoreEnvironmentFlag = python_ignore_environment ? 1 : 0; #endif +#endif } /* When this is turned on before Python is initialised then Python will @@ -1849,6 +1855,24 @@ show_python_dont_write_bytecode (struct ui_file *file, int from_tty, value); } +/* Return value to assign to PyConfig.write_bytecode or, when + negated (via !), Py_DontWriteBytecodeFlag. Py_DontWriteBytecodeFlag + is deprecated in Python 3.12. */ + +static int +python_write_bytecode () +{ + int wbc = 0; + + if (python_dont_write_bytecode == AUTO_BOOLEAN_AUTO) + wbc = (!python_ignore_environment + && getenv ("PYTHONDONTWRITEBYTECODE") != nullptr) ? 0 : 1; + else + wbc = python_dont_write_bytecode == AUTO_BOOLEAN_TRUE ? 0 : 1; + + return wbc; +} + /* Implement 'set python dont-write-bytecode'. This sets Python's internal flag no matter when the command is issued, however, if this is used after Py_Initialize has been called then many modules could already @@ -1859,13 +1883,13 @@ set_python_dont_write_bytecode (const char *args, int from_tty, struct cmd_list_element *c) { #ifdef HAVE_PYTHON - if (python_dont_write_bytecode == AUTO_BOOLEAN_AUTO) - Py_DontWriteBytecodeFlag - = (!python_ignore_environment - && getenv ("PYTHONDONTWRITEBYTECODE") != nullptr) ? 1 : 0; - else - Py_DontWriteBytecodeFlag - = python_dont_write_bytecode == AUTO_BOOLEAN_TRUE ? 1 : 0; + /* Py_DontWriteBytecodeFlag is deprecated in Python 3.12. Disable + its usage in Python 3.10 and above since the PyConfig mechanism + is now (also) used in 3.10 and higher. See do_start_initialization() + in this file. */ +#if PY_VERSION_HEX < 0x030a0000 + Py_DontWriteBytecodeFlag = !python_write_bytecode (); +#endif #endif /* HAVE_PYTHON */ } @@ -2001,16 +2025,51 @@ do_start_initialization () } setlocale (LC_ALL, oldloc.c_str ()); + /* Py_SetProgramName was deprecated in Python 3.11. Use PyConfig + mechanisms for Python 3.10 and newer. */ +#if PY_VERSION_HEX < 0x030a0000 /* Note that Py_SetProgramName expects the string it is passed to remain alive for the duration of the program's execution, so it is not freed after this call. */ Py_SetProgramName (progname_copy); - /* Define _gdb as a built-in module. */ PyImport_AppendInittab ("_gdb", init__gdb_module); -#endif + Py_Initialize (); +#else + PyConfig config; + bool fail = false; + + PyConfig_InitPythonConfig (&config); + PyStatus status = PyConfig_SetString (&config, &config.program_name, + progname_copy); + if (PyStatus_Exception (status)) + goto init_done; + + config.write_bytecode = python_write_bytecode (); + config.use_environment = !python_ignore_environment; + + status = PyConfig_Read (&config); + if (PyStatus_Exception (status)) + goto init_done; + + /* Define _gdb as a built-in module. */ + if (PyImport_AppendInittab ("_gdb", init__gdb_module) == -1) + { + fail = true; + goto init_done; + } + status = Py_InitializeFromConfig (&config); + +init_done: + PyConfig_Clear (&config); + if (PyStatus_Exception (status) || fail) + return false; +#endif +#else Py_Initialize (); +#endif + #if PY_VERSION_HEX < 0x03090000 /* PyEval_InitThreads became deprecated in Python 3.9 and will be removed in Python 3.11. Prior to Python 3.7, this call was @@ -2317,12 +2376,23 @@ do_initialize (const struct extension_language_defn *extlang) sys_path = PySys_GetObject ("path"); + /* PySys_SetPath was deprecated in Python 3.11. Disable this + deprecated code for Python 3.10 and newer. Also note that this + ifdef eliminates potential initialization of sys.path via + PySys_SetPath. My (kevinb's) understanding of PEP 587 suggests + that it's not necessary due to module_search_paths being + initialized to an empty list following any of the PyConfig + initialization functions. If it does turn out that some kind of + initialization is still needed, it should be added to the + PyConfig-based initialization in do_start_initialize(). */ +#if PY_VERSION_HEX < 0x030a0000 /* If sys.path is not defined yet, define it first. */ if (!(sys_path && PyList_Check (sys_path))) { PySys_SetPath (L""); sys_path = PySys_GetObject ("path"); } +#endif if (sys_path && PyList_Check (sys_path)) { gdbpy_ref<> pythondir (PyUnicode_FromString (gdb_pythondir.c_str ())); -- 2.36.1