[PATCH] Make GDB compile with Python 3 on MinGW

[PATCH] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

[Despite my comment on IRC, I made this approach work!]

PyFile_FromString and PyFile_AsFile have been removed in Python 3.
There is no obvious replacement that works here, and we can't just
pass our FILE* to a DLL in Windows because it may use a different
C runtime.

So we just call a Python function which reads and executes file
contents. Care must be taken to execute it in the context of
__main__.

Tested by inverting the ifdef and running the testsuite on Debian
Linux (even without the patch, I failed at running the testsuite
on Windows).

gdb/ChangeLog:

2019-08-13  Christian Biesinger  <cbiesinger@google.com>

	* python/lib/gdb/__init__.py: Add an execute_file function.
	* python/python.c (python_run_simple_file): Call gdb.execute_file
	on Windows.
---
 gdb/python/lib/gdb/__init__.py | 23 +++++++++++++++++++++++
 gdb/python/python.c            | 18 +++++++++++++-----
 2 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
index af74df80c8..f1adc5ffbe 100644
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -106,6 +106,29 @@ def execute_unwinders(pending_frame):
 
     return None
 
+def execute_file(filepath):
+    """This function is used to replace Python 2's PyRun_SimpleFile.
+
+    Loads and executes the given file.
+
+    We could use the runpy module, but its documentation says:
+    "Furthermore, any functions and classes defined by the executed code are
+    not guaranteed to work correctly after a runpy function has returned."
+    """
+    globals = sys.modules['__main__'].__dict__
+    set_file = False
+    if not hasattr(globals, '__file__'):
+        globals['__file__'] = filepath
+        set_file = True
+    try:
+        with open(filepath, 'rb') as file:
+            # We pass globals also as locals to match what Python does
+            # in PyRun_SimpleFile.
+            exec(compile(file.read(), filepath, 'exec'), globals, globals)
+    finally:
+        if set_file:
+            del globals['__file__']
+
 
 # Convenience variable to GDB's python directory
 PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
diff --git a/gdb/python/python.c b/gdb/python/python.c
index 162470dcc0..18d0562212 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -339,14 +339,22 @@ python_run_simple_file (FILE *file, const char *filename)
   /* Because we have a string for a filename, and are using Python to
      open the file, we need to expand any tilde in the path first.  */
   gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
-  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
-  if (python_file == NULL)
+
+  if (gdb_python_module == nullptr
+      || ! PyObject_HasAttrString (gdb_python_module, "execute_file"))
     {
-      gdbpy_print_stack ();
-      error (_("Error while opening file: %s"), full_path.get ());
+      error (_("Installation error: gdb.execute_file function is missing"));
+      return;
     }
 
-  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
+  gdbpy_ref<> return_value
+    (PyObject_CallMethod (gdb_python_module, "execute_file", "s", full_path.get ()));
+  if (return_value == nullptr)
+    {
+      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
+         behavior of the non-Windows codepath.  */
+      PyErr_PrintEx(0);
+    }
 
 #endif /* _WIN32 */
 }
-- 
2.23.0.rc1.153.gdeed80330f-goog

[PATCH v2] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

[Now with the comment updated too]

PyFile_FromString and PyFile_AsFile have been removed in Python 3.
There is no obvious replacement that works here, and we can't just
pass our FILE* to a DLL in Windows because it may use a different
C runtime.

So we just call a Python function which reads and executes file
contents. Care must be taken to execute it in the context of
__main__.

Tested by inverting the ifdef and running the testsuite on Debian
Linux (even without the patch, I failed at running the testsuite
on Windows).

gdb/ChangeLog:

2019-08-13  Christian Biesinger  <cbiesinger@google.com>

	* python/lib/gdb/__init__.py: Add an execute_file function.
	* python/python.c (python_run_simple_file): Call gdb.execute_file
	on Windows.
---
 gdb/python/lib/gdb/__init__.py | 23 +++++++++++++++++++++++
 gdb/python/python.c            | 23 +++++++++++++++--------
 2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
index af74df80c8..f1adc5ffbe 100644
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -106,6 +106,29 @@ def execute_unwinders(pending_frame):
 
     return None
 
+def execute_file(filepath):
+    """This function is used to replace Python 2's PyRun_SimpleFile.
+
+    Loads and executes the given file.
+
+    We could use the runpy module, but its documentation says:
+    "Furthermore, any functions and classes defined by the executed code are
+    not guaranteed to work correctly after a runpy function has returned."
+    """
+    globals = sys.modules['__main__'].__dict__
+    set_file = False
+    if not hasattr(globals, '__file__'):
+        globals['__file__'] = filepath
+        set_file = True
+    try:
+        with open(filepath, 'rb') as file:
+            # We pass globals also as locals to match what Python does
+            # in PyRun_SimpleFile.
+            exec(compile(file.read(), filepath, 'exec'), globals, globals)
+    finally:
+        if set_file:
+            del globals['__file__']
+
 
 # Convenience variable to GDB's python directory
 PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
diff --git a/gdb/python/python.c b/gdb/python/python.c
index 162470dcc0..617bc0b84c 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -323,9 +323,8 @@ python_interactive_command (const char *arg, int from_tty)
    A FILE * from one runtime does not necessarily operate correctly in
    the other runtime.
 
-   To work around this potential issue, we create on Windows hosts the
-   FILE object using Python routines, thus making sure that it is
-   compatible with the Python library.  */
+   To work around this potential issue, we run code in Python to load
+   the script.  */
 
 static void
 python_run_simple_file (FILE *file, const char *filename)
@@ -339,14 +338,22 @@ python_run_simple_file (FILE *file, const char *filename)
   /* Because we have a string for a filename, and are using Python to
      open the file, we need to expand any tilde in the path first.  */
   gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
-  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
-  if (python_file == NULL)
+
+  if (gdb_python_module == nullptr
+      || ! PyObject_HasAttrString (gdb_python_module, "execute_file"))
     {
-      gdbpy_print_stack ();
-      error (_("Error while opening file: %s"), full_path.get ());
+      error (_("Installation error: gdb.execute_file function is missing"));
+      return;
     }
 
-  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
+  gdbpy_ref<> return_value
+    (PyObject_CallMethod (gdb_python_module, "execute_file", "s", full_path.get ()));
+  if (return_value == nullptr)
+    {
+      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
+         behavior of the non-Windows codepath.  */
+      PyErr_PrintEx(0);
+    }
 
 #endif /* _WIN32 */
 }
-- 
2.23.0.rc1.153.gdeed80330f-goog

Re: [PATCH v2] Make GDB compile with Python 3 on MinGW

[Simon Marchi]

On 2019-08-14 12:06 p.m., Christian Biesinger via gdb-patches wrote:
> [Now with the comment updated too]
> 
> PyFile_FromString and PyFile_AsFile have been removed in Python 3.
> There is no obvious replacement that works here, and we can't just
> pass our FILE* to a DLL in Windows because it may use a different
> C runtime.
> 
> So we just call a Python function which reads and executes file
> contents. Care must be taken to execute it in the context of
> __main__.
> 
> Tested by inverting the ifdef and running the testsuite on Debian
> Linux (even without the patch, I failed at running the testsuite
> on Windows).

I haven't deciphered the Python part yet, but here are some early comments.

Did you test with both Python 2 and 3?

> gdb/ChangeLog:
> 
> 2019-08-13  Christian Biesinger  <cbiesinger@google.com>
> 
> 	* python/lib/gdb/__init__.py: Add an execute_file function.
> 	* python/python.c (python_run_simple_file): Call gdb.execute_file
> 	on Windows.
> ---
>  gdb/python/lib/gdb/__init__.py | 23 +++++++++++++++++++++++
>  gdb/python/python.c            | 23 +++++++++++++++--------
>  2 files changed, 38 insertions(+), 8 deletions(-)
> 
> diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
> index af74df80c8..f1adc5ffbe 100644
> --- a/gdb/python/lib/gdb/__init__.py
> +++ b/gdb/python/lib/gdb/__init__.py
> @@ -106,6 +106,29 @@ def execute_unwinders(pending_frame):
>  
>      return None
>  
> +def execute_file(filepath):

I would suggest naming this "_execute_file", to emphasize that's internal,
not to be used by users.

> +    """This function is used to replace Python 2's PyRun_SimpleFile.
> +
> +    Loads and executes the given file.
> +
> +    We could use the runpy module, but its documentation says:
> +    "Furthermore, any functions and classes defined by the executed code are
> +    not guaranteed to work correctly after a runpy function has returned."
> +    """
> +    globals = sys.modules['__main__'].__dict__
> +    set_file = False
> +    if not hasattr(globals, '__file__'):
> +        globals['__file__'] = filepath
> +        set_file = True

What does setting __file__ help with? A comment explaining why this is needed
would be nice.

> +    try:
> +        with open(filepath, 'rb') as file:
> +            # We pass globals also as locals to match what Python does
> +            # in PyRun_SimpleFile.
> +            exec(compile(file.read(), filepath, 'exec'), globals, globals)

Could you split this line (the exec and compile) in two?  If there is an error
coming out of either of them, and we get a backtrace, it will be easier to know
which one it is if they are on separate lines.

> +    finally:
> +        if set_file:
> +            del globals['__file__']
> +
>  
>  # Convenience variable to GDB's python directory
>  PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
> diff --git a/gdb/python/python.c b/gdb/python/python.c
> index 162470dcc0..617bc0b84c 100644
> --- a/gdb/python/python.c
> +++ b/gdb/python/python.c
> @@ -323,9 +323,8 @@ python_interactive_command (const char *arg, int from_tty)
>     A FILE * from one runtime does not necessarily operate correctly in
>     the other runtime.
>  
> -   To work around this potential issue, we create on Windows hosts the
> -   FILE object using Python routines, thus making sure that it is
> -   compatible with the Python library.  */
> +   To work around this potential issue, we run code in Python to load
> +   the script.  */
>  
>  static void
>  python_run_simple_file (FILE *file, const char *filename)
> @@ -339,14 +338,22 @@ python_run_simple_file (FILE *file, const char *filename)
>    /* Because we have a string for a filename, and are using Python to
>       open the file, we need to expand any tilde in the path first.  */
>    gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
> -  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
> -  if (python_file == NULL)
> +
> +  if (gdb_python_module == nullptr
> +      || ! PyObject_HasAttrString (gdb_python_module, "execute_file"))
>      {
> -      gdbpy_print_stack ();
> -      error (_("Error while opening file: %s"), full_path.get ());
> +      error (_("Installation error: gdb.execute_file function is missing"));
> +      return;
>      }
>  
> -  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
> +  gdbpy_ref<> return_value
> +    (PyObject_CallMethod (gdb_python_module, "execute_file", "s", full_path.get ()));

Wrap the line above to fit in 80 columns:

  gdbpy_ref<> return_value
    (PyObject_CallMethod (gdb_python_module, "execute_file", "s",
			  full_path.get ()));

> +  if (return_value == nullptr)
> +    {
> +      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
> +         behavior of the non-Windows codepath.  */
> +      PyErr_PrintEx(0);

I don't think PyErr_PrintEx clears the Python error indicator, and I think we want
it cleared when we exit this function, so should we call PyErr_Clear here?

Simon

Re: [PATCH v2] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

On Wed, Aug 14, 2019 at 9:58 PM Simon Marchi <simark@simark.ca> wrote:
>
> On 2019-08-14 12:06 p.m., Christian Biesinger via gdb-patches wrote:
> > [Now with the comment updated too]
> >
> > PyFile_FromString and PyFile_AsFile have been removed in Python 3.
> > There is no obvious replacement that works here, and we can't just
> > pass our FILE* to a DLL in Windows because it may use a different
> > C runtime.
> >
> > So we just call a Python function which reads and executes file
> > contents. Care must be taken to execute it in the context of
> > __main__.
> >
> > Tested by inverting the ifdef and running the testsuite on Debian
> > Linux (even without the patch, I failed at running the testsuite
> > on Windows).
>
> I haven't deciphered the Python part yet, but here are some early comments.
>
> Did you test with both Python 2 and 3?

I did, yeah.

> > gdb/ChangeLog:
> >
> > 2019-08-13  Christian Biesinger  <cbiesinger@google.com>
> >
> >       * python/lib/gdb/__init__.py: Add an execute_file function.
> >       * python/python.c (python_run_simple_file): Call gdb.execute_file
> >       on Windows.
> > ---
> >  gdb/python/lib/gdb/__init__.py | 23 +++++++++++++++++++++++
> >  gdb/python/python.c            | 23 +++++++++++++++--------
> >  2 files changed, 38 insertions(+), 8 deletions(-)
> >
> > diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
> > index af74df80c8..f1adc5ffbe 100644
> > --- a/gdb/python/lib/gdb/__init__.py
> > +++ b/gdb/python/lib/gdb/__init__.py
> > @@ -106,6 +106,29 @@ def execute_unwinders(pending_frame):
> >
> >      return None
> >
> > +def execute_file(filepath):
>
> I would suggest naming this "_execute_file", to emphasize that's internal,
> not to be used by users.

Done (and sent a separate patch to rename other internal functions)

> > +    """This function is used to replace Python 2's PyRun_SimpleFile.
> > +
> > +    Loads and executes the given file.
> > +
> > +    We could use the runpy module, but its documentation says:
> > +    "Furthermore, any functions and classes defined by the executed code are
> > +    not guaranteed to work correctly after a runpy function has returned."
> > +    """
> > +    globals = sys.modules['__main__'].__dict__
> > +    set_file = False
> > +    if not hasattr(globals, '__file__'):
> > +        globals['__file__'] = filepath
> > +        set_file = True
>
> What does setting __file__ help with? A comment explaining why this is needed
> would be nice.

Done

> > +    try:
> > +        with open(filepath, 'rb') as file:
> > +            # We pass globals also as locals to match what Python does
> > +            # in PyRun_SimpleFile.
> > +            exec(compile(file.read(), filepath, 'exec'), globals, globals)
>
> Could you split this line (the exec and compile) in two?  If there is an error
> coming out of either of them, and we get a backtrace, it will be easier to know
> which one it is if they are on separate lines.

Done

> > +    finally:
> > +        if set_file:
> > +            del globals['__file__']
> > +
> >
> >  # Convenience variable to GDB's python directory
> >  PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
> > diff --git a/gdb/python/python.c b/gdb/python/python.c
> > index 162470dcc0..617bc0b84c 100644
> > --- a/gdb/python/python.c
> > +++ b/gdb/python/python.c
> > @@ -323,9 +323,8 @@ python_interactive_command (const char *arg, int from_tty)
> >     A FILE * from one runtime does not necessarily operate correctly in
> >     the other runtime.
> >
> > -   To work around this potential issue, we create on Windows hosts the
> > -   FILE object using Python routines, thus making sure that it is
> > -   compatible with the Python library.  */
> > +   To work around this potential issue, we run code in Python to load
> > +   the script.  */
> >
> >  static void
> >  python_run_simple_file (FILE *file, const char *filename)
> > @@ -339,14 +338,22 @@ python_run_simple_file (FILE *file, const char *filename)
> >    /* Because we have a string for a filename, and are using Python to
> >       open the file, we need to expand any tilde in the path first.  */
> >    gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
> > -  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
> > -  if (python_file == NULL)
> > +
> > +  if (gdb_python_module == nullptr
> > +      || ! PyObject_HasAttrString (gdb_python_module, "execute_file"))
> >      {
> > -      gdbpy_print_stack ();
> > -      error (_("Error while opening file: %s"), full_path.get ());
> > +      error (_("Installation error: gdb.execute_file function is missing"));
> > +      return;
> >      }
> >
> > -  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
> > +  gdbpy_ref<> return_value
> > +    (PyObject_CallMethod (gdb_python_module, "execute_file", "s", full_path.get ()));
>
> Wrap the line above to fit in 80 columns:
>
>   gdbpy_ref<> return_value
>     (PyObject_CallMethod (gdb_python_module, "execute_file", "s",
>                           full_path.get ()));

Oops, done.

> > +  if (return_value == nullptr)
> > +    {
> > +      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
> > +         behavior of the non-Windows codepath.  */
> > +      PyErr_PrintEx(0);
>
> I don't think PyErr_PrintEx clears the Python error indicator, and I think we want
> it cleared when we exit this function, so should we call PyErr_Clear here?

It does, per https://docs.python.org/3/c-api/exceptions.html and
https://docs.python.org/2/c-api/exceptions.html

Christian

[PATCH] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

[Updated per Simon's comments]

PyFile_FromString and PyFile_AsFile have been removed in Python 3.
There is no obvious replacement that works here, and we can't just
pass our FILE* to a DLL in Windows because it may use a different
C runtime.

So we just call a Python function which reads and executes file
contents. Care must be taken to execute it in the context of
__main__.

Tested by inverting the ifdef and running the testsuite on Debian
Linux (even without the patch, I failed at running the testsuite
on Windows). I did test with both Python 2 and 3.

gdb/ChangeLog:

2019-08-13  Christian Biesinger  <cbiesinger@google.com>

	* python/lib/gdb/__init__.py: Add an execute_file function.
	* python/python.c (python_run_simple_file): Call gdb.execute_file
	on Windows.
---
 gdb/python/lib/gdb/__init__.py | 26 ++++++++++++++++++++++++++
 gdb/python/python.c            | 24 ++++++++++++++++--------
 2 files changed, 42 insertions(+), 8 deletions(-)

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
index af74df80c8..d1a49499b0 100644
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -106,6 +106,32 @@ def execute_unwinders(pending_frame):
 
     return None
 
+def _execute_file(filepath):
+    """This function is used to replace Python 2's PyRun_SimpleFile.
+
+    Loads and executes the given file.
+
+    We could use the runpy module, but its documentation says:
+    "Furthermore, any functions and classes defined by the executed code are
+    not guaranteed to work correctly after a runpy function has returned."
+    """
+    globals = sys.modules['__main__'].__dict__
+    set_file = False
+    # Set file (if not set) so that the imported file can use it (e.g. to
+    # access file-relative paths). This matches what PyRun_SimpleFile does.
+    if not hasattr(globals, '__file__'):
+        globals['__file__'] = filepath
+        set_file = True
+    try:
+        with open(filepath, 'rb') as file:
+            # We pass globals also as locals to match what Python does
+            # in PyRun_SimpleFile.
+	    compiled = compile(file.read(), filepath, 'exec')
+            exec(compiled, globals, globals)
+    finally:
+        if set_file:
+            del globals['__file__']
+
 
 # Convenience variable to GDB's python directory
 PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
diff --git a/gdb/python/python.c b/gdb/python/python.c
index 162470dcc0..0dd8dd5f35 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -323,9 +323,8 @@ python_interactive_command (const char *arg, int from_tty)
    A FILE * from one runtime does not necessarily operate correctly in
    the other runtime.
 
-   To work around this potential issue, we create on Windows hosts the
-   FILE object using Python routines, thus making sure that it is
-   compatible with the Python library.  */
+   To work around this potential issue, we run code in Python to load
+   the script.  */
 
 static void
 python_run_simple_file (FILE *file, const char *filename)
@@ -339,14 +338,23 @@ python_run_simple_file (FILE *file, const char *filename)
   /* Because we have a string for a filename, and are using Python to
      open the file, we need to expand any tilde in the path first.  */
   gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
-  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
-  if (python_file == NULL)
+
+  if (gdb_python_module == nullptr
+      || ! PyObject_HasAttrString (gdb_python_module, "_execute_file"))
     {
-      gdbpy_print_stack ();
-      error (_("Error while opening file: %s"), full_path.get ());
+      error (_("Installation error: gdb.execute_file function is missing"));
+      return;
     }
 
-  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
+  gdbpy_ref<> return_value
+    (PyObject_CallMethod (gdb_python_module, "_execute_file", "s",
+			  full_path.get ()));
+  if (return_value == nullptr)
+    {
+      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
+         behavior of the non-Windows codepath.  */
+      PyErr_PrintEx(0);
+    }
 
 #endif /* _WIN32 */
 }
-- 
2.23.0.rc1.153.gdeed80330f-goog

[PATCH v3] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

[Fixes two minor typos]

PyFile_FromString and PyFile_AsFile have been removed in Python 3.
There is no obvious replacement that works here, and we can't just
pass our FILE* to a DLL in Windows because it may use a different
C runtime.

So we just call a Python function which reads and executes file
contents. Care must be taken to execute it in the context of
__main__.

Tested by inverting the ifdef and running the testsuite on Debian
Linux (even without the patch, I failed at running the testsuite
on Windows). I did test with both Python 2 and 3.

gdb/ChangeLog:

2019-08-13  Christian Biesinger  <cbiesinger@google.com>

	* python/lib/gdb/__init__.py: Add an execute_file function.
	* python/python.c (python_run_simple_file): Call gdb.execute_file
	on Windows.
---
 gdb/python/lib/gdb/__init__.py | 26 ++++++++++++++++++++++++++
 gdb/python/python.c            | 26 +++++++++++++++++---------
 2 files changed, 43 insertions(+), 9 deletions(-)

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
index af74df80c8..afe5b08f3a 100644
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -106,6 +106,32 @@ def execute_unwinders(pending_frame):
 
     return None
 
+def _execute_file(filepath):
+    """This function is used to replace Python 2's PyRun_SimpleFile.
+
+    Loads and executes the given file.
+
+    We could use the runpy module, but its documentation says:
+    "Furthermore, any functions and classes defined by the executed code are
+    not guaranteed to work correctly after a runpy function has returned."
+    """
+    globals = sys.modules['__main__'].__dict__
+    set_file = False
+    # Set file (if not set) so that the imported file can use it (e.g. to
+    # access file-relative paths). This matches what PyRun_SimpleFile does.
+    if not hasattr(globals, '__file__'):
+        globals['__file__'] = filepath
+        set_file = True
+    try:
+        with open(filepath, 'rb') as file:
+            # We pass globals also as locals to match what Python does
+            # in PyRun_SimpleFile.
+            compiled = compile(file.read(), filepath, 'exec')
+            exec(compiled, globals, globals)
+    finally:
+        if set_file:
+            del globals['__file__']
+
 
 # Convenience variable to GDB's python directory
 PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
diff --git a/gdb/python/python.c b/gdb/python/python.c
index 162470dcc0..46aedba0ed 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -323,14 +323,13 @@ python_interactive_command (const char *arg, int from_tty)
    A FILE * from one runtime does not necessarily operate correctly in
    the other runtime.
 
-   To work around this potential issue, we create on Windows hosts the
-   FILE object using Python routines, thus making sure that it is
-   compatible with the Python library.  */
+   To work around this potential issue, we run code in Python to load
+   the script.  */
 
 static void
 python_run_simple_file (FILE *file, const char *filename)
 {
-#ifndef _WIN32
+#ifdef _WIN32
 
   PyRun_SimpleFile (file, filename);
 
@@ -339,14 +338,23 @@ python_run_simple_file (FILE *file, const char *filename)
   /* Because we have a string for a filename, and are using Python to
      open the file, we need to expand any tilde in the path first.  */
   gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
-  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
-  if (python_file == NULL)
+
+  if (gdb_python_module == nullptr
+      || ! PyObject_HasAttrString (gdb_python_module, "_execute_file"))
     {
-      gdbpy_print_stack ();
-      error (_("Error while opening file: %s"), full_path.get ());
+      error (_("Installation error: gdb._execute_file function is missing"));
+      return;
     }
 
-  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
+  gdbpy_ref<> return_value
+    (PyObject_CallMethod (gdb_python_module, "_execute_file", "s",
+			  full_path.get ()));
+  if (return_value == nullptr)
+    {
+      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
+         behavior of the non-Windows codepath.  */
+      PyErr_PrintEx(0);
+    }
 
 #endif /* _WIN32 */
 }
-- 
2.23.0.rc1.153.gdeed80330f-goog

Re: [PATCH v3] Make GDB compile with Python 3 on MinGW

[Pedro Alves]

On 8/15/19 6:42 PM, Christian Biesinger via gdb-patches wrote:
> +      error (_("Installation error: gdb._execute_file function is missing"));
> +      return;

That "return" statement is dead code, error throws an exception.

Thanks,
Pedro Alves

Re: [PATCH v2] Make GDB compile with Python 3 on MinGW

[Simon Marchi]

On 2019-08-15 1:14 p.m., Christian Biesinger wrote:
>> I don't think PyErr_PrintEx clears the Python error indicator, and I think we want
>> it cleared when we exit this function, so should we call PyErr_Clear here?
> 
> It does, per https://docs.python.org/3/c-api/exceptions.html and
> https://docs.python.org/2/c-api/exceptions.html

Ah yeah, I'm blond: "and clear the error indicator".

Simon

[PATCH v4] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

PyFile_FromString and PyFile_AsFile have been removed in Python 3.
There is no obvious replacement that works here, and we can't just
pass our FILE* to a DLL in Windows because it may use a different
C runtime.

So we just call a Python function which reads and executes file
contents. Care must be taken to execute it in the context of
__main__.

Tested by inverting the ifdef and running the testsuite on Debian
Linux (even without the patch, I failed at running the testsuite
on Windows). I did test with both Python 2 and 3.

gdb/ChangeLog:

2019-08-13  Christian Biesinger  <cbiesinger@google.com>

	* python/lib/gdb/__init__.py: Add an execute_file function.
	* python/python.c (python_run_simple_file): Call gdb.execute_file
	on Windows.
---
 gdb/python/lib/gdb/__init__.py | 26 ++++++++++++++++++++++++++
 gdb/python/python.c            | 23 ++++++++++++++---------
 2 files changed, 40 insertions(+), 9 deletions(-)

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
index af74df80c8..afe5b08f3a 100644
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -106,6 +106,32 @@ def execute_unwinders(pending_frame):
 
     return None
 
+def _execute_file(filepath):
+    """This function is used to replace Python 2's PyRun_SimpleFile.
+
+    Loads and executes the given file.
+
+    We could use the runpy module, but its documentation says:
+    "Furthermore, any functions and classes defined by the executed code are
+    not guaranteed to work correctly after a runpy function has returned."
+    """
+    globals = sys.modules['__main__'].__dict__
+    set_file = False
+    # Set file (if not set) so that the imported file can use it (e.g. to
+    # access file-relative paths). This matches what PyRun_SimpleFile does.
+    if not hasattr(globals, '__file__'):
+        globals['__file__'] = filepath
+        set_file = True
+    try:
+        with open(filepath, 'rb') as file:
+            # We pass globals also as locals to match what Python does
+            # in PyRun_SimpleFile.
+            compiled = compile(file.read(), filepath, 'exec')
+            exec(compiled, globals, globals)
+    finally:
+        if set_file:
+            del globals['__file__']
+
 
 # Convenience variable to GDB's python directory
 PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
diff --git a/gdb/python/python.c b/gdb/python/python.c
index 162470dcc0..cca7c4cd6b 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -323,9 +323,8 @@ python_interactive_command (const char *arg, int from_tty)
    A FILE * from one runtime does not necessarily operate correctly in
    the other runtime.
 
-   To work around this potential issue, we create on Windows hosts the
-   FILE object using Python routines, thus making sure that it is
-   compatible with the Python library.  */
+   To work around this potential issue, we run code in Python to load
+   the script.  */
 
 static void
 python_run_simple_file (FILE *file, const char *filename)
@@ -339,15 +338,21 @@ python_run_simple_file (FILE *file, const char *filename)
   /* Because we have a string for a filename, and are using Python to
      open the file, we need to expand any tilde in the path first.  */
   gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
-  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
-  if (python_file == NULL)
+
+  if (gdb_python_module == nullptr
+      || ! PyObject_HasAttrString (gdb_python_module, "_execute_file"))
+    error (_("Installation error: gdb._execute_file function is missing"));
+
+  gdbpy_ref<> return_value
+    (PyObject_CallMethod (gdb_python_module, "_execute_file", "s",
+			  full_path.get ()));
+  if (return_value == nullptr)
     {
-      gdbpy_print_stack ();
-      error (_("Error while opening file: %s"), full_path.get ());
+      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
+         behavior of the non-Windows codepath.  */
+      PyErr_PrintEx(0);
     }
 
-  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
-
 #endif /* _WIN32 */
 }
 
-- 
2.23.0.rc1.153.gdeed80330f-goog

Re: [PATCH v3] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

Thanks, will send a patch in a second with that fixed (and which
undoes the WIN32 ifdef change I accidentally had in the last version)

Christian

On Thu, Aug 15, 2019 at 1:02 PM Pedro Alves <palves@redhat.com> wrote:
>
> On 8/15/19 6:42 PM, Christian Biesinger via gdb-patches wrote:
> > +      error (_("Installation error: gdb._execute_file function is missing"));
> > +      return;
>
> That "return" statement is dead code, error throws an exception.
>
> Thanks,
> Pedro Alves

[PING] [PATCH v4] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

Ping

On Thu, Aug 15, 2019 at 1:49 PM Christian Biesinger
<cbiesinger@google.com> wrote:
>
> PyFile_FromString and PyFile_AsFile have been removed in Python 3.
> There is no obvious replacement that works here, and we can't just
> pass our FILE* to a DLL in Windows because it may use a different
> C runtime.
>
> So we just call a Python function which reads and executes file
> contents. Care must be taken to execute it in the context of
> __main__.
>
> Tested by inverting the ifdef and running the testsuite on Debian
> Linux (even without the patch, I failed at running the testsuite
> on Windows). I did test with both Python 2 and 3.
>
> gdb/ChangeLog:
>
> 2019-08-13  Christian Biesinger  <cbiesinger@google.com>
>
>         * python/lib/gdb/__init__.py: Add an execute_file function.
>         * python/python.c (python_run_simple_file): Call gdb.execute_file
>         on Windows.
> ---
>  gdb/python/lib/gdb/__init__.py | 26 ++++++++++++++++++++++++++
>  gdb/python/python.c            | 23 ++++++++++++++---------
>  2 files changed, 40 insertions(+), 9 deletions(-)
>
> diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
> index af74df80c8..afe5b08f3a 100644
> --- a/gdb/python/lib/gdb/__init__.py
> +++ b/gdb/python/lib/gdb/__init__.py
> @@ -106,6 +106,32 @@ def execute_unwinders(pending_frame):
>
>      return None
>
> +def _execute_file(filepath):
> +    """This function is used to replace Python 2's PyRun_SimpleFile.
> +
> +    Loads and executes the given file.
> +
> +    We could use the runpy module, but its documentation says:
> +    "Furthermore, any functions and classes defined by the executed code are
> +    not guaranteed to work correctly after a runpy function has returned."
> +    """
> +    globals = sys.modules['__main__'].__dict__
> +    set_file = False
> +    # Set file (if not set) so that the imported file can use it (e.g. to
> +    # access file-relative paths). This matches what PyRun_SimpleFile does.
> +    if not hasattr(globals, '__file__'):
> +        globals['__file__'] = filepath
> +        set_file = True
> +    try:
> +        with open(filepath, 'rb') as file:
> +            # We pass globals also as locals to match what Python does
> +            # in PyRun_SimpleFile.
> +            compiled = compile(file.read(), filepath, 'exec')
> +            exec(compiled, globals, globals)
> +    finally:
> +        if set_file:
> +            del globals['__file__']
> +
>
>  # Convenience variable to GDB's python directory
>  PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
> diff --git a/gdb/python/python.c b/gdb/python/python.c
> index 162470dcc0..cca7c4cd6b 100644
> --- a/gdb/python/python.c
> +++ b/gdb/python/python.c
> @@ -323,9 +323,8 @@ python_interactive_command (const char *arg, int from_tty)
>     A FILE * from one runtime does not necessarily operate correctly in
>     the other runtime.
>
> -   To work around this potential issue, we create on Windows hosts the
> -   FILE object using Python routines, thus making sure that it is
> -   compatible with the Python library.  */
> +   To work around this potential issue, we run code in Python to load
> +   the script.  */
>
>  static void
>  python_run_simple_file (FILE *file, const char *filename)
> @@ -339,15 +338,21 @@ python_run_simple_file (FILE *file, const char *filename)
>    /* Because we have a string for a filename, and are using Python to
>       open the file, we need to expand any tilde in the path first.  */
>    gdb::unique_xmalloc_ptr<char> full_path (tilde_expand (filename));
> -  gdbpy_ref<> python_file (PyFile_FromString (full_path.get (), (char *) "r"));
> -  if (python_file == NULL)
> +
> +  if (gdb_python_module == nullptr
> +      || ! PyObject_HasAttrString (gdb_python_module, "_execute_file"))
> +    error (_("Installation error: gdb._execute_file function is missing"));
> +
> +  gdbpy_ref<> return_value
> +    (PyObject_CallMethod (gdb_python_module, "_execute_file", "s",
> +                         full_path.get ()));
> +  if (return_value == nullptr)
>      {
> -      gdbpy_print_stack ();
> -      error (_("Error while opening file: %s"), full_path.get ());
> +      /* Use PyErr_PrintEx instead of gdbpy_print_stack to better match the
> +         behavior of the non-Windows codepath.  */
> +      PyErr_PrintEx(0);
>      }
>
> -  PyRun_SimpleFile (PyFile_AsFile (python_file.get ()), filename);
> -
>  #endif /* _WIN32 */
>  }
>
> --
> 2.23.0.rc1.153.gdeed80330f-goog
>

Re: [PATCH v4] Make GDB compile with Python 3 on MinGW

[Simon Marchi]

On 2019-08-15 2:49 p.m., Christian Biesinger via gdb-patches wrote:
> PyFile_FromString and PyFile_AsFile have been removed in Python 3.
> There is no obvious replacement that works here, and we can't just
> pass our FILE* to a DLL in Windows because it may use a different
> C runtime.
> 
> So we just call a Python function which reads and executes file
> contents. Care must be taken to execute it in the context of
> __main__.
> 
> Tested by inverting the ifdef and running the testsuite on Debian
> Linux (even without the patch, I failed at running the testsuite
> on Windows). I did test with both Python 2 and 3.
> 
> gdb/ChangeLog:
> 
> 2019-08-13  Christian Biesinger  <cbiesinger@google.com>
> 
> 	* python/lib/gdb/__init__.py: Add an execute_file function.

This can be written as:

  * python/lib/gdb/__init__.py (_execute_file): New function.

> 	* python/python.c (python_run_simple_file): Call gdb.execute_file
> 	on Windows.

Please update "execute_file" to "_execute_file" in both entries.

The patch LGTM with this fixed.

Thanks,

Simon

Re: [PATCH v4] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

On Thu, Aug 22, 2019 at 5:24 PM Simon Marchi <simark@simark.ca> wrote:
>
> On 2019-08-15 2:49 p.m., Christian Biesinger via gdb-patches wrote:
> > PyFile_FromString and PyFile_AsFile have been removed in Python 3.
> > There is no obvious replacement that works here, and we can't just
> > pass our FILE* to a DLL in Windows because it may use a different
> > C runtime.
> >
> > So we just call a Python function which reads and executes file
> > contents. Care must be taken to execute it in the context of
> > __main__.
> >
> > Tested by inverting the ifdef and running the testsuite on Debian
> > Linux (even without the patch, I failed at running the testsuite
> > on Windows). I did test with both Python 2 and 3.
> >
> > gdb/ChangeLog:
> >
> > 2019-08-13  Christian Biesinger  <cbiesinger@google.com>
> >
> >       * python/lib/gdb/__init__.py: Add an execute_file function.
>
> This can be written as:
>
>   * python/lib/gdb/__init__.py (_execute_file): New function.
>
> >       * python/python.c (python_run_simple_file): Call gdb.execute_file
> >       on Windows.
>
> Please update "execute_file" to "_execute_file" in both entries.
>
> The patch LGTM with this fixed.

Thanks, pushing now with that fixed.


Christian

Re: [PATCH v4] Make GDB compile with Python 3 on MinGW

[Christian Biesinger via gdb-patches]

On Thu, Aug 22, 2019 at 6:47 PM Christian Biesinger
<cbiesinger@google.com> wrote:
>
> On Thu, Aug 22, 2019 at 5:24 PM Simon Marchi <simark@simark.ca> wrote:
> >
> > On 2019-08-15 2:49 p.m., Christian Biesinger via gdb-patches wrote:
> > > PyFile_FromString and PyFile_AsFile have been removed in Python 3.
> > > There is no obvious replacement that works here, and we can't just
> > > pass our FILE* to a DLL in Windows because it may use a different
> > > C runtime.
> > >
> > > So we just call a Python function which reads and executes file
> > > contents. Care must be taken to execute it in the context of
> > > __main__.
> > >
> > > Tested by inverting the ifdef and running the testsuite on Debian
> > > Linux (even without the patch, I failed at running the testsuite
> > > on Windows). I did test with both Python 2 and 3.
[...]
> > The patch LGTM with this fixed.
>
> Thanks, pushing now with that fixed.

Thinking about this patch -- is it worth a NEWS entry to mention that
gdb can now be compiled with Python 3 on Windows?

Christian

Re: [PATCH v4] Make GDB compile with Python 3 on MinGW

[Eli Zaretskii]

> From: "Christian Biesinger via gdb-patches" <gdb-patches@sourceware.org>
> Date: Sat, 14 Sep 2019 09:08:51 -0400
> Cc: gdb-patches <gdb-patches@sourceware.org>
> 
> Thinking about this patch -- is it worth a NEWS entry to mention that
> gdb can now be compiled with Python 3 on Windows?

You are right, it probably is.

[PATCH] Add a NEWS entry that gdb can be compiled with py3 on Windows.

[Christian Biesinger via gdb-patches]

[Eli -- how about this one?]

This was fixed in commit 272044897e178835f596c96740c5a1800ec6f9fb.

2019-09-11  Christian Biesinger  <cbiesinger@google.com>

	* NEWS: Mention that gdb can now be compiled with Python 3
	on Windows.
---
 gdb/NEWS | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gdb/NEWS b/gdb/NEWS
index d502f08984c..947b7435337 100644
--- a/gdb/NEWS
+++ b/gdb/NEWS
@@ -32,6 +32,8 @@
 * GDB now shows the Ada task names at more places, e.g. in task switching
   messages.
 
+* GDB can now be compiled with Python 3 on Windows.
+
 * Python API
 
   ** The gdb.Value type has a new method 'format_string' which returns a
-- 
2.23.0.237.gc6a4ce50a0-goog

Re: [PATCH] Add a NEWS entry that gdb can be compiled with py3 on Windows.

[Eli Zaretskii]

> Date: Sat, 14 Sep 2019 15:43:20 -0500
> From: "Christian Biesinger via gdb-patches" <gdb-patches@sourceware.org>
> Cc: Christian Biesinger <cbiesinger@google.com>
> 
> [Eli -- how about this one?]
> 
> This was fixed in commit 272044897e178835f596c96740c5a1800ec6f9fb.
> 
> 2019-09-11  Christian Biesinger  <cbiesinger@google.com>
> 
> 	* NEWS: Mention that gdb can now be compiled with Python 3
> 	on Windows.
> ---
>  gdb/NEWS | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/gdb/NEWS b/gdb/NEWS
> index d502f08984c..947b7435337 100644
> --- a/gdb/NEWS
> +++ b/gdb/NEWS
> @@ -32,6 +32,8 @@
>  * GDB now shows the Ada task names at more places, e.g. in task switching
>    messages.
>  
> +* GDB can now be compiled with Python 3 on Windows.
> +

Fine with me, thanks.

Re: [PATCH] Add a NEWS entry that gdb can be compiled with py3 on Windows.

[Christian Biesinger via gdb-patches]

On Sat, Sep 14, 2019 at 10:34 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Date: Sat, 14 Sep 2019 15:43:20 -0500
> > From: "Christian Biesinger via gdb-patches" <gdb-patches@sourceware.org>
> > Cc: Christian Biesinger <cbiesinger@google.com>
> >
> > [Eli -- how about this one?]
> >
> > This was fixed in commit 272044897e178835f596c96740c5a1800ec6f9fb.
> >
> > 2019-09-11  Christian Biesinger  <cbiesinger@google.com>
> >
> >       * NEWS: Mention that gdb can now be compiled with Python 3
> >       on Windows.
> > ---
> >  gdb/NEWS | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/gdb/NEWS b/gdb/NEWS
> > index d502f08984c..947b7435337 100644
> > --- a/gdb/NEWS
> > +++ b/gdb/NEWS
> > @@ -32,6 +32,8 @@
> >  * GDB now shows the Ada task names at more places, e.g. in task switching
> >    messages.
> >
> > +* GDB can now be compiled with Python 3 on Windows.
> > +
>
> Fine with me, thanks.

Thanks, pushed as 6a062a93d70.

Christian