[PATCH] gdb/python: fix memory leak in python inferior code

[Andrew Burgess <andrew.burgess@embecosm.com>]

When a user creates a gdb.Inferior object for the first time a new
Python object is created.  This object is then cached within GDB's
inferior object using the registry mechanism (see
inferior_to_inferior_object in py-inferior.c, specifically the calls
to inferior_data and set_inferior_data).

This reference to the Python object, held within the real inferior
object, counts as a reference to the Python object, and so holds the
reference count of the Python object above zero.

At the same time, the Python object maintains a pointer back to GDB's
real inferior object.

When GDB's inferior object is deleted (say the inferior exits) then
py_free_inferior is called (thanks to the registry system), this
function looks up the Python object and sets the inferior pointer to
nullptr and finally deletes reduces the reference count on the Python
object.

If at this point the user still holds a reference to the Python object
then nothing happens.  The gdb.Inferior Python object is now in the
non-valid state (see infpy_is_valid in py-inferior.c), but otherwise,
everything is fine.

However, if there are no further references to the gdb.Inferior
object, or, once the user has given up all their references to the
gdb.Inferior object, then infpy_dealloc is called.

This function currently checks to see if the inferior pointer within
the gdb.Inferior object is nullptr or not.  If the pointer is nullptr
then infpy_dealloc immediately returns.

Only when the inferior point in the gdb.Inferior is not nullptr do
we (a) set the gdb.Inferior reference inside GDB's inferior to
nullptr, and (b) call the underlying Python tp_free function.

There are a number things wrong here:

  1. The gdb.Inferior reference within GDB's inferior object hold a
  reference count, thus, setting this reference to nullptr without
  first decrementing the reference count would leak a reference.  This
  could be solved by having the inferior hold a gdbpy_ref object,
  however...

  2. As GDB's inferior holds a reference then infpy_dealloc will never
  be called until GDB's inferior object is deleted, calls
  py_free_inferior, and so gives up the reference.  At this point
  there is no longer a need to call set_inferior_data to set the field
  back to NULL, that field must have been cleared in order to get the
  reference count to zero, which means...

  3. If we know that py_free_inferior must be called before
  infpy_dealloc, then we know that the inferior pointer in
  gdb.Inferior will always be nullptr when infpy_dealloc is called,
  this means that the call to the underlying tp_free function will
  always be skipped.  Skipping this call at all always seems like a
  bad idea, but it turns out we will always skip it.

Given all of the above, I assert that the gdb.Inferior field will
always be nullptr when infpy_dealloc is called.  That's what this
patch does.

I assume that failing to call tp_free will mean that Python leaks some
memory associated with the gdb.Inferior object, but I don't know any
way that I could show that in a test.  As such, I've got no new tests
for this commit, but if anyone has any ideas I'd be happy to add
something.
---
 gdb/python/py-inferior.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/gdb/python/py-inferior.c b/gdb/python/py-inferior.c
index cfbc2f6574f..c36f275b23e 100644
--- a/gdb/python/py-inferior.c
+++ b/gdb/python/py-inferior.c
@@ -864,12 +864,20 @@ static void
 infpy_dealloc (PyObject *obj)
 {
   inferior_object *inf_obj = (inferior_object *) obj;
-  struct inferior *inf = inf_obj->inferior;
 
-  if (! inf)
-    return;
+  /* The inferior itself holds a reference to this Python object, which
+     will keep the reference count of this object above zero until GDB
+     deletes the inferior and py_free_inferior is called.
+
+     Once py_free_inferior has been called then the link between this
+     Python object and the inferior is set to nullptr, and then the
+     reference count on this Python object is decremented.
+
+     The result of all this is that the link between this Python object and
+     the inferior should always have been set to nullptr before this
+     function is called.  */
+  gdb_assert (inf_obj->inferior == nullptr);
 
-  set_inferior_data (inf, infpy_inf_data_key, NULL);
   Py_TYPE (obj)->tp_free (obj);
 }
 
-- 
2.25.4