GDB crashing because of Python

GDB crashing because of Python

[Kevin Pouget]

Hello,

I notice today a lot for GDB crash because of Python, am I the only one?
(I didn't refresh my git tree since a while, so it might not be
directly related to the last Python patches)

It's hard to tell exactly what cause it, but for instance I see repeatedly:

> SystemError: /builddir/build/BUILD/Python-2.7.3/Objects/listobject.c:178: bad argument to internal function

This one aborts() GDB

> Fatal Python error: GC object already tracked

from
> #2  0x0000003f2c4f7b3e in Py_FatalError (msg=msg@entry=
>     0x3f2c520ae8 "GC object already tracked")
>     at /usr/src/debug/Python-2.7.3/Python/pythonrun.c:1685
> #3  0x0000003f2c4739cb in PyList_New (size=size@entry=0)
>     at /usr/src/debug/Python-2.7.3/Objects/listobject.c:170
> #4  0x0000003f2c4f58a5 in PyArena_New ()
>     at /usr/src/debug/Python-2.7.3/Python/pyarena.c:143
> #5  0x0000003f2c4f793a in PyRun_FileExFlags (...)

and this one raises a segfault:
> Program received signal SIGSEGV, Segmentation fault.
> PyObject_Malloc (nbytes=32)
>     at /usr/src/debug/Python-2.7.3/Objects/obmalloc.c:784
> 784	            if ((pool->freeblock = *(block **)bp) != NULL) {

from:
> #0  PyObject_Malloc (nbytes=32)
>     at /usr/src/debug/Python-2.7.3/Objects/obmalloc.c:784
> #1  0x0000003f2c475291 in _PyLong_New (size=1)
>     at /usr/src/debug/Python-2.7.3/Objects/longobject.c:76
> #2  0x0000003f2c479e9f in PyLong_FromUnsignedLongLong (
>     ival=ival@entry=135084326)
>     at /usr/src/debug/Python-2.7.3/Objects/longobject.c:883
> #3  0x00000000004df100 in frapy_pc (self=
>     <gdb.Frame at remote 0x2c1d130>, args=<optimized out>)
>     at ../../../git/gdb/gdb/python/py-frame.c:209
> #4  0x0000003f2c4dce36 in call_function (oparg=<optimized out>,
>     pp_stack=0x7fff850c3638)
>     at /usr/src/debug/Python-2.7.3/Python/ceval.c:4082
> #5  PyEval_EvalFrameEx (f=f@entry=

I'm on Fedora 17, x86_64, Python seems to be at version 2.7.3, gdb is
up to date against the trunk (7.5.50.20120823-cvs)


Cordially,

Kevin

Re: GDB crashing because of Python

[Tom Tromey]

>>>>> "Kevin" == Kevin Pouget <kevin.pouget@gmail.com> writes:

Kevin> I notice today a lot for GDB crash because of Python, am I the
Kevin> only one?  (I didn't refresh my git tree since a while, so it
Kevin> might not be directly related to the last Python patches)

Kevin> It's hard to tell exactly what cause it, but for instance I see
Kevin> repeatedly:
[...]

I haven't seen it.
I updated from CVS just now, rebuilt, and ran the gdb.python tests
without any problems.

Kevin> I'm on Fedora 17, x86_64, Python seems to be at version 2.7.3, gdb is
Kevin> up to date against the trunk (7.5.50.20120823-cvs)

I'm still on Fedora 16.  I wouldn't expect 17 to have particular
problems like this though.

What were you doing when you got these crashes?  Is it something simple
that I could try?  What happens if you run the gdb.python tests?

Tom

Re: GDB crashing because of Python

[Kevin Pouget]

[-- Attachment #1: Type: text/plain, Size: 2275 bytes --]

On Thu, Aug 23, 2012 at 3:55 PM, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Kevin" == Kevin Pouget <kevin.pouget@gmail.com> writes:
>
> Kevin> I notice today a lot for GDB crash because of Python, am I the
> Kevin> only one?  (I didn't refresh my git tree since a while, so it
> Kevin> might not be directly related to the last Python patches)
>
> Kevin> It's hard to tell exactly what cause it, but for instance I see
> Kevin> repeatedly:
> [...]
>
> I haven't seen it.
> I updated from CVS just now, rebuilt, and ran the gdb.python tests
> without any problems.
>
> Kevin> I'm on Fedora 17, x86_64, Python seems to be at version 2.7.3, gdb is
> Kevin> up to date against the trunk (7.5.50.20120823-cvs)
>
> I'm still on Fedora 16.  I wouldn't expect 17 to have particular
> problems like this though.
>
> What were you doing when you got these crashes?  Is it something simple
> that I could try?  What happens if you run the gdb.python tests?
>
> Tom

Hello,

> What happens if you run the gdb.python tests?
unfortunately, the run 100% fine ...

> What were you doing when you got these crashes?
just "normal" things, I mean, running code which have been running
fine until today.

> SystemError: /builddir/build/BUILD/Python-2.7.3/Objects/listobject.c:178: bad argument to internal function
this one is triggered in the middle of a Breakpoint callback, so it
appears quite often

> Program received signal SIGSEGV, Segmentation fault.
> PyObject_Malloc (nbytes=32)
>     at /usr/src/debug/Python-2.7.3/Objects/obmalloc.c:784
this one is when I want to attach gdb to another gdb instance (gdb2
--pid $(pidof gdb))

and finally:
> Fatal Python error: GC object already tracked
this one appears when I start the inferior execution, and start it again

I've attached the complete stack of this crash (from a third gdb!).

We can see that it crashed while auto-loading
"/usr/lib/debug/usr/bin/gdb-gdb.py", which is the original file from
Fedora gdb package.

I can also see that the stack contains "prompt_hook", I've seen another crash
which was mentioned this file [1], which deals with prompt customization.
It *might* be related, but I'm not sure at all.


Thanks,

Kevin


[1] : http://gitorious.org/misc-gdb-stuff/misc-gdb-stuff/blobs/master/misc_gdb/gaudy_prompt.py

[-- Attachment #2: stack.txt --]
[-- Type: text/plain, Size: 3717 bytes --]

(gdb) where
#0  0x0000003f19035925 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003f190370d8 in __GI_abort () at abort.c:91
#2  0x0000003f2c4f7b3e in Py_FatalError (msg=msg@entry=
    0x3f2c520ae8 "GC object already tracked")
    at /usr/src/debug/Python-2.7.3/Python/pythonrun.c:1685
#3  0x0000003f2c4739cb in PyList_New (size=size@entry=0)
    at /usr/src/debug/Python-2.7.3/Objects/listobject.c:170
#4  0x0000003f2c4f58a5 in PyArena_New ()
    at /usr/src/debug/Python-2.7.3/Python/pyarena.c:143
#5  0x0000003f2c4f793a in PyRun_FileExFlags (fp=fp@entry=
    0x478ec40, filename=filename@entry=
    0x3d8f910 "/usr/lib/debug/usr/bin/gdb-gdb.py", 
    start=start@entry=257, globals=globals@entry=
    {'GdbOutputFile': <classobj at remote 0x23d8460>, '__builtins__': <module at remote 0x7f0a6e5f4ad0>, '__file__': '\x00\x00\x00\x00/lib/deb\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'GdbRemoveReadlineFinder': <classobj at remote 0x23d83f8>, 'init': <module at remote 0x23ff718>, '__package__': None, 'sys': <module at remote 0x7f0a6e5f4b78>, 'GdbOutputErrorFile': <classobj at remote 0x23d84c8>, 'gdb': <module at remote 0x23f5830>, 'prompt_hook': None, '__name__': '__main__', 'GdbSetPythonDirectory': <function at remote 0x23f2758>, 'os': <module at remote 0x7f0a6e5bacc8>, '__doc__': None}, 
    locals=locals@entry=
    {'GdbOutputFile': <classobj at remote 0x23d8460>, '__builtins__': <module at remote 0x7f0a6e5f4ad0>, '__file__': '\x00\x00\x00\x00/lib/deb\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'GdbRemoveReadlineFinder': <classobj at remote 0x23d83f8>, 'init': <module at remote 0x23ff718>, '__package__': None, 'sys': <module at remote 0x7f0a6e5f4b78>, 'GdbOutputErrorFile': <classobj at remote 0x23d84c8>, 'gdb': <module at remote 0x23f5830>, 'prompt_hook': None, '__name__': '__main__', 'GdbSetPythonDirectory': <function at remote 0x23f2758>, 'os': <module at remote 0x7f0a6e5bacc8>, '__doc__': None}, 
    closeit=closeit@entry=0, flags=flags@entry=0x0)
    at /usr/src/debug/Python-2.7.3/Python/pythonrun.c:1335
#6  0x0000003f2c4f83ab in PyRun_SimpleFileExFlags (fp=fp@entry=
    0x478ec40, filename=filename@entry=
    0x3d8f910 "/usr/lib/debug/usr/bin/gdb-gdb.py", 
    closeit=closeit@entry=0, flags=flags@entry=0x0)
    at /usr/src/debug/Python-2.7.3/Python/pythonrun.c:951
#7  0x00000000004d9b11 in python_run_simple_file (filename=
    0x3d8f910 "/usr/lib/debug/usr/bin/gdb-gdb.py", file=
    0x478ec40) at ../../../git/gdb/gdb/python/python.c:269
#8  source_python_script_for_objfile (objfile=0x26ed780, file=
    0x478ec40, filename=
    0x3d8f910 "/usr/lib/debug/usr/bin/gdb-gdb.py")
    at ../../../git/gdb/gdb/python/python.c:1134
#9  0x000000000050b1cc in auto_load_objfile_script (
    objfile=objfile@entry=0x26ed780, language=language@entry=
    0x748390 <script_language_python>)
    at ../../../git/gdb/gdb/auto-load.c:770
#10 0x00000000004da3df in gdbpy_load_auto_scripts_for_objfile (
    objfile=0x26ed780)
    at ../../../git/gdb/gdb/python/py-auto-load.c:233
#11 0x0000000000583d5d in captured_main (data=data@entry=
    0x7fffe49b9d30) at ../../../git/gdb/gdb/main.c:964
#12 0x000000000058213e in catch_errors (func=func@entry=
    0x583780 <captured_main>, func_args=func_args@entry=
    0x7fffe49b9d30, errstring=errstring@entry=0x733ab2 "", 
    mask=mask@entry=6) at ../../../git/gdb/gdb/exceptions.c:546
#13 0x0000000000584794 in gdb_main (args=args@entry=
    0x7fffe49b9d30) at ../../../git/gdb/gdb/main.c:1008
#14 0x000000000045894e in main (argc=<optimized out>, 
    argv=<optimized out>) at ../../../git/gdb/gdb/gdb.c:34
(gdb) 

Re: GDB crashing because of Python

[Tom Tromey]

Kevin> I'm on Fedora 17, x86_64, Python seems to be at version 2.7.3, gdb is
Kevin> up to date against the trunk (7.5.50.20120823-cvs)

I installed F17 in a mock chroot and ran the latest gdb there.
This isn't an identical scenario due to possible kernel differences, but
I wouldn't expect those to matter.

>> Fatal Python error: GC object already tracked

Kevin> this one appears when I start the inferior execution, and start it again

I couldn't make it crash.
What happens if you run under valgrind?
When I do that, I see some output, but nothing that looks too serious
and/or new.

Tom

Re: GDB crashing because of Python

[Kevin Pouget]

On Thu, Aug 23, 2012 at 8:52 PM, Tom Tromey <tromey@redhat.com> wrote:
>
> Kevin> I'm on Fedora 17, x86_64, Python seems to be at version 2.7.3, gdb is
> Kevin> up to date against the trunk (7.5.50.20120823-cvs)
>
> I installed F17 in a mock chroot and ran the latest gdb there.
> This isn't an identical scenario due to possible kernel differences, but
> I wouldn't expect those to matter.
>
> >> Fatal Python error: GC object already tracked
>
> Kevin> this one appears when I start the inferior execution, and start it again
>
> I couldn't make it crash.
> What happens if you run under valgrind?
> When I do that, I see some output, but nothing that looks too serious
> and/or new.
>
> Tom


Hello,

so based on `git bisect`, it looks like the errors were introduced by
this commit:

--> SystemError:
/builddir/build/BUILD/Python-2.7.3/Objects/listobject.c:178: bad
argument to internal function
--> Fatal Python error: GC object already tracked
# http://sourceware.org/ml/gdb-patches/2012-08/msg00434.html
> commit 97143778fc8aceaca6895de13b93c88811402441
> Author: Tom Tromey <tromey@redhat.com>
> Date:   Wed Aug 15 14:21:57 2012 +0000
>       PR python/14387:


I'm not sure about what relevant information I can provide with
valgrind, so here is the complete log:

==29717== Memcheck, a memory error detector
==29717== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==29717== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==29717== Command: gdb-master --pid 26304 -ex quit
==29717==
GNU gdb (GDB) 7.5.50.20120815-cvs
[... gdb starts ...]
Attaching to process 26304
Reading symbols from /home/kevin/travail/i386/gdb-trunk.build/gdb/gdb...done.
==29719==
==29719== HEAP SUMMARY:
==29719==     in use at exit: 18,910,812 bytes in 38,905 blocks
==29719==   total heap usage: 174,966 allocs, 136,061 frees,
132,543,220 bytes allocated
==29719==
==29719== LEAK SUMMARY:
==29719==    definitely lost: 5,598 bytes in 33 blocks
==29719==    indirectly lost: 24 bytes in 1 blocks
==29719==      possibly lost: 1,819,580 bytes in 10,752 blocks
==29719==    still reachable: 17,085,610 bytes in 28,119 blocks
==29719==         suppressed: 0 bytes in 0 blocks
==29719== Rerun with --leak-check=full to see details of leaked memory
==29719==
==29719== For counts of detected and suppressed errors, rerun with: -v
==29719== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
Reading symbols from /lib64/libdl.so.2...Reading symbols from
/usr/lib/debug/lib64/libdl-2.15.so.debug...
[... symbols loading ...]
==29717== Invalid read of size 8
==29717==    at 0x501076: objfpy_get_printers (py-objfile.c:88)
==29717==    by 0x502ACD: find_pretty_printer_from_objfiles
(py-prettyprint.c:114)
==29717==    by 0x502D75: find_pretty_printer (py-prettyprint.c:190)
==29717==    by 0x503EEE: apply_val_pretty_printer (py-prettyprint.c:719)
==29717==    by 0x5819E8: val_print (valprint.c:727)
==29717==    by 0x581CAD: common_val_print (valprint.c:806)
==29717==    by 0x5C33DA: print_frame_arg (stack.c:282)
==29717==    by 0x5C3F5D: print_frame_args (stack.c:647)
==29717==    by 0x5C4E72: print_frame (stack.c:1171)
==29717==    by 0x5C44A9: print_frame_info (stack.c:826)
==29717==    by 0x5C2F4E: print_stack_frame (stack.c:165)
==29717==    by 0x5BCB09: normal_stop (infrun.c:6053)
==29717==  Address 0xbf8d638 is 24 bytes inside a block of size 240 free'd
==29717==    at 0x4A079AE: free (vg_replace_malloc.c:427)
==29717==    by 0x6F9D9C: xfree (common-utils.c:107)
==29717==    by 0x5CAFAD: catcher_pop (exceptions.c:123)
==29717==    by 0x5CB03D: exceptions_state_mc (exceptions.c:149)
==29717==    by 0x5CB1C8: exceptions_state_mc_action_iter (exceptions.c:210)
==29717==    by 0x6CEE1B: get_frame_pc_if_available (frame.c:1990)
==29717==    by 0x6CEBD2: get_prev_frame (frame.c:1891)
==29717==    by 0x6594F2: value_of_dwarf_reg_entry (dwarf2loc.c:1238)
==29717==    by 0x65969B: value_of_dwarf_block_entry (dwarf2loc.c:1300)
==29717==    by 0x65F125: loclist_read_variable_at_entry (dwarf2loc.c:4009)
==29717==    by 0x5C3632: read_frame_arg (stack.c:335)
==29717==    by 0x5C3F3F: print_frame_args (stack.c:644)
==29717==
==29717==
==29717== Process terminating with default action of signal 11 (SIGSEGV)
==29717==  Bad permissions for mapped region at address 0x456EF0
==29717==    at 0x501081: objfpy_get_printers (py-objfile.c:88)
==29717==    by 0x502ACD: find_pretty_printer_from_objfiles
(py-prettyprint.c:114)
==29717==    by 0x502D75: find_pretty_printer (py-prettyprint.c:190)
==29717==    by 0x503EEE: apply_val_pretty_printer (py-prettyprint.c:719)
==29717==    by 0x5819E8: val_print (valprint.c:727)
==29717==    by 0x581CAD: common_val_print (valprint.c:806)
==29717==    by 0x5C33DA: print_frame_arg (stack.c:282)
==29717==    by 0x5C3F5D: print_frame_args (stack.c:647)
==29717==    by 0x5C4E72: print_frame (stack.c:1171)
==29717==    by 0x5C44A9: print_frame_info (stack.c:826)
==29717==    by 0x5C2F4E: print_stack_frame (stack.c:165)
==29717==    by 0x5BCB09: normal_stop (infrun.c:6053)
0x0000003f190e8b84 in __GI___poll (==29717==
==29717== HEAP SUMMARY:
==29717==     in use at exit: 37,857,799 bytes in 42,044 blocks
==29717==   total heap usage: 849,325 allocs, 807,281 frees,
807,618,669 bytes allocated
==29717==
==29717== LEAK SUMMARY:
==29717==    definitely lost: 5,598 bytes in 33 blocks
==29717==    indirectly lost: 24 bytes in 1 blocks
==29717==      possibly lost: 1,839,074 bytes in 10,983 blocks
==29717==    still reachable: 36,013,103 bytes in 31,027 blocks
==29717==         suppressed: 0 bytes in 0 blocks
==29717== Rerun with --leak-check=full to see details of leaked memory
==29717==
==29717== For counts of detected and suppressed errors, rerun with: -v
==29717== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
[1]    29717 segmentation fault  valgrind gdb-master --pid 26304 -ex quit


Thanks for your help,

Kevin

Re: GDB crashing because of Python

[Tom Tromey]

>>>>> "Kevin" == Kevin Pouget <kevin.pouget@gmail.com> writes:

Kevin> so based on `git bisect`, it looks like the errors were introduced by
Kevin> this commit:

Can you please try the appended patch?

I think the bug here is that this code assumes that
objfile_to_objfile_object returns a new reference, but in fact it does
not.

While looking at this I think I found even more reference counting bugs.
Despair.

Tom

diff --git a/gdb/python/py-newobjfileevent.c b/gdb/python/py-newobjfileevent.c
index 3059ae4..d014be6 100644
--- a/gdb/python/py-newobjfileevent.c
+++ b/gdb/python/py-newobjfileevent.c
@@ -25,7 +25,7 @@ static PyObject *
 create_new_objfile_event_object (struct objfile *objfile)
 {
   PyObject *objfile_event;
-  PyObject *py_objfile = NULL;
+  PyObject *py_objfile;
 
   objfile_event = create_event_object (&new_objfile_event_object_type);
   if (!objfile_event)
@@ -36,12 +36,10 @@ create_new_objfile_event_object (struct objfile *objfile)
                                          "new_objfile",
                                          py_objfile) < 0)
     goto fail;
-  Py_DECREF (py_objfile);
 
   return objfile_event;
 
  fail:
-  Py_XDECREF (py_objfile);
   Py_XDECREF (objfile_event);
   return NULL;
 }

Re: GDB crashing because of Python

[Tom Tromey]

Tom> While looking at this I think I found even more reference counting bugs.

I thought perhaps the weird logic in inferior_to_inferior_object would
cause problems.  But it turns out that the struct inferior doesn't own a
reference, and infpy_dealloc solves the problem I thought I was seeing.

Tom

Re: GDB crashing because of Python

[Tom Tromey]

Kevin> so based on `git bisect`, it looks like the errors were introduced by
Kevin> this commit:

Tom> Can you please try the appended patch?

I spent some time today trying to make a robust test case for this.
I can see the error clearly with valgrind, but I can't make it crash.
Printing the Objfile reference counts "works" but it seems to not be
very robust -- in the failing case the count appears to be 12 on my
machine, but of course this is just a fluke, and it could well be
anything.

-lmcheck unfortunately doesn't help, because Python is typically built
using its own allocator.  It would be nice if we could disable this for
the test suite -- it is automatically disabled under valgrind (under
some typical configurations), so it could be done cheaply -- but this
isn't available.

Anyway, I'm going to write a ChangeLog and send it to the patch list.

Tom

Re: GDB crashing because of Python

[Kevin Pouget]

On Tue, Aug 28, 2012 at 9:44 PM, Tom Tromey <tromey@redhat.com> wrote:
> Kevin> so based on `git bisect`, it looks like the errors were introduced by
> Kevin> this commit:
>
> Tom> Can you please try the appended patch?
>
> I spent some time today trying to make a robust test case for this.
> I can see the error clearly with valgrind, but I can't make it crash.
> Printing the Objfile reference counts "works" but it seems to not be
> very robust -- in the failing case the count appears to be 12 on my
> machine, but of course this is just a fluke, and it could well be
> anything.
>
> -lmcheck unfortunately doesn't help, because Python is typically built
> using its own allocator.  It would be nice if we could disable this for
> the test suite -- it is automatically disabled under valgrind (under
> some typical configurations), so it could be done cheaply -- but this
> isn't available.
>
> Anyway, I'm going to write a ChangeLog and send it to the patch list.
>
> Tom


Hello,

Thanks, the patch of the previous email seems to resolve all the bugs
I detailed!

I was thinking that it's strange that I'm the only one seeing this bug
from a file *I* committed ... but maybe I'm the only one actually
using the feature! Did you try to register a Python handler to
`gdb.events.new_objfile` to trigger the bug? (gdb/trunk doesn't not
crash anymore if I remove the connection from my init files)



Cordially,

Kevin

Re: GDB crashing because of Python

[Tom Tromey]

>>>>> "Kevin" == Kevin Pouget <kevin.pouget@gmail.com> writes:

Kevin> Thanks, the patch of the previous email seems to resolve all the bugs
Kevin> I detailed!

I'm going to check it in today.

Kevin> I was thinking that it's strange that I'm the only one seeing this bug
Kevin> from a file *I* committed ... but maybe I'm the only one actually
Kevin> using the feature! Did you try to register a Python handler to
Kevin> `gdb.events.new_objfile` to trigger the bug? (gdb/trunk doesn't not
Kevin> crash anymore if I remove the connection from my init files)

It is a regression due to an earlier, but post-7.5, patch.

Tom