Interpret object causing crash in __cxa_finalize (have core)

Interpret object causing crash in __cxa_finalize (have core)

[Jeffrey Walton]

Hi All,

I'm observing an intermittent crash during unloading of a shared
object. The library and test harness were built with '-g3 -ggdb -O0',
and 'ulimit -c unlimited', so I am able to get it under GDB. But I'm
having problems interpreting what object is being problematic.

$ gdb
...
(gdb) file <path to self test> # loads OK
(gdb) core core # loads OK
(gdb) bt full
...
#21 0x00002ba649808630 in __cxa_finalize (d=0x2ba64901c3d8)
    at cxa_finalize.c:56
        check = <value optimized out>
        cxafn = 0x6
        cxaarg = 0xa46
        f = 0x13925e0
        funcs = 0x13924f0
(gdb) po 0x13925e0
evaluation of this expression requires the target program to be active

Any ideas how to proceed?

Thanks in advance,
Jeff

(gdb) bt full
#0  0x00002ba649802a75 in *__GI_raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
        pid = <value optimized out>
        selftid = <value optimized out>
#1  0x00002ba6498065c0 in *__GI_abort () at abort.c:92
        act = {__sigaction_handler = {sa_handler = 0x7fffecc59ff0,
            sa_sigaction = 0x7fffecc59ff0}, sa_mask = {__val = {
              140737165762736, 140737165774648, 22, 47993198828606, 1,
              4256528, 99, 47993198832910, 3, 140737165762734, 2,
              47993198828632, 1, 47993198837551, 3, 140737165762746}},
          sa_flags = 6, sa_restorer = 0x2ba649917f33}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00002ba64983c4fb in __libc_message (do_abort=<value optimized out>,
    fmt=<value optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
        ap = {{gp_offset = 40, fp_offset = 48,
            overflow_arg_area = 0x7fffecc5aa30,
            reg_save_area = 0x7fffecc5a940}}
        ap_copy = {{gp_offset = 16, fp_offset = 48,
            overflow_arg_area = 0x7fffecc5aa30,
            reg_save_area = 0x7fffecc5a940}}
        fd = 3
        on_2 = <value optimized out>
        list = <value optimized out>
        nlist = 1024
        cp = <value optimized out>
        written = false
#3  0x00002ba6498465b6 in malloc_printerr (action=3,
    str=0x2ba649919cd8 "double free or corruption (!prev)",
    ptr=<value optimized out>) at malloc.c:6266
        buf = "00000000013a21c0"
        cp = 0x0
#4  0x00002ba64984ce83 in *__GI___libc_free (mem=<value optimized out>)
    at malloc.c:3738
        ar_ptr = 0x2ba649b4de40
        p = 0x0
#5  0x0000000000470ac0 in
__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<char const,
std::string> > >::deallocate (this=0x83dfc0, __p=0x13a21c0)
    at /usr/include/c++/4.4/ext/new_allocator.h:95
No locals.
#6  0x00000000004708ca in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_put_node (this=0x83dfc0,
    __p=0x13a21c0) at /usr/include/c++/4.4/bits/stl_tree.h:363
No locals.
#7  0x0000000000470326 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_destroy_node (
    this=0x83dfc0, __p=0x13a21c0) at /usr/include/c++/4.4/bits/stl_tree.h:409
No locals.
#8  0x000000000046fc65 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a21c0) at /usr/include/c++/4.4/bits/stl_tree.h:972
        __y = 0x0
#9  0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a2140) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a2100
#10 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a2100) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a20c0
#11 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a20c0) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a2080
#12 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a2080) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a2040
#13 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a2040) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a2000
#14 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a2000) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a1fc0
#15 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a1fc0) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a1f80
#16 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a1f80) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a1f40
#17 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a1f40) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x13a1f00
#18 0x000000000046fc42 in std::_Rb_tree<char, std::pair<char const,
std::string>, std::_Select1st<std::pair<char const, std::string> >,
std::less<char>, std::allocator<std::pair<char const, std::string> >
>::_M_erase (this=0x83dfc0,
    __x=0x13a1f00) at /usr/include/c++/4.4/bits/stl_tree.h:970
        __y = 0x83dfc0
#19 0x000000000046f7db in ~_Rb_tree (this=0x83dfc0,
    __in_chrg=<value optimized out>)
    at /usr/include/c++/4.4/bits/stl_tree.h:614
No locals.
#20 0x0000000000470c0a in ~map (this=0x83dfc0, __in_chrg=<value optimized out>)
    at /usr/include/c++/4.4/bits/stl_map.h:87
No locals.
#21 0x00002ba649808630 in __cxa_finalize (d=0x2ba64901c3d8)
    at cxa_finalize.c:56
        check = <value optimized out>
        cxafn = 0x6
        cxaarg = 0xa46
        f = 0x13925e0
        funcs = 0x13924f0
#22 0x00002ba648c59076 in __do_global_dtors_aux () from lib/libesapi-c++.so
No symbol table info available.
#23 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

Re: Interpret object causing crash in __cxa_finalize (have core)

[Jan Kratochvil]

On Wed, 24 Aug 2011 21:31:14 +0200, Jeffrey Walton wrote:
> I'm observing an intermittent crash during unloading of a shared
> object. The library and test harness were built with '-g3 -ggdb -O0',
> and 'ulimit -c unlimited', so I am able to get it under GDB.

If you have it reproducible run it under valgrind first.


> But I'm having problems interpreting what object is being problematic.
[...]
> #21 0x00002ba649808630 in __cxa_finalize (d=0x2ba64901c3d8)
>     at cxa_finalize.c:56
>         check = <value optimized out>
>         cxafn = 0x6
>         cxaarg = 0xa46
>         f = 0x13925e0
>         funcs = 0x13924f0
> (gdb) po 0x13925e0
> evaluation of this expression requires the target program to be active

print-object is for Objecttive-C.  And you will decode there just the info you
see in the frame #20.


This looks as a memory corruption, it is better to debug instrument the code
leading to this state, this core file may no longer be too useful.


Regards,
Jan


[...]
> #20 0x0000000000470c0a in ~map (this=0x83dfc0, __in_chrg=<value optimized out>)
>     at /usr/include/c++/4.4/bits/stl_map.h:87
> No locals.
> #21 0x00002ba649808630 in __cxa_finalize (d=0x2ba64901c3d8)
>     at cxa_finalize.c:56
>         check = <value optimized out>
>         cxafn = 0x6
>         cxaarg = 0xa46
>         f = 0x13925e0
>         funcs = 0x13924f0
> #22 0x00002ba648c59076 in __do_global_dtors_aux () from lib/libesapi-c++.so
> No symbol table info available.
> #23 0x0000000000000000 in ?? ()
> No symbol table info available.
> (gdb)

Re: Interpret object causing crash in __cxa_finalize (have core)

[Jeffrey Walton]

On Wed, Aug 24, 2011 at 3:39 PM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> On Wed, 24 Aug 2011 21:31:14 +0200, Jeffrey Walton wrote:
>> I'm observing an intermittent crash during unloading of a shared
>> object. The library and test harness were built with '-g3 -ggdb -O0',
>> and 'ulimit -c unlimited', so I am able to get it under GDB.
>
> If you have it reproducible run it under valgrind first.
Boost has made Valgrind useless (15000 line of output). And I have not
been successful in getting suppression rules:
http://lists.boost.org/boost-users/2011/08/70235.php and
http://sourceforge.net/mailarchive/forum.php?thread_name=CAH8yC8k0QAqj%2B4eyQ%3D20aH11Tnb7m43%3DxjCdkxKZY8ssgf3rfg%40mail.gmail.com&forum_name=valgrind-users.

>> But I'm having problems interpreting what object is being problematic.
> [...]
>> #21 0x00002ba649808630 in __cxa_finalize (d=0x2ba64901c3d8)
>>     at cxa_finalize.c:56
>>         check = <value optimized out>
>>         cxafn = 0x6
>>         cxaarg = 0xa46
>>         f = 0x13925e0
>>         funcs = 0x13924f0
>> (gdb) po 0x13925e0
>> evaluation of this expression requires the target program to be active
>
> print-object is for Objecttive-C.  And you will decode there just the info you
> see in the frame #20.
Indeed! I've been working on iPhones/iPads for too long.

> This looks as a memory corruption, it is better to debug instrument the code
> leading to this state, this core file may no longer be too useful.
Ok, thanks.

To retain info on the objects in question, do I need to compile with
g++ -v and save the intermediate (ii?) files?

Jeff

> [...]
>> #20 0x0000000000470c0a in ~map (this=0x83dfc0, __in_chrg=<value optimized out>)
>>     at /usr/include/c++/4.4/bits/stl_map.h:87
>> No locals.
>> #21 0x00002ba649808630 in __cxa_finalize (d=0x2ba64901c3d8)
>>     at cxa_finalize.c:56
>>         check = <value optimized out>
>>         cxafn = 0x6
>>         cxaarg = 0xa46
>>         f = 0x13925e0
>>         funcs = 0x13924f0
>> #22 0x00002ba648c59076 in __do_global_dtors_aux () from lib/libesapi-c++.so
>> No symbol table info available.
>> #23 0x0000000000000000 in ?? ()
>> No symbol table info available.
>> (gdb)
>

Re: Interpret object causing crash in __cxa_finalize (have core)

[Jan Kratochvil]

On Wed, 24 Aug 2011 21:54:09 +0200, Jeffrey Walton wrote:
> Boost has made Valgrind useless (15000 line of output). And I have not
> been successful in getting suppression rules:
> http://lists.boost.org/boost-users/2011/08/70235.php and
> http://sourceforge.net/mailarchive/forum.php?thread_name=CAH8yC8k0QAqj%2B4eyQ%3D20aH11Tnb7m43%3DxjCdkxKZY8ssgf3rfg%40mail.gmail.com&forum_name=valgrind-users.

You do not need to track memory leaks but you should track memory corruptions.
You was told the same in the mails.


> To retain info on the objects in question, do I need to compile with
> g++ -v and save the intermediate (ii?) files?

I do not see any missing debug info in your backtrace.

g++ uses -g for debug info, not -v.

You did not tell which platform do you run on but it seems to me like
GNU/Linux, debug info is stored there in the binary files or in separate
.debug info files (one file per one library) in /usr/lib/debug.  The debug
info stored in object files is specific only to Apple OSes.

But after all you have all the debug info you can have in that backtrace.


Regards,
Jan

Re: Interpret object causing crash in __cxa_finalize (have core)

[Jeffrey Walton]

On Wed, Aug 24, 2011 at 4:03 PM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> On Wed, 24 Aug 2011 21:54:09 +0200, Jeffrey Walton wrote:
>> Boost has made Valgrind useless (15000 line of output). And I have not
>> been successful in getting suppression rules:
>> http://lists.boost.org/boost-users/2011/08/70235.php and
>> http://sourceforge.net/mailarchive/forum.php?thread_name=CAH8yC8k0QAqj%2B4eyQ%3D20aH11Tnb7m43%3DxjCdkxKZY8ssgf3rfg%40mail.gmail.com&forum_name=valgrind-users.
>
> You do not need to track memory leaks but you should track memory corruptions.
> You was told the same in the mails.
We are also interested in memory leaks - other libraries affect our integrity.

OT: we're finding these other libraries are somewhat sloppy, and are
affecting our ability to analyze our stuff. They need to fix their
gear, or we need to find suppression rules.

>> To retain info on the objects in question, do I need to compile with
>> g++ -v and save the intermediate (ii?) files?
>
> I do not see any missing debug info in your backtrace.
>
> g++ uses -g for debug info, not -v.
Yes, we have -g3 -ggdb. But we seem to be missing diagnostic
information from __do_global_dtors_aux and __cxa_finalize.

> You did not tell which platform do you run on but it seems to me like
> GNU/Linux, debug info is stored there in the binary files or in separate
> .debug info files (one file per one library) in /usr/lib/debug.  The debug
> info stored in object files is specific only to Apple OSes.
My bad. This is a C++ library (non-Apple).

$ uname -a
Linux studio 2.6.32-33-generic #72-Ubuntu SMP Fri Jul 29 21:07:13 UTC
2011 x86_64 GNU/Linux

> But after all you have all the debug info you can have in that backtrace.
OK, I' seem to have a misconception. Is there no debug information
associated with global constructors and destructors? If not, how does
one determine the problematic object destructor? Lumping everything
into __cxa_finalize is not helpful.

Jeff

Re: Interpret object causing crash in __cxa_finalize (have core)

[Jan Kratochvil]

On Wed, 24 Aug 2011 22:18:45 +0200, Jeffrey Walton wrote:
> We are also interested in memory leaks - other libraries affect our integrity.

Then please do not complain on too many messages, the mails said Boost is not
conforming to memory leak checkers (that does not mean it really leaks).
This mail thread is about memory corruption, not about memory leaks.


> Yes, we have -g3 -ggdb.

-ggdb in fact has no effect.


> But we seem to be missing diagnostic
> information from __do_global_dtors_aux and __cxa_finalize.

You have full debug info from __cxa_finalize, what more info would you like?

__do_global_dtors_aux just executes all the destructors, there isn't anything
interesting inside.  It is assembled from gcc sources some special ways so its
debug info is missing.


> OK, I' seem to have a misconception. Is there no debug information
> associated with global constructors and destructors?

Global destructor is __cxa_finalize which has debug info.  It runs destructors
for all the existing instances, for instance 0x83dfc0 is run the destructor
~map, it also has debug info.


> how does one determine the problematic object destructor?

There probably isn't any problematic object destructor.  Just some code before
corrupted memory so the correct object destructor crashes on it later.

Please read more about memory corruption debugging, there is a wide range of
tools for it, GDB is not one of them.  I have written one brief list of such
tools in the part 1 of:
	http://people.redhat.com/jkratoch/DeveloperConference2011-debug.pdf


Regards,
Jan