[Bug threads/10757] New: GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[Bug threads/10757] New: GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[ppluzhnikov at google dot com]

Possibly related to PR8963.

This was originally reported with gdbserver across a relatively high-latency
connection: attach remote process, set breakpoints, continue:

  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  The program no longer exists.

Reproduces with attached test case on Fedora 11 with both
  GNU gdb (GDB) Fedora (6.8.50.20090302-38.fc11)
  GNU gdb (GDB) 7.0.50.20091011-cvs

$ ulimit -s 32
$ ./a.out 20 &
[2] 16717

$ ~/gdb-cvs/build/gdb/gdb --pid $(pgrep a.out)
GNU gdb (GDB) 7.0.50.20091011-cvs
...
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 16717
Reading symbols from /home/paul/tmp/a.out...done.
Reading symbols from /lib/libpthread.so.0...Reading symbols from
/usr/lib/debug/lib/libpthread-2.10.1.so.debug...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f3eb70 (LWP 5139)]
(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...Reading symbols from
/usr/lib/debug/lib/libc-2.10.1.so.debug...done.
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...Reading symbols from
/usr/lib/debug/lib/ld-2.10.1.so.debug...done.
(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
0x001ef424 in __kernel_vsyscall ()
(gdb) b foo
Breakpoint 1 at 0x80485c7: file manythreads2.c, line 10.
(gdb) commands 1
>c
>end
(gdb) c
[Thread 0xb7f3eb70 (LWP 5139) exited]

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
The program no longer exists.
(gdb) quit
[2]+  Trace/breakpoint trap   (core dumped) ./a.out 20


It appears that to trigger the bug, I need a program which rapidly creates
threads while gdb/gdbserver is attaching.

The same program works fine when started from within GDB. Race in attach?

-- 
           Summary: GDB does not attach all threads of a multithreaded
                    process => inferior gets SIGTRAP
           Product: gdb
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: threads
        AssignedTo: unassigned at sourceware dot org
        ReportedBy: ppluzhnikov at google dot com
                CC: gdb-prs at sourceware dot org
 GCC build triplet: i686-pc-linux-gnu
  GCC host triplet: i686-pc-linux-gnu
GCC target triplet: i686-pc-linux-gnu


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2009-10-11 17:22 -------
Created an attachment (id=4270)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=4270&action=view)
test case


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2009-10-11 21:47 -------
The same problem appears to happen on Solaris-10/i686 with GDB-6.6 (which Sun
ships), but not with GDB-cvs.

[1]+  Running                 ./a.out 40 &
$ gdb ./a.out 26519                    
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-pc-solaris2.10"...
Attaching to program `/export/home/paul/tmp/a.out', process 26519
Reading symbols from /lib/libpthread.so.1...
warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 00000074
done.
Loaded symbols for /lib/libpthread.so.1
Reading symbols from /lib/libc.so.1...done.
Loaded symbols for /lib/libc.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
sol-thread active.
Retry #1:
Retry #2:
Retry #3:
Retry #4:
[New LWP    1        ]
[New Thread 1 (LWP 1)]
Symbols already loaded for /lib/libpthread.so.1
Symbols already loaded for /lib/libc.so.1
Symbols already loaded for /lib/ld.so.1
[Switching to Thread 1 (LWP 1)]
0xfef19735 in ___nanosleep () from /lib/libc.so.1
(gdb) b foo
Breakpoint 1 at 0x8050a83: file manythreads.c, line 10.
(gdb) c
Continuing.
[New LWP    183438        ]
[New LWP    186777        ]
[New LWP    186756        ]
[New LWP    186778        ]
[New Thread 183438 (LWP 183438)]
[Switching to Thread 183438 (LWP 183438)]

Breakpoint 1, foo () at manythreads.c:10
10        return NULL;
(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>silent
>c
>end
(gdb) c
Continuing.
[LWP    183438         exited]
[New LWP    186738        ]
procfs: fetch_registers, get_gregs line 3706, /proc/26519/lwp/183438: No such
file or directory.
(gdb) q
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /export/home/paul/tmp/a.out, process 26519
$ 
[1]+  Trace/Breakpoint Trap   (core dumped) ./a.out 40


With GDB-cvs, not all is smooth either:

./a.out 40 &
[1] 26525
$ 
$ 
$ 
$ ~/gdb-cvs/build/gdb/gdb ./a.out 26525
GNU gdb (GDB) 7.0.50.20091011-cvs
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-pc-solaris2.10".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /export/home/paul/tmp/a.out...done.
Attaching to program `/export/home/paul/tmp/a.out', process 26525
[New process 26525]
Retry #1:
Retry #2:
Retry #3:
Retry #4:
Reading symbols from /lib/libpthread.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.1
Reading symbols from /lib/libc.so.1...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New LWP    57697        ]
[New LWP    57681        ]
[New LWP    57678        ]
[New LWP    57698        ]
[New LWP    57690        ]
[New LWP    57644        ]
[New LWP    57703        ]
[New LWP    57708        ]
[New LWP    57711        ]
[New LWP    57640        ]
[New LWP    57652        ]
[New LWP    57653        ]
[New LWP    57662        ]
[New LWP    57663        ]
[New LWP    57672        ]
[New LWP    57692        ]
[New LWP    57707        ]
[New LWP    57685        ]
[New LWP    57704        ]
[New LWP    57673        ]
[New LWP    57661        ]
[New LWP    57683        ]
[New LWP    57686        ]
[New LWP    57674        ]
[New LWP    57712        ]
[New LWP    57717        ]
[New LWP    57675        ]
[New LWP    57650        ]
[New LWP    57695        ]
[New LWP    57651        ]
[New LWP    57684        ]
[New LWP    57706        ]
[New LWP    57658        ]
[New LWP    57647        ]
[New LWP    57680        ]
[New LWP    57699        ]
[New LWP    57718        ]
[New LWP    57669        ]
[New LWP    57642        ]
[New LWP    57657        ]
[New LWP    57659        ]
[New LWP    57641        ]
[New LWP    57719        ]
[New LWP    57649        ]
[New LWP    57710        ]
[New LWP    57682        ]
[New LWP    57702        ]
[New LWP    57667        ]
[New LWP    57670        ]
[New LWP    57679        ]
[New LWP    57688        ]
[New LWP    57696        ]
[New LWP    57689        ]
[New LWP    57705        ]
[New LWP    57713        ]
[New LWP    57654        ]
[New LWP    57666        ]
[New LWP    57648        ]
[New LWP    57664        ]
[New LWP    57714        ]
[New LWP    57701        ]
[New LWP    57645        ]
[New LWP    57656        ]
[New LWP    57716        ]
[New LWP    57671        ]
[New LWP    57676        ]
[New LWP    57715        ]
[New LWP    57668        ]
[New LWP    57655        ]
[New LWP    57691        ]
[New LWP    57646        ]
[New LWP    57665        ]
[New LWP    57694        ]
[New LWP    57687        ]
[New LWP    57643        ]
[New LWP    57660        ]
[New LWP    57677        ]
[New LWP    57709        ]
[New LWP    57700        ]
[New LWP    57693        ]
[New Thread 1 (LWP 1)]
[New Thread 57680 (LWP 57680)]
[New Thread 57681 (LWP 57681)]
[New Thread 57682 (LWP 57682)]
[New Thread 57683 (LWP 57683)]
[New Thread 57684 (LWP 57684)]
[New Thread 57685 (LWP 57685)]
[New Thread 57686 (LWP 57686)]
[New Thread 57687 (LWP 57687)]
[New Thread 57688 (LWP 57688)]
[New Thread 57689 (LWP 57689)]
[New Thread 57690 (LWP 57690)]
[New Thread 57691 (LWP 57691)]
[New Thread 57692 (LWP 57692)]
[New Thread 57693 (LWP 57693)]
[New Thread 57694 (LWP 57694)]
[New Thread 57695 (LWP 57695)]
[New Thread 57696 (LWP 57696)]
[New Thread 57697 (LWP 57697)]
[New Thread 57698 (LWP 57698)]
[New Thread 57699 (LWP 57699)]
[New Thread 57700 (LWP 57700)]
[New Thread 57701 (LWP 57701)]
[New Thread 57702 (LWP 57702)]
[New Thread 57703 (LWP 57703)]
[New Thread 57704 (LWP 57704)]
[New Thread 57705 (LWP 57705)]
[New Thread 57706 (LWP 57706)]
[New Thread 57707 (LWP 57707)]
[New Thread 57708 (LWP 57708)]
[New Thread 57709 (LWP 57709)]
[New Thread 57710 (LWP 57710)]
[New Thread 57711 (LWP 57711)]
[New Thread 57712 (LWP 57712)]
[New Thread 57713 (LWP 57713)]
[New Thread 57714 (LWP 57714)]
[New Thread 57715 (LWP 57715)]
[New Thread 57716 (LWP 57716)]
[New Thread 57717 (LWP 57717)]
[New Thread 57718 (LWP 57718)]
[New Thread 57719 (LWP 57719)]
[New Thread 57640        ]
[New Thread 57641        ]
[New Thread 57642        ]
[New Thread 57643        ]
[New Thread 57644        ]
[New Thread 57645        ]
[New Thread 57646        ]
[New Thread 57647        ]
[New Thread 57648        ]
[New Thread 57649        ]
[New Thread 57650        ]
[New Thread 57651        ]
[New Thread 57652        ]
[New Thread 57653        ]
[New Thread 57654        ]
[New Thread 57655        ]
[New Thread 57656        ]
[New Thread 57657        ]
[New Thread 57658        ]
[New Thread 57659        ]
[New Thread 57660        ]
[New Thread 57661        ]
[New Thread 57662        ]
[New Thread 57663        ]
[New Thread 57664        ]
[New Thread 57665        ]
[New Thread 57666        ]
[New Thread 57667        ]
[New Thread 57668        ]
[New Thread 57669        ]
[New Thread 57670        ]
[New Thread 57671        ]
[New Thread 57672        ]
[New Thread 57673        ]
[New Thread 57674        ]
[New Thread 57675        ]
[New Thread 57676        ]
[New Thread 57677        ]
[New Thread 57678        ]
[New Thread 57679        ]
Loaded symbols for /lib/libc.so.1
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
[Switching to Thread 1 (LWP 1)]
0xfef19735 in ___nanosleep () from /lib/libc.so.1
(gdb) b foo
Breakpoint 1 at 0x8050a83: file manythreads.c, line 10.
(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>silent
>c
>end
(gdb) c
Continuing.
[New LWP    57720        ]
[Switching to Thread 57680 (LWP 57680)]
[LWP    57680         exited]
procfs: fetch_registers, get_gregs line 3799, /proc/26525/lwp/57680: No such
file or directory.
(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>end
c
procfs: fetch_registers, get_gregs line 3799, /proc/26525/lwp/57680: No such
file or directory.
(gdb) c                         
Continuing.
procfs: fetch_registers, get_gregs line 3799, /proc/26525/lwp/57680: No such
file or directory.
(gdb) c
Continuing.
procfs: fetch_registers, get_gregs line 3799, /proc/26525/lwp/57680: No such
file or directory.
(gdb) q
A debugging session is active.

        Inferior 1 [process 26525    ] will be detached.

Quit anyway? (y or n) y
Detaching from program: /export/home/paul/tmp/a.out, process 26525
$ 
$ jobs
[1]+  Running                 ./a.out 40 &


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2009-10-11 22:28 -------
Subject: Re:  New: GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

On Sunday 11 October 2009 18:19:08, ppluzhnikov at google dot com wrote:
> The same program works fine when started from within GDB. Race in attach?

Sounds like it.   gdb/gdbserver/linux-low.c:linux_attach_lwp_1
describes some races.  Check if this case fits one of the described
there.

It looks like we manage to install (a) breakpoint(s) in the inferior's
address space before attaching to all threads.  This looks suspicious:

gdbserver/thread-db.c:

>  if (thread_db_load_search ())
>    {
>      if (use_events && thread_db_enable_reporting () == 0)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^

This installs the thread event breakpoint.

>	{
>	  /* Keep trying; maybe event reporting will work later.  */
>	  thread_db_free (proc);
>	  return 0;
>	}
>      thread_db_find_new_threads ();

And only after do we go about attaching to all threads.

>      thread_db_look_up_symbols ();
>      proc->all_symbols_looked_up = 1;
>      return 1;
>    }

Do check if your setup needed thread event breakpoints
at all (use_events), there could be other races.  It could be
a breakpoint set from GDB.  Enable "set debug remote 1", and check
if GDB installs any breakpoint in the inferior before
the "qSymbol" packet that makes gdbserver able to find and
load thread_db.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2009-10-11 22:33 -------
Subject: Re:  GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

Also, if you let the app that crashed with a SIGTRAP dump
core, you could load its core into gdb to check which breakpoint
trapped (well, the address of the breakpoint).  Remember to disable
address space randomization to make your life easier.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2009-10-11 23:27 -------
(In reply to comment #3)
> gdb/gdbserver/linux-low.c:linux_attach_lwp_1
> describes some races.

Please note that gdbserver is not involved in this bug report.

SIGTRAP is happening with 'gdb --pid' or '(gdb) attach'.
When using '(gdb) run' instead, everything works fine.

(In reply to comment #4)

> Also, if you let the app that crashed with a SIGTRAP dump
> core, you could load its core into gdb to check which breakpoint
> trapped

In all cores I've examined, it was the breakpoint on foo().
If I don't insert it, then I never get SIGTRAP.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2009-10-11 23:51 -------
Subject: Re:  GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

On Monday 12 October 2009 00:27:46, ppluzhnikov at google dot com wrote:

> Please note that gdbserver is not involved in this bug report.
> SIGTRAP is happening with 'gdb --pid' or '(gdb) attach'.
> When using '(gdb) run' instead, everything works fine.

Ah, I got confused with the "This was originally reported
with gdbserver".  gdb/linux-thread-db.c seems to have the same
race anyway, but only if "set breakpoint always-inserted on",
as otherwise, breakpoint are only installed on the next
resume, after finding threads.

> In all cores I've examined, it was the breakpoint on foo().
> If I don't insert it, then I never get SIGTRAP.

This seems to confirm that GDB is inserting breakpoints
into the inferior's address space before having attached
to all threads.  Either that, or for some reason GDB is
managing to end up not attached to some but not all
threads when attaching to an inferior that is
spawning threads.  On an NTPL system, clone children of
ptraced lwps are already ptraced attached as well, although
I wouldn't discount a kernel bug (RedHat systems
being more suspectible to such bugs due to ptrace
on top of utrace reimplementation).  There could also
be an problem with new threads being added to glibc's
pthreads list just while thread_db is iterating
over them so that the iterator could miss new threads.
Dunno.

The basic premise is that GDB needs to attach to every
LWP running in the inferior, not just the main LWP.  If
an LWP hits a breakpoint before GDB having PTRACE_ATTACHed
to it, it will simply die with a SIGTRAP, because ptrace
won't intercept it.

Perhaps simply a breakpoint on "insert_breakpoints" can
confirm that gdb is inserting breakpoints too soon.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2009-10-12 00:56 -------
Subject: Re:  GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

On my machine (ubuntu 8.04, 2.6.24-24-generic x86_64) I haven't
managed to get a SIGTRAP, but I see badness nonetheless.

>ulimit -s 32
>~/manythreads2 20&
[1] 26139
>./gdb --pid $(pidof manythreads2)
(...)
[New Thread 0x4d686950 (LWP 418)]
[New Thread 0x40b46950 (LWP 434)]

warning: Can't attach LWP 449: Operation not permitted
[New Thread 0x4de87950 (LWP 476)]
[New Thread 0x4fe8b950 (LWP 489)]
[New Thread 0x4be83950 (LWP 510)]
[New Thread 0x51e8f950 (LWP 529)]

warning: Can't attach LWP -1: No such process
[New Thread 0x49e7f950 (LWP 537)]
(...)

>./gdb -ex "set pagination off" --pid $(pidof manythreads2)
GNU gdb (GDB) 7.0.50.20091011-cvs
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
/home/pedro/.gdbinit:35: Error in sourced command file:
Undefined set remote command: "username gemini".  Try "help set remote".
Attaching to process 8699
Reading symbols from /home/pedro/manythreads2...done.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x4087f950 (LWP 3678)]
[New Thread 0x40481950 (LWP 3698)]

warning: Can't attach LWP 3697: No such process
[New Thread 0x4065b950 (LWP 3733)]
[New Thread 0x41ecc950 (LWP 3753)]
[New Thread 0x404b1950 (LWP 3752)]
[New Thread 0x41bc7950 (LWP 3750)]
[New Thread 0x403f5950 (LWP 3748)]
[New Thread 0x40353950 (LWP 3746)]
[New Thread 0x40ae0950 (LWP 3745)]
[New Thread 0x40322950 (LWP 3744)]
[New Thread 0x4040d950 (LWP 3743)]
[New Thread 0x401a2950 (LWP 3742)]
[New Thread 0x403b5950 (LWP 3741)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f7cdd611b81 in nanosleep () from /lib/libc.so.6
Setting up the environment for debugging gdb.
Function "internal_error" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
Function "info_command" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
/home/pedro/gdb/baseline/build/gdb/.gdbinit:8: Error in sourced command file:
No breakpoint number 0.
(gdb) b foo
During symbol reading, DW_AT_name missing from DW_TAG_base_type.
During symbol reading, debug info gives in-file macro definition with zero line 0: __STDC__ 1.
Breakpoint 1 at 0x40077c: file manythreads2.c, line 10.
(gdb) commands
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>c
>end
(gdb) c
Continuing.
[Thread 0x403b5950 (LWP 3741) exited]
[Thread 0x401a2950 (LWP 3742) exited]
[Thread 0x4040d950 (LWP 3743) exited]
[Thread 0x40322950 (LWP 3744) exited]
[Thread 0x40ae0950 (LWP 3745) exited]
[Thread 0x40353950 (LWP 3746) exited]
[Thread 0x403f5950 (LWP 3748) exited]
[Thread 0x41bc7950 (LWP 3750) exited]
[Thread 0x404b1950 (LWP 3752) exited]
[Thread 0x41ecc950 (LWP 3753) exited]
[Thread 0x40481950 (LWP 3698) exited]
[Thread 0x4087f950 (LWP 3678) exited]

<ctrl-c> doesn't work.

>cat /proc/8699/status | grep State
State:  Z (zombie)

and 

>ls /proc/8699/task/
13546/ 13547/ 13548/ 13549/ 13550/ 13551/ 13552/ 13553/ 13554/ 3733/  8699/

and they're all zombie.


I'm suspecting something like this:

> There could also
> be an problem with new threads being added to glibc's
> pthreads list just while thread_db is iterating
> over them so that the iterator could miss new threads.

Look here:

>./gdb -ex "set pagination off" --pid $(pidof manythreads2)
GNU gdb (GDB) 7.0.50.20091011-cvs
(...)
Attaching to process 14235
Reading symbols from /home/pedro/manythreads2...done.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x40019950 (LWP 31166)]
[New Thread 0x403d4950 (LWP 31184)]
[New Thread 0x4162e950 (LWP 31200)]
[New Thread 0x41f75950 (LWP 31214)]
[New Thread 0x41636950 (LWP 31226)]
[New Thread 0x41666950 (LWP 31238)]
[New Thread 0x4163e950 (LWP 31237)]
[New Thread 0x418bc950 (LWP 31236)]
[New Thread 0x4055b950 (LWP 31281)]
[New Thread 0x403cc950 (LWP 31299)]
[New Thread 0x41646950 (LWP 31315)]
[New Thread 0x4166e950 (LWP 31329)]
[New Thread 0x40a5a950 (LWP 31341)]
[New Thread 0x41144950 (LWP 31353)]
[New Thread 0x4071f950 (LWP 31352)]
[New Thread 0x41656950 (LWP 31347)]
[New Thread 0x41626950 (LWP 31279)]
[New Thread 0x4165e950 (LWP 31260)]
[New Thread 0x404df950 (LWP 31248)]
[New Thread 0x41776950 (LWP 31232)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007fe54bab4b81 in nanosleep () from /lib/libc.so.6
Setting up the environment for debugging gdb.
Function "internal_error" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
Function "info_command" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
/home/pedro/gdb/baseline/build/gdb/.gdbinit:8: Error in sourced command file:
No breakpoint number 0.


Now, if we had attached and managed to freeze all threads, "info threads"
wouldn't show "New Thread"s, but it does:

(gdb) info threads
[New Thread 0x4000e950 (LWP 25558)]
[New Thread 0x4167e950 (LWP 25557)]
[New Thread 0x4002e950 (LWP 25556)]
[New Thread 0x403e4950 (LWP 25394)]
[New Thread 0x403ec950 (LWP 25560)]
[New Thread 0x41d49950 (LWP 31359)]
[New Thread 0x40727950 (LWP 31354)]
...
  28 Thread 0x40727950 (LWP 31354)  0x00007fe54bd80796 in pthread_join () from /lib/libpthread.so.0
  27 Thread 0x41d49950 (LWP 31359)  0x00007fe54bd80796 in pthread_join () from /lib/libpthread.so.0
...


This means that GDB hadn't been attached to those threads yet.  They
were still running free.  One of those would be your SIGTRAP thread.
If one does enough "info threads" (perhaps one or two more), GDB should
manage to attach to all threads (and stop reporting "New Threads"s).

After that, you shouldn't be able to reproduce the SIGTRAP.

I guess one way to fix it would be to to have try_thread_db_load_1
know that we're attaching, and make it keep looking for new
threads until no new thread comes out.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2009-10-12 01:10 -------
Subject: Re:  GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

On Monday 12 October 2009 01:56:10, pedro at codesourcery dot com wrote:

> On my machine (ubuntu 8.04, 2.6.24-24-generic x86_64) I haven't
> managed to get a SIGTRAP, but I see badness nonetheless.

Bah, update: I've just gotten a SIGTRAP.  I just needed:

 "attach; b foo; c"

Indeed I never see the problem when I do 

 "attach; info threads; b foo; c"

Sorry for not having confirmed what was going on
before digressing into breakpoint insertion issues.  :-)

Sometimes I also see this when attaching...:

 warning: Can't attach LWP 15338: No such process
 ../../src/gdb/linux-nat.c:1341: internal-error: linux_nat_post_attach_wait: Assertion `pid == new_pid && WIFSTOPPED (status)' failed.
 A problem internal to GDB has been detected,
 further debugging may prove unreliable.
 Quit this debugging session? (y or n)     

It seems that the first event we get right after PTRACE_ATTACH can be an
lwp exit.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[jan dot kratochvil at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jan dot kratochvil at redhat
                   |                            |dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2009-10-15 18:17 -------
Subject: Bug 10757

CVSROOT:	/cvs/src
Module name:	src
Changes by:	ppluzhnikov@sourceware.org	2009-10-15 18:17:40

Modified files:
	gdb            : ChangeLog 

Log message:
	Forgot to mention PR gdb/10757.

Patches:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog.diff?cvsroot=src&r1=1.10966&r2=1.10967



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2009-10-27 21:33 -------
Subject: Bug 10757

CVSROOT:	/cvs/src
Module name:	src
Changes by:	ppluzhnikov@sourceware.org	2009-10-27 21:32:49

Modified files:
	gdb            : ChangeLog linux-thread-db.c 

Log message:
	2009-10-27  Paul Pluzhnikov  <ppluzhnikov@google.com>
	
	PR gdb/10757
	* linux-thread-db.c (attach_thread): Return success/failure
	indicator.
	(thread_db_find_new_threads_silently): Retry until no new threads.
	(struct callback_data): New.
	(find_new_threads_callback): Count new threads, stop iteration
	on error.
	(find_new_threads_once): New function.
	(thread_db_find_new_threads_2): Rename from
	thread_db_find_new_threads_1 and adjust.
	(thread_db_find_new_threads_1): New function.

Patches:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog.diff?cvsroot=src&r1=1.11007&r2=1.11008
http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/gdb/linux-thread-db.c.diff?cvsroot=src&r1=1.65&r2=1.66



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2009-10-28 17:03 -------
Subject: Bug 10757

CVSROOT:	/cvs/src
Module name:	src
Changes by:	ppluzhnikov@sourceware.org	2009-10-28 17:03:17

Modified files:
	gdb/gdbserver  : ChangeLog thread-db.c 

Log message:
	2009-10-28  Paul Pluzhnikov  <ppluzhnikov@google.com>
	
	PR gdb/10757
	* thread-db.c (attach_thread): New function.
	(maybe_attach_thread): Return success/failure.
	(find_new_threads_callback): Adjust.
	(thread_db_find_new_threads): Loop until no new threads.

Patches:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/gdb/gdbserver/ChangeLog.diff?cvsroot=src&r1=1.292&r2=1.293
http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/gdb/gdbserver/thread-db.c.diff?cvsroot=src&r1=1.24&r2=1.25



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2009-10-31 15:05 -------
Paul, should we close this?


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2009-10-31 17:26 -------
Solaris problem still exists, but it's apparently sufficiently different. I've
opened PR10879 for it.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2009-12-08 17:11 -------
*** Bug 9892 has been marked as a duplicate of this bug. ***

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ich at az2000 dot de


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug threads/10757] GDB does not attach all threads of a multithreaded process => inferior gets SIGTRAP

[esigra at gmail dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |esigra at gmail dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10757

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.