gdbserver and "terminated with signal SIGTRAP"

gdbserver and "terminated with signal SIGTRAP"

[Atsushi Nemoto]

I got "Child terminated with signal = 5" message when debugging this
tiny program with gdbserver.  Setting breakpoint on "func" (which is
an entry point of thread) and doing "continue" can reproduce this
problem.  I'm running gdb and gdbserver on same host (linux/x86).

--- foo.c ---
#include <stdlib.h>
#include <pthread.h>

void *func(void *data)
{
	return data;
}

int main(int argc, char **argv) 
{
        pthread_t id;
	int i;

	for (i = 0; i < 4; i++) {
		pthread_create(&id, NULL, func, NULL);
		pthread_join(id, NULL);
	}
	return 0;
}
--- --- ---

I tried gdb 6.4.90 (Debian 4.0), 6.5 (FC6), 6.6 (FC7), vanilla 6.7.1
on i386 and got same results on all of them.  The current cvs snapshot
(gdb-6.7.50.20071129) did not cause this problem, but I want to know
how to fix older versions.


Here is a session log (both side).

--- host (gdb) side ---
$ gdb ./foo
GNU gdb Red Hat Linux (6.5-15.fc6rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) target remote localhost:50000
Remote debugging using localhost:50000
0x00b8f810 in ?? ()
(gdb) b func
Breakpoint 1 at 0x8048437: file foo.c, line 6.
(gdb) c
Continuing.
[New thread 19500]
[New thread 19502]
[Switching to thread 19502]

Breakpoint 1, func (data=0x0) at foo.c:6
6               return data;
(gdb) c
Continuing.

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
The program no longer exists.
--- --- ---

--- target (gdbserver) side ---
$ gdbserver localhost:50000 ./foo
Process ./foo created; pid = 19500
Listening on port 50000
Remote debugging from host 127.0.0.1

Child terminated with signal = 5 

Child terminated with signal = 0x5
GDBserver exiting
--- --- ---

And here is a session log on gdbserver side (gdb-6.7.1) with
debug_threads=1.

--- target (gdbserver) side (debug_thread=1) ---
Process ./foo created; pid = 10926
Got an event from 10926 (57f)
stop pc (before any decrement) is 00b8f810
stop pc (before any decrement) is 00b8f810
Hit a (non-reinsert) breakpoint.
Listening on port 50000
Remote debugging from host 127.0.0.1
Writing 24 to 00b9d7a0
Writing cc to 00b9d7a0
Writing cc to 08048437
Resuming, no pending status
Resuming process 10926 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00b8f810
Got an event from 10926 (57f)
stop pc (before any decrement) is 00b9d7a1
stop pc (before any decrement) is 00b9d7a1
Hit a (non-reinsert) breakpoint.
Writing 55 to 00b9d7a0
Writing 8b to 08048437
Resuming, no pending status
Resuming process 10926 (step, signal 0, stop not expected)
Checking for breakpoint in process 10926.
stop pc (before any decrement) is 00b9d7a0
Ignoring, PC was changed.  Old PC was 0x00b9d7a0
  stop pc (before any decrement) is 00b9d7a0
Got an event from 10926 (57f)
stop pc (before any decrement) is 00b9d7a1
stop pc (before any decrement) is 00b9d7a1
Hit a (non-reinsert) breakpoint.
Writing cc to 00b9d7a0
Writing cc to 08048437
Resuming, no pending status
Resuming process 10926 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00b9d7a1
Got an event from 10926 (57f)
stop pc (before any decrement) is 00b9d7a1
stop pc (before any decrement) is 00b9d7a1
Hit a (non-reinsert) breakpoint.
Writing 55 to 00b9d7a0
Writing 8b to 08048437
Writing 80 to 00dfa16c
Writing cc to 00de87d0
Found first thread -1208305984 (LWP 10926)
Writing 01 to b7fab8a9
Resuming, no pending status
Resuming process 10926 (step, signal 0, stop not expected)
Checking for breakpoint in process 10926.
stop pc (before any decrement) is 00b9d7a1
Removed breakpoint.
set pc to 00b9d7a0
  stop pc (before any decrement) is 00b9d7a0
Got an event from 10926 (57f)
stop pc (before any decrement) is 00b9d7a1
stop pc (before any decrement) is 00b9d7a1
Hit a (non-reinsert) breakpoint.
Writing resume reply for -1208305984

Writing cc to 00b9d7a0
Writing cc to 08048437
Resuming, no pending status
Resuming process 10926 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00b9d7a1
Got an event from 10926 (57f)
stop pc (before any decrement) is 00de87d1
stop pc (before any decrement) is 00de87d1
Hit a (non-reinsert) breakpoint.
Thread creation event.			# The 1st thread creation
Writing 00 to b7faadac
Writing 00 to 00dfa174
Attaching to thread -1208308848 (LWP 10927)	# 1st attach
Writing 01 to b7faad79
Writing 55 to 00de87d0
Resuming process 10926 (step, signal 0, stop not expected)
  pending reinsert at 00de87d0Checking for breakpoint in process 10926.
stop pc (before any decrement) is 00de87d1
Removed breakpoint.
set pc to 00de87d0
  stop pc (before any decrement) is 00de87d0
Got an event from 10926 (57f)
stop pc (before any decrement) is 00de87d1
stop pc (before any decrement) is 00de87d1
Reinserted breakpoint.
Writing cc to 00de87d0
Resuming process 10926 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00de87d1
Got an event from 10927 (137f)
stop pc (before any decrement) is 00110402
Expected stop.
Resuming process 10927 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00110402
Got an event from 10927 (57f)
stop pc (before any decrement) is 08048438
stop pc (before any decrement) is 08048438
Hit a (non-reinsert) breakpoint.
Sending sigstop to process 10926
Got an event from 10926 (137f)
stop pc (before any decrement) is 00110402
Writing resume reply for -1208308848

Writing 55 to 00b9d7a0
Writing 8b to 08048437
---
# stop at first breakpoint here.
---
Resuming, no pending status
Resuming process 10927 (step, signal 0, stop not expected)
Checking for breakpoint in process 10927.
stop pc (before any decrement) is 08048437
Ignoring, PC was changed.  Old PC was 0x08048437
  stop pc (before any decrement) is 08048437
Got an event from 10927 (57f)
stop pc (before any decrement) is 0804843a
stop pc (before any decrement) is 0804843a
Hit a (non-reinsert) breakpoint.
Writing resume reply for -1208308848

Writing cc to 00b9d7a0
Writing cc to 08048437
Resuming, no pending status
Resuming process 10926 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00110402
Resuming process 10927 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 0804843a
Got an event from 10926 (57f)
stop pc (before any decrement) is 00de87d1
stop pc (before any decrement) is 00de87d1
Hit a (non-reinsert) breakpoint.
Thread creation event.			# The 2nd thread creation
Writing 00 to b7faadac
Writing 00 to 00dfa174
Writing 55 to 00de87d0			# no attach here...
Resuming process 10926 (step, signal 0, stop not expected)
  pending reinsert at 00de87d0Checking for breakpoint in process 10926.
stop pc (before any decrement) is 00de87d1
Removed breakpoint.
set pc to 00de87d0
  stop pc (before any decrement) is 00de87d0
Got an event from 10927 (0)
Thread -1208308848 (LWP 10927) exiting	# detect death of 1st thread here???
Got an event from 10926 (57f)
stop pc (before any decrement) is 00de87d1
stop pc (before any decrement) is 00de87d1
Reinserted breakpoint.
Writing cc to 00de87d0
Resuming process 10926 (continue, signal 0, stop not expected)
  stop pc (before any decrement) is 00de87d1
Got an event from 10926 (5)		# terminated (not stopped) by bp
Thread -1208305984 (LWP 10926) exiting

Child terminated with signal = 5 

Child terminated with signal = 0x5 (SIGTRAP)
GDBserver exiting

Program exited normally.
--- --- ---


After second "Thread creation event." message, there is no "Attaching
to thread ..." message.  So the new thread did not attached, then
breakpoint just terminate (not stop) the program.

I confirmed that maybe_attach_thread() in thread-db.c did not call
linux_attach_lwp() on the second time.  That was because
find_inferior_id() returned non NULL value.  It seems
find_inferior_id() returned the first thread (which should be dead
before the second thread creation).


Any suggestion to how to fix this (or how this is fixed on current
CVS) are welcome.  Thank you.

---
Atsushi Nemoto

Re: gdbserver and "terminated with signal SIGTRAP"

[Atsushi Nemoto]

On Thu, 29 Nov 2007 23:55:02 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> I confirmed that maybe_attach_thread() in thread-db.c did not call
> linux_attach_lwp() on the second time.  That was because
> find_inferior_id() returned non NULL value.  It seems
> find_inferior_id() returned the first thread (which should be dead
> before the second thread creation).

Now I understand why find_inferior_id() misbehaves.  The "id" is
actually pthread_t and it might be recycled.

For example, This program shows same id values!

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>

void *func(void *data)
{
	return data;
}

int main(int argc, char **argv) 
{
        pthread_t id;
	int i;

	for (i = 0; i < 4; i++) {
		pthread_create(&id, NULL, func, NULL);
		printf("%x\n", id);
		pthread_join(id, NULL);
	}
	return 0;
}

$ ./foo
b7fdbb90
b7fdbb90
b7fdbb90
b7fdbb90


But... how can this be avoided?

---
Atsushi Nemoto

Re: gdbserver and "terminated with signal SIGTRAP"

[Daniel Jacobowitz]

On Fri, Nov 30, 2007 at 05:58:14PM +0900, Atsushi Nemoto wrote:
> On Thu, 29 Nov 2007 23:55:02 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> > I confirmed that maybe_attach_thread() in thread-db.c did not call
> > linux_attach_lwp() on the second time.  That was because
> > find_inferior_id() returned non NULL value.  It seems
> > find_inferior_id() returned the first thread (which should be dead
> > before the second thread creation).
> 
> Now I understand why find_inferior_id() misbehaves.  The "id" is
> actually pthread_t and it might be recycled.

Right.

Have you tried HEAD?  The new code I've added since 6.7.1 for thread
creation might help.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdbserver and "terminated with signal SIGTRAP"

[Atsushi Nemoto]

On Fri, 30 Nov 2007 09:47:38 -0500, Daniel Jacobowitz <drow@false.org> wrote:
> Have you tried HEAD?  The new code I've added since 6.7.1 for thread
> creation might help.

Yes.  It seems work but still I cannot see exactly when/how.

http://sourceware.org/ml/gdb-patches/2007-10/msg00547.html ("[commit]
gdbserver support for debugging without libthread_db") is the commit?

---
Atsushi Nemoto

Re: gdbserver and "terminated with signal SIGTRAP"

[Daniel Jacobowitz]

On Sat, Dec 01, 2007 at 12:00:04AM +0900, Atsushi Nemoto wrote:
> On Fri, 30 Nov 2007 09:47:38 -0500, Daniel Jacobowitz <drow@false.org> wrote:
> > Have you tried HEAD?  The new code I've added since 6.7.1 for thread
> > creation might help.
> 
> Yes.  It seems work but still I cannot see exactly when/how.
> 
> http://sourceware.org/ml/gdb-patches/2007-10/msg00547.html ("[commit]
> gdbserver support for debugging without libthread_db") is the commit?

Yes.  We now attach to new threads based on kernel events, and we no
longer track things based on the pthread_t ID.  Those two together
eliminate this race.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdbserver and "terminated with signal SIGTRAP"

[Atsushi Nemoto]

On Fri, 30 Nov 2007 10:13:53 -0500, Daniel Jacobowitz <drow@false.org> wrote:
> > http://sourceware.org/ml/gdb-patches/2007-10/msg00547.html ("[commit]
> > gdbserver support for debugging without libthread_db") is the commit?
> 
> Yes.  We now attach to new threads based on kernel events, and we no
> longer track things based on the pthread_t ID.  Those two together
> eliminate this race.

Thank you.  I'll look at the patch deeper.  Do you think it can be
backported to gdb 6.6?  Could you give me any suggestions?

---
Atsushi Nemoto

Re: gdbserver and "terminated with signal SIGTRAP"

[Daniel Jacobowitz]

On Sat, Dec 01, 2007 at 12:38:11AM +0900, Atsushi Nemoto wrote:
> On Fri, 30 Nov 2007 10:13:53 -0500, Daniel Jacobowitz <drow@false.org> wrote:
> > > http://sourceware.org/ml/gdb-patches/2007-10/msg00547.html ("[commit]
> > > gdbserver support for debugging without libthread_db") is the commit?
> > 
> > Yes.  We now attach to new threads based on kernel events, and we no
> > longer track things based on the pthread_t ID.  Those two together
> > eliminate this race.
> 
> Thank you.  I'll look at the patch deeper.  Do you think it can be
> backported to gdb 6.6?  Could you give me any suggestions?

I'd recommend just using the newer gdbserver wholesale.  It should
work OK.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdbserver and "terminated with signal SIGTRAP"

[Atsushi Nemoto]

On Fri, 30 Nov 2007 10:51:02 -0500, Daniel Jacobowitz <drow@false.org> wrote:
> I'd recommend just using the newer gdbserver wholesale.  It should
> work OK.

Well, I have backported the patch with some other patches to 6.6 and
it works well.  Thank you anyway.

---
Atsushi Nemoto