gdb and threads

gdb and threads

[Folkert van Heusden]

Hi,

I'm trying to debug a mulithreaded application running on a multicore cpu.
It is a C++ application which uses the new std::thread (which uses
pthreads underneath).
Now when my program gets a SIGSEGV, I can use 'bt' in gdb to get a backtrace.
Can I be sure that the backtrace I get is the one from the thread
causing the SIGSEGV?


Thanks,

Folkert.
www.vanheusden.com

Re: gdb and threads

[Jan Kratochvil]

On Thu, 12 Apr 2012 11:44:26 +0200, Folkert van Heusden wrote:
> Now when my program gets a SIGSEGV, I can use 'bt' in gdb to get a backtrace.
> Can I be sure that the backtrace I get is the one from the thread
> causing the SIGSEGV?

Assuming GDB default mode, not async/non-stop modes.

With live process yes.  But be aware multiple threads may segfault "at once".

With core files it depends which thread is put first in the core file, with
Linux kernel in practice it works, I do not know guarantees of it in Linux
kernel.


Regards,
Jan

Re: GDB and threads...

[Jiva DeVoe]

Oop! I just tried this on a Mandrake 8.0 box.  And lo and behold, IT
WORKS!  Soooo I'll figure out what the diff is in version, or upgrade
to 8.0!  Thanks again!

On Mon, Apr 30, 2001 at 06:32:06PM -0700, Kevin Buettner wrote:
> On Apr 26,  5:43pm, Jiva DeVoe wrote:
> 
> > Does anyone have any tips on debugging multithreaded apps using gdb?
> > I have been trying for a couple of months now, and IMO gdb is REALLY
> > not very effective at it (at least in my experience.)  I've used
> > windows debuggers to do this, and when I would have a fault in a
> > subthread, the debugger would actually break *in* that thread.  With
> > gdb, it breaks, but it still breaks in the main thread.  I can't even
> > get the thread info commands to work.
> 
> Jiva,
> 
> I constructed a small program with three threads (counting the main
> thread) to see if I could reproduce the problem that you describe.
> Here's the program:
> 
> #include <stdio.h>
> #include <unistd.h>
> #include <pthread.h>
> 
> void *
> worker (void *args)
> {
>   int i;
>   char **a = args;
> 
>   for (i = 0; i < 5; i++)
>     {
>       printf ("%c\n", a[i][0]);
>       sleep (1);
>     }
>   return NULL;
> }
> 
> int
> main(void)
> {
>   pthread_t tid1, tid2;
>   char *a1[] = { "a", "b", "c", "d", "e" };
>   char *a2[] = { "A", "B", "C", 0, "E" };
> 
>   pthread_create (&tid1, NULL, worker, a1);
>   pthread_create (&tid2, NULL, worker, a2);
> 
> 
>   pthread_join (tid1, NULL);
>   pthread_join (tid2, NULL);
>   return 0;
> }
> 
> I compile it (on my Red Hat 7.0 machine) as follows:
> 
>     gcc -Wall -o threads-segv -g threads-segv.c -lpthread
> 
> When I run it (without GDB), it terminates (as intended) with a
> SIGSEGV.  When I debug it with GDB (using current development
> sources), I see the following:
> 
>     (gdb) r
>     Starting program: /home/kev/ctests/threads-segv 
>     [New Thread 1024 (LWP 5462)]
>     [New Thread 2049 (LWP 5463)]
>     Delayed SIGSTOP caught for LWP 5463.
>     [New Thread 1026 (LWP 5464)]
>     Delayed SIGSTOP caught for LWP 5464.
>     [New Thread 2051 (LWP 5465)]
>     Delayed SIGSTOP caught for LWP 5465.
>     a
>     A
>     b
>     B
>     C
>     c
>     d
> 
>     Program received signal SIGSEGV, Segmentation fault.
>     [Switching to Thread 2051 (LWP 5465)]
>     0x08048577 in worker (args=0xbffff968) at threads-segv.c:13
>     warning: Source file is more recent than executable.
> 
>     13            printf ("%c\n", a[i][0]);
>     (gdb) print i
>     $1 = 3
>     (gdb) print a[i]
>     $2 = 0x0
>     (gdb) info thread
>     * 4 Thread 2051 (LWP 5465)  0x08048577 in worker (args=0xbffff968)
> 	at threads-segv.c:13
>       3 Thread 1026 (LWP 5464)  0x400f3241 in __libc_nanosleep ()
>        from /lib/libc.so.6
>       2 Thread 2049 (LWP 5463)  0x40113b97 in __poll (fds=0x804b684, nfds=1, 
> 	timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:63
>       1 Thread 1024 (LWP 5462)  0x40067e75 in __sigsuspend (set=0xbffff7c8)
> 	at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
>     (gdb) bt
>     #0  0x08048577 in worker (args=0xbffff968) at threads-segv.c:13
>     #1  0x4002bc8e in pthread_start_thread_event (arg=0xbf5ffc00) at manager.c:274
>     (gdb) 
> 
> So, with my admittedly simple test case, I've been unable to reproduce
> the problems that you've described.
> 
> Have you tried the current development sources?  If not, see
> 
>     http://sources.redhat.com/gdb/#download
> 
> and read the section about downloading the development version.
> 
> Also, it would be useful if you'd send us your own testcase(s) which
> illustrate the problems that you're seeing.
> 
> Kevin

-- 
Jiva DeVoe
VP Of Software Development
Opnix, Inc. - Simply cool bandwidth.
GPG Fingerprint: 0A17 DF84 516A 1DC4 B837  FE6D 3128 41CD 97CB 4AA7

Re: GDB and threads...

[Jiva DeVoe]

Kevin, thanks for doing this.  I'm sorry I haven't sent you a test
case yet, been REAL busy tracking down bugs in multithreaded apps. ;D

That said...

I used this exact test case here on my machine, using Mandrake 7.2 and
gdb5.0.  I compiled it using the flags you give below, and I had the
same result I had before in my apps.  The threads don't show up in
info threads, and when it segv's it stays in the main thread, and does
not go to the secondary thread.

Here's the output from gdb:

GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i686-pc-linux-gnu"...
(gdb) run
Starting program: /home/jiva/tmp/./bug6 
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIG32, Real-time event 32.
a

Program received signal SIG32, Real-time event 32.
A
b
B
c
C
d

Program received signal SIGSEGV, Segmentation fault.
0x4005517e in sigsuspend () from /lib/libc.so.6
(gdb) bt
#0  0x4005517e in sigsuspend () from /lib/libc.so.6
#1  0x4002b1a0 in pthread_setconcurrency () from /lib/libpthread.so.0
#2  0x40028847 in pthread_join () from /lib/libpthread.so.0
#3  0x80485ce in main () at bug6.c:30
#4  0x4004ecbe in __libc_start_main () from /lib/libc.so.6
(gdb) 

This is on Mandrake 7.2  I have tried the latest CVS *stable* version,
but not the devel version.  I'll give that a try tonight and let you
know.


On Mon, Apr 30, 2001 at 06:32:06PM -0700, Kevin Buettner wrote:
> On Apr 26,  5:43pm, Jiva DeVoe wrote:
> 
> > Does anyone have any tips on debugging multithreaded apps using gdb?
> > I have been trying for a couple of months now, and IMO gdb is REALLY
> > not very effective at it (at least in my experience.)  I've used
> > windows debuggers to do this, and when I would have a fault in a
> > subthread, the debugger would actually break *in* that thread.  With
> > gdb, it breaks, but it still breaks in the main thread.  I can't even
> > get the thread info commands to work.
> 
> Jiva,
> 
> I constructed a small program with three threads (counting the main
> thread) to see if I could reproduce the problem that you describe.
> Here's the program:
> 
> #include <stdio.h>
> #include <unistd.h>
> #include <pthread.h>
> 
> void *
> worker (void *args)
> {
>   int i;
>   char **a = args;
> 
>   for (i = 0; i < 5; i++)
>     {
>       printf ("%c\n", a[i][0]);
>       sleep (1);
>     }
>   return NULL;
> }
> 
> int
> main(void)
> {
>   pthread_t tid1, tid2;
>   char *a1[] = { "a", "b", "c", "d", "e" };
>   char *a2[] = { "A", "B", "C", 0, "E" };
> 
>   pthread_create (&tid1, NULL, worker, a1);
>   pthread_create (&tid2, NULL, worker, a2);
> 
> 
>   pthread_join (tid1, NULL);
>   pthread_join (tid2, NULL);
>   return 0;
> }
> 
> I compile it (on my Red Hat 7.0 machine) as follows:
> 
>     gcc -Wall -o threads-segv -g threads-segv.c -lpthread
> 
> When I run it (without GDB), it terminates (as intended) with a
> SIGSEGV.  When I debug it with GDB (using current development
> sources), I see the following:
> 
>     (gdb) r
>     Starting program: /home/kev/ctests/threads-segv 
>     [New Thread 1024 (LWP 5462)]
>     [New Thread 2049 (LWP 5463)]
>     Delayed SIGSTOP caught for LWP 5463.
>     [New Thread 1026 (LWP 5464)]
>     Delayed SIGSTOP caught for LWP 5464.
>     [New Thread 2051 (LWP 5465)]
>     Delayed SIGSTOP caught for LWP 5465.
>     a
>     A
>     b
>     B
>     C
>     c
>     d
> 
>     Program received signal SIGSEGV, Segmentation fault.
>     [Switching to Thread 2051 (LWP 5465)]
>     0x08048577 in worker (args=0xbffff968) at threads-segv.c:13
>     warning: Source file is more recent than executable.
> 
>     13            printf ("%c\n", a[i][0]);
>     (gdb) print i
>     $1 = 3
>     (gdb) print a[i]
>     $2 = 0x0
>     (gdb) info thread
>     * 4 Thread 2051 (LWP 5465)  0x08048577 in worker (args=0xbffff968)
> 	at threads-segv.c:13
>       3 Thread 1026 (LWP 5464)  0x400f3241 in __libc_nanosleep ()
>        from /lib/libc.so.6
>       2 Thread 2049 (LWP 5463)  0x40113b97 in __poll (fds=0x804b684, nfds=1, 
> 	timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:63
>       1 Thread 1024 (LWP 5462)  0x40067e75 in __sigsuspend (set=0xbffff7c8)
> 	at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
>     (gdb) bt
>     #0  0x08048577 in worker (args=0xbffff968) at threads-segv.c:13
>     #1  0x4002bc8e in pthread_start_thread_event (arg=0xbf5ffc00) at manager.c:274
>     (gdb) 
> 
> So, with my admittedly simple test case, I've been unable to reproduce
> the problems that you've described.
> 
> Have you tried the current development sources?  If not, see
> 
>     http://sources.redhat.com/gdb/#download
> 
> and read the section about downloading the development version.
> 
> Also, it would be useful if you'd send us your own testcase(s) which
> illustrate the problems that you're seeing.
> 
> Kevin

-- 
Jiva DeVoe
VP Of Software Development
Opnix, Inc. - Simply cruisin bandwidth.
GPG Fingerprint: 0A17 DF84 516A 1DC4 B837  FE6D 3128 41CD 97CB 4AA7

Re: GDB and threads...

[Kevin Buettner]

On Apr 26,  5:43pm, Jiva DeVoe wrote:

> Does anyone have any tips on debugging multithreaded apps using gdb?
> I have been trying for a couple of months now, and IMO gdb is REALLY
> not very effective at it (at least in my experience.)  I've used
> windows debuggers to do this, and when I would have a fault in a
> subthread, the debugger would actually break *in* that thread.  With
> gdb, it breaks, but it still breaks in the main thread.  I can't even
> get the thread info commands to work.

Jiva,

I constructed a small program with three threads (counting the main
thread) to see if I could reproduce the problem that you describe.
Here's the program:

--- threads-segv.c ---
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

void *
worker (void *args)
{
  int i;
  char **a = args;

  for (i = 0; i < 5; i++)
    {
      printf ("%c\n", a[i][0]);
      sleep (1);
    }
  return NULL;
}

int
main(void)
{
  pthread_t tid1, tid2;
  char *a1[] = { "a", "b", "c", "d", "e" };
  char *a2[] = { "A", "B", "C", 0, "E" };

  pthread_create (&tid1, NULL, worker, a1);
  pthread_create (&tid2, NULL, worker, a2);


  pthread_join (tid1, NULL);
  pthread_join (tid2, NULL);
  return 0;
}
--- end threads-segv.c ---

I compile it (on my Red Hat 7.0 machine) as follows:

    gcc -Wall -o threads-segv -g threads-segv.c -lpthread

When I run it (without GDB), it terminates (as intended) with a
SIGSEGV.  When I debug it with GDB (using current development
sources), I see the following:

    (gdb) r
    Starting program: /home/kev/ctests/threads-segv 
    [New Thread 1024 (LWP 5462)]
    [New Thread 2049 (LWP 5463)]
    Delayed SIGSTOP caught for LWP 5463.
    [New Thread 1026 (LWP 5464)]
    Delayed SIGSTOP caught for LWP 5464.
    [New Thread 2051 (LWP 5465)]
    Delayed SIGSTOP caught for LWP 5465.
    a
    A
    b
    B
    C
    c
    d

    Program received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 2051 (LWP 5465)]
    0x08048577 in worker (args=0xbffff968) at threads-segv.c:13
    warning: Source file is more recent than executable.

    13            printf ("%c\n", a[i][0]);
    (gdb) print i
    $1 = 3
    (gdb) print a[i]
    $2 = 0x0
    (gdb) info thread
    * 4 Thread 2051 (LWP 5465)  0x08048577 in worker (args=0xbffff968)
	at threads-segv.c:13
      3 Thread 1026 (LWP 5464)  0x400f3241 in __libc_nanosleep ()
       from /lib/libc.so.6
      2 Thread 2049 (LWP 5463)  0x40113b97 in __poll (fds=0x804b684, nfds=1, 
	timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:63
      1 Thread 1024 (LWP 5462)  0x40067e75 in __sigsuspend (set=0xbffff7c8)
	at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
    (gdb) bt
    #0  0x08048577 in worker (args=0xbffff968) at threads-segv.c:13
    #1  0x4002bc8e in pthread_start_thread_event (arg=0xbf5ffc00) at manager.c:274
    (gdb) 

So, with my admittedly simple test case, I've been unable to reproduce
the problems that you've described.

Have you tried the current development sources?  If not, see

    http://sources.redhat.com/gdb/#download

and read the section about downloading the development version.

Also, it would be useful if you'd send us your own testcase(s) which
illustrate the problems that you're seeing.

Kevin

Re: GDB and threads...

[Jeffrey W. Baker]

On Thu, 26 Apr 2001, Jiva DeVoe wrote:

> Does anyone have any tips on debugging multithreaded apps using gdb?
> I have been trying for a couple of months now, and IMO gdb is REALLY
> not very effective at it (at least in my experience.)  I've used
> windows debuggers to do this, and when I would have a fault in a
> subthread, the debugger would actually break *in* that thread.  With
> gdb, it breaks, but it still breaks in the main thread.  I can't even
> get the thread info commands to work.
>
> HELP!
>
> FYI, I am using version 5.0 of gdb on Mandrake 7.2 with kernel 2.2.x

I think you'd better get a newer version of gdb.  I was flustered so I
downloaded a CVS snapshot.  Now it works fine with my threaded programs as
long as I don't spawn a lot of threads.

Linux 2.4.3-ac<mumble> and glibc 2.2.2 here.

-jwb

GDB and threads...

[Jiva DeVoe]

Does anyone have any tips on debugging multithreaded apps using gdb?
I have been trying for a couple of months now, and IMO gdb is REALLY
not very effective at it (at least in my experience.)  I've used
windows debuggers to do this, and when I would have a fault in a
subthread, the debugger would actually break *in* that thread.  With
gdb, it breaks, but it still breaks in the main thread.  I can't even
get the thread info commands to work.

HELP!

FYI, I am using version 5.0 of gdb on Mandrake 7.2 with kernel 2.2.x

Alternatively, when is gdb going to have better thread support?

-- 
Jiva DeVoe
VP Of Software Development
Opnix, Inc. - Simply funkadelic bandwidth.
GPG Fingerprint: 0A17 DF84 516A 1DC4 B837  FE6D 3128 41CD 97CB 4AA7