Problem debugging multi-threaded application

Problem debugging multi-threaded application

[Arvid Brodin]

We have a program, a big beast, which on some platforms cannot be debugged, and I'm trying to figure out why.

The program is supposed to print out some welcome/init messages and then display a prompt. It also starts a bunch of threads (about 200 of them). When run under gdb on some platforms, I only get the [New Thread 0x... (LWP ...)] messages, but no text output. The program seems to run - only I get no text output (or input). 

If I invoke gdb under strace, i.e. 'strace -f -o strace-out.txt gdb <my-program>', and then run my program in gdb, I instead get no messages about created threads, but I do get some printouts from my program before it hangs. The strace log among other things contain messages about "--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---" - I don't know if this is important.

If run under gdbtui - 'gdbtui <my-program>' - as soon as I 'run' the program gdbtui exits to the shell with
'[1]+  Stopped       gdbtui <my-program>'.

If I comment out either the call to create_thread_event_breakpoint() in gdb/linux-thread-db.c:468 or the call to td_thr_event_enable() in gdb/linux-thread-db.c:716 I get the printouts from my program (although I guess it also messes something up - the program hangs shortly after initialization in this case).

The problem exists with both gdb-6.3.0 and gdb-6.7.1.

The problem exists on these platforms: 

---------------
$ uname -r
2.4.21-53.ELsmp

$ /lib/tls/libc.so.6 
GNU C Library stable release version 2.3.2, by Roland McGrath et al.
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.2.3 20030502 (Red Hat Linux 3.2.3-58).
Compiled on a Linux 2.4.20 system on 2007-04-19.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        NPTL 0.60 by Ulrich Drepper
        RT using linux kernel aio
        The C stubs add-on version 2.1.2.
        BIND-8.2.3-T5B
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Glibc-2.0 compatibility add-on by Cristian Gafton 
Thread-local storage support included.
---------------

and

---------------
$~> uname -r
2.6.5-7.287.3-smp

$~> /lib/tls/libc.so.6 
GNU C Library stable release version 2.3.5 (20050720), by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 3.3.3 (SuSE Linux).
Compiled on a Linux 2.6.5 system on 2007-10-10.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        Native POSIX Threads Library by Ulrich Drepper et al
        GNU Libidn by Simon Josefsson
        NoVersion patch for broken glibc 2.0 binaries
        BIND-8.2.3-T5B
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
---------------

but not on

---------------
$~> uname -r
2.6.16.46-0.14-smp

$~> /lib/libc.so.6 
GNU C Library development release version 2.4 (20070506), by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 4.1.0 (SUSE Linux).
Compiled on a Linux 2.6.16 system on 2007-05-06.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        NoVersion patch for broken glibc 2.0 binaries
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
Thread-local storage support included.
---------------

The paths to glibc above is taken from 'ldd <path to libthread_db.so.1 as printed by gdb>, i.e. on the systems that fails /lib/tls/libc.so.1 and /lib/tls/libthread_db.so.1 are used, while on the working system the files are found under /lib.

To recapitulate, the program runs fine on all these platforms when not being debugged, and works under gdb on the "newest" one. On the older platforms the symptoms are text input and output does not work when the program is being debugged, unless gdb is run under strace, when it works "better", at least.

I have no idea if this is a bug in gdb, if we do something bad in our program, if it is a bug in nptl, or ... 

Any ideas?

--
Arvid Brodin
Enea LCC

Re: Problem debugging multi-threaded application

[Daniel Jacobowitz]

On Mon, Jan 21, 2008 at 07:26:37PM +0100, Arvid Brodin wrote:
> If I invoke gdb under strace, i.e. 'strace -f -o strace-out.txt gdb
> <my-program>'

Don't do that :-)  strace and gdb both use the ptrace interface; if
you tell strace to follow children with -f, then it will win and
gdb will not be able to debug the program it started.

> The problem exists on these platforms: 

Since the problem exists on older platforms and not on newer ones, it
may be a bug in the Linux kernel or glibc which has since been fixed.
GDB is very sensitive to such bugs.

> I have no idea if this is a bug in gdb, if we do something bad in
> our program, if it is a bug in nptl, or ...

It sounds sort of like a problem with job control and terminal groups,
but I'm not sure I have enough to guess... can you run it and use the
gdb "attach" command?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Problem debugging multi-threaded application

[Arvid Brodin]

On 2008-01-21 19:35, Daniel Jacobowitz wrote:
> On Mon, Jan 21, 2008 at 07:26:37PM +0100, Arvid Brodin wrote:
>> If I invoke gdb under strace, i.e. 'strace -f -o strace-out.txt gdb
>> <my-program>'
> 
> Don't do that :-)  strace and gdb both use the ptrace interface; if
> you tell strace to follow children with -f, then it will win and
> gdb will not be able to debug the program it started.

That explains a lot. Thanks for the info!


>> I have no idea if this is a bug in gdb, if we do something bad in
>> our program, if it is a bug in nptl, or ...
> 
> It sounds sort of like a problem with job control and terminal groups,
> but I'm not sure I have enough to guess... can you run it and use the
> gdb "attach" command?

Actually, that was one of the first things I tried, and yes, that works fine. Sorry for not mentioning it before. I also found this old message regarding a similar symptom and foregrounding/backgrounding and SIGTTOU, which suggests commenting out the [New Thread ...] message. I tried that but it made no difference. I suspect that particular bug has been long fixed:

http://sourceware.org/ml/gdb/2004-03/msg00251.html

--
Arvid Brodin
Enea LCC

Re: Problem debugging multi-threaded application

[Arvid Brodin]

After further testing, this seems to be a thread locking problem within our program. Our program works - or not - depending on how busy the machine is. The only reason it turned up only during debugging was because the debugging of course required more cpu cycles. Sorry for the noise.

-- 
Arvid Brodin
Enea LCC