Linux threads support in GDB

Linux threads support in GDB

[Mark Kettenis]

To all who have reported problems with GDB's threads support on Linux,

Today, I've checked in a rewrite of GDB's threads support for Linux
into the GDB CVS tree.  This should really make debugging
multi-threaded code more robust.  However, to be sure that it really
is more robust, it needs to be tested.  Multi-threaded code is
timing-critical and me not noticing any problems doesn't mean they are
not there.  I welcome everybody to check out the new code and report
any problems.  Please provide a (small) test program and enough
information to reproduce the problem if you do.

To avoid disappointments:

 * You'll need glibc 2.1.3.  Older versions are no longer supported.

 * The new threads stuff is only enabled for Linux/x86.

 * The "zombie problem" isn't fixed yet ...

 * ... which doesn't really matter since the limit of ~30 threads
   isn't fixed either.

 * You'll see quite a bit of debugging output, that tells you (or me)
   what's going on, but interferes with the program output.

See < http://sources.redhat.com/gdb > for information on how the check out
the code.

I'll be looking forward to your reports,

Mark

Re: Linux threads support in GDB

[Eric Paire]

> One application that uses SIGSTOP for suspending a running thread is Sun's J2SE.
> 
Two years ago, we ported the Sun JDK-1.1.6 on native LinuxThreads and
did *NOT* use SIGSTOP(because, if I remember correctly, this schema
was not sufficient to insure correct thread synchronization on
multiprocessors); at the same time, I implemented the very first
version of the LinuxThread support within GDB-4.17 to debug it,
and it worked !

-Eric
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ Eric PAIRE
Web  : http://www.ri.silicomp.com/~paire  | Groupe SILICOMP - Research Institute
Email: eric.paire@ri.silicomp.com         | 2, avenue de Vignate
Phone: +33 (0) 476 63 48 71               | F-38610 Gieres
Fax  : +33 (0) 476 51 05 32               | FRANCE

Re: Linux threads support in GDB

[gdb]

> Explicitly stopping individual threads is considered bad practice by
> most threads experts.  And of course using SIGSTOP is unportable.
> That's why I don't consider this a critical problem.

I can believe that it is considered bad, and right now the only implementation
of thread stopping is used when creating a new thread in certain instances
(Some applications expect threads to be created in a stopped state and use
SIGSTOP to do so. I could implement support for this differently, waiting on a
condition until the resume is given, and if there is a stop/resume pair used
outside of this case then it would break GDB, but at this time it isn't used.)

One application that uses SIGSTOP for suspending a running thread is Sun's J2SE.

> I'd be delighted if you can find a way to make things work.  I just
> don't see how.

If I get enough time to do so, I will think on the problem some more (so far
I've only done brief poking around.)

--Randy

Contrary to popular belief, UNIX is user friendly.
  It just happens to be selective about who it makes friends with.

Re: Linux threads support in GDB

[Mark Kettenis]

   Date: Mon, 02 Oct 2000 17:03:48 -0600 (MDT)
   From: gdb@dontknow.org

   On 02-Oct-00 Mark Kettenis wrote:
   >    Date: Wed, 13 Sep 2000 18:25:21 -0600 (MDT)
   >    From: gdb@dontknow.org
   > 
   > Appologies for the this late reply.  Meanwhile, the particular failing
   > assertions you were seeing should be gone.
   > 
   >    Ok, here is a test case that will reproduce the problem (to some
   >    extent anyways). I guess for some of my testing trying to reproduce
   >    the problem with something similiar I was using GDB 5.0, not the
   >    cvs version. However that version also has problems with the
   >    following code.) Normally when I run the real application I have
   >    SIGSTOP and SIGCONT set to nostop, which may be why it isn't seen
   >    until a breakpoint is used, whereas here it happens after the
   >    implicit SIGCONT breakpoint.
   > 
   > Hmm.  GDB heavily relies on SIGSTOP itself.  There is no other way to
   > stop threads in Linux :-(.  Due to the nature of signals (particularly
   > the fact that multiple signals might be merged into one), this rather
   > hoplessly interferes with the SIGSTOPs you're using in your program.
   > I really don't intend to fix this.  Sorry.

   And my application requires the occassional use of SIGSTOP for
   stopping threads (as you said, there is no other way to do it, at
   least not that I can find.)  However I am not exactly sure what the
   cause of the problem is. Especially considering that a signal is
   sent to a single thread and not all threads (with the possible
   exceptions of the main thread or signals from the shell). This
   behavior is where Linux is different than standard PTHREADS
   (otherwise the use of SIGSTOP would be a bad thing for my
   application.)

Explicitly stopping individual threads is considered bad practice by
most threads experts.  And of course using SIGSTOP is unportable.
That's why I don't consider this a critical problem.

I'd be delighted if you can find a way to make things work.  I just
don't see how.

   Another problem that I am seeing with the threading is that exiting
   threads are ending up defunct rather than going away while using
   GDB (correct behavior is seen w/o GDB). Is SIGCHLD is being handled
   correctly? IIRC that is the cause of that.

It's a kernel bug (actually two in the latest kernels).  I sent Linus
some patches but thus far they didn't make it into the kernel.

Mark

Re: Linux threads support in GDB

[gdb]

On 02-Oct-00 Mark Kettenis wrote:
>    Date: Wed, 13 Sep 2000 18:25:21 -0600 (MDT)
>    From: gdb@dontknow.org
> 
> Appologies for the this late reply.  Meanwhile, the particular failing
> assertions you were seeing should be gone.

I actually tested it, and it gets rid of the assertion, though the application
still exits when the SIGALRM happens while the processes is STOP'ed. Is there
any way to queue up the signals and send them upon program/thread resumption?

I did however realize that part of what I was missing about the problem is
that all threads are stopped during program breakpoints. So I guess that I can
kinda see how signals happening while the program is stopped. Can the itimer be
suspended somehow when trying to stop all threads? (Yes I know, timers are only
one possible cause, and finding all causes would be impossible since user
applications could cause the problem, however itimer might be the most common.
And something special must be happening for SIGUSR1 which pthread uses, but I
haven't looked at too many of the GDB threading internals.)

>    Ok, here is a test case that will reproduce the problem (to some
>    extent anyways). I guess for some of my testing trying to reproduce
>    the problem with something similiar I was using GDB 5.0, not the
>    cvs version. However that version also has problems with the
>    following code.) Normally when I run the real application I have
>    SIGSTOP and SIGCONT set to nostop, which may be why it isn't seen
>    until a breakpoint is used, whereas here it happens after the
>    implicit SIGCONT breakpoint.
> 
> Hmm.  GDB heavily relies on SIGSTOP itself.  There is no other way to
> stop threads in Linux :-(.  Due to the nature of signals (particularly
> the fact that multiple signals might be merged into one), this rather
> hoplessly interferes with the SIGSTOPs you're using in your program.
> I really don't intend to fix this.  Sorry.
> 
> Mark
> 
> PS Please keep CC'ing the GDB mailing list.  These discussions could
> be useful for other people too.



Contrary to popular belief, UNIX is user friendly.
  It just happens to be selective about who it makes friends with.

Re: Linux threads support in GDB

[gdb]

On 02-Oct-00 Mark Kettenis wrote:
>    Date: Wed, 13 Sep 2000 18:25:21 -0600 (MDT)
>    From: gdb@dontknow.org
> 
> Appologies for the this late reply.  Meanwhile, the particular failing
> assertions you were seeing should be gone.
> 
>    Ok, here is a test case that will reproduce the problem (to some
>    extent anyways). I guess for some of my testing trying to reproduce
>    the problem with something similiar I was using GDB 5.0, not the
>    cvs version. However that version also has problems with the
>    following code.) Normally when I run the real application I have
>    SIGSTOP and SIGCONT set to nostop, which may be why it isn't seen
>    until a breakpoint is used, whereas here it happens after the
>    implicit SIGCONT breakpoint.
> 
> Hmm.  GDB heavily relies on SIGSTOP itself.  There is no other way to
> stop threads in Linux :-(.  Due to the nature of signals (particularly
> the fact that multiple signals might be merged into one), this rather
> hoplessly interferes with the SIGSTOPs you're using in your program.
> I really don't intend to fix this.  Sorry.

And my application requires the occassional use of SIGSTOP for stopping threads
(as you said, there is no other way to do it, at least not that I can find.)
However I am not exactly sure what the cause of the problem is. Especially
considering that a signal is sent to a single thread and not all threads (with
the possible exceptions of the main thread or signals from the shell). This
behavior is where Linux is different than standard PTHREADS (otherwise the use
of SIGSTOP would be a bad thing for my application.)

Another problem that I am seeing with the threading is that exiting threads
are ending up defunct rather than going away while using GDB (correct behavior
is seen w/o GDB). Is SIGCHLD is being handled correctly? IIRC that is the cause
of that.

> Mark
> 
> PS Please keep CC'ing the GDB mailing list.  These discussions could
> be useful for other people too.

Ok, I'll try to remember. I'm subscribed to the gdb-bug list but not the gdb
list.


Contrary to popular belief, UNIX is user friendly.
  It just happens to be selective about who it makes friends with.

Re: Linux threads support in GDB

[Mark Kettenis]

   Date: Wed, 13 Sep 2000 18:25:21 -0600 (MDT)
   From: gdb@dontknow.org

Appologies for the this late reply.  Meanwhile, the particular failing
assertions you were seeing should be gone.

   Ok, here is a test case that will reproduce the problem (to some
   extent anyways). I guess for some of my testing trying to reproduce
   the problem with something similiar I was using GDB 5.0, not the
   cvs version. However that version also has problems with the
   following code.) Normally when I run the real application I have
   SIGSTOP and SIGCONT set to nostop, which may be why it isn't seen
   until a breakpoint is used, whereas here it happens after the
   implicit SIGCONT breakpoint.

Hmm.  GDB heavily relies on SIGSTOP itself.  There is no other way to
stop threads in Linux :-(.  Due to the nature of signals (particularly
the fact that multiple signals might be merged into one), this rather
hoplessly interferes with the SIGSTOPs you're using in your program.
I really don't intend to fix this.  Sorry.

Mark

PS Please keep CC'ing the GDB mailing list.  These discussions could
be useful for other people too.