[Bug gdb/12702] New: gdb can hang waiting for thread group leader

[Bug gdb/12702] New: gdb can hang waiting for thread group leader

[dje at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

           Summary: gdb can hang waiting for thread group leader
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: gdb
        AssignedTo: unassigned@sourceware.org
        ReportedBy: dje@google.com


Created attachment 5685
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5685
Patch to workaround the issue.

When ptracing, waitpid with options == 0 will hang if the thread group leader
exits while there are still other threads around.
[Blech!  IWBN if it could at least return an error instead of hanging.]

When gdb detects a thread has stopped, it will stop other threads before
returning control to the user (in "all-stop mode").
A race occurs when the main thread exits between gdb detecting a 2nd thread has
stopped and gdb waits for the main thread to stop.
If the main thread has exited and there are other threads then waitpid
(main_thread_pid, &status, 0) will hang.

This patch to gdb and the accompanying testcase illustrates the issue.
Apply to cvs head as of 11apr25.

diff -u -p -r1.199 linux-nat.c
--- linux-nat.c 9 Mar 2011 12:48:55 -0000       1.199
+++ linux-nat.c 25 Apr 2011 23:48:13 -0000
@@ -4047,6 +4108,21 @@ linux_thread_alive (ptid_t ptid)
                        target_pid_to_str (ptid),
                        err ? safe_strerror (tmp_errno) : "OK");

+  if (debug_linux_nat && GET_PID (ptid) != GET_LWP (ptid))
+    {
+      char buf[200];
+      sprintf (buf, "cat /proc/%ld/task/%ld/status",
+              (long) GET_PID (ptid), (long) GET_LWP (ptid));
+      system (buf);
+      sleep (3);
+      err = kill_lwp (GET_LWP (ptid), 0);
+      fprintf_unfiltered (gdb_stdlog,
+                         "LLTA: KILL(SIG0) %s (%s)\n",
+                         target_pid_to_str (ptid),
+                         err ? safe_strerror (err) : "OK");
+      system (buf);
+    }
+
   if (err != 0)
     return 0;


bash$ cat testcase.c

#include <stdio.h>
#include <pthread.h>
#include <syscall.h>
#include <stdarg.h>
#include <stdlib.h>

int
gettid ()
{
 return syscall (__NR_gettid);
}

void
printf_flushed (const char *msg, ...)
{
 va_list args;
 va_start (args, msg);
 vprintf (msg, args);
 va_end (args);
 fflush (stdout);
}

void*
thread_function (void* dummy_ptr)
{
 printf_flushed ("Thread self 0x%x, pid %d, lwp %d\n",
                 pthread_self (), getpid (), gettid ());
 asm volatile ("int3");
 pthread_exit ((void *) 0);
 abort ();
}

int
main ()
{
 pthread_t thread_id;
 pthread_create (&thread_id, NULL, thread_function, NULL);
 sleep (1);
 return 0;
}
bash$ gcc -g testcase.c -o testcase.x64 -lpthread
bash$ ./gdb --batch -nx -ex "set debug infrun 1" -ex "set debug lin-lwp 1" -ex
run -ex quit ./testcase.x64
[...]
LLW: waitpid 15050 received Trace/breakpoint trap (stopped)
LLTA: KILL(SIG0) Thread 0x7ffff783c700 (LWP 15050) (OK)
Name:   testcase.x64
State:  T (tracing stop)
Tgid:   15047
Pid:    15050
PPid:   15045
TracerPid:      15045
[...]
State:  Z (zombie)
Tgid:   15047
Pid:    15050
PPid:   15045
TracerPid:      15045
[...]
LLW: Candidate event Trace/breakpoint trap (stopped) in Thread 0x7ffff783c700
(LWP 15050).
SC:  kill Thread 0x7ffff7fd7700 (LWP 15047) **<SIGSTOP>**
SC:  lwp kill 0 ERRNO-OK
hang

The attached patch works around the bug^wissue.

I suspect all calls to waitpid with options == 0 need to be audited and either
fixed or documented that they can't trip over this issue.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[jan.kratochvil at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

Jan Kratochvil <jan.kratochvil at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jan.kratochvil at redhat
                   |                            |dot com

--- Comment #1 from Jan Kratochvil <jan.kratochvil at redhat dot com> 2011-04-26 17:57:27 UTC ---
Without comparing it with this fix Fedora carries this one:
http://pkgs.fedoraproject.org/gitweb/?p=gdb.git;a=blob_plain;f=gdb-6.6-bz247354-leader-exit-fix.patch;hb=f15/master
http://pkgs.fedoraproject.org/gitweb/?p=gdb.git;a=blob_plain;f=gdb-6.6-bz247354-leader-exit-test.patch;hb=f15/master

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[dje at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

--- Comment #2 from dje at google dot com 2011-04-27 01:09:26 UTC ---
Actually, options == 0 is a red herring.
!WNOHANG seems to be the only requirement as far as options goes.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[dje at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

--- Comment #3 from dje at google dot com 2011-05-02 17:01:31 UTC ---
This is similar to, if not identical to 10970.
If one wants to close it as a dup, feel free to,
PROVIDED, one documents in 10970, e.g. with a link
to this one, that the bug can also happen when
the main program exits, not just pthread_exits.

It's seems highly unfortunate that a waitpid call in a ptracer has to hang in
this way.
[Heh, I wonder if setting up an alarm would work well enough, assuming the
alarm can unhang the waitpid call.]

I gather that no SIGCHLD will get generated, so gdbserver can suffer from this
too: it does a WNOHANG waitpid call followed by a sigsuspend, though maybe it
avoids the issue via some other means (quite likely accidentally though).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[dje at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

--- Comment #4 from dje at google dot com 2011-05-17 15:55:46 UTC ---
*** Bug 10095 has been marked as a duplicate of this bug. ***

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[jan.kratochvil at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

--- Comment #5 from Jan Kratochvil <jan.kratochvil at redhat dot com> 2011-05-17 16:06:04 UTC ---
Comment 5 updated links:
http://pkgs.fedoraproject.org/gitweb/?p=gdb.git;a=blob_plain;f=gdb-6.6-bz247354-leader-exit-fix.patch;hb=master
http://pkgs.fedoraproject.org/gitweb/?p=gdb.git;a=blob_plain;f=gdb-6.6-bz247354-leader-exit-test.patch;hb=master

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[cvs-commit at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

--- Comment #6 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> 2011-05-27 16:55:43 UTC ---
CVSROOT:    /cvs/src
Module name:    src
Changes by:    jkratoch@sourceware.org    2011-05-27 16:55:39

Modified files:
    gdb            : ChangeLog linux-nat.c 
    gdb/testsuite  : ChangeLog 
Added files:
    gdb/testsuite/gdb.threads: leader-exit.c leader-exit.exp 

Log message:
    gdb/
    Fix PR 10970, PR 12702.
    * linux-nat.c (linux_lwp_is_zombie): New function.
    (wait_lwp): Initialize status.  New variable prev_mask.  Block signals.
    Check for linux_lwp_is_zombie.  Use WNOHANG and sigsuspend.

    gdb/testsuite/
    * gdb.threads/leader-exit.c: New file.
    * gdb.threads/leader-exit.exp: New file.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/ChangeLog.diff?cvsroot=src&r1=1.13060&r2=1.13061
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/linux-nat.c.diff?cvsroot=src&r1=1.205&r2=1.206
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.2728&r2=1.2729
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.threads/leader-exit.c.diff?cvsroot=src&r1=NONE&r2=1.1
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.threads/leader-exit.exp.diff?cvsroot=src&r1=NONE&r2=1.1

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[jan.kratochvil at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

Jan Kratochvil <jan.kratochvil at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #7 from Jan Kratochvil <jan.kratochvil at redhat dot com> 2011-05-27 17:03:28 UTC ---
Doug, I do not have the original problem reproducible but I also run on
ptrace-on-utrace which behaves a bit differently.
I believe this Bug can be closed now.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[teawater at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

teawater <teawater at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |ASSIGNED
                 CC|                            |teawater at gmail dot com

--- Comment #8 from teawater <teawater at gmail dot com> ---
Found this issue still affect gdbserver.  Post a patch for it.
http://sourceware.org/ml/gdb-patches/2013-07/msg00475.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[cvs-commit at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12702

--- Comment #9 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "gdb and binutils".

The branch, master has been updated
       via  fa96cb382c12b099675c5cc238aaa7352a3fd3d7 (commit)
       via  d632a0971cf9c8044e2b59da45ba6bbc27714c84 (commit)
       via  2ebd5a35208e40dab7df46c7d6d58f4d614c6e21 (commit)
      from  5063daf7357ccdc2c9e899ca0421c2e8e580c88d (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=fa96cb382c12b099675c5cc238aaa7352a3fd3d7

commit fa96cb382c12b099675c5cc238aaa7352a3fd3d7
Author: Pedro Alves <palves@redhat.com>
Date:   Thu Feb 27 14:30:08 2014 +0000

    Teach GDBserver's Linux backend about no unwaited-for children
(TARGET_WAITKIND_NO_RESUMED).

    GDBserver currently hangs forever in waitpid if the leader thread
    exits before other threads, or if all resumed threads exit - e.g.,
    next over a thread exit with sched-locking on.  This is exposed by
    leader-exit.exp.  leader-exit.exp is part of a series of tests for a
    set of related problems.  See
    <http://www.sourceware.org/ml/gdb-patches/2011-10/msg00704.html>:

     "
     To recap, on the Linux kernel, ptrace/waitpid don't allow reaping the
     leader thread until all other threads in the group are reaped.  When
     the leader exits, it goes zombie, but waitpid will not return an exit
     status until the other threads are gone.  This is presently exercised
     by the gdb.threads/leader-exit.exp test.  The fix for that test, in
     linux-nat.c:wait_lwp, handles the case where we see the leader gone
     when we're stopping all threads to report an event to some other
     thread to the core.

     (...)

     The latter bit about not blocking if there no resumed threads in the
     process also applies to some other thread exiting, not just the main
     thread.  E.g., this test starts a thread, and runs to a breakpoint in
     that thread:

     ...
     (gdb) c
     Continuing.
     [New Thread 0x7ffff75a4700 (LWP 23397)]
     [Switching to Thread 0x7ffff75a4700 (LWP 23397)]

     Breakpoint 2, thread_a (arg=0x0) at
../../../src/gdb/testsuite/gdb.threads/no-unwaited-for-left.c:28
     28        return 0; /* break-here */
     (gdb) info threads
     * 2 Thread 0x7ffff75a4700 (LWP 23397)  thread_a (arg=0x0) at
../../../src/gdb/testsuite/gdb.threads/no-unwaited-for-left.c:28
       1 Thread 0x7ffff7fcb720 (LWP 23391)  0x00007ffff7bc606d in pthread_join
(threadid=140737343276800, thread_return=0x0) at pthread_join.c:89

     The thread will exit as soon as we resume it.  But if we only resume
     that thread, leaving the rest of the threads stopped:

     (gdb) set scheduler-locking on
     (gdb) c
     Continuing.
     ^C^C^C^C^C^C^C^C
     "

    This patch fixes the issues by implementing TARGET_WAITKIND_NO_RESUMED
    on GDBserver, similarly to what the patch above did for native
    Linux GDB.

    gdb.threads/leader-exit.exp now passes.

    gdb.threads/no-unwaited-for-left.exp now at least errors out instead
    of hanging:

     continue
     Continuing.
     warning: Remote failure reply: E.No unwaited-for children left.

     [Thread 15454] #1 stopped.
     0x00000034cf408e60 in pthread_join (threadid=140737353922368,
thread_return=0x0) at pthread_join.c:93
     93          lll_wait_tid (pd->tid);
     (gdb) FAIL: gdb.threads/no-unwaited-for-left.exp: continue stops when the
main thread exits

    The gdb.threads/non-ldr-exc-*.exp tests are skipped because GDBserver
    unfortunately doesn't support fork/exec yet, but I'm confident this
    fixes the related issues.

    I'm leaving modeling TARGET_WAITKIND_NO_RESUMED in the RSP for a
    separate pass.

    (BTW, in case of error in response to a vCont, it would be better for
    GDB to query the target for the current thread, or re-select one,
    instead of assuming current inferior_ptid is still the selected
    thread.)

    This implementation is a little different from GDB's, because I'm
    avoiding bringing in more of this broken use of waitpid(PID) into
    GDBserver.  Specifically, this avoids waitpid(PID) when stopping all
    threads.  There's really no need for wait_for_sigstop to wait for each
    LWP in turn.  Instead, with some refactoring, we make it reuse
    linux_wait_for_event.

    gdb/gdbserver/
    2014-02-27  Pedro Alves  <palves@redhat.com>

        PR 12702
        * inferiors.h (A_I_NEXT, ALL_INFERIORS_TYPE, ALL_PROCESSES): New
        macros.
        * linux-low.c (delete_lwp, handle_extended_wait): Add debug
        output.
        (last_thread_of_process_p): Take a PID argument instead of a
        thread pointer.
        (linux_wait_for_lwp): Delete.
        (num_lwps, check_zombie_leaders, not_stopped_callback): New
        functions.
        (linux_low_filter_event): New function, party factored out from
        linux_wait_for_event.
        (linux_wait_for_event): Rename to ...
        (linux_wait_for_event_filtered): ... this.  Add new filter ptid
        argument.  Partly rewrite.  Always use waitpid(-1, WNOHANG) and
        sigsuspend.  Check for zombie leaders.
        (linux_wait_for_event): Reimplement as wrapper around
        linux_wait_for_event_filtered.
        (linux_wait_1): Handle TARGET_WAITKIND_NO_RESUMED.  Assume that if
        a normal or signal exit is seen, it's the whole process exiting.
        (wait_for_sigstop): No longer a for_each_inferior callback.
        Rewrite on top of linux_wait_for_event_filtered.
        (stop_all_lwps): Call wait_for_sigstop directly.
        * server.c (resume, handle_target_event): Handle
        TARGET_WAITKIND_NO_RESUMED.

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=d632a0971cf9c8044e2b59da45ba6bbc27714c84

commit d632a0971cf9c8044e2b59da45ba6bbc27714c84
Author: Pedro Alves <palves@redhat.com>
Date:   Thu Feb 27 14:30:08 2014 +0000

    Move linux-nat.c:status_to_str to nat/linux-waitpid.c.

    So that gdbserver's Linux backend can use it too.

    gdb/
    2014-02-27  Pedro Alves  <palves@redhat.com>

        PR 12702
        * linux-nat.c (status_to_str): Moved to nat/linux-waitpid.c.
        * nat/linux-waitpid.c: Include string.h.
        (status_to_str): Moved here and made extern.
        * nat/linux-waitpid.h (status_to_str): New declaration.

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=2ebd5a35208e40dab7df46c7d6d58f4d614c6e21

commit 2ebd5a35208e40dab7df46c7d6d58f4d614c6e21
Author: Hui Zhu <hui@codesourcery.com>
Date:   Thu Feb 27 14:30:07 2014 +0000

    Move ptid_match to common/ptid.c.

    So that gdbserver can use it too.

    gdb/
    2014-02-27  Hui Zhu  <hui@codesourcery.com>

        PR 12702
        * infrun.c (ptid_match): Move ...
        * common/ptid.c (ptid_match): ... here.
        * inferior.h (ptid_match): Move ...
        * common/ptid.h (ptid_match): ... here.

-----------------------------------------------------------------------

Summary of changes:
 gdb/ChangeLog             |   16 +
 gdb/common/ptid.c         |   14 +
 gdb/common/ptid.h         |   10 +
 gdb/gdbserver/ChangeLog   |   28 ++
 gdb/gdbserver/inferiors.h |   22 ++
 gdb/gdbserver/linux-low.c |  852 ++++++++++++++++++++++++++-------------------
 gdb/gdbserver/server.c    |   18 +-
 gdb/inferior.h            |   10 -
 gdb/infrun.c              |   14 -
 gdb/linux-nat.c           |   25 --
 gdb/nat/linux-waitpid.c   |   28 ++
 gdb/nat/linux-waitpid.h   |    4 +
 12 files changed, 640 insertions(+), 401 deletions(-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[palves at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12702

Pedro Alves <palves at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |palves at redhat dot com

--- Comment #10 from Pedro Alves <palves at redhat dot com> ---
This is now fixed on GDBserver too.

  https://sourceware.org/ml/gdb-patches/2014-02/msg00826.html

I've left out the original "[PATCH 5/5] Add TARGET_WAITKIND_NO_RESUMED support
to the RSP" at https://sourceware.org/ml/gdb-patches/2014-01/msg00897.html ,
because I worried about races in non-stop mode.  Assume N is the new stop reply
for TARGET_WAITKIND_NO_RESUMED, and say, the traffic goes like this:

(no thread was resumed before #1)

GDB sends two steps in a row.

 #1 -> vCont;s:555
 #2 <- OK

(555 exits)

 #3 <- %Stop:N   (no more resumed threads left)

 #4 -> vCont;s:666
 #5 <- OK

And now GDB goes and processes #3, and mistakenly believes no thread is left
running on the target, while in fact, 666 is left running.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/12702] gdb can hang waiting for thread group leader

[palves at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12702

Pedro Alves <palves at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #11 from Pedro Alves <palves at redhat dot com> ---
The hangs are gone, so marking this closed.  The GDBserver/RSP issue is now
tracked in PR14618.

-- 
You are receiving this mail because:
You are on the CC list for the bug.