[Bug threads/31579] New: Cannot find user-level thread for LWP

[Bug threads/31579] New: Cannot find user-level thread for LWP

[ssbssa at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

            Bug ID: 31579
           Summary: Cannot find user-level thread for LWP
           Product: gdb
           Version: 14.1
            Status: NEW
          Severity: normal
          Priority: P2
         Component: threads
          Assignee: unassigned at sourceware dot org
          Reporter: ssbssa at sourceware dot org
  Target Milestone: ---

With gdb-14.2 I get the following error:
```
Attaching to process 14793
[New LWP 14799]
[New LWP 14800]
[New LWP 14804]
[New LWP 14805]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x0000003ee780b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) c
Continuing.
Cannot find user-level thread for LWP 14805: generic error
```

The same program runs without problem with gdb-13:
```
Attaching to process 16559
[New LWP 16561]
[New LWP 16562]
[New LWP 16563]
[New LWP 16564]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x0000003ee780b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) c
Continuing.
[Detaching after fork from child process 16699]
[Detaching after fork from child process 16701]
[Detaching after fork from child process 16705]
[Detaching after fork from child process 16709]
[Detaching after fork from child process 16711]
[Detaching after fork from child process 16715]
```

I bisected it to this commit:
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=6d30ada87b7a515a0f623687e2faadc1d4acf440
In particular, it's the proc-service.c hunk, and I got it to work again by
partly reverting the commit like this:
```
diff --git a/gdb/inferior.h b/gdb/inferior.h
index 712f9b3..0753c34 100644
--- a/gdb/inferior.h
+++ b/gdb/inferior.h
@@ -771,12 +771,12 @@ class scoped_restore_current_inferior_for_memory
   /* Save the current globals and switch to the given inferior and the
      inferior's program space.  inferior_ptid is set to point to the
      inferior's process id (and not to any particular thread).  */
-  explicit scoped_restore_current_inferior_for_memory (inferior *inf)
+  explicit scoped_restore_current_inferior_for_memory (inferior *inf, ptid_t
ptid = null_ptid)
     : m_save_ptid (&inferior_ptid)
   {
     set_current_inferior (inf);
     set_current_program_space (inf->pspace);
-    inferior_ptid = ptid_t (inf->pid);
+    inferior_ptid = ptid == null_ptid ? ptid_t (inf->pid) : ptid;
   }

   DISABLE_COPY_AND_ASSIGN (scoped_restore_current_inferior_for_memory);
diff --git a/gdb/proc-service.c b/gdb/proc-service.c
index f735eb0..dcb8e2b 100644
--- a/gdb/proc-service.c
+++ b/gdb/proc-service.c
@@ -72,7 +72,7 @@
 ps_xfer_memory (const struct ps_prochandle *ph, psaddr_t addr,
                gdb_byte *buf, size_t len, int write)
 {
-  scoped_restore_current_inferior_for_memory save_inferior (ph->thread->inf);
+  scoped_restore_current_inferior_for_memory save_inferior (ph->thread->inf,
ph->thread->ptid);

   CORE_ADDR core_addr = ps_addr_to_core_addr (addr);

```

When I set a breakpoint at ps_xfer_memory, it stops there twice for each
"Detaching after fork from child process" message,
and it looks like this:
```
(top-gdb) bt
#0  ps_xfer_memory (ph=0x8f94c18, addr=0x3ee7a1c2a0 <__stack_user>,
buf=0x7ffec1dc5dc0 "", len=8, write=0) at ../../gdb/proc-service.c:74
#1  0x00007f46c7add770 in _td_fetch_value () from /lib64/libthread_db.so.1
#2  0x00007f46c7adad6d in td_ta_map_lwp2thr () from /lib64/libthread_db.so.1
#3  0x00000000008c07c5 in thread_from_lwp (stopped=0x249d420, ptid=...) at
../../gdb/linux-thread-db.c:415
#4  0x00000000008c179b in thread_db_target::wait (this=<optimized out>,
ptid=..., ourstatus=0x7ffec1dc6568, options=...) at
../../gdb/linux-thread-db.c:1421
#5  0x0000000000a449e1 in target_wait (ptid=...,
status=status@entry=0x7ffec1dc6568, options=..., options@entry=...) at
../../gdb/target.c:2613
#6  0x00000000008788d5 in do_target_wait_1 (inf=inf@entry=0x2274460, ptid=...,
status=status@entry=0x7ffec1dc6568, options=options@entry=...) at
../../gdb/infrun.c:3956
#7  0x0000000000889ecf in __lambda6::operator() (inf=0x2274460,
__closure=<synthetic pointer>) at ../../gdb/infrun.c:4015
#8  do_target_wait (options=..., ecs=0x7ffec1dc6540) at ../../gdb/infrun.c:4034
#9  fetch_inferior_event () at ../../gdb/infrun.c:4445
#10 0x0000000000bbdf2d in gdb_wait_for_event (block=block@entry=0) at
../../gdbsupport/event-loop.cc:694
#11 0x0000000000bbe6d7 in gdb_do_one_event (mstimeout=mstimeout@entry=-1) at
../../gdbsupport/event-loop.cc:217
#12 0x00000000008d07ca in start_event_loop () at ../../gdb/main.c:411
#13 captured_command_loop () at ../../gdb/main.c:475
#14 0x00000000008d25e5 in captured_main (data=data@entry=0x7ffec1dc6690) at
../../gdb/main.c:1348
#15 gdb_main (args=args@entry=0x7ffec1dc66b0) at ../../gdb/main.c:1367
#16 0x0000000000621e35 in main (argc=6, argv=0x7ffec1dc67c8) at
../../gdb/gdb.c:39
(top-gdb) p ph->thread->inf->pid
$1 = 20298
(top-gdb) p ph->thread->ptid
$2 = {m_pid = 20298, m_lwp = 20304, m_tid = 0}
```

I can't share the executable, but can try to make a reproducer if someone has a
suggestion what it should look like.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[ssbssa at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

Hannes Domani <ssbssa at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pedro at palves dot net,
                   |                            |ssbssa at sourceware dot org
               Host|                            |x86_64-redhat-linux

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[ssbssa at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

--- Comment #1 from Hannes Domani <ssbssa at sourceware dot org> ---
I figured out a simple reproducer, just forking outside the main thread:
```
#include <pthread.h>
#include <unistd.h>

void *thread(void *arg)
{
  fork();
  return arg;
}

int main( void )
{
  pthread_t t;
  pthread_create(&t, NULL, &thread, NULL);
  pthread_join(t, NULL);
  return( 0 );
}
```

Then I always get this error:
```
$ gcc -g -o forking forking.c -Wall -Wextra -pthread
$ gdb -q forking
Reading symbols from forking...
(gdb) r
Starting program: /tmp/forking
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7fd4700 (LWP 30716)]
Cannot find user-level thread for LWP 30716: generic error
(gdb)
```

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[pedro at palves dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

Pedro Alves <pedro at palves dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at sourceware dot org   |pedro at palves dot net
             Status|NEW                         |ASSIGNED

--- Comment #2 from Pedro Alves <pedro at palves dot net> ---
Patch posted:

 [PATCH] gdb/linux-nat: Fix mem access ptrace fallback (PR threads/31579)

https://inbox.sourceware.org/gdb-patches/20240411113941.204268-1-pedro@palves.net/T/#u

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[pedro at palves dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

--- Comment #3 from Pedro Alves <pedro at palves dot net> ---
BTW, for completeness, you don't need fork to trigger this.  All you need is to
set a breakpoint that is hit by any thread other than the leader.  E.g., using
the same test program:

(gdb) b thread
Breakpoint 1 at 0x11b9: file forking.c, line 6.
(gdb) r
Starting program: /tmp/forking
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7bff640 (LWP 3266257)]
Cannot find user-level thread for LWP 3266257: generic error

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

--- Comment #4 from Sourceware Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Pedro Alves <palves@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=5e86aab85118ae705bbf9b247497167a8d801d2c

commit 5e86aab85118ae705bbf9b247497167a8d801d2c
Author: Pedro Alves <pedro@palves.net>
Date:   Wed Apr 10 20:00:26 2024 +0100

    gdb/linux-nat: Fix mem access ptrace fallback (PR threads/31579)

    Old RHEL systems have a kernel that does not support writing memory
    via /proc/pid/mem.  On such systems, we fallback to accessing memory
    via ptrace.  That has a few downsides described in the "Accessing
    inferior memory" section at the top of linux-nat.c.

    The target_xfer interface for memory access uses inferior_ptid as
    sideband argument to indicate which process to access.  Memory access
    is process-wide, it is not thread-specific, so inferior_ptid is
    sometimes pointed at a process-wide ptid_t for the memory access
    (i.e., a ptid that looks like {pid, 0, 0}).  That is the case for any
    code that uses scoped_restore_current_inferior_for_memory, for
    example.

    That is what causes the issue described in PR 31579, where thread_db
    calls into the debugger to read memory, which reaches our
    ps_xfer_memory function, which does:

      static ps_err_e
      ps_xfer_memory (const struct ps_prochandle *ph, psaddr_t addr,
                      gdb_byte *buf, size_t len, int write)
      {
        scoped_restore_current_inferior_for_memory save_inferior
(ph->thread->inf);

        ...
          ret = target_read_memory (core_addr, buf, len);
        ...
      }

    If linux_nat_target::xfer_partial falls back to inf_ptrace_target with
    a pid-ptid, then the ptrace code will do the ptrace call targeting
    pid, the leader LWP.  That may fail with ESRCH if the leader is
    currently running, or zombie.  That is the case in the scenario in
    question, because thread_db is consulted for an event of a non-leader
    thread, before we've stopped the whole process.

    Fix this by having the ptrace fallback code try to find a stopped LWP
    to use with ptrace.

    I chose to handle this in the linux-nat target instead of in common
    code because (global) memory is a process-wide property, and this
    avoids having to teach all the code paths that use
    scoped_restore_current_inferior_for_memory to find some stopped thread
    to access memory through, which is a ptrace quirk.  That is
    effectively what we used to do before we started relying on writable
    /proc/pid/mem.  I'd rather not go back there.

    To trigger this on modern kernels you have to hack linux-nat.c to
    force the ptrace fallback code, like so:

     --- a/gdb/linux-nat.c
     +++ b/gdb/linux-nat.c
     @@ -3921,7 +3921,7 @@ linux_nat_target::xfer_partial (enum target_object
object,
              poke would incorrectly write memory to the post-exec address
              space, while the core was trying to write to the pre-exec
              address space.  */
     -      if (proc_mem_file_is_writable ())
     +      if (0 && proc_mem_file_is_writable ())

    With that hack, I was able to confirm that the fix fixes hundreds of
    testsuite failures.  Compared to a test run with pristine master, the
    hack above + this commit's fix shows that some non-stop-related tests
    fail, but that is expected, because those are tests that need to
    access memory while the program is running.  (I made no effort to
    temporarily pause an lwp if no ptrace-stopped lwp is found.)

    Change-Id: I24a4f558e248aff7bc7c514a88c698f379f23180
    Tested-By: Hannes Domani <ssbssa@yahoo.de>
    Approved-By: Andrew Burgess <aburgess@redhat.com>

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[pedro at palves dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

Pedro Alves <pedro at palves dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |15.1

--- Comment #5 from Pedro Alves <pedro at palves dot net> ---
Fixed on master.  (Somehow I had thought that the 15 branch had already been
cut.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug threads/31579] Cannot find user-level thread for LWP

[pedro at palves dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=31579

Pedro Alves <pedro at palves dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #6 from Pedro Alves <pedro at palves dot net> ---
Really, it's fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.