[Bug rust/29550] New: Segmentation fault in infrun.c

[Bug rust/29550] New: Segmentation fault in infrun.c

[eigelc at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29550

            Bug ID: 29550
           Summary: Segmentation fault in infrun.c
           Product: gdb
           Version: 12.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: rust
          Assignee: unassigned at sourceware dot org
          Reporter: eigelc at gmail dot com
  Target Milestone: ---

I am debugging a rather complex rust application. gdb crashes.

(gdb) set follow-fork-mode child
(gdb) b src/runtime/fork/init/mod.rs:150
Breakpoint 1 at 0x559ebe2aab9d: file mod.rs, line 150.
(gdb) c
Continuing.
[Attaching after Thread 0x7fdec5453cc0 (LWP 28777) fork to child process 46624]
[New inferior 2 (process 46624)]
[Detaching after fork from parent process 28777]
[Inferior 1 (process 28777) detached]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Attaching after Thread 0x7fdec5453cc0 (LWP 46624) fork to child process 46625]
[New inferior 3 (process 46625)]
[Detaching after fork from parent process 46624]
[Inferior 2 (process 46624) detached]
warning: Target and debugger are in different PID namespaces; thread lists and
other data are likely unreliable.  Connect to gdbserver inside the container.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Attaching after Thread 0x7fdec5453cc0 (LWP 46625) fork to child process 46626]
[New inferior 4 (process 46626)]


Fatal signal: Segmentation fault
----- Backtrace -----
0x562325f90c20 gdb_internal_backtrace_1
        /home/cristianeigel/tmp/gdb-12.1/gdb/bt-utils.c:122
0x562325f90c20 _Z22gdb_internal_backtracev
        /home/cristianeigel/tmp/gdb-12.1/gdb/bt-utils.c:168
0x5623260931f9 handle_fatal_signal
        /home/cristianeigel/tmp/gdb-12.1/gdb/event-top.c:904
0x5623260933b8 handle_sigsegv
        /home/cristianeigel/tmp/gdb-12.1/gdb/event-top.c:977
0x7f53b8ee151f ???
        ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x5623260fb6c7 maybe_set_commit_resumed_all_targets
        /home/cristianeigel/tmp/gdb-12.1/gdb/infrun.c:2785
0x562325ed4d42 _ZN29scoped_disable_commit_resumed5resetEv
        /home/cristianeigel/tmp/gdb-12.1/gdb/infrun.c:2895
0x562325ed4d42 _ZN29scoped_disable_commit_resumedD4Ev
        /home/cristianeigel/tmp/gdb-12.1/gdb/infrun.c:2927
0x562325ed4d42 _Z20fetch_inferior_eventv
        /home/cristianeigel/tmp/gdb-12.1/gdb/infrun.c:4151
0x562326425395 gdb_wait_for_event
        /home/cristianeigel/tmp/gdb-12.1/gdbsupport/event-loop.cc:700
0x562326425879 gdb_wait_for_event
        /home/cristianeigel/tmp/gdb-12.1/gdbsupport/event-loop.cc:596
0x562326425879 _Z16gdb_do_one_eventv
        /home/cristianeigel/tmp/gdb-12.1/gdbsupport/event-loop.cc:212
0x562326152d64 start_event_loop
        /home/cristianeigel/tmp/gdb-12.1/gdb/main.c:421
0x562326152d64 captured_command_loop
        /home/cristianeigel/tmp/gdb-12.1/gdb/main.c:481
0x562326154964 captured_main
        /home/cristianeigel/tmp/gdb-12.1/gdb/main.c:1351
0x562326154964 _Z8gdb_mainP18captured_main_args
        /home/cristianeigel/tmp/gdb-12.1/gdb/main.c:1366
0x562325ee87ff main
        /home/cristianeigel/tmp/gdb-12.1/gdb/gdb.c:32
---------------------
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

This is a bug, please report it.  For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.

[1]    46554 segmentation fault  sudo -E
/home/cristianeigel/.cargo/bin/rust-gdb --pid 28777

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug rust/29550] Segmentation fault in infrun.c

[simon.marchi at polymtl dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=29550

Simon Marchi <simon.marchi at polymtl dot ca> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |simon.marchi at polymtl dot ca

--- Comment #1 from Simon Marchi <simon.marchi at polymtl dot ca> ---
If you can provide a reproducer, I'd be interested to dig into it.

The code is:

  for (inferior *inf : all_non_exited_inferiors ())
    {
      process_stratum_target *proc_target = inf->process_target ();

      if (proc_target->commit_resumed_state)  <-- segfaults here

This means we have an inferior marked as not exited (inf->pid != 0), while
having no process target.  This is an invalid state.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug rust/29550] Segmentation fault in infrun.c

[eigelc at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29550

--- Comment #2 from Cristian Eigel <eigelc at gmail dot com> ---
This is going to be rather complex and you need rust 1.63 to reproduce (the
project is written in rust). This can only be done in Linux. The project I was
debugging is this: https://github.com/esrlabs/northstar. Please download it.

From the quickstart (in the Readme) run these steps:
```
sudo apt-get install build-essential libclang1 squashfs-tools
./examples/build_examples.sh
cargo run --release --bin northstar
```

The last command will ask for an elevation of privilege. The script which is
doing that is .cargo/runner-x86_64-unknown-linux-gnu (in case you want to
inspect what's it trying to do). The script is running because the
.cargo/config has it set as a runner.


In yet another console, run ps -a and get the pid of northstar-fork. Then run
sudo gdb --pid {THAT_PID_ABOVE}
Once in gdb run these commands

```
set follow-fork-mode child
b northstar-runtime/src/runtime/fork/init/mod.rs:150
continue
```

These should be after the fork call in the function run.

Now in another console:
```
cargo run --release --bin northstar-nstar -- start console
```
At these point the gdb would get the seg-fault.

If you want to stop just before the seg-fault, then put the break at line 144
(or before the fork is called) and then next from there.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug rust/29550] Segmentation fault in infrun.c

[simon.marchi at polymtl dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=29550

--- Comment #3 from Simon Marchi <simon.marchi at polymtl dot ca> ---
Thanks, I was able to reproduce with those instructions.

We arrive in this state because an exception is thrown here:

(gdb) bt
#0  0x00007f6f7baa5f91 in __cxxabiv1::__cxa_throw (obj=0x60e0000063a0,
tinfo=0x561aa4ff23e8 <typeinfo for gdb_exception_error>, dest=0x561aa416284c
<gdb_exception_error::~gdb_exception_error()>)
    at /usr/src/debug/gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
#1  0x0000561aa4ecc871 in throw_it(return_reason, errors, const char *, typedef
__va_list_tag __va_list_tag *) (reason=RETURN_ERROR, error=GENERIC_ERROR, 
    fmt=0x561aa0735640 "\"%s\": could not open as an executable file: %s.",
ap=0x7ffff0b73440) at
/home/smarchi/src/binutils-gdb/gdbsupport/common-exceptions.cc:200
#2  0x0000561aa4ecc913 in throw_verror (error=GENERIC_ERROR, fmt=0x561aa0735640
"\"%s\": could not open as an executable file: %s.", ap=0x7ffff0b73440)
    at /home/smarchi/src/binutils-gdb/gdbsupport/common-exceptions.cc:208
#3  0x0000561aa49deeda in verror (string=0x561aa0735640 "\"%s\": could not open
as an executable file: %s.", args=0x7ffff0b73440) at
/home/smarchi/src/binutils-gdb/gdb/utils.c:164
#4  0x0000561aa4ee117c in error (fmt=0x561aa0735640 "\"%s\": could not open as
an executable file: %s.") at
/home/smarchi/src/binutils-gdb/gdbsupport/errors.cc:46
#5  0x0000561aa30a213c in exec_file_attach (filename=0x6060000367a0
"target:/home/smarchi/src/northstar/target/release/northstar", from_tty=0) at
/home/smarchi/src/binutils-gdb/gdb/exec.c:457
#6  0x0000561aa3bc99f8 in clone_program_space (dest=0x613000008540,
src=0x613000004640) at /home/smarchi/src/binutils-gdb/gdb/progspace.c:210
#7  0x0000561aa345bf83 in follow_fork_inferior (follow_child=true,
detach_fork=true) at /home/smarchi/src/binutils-gdb/gdb/infrun.c:578
#8  0x0000561aa345eb42 in follow_fork () at
/home/smarchi/src/binutils-gdb/gdb/infrun.c:794
#9  0x0000561aa3493342 in handle_inferior_event (ecs=0x7ffff0b744c0) at
/home/smarchi/src/binutils-gdb/gdb/infrun.c:5728
#10 0x0000561aa3480cae in fetch_inferior_event () at
/home/smarchi/src/binutils-gdb/gdb/infrun.c:4233
#11 0x0000561aa33c96d4 in inferior_event_handler (event_type=INF_REG_EVENT) at
/home/smarchi/src/binutils-gdb/gdb/inf-loop.c:41
#12 0x0000561aa36a2c1e in handle_target_event (error=0, client_data=0x0) at
/home/smarchi/src/binutils-gdb/gdb/linux-nat.c:4216
#13 0x0000561aa4ee32fe in handle_file_event (file_ptr=0x60700001a650,
ready_mask=1) at /home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:574
#14 0x0000561aa4ee3c3a in gdb_wait_for_event (block=0) at
/home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:695
#15 0x0000561aa4ee16e8 in gdb_do_one_event (mstimeout=-1) at
/home/smarchi/src/binutils-gdb/gdbsupport/event-loop.cc:217
#16 0x0000561aa461e45d in wait_sync_command_done () at
/home/smarchi/src/binutils-gdb/gdb/top.c:546
#17 0x0000561aa461e675 in maybe_wait_sync_command_done (was_sync=0) at
/home/smarchi/src/binutils-gdb/gdb/top.c:563
#18 0x0000561aa461fb69 in execute_command (p=0x7ffff0b77633 "", from_tty=1) at
/home/smarchi/src/binutils-gdb/gdb/top.c:694
#19 0x0000561aa37a3799 in catch_command_errors (command=0x561aa461e692
<execute_command(char const*, int)>, arg=0x7ffff0b77632 "c", from_tty=1,
do_bp_actions=true)
    at /home/smarchi/src/binutils-gdb/gdb/main.c:513
#20 0x0000561aa37a4035 in execute_cmdargs (cmdarg_vec=0x7ffff0b752d0,
file_type=CMDARG_FILE, cmd_type=CMDARG_COMMAND, ret=0x7ffff0b74f90) at
/home/smarchi/src/binutils-gdb/gdb/main.c:608
#21 0x0000561aa37a8300 in captured_main_1 (context=0x7ffff0b75770) at
/home/smarchi/src/binutils-gdb/gdb/main.c:1299
#22 0x0000561aa37a8981 in captured_main (data=0x7ffff0b75770) at
/home/smarchi/src/binutils-gdb/gdb/main.c:1320
#23 0x0000561aa37a8a63 in gdb_main (args=0x7ffff0b75770) at
/home/smarchi/src/binutils-gdb/gdb/main.c:1345
#24 0x0000561aa1dd88ce in main (argc=16, argv=0x7ffff0b75908) at
/home/smarchi/src/binutils-gdb/gdb/gdb.c:32

The exception is thrown by exec_file_attach before we call target_follow_fork. 
The latter is responsible for pushing the process target on the child
inferior's target stack.  So inf->pid was assigned, but the process target slot
stays nullptr.  While unwinding (while the exception is propagating),
maybe_set_commit_resumed_all_targets gets called and we hit the segfault.

After investigating for a bit, I think this is what happens:

 - Process A, is initially in the same mnt namespace as GDB.  Its
program_space::exec_filename contains
/home/smarchi/src/northstar/target/release/northstar, which is a path in that
mnt namespace.
 - Process A changes to a different mnt namespace.
 - Process A forks, which creates process B, which is automatically in that
other mnt namespace.
 - While handling the fork and setting up the inferior for process B, GDB
clones process A's program_space, calling exec_file_attach with
/home/smarchi/src/northstar/target/release/northstar, the path still in process
A's program_space::exec_filename.
 - GDB sees that process B is in a different mnt namespace, so it tries to
access /home/smarchi/src/northstar/target/release/northstar in that other mnt
namespace.  This fails, because
/home/smarchi/src/northstar/target/release/northstar is a path in the original
namespace.

I was able to reproduce using a program that unlinks itself before forking:

$ cat fork.c
#include <unistd.h>
#include <stdlib.h>

int main(int argc, const char **argv)
{
  unlink (argv[0]);
  fork ();
  return 0;
}                                                                               
$ gcc fork.c -g                                                                 
$ ./gdb -nx -q --data-directory=data-directory a.out -ex "set follow-fork-mode
child" -ex r
Reading symbols from a.out...
Starting program: /home/smarchi/build/binutils-gdb/gdb/a.out 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/../lib/libthread_db.so.1".
[Attaching after Thread 0x7ffff7dbd740 (LWP 227302) fork to child process
227305]
[New inferior 2 (process 227305)]
/home/smarchi/src/binutils-gdb/gdb/infrun.c:2899:24: runtime error: member
access within null pointer of type 'struct process_stratum_target'   

This causes the same crash, because exec_file_attach can't find the file.

However, it's not exactly the same situation.  In the situation with the mnt
namespaces, the file still (typically) exists on the original namespace, so GDB
could (and should, I think) open it and read it.  GDB would need to remember
which mnt namespace that path came from.  So, these two cases should be fixed
and tested separately.

-- 
You are receiving this mail because:
You are on the CC list for the bug.