RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

["Yan, Zhiyong" <Zhiyong.Yan@windriver.com>]

[-- Attachment #1: Type: text/plain, Size: 4482 bytes --]

Hi Kevin,
    The callstack of assert is attached.
    Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.

    Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.

Best Regards.
Zhiyong

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com> 
Sent: Saturday, July 22, 2023 4:50 AM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.

I tested gdb.threads/*.exp using --target_board=native-gdbserver both with and without your patch.  Some of these tests are racy, but my conclusion from just looking at the PASSes and FAILs (after many test
runs) is that there are no regressions.

But then I remembered to enable core dumps on the Pi and after running gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main-vfork
by itself, I saw that it left a core file...

$ make check RUNTESTFLAGS="--target_board=native-gdbserver" TESTS=gdb.threads/pending-fork-event-detach.exp
...
                === gdb Summary ===

# of unexpected core files      1
# of expected passes            240

The core file was from the running test case, not gdbserver, nor gdb.

Looking at the core file in GDB shows...

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
29        x++;
[Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
(gdb) x/i $pc
=> 0x10624 <break_here+12>:     udf     #16
(gdb) x/x $pc
0x10624 <break_here+12>:        0xe7f001f0

...and in gdbserver/linux-aarch32-low.cc:

#define arm_eabi_breakpoint 0xe7f001f0UL

I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.

This core file is not created when I run the test using a gdbserver without your patch.

I'm suspicious of the assert in linux_process_target::maybe_hw_step.
Currently, it looks like this:

bool
linux_process_target::maybe_hw_step (thread_info *thread) {
  if (supports_hardware_single_step ())
    return true;
  else
    {
      /* GDBserver must insert single-step breakpoint for software
         single step.  */
      gdb_assert (has_single_step_breakpoints (thread));
      return false;
    }
}

But, when Yao Qi introduced it back in June, 2016, it looked like
this:

static int
maybe_hw_step (struct thread_info *thread) {
  if (can_hardware_single_step ())
    return 1;
  else
    {
      struct process_info *proc = get_thread_process (thread);

      /* GDBserver must insert reinsert breakpoint for software
     single step.  */
      gdb_assert (has_reinsert_breakpoints (proc));
      return 0;
    }
}

So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.

Also, back in 2016, maybe_hw_step() was only called from two locations; in each case it was in a block in which the condition
lwp->bp_reinsert != 0 was true.  But now there are two other
calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.

In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?

Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.

Kevin


[-- Attachment #2: callstack.txt --]
[-- Type: text/plain, Size: 5798 bytes --]

maybe_hw_step: Assertion `has_single_step_breakpoints (thread)' failed.
Aborted (core dumped)
EC@mpr3.1> ls -l
total 7464
-rw------- 1 root root  630784 Jun 28 12:14 core-gdbserver-11475-1656418477
-rwxr-xr-x 1 root root 7265572 Jun 28 12:12 gdbserver
EC@mpr3.1> gdb ./gdbserver ./core-gdbserver-11475-1656418477
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-wrs-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./gdbserver...
[New LWP 11475]

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `./gdbserver --once --attach :1234 325'.
Program terminated with signal SIGABRT, Aborted.
#0  0x76ca6216 in ?? () from /lib/libc.so.6
(gdb) bt
#0  0x76ca6216 in ?? () from /lib/libc.so.6
#1  0x76cb49d4 in raise () from /lib/libc.so.6
#2  0x76ca5ca0 in abort () from /lib/libc.so.6
#3  0x004a8280 in abort_or_exit () at ../../gdb-13.0.50.20221021/gdbserver/utils.cc:39
#4  internal_verror (file=<optimized out>, line=line@entry=2448, fmt=0x0, fmt@entry=0x7ee8c2bc "\210\316N", args=...,
    args@entry=...) at ../../gdb-13.0.50.20221021/gdbserver/utils.cc:108
#5  0x004d83ce in internal_error_loc (file=<optimized out>, line=line@entry=2448, fmt=0x4e4bc8 "%s: Assertion `%s' failed.")
    at ../../gdb-13.0.50.20221021/gdbsupport/errors.cc:58
#6  0x004c5c28 in linux_process_target::maybe_hw_step (this=<optimized out>, thread=<optimized out>)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2448
#7  linux_process_target::maybe_hw_step (this=<optimized out>, thread=<optimized out>)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2440
#8  0x004c660a in linux_process_target::resume_stopped_resumed_lwps (this=this@entry=0x50a3cc <the_arm_target>, thread=0x209fb28)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2466
#9  0x004c7614 in <lambda(thread_info*)>::operator() (__closure=<synthetic pointer>, thread=<optimized out>)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2606
#10 for_each_thread<linux_process_target::wait_for_event_filtered(ptid_t, ptid_t, int*, int)::<lambda(thread_info*)> > (func=...)
    at ../../gdb-13.0.50.20221021/gdbserver/gdbthread.h:159
#11 linux_process_target::wait_for_event_filtered (this=this@entry=0x50a3cc <the_arm_target>, wait_ptid=..., filter_ptid=...,
    wstatp=0x7ee8c544, options=1073741824) at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2604
#12 0x004c9426 in linux_process_target::wait_for_event (options=1079026852, wstatp=0x7ee8c544, ptid=...,
    this=0x50a3cc <the_arm_target>) at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2676
#13 linux_process_target::wait_1 (this=this@entry=0x50a3cc <the_arm_target>, ptid=...,
    ourstatus=ourstatus@entry=0x50c9f8 <g_client_state+1288>, target_options=..., target_options@entry=...)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2970
#14 0x004cab6c in linux_process_target::wait (this=0x50a3cc <the_arm_target>, ptid=..., ourstatus=0x50c9f8 <g_client_state+1288>,
    target_options=...) at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:3624
#15 0x004b9778 in target_wait (options=..., status=0x50c9f8 <g_client_state+1288>, ptid=...)
    at ../../gdb-13.0.50.20221021/gdbserver/target.cc:197
#16 mywait (ptid=..., ourstatus=ourstatus@entry=0x50c9f8 <g_client_state+1288>, options=...,
    connected_wait=connected_wait@entry=1) at ../../gdb-13.0.50.20221021/gdbserver/target.cc:142
#17 0x004b3428 in resume (actions=<optimized out>, num_actions=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--c
    at ../../gdb-13.0.50.20221021/gdbserver/server.cc:2916
#18 resume (actions=0x20a6778, num_actions=<optimized out>) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:2888
#19 0x004b3ed0 in handle_v_cont (own_buf=0x20a6778 "E\001") at ../../gdb-13.0.50.20221021/gdbserver/server.cc:2875
#20 handle_v_requests (own_buf=own_buf@entry=0x208b0a8 "vCont;r45b444,45b44c:p145.18a;c:p145.-1", packet_len=packet_len@entry=39, new_packet_len=new_packet_len@entry=0x7ee8c8ac) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:3135
#21 0x004b6b76 in process_serial_event () at ../../gdb-13.0.50.20221021/gdbserver/server.cc:4481
#22 handle_serial_event (err=<optimized out>, client_data=<optimized out>) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:4513
#23 0x004d8904 in gdb_wait_for_event (block=block@entry=1) at ../../gdb-13.0.50.20221021/gdbsupport/event-loop.cc:694
#24 0x004d9034 in gdb_wait_for_event (block=1) at ../../gdb-13.0.50.20221021/gdbsupport/event-loop.cc:593
#25 gdb_do_one_event (mstimeout=mstimeout@entry=-1) at ../../gdb-13.0.50.20221021/gdbsupport/event-loop.cc:264
#26 0x004a8a6c in start_event_loop () at ../../gdb-13.0.50.20221021/gdbserver/server.cc:3511
#27 captured_main (argv=<optimized out>, argc=5) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:3991
#28 main (argc=5, argv=<optimized out>) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:4077
(gdb)
(gdb)
(gdb) info thread
  Id   Target Id         Frame
* 1    LWP 11475         0x76ca6216 in ?? () from /lib/libc.so.6
(gdb)

[-- Attachment #3: gdbx2 --]
[-- Type: application/octet-stream, Size: 415 bytes --]

# gdbserver --once --attach :1234 $(pgrep eqptCntrl)

set pagination off

#target remote 128.224.176.94:1234
target remote 128.224.178.123:1234

#set logging file pr29742-host.log
#set logging enabled

#set debug remote 1
#set debug remote-packet-max-chars unlimited

handle SIG32 nostop noprint
handle SIGSTOP nostop noprint

b lpudated.c:92

c
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n