RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

["Yan, Zhiyong" <Zhiyong.Yan@windriver.com>]

Hi Kevin,
     Below is my PI's info:

root@bcm-2xxx-rpi4:~# uname -a
Linux bcm-2xxx-rpi4 5.15.110-yocto-standard #1 SMP PREEMPT Wed May 3 01:43:11 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

root@bcm-2xxx-rpi4:~# file gdbserver
gdbserver: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-aarch64.so.1, BuildID[sha1]=1e6ee58be0809d620fbdf3f86c8c4541f01ad9e9, for GNU/Linux 3.14.0, with debug_info, not stripped

root@bcm-2xxx-rpi4:~# file osm
osm: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=da7ee15aec73080ae3954d803b542bd9e8185c44, for GNU/Linux 3.14.0, with debug_info, not stripped
root@bcm-2xxx-rpi4:~#

     Both user app and OS are aarch64. 
[zyan1] On this, gdbserver supports hardware single step, assert doesn't happen.
-----------------------------------------------------------
Below is my Xilinx-zynq info: 
Last login: Wed Jul 26 03:03:09 2023
root@xilinx-zynq:~# uname -a
Linux xilinx-zynq 5.15.106-yocto-standard #1 SMP PREEMPT Tue Apr 11 03:06:10 UTC 2023 armv7l armv7l armv7l GNU/Linux
root@xilinx-zynq:~# which ls
/bin/ls
root@xilinx-zynq:~# file /bin/ls
/bin/ls: symbolic link to /bin/ls.coreutils
root@xilinx-zynq:~# file /bin/ls.coreutils
/bin/ls.coreutils: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=f8bf6bdad65965d53cdd9fd1ebfd00d191f4cbbc, for GNU/Linux 3.2.0, stripped
root@xilinx-zynq:~#

[zyan1] On this, gdbserver doesn't support hardware single step, assert can happen.

Best Regards.

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com> 
Sent: Tuesday, July 25, 2023 2:32 PM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

One problem that I encountered on my Pi, which may explain the behavior that you're seeing, is that recent 32-bit versions of the Raspberry Pi OS are running a 64-bit/aarch64 kernel, but the userland is 32-bit.

root@rpi4-2:~# /usr/bin/uname -m
aarch64
root@rpi4-2:~# file /usr/bin/ls
/usr/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=81004d065160807541b79235b23eea0e00a2d44e, for GNU/Linux 3.2.0, stripped

Note that uname -m returns aarch64, but that "ls" and other executables are "ELF 32-bit ...".

The binutils-gdb configury uses uname -m to figure out for what gdbserver host/target to build.  (host and target must be the same, otherwise gdbserver won't build.) So it may be the case that you built an aarch64 gdbserver instead of an arm gdbserver.  I think you can check this as follows:

kev@rpi4-2:/mesquite2/sourceware-git/rpi-master/bld/gdbserver$ ls linux-{arm,aarch}* linux-aarch32-low.o  linux-aarch32-tdesc.o  linux-arm-low.o  linux-arm-tdesc.o

If you also/instead see linux-aarch64-low.o and linux-aarch64-tdesc.o in that list, then you probably have an aarch64 gdbserver.

When I tried my build with uname -m returning aarch64, the build errored out because (I think) I was missing certain aarch64 header files.  But I knew that I didn't want to build for aarch64, so I abandoned that build.  What I ended up doing was making a wrapper for uname which substituted 'arm' for 'aarch64'.  I put it in /usr/local/bin, and /usr/local/bin is early in my PATH, so the configury finds it first...

root@rpi4-2:~# uname -m
arm

Here's my /usr/local/bin/uname script:
- - - -
root@rpi4-2:~# cat /usr/local/bin/uname
#!/bin/bash

/usr/bin/uname $* | sed -e s/aarch64/arm/
- - - -

[ Yes, this is a hack, but I couldn't think of a cleaner way to do it.
  I tried a configure line with "--host=arm-linux --target=host-linux",
  but that didn't work because something in the build wanted
  arm-linux-ar to exist and it didn't.  I could have made some
  symlinks, e.g. "ln -s /usr/bin/ar /usr/local/bin/arm-linux-ar", with
  similar symlinks for gcc, g++, ln, ranlib, etc, but that seemed like
  more work than my uname wrapper hack.]

I just checked my gdbserver build.  It's definitely getting into
arm_target::supports_hardware_single_step:

Breakpoint 1, linux_process_target::maybe_hw_step (
    this=0x8645c <the_arm_target>, thread=0x9df38)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-low.cc:2442
2442      if (supports_hardware_single_step ())
(gdb) s
arm_target::supports_hardware_single_step (this=0x8645c <the_arm_target>)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-arm-low.cc:1042
1042      return false;

Kevin

On Tue, 25 Jul 2023 04:21:00 +0000
"Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:

> Hi Kevin,
>      I test gdb11 on RaspBerry Pi4.
>      As you said, I can't produce this assert issue.
>      The direct reason is because supports_hardware_single_step () returns on RaspBerry Pi4, not like xilinux-zynq.
>      Please see attached pictures, we can see arm_target::supports_hardware_single_step () is never entered.
>      This assert only happens when supports_hardware_single_step () returns 'false'. On Raspberry Pi4, when I hardcoded supports_hardware_single_step () returns 'false', then assert happened.
>      For more information about " This assert only happens when supports_hardware_single_step () returns 'false'".
>      You can check 
> https://sourceware.org/bugzilla/show_bug.cgi?id=30387
>
>      So, the new question is why arm_target::supports_hardware_single_step () is never entered on Raspberry Pi4.
>
> Best Regards.
> Zhiyong
>
>
> -----Original Message-----,
> From: Kevin Buettner <kevinb@redhat.com>
> Sent: Tuesday, July 25, 2023 11:37 AM
> To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> pending thread whose last_resume_kind is resume_step
>
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> Hi Zhiyong,
>
> I looked at the backtrace that you provided and see that 
> maybe_hw_step() is being called from 
> linux_process_target::resume_stopped_resumed_lwps,
> which is the one location where I wasn't able to convince myself that the assert should hold.
>
> I was running your test case executable (osm) as an unprivileged user, 
> so neither the syslog calls nor the sudo were working.  (Sudo could 
> perhaps work, but it wanted to prompt for a password and stdin and 
> stdout were closed.)  I've since modified it so that sudo isn't used 
> and I'm using 'fprintf(stderr, ...)' instead of syslog - which is how 
> I discovered that sudo wasn't working.  I've tried next'ing quite a 
> lot, but so far I haven't reproduced the bug.  (Hopefully, the sudo 
> isn't required to reproduce the problem.)
>
> If you manage to reproduce the bug on a Raspberry Pi 4 (and tell me how to do it), that'd be great!
>
> So, what I'm doing, using three separate terminals, in an attempt to reproduce the bug is:
>
> 1) Run osm in terminal 1.  (I didn't want to mess with systemd.)  Once I start running it, I see a bunch of messages from the dd command.
>
> 2) In terminal 2, I run:
>
>    /path/to/gdbserver --debug --debug-format=all --remote-debug 
> --event-loop-debug --once --attach :1234 $(pgrep osm)
>
> 3) In terminal 3, I run:
>
>    /path/to/gdb osm -x ./gdbx2
>
> (I've changed the target remote command in gdbx2 to refer to 
> localhost.)
>
> I'm also attaching my hacked lupdated.c.  If you see anything wrong with what I'm trying, please let me know.
>
> Kevin
>
> On Mon, 24 Jul 2023 13:36:24 +0000
> "Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:
>
> > Hi Kevin,
> >     The callstack of assert is attached.
> >     Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.
> >
> >     Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.
> >
> > Best Regards.
> > Zhiyong
> >
> > -----Original Message-----
> > From: Kevin Buettner <kevinb@redhat.com>
> > Sent: Saturday, July 22, 2023 4:50 AM
> > To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> > Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> > Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> > pending thread whose last_resume_kind is resume_step
> >
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> > Hi Zhiyong,
> >
> > I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.
> >
> > I tested gdb.threads/*.exp using --target_board=native-gdbserver 
> > both with and without your patch.  Some of these tests are racy, but 
> > my conclusion from just looking at the PASSes and FAILs (after many 
> > test
> > runs) is that there are no regressions.
> >
> > But then I remembered to enable core dumps on the Pi and after 
> > running 
> > gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main
> > -v fork by itself, I saw that it left a core file...
> >
> > $ make check RUNTESTFLAGS="--target_board=native-gdbserver"
> > TESTS=gdb.threads/pending-fork-event-detach.exp
> > ...
> >                 === gdb Summary ===
> >
> > # of unexpected core files      1
> > # of expected passes            240
> >
> > The core file was from the running test case, not gdbserver, nor gdb.
> >
> > Looking at the core file in GDB shows...
> >
> > Program terminated with signal SIGTRAP, Trace/breakpoint trap.
> > #0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
> > 29        x++;
> > [Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
> > (gdb) x/i $pc
> > => 0x10624 <break_here+12>:     udf     #16
> > (gdb) x/x $pc
> > 0x10624 <break_here+12>:        0xe7f001f0
> >
> > ...and in gdbserver/linux-aarch32-low.cc:
> >
> > #define arm_eabi_breakpoint 0xe7f001f0UL
> >
> > I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.
> >
> > This core file is not created when I run the test using a gdbserver without your patch.
> >
> > I'm suspicious of the assert in linux_process_target::maybe_hw_step.
> > Currently, it looks like this:
> >
> > bool
> > linux_process_target::maybe_hw_step (thread_info *thread) {
> >   if (supports_hardware_single_step ())
> >     return true;
> >   else
> >     {
> >       /* GDBserver must insert single-step breakpoint for software
> >          single step.  */
> >       gdb_assert (has_single_step_breakpoints (thread));
> >       return false;
> >     }
> > }
> >
> > But, when Yao Qi introduced it back in June, 2016, it looked like
> > this:
> >
> > static int
> > maybe_hw_step (struct thread_info *thread) {
> >   if (can_hardware_single_step ())
> >     return 1;
> >   else
> >     {
> >       struct process_info *proc = get_thread_process (thread);
> >
> >       /* GDBserver must insert reinsert breakpoint for software
> >      single step.  */
> >       gdb_assert (has_reinsert_breakpoints (proc));
> >       return 0;
> >     }
> > }
> >
> > So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.
> >
> > Also, back in 2016, maybe_hw_step() was only called from two 
> > locations; in each case it was in a block in which the condition
> > lwp->bp_reinsert != 0 was true.  But now there are two other
> > calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.
> >
> > In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?
> >
> > Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.
> >
> > Kevin
> >