Context switch during stepping causes weird behavior

[Adrian Oltean <adrian.oltean@nxp.com>]

[-- Attachment #1: Type: text/plain, Size: 31862 bytes --]

Hi everyone,

I'm currently facing an issue that occurs while stepping over code running in a
kernel thread that gets moved by the target OS (Linux) on a different core.

To give you a little bit of background on my setup:
- I have a custom GDB server able to control ARMv8 targets;
- I'm using GDB 7.11.1 and GDB 11.1 but seeing the same behavior;
- I'm running GDB in all-stop mode;
- A thread from GDB is actually associated to a physical core from target;
- I'm actually debugging a Linux kernel 5.15 with GDB (a bare-metal debug
session but with an extra layer of python scripts to help control the target
Linux kernel).
What is the problem? I have a HW break somewhere inside the initialization
function of a kernel module. Target stops in the breakpoint but the problem
I face happens during a step over inside this init sequence. While GDB performing
all the step actions (single stepping, range stepping, resuming, setting temp
breaks, etc.) the Linux kernel decides to move the execution to a different
core (so a different thread in my debug model). As a result, temp breaks set
during stepping are hit on a different thread than the one used for initiating
the step over. This completely messes-up the debug session. In other words,
GDB ends-up in an infinite loop trying to finish the step over by switching
to the initial thread, resuming it, setting other breaks that are than hit by
other threads, resuming from those temp breaks, etc. Also, once GDB looses
control of the stepping, the target Linux enters the "idle" loop, making GDB's
job even more complicated when it comes to resuming from pointless breaks
set during stepping. Note that it has to deal with 16 threads (actual HW cores)
that constantly loop inside the "idle" subsystem.

I'm attaching below some logs. Maybe some trained eyes can help with some hints
about how to avoid such an issue. Note that control is lost around address
0xffff80000945803c, when target is resumed and the actual kernel thread is moved
from thread 3 to thread 4. Moreover, addresses 0xffff8000113dfb1c, 0xffff8000100a40a0
or 0xffff8000113dfb20 are somewhere in the "idle" loop inside the Linux kernel.

Any help would be appreciated.

Thank you,
Adrian

-------------------------------------------------------------------------------

(gdb) list
warning: Source file is more recent than executable.
1191      static struct ctl_table_header *verbosity_sysctl_header;
1192      static int __init init_cryptodev(void)
1193      {
1194                     int rc;
1195
1196                     cryptodev_wq = create_workqueue("cryptodev_queue");
1197                     if (unlikely(!cryptodev_wq)) {
1198                                    pr_err(PFX "failed to allocate the cryptodev workqueue\n");
1199                                    return -EFAULT;
1200                     }
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  2    Thread 6          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  3    Thread 2          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
* 4    Thread 3          init_cryptodev () at /cryptodev_linux/ioctl.c:1196
  5    Thread 4          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  6    Thread 5          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  7    Thread 7          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  8    Thread 8          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  9    Thread 9          0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  10   Thread 10         __delay (cycles=25000) at /linux/arch/arm64/lib/delay.c:34
  11   Thread 11         0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  12   Thread 12         0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  13   Thread 13         0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  14   Thread 14         0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  15   Thread 15         0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
  16   Thread 16         0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
(gdb) frame 0
#0  init_cryptodev () at /cryptodev_linux/ioctl.c:1196
1196      /cryptodev_linux/ioctl.c: No such file or directory.
(gdb) show schedule-multiple
Resuming the execution of threads of all processes is off.
(gdb) next
infrun: clear_proceed_status_thread (Thread 1)
infrun: clear_proceed_status_thread (Thread 6)
infrun: clear_proceed_status_thread (Thread 2)
infrun: clear_proceed_status_thread (Thread 3)
infrun: clear_proceed_status_thread (Thread 4)
infrun: clear_proceed_status_thread (Thread 5)
infrun: clear_proceed_status_thread (Thread 7)
infrun: clear_proceed_status_thread (Thread 8)
infrun: clear_proceed_status_thread (Thread 9)
infrun: clear_proceed_status_thread (Thread 10)
infrun: clear_proceed_status_thread (Thread 11)
infrun: clear_proceed_status_thread (Thread 12)
infrun: clear_proceed_status_thread (Thread 13)
infrun: clear_proceed_status_thread (Thread 14)
infrun: clear_proceed_status_thread (Thread 15)
infrun: clear_proceed_status_thread (Thread 16)
infrun: proceed (addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT)
infrun: step-over queue now empty
infrun: resuming [Thread 3] for step-over
infrun: skipping breakpoint: stepping past insn at: 0xffff800009458000
infrun: skipping breakpoint: stepping past insn at: 0xffff800009458000
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 3] at 0xffff800009458000
infrun: infrun_async(1)
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   -1.0.0 [Thread 0],
infrun:   status->kind = ignore
infrun: TARGET_WAITKIND_IGNORE
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.3.0 [Thread 3],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff800009458004
infrun: stepping inside range [0xffff800009458000-0xffff800009458040]
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=0, current thread [Thread 3] at 0xffff800009458004
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.3.0 [Thread 3],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000100a40a0
infrun: stepped into subroutine
infrun: inserting step-resume breakpoint at 0xffff80000945803c
infrun: resume (step=0, signal=GDB_SIGNAL_0), trap_expected=0, current thread [Thread 3] at 0xffff8000100a40a0
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.4.0 [Thread 4],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff80000945803c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 4
infrun: BPSTAT_WHAT_SINGLE
infrun: thread [Thread 4] still needs step-over
infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 4] at 0xffff80000945803c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.4.0 [Thread 4],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff800009458040
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.1.0 [Thread 1],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun:   1.2.0 [Thread 2],
infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
///..... And a lot more...