* Context switch during stepping causes weird behavior
@ 2022-10-04 12:19 Adrian Oltean
2022-10-04 16:57 ` John Baldwin
2022-10-10 9:52 ` Luis Machado
0 siblings, 2 replies; 10+ messages in thread
From: Adrian Oltean @ 2022-10-04 12:19 UTC (permalink / raw)
To: gdb
[-- Attachment #1: Type: text/plain, Size: 31862 bytes --]
Hi everyone,
I'm currently facing an issue that occurs while stepping over code running in a
kernel thread that gets moved by the target OS (Linux) on a different core.
To give you a little bit of background on my setup:
- I have a custom GDB server able to control ARMv8 targets;
- I'm using GDB 7.11.1 and GDB 11.1 but seeing the same behavior;
- I'm running GDB in all-stop mode;
- A thread from GDB is actually associated to a physical core from target;
- I'm actually debugging a Linux kernel 5.15 with GDB (a bare-metal debug
session but with an extra layer of python scripts to help control the target
Linux kernel).
What is the problem? I have a HW break somewhere inside the initialization
function of a kernel module. Target stops in the breakpoint but the problem
I face happens during a step over inside this init sequence. While GDB performing
all the step actions (single stepping, range stepping, resuming, setting temp
breaks, etc.) the Linux kernel decides to move the execution to a different
core (so a different thread in my debug model). As a result, temp breaks set
during stepping are hit on a different thread than the one used for initiating
the step over. This completely messes-up the debug session. In other words,
GDB ends-up in an infinite loop trying to finish the step over by switching
to the initial thread, resuming it, setting other breaks that are than hit by
other threads, resuming from those temp breaks, etc. Also, once GDB looses
control of the stepping, the target Linux enters the "idle" loop, making GDB's
job even more complicated when it comes to resuming from pointless breaks
set during stepping. Note that it has to deal with 16 threads (actual HW cores)
that constantly loop inside the "idle" subsystem.
I'm attaching below some logs. Maybe some trained eyes can help with some hints
about how to avoid such an issue. Note that control is lost around address
0xffff80000945803c, when target is resumed and the actual kernel thread is moved
from thread 3 to thread 4. Moreover, addresses 0xffff8000113dfb1c, 0xffff8000100a40a0
or 0xffff8000113dfb20 are somewhere in the "idle" loop inside the Linux kernel.
Any help would be appreciated.
Thank you,
Adrian
-------------------------------------------------------------------------------
(gdb) list
warning: Source file is more recent than executable.
1191 static struct ctl_table_header *verbosity_sysctl_header;
1192 static int __init init_cryptodev(void)
1193 {
1194 int rc;
1195
1196 cryptodev_wq = create_workqueue("cryptodev_queue");
1197 if (unlikely(!cryptodev_wq)) {
1198 pr_err(PFX "failed to allocate the cryptodev workqueue\n");
1199 return -EFAULT;
1200 }
(gdb) info threads
Id Target Id Frame
1 Thread 1 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
2 Thread 6 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
3 Thread 2 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
* 4 Thread 3 init_cryptodev () at /cryptodev_linux/ioctl.c:1196
5 Thread 4 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
6 Thread 5 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
7 Thread 7 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
8 Thread 8 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
9 Thread 9 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
10 Thread 10 __delay (cycles=25000) at /linux/arch/arm64/lib/delay.c:34
11 Thread 11 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
12 Thread 12 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
13 Thread 13 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
14 Thread 14 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
15 Thread 15 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
16 Thread 16 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
(gdb) frame 0
#0 init_cryptodev () at /cryptodev_linux/ioctl.c:1196
1196 /cryptodev_linux/ioctl.c: No such file or directory.
(gdb) show schedule-multiple
Resuming the execution of threads of all processes is off.
(gdb) next
infrun: clear_proceed_status_thread (Thread 1)
infrun: clear_proceed_status_thread (Thread 6)
infrun: clear_proceed_status_thread (Thread 2)
infrun: clear_proceed_status_thread (Thread 3)
infrun: clear_proceed_status_thread (Thread 4)
infrun: clear_proceed_status_thread (Thread 5)
infrun: clear_proceed_status_thread (Thread 7)
infrun: clear_proceed_status_thread (Thread 8)
infrun: clear_proceed_status_thread (Thread 9)
infrun: clear_proceed_status_thread (Thread 10)
infrun: clear_proceed_status_thread (Thread 11)
infrun: clear_proceed_status_thread (Thread 12)
infrun: clear_proceed_status_thread (Thread 13)
infrun: clear_proceed_status_thread (Thread 14)
infrun: clear_proceed_status_thread (Thread 15)
infrun: clear_proceed_status_thread (Thread 16)
infrun: proceed (addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT)
infrun: step-over queue now empty
infrun: resuming [Thread 3] for step-over
infrun: skipping breakpoint: stepping past insn at: 0xffff800009458000
infrun: skipping breakpoint: stepping past insn at: 0xffff800009458000
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 3] at 0xffff800009458000
infrun: infrun_async(1)
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: -1.0.0 [Thread 0],
infrun: status->kind = ignore
infrun: TARGET_WAITKIND_IGNORE
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.3.0 [Thread 3],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff800009458004
infrun: stepping inside range [0xffff800009458000-0xffff800009458040]
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=0, current thread [Thread 3] at 0xffff800009458004
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.3.0 [Thread 3],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000100a40a0
infrun: stepped into subroutine
infrun: inserting step-resume breakpoint at 0xffff80000945803c
infrun: resume (step=0, signal=GDB_SIGNAL_0), trap_expected=0, current thread [Thread 3] at 0xffff8000100a40a0
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.4.0 [Thread 4],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff80000945803c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 4
infrun: BPSTAT_WHAT_SINGLE
infrun: thread [Thread 4] still needs step-over
infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 4] at 0xffff80000945803c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.4.0 [Thread 4],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff800009458040
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 1
infrun: [Thread 1] hit another thread's single-step breakpoint
infrun: need to step [Thread 1] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.1.0 [Thread 1],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
infrun: switching back to stepped thread
infrun: resuming previously stepped thread
infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
infrun: clear_step_over_info
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0xffff8000113dfb1c
infrun: context switch
infrun: Switching context from Thread 3 to Thread 2
infrun: [Thread 2] hit another thread's single-step breakpoint
infrun: need to step [Thread 2] over single-step breakpoint
infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
infrun: prepare_to_wait
infrun: target_wait (-1.0.0, status) =
infrun: 1.2.0 [Thread 2],
infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
infrun: TARGET_WAITKIND_STOPPED
infrun: clear_step_over_info
infrun: stop_pc = 0xffff8000113dfb20
///..... And a lot more...
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Context switch during stepping causes weird behavior
2022-10-04 12:19 Context switch during stepping causes weird behavior Adrian Oltean
@ 2022-10-04 16:57 ` John Baldwin
2022-10-05 6:48 ` Adrian Oltean
2022-10-10 9:52 ` Luis Machado
1 sibling, 1 reply; 10+ messages in thread
From: John Baldwin @ 2022-10-04 16:57 UTC (permalink / raw)
To: Adrian Oltean, gdb
On 10/4/22 5:19 AM, Adrian Oltean via Gdb wrote:
> Hi everyone,
>
> I'm currently facing an issue that occurs while stepping over code running in a
> kernel thread that gets moved by the target OS (Linux) on a different core.
>
> To give you a little bit of background on my setup:
> - I have a custom GDB server able to control ARMv8 targets;
> - I'm using GDB 7.11.1 and GDB 11.1 but seeing the same behavior;
> - I'm running GDB in all-stop mode;
> - A thread from GDB is actually associated to a physical core from target;
> - I'm actually debugging a Linux kernel 5.15 with GDB (a bare-metal debug
> session but with an extra layer of python scripts to help control the target
> Linux kernel).
> What is the problem? I have a HW break somewhere inside the initialization
> function of a kernel module. Target stops in the breakpoint but the problem
> I face happens during a step over inside this init sequence. While GDB performing
> all the step actions (single stepping, range stepping, resuming, setting temp
> breaks, etc.) the Linux kernel decides to move the execution to a different
> core (so a different thread in my debug model). As a result, temp breaks set
> during stepping are hit on a different thread than the one used for initiating
> the step over. This completely messes-up the debug session. In other words,
> GDB ends-up in an infinite loop trying to finish the step over by switching
> to the initial thread, resuming it, setting other breaks that are than hit by
> other threads, resuming from those temp breaks, etc. Also, once GDB looses
> control of the stepping, the target Linux enters the "idle" loop, making GDB's
> job even more complicated when it comes to resuming from pointless breaks
> set during stepping. Note that it has to deal with 16 threads (actual HW cores)
> that constantly loop inside the "idle" subsystem.
>
> I'm attaching below some logs. Maybe some trained eyes can help with some hints
> about how to avoid such an issue. Note that control is lost around address
> 0xffff80000945803c, when target is resumed and the actual kernel thread is moved
> from thread 3 to thread 4. Moreover, addresses 0xffff8000113dfb1c, 0xffff8000100a40a0
> or 0xffff8000113dfb20 are somewhere in the "idle" loop inside the Linux kernel.
>
> Any help would be appreciated.
Interrupts make single-stepping on bare-metal or OS kernels hard. When doing
similar things for debugging FreeBSD's kernel via bare-metal stubs (e.g. in
QEMU or in FreeBSD's hypervisor) I generally use 'until' and/or breakpoints
instead to work around this rather than using normal stepping.
On the GDB server side (for the GDB stub in FreeBSD's hypervisor) I've thought
about doing odd things like trying to defer interrupts while stepping but there
isn't a really good way to deal with that in general.
(Most of the time when trying to step what happens for me is that I get a
reported stop back for a PC in the timer interrupt handler for the local APIC
timer interrupt, and then GDB sees that the PC is "out of range" and just stops
at that point)
--
John Baldwin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Context switch during stepping causes weird behavior
2022-10-04 16:57 ` John Baldwin
@ 2022-10-05 6:48 ` Adrian Oltean
2022-10-05 16:53 ` John Baldwin
0 siblings, 1 reply; 10+ messages in thread
From: Adrian Oltean @ 2022-10-05 6:48 UTC (permalink / raw)
To: John Baldwin, gdb
Hi John,
Thanks for sharing your experience. I'm also planning to recommend our customers to use breakpoints instead of actual step operations. However, I still don't like the uncontrollable GDB when stepping and kernel thread gets moved. In your case, I see a friendlier behavior - at least stepping stops at some point, even though not reaching the expected code section. This would be my goal as well. Maybe someone can suggest how to patch the GDB code in order to suspend the step when GDB detects that stepping thread is actually changed. Unfortunately, I don't have much knowledge about the GDB code and all my attempts failed so far.
Thank you,
Adrian
-----Original Message-----
Interrupts make single-stepping on bare-metal or OS kernels hard. When doing similar things for debugging FreeBSD's kernel via bare-metal stubs (e.g. in QEMU or in FreeBSD's hypervisor) I generally use 'until' and/or breakpoints instead to work around this rather than using normal stepping.
On the GDB server side (for the GDB stub in FreeBSD's hypervisor) I've thought about doing odd things like trying to defer interrupts while stepping but there isn't a really good way to deal with that in general.
(Most of the time when trying to step what happens for me is that I get a reported stop back for a PC in the timer interrupt handler for the local APIC timer interrupt, and then GDB sees that the PC is "out of range" and just stops at that point)
--
John Baldwin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Context switch during stepping causes weird behavior
2022-10-05 6:48 ` Adrian Oltean
@ 2022-10-05 16:53 ` John Baldwin
2022-10-07 15:40 ` Keith Seitz
0 siblings, 1 reply; 10+ messages in thread
From: John Baldwin @ 2022-10-05 16:53 UTC (permalink / raw)
To: Adrian Oltean, gdb
On 10/4/22 11:48 PM, Adrian Oltean wrote:
> Hi John,
>
> Thanks for sharing your experience. I'm also planning to recommend our customers to use breakpoints instead of actual step operations. However, I still don't like the uncontrollable GDB when stepping and kernel thread gets moved. In your case, I see a friendlier behavior - at least stepping stops at some point, even though not reaching the expected code section. This would be my goal as well. Maybe someone can suggest how to patch the GDB code in order to suspend the step when GDB detects that stepping thread is actually changed. Unfortunately, I don't have much knowledge about the GDB code and all my attempts failed so far.
I wonder if the difference is due to the use of displaced stepping?
FreeBSD doesn't support displaced stepping. I don't know off the top
of my head if there is a knob to disable displaced stepping. If there
is, you might try doing that to see if you get the type of behavior I
see where the debugger stops because a step ends up out of bounds.
> Thank you,
> Adrian
>
> -----Original Message-----
>
> Interrupts make single-stepping on bare-metal or OS kernels hard. When doing similar things for debugging FreeBSD's kernel via bare-metal stubs (e.g. in QEMU or in FreeBSD's hypervisor) I generally use 'until' and/or breakpoints instead to work around this rather than using normal stepping.
>
> On the GDB server side (for the GDB stub in FreeBSD's hypervisor) I've thought about doing odd things like trying to defer interrupts while stepping but there isn't a really good way to deal with that in general.
>
> (Most of the time when trying to step what happens for me is that I get a reported stop back for a PC in the timer interrupt handler for the local APIC timer interrupt, and then GDB sees that the PC is "out of range" and just stops at that point)
>
> --
> John Baldwin
--
John Baldwin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Context switch during stepping causes weird behavior
2022-10-05 16:53 ` John Baldwin
@ 2022-10-07 15:40 ` Keith Seitz
2022-10-12 6:46 ` [EXT] " Adrian Oltean
0 siblings, 1 reply; 10+ messages in thread
From: Keith Seitz @ 2022-10-07 15:40 UTC (permalink / raw)
To: John Baldwin, Adrian Oltean, gdb
On 10/5/22 09:53, John Baldwin wrote:
> I don't know off the top of my head if there is a knob to disable
> displaced stepping.
FYI:
(gdb) apropos displaced
set debug displaced -- Set displaced stepping debugging.
set displaced-stepping -- Set debugger's willingness to use displaced stepping.
show debug displaced -- Show displaced stepping debugging.
show displaced-stepping -- Show debugger's willingness to use displaced stepping.
Keith
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [EXT] Re: Context switch during stepping causes weird behavior
2022-10-07 15:40 ` Keith Seitz
@ 2022-10-12 6:46 ` Adrian Oltean
2022-10-13 6:34 ` Luis Machado
0 siblings, 1 reply; 10+ messages in thread
From: Adrian Oltean @ 2022-10-12 6:46 UTC (permalink / raw)
To: Keith Seitz, John Baldwin, gdb
Thanks guys. Looks like displaced stepping is off by default when GDB is running
in all-stop mode - my case. So I'll have to look for other ways to work around
the behavior I described.
Adrian
> Caution: EXT Email
>
> On 10/5/22 09:53, John Baldwin wrote:
>
> > I don't know off the top of my head if there is a knob to disable
> > displaced stepping.
> FYI:
>
> (gdb) apropos displaced
> set debug displaced -- Set displaced stepping debugging.
> set displaced-stepping -- Set debugger's willingness to use displaced stepping.
> show debug displaced -- Show displaced stepping debugging.
> show displaced-stepping -- Show debugger's willingness to use displaced
> stepping.
>
> Keith
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [EXT] Re: Context switch during stepping causes weird behavior
2022-10-12 6:46 ` [EXT] " Adrian Oltean
@ 2022-10-13 6:34 ` Luis Machado
2022-10-13 6:41 ` Adrian Oltean
0 siblings, 1 reply; 10+ messages in thread
From: Luis Machado @ 2022-10-13 6:34 UTC (permalink / raw)
To: Adrian Oltean, Keith Seitz, John Baldwin, gdb
Hi Adrian,
On 10/12/22 07:46, Adrian Oltean via Gdb wrote:
> Thanks guys. Looks like displaced stepping is off by default when GDB is running
> in all-stop mode - my case. So I'll have to look for other ways to work around
> the behavior I described.
That is a bit misleading. It used to be the case it was only used for non-stop mode, but
nowadays it is supported if the underlying target supports it.
I don't think Arm (As in arm-none-elf, I'm assuming this is the case here) supports it, but when
debugging linux-hosted Arm binaries, then it is supported.
See gdb/infrun.c:gdbarch_supports_displaced_stepping.
>
> Adrian
>
>> Caution: EXT Email
>>
>> On 10/5/22 09:53, John Baldwin wrote:
>>
>>> I don't know off the top of my head if there is a knob to disable
>>> displaced stepping.
>> FYI:
>>
>> (gdb) apropos displaced
>> set debug displaced -- Set displaced stepping debugging.
>> set displaced-stepping -- Set debugger's willingness to use displaced stepping.
>> show debug displaced -- Show displaced stepping debugging.
>> show displaced-stepping -- Show debugger's willingness to use displaced
>> stepping.
>>
>> Keith
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [EXT] Re: Context switch during stepping causes weird behavior
2022-10-13 6:34 ` Luis Machado
@ 2022-10-13 6:41 ` Adrian Oltean
0 siblings, 0 replies; 10+ messages in thread
From: Adrian Oltean @ 2022-10-13 6:41 UTC (permalink / raw)
To: Luis Machado, Keith Seitz, John Baldwin, gdb
Hi Luis,
Thank you for clarifying this. I concluded that displaced stepping is off
based on the description I found in the GDB user guide. Maybe it's worth
updating the docs as well.
Adrian
> -----Original Message-----
>
> On 10/12/22 07:46, Adrian Oltean via Gdb wrote:
> > Thanks guys. Looks like displaced stepping is off by default when GDB
> > is running in all-stop mode - my case. So I'll have to look for other
> > ways to work around the behavior I described.
>
> That is a bit misleading. It used to be the case it was only used for non-stop
> mode, but nowadays it is supported if the underlying target supports it.
>
> I don't think Arm (As in arm-none-elf, I'm assuming this is the case here) supports
> it, but when debugging linux-hosted Arm binaries, then it is supported.
>
> See gdb/infrun.c:gdbarch_supports_displaced_stepping.
> >
> > Adrian
> >
> >> Caution: EXT Email
> >>
> >> On 10/5/22 09:53, John Baldwin wrote:
> >>
> >>> I don't know off the top of my head if there is a knob to disable
> >>> displaced stepping.
> >> FYI:
> >>
> >> (gdb) apropos displaced
> >> set debug displaced -- Set displaced stepping debugging.
> >> set displaced-stepping -- Set debugger's willingness to use displaced stepping.
> >> show debug displaced -- Show displaced stepping debugging.
> >> show displaced-stepping -- Show debugger's willingness to use
> >> displaced stepping.
> >>
> >> Keith
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Context switch during stepping causes weird behavior
2022-10-04 12:19 Context switch during stepping causes weird behavior Adrian Oltean
2022-10-04 16:57 ` John Baldwin
@ 2022-10-10 9:52 ` Luis Machado
2022-10-12 6:40 ` [EXT] " Adrian Oltean
1 sibling, 1 reply; 10+ messages in thread
From: Luis Machado @ 2022-10-10 9:52 UTC (permalink / raw)
To: Adrian Oltean, gdb
On 10/4/22 13:19, Adrian Oltean via Gdb wrote:
> Hi everyone,
>
> I'm currently facing an issue that occurs while stepping over code running in a
> kernel thread that gets moved by the target OS (Linux) on a different core.
>
> To give you a little bit of background on my setup:
> - I have a custom GDB server able to control ARMv8 targets;
> - I'm using GDB 7.11.1 and GDB 11.1 but seeing the same behavior;
> - I'm running GDB in all-stop mode;
> - A thread from GDB is actually associated to a physical core from target;
> - I'm actually debugging a Linux kernel 5.15 with GDB (a bare-metal debug
> session but with an extra layer of python scripts to help control the target
> Linux kernel).
> What is the problem? I have a HW break somewhere inside the initialization
> function of a kernel module. Target stops in the breakpoint but the problem
> I face happens during a step over inside this init sequence. While GDB performing
> all the step actions (single stepping, range stepping, resuming, setting temp
> breaks, etc.) the Linux kernel decides to move the execution to a different
> core (so a different thread in my debug model). As a result, temp breaks set
> during stepping are hit on a different thread than the one used for initiating
> the step over. This completely messes-up the debug session. In other words,
> GDB ends-up in an infinite loop trying to finish the step over by switching
> to the initial thread, resuming it, setting other breaks that are than hit by
> other threads, resuming from those temp breaks, etc. Also, once GDB looses
> control of the stepping, the target Linux enters the "idle" loop, making GDB's
> job even more complicated when it comes to resuming from pointless breaks
> set during stepping. Note that it has to deal with 16 threads (actual HW cores)
> that constantly loop inside the "idle" subsystem.
>
> I'm attaching below some logs. Maybe some trained eyes can help with some hints
> about how to avoid such an issue. Note that control is lost around address
> 0xffff80000945803c, when target is resumed and the actual kernel thread is moved
> from thread 3 to thread 4. Moreover, addresses 0xffff8000113dfb1c, 0xffff8000100a40a0
> or 0xffff8000113dfb20 are somewhere in the "idle" loop inside the Linux kernel.
>
> Any help would be appreciated.
>
> Thank you,
> Adrian
>
> -------------------------------------------------------------------------------
>
> (gdb) list
> warning: Source file is more recent than executable.
> 1191 static struct ctl_table_header *verbosity_sysctl_header;
> 1192 static int __init init_cryptodev(void)
> 1193 {
> 1194 int rc;
> 1195
> 1196 cryptodev_wq = create_workqueue("cryptodev_queue");
> 1197 if (unlikely(!cryptodev_wq)) {
> 1198 pr_err(PFX "failed to allocate the cryptodev workqueue\n");
> 1199 return -EFAULT;
> 1200 }
> (gdb) info threads
> Id Target Id Frame
> 1 Thread 1 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 2 Thread 6 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 3 Thread 2 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> * 4 Thread 3 init_cryptodev () at /cryptodev_linux/ioctl.c:1196
> 5 Thread 4 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 6 Thread 5 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 7 Thread 7 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 8 Thread 8 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 9 Thread 9 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 10 Thread 10 __delay (cycles=25000) at /linux/arch/arm64/lib/delay.c:34
> 11 Thread 11 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 12 Thread 12 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 13 Thread 13 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 14 Thread 14 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 15 Thread 15 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> 16 Thread 16 0xffff8000113dfb1c in __cpu_do_idle () at /linux/arch/arm64/kernel/process.c:78
> (gdb) frame 0
> #0 init_cryptodev () at /cryptodev_linux/ioctl.c:1196
> 1196 /cryptodev_linux/ioctl.c: No such file or directory.
> (gdb) show schedule-multiple
> Resuming the execution of threads of all processes is off.
> (gdb) next
> infrun: clear_proceed_status_thread (Thread 1)
> infrun: clear_proceed_status_thread (Thread 6)
> infrun: clear_proceed_status_thread (Thread 2)
> infrun: clear_proceed_status_thread (Thread 3)
> infrun: clear_proceed_status_thread (Thread 4)
> infrun: clear_proceed_status_thread (Thread 5)
> infrun: clear_proceed_status_thread (Thread 7)
> infrun: clear_proceed_status_thread (Thread 8)
> infrun: clear_proceed_status_thread (Thread 9)
> infrun: clear_proceed_status_thread (Thread 10)
> infrun: clear_proceed_status_thread (Thread 11)
> infrun: clear_proceed_status_thread (Thread 12)
> infrun: clear_proceed_status_thread (Thread 13)
> infrun: clear_proceed_status_thread (Thread 14)
> infrun: clear_proceed_status_thread (Thread 15)
> infrun: clear_proceed_status_thread (Thread 16)
> infrun: proceed (addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT)
> infrun: step-over queue now empty
> infrun: resuming [Thread 3] for step-over
> infrun: skipping breakpoint: stepping past insn at: 0xffff800009458000
> infrun: skipping breakpoint: stepping past insn at: 0xffff800009458000
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 3] at 0xffff800009458000
> infrun: infrun_async(1)
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: -1.0.0 [Thread 0],
> infrun: status->kind = ignore
> infrun: TARGET_WAITKIND_IGNORE
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.3.0 [Thread 3],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff800009458004
> infrun: stepping inside range [0xffff800009458000-0xffff800009458040]
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=0, current thread [Thread 3] at 0xffff800009458004
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.3.0 [Thread 3],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000100a40a0
> infrun: stepped into subroutine
> infrun: inserting step-resume breakpoint at 0xffff80000945803c
> infrun: resume (step=0, signal=GDB_SIGNAL_0), trap_expected=0, current thread [Thread 3] at 0xffff8000100a40a0
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.4.0 [Thread 4],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff80000945803c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 4
> infrun: BPSTAT_WHAT_SINGLE
> infrun: thread [Thread 4] still needs step-over
> infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
> infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
> infrun: skipping breakpoint: stepping past insn at: 0xffff80000945803c
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 4] at 0xffff80000945803c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.4.0 [Thread 4],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff800009458040
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 1
> infrun: [Thread 1] hit another thread's single-step breakpoint
> infrun: need to step [Thread 1] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 1] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.1.0 [Thread 1],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> infrun: switching back to stepped thread
> infrun: resuming previously stepped thread
> infrun: expected thread advanced also (0xffff8000100a40a0 -> 0xffff8000113dfb1c)
> infrun: clear_step_over_info
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0xffff8000113dfb1c
> infrun: context switch
> infrun: Switching context from Thread 3 to Thread 2
> infrun: [Thread 2] hit another thread's single-step breakpoint
> infrun: need to step [Thread 2] over single-step breakpoint
> infrun: resume (step=1, signal=GDB_SIGNAL_0), trap_expected=1, current thread [Thread 2] at 0xffff8000113dfb1c
> infrun: prepare_to_wait
> infrun: target_wait (-1.0.0, status) =
> infrun: 1.2.0 [Thread 2],
> infrun: status->kind = stopped, signal = GDB_SIGNAL_TRAP
> infrun: TARGET_WAITKIND_STOPPED
> infrun: clear_step_over_info
> infrun: stop_pc = 0xffff8000113dfb20
> ///..... And a lot more...
GDB doesn't really like things changing behind its back, and I don't think this case is handled very well (or at all).
When the thread migrates to a different core, would it be possible to re-order the thread numbering so GDB's view of the threads
is unchanged?
^ permalink raw reply [flat|nested] 10+ messages in thread* RE: [EXT] Re: Context switch during stepping causes weird behavior
2022-10-10 9:52 ` Luis Machado
@ 2022-10-12 6:40 ` Adrian Oltean
0 siblings, 0 replies; 10+ messages in thread
From: Adrian Oltean @ 2022-10-12 6:40 UTC (permalink / raw)
To: Luis Machado, gdb
Hi Luis,
Indeed, GDB does not handle very well such a case...
I'll have to think about your suggestion. The idea is not bad but my GDB server is not
aware about the underlying debugged app (Linux kernel), thus making it complicated
to map kernel threads from target to actual GDB threads.
Thanks,
Adrian
> -----Original Message-----
> GDB doesn't really like things changing behind its back, and I don't think this case
> is handled very well (or at all).
>
> When the thread migrates to a different core, would it be possible to re-order
> the thread numbering so GDB's view of the threads is unchanged?
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-10-13 6:41 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-10-04 12:19 Context switch during stepping causes weird behavior Adrian Oltean
2022-10-04 16:57 ` John Baldwin
2022-10-05 6:48 ` Adrian Oltean
2022-10-05 16:53 ` John Baldwin
2022-10-07 15:40 ` Keith Seitz
2022-10-12 6:46 ` [EXT] " Adrian Oltean
2022-10-13 6:34 ` Luis Machado
2022-10-13 6:41 ` Adrian Oltean
2022-10-10 9:52 ` Luis Machado
2022-10-12 6:40 ` [EXT] " Adrian Oltean
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).