Re: Context switch during stepping causes weird behavior

[John Baldwin <jhb@FreeBSD.org>]

On 10/4/22 5:19 AM, Adrian Oltean via Gdb wrote:
> Hi everyone,
> 
> I'm currently facing an issue that occurs while stepping over code running in a
> kernel thread that gets moved by the target OS (Linux) on a different core.
> 
> To give you a little bit of background on my setup:
> - I have a custom GDB server able to control ARMv8 targets;
> - I'm using GDB 7.11.1 and GDB 11.1 but seeing the same behavior;
> - I'm running GDB in all-stop mode;
> - A thread from GDB is actually associated to a physical core from target;
> - I'm actually debugging a Linux kernel 5.15 with GDB (a bare-metal debug
> session but with an extra layer of python scripts to help control the target
> Linux kernel).
> What is the problem? I have a HW break somewhere inside the initialization
> function of a kernel module. Target stops in the breakpoint but the problem
> I face happens during a step over inside this init sequence. While GDB performing
> all the step actions (single stepping, range stepping, resuming, setting temp
> breaks, etc.) the Linux kernel decides to move the execution to a different
> core (so a different thread in my debug model). As a result, temp breaks set
> during stepping are hit on a different thread than the one used for initiating
> the step over. This completely messes-up the debug session. In other words,
> GDB ends-up in an infinite loop trying to finish the step over by switching
> to the initial thread, resuming it, setting other breaks that are than hit by
> other threads, resuming from those temp breaks, etc. Also, once GDB looses
> control of the stepping, the target Linux enters the "idle" loop, making GDB's
> job even more complicated when it comes to resuming from pointless breaks
> set during stepping. Note that it has to deal with 16 threads (actual HW cores)
> that constantly loop inside the "idle" subsystem.
> 
> I'm attaching below some logs. Maybe some trained eyes can help with some hints
> about how to avoid such an issue. Note that control is lost around address
> 0xffff80000945803c, when target is resumed and the actual kernel thread is moved
> from thread 3 to thread 4. Moreover, addresses 0xffff8000113dfb1c, 0xffff8000100a40a0
> or 0xffff8000113dfb20 are somewhere in the "idle" loop inside the Linux kernel.
> 
> Any help would be appreciated.

Interrupts make single-stepping on bare-metal or OS kernels hard.  When doing
similar things for debugging FreeBSD's kernel via bare-metal stubs (e.g. in
QEMU or in FreeBSD's hypervisor) I generally use 'until' and/or breakpoints
instead to work around this rather than using normal stepping.

On the GDB server side (for the GDB stub in FreeBSD's hypervisor) I've thought
about doing odd things like trying to defer interrupts while stepping but there
isn't a really good way to deal with that in general.

(Most of the time when trying to step what happens for me is that I get a
reported stop back for a PC in the timer interrupt handler for the local APIC
timer interrupt, and then GDB sees that the PC is "out of range" and just stops
at that point)

-- 
John Baldwin