Hardware watchpoints on GNU/Linux AArch64

Hardware watchpoints on GNU/Linux AArch64

[Florian Weimer]

Running

gdb-12.1-10.fc38.aarch64
kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64

I do not seem to be able to use hardware watchpoints.  GDB just logs
warnings:

warning: Unable to determine the number of hardware watchpoints available.
warning: Unable to determine the number of hardware breakpoints available.

I see AArch64-specific code in GDB, so this is a bit of a surprise to
me.

The machine uses virtualization:

Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 3
  On-line CPU(s) list:  0-2
Vendor ID:              ARM
  BIOS Vendor ID:       QEMU
  Model name:           Neoverse-N1
    BIOS Model name:    virt-rhel8.6.0  CPU @ 2.0GHz
    BIOS CPU family:    1
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 1
    Socket(s):          3
    Stepping:           r3p1
    BogoMIPS:           50.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp
                         asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs

Is this actually expected to work?  Would we have more luck on a
bare-metal system?

Thanks,
Florian

Re: Hardware watchpoints on GNU/Linux AArch64

[Andreas Schwab]

On Jan 02 2023, Florian Weimer via Gdb wrote:

> Is this actually expected to work?  Would we have more luck on a
> bare-metal system?

Worksforme on real hardware, probably a missing qemu emulation.

You can try maintenance set show-debug-regs 1 to see if you get more
information.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Hardware watchpoints on GNU/Linux AArch64

[Florian Weimer]

* Andreas Schwab:

> On Jan 02 2023, Florian Weimer via Gdb wrote:
>
>> Is this actually expected to work?  Would we have more luck on a
>> bare-metal system?
>
> Worksforme on real hardware, probably a missing qemu emulation.
>
> You can try maintenance set show-debug-regs 1 to see if you get more
> information.

It doesn't show more details unfortunately.

This is KVM running on real hardware.  Admittedly, it could be an
outdated kernel.

Thanks,
Florian

Re: Hardware watchpoints on GNU/Linux AArch64

[Andrew Pinski]

On Mon, Jan 2, 2023 at 8:49 AM Florian Weimer via Gdb
<gdb@sourceware.org> wrote:
>
> Running
>
> gdb-12.1-10.fc38.aarch64
> kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64
>
> I do not seem to be able to use hardware watchpoints.  GDB just logs
> warnings:
>
> warning: Unable to determine the number of hardware watchpoints available.
> warning: Unable to determine the number of hardware breakpoints available.

This shows up when the ptrace either fails or ptrace returns an
unknown value for AARCH64_DEBUG_ARCH value.
In the boot log there should be a message saying how many HW
breakpoint and watchpoints there are and what version of the debug
arch is there.
The supported values for debug arch are:
#define AARCH64_DEBUG_ARCH_V8 0x6
#define AARCH64_DEBUG_ARCH_V8_1 0x7
#define AARCH64_DEBUG_ARCH_V8_2 0x8
#define AARCH64_DEBUG_ARCH_V8_4 0x9

I don't know of any other values that can be reported, especially for
Neoverse-N1 (which should be 8_2). Neoverse-N2 is 8_4.

Thanks,
Andrew Pinski

>
> I see AArch64-specific code in GDB, so this is a bit of a surprise to
> me.
>
> The machine uses virtualization:
>
> Architecture:           aarch64
>   CPU op-mode(s):       32-bit, 64-bit
>   Byte Order:           Little Endian
> CPU(s):                 3
>   On-line CPU(s) list:  0-2
> Vendor ID:              ARM
>   BIOS Vendor ID:       QEMU
>   Model name:           Neoverse-N1
>     BIOS Model name:    virt-rhel8.6.0  CPU @ 2.0GHz
>     BIOS CPU family:    1
>     Model:              1
>     Thread(s) per core: 1
>     Core(s) per socket: 1
>     Socket(s):          3
>     Stepping:           r3p1
>     BogoMIPS:           50.00
>     Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp
>                          asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
>
> Is this actually expected to work?  Would we have more luck on a
> bare-metal system?
>
> Thanks,
> Florian
>

Re: Hardware watchpoints on GNU/Linux AArch64

[Andrew Pinski]

On Mon, Jan 2, 2023 at 9:28 AM Andrew Pinski <pinskia@gmail.com> wrote:
>
> On Mon, Jan 2, 2023 at 8:49 AM Florian Weimer via Gdb
> <gdb@sourceware.org> wrote:
> >
> > Running
> >
> > gdb-12.1-10.fc38.aarch64
> > kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64
> >
> > I do not seem to be able to use hardware watchpoints.  GDB just logs
> > warnings:
> >
> > warning: Unable to determine the number of hardware watchpoints available.
> > warning: Unable to determine the number of hardware breakpoints available.
>
> This shows up when the ptrace either fails or ptrace returns an
> unknown value for AARCH64_DEBUG_ARCH value.
> In the boot log there should be a message saying how many HW
> breakpoint and watchpoints there are and what version of the debug
> arch is there.
> The supported values for debug arch are:
> #define AARCH64_DEBUG_ARCH_V8 0x6
> #define AARCH64_DEBUG_ARCH_V8_1 0x7
> #define AARCH64_DEBUG_ARCH_V8_2 0x8
> #define AARCH64_DEBUG_ARCH_V8_4 0x9
>
> I don't know of any other values that can be reported, especially for
> Neoverse-N1 (which should be 8_2). Neoverse-N2 is 8_4.

The kernel fills in that field with ptrace via:
u8 debug_monitors_arch(void)
{
        return cpuid_feature_extract_unsigned_field(read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1),
                                                ID_AA64DFR0_EL1_DebugVer_SHIFT);
}

So either kvm/qemu is setting that EL1 (system) register incorrectly
or the KVM Invocation didn't turn on HW breakpoints. Or KVM does not
support HW breakpoints yet.

Thanks,
Andrew


>
> Thanks,
> Andrew Pinski
>
> >
> > I see AArch64-specific code in GDB, so this is a bit of a surprise to
> > me.
> >
> > The machine uses virtualization:
> >
> > Architecture:           aarch64
> >   CPU op-mode(s):       32-bit, 64-bit
> >   Byte Order:           Little Endian
> > CPU(s):                 3
> >   On-line CPU(s) list:  0-2
> > Vendor ID:              ARM
> >   BIOS Vendor ID:       QEMU
> >   Model name:           Neoverse-N1
> >     BIOS Model name:    virt-rhel8.6.0  CPU @ 2.0GHz
> >     BIOS CPU family:    1
> >     Model:              1
> >     Thread(s) per core: 1
> >     Core(s) per socket: 1
> >     Socket(s):          3
> >     Stepping:           r3p1
> >     BogoMIPS:           50.00
> >     Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp
> >                          asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
> >
> > Is this actually expected to work?  Would we have more luck on a
> > bare-metal system?
> >
> > Thanks,
> > Florian
> >

Re: Hardware watchpoints on GNU/Linux AArch64

[Florian Weimer]

* Andrew Pinski:

> On Mon, Jan 2, 2023 at 8:49 AM Florian Weimer via Gdb
> <gdb@sourceware.org> wrote:
>>
>> Running
>>
>> gdb-12.1-10.fc38.aarch64
>> kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64
>>
>> I do not seem to be able to use hardware watchpoints.  GDB just logs
>> warnings:
>>
>> warning: Unable to determine the number of hardware watchpoints available.
>> warning: Unable to determine the number of hardware breakpoints available.
>
> This shows up when the ptrace either fails or ptrace returns an
> unknown value for AARCH64_DEBUG_ARCH value.
> In the boot log there should be a message saying how many HW
> breakpoint and watchpoints there are and what version of the debug
> arch is there.

I only see this in the kernel log:

hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.

Nothing else in the vicinity.

Thanks,
Florian

Re: Hardware watchpoints on GNU/Linux AArch64

[Florian Weimer]

* Florian Weimer:

> * Andrew Pinski:
>
>> On Mon, Jan 2, 2023 at 8:49 AM Florian Weimer via Gdb
>> <gdb@sourceware.org> wrote:
>>>
>>> Running
>>>
>>> gdb-12.1-10.fc38.aarch64
>>> kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64
>>>
>>> I do not seem to be able to use hardware watchpoints.  GDB just logs
>>> warnings:
>>>
>>> warning: Unable to determine the number of hardware watchpoints available.
>>> warning: Unable to determine the number of hardware breakpoints available.
>>
>> This shows up when the ptrace either fails or ptrace returns an
>> unknown value for AARCH64_DEBUG_ARCH value.
>> In the boot log there should be a message saying how many HW
>> breakpoint and watchpoints there are and what version of the debug
>> arch is there.
>
> I only see this in the kernel log:
>
> hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
>
> Nothing else in the vicinity.

GDB on bare-metal (on a different machine, not the hypervisor) has
hardware watchpoint support, as expected.

Probably the next step is to figure out what software is running on the
hypervisor.

Thanks,
Florian

Re: Hardware watchpoints on GNU/Linux AArch64

[Andrew Pinski]

On Mon, Jan 2, 2023 at 9:39 AM Andrew Pinski <pinskia@gmail.com> wrote:
>
> On Mon, Jan 2, 2023 at 9:28 AM Andrew Pinski <pinskia@gmail.com> wrote:
> >
> > On Mon, Jan 2, 2023 at 8:49 AM Florian Weimer via Gdb
> > <gdb@sourceware.org> wrote:
> > >
> > > Running
> > >
> > > gdb-12.1-10.fc38.aarch64
> > > kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64
> > >
> > > I do not seem to be able to use hardware watchpoints.  GDB just logs
> > > warnings:
> > >
> > > warning: Unable to determine the number of hardware watchpoints available.
> > > warning: Unable to determine the number of hardware breakpoints available.
> >
> > This shows up when the ptrace either fails or ptrace returns an
> > unknown value for AARCH64_DEBUG_ARCH value.
> > In the boot log there should be a message saying how many HW
> > breakpoint and watchpoints there are and what version of the debug
> > arch is there.
> > The supported values for debug arch are:
> > #define AARCH64_DEBUG_ARCH_V8 0x6
> > #define AARCH64_DEBUG_ARCH_V8_1 0x7
> > #define AARCH64_DEBUG_ARCH_V8_2 0x8
> > #define AARCH64_DEBUG_ARCH_V8_4 0x9
> >
> > I don't know of any other values that can be reported, especially for
> > Neoverse-N1 (which should be 8_2). Neoverse-N2 is 8_4.
>
> The kernel fills in that field with ptrace via:
> u8 debug_monitors_arch(void)
> {
>         return cpuid_feature_extract_unsigned_field(read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1),
>                                                 ID_AA64DFR0_EL1_DebugVer_SHIFT);
> }
>
> So either kvm/qemu is setting that EL1 (system) register incorrectly
> or the KVM Invocation didn't turn on HW breakpoints. Or KVM does not
> support HW breakpoints yet.

Looking KVM sources (arch/arm64/kvm/sys_regs.c) inside the kernel we see:
                /* Limit debug to ARMv8.0 */
                val &= ~ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_DebugVer);
                val |=
FIELD_PREP(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_DebugVer), 6);
Which was added by 94893fc9ad8cd for 5.12 unless this changed post Linux 6.0 .
So a value of 6 should be good to go.

Thanks,
Andrew Pinski


>
> Thanks,
> Andrew
>
>
> >
> > Thanks,
> > Andrew Pinski
> >
> > >
> > > I see AArch64-specific code in GDB, so this is a bit of a surprise to
> > > me.
> > >
> > > The machine uses virtualization:
> > >
> > > Architecture:           aarch64
> > >   CPU op-mode(s):       32-bit, 64-bit
> > >   Byte Order:           Little Endian
> > > CPU(s):                 3
> > >   On-line CPU(s) list:  0-2
> > > Vendor ID:              ARM
> > >   BIOS Vendor ID:       QEMU
> > >   Model name:           Neoverse-N1
> > >     BIOS Model name:    virt-rhel8.6.0  CPU @ 2.0GHz
> > >     BIOS CPU family:    1
> > >     Model:              1
> > >     Thread(s) per core: 1
> > >     Core(s) per socket: 1
> > >     Socket(s):          3
> > >     Stepping:           r3p1
> > >     BogoMIPS:           50.00
> > >     Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp
> > >                          asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
> > >
> > > Is this actually expected to work?  Would we have more luck on a
> > > bare-metal system?
> > >
> > > Thanks,
> > > Florian
> > >

Re: Hardware watchpoints on GNU/Linux AArch64

[Luis Machado]

Hi,

On 1/2/23 16:48, Florian Weimer via Gdb wrote:
> Running
> 
> gdb-12.1-10.fc38.aarch64
> kernel-6.2.0-0.rc1.20221230gitbff687b3dad6.15.fc38.aarch64
> 
> I do not seem to be able to use hardware watchpoints.  GDB just logs
> warnings:
> 
> warning: Unable to determine the number of hardware watchpoints available.
> warning: Unable to determine the number of hardware breakpoints available.
> 
> I see AArch64-specific code in GDB, so this is a bit of a surprise to
> me.
> 
> The machine uses virtualization:
> 
> Architecture:           aarch64
>    CPU op-mode(s):       32-bit, 64-bit
>    Byte Order:           Little Endian
> CPU(s):                 3
>    On-line CPU(s) list:  0-2
> Vendor ID:              ARM
>    BIOS Vendor ID:       QEMU
>    Model name:           Neoverse-N1
>      BIOS Model name:    virt-rhel8.6.0  CPU @ 2.0GHz
>      BIOS CPU family:    1
>      Model:              1
>      Thread(s) per core: 1
>      Core(s) per socket: 1
>      Socket(s):          3
>      Stepping:           r3p1
>      BogoMIPS:           50.00
>      Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp
>                           asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
> 
> Is this actually expected to work?  Would we have more luck on a
> bare-metal system?
> 
> Thanks,
> Florian
> 

Apologies, I was out for a few days during end of year. Hardware watchpoints should work
on AArch64/Linux. They should also work on system QEMU (though there might be differences in behavior).

I'm not sure about KVM though.