How to use bts recording?

How to use bts recording?

[Simon Sobisch]

(gdb) record btrace bts
Could not enable branch tracing for Thread 0x7ffff7fe6740 (LWP 2464935): 
Failed to start recording: No such file or directory.

What does this message, raised by GDB 9.2, want to tell me?
Looks like GDB 11.1 is a bit better:

(gdb) record btrace bts
Could not enable branch tracing for Thread 0x7ffff7fe4740 (LWP 2756): 
Failed to start recording: Operation not supported

How to get that operation working?

... still on the road to understand GDBs recording...
Simon

RE: How to use bts recording?

[Metzger, Markus T]

[re-adding gdb, which got removed accidentally]

Hello Simon,

>using Intel(R) Xeon(R) Gold 6136 CPU on 4.18.0-80.el8.x86_64
>   and Intel(R) Xeon(R) E5-2697 v4    on 3.10.0-514.16.1.el7.x86_64

Both kernels should support BTS and both processors should support BTS as well as PT.  Given the choice, I'd always go with PT as the recording overhead is significantly lower and the trace format is much more compact giving you a significantly longer history.


>How can I check for BTS support?

Hardware support is enumerated by cpuid and published by the kernel in /proc/cpuinfo under flags (look for 'bts').  Same for PT (look for 'intel_pt').

Kernel support for BTS would be checked by trying to enable it via perf_event_open as GDB does.  I'm not aware of some kernel feature enumeration that one could use, instead.

For PT, kernel support could be checked by looking at /sys/bus/event_source/devices/intel_pt/type, which gives the perf event type to be used for PT.  BTS does not need that as it is using a fixed type.


>I've temporarily adjusted /proc/sys/kernel/perf_event_paranoid but even
>-1 makes no difference.

This is one common pitfall and that's why GDB is checking it and giving instructions how to fix it.  The ' No such file or directory ' error you listed below may be a result of GDB trying to access that file and not finding it.

One common reason for BTS not working is that someone on the system (including yourself) is using PT.  The kernel does not allow them to be used at the same time, even by different processes.

For PT, I'd ask you to try 'perf record -e intel_pt//u -- true' so see if PT is working in general on your system for your user.  For BTS, I don't know of another tool we could try.

The error message should come from diagnose_perf_event_open_fail(), which means that the perf_event_open syscall failed.  Could you check if perf is working at all on your system for your user?

Regards,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Re: How to use bts recording?

[Simon Sobisch]

Am 24.01.2022 um 10:27 schrieb Metzger, Markus T:
> Hello Simon,
> 
>> using Intel(R) Xeon(R) Gold 6136 CPU on 4.18.0-80.el8.x86_64
>>    and Intel(R) Xeon(R) E5-2697 v4   on 3.10.0-514.16.1.el7.x86_64
> 
> Both kernels should support BTS and both processors should support BTS as well as PT.  Given the choice, I'd always go with PT as the recording overhead is significantly lower and the trace format is much more compact giving you a significantly longer history.

PT has the downside of GDB having to be configured for that - which I 
guess isn't common in distro versions - and somehow a manual build n the 
4.x kernel where libipt was available also did not pick it up (will try 
to force it with a GDB 11.2 build soon, thanks for pointing out the 
configure flags; I've looked in the tarball in the ./configure file but 
that was in gdb/configure).

>> How can I check for BTS support?
> 
> Hardware support is enumerated by cpuid and published by the kernel in /proc/cpuinfo under flags (look for 'bts').  Same for PT (look for 'intel_pt').

That may be the reason that it wasn't picked up; those aren't in there:

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb 
rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable 
nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic 
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor 
lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb 
stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq 
rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt 
xsavec xsaves arat pku ospke flush_l1d arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm 
constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid 
sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c 
rdrand hypervisor lahf_lm abm 3dnowprefetch arat fsgsbase bmi1 hle avx2 
smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt


> Kernel support for BTS would be checked by trying to enable it via perf_event_open as GDB does.  I'm not aware of some kernel feature enumeration that one could use, instead.
> 
> For PT, kernel support could be checked by looking at /sys/bus/event_source/devices/intel_pt/type, which gives the perf event type to be used for PT.  BTS does not need that as it is using a fixed type.

There's no /sys/bus/event_source/devices/intel_pt/ either.

>> I've temporarily adjusted /proc/sys/kernel/perf_event_paranoid but even
>> -1 makes no difference.
> 
> This is one common pitfall and that's why GDB is checking it and giving instructions how to fix it.  The ' No such file or directory ' error you listed below may be a result of GDB trying to access that file and not finding it.

I guess this isn't the issue here , I can do
    cat proc/sys/kernel/perf_event_paranoid
without any problems (showing 1 or 2, when not temporarily adjusted).

> One common reason for BTS not working is that someone on the system (including yourself) is using PT.  The kernel does not allow them to be used at the same time, even by different processes.
> 
> For PT, I'd ask you to try 'perf record -e intel_pt//u -- true' so see if PT is working in general on your system for your user.  For BTS, I don't know of another tool we could try.

In both environment this raises

event syntax error:
    'intel_pt//u'
     \___ Cannot find PMU `intel_pt'. Missing kernel support?


I guess in this case there is no use in trying to build with intel-pt, 
is it?

Not sure how to enable intel_pt in the kernel either.

> The error message should come from diagnose_perf_event_open_fail(), which means that the perf_event_open syscall failed.  Could you check if perf is working at all on your system for your user?

Already checked that - works fine in general.


Thanks for trying to get this working, ideally we have some hints the 
next time someone looks out for this and can add a better error message 
for bts to GDB.

Simon

> 
> Regards,
> Markus.
> 
> Intel Deutschland GmbH
> Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
> Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
> Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva
> Chairperson of the Supervisory Board: Nicole Lau
> Registered Office: Munich
> Commercial Register: Amtsgericht Muenchen HRB 186928

RE: How to use bts recording?

[Metzger, Markus T]

Hello Simon,

>PT has the downside of GDB having to be configured for that - which I
>guess isn't common in distro versions - and somehow a manual build n the
>4.x kernel where libipt was available also did not pick it up (will try
>to force it with a GDB 11.2 build soon, thanks for pointing out the
>configure flags; I've looked in the tarball in the ./configure file but
>that was in gdb/configure).

Let's work on that, then.  Fedora configures GDB with PT support and I thought RHEL would too.  Not sure at which version they started.  The Debian GDB maintainer promised to enable it but I have not checked, since.


>>> How can I check for BTS support?
>>
>> Hardware support is enumerated by cpuid and published by the kernel in
>/proc/cpuinfo under flags (look for 'bts').  Same for PT (look for 'intel_pt').
>
>That may be the reason that it wasn't picked up; those aren't in there:
>
>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb
>rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable
>nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic
>movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
>lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb
>stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq
>rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt
>xsavec xsaves arat pku ospke flush_l1d arch_capabilities
>bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
>
>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
>constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid
>sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c
>rdrand hypervisor lahf_lm abm 3dnowprefetch arat fsgsbase bmi1 hle avx2
>smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt

Are you maybe using virtualization?


>> For PT, kernel support could be checked by looking at
>/sys/bus/event_source/devices/intel_pt/type, which gives the perf event type to
>be used for PT.  BTS does not need that as it is using a fixed type.
>
>There's no /sys/bus/event_source/devices/intel_pt/ either.

OK, so your system definitely does not support PT.  This also explains this error...

>event syntax error:
>    'intel_pt//u'
>     \___ Cannot find PMU `intel_pt'. Missing kernel support?
>
>
>I guess in this case there is no use in trying to build with intel-pt,
>is it?

Yes.  You should be able to build GDB with PT support but it wouldn't
work on that system.


>Thanks for trying to get this working, ideally we have some hints the
>next time someone looks out for this and can add a better error message
>for bts to GDB.

If this turns out to be something that GDB can check with reasonable effort,
diagnosing this would certainly help.  Like we do for perf_event_paranoid.

Regards,
Markus.
Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Re: How to use bts recording?

[Simon Sobisch]

Am 26.01.2022 um 13:41 schrieb Metzger, Markus T:
> Hello Simon,
> 
>> PT has the downside of GDB having to be configured for that - which I
>> guess isn't common in distro versions - and somehow a manual build n the
>> 4.x kernel where libipt was available also did not pick it up (will try
>> to force it with a GDB 11.2 build soon, thanks for pointing out the
>> configure flags; I've looked in the tarball in the ./configure file but
>> that was in gdb/configure).
> 
> Let's work on that, then.  Fedora configures GDB with PT support and I thought RHEL would too.  Not sure at which version they started.  The Debian GDB maintainer promised to enable it but I have not checked, since.

I've just rechecked current Debian, the gdb binary, which shows as
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
has a link to /usr/lib/x86_64-linux-gnu/libipt.so.2, so I guess it is in 
there.

Running there actually does show a different message

(gdb) record btrace
Could not enable branch tracing for Thread 0x7ffff79833c0 (LWP 334): BTS 
support has been disabled for the target cpu.
(gdb) record btrace pt
Could not enable branch tracing for Thread 0x7ffff79833c0 (LWP 334): 
Failed to open /sys/bus/event_source/devices/intel_pt/type: No such file 
or directory.

But this machine is an AMD one...

Both messages are very clear - it would be nice to get those with GDB 
11.x, too (not sure if they are produced by a Debian patch or by a 
different code path).


>>>> How can I check for BTS support?
>>>
>>> Hardware support is enumerated by cpuid and published by the kernel in
>> /proc/cpuinfo under flags (look for 'bts').  Same for PT (look for 'intel_pt').
>>
>> That may be the reason that it wasn't picked up; those aren't in there:
>>
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb
>> rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable
>> nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic
>> movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
>> lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb
>> stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq
>> rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt
>> xsavec xsaves arat pku ospke flush_l1d arch_capabilities
>> bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
>>
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
>> constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid
>> sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c
>> rdrand hypervisor lahf_lm abm 3dnowprefetch arat fsgsbase bmi1 hle avx2
>> smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt
> 
> Are you maybe using virtualization?

Rechecked with the server team: yes, the old RHEL kernel runs on Redhat 
Virtualization Manager; the newer one on VMware vSphere.

If someone knows how to enable Intel PT and/or BTS there I'd take a 
pointer and pass it to the server team.

> [...]
> 
> 
>> Thanks for trying to get this working, ideally we have some hints the
>> next time someone looks out for this and can add a better error message
>> for bts to GDB.
> 
> If this turns out to be something that GDB can check with reasonable effort,
> diagnosing this would certainly help.  Like we do for perf_event_paranoid.

I totally agree. The "only" missing parts are:

* How could GDB check this?
* Who applies this change, possibly directly after the bts error is 
recognized?

I'm available for testing a patched GDB 11.2 on both kernels :-)

> Regards,
> Markus.
> Intel Deutschland GmbH

Thanks again,
Simon

RE: How to use bts recording?

[Metzger, Markus T]

Hello Simon,

>> Are you maybe using virtualization?
>
>Rechecked with the server team: yes, the old RHEL kernel runs on Redhat
>Virtualization Manager; the newer one on VMware vSphere.

Here's the problem.  AFAIK neither BTS nor PT are virtualized.  This is simply
missing enabling in hypervisors.  They do properly indicate this by omitting
the respective cpuid enumeration.  This results in an EOPNOTSUPP when
GDB tries to enable it, which is printed by GDB.

From that perspective, the error message was all correct.  What's missing is
why it is not supported.  All that GDB knows is that the OS told it that this is
not supported.

I'm afraid that's a fairly common problem, these days, but I don't see how
GDB could check that it is running in a guest OS and that the hypervisor did
not virtualize PT or BTS.

GDB could grep for the respective flags in /proc/cpuinfo but it would really
only confirm what the OS already told it: it is not supported.

Regards,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Re: How to use bts recording?

[Simon Sobisch]

Am 26.01.2022 um 18:11 schrieb Metzger, Markus T:
> Hello Simon,
> 
>>> Are you maybe using virtualization?
>>
>> Rechecked with the server team: yes, the old RHEL kernel runs on Redhat
>> Virtualization Manager; the newer one on VMware vSphere.
> 
> Here's the problem.  AFAIK neither BTS nor PT are virtualized.  This is simply
> missing enabling in hypervisors.  They do properly indicate this by omitting
> the respective cpuid enumeration.  This results in an EOPNOTSUPP when
> GDB tries to enable it, which is printed by GDB.

I see - but something is still not ideal.
At least on this machine GDB 11.1 said:

(gdb) record btrace bts
Could not enable branch tracing for Thread 0x7ffff7fe4740 (LWP 2756): 
**Failed to start recording: Operation not supported**

While the Debian GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git reports

Could not enable branch tracing for Thread 0x7ffff79833c0 (LWP 334): 
**BTS support has been disabled for the target cpu**

That's a clear message, I like that.

Is this a special Debian patch or is because of the target cpu being AMD?
Is there a path that should be taken when the OS reports EOPNOTSUPP?

A message similar to the one above would be useful.

> I'm afraid that's a fairly common problem, these days, but I don't see how
> GDB could check that it is running in a guest OS and that the hypervisor did
> not virtualize PT or BTS.

As you've said: use of virtualized environments is quite common, so it 
may be reasonable to adjust the docs to at least hint at that issue.

I suggest changing

- Hardware-supported instruction recording, supported on Intel processors
+ Hardware-supported instruction recording, supported on Intel 
processors if not running in a virtual machine.

... or something like that.


BTW: rechecked newer RHEL:
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-12.el8 also has the link to 
libipt and so also says
Failed to open /sys/bus/event_source/devices/intel_pt/type

[note: that message possibly could hint at "Intel PT not available"]

Also: not sure if libipt is the one that is checking that directory, but 
possibly GDB could do this _before_ printing
 > Intel Processor Trace support was disabled at compile time.

... because this led me to believe "just reconfigure and possible 
install libipt before and everything will be fine" - but if I had done 
this the only result would have been another error message.

RE: How to use bts recording?

[Metzger, Markus T]

Hello Simon,

>I see - but something is still not ideal.
>At least on this machine GDB 11.1 said:
>
>(gdb) record btrace bts
>Could not enable branch tracing for Thread 0x7ffff7fe4740 (LWP 2756):
>**Failed to start recording: Operation not supported**
>
>While the Debian GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git reports
>
>Could not enable branch tracing for Thread 0x7ffff79833c0 (LWP 334):
>**BTS support has been disabled for the target cpu**
>
>That's a clear message, I like that.

The first message comes from failing to enable BTS and from printing errno.

The second message comes from cpu_supports_bts() returning false for AMD.
We have not even tried enabling it.


>> I'm afraid that's a fairly common problem, these days, but I don't see how
>> GDB could check that it is running in a guest OS and that the hypervisor did
>> not virtualize PT or BTS.
>
>As you've said: use of virtualized environments is quite common, so it
>may be reasonable to adjust the docs to at least hint at that issue.
>
>I suggest changing
>
>- Hardware-supported instruction recording, supported on Intel processors
>+ Hardware-supported instruction recording, supported on Intel
>processors if not running in a virtual machine.

That's a feature of the hypervisor, not of GDB.  Hypervisors that do virtualize BTS
or PT won't be very happy with such wording inside the GDB manual.


>Failed to open /sys/bus/event_source/devices/intel_pt/type
>
>[note: that message possibly could hint at "Intel PT not available"]

We can probably turn the file open failure into something like "the system does
not support PT".  I agree that this would be a lot more helpful.


>Also: not sure if libipt is the one that is checking that directory, but
>possibly GDB could do this _before_ printing
> > Intel Processor Trace support was disabled at compile time.
>
>... because this led me to believe "just reconfigure and possible
>install libipt before and everything will be fine" - but if I had done
>this the only result would have been another error message.

This would be another check we could do.  Right now, we don't even try since
GDB wouldn't be able to decode the trace, anyway.

Which error message is more helpful is probably subjective.  If you went through
the effort of setting up a bare metal system only to find out that your version of
GDB doesn't support PT, anyway, the other order would be more useful.

Given that most distros meanwhile build GDB with PT support, I'd keep the current
order of errors.

Regards,
Markus.
Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928