public inbox for systemtap@sourceware.org
 help / color / mirror / Atom feed
* [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
@ 2024-03-11 13:54 wcohen at redhat dot com
  2024-03-11 13:57 ` [Bug runtime/31472] " fche at redhat dot com
                   ` (5 more replies)
  0 siblings, 6 replies; 7+ messages in thread
From: wcohen at redhat dot com @ 2024-03-11 13:54 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

            Bug ID: 31472
           Summary: systemtap.base/tracepoints.stp causes reboot with
                    RHEL9 linux 5.14.0-428.el9.x86_64
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: wcohen at redhat dot com
  Target Milestone: ---

When running "make installcheck" on x86_64 RHEL9 with a very recently built
kernel, linux 5.14.0-428.el9.x86_64, the machine will reboot.  This can be
demonstrated with the following steps:

$ cd systemtap_write/systemtap/
$ ../install/bin/stap -p4 -k -mtracepoints -v
testsuite/systemtap.base/tracepoints.stp
Pass 1: parsed user script and 503 library scripts using
204124virt/175760res/15488shr/160016data kb, in 290usr/60sys/461real ms.
Pass 2: analyzed script: 2174 probes, 1 function, 0 embeds, 1 global using
233300virt/208684res/18960shr/189192data kb, in 81560usr/10520sys/94518real ms.
Pass 3: translated to C into "/tmp/stapoffzQT/tracepoints_src.c" using
233300virt/208684res/18960shr/189192data kb, in 190usr/20sys/226real ms.
tracepoints.ko
Pass 4: compiled C into "tracepoints.ko" in 177710usr/9790sys/190275real ms.
Keeping temporary directory "/tmp/stapoffzQT"
$ cp -a /tmp/stapoffzQT .
$ ../install/bin/staprun tracepoints.ko
[ 6315.911726] jump_label: Fatal kernel bug, unexpected op at
__access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints] [0000000071a5b3f1]
(e9 eb 00 00 00 != 0f 1f 44 00 00)) size:5 type:1
[ 6315.911834] ------------[ cut here ]------------
[ 6315.911836] kernel BUG at arch/x86/kernel/jump_label.c:73!
[ 6315.911844] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 6315.912830] CPU: 0 PID: 13174 Comm: stapio Kdump: loaded Tainted: G        W
 OE     -------  ---  5.14.0-428.el9.x86_64 #1
[ 6315.912830] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.16.3-1.fc39 04/01/2014
[ 6315.912830] RIP: 0010:__jump_label_patch.cold+0x24/0x26
[ 6315.912830] Code: ba e9 6a ff 40 ff 49 c7 c7 c8 95 83 bb 41 56 45 89 e1 4d
89 f8 4c 89 e9 4c 89 ea 4c 89 ee 48 c7 c7 58 a8 30 ba e8 ad d2 00 00 <0f> 0b 53
48 c7 c6 58 b2 30 ba 48 89 fb 48 8d bf c0 00 00 00 e8 9a
[ 6315.912830] RSP: 0018:ffffa7ba03657b50 EFLAGS: 00010246
[ 6315.912830] RAX: 00000000000000ad RBX: 00000000000000eb RCX:
0000000000000000
[ 6315.912830] RDX: 0000000000000000 RSI: ffff897b3bc20840 RDI:
ffff897b3bc20840
[ 6315.912830] RBP: ffffa7ba03657b88 R08: 80000000ffff8388 R09:
0000000000ffff0a
[ 6315.912830] R10: 0000000000000000 R11: 000000000000000f R12:
0000000000000005
[ 6315.912830] R13: ffffffffc0f7339d R14: 0000000000000001 R15:
ffffffffb9e0a26a
[ 6315.912830] FS:  00007f02e5506000(0000) GS:ffff897b3bc00000(0000)
knlGS:0000000000000000
[ 6315.912830] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6315.912830] CR2: 00007f0ab50f6030 CR3: 00000000011b2005 CR4:
0000000000370ef0
[ 6315.912830] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 6315.912830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 6315.912830] Call Trace:
[ 6315.912830]  <TASK>
[ 6315.912830]  ? show_trace_log_lvl+0x1c4/0x2df
[ 6315.912830]  ? show_trace_log_lvl+0x1c4/0x2df
[ 6315.912830]  ? arch_jump_label_transform_queue+0x34/0x80
[ 6315.912830]  ? __die_body.cold+0x8/0xd
[ 6315.912830]  ? die+0x2b/0x50
[ 6315.912830]  ? do_trap+0xce/0x120
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  ? do_error_trap+0x65/0x80
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  ? exc_invalid_op+0x4e/0x70
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  ? asm_exc_invalid_op+0x16/0x20
[ 6315.912830]  ? __access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints]
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  arch_jump_label_transform_queue+0x34/0x80
[ 6315.912830]  __jump_label_update+0x6a/0x120
[ 6315.912830]  jump_label_update+0x98/0xf0
[ 6315.912830]  static_key_enable_cpuslocked+0x5f/0x90
[ 6315.912830]  static_key_enable+0x16/0x20
[ 6315.912830]  tracepoint_add_func+0x2e5/0x3f0
[ 6315.912830]  ? unregister_tracepoint_probe_2100+0x30/0x30 [tracepoints]
[ 6315.912830]  tracepoint_probe_register+0x58/0x90
[ 6315.912830]  ? unregister_tracepoint_probe_2100+0x30/0x30 [tracepoints]
[ 6315.912830]  ? unregister_tracepoint_probe_2100+0x30/0x30 [tracepoints]
[ 6315.912830]  stp_tracepoint_probe_register+0x127/0x1a0 [tracepoints]
[ 6315.912830]  ? systemtap_module_init.cold+0xea/0x190 [tracepoints]
[ 6315.912830]  ? _stp_ctl_write_cmd+0x2b4/0x440 [tracepoints]
[ 6315.912830]  ? proc_reg_write+0x53/0xa0
[ 6315.912830]  ? vfs_write+0xe4/0x410
[ 6315.912830]  ? __fget_light+0x9f/0x130
[ 6315.912830]  ? ksys_write+0x5f/0xe0
[ 6315.912830]  ? do_syscall_64+0x59/0x90
[ 6315.912830]  ? sigprocmask+0xb4/0xe0
[ 6315.912830]  ? sigprocmask+0xb4/0xe0
[ 6315.912830]  ? syscall_exit_work+0x103/0x130
[ 6315.912830]  ? syscall_exit_to_user_mode+0x22/0x40
[ 6315.912830]  ? do_syscall_64+0x69/0x90
[ 6315.912830]  ? exc_page_fault+0x62/0x150
[ 6315.912830]  ? entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 6315.912830]  </TASK>
[ 6315.912830] Modules linked in: tracepoints(OE) uinput snd_seq_dummy
snd_hrtimer nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink
qrtr sunrpc intel_rapl_msr intel_rapl_common intel_pmc_core intel_vsec
pmt_telemetry pmt_class kvm_intel snd_hda_codec_generic ledtrig_audio
snd_hda_intel kvm snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec
snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm irqbypass rapl snd_timer
iTCO_wdt snd iTCO_vendor_support i2c_i801 pcspkr lpc_ich virtio_balloon
i2c_smbus soundcore joydev xfs libcrc32c qxl drm_ttm_helper ttm drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ahci libahci drm libata
virtio_net crct10dif_pclmul crc32_pclmul crc32c_intel virtio_scsi virtio_blk
ghash_clmulni_intel virtio_console net_failover failover serio_raw dm_mirror
dm_region_hash dm_log dm_mod fuse


This does not happen with the RHEL9 linux 5.14.0-427.el9.x86_64 as seen below:

$ cd systemtap_write/systemtap/
$ uname -r
5.14.0-427.el9.x86_64
$  ../install/bin/stap -p4 -k -mtracepointsx -v
testsuite/systemtap.base/tracepoints.stp
Pass 1: parsed user script and 503 library scripts using
204116virt/175744res/15488shr/160008data kb, in 250usr/30sys/290real ms.
Pass 2: analyzed script: 22 probes, 1 function, 0 embeds, 1 global using
209860virt/185180res/18828shr/165752data kb, in 40usr/70sys/115real ms.
Pass 3: translated to C into "/tmp/stapAKS5lu/tracepointsx_src.c" using
209860virt/185180res/18828shr/165752data kb, in 10usr/0sys/10real ms.
tracepointsx.ko
Pass 4: compiled C into "tracepointsx.ko" in 16100usr/2020sys/18336real ms.
Keeping temporary directory "/tmp/stapAKS5lu"
$ cp -a /tmp/stapAKS5lu
$  ../install/bin/staprun tracepointsx.ko
tracepoints OK
$

Looking at what is creating the error message:

[ 6315.911726] jump_label: Fatal kernel bug, unexpected op at
__access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints] [0000000071a5b3f1]
(e9 eb 00 00 00 != 0f 1f 44 00 00)) size:5 type:1

It appears to be coming from:

https://elixir.bootlin.com/linux/v5.14/source/arch/x86/kernel/jump_label.c#L71

Next step is to look to see which specific tracepoint(s) is/are triggering
this.  I have the temp stap build files save so can look over those.

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [Bug runtime/31472] systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
  2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
@ 2024-03-11 13:57 ` fche at redhat dot com
  2024-03-12 16:13 ` wcohen at redhat dot com
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: fche at redhat dot com @ 2024-03-11 13:57 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
worth also comparing: objtool changes; whether tracepoint call sites are
differently marked up by the inline-asm / kbuild system than before

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [Bug runtime/31472] systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
  2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
  2024-03-11 13:57 ` [Bug runtime/31472] " fche at redhat dot com
@ 2024-03-12 16:13 ` wcohen at redhat dot com
  2024-03-12 18:16 ` wcohen at redhat dot com
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: wcohen at redhat dot com @ 2024-03-12 16:13 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #2 from William Cohen <wcohen at redhat dot com> ---
The following message appears to be saying it is attempting to put in a jump
patch on the tracepoint module which is very odd (it should be trying to
instrument itself):

[ 6315.911726] jump_label: Fatal kernel bug, unexpected op at
__access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints] [0000000071a5b3f1]
(e9 eb 00 00 00 != 0f 1f 44 00 00)) size:5 type:1

The kernel checks to determine whether the location is a valid location for a
jump patch if it doesn't match the expected form the following line prints out
the message followed by a BUG() which cause the reboot:

https://elixir.bootlin.com/linux/v5.14/source/arch/x86/kernel/jump_label.c#L71

There seems to be something that is triggering this due to the number of
tracepoints being instrumented.  The RHEL 427 kernel only list out 21
tracepoints which the 428 kernel that has a problem with the tracepoints.stp
test has 2173 tracepoints listed.  Playing around with the list of tracepoints
in all_tracepoint to see see if can get a smaller set tracepoints that trigger
the reboot.

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [Bug runtime/31472] systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
  2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
  2024-03-11 13:57 ` [Bug runtime/31472] " fche at redhat dot com
  2024-03-12 16:13 ` wcohen at redhat dot com
@ 2024-03-12 18:16 ` wcohen at redhat dot com
  2024-03-12 18:29 ` wcohen at redhat dot com
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: wcohen at redhat dot com @ 2024-03-12 18:16 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

--- Comment #3 from William Cohen <wcohen at redhat dot com> ---
Created attachment 15400
  --> https://sourceware.org/bugzilla/attachment.cgi?id=15400&action=edit
smaller reproducer using just mmap_lock:* tracepoints for the reboot

Took a look at the difference between the RHEL9 427 and 428 kernel tracepoints.
 Suspected there was a set of tracepoints that were triggering the reboot in
the RHEL9 428 kernel.  With a pared down tracepoints.stp check which wildcard
tracepoints were causing the problem.  The following three tracepoints are
causing the reboot on RHEL9 428 kernel:

kernel.trace("mmap_lock:mmap_lock_acquire_returned")
kernel.trace("mmap_lock:mmap_lock_released")
kernel.trace("mmap_lock:mmap_lock_start_locking")

Check f39 and fedora rawhide and they both have those tracepoints. Attempted to
fedora 39 kernel-6.7.9-200.fc39.x86_64 and the reproducer ran with out issue.
The reproduce also ran without issue on rawhide with
kernel-6.8.0-0.rc7.20240308git3aaa8ce7a335.60.fc41.x86_64

Looks like the next step is to compare the difference between RHEL9 and
f39/rawhide.

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [Bug runtime/31472] systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
  2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
                   ` (2 preceding siblings ...)
  2024-03-12 18:16 ` wcohen at redhat dot com
@ 2024-03-12 18:29 ` wcohen at redhat dot com
  2024-03-14 18:51 ` wcohen at redhat dot com
  2024-03-18 21:06 ` wcohen at redhat dot com
  5 siblings, 0 replies; 7+ messages in thread
From: wcohen at redhat dot com @ 2024-03-12 18:29 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

--- Comment #4 from William Cohen <wcohen at redhat dot com> ---
Traceback from the scaled down reproducer:


$ ../install/bin/stap -k -p4 -v -mpr31472 pr31472_mmap_lock.stp 
Pass 1: parsed user script and 503 library scripts using
204124virt/175892res/15616shr/160016data kb, in 270usr/30sys/309real ms.
Pass 2: analyzed script: 4 probes, 1 function, 0 embeds, 0 globals using
211220virt/186660res/18948shr/167112data kb, in 100usr/110sys/210real ms.
Pass 3: translated to C into "/tmp/stap0VnT14/pr31472_src.c" using
211220virt/186660res/18948shr/167112data kb, in 0usr/0sys/0real ms.
pr31472.ko
Pass 4: compiled C into "pr31472.ko" in 2210usr/460sys/2709real ms.
Keeping temporary directory "/tmp/stap0VnT14"
$ sudo ../install/bin/staprun pr31472.ko 
[sudo] password for wcohen: 
[  224.637382] pr31472 (pr31472_mmap_lock.stp): systemtap: 5.1/0.190, base:
ffffffffc0cd0000, memory: 40data/28text/10ctx/16486net/121alloc kb, probes: 4
[  224.637402] jump_label: Fatal kernel bug, unexpected op at
__access_process_vm_.constprop.0+0x3d/0x1f0 [pr31472] [00000000c7948df7] (e9 eb
00 00 00 != 0f 1f 44 00 00)) size:5 type:1
[  224.637415] ------------[ cut here ]------------
[  224.637416] kernel BUG at arch/x86/kernel/jump_label.c:73!
[  224.637423] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[  224.638345] CPU: 0 PID: 2304 Comm: stapio Kdump: loaded Tainted: G          
OE     -------  ---  5.14.0-428.el9.x86_64 #1
[  224.638345] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.16.3-1.fc39 04/01/2014
[  224.638345] RIP: 0010:__jump_label_patch.cold+0x24/0x26
[  224.638345] Code: ba e9 6a ff 40 ff 49 c7 c7 c8 95 a3 bb 41 56 45 89 e1 4d
89 f8 4c 89 e9 4c 89 ea 4c 89 ee 48 c7 c7 58 a8 50 ba e8 ad d2 00 00 <0f> 0b 53
48 c7 c6 58 b2 50 ba 48 89 fb 48 8d bf c0 00 00 00 e8 9a
[  224.638345] RSP: 0018:ffffb2c78121fb38 EFLAGS: 00010246
[  224.638345] RAX: 00000000000000a9 RBX: 00000000000000eb RCX:
0000000000000027
[  224.638345] RDX: 0000000000000000 RSI: ffffffffbac67a20 RDI:
ffff9edc7bc20848
[  224.638345] RBP: ffffb2c78121fb70 R08: 80000000ffff8328 R09:
0000000000ffff0a
[  224.638345] R10: 0000000000000000 R11: 000000000000000f R12:
0000000000000005
[  224.638345] R13: ffffffffc0cd119d R14: 0000000000000001 R15:
ffffffffba00a26a
[  224.638345] FS:  00007fba1baf1000(0000) GS:ffff9edc7bc00000(0000)
knlGS:0000000000000000
[  224.638345] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  224.638345] CR2: 00007ffe20128084 CR3: 0000000102f4c006 CR4:
0000000000370ef0
[  224.638345] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  224.638345] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  224.638345] Call Trace:
[  224.638345]  <TASK>
[  224.638345]  ? show_trace_log_lvl+0x1c4/0x2df
[  224.638345]  ? show_trace_log_lvl+0x1c4/0x2df
[  224.638345]  ? arch_jump_label_transform_queue+0x34/0x80
[  224.638345]  ? __die_body.cold+0x8/0xd
[  224.638345]  ? die+0x2b/0x50
[  224.638345]  ? do_trap+0xce/0x120
[  224.638345]  ? __jump_label_patch.cold+0x24/0x26
[  224.638345]  ? do_error_trap+0x65/0x80
[  224.638345]  ? __jump_label_patch.cold+0x24/0x26
[  224.638345]  ? exc_invalid_op+0x4e/0x70
[  224.638345]  ? __jump_label_patch.cold+0x24/0x26
[  224.638345]  ? asm_exc_invalid_op+0x16/0x20
[  224.638345]  ? __access_process_vm_.constprop.0+0x3d/0x1f0 [pr31472]
[  224.638345]  ? __jump_label_patch.cold+0x24/0x26
[  224.638345]  ? __jump_label_patch.cold+0x24/0x26
[  224.638345]  arch_jump_label_transform_queue+0x34/0x80
[  224.638345]  __jump_label_update+0x6a/0x120
[  224.638345]  jump_label_update+0x98/0xf0
[  224.638345]  static_key_enable_cpuslocked+0x5f/0x90
[  224.638345]  static_key_enable+0x16/0x20
[  224.638345]  tracepoint_add_func+0x2e5/0x3f0
[  224.638345]  ? contexts+0xffffffffc0c99f98/0xffffffffc0c99f98 [pr31472]
[  224.638345]  tracepoint_probe_register+0x58/0x90
[  224.638345]  ? contexts+0xffffffffc0c99f98/0xffffffffc0c99f98 [pr31472]
[  224.638345]  ? contexts+0xffffffffc0c99f98/0xffffffffc0c99f98 [pr31472]
[  224.638345]  stp_tracepoint_probe_register+0x127/0x1a0 [pr31472]
[  224.638345]  ? systemtap_module_init.cold+0xc8/0x284 [pr31472]
[  224.638345]  ? _stp_ctl_write_cmd+0x2b4/0x440 [pr31472]
[  224.638345]  ? proc_reg_write+0x53/0xa0
[  224.638345]  ? vfs_write+0xe4/0x410
[  224.638345]  ? __fget_light+0x9f/0x130
[  224.638345]  ? ksys_write+0x5f/0xe0
[  224.638345]  ? do_syscall_64+0x59/0x90
[  224.638345]  ? syscall_exit_to_user_mode+0x22/0x40
[  224.638345]  ? do_syscall_64+0x69/0x90
[  224.638345]  ? do_syscall_64+0x69/0x90
[  224.638345]  ? syscall_exit_work+0x103/0x130
[  224.638345]  ? syscall_exit_to_user_mode+0x22/0x40
[  224.638345]  ? do_syscall_64+0x69/0x90
[  224.638345]  ? do_user_addr_fault+0x1d6/0x6a0
[  224.638345]  ? exc_page_fault+0x62/0x150
[  224.638345]  ? entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  224.638345]  </TASK>
[  224.638345] Modules linked in: pr31472(OE) uinput snd_seq_dummy snd_hrtimer
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr sunrpc
intel_rapl_msr intel_rapl_common intel_pmc_core intel_vsec pmt_telemetry
pmt_class kvm_intel snd_hda_codec_generic ledtrig_audio snd_hda_intel kvm
snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep
snd_seq snd_seq_device snd_pcm irqbypass rapl snd_timer iTCO_wdt
iTCO_vendor_support snd pcspkr i2c_i801 virtio_balloon lpc_ich soundcore joydev
i2c_smbus xfs libcrc32c qxl drm_ttm_helper ttm drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ahci drm libahci libata crct10dif_pclmul
crc32_pclmul crc32c_intel virtio_net virtio_scsi virtio_blk ghash_clmulni_intel
virtio_console net_failover failover serio_raw dm_mirror dm_region_hash dm_log
dm_mod fuse

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [Bug runtime/31472] systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
  2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
                   ` (3 preceding siblings ...)
  2024-03-12 18:29 ` wcohen at redhat dot com
@ 2024-03-14 18:51 ` wcohen at redhat dot com
  2024-03-18 21:06 ` wcohen at redhat dot com
  5 siblings, 0 replies; 7+ messages in thread
From: wcohen at redhat dot com @ 2024-03-14 18:51 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

--- Comment #5 from William Cohen <wcohen at redhat dot com> ---
Did a git bisect running against the RHEL9 428 kernel and verified that the
problem on RHEL9 is caused by:

commit 155c689b2a75dcb217a1c52886c04982f4c169f2
Author: William Cohen <wcohen@redhat.com>
Date:   Thu Mar 7 13:44:06 2024 -0500

    PR30716: Turn off objtool warnings on systemtap instrumentation modules

    The previous approaches to turning off the objtool warnings did not
    work for x86_64 RHEL9.  The systemtap generated code is not on the
    whitelist to use certain kernel functions. The additional objtool
    warning output mentioning the systemtap code using those functions
    with UACCESS enabled caused a number of the tests in the testsuite to
    fail.  The generated Makefile now includes a line to turn off running
    objtool on the systemtap generated module and eliminates those
    warnings.

There are additional sections on generated .ko module when objtool is run.  The
missing orc_unwind explains why the backtrace is looking a bit odd. Below shows
what sections the bad .ko is missing:

 diff -u0 a b
--- a   2024-03-13 15:26:23.637081062 -0400
+++ b   2024-03-13 15:25:58.343950133 -0400
@@ -7 +6,0 @@
-.call_sites
@@ -21,2 +19,0 @@
-.orc_unwind
-.orc_unwind_ip
@@ -27 +23,0 @@
-.rela.call_sites
@@ -32 +27,0 @@
-.rela.orc_unwind_ip
@@ -36,2 +30,0 @@
-.rela.retpoline_sites
-.rela.return_sites
@@ -40 +32,0 @@
-.rela.static_call_sites
@@ -43,2 +34,0 @@
-.retpoline_sites
-.return_sites
@@ -51 +40,0 @@
-.static_call_sites

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [Bug runtime/31472] systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64
  2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
                   ` (4 preceding siblings ...)
  2024-03-14 18:51 ` wcohen at redhat dot com
@ 2024-03-18 21:06 ` wcohen at redhat dot com
  5 siblings, 0 replies; 7+ messages in thread
From: wcohen at redhat dot com @ 2024-03-18 21:06 UTC (permalink / raw)
  To: systemtap

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #6 from William Cohen <wcohen at redhat dot com> ---
Fixed by:

commit deeee3e3667674a00fed6c42b8f028465271eab5
Author: William Cohen <wcohen@redhat.com>
Date:   Thu Mar 14 15:27:17 2024 -0400

    Revert "PR30716: Turn off objtool warnings on systemtap instrumentation
modules"

    This reverts commit 155c689b2a75dcb217a1c52886c04982f4c169f2.

    There are other things that objtool is doing in addition to checking
    user accesses and disabling objtool with newer RHEL9
    5.14.0-428.el9.x86_64 causes the system to reboot when setting up some
    tracepoint probes (PR30472).

-- 
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2024-03-18 21:06 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-11 13:54 [Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64 wcohen at redhat dot com
2024-03-11 13:57 ` [Bug runtime/31472] " fche at redhat dot com
2024-03-12 16:13 ` wcohen at redhat dot com
2024-03-12 18:16 ` wcohen at redhat dot com
2024-03-12 18:29 ` wcohen at redhat dot com
2024-03-14 18:51 ` wcohen at redhat dot com
2024-03-18 21:06 ` wcohen at redhat dot com

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).