[Bug runtime/31472] New: systemtap.base/tracepoints.stp causes reboot with RHEL9 linux 5.14.0-428.el9.x86_64

["wcohen at redhat dot com" <sourceware-bugzilla@sourceware.org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=31472

            Bug ID: 31472
           Summary: systemtap.base/tracepoints.stp causes reboot with
                    RHEL9 linux 5.14.0-428.el9.x86_64
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: wcohen at redhat dot com
  Target Milestone: ---

When running "make installcheck" on x86_64 RHEL9 with a very recently built
kernel, linux 5.14.0-428.el9.x86_64, the machine will reboot.  This can be
demonstrated with the following steps:

$ cd systemtap_write/systemtap/
$ ../install/bin/stap -p4 -k -mtracepoints -v
testsuite/systemtap.base/tracepoints.stp
Pass 1: parsed user script and 503 library scripts using
204124virt/175760res/15488shr/160016data kb, in 290usr/60sys/461real ms.
Pass 2: analyzed script: 2174 probes, 1 function, 0 embeds, 1 global using
233300virt/208684res/18960shr/189192data kb, in 81560usr/10520sys/94518real ms.
Pass 3: translated to C into "/tmp/stapoffzQT/tracepoints_src.c" using
233300virt/208684res/18960shr/189192data kb, in 190usr/20sys/226real ms.
tracepoints.ko
Pass 4: compiled C into "tracepoints.ko" in 177710usr/9790sys/190275real ms.
Keeping temporary directory "/tmp/stapoffzQT"
$ cp -a /tmp/stapoffzQT .
$ ../install/bin/staprun tracepoints.ko
[ 6315.911726] jump_label: Fatal kernel bug, unexpected op at
__access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints] [0000000071a5b3f1]
(e9 eb 00 00 00 != 0f 1f 44 00 00)) size:5 type:1
[ 6315.911834] ------------[ cut here ]------------
[ 6315.911836] kernel BUG at arch/x86/kernel/jump_label.c:73!
[ 6315.911844] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 6315.912830] CPU: 0 PID: 13174 Comm: stapio Kdump: loaded Tainted: G        W
 OE     -------  ---  5.14.0-428.el9.x86_64 #1
[ 6315.912830] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.16.3-1.fc39 04/01/2014
[ 6315.912830] RIP: 0010:__jump_label_patch.cold+0x24/0x26
[ 6315.912830] Code: ba e9 6a ff 40 ff 49 c7 c7 c8 95 83 bb 41 56 45 89 e1 4d
89 f8 4c 89 e9 4c 89 ea 4c 89 ee 48 c7 c7 58 a8 30 ba e8 ad d2 00 00 <0f> 0b 53
48 c7 c6 58 b2 30 ba 48 89 fb 48 8d bf c0 00 00 00 e8 9a
[ 6315.912830] RSP: 0018:ffffa7ba03657b50 EFLAGS: 00010246
[ 6315.912830] RAX: 00000000000000ad RBX: 00000000000000eb RCX:
0000000000000000
[ 6315.912830] RDX: 0000000000000000 RSI: ffff897b3bc20840 RDI:
ffff897b3bc20840
[ 6315.912830] RBP: ffffa7ba03657b88 R08: 80000000ffff8388 R09:
0000000000ffff0a
[ 6315.912830] R10: 0000000000000000 R11: 000000000000000f R12:
0000000000000005
[ 6315.912830] R13: ffffffffc0f7339d R14: 0000000000000001 R15:
ffffffffb9e0a26a
[ 6315.912830] FS:  00007f02e5506000(0000) GS:ffff897b3bc00000(0000)
knlGS:0000000000000000
[ 6315.912830] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6315.912830] CR2: 00007f0ab50f6030 CR3: 00000000011b2005 CR4:
0000000000370ef0
[ 6315.912830] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 6315.912830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 6315.912830] Call Trace:
[ 6315.912830]  <TASK>
[ 6315.912830]  ? show_trace_log_lvl+0x1c4/0x2df
[ 6315.912830]  ? show_trace_log_lvl+0x1c4/0x2df
[ 6315.912830]  ? arch_jump_label_transform_queue+0x34/0x80
[ 6315.912830]  ? __die_body.cold+0x8/0xd
[ 6315.912830]  ? die+0x2b/0x50
[ 6315.912830]  ? do_trap+0xce/0x120
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  ? do_error_trap+0x65/0x80
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  ? exc_invalid_op+0x4e/0x70
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  ? asm_exc_invalid_op+0x16/0x20
[ 6315.912830]  ? __access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints]
[ 6315.912830]  ? __jump_label_patch.cold+0x24/0x26
[ 6315.912830]  arch_jump_label_transform_queue+0x34/0x80
[ 6315.912830]  __jump_label_update+0x6a/0x120
[ 6315.912830]  jump_label_update+0x98/0xf0
[ 6315.912830]  static_key_enable_cpuslocked+0x5f/0x90
[ 6315.912830]  static_key_enable+0x16/0x20
[ 6315.912830]  tracepoint_add_func+0x2e5/0x3f0
[ 6315.912830]  ? unregister_tracepoint_probe_2100+0x30/0x30 [tracepoints]
[ 6315.912830]  tracepoint_probe_register+0x58/0x90
[ 6315.912830]  ? unregister_tracepoint_probe_2100+0x30/0x30 [tracepoints]
[ 6315.912830]  ? unregister_tracepoint_probe_2100+0x30/0x30 [tracepoints]
[ 6315.912830]  stp_tracepoint_probe_register+0x127/0x1a0 [tracepoints]
[ 6315.912830]  ? systemtap_module_init.cold+0xea/0x190 [tracepoints]
[ 6315.912830]  ? _stp_ctl_write_cmd+0x2b4/0x440 [tracepoints]
[ 6315.912830]  ? proc_reg_write+0x53/0xa0
[ 6315.912830]  ? vfs_write+0xe4/0x410
[ 6315.912830]  ? __fget_light+0x9f/0x130
[ 6315.912830]  ? ksys_write+0x5f/0xe0
[ 6315.912830]  ? do_syscall_64+0x59/0x90
[ 6315.912830]  ? sigprocmask+0xb4/0xe0
[ 6315.912830]  ? sigprocmask+0xb4/0xe0
[ 6315.912830]  ? syscall_exit_work+0x103/0x130
[ 6315.912830]  ? syscall_exit_to_user_mode+0x22/0x40
[ 6315.912830]  ? do_syscall_64+0x69/0x90
[ 6315.912830]  ? exc_page_fault+0x62/0x150
[ 6315.912830]  ? entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 6315.912830]  </TASK>
[ 6315.912830] Modules linked in: tracepoints(OE) uinput snd_seq_dummy
snd_hrtimer nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink
qrtr sunrpc intel_rapl_msr intel_rapl_common intel_pmc_core intel_vsec
pmt_telemetry pmt_class kvm_intel snd_hda_codec_generic ledtrig_audio
snd_hda_intel kvm snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec
snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm irqbypass rapl snd_timer
iTCO_wdt snd iTCO_vendor_support i2c_i801 pcspkr lpc_ich virtio_balloon
i2c_smbus soundcore joydev xfs libcrc32c qxl drm_ttm_helper ttm drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ahci libahci drm libata
virtio_net crct10dif_pclmul crc32_pclmul crc32c_intel virtio_scsi virtio_blk
ghash_clmulni_intel virtio_console net_failover failover serio_raw dm_mirror
dm_region_hash dm_log dm_mod fuse


This does not happen with the RHEL9 linux 5.14.0-427.el9.x86_64 as seen below:

$ cd systemtap_write/systemtap/
$ uname -r
5.14.0-427.el9.x86_64
$  ../install/bin/stap -p4 -k -mtracepointsx -v
testsuite/systemtap.base/tracepoints.stp
Pass 1: parsed user script and 503 library scripts using
204116virt/175744res/15488shr/160008data kb, in 250usr/30sys/290real ms.
Pass 2: analyzed script: 22 probes, 1 function, 0 embeds, 1 global using
209860virt/185180res/18828shr/165752data kb, in 40usr/70sys/115real ms.
Pass 3: translated to C into "/tmp/stapAKS5lu/tracepointsx_src.c" using
209860virt/185180res/18828shr/165752data kb, in 10usr/0sys/10real ms.
tracepointsx.ko
Pass 4: compiled C into "tracepointsx.ko" in 16100usr/2020sys/18336real ms.
Keeping temporary directory "/tmp/stapAKS5lu"
$ cp -a /tmp/stapAKS5lu
$  ../install/bin/staprun tracepointsx.ko
tracepoints OK
$

Looking at what is creating the error message:

[ 6315.911726] jump_label: Fatal kernel bug, unexpected op at
__access_process_vm_.constprop.0+0x3d/0x1f0 [tracepoints] [0000000071a5b3f1]
(e9 eb 00 00 00 != 0f 1f 44 00 00)) size:5 type:1

It appears to be coming from:

https://elixir.bootlin.com/linux/v5.14/source/arch/x86/kernel/jump_label.c#L71

Next step is to look to see which specific tracepoint(s) is/are triggering
this.  I have the temp stap build files save so can look over those.

-- 
You are receiving this mail because:
You are the assignee for the bug.