* last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
@ 2015-06-19 15:15 William Cohen
2015-06-19 16:29 ` William Cohen
2015-06-19 16:38 ` Frank Ch. Eigler
0 siblings, 2 replies; 10+ messages in thread
From: William Cohen @ 2015-06-19 15:15 UTC (permalink / raw)
To: systemtap, Mark Wielaard, Pratyush Anand, Dave Long
Pratyush has been working on uprobes support for aarch64 and I have been exercising it with systemtap. When running the tests on a locally built checkout of systemtap sometimes the last_100_frees.stp example causes the machine to crash. It looks like this is being triggered by the scrip doing the userspace backtrace. Any suggestions on how to further diagnose this? Other nuggets of information available in the output below?
-Will
Jun 19 10:36:45 apm-mustang-ev3-11 kernel: Unable to handle kernel paging request at virtual address fffffc0001707e90
Jun 19 10:36:45 apm-mustang-ev3-11 kernel: pgd = fffffe01ddab0000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [fffffc0001707e90] *pgd=00000041ddd20003, *pud=00000041ddd20003, *pmd=00000041ddd20003, *pte=00000000000000\
00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Internal error: Oops: 96000007 [#1] SMP
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Modules linked in: stap_30b4cb5617d66b47c47d1ba687c18f92_2825(O) vfat fat xfs libcrc32c realtek [last unloa\
ded: stap_30b4cb5617d66b47c47d1ba687c18f92_2814]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: CPU: 6 PID: 2826 Comm: stap Tainted: G W O 4.1.0-rc3+ #1
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Apr 22 2015
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: task: fffffe01dda97300 ti: fffffe00bd960000 task.ti: fffffe00bd960000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: PC is at processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: LR is at processCFI.constprop.119+0x75c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: pc : [<fffffdfffc4fa2e0>] lr : [<fffffdfffc4fa2c0>] pstate: 20000145
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: sp : fffffe00bd963b30
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x29: fffffe00bd963b30 x28: 0000000000005330
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x27: fffffdfffc501e88 x26: fffffdfffc76b119
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x25: 000003ffb3e36d1b x24: fffffc00016e04e0
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x23: fffffdfffc5034e8 x22: 0000000000000001
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x21: 000000000000001b x20: fffffdfffc501e8c
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x19: fffffdfffc76b140 x18: 000003ffdafdd170
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x17: 0000000000675f50 x16: 000003ffb3e6d68c
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x15: 0000000000000000 x14: fffffffffffff928
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x13: 0000000000000002 x12: fffffffffffff920
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x11: 0000000000000002 x10: fffffffffffff978
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x9 : 0000000000000040 x8 : 0000000000000039
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x7 : 000000000000004f x6 : 000000000000004f
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x5 : 000000000000000e x4 : 000000000000004f
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x3 : fffffdfffc76b11a x2 : 0000000000000228
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x1 : fffffc0001707e90 x0 : 000000000000270f
Jun 19 10:36:46 apm-mustang-ev3-11 kernel:
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Process stap (pid: 2826, stack limit = 0xfffffe00bd960020)
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Stack: (0xfffffe00bd963b30 to 0xfffffe00bd964000)
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b20: bd963bd0 fffffe00 fc4fb3f8 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b40: fc502000 fffffdff 016e03b8 fffffc00 00000001 00000000 fc502510 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b60: 00000228 00000000 fc7537d0 fffffdff 016e04e0 fffffc00 b3e3b930 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b80: fc761820 fffffdff 00005330 00000000 fc766969 fffffdff fc76b11a fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ba0: bd963c68 fffffe00 bd963c70 fffffe00 016e04e8 fffffc00 016e04f0 fffffc00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3bc0: bd963c98 fffffe00 bd963c64 fffffe00 bd963cb0 fffffe00 fc4fbe84 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3be0: fc753750 fffffdff 016e03b8 fffffc00 00000001 00000000 b3e36d1b 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c00: 016e03b8 fffffc00 00000000 00000400 00000004 00000000 bd960000 fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c20: 00ef0a00 fffffe00 ff070000 00000001 fc766954 fffffdff 00134c38 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c40: 00000004 00000000 00000004 00000008 fc766958 fffffdff b3e36d1b 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c60: 0000001b 00000001 fc766969 fffffdff fc766970 fffffdff fc76b0c1 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c80: fc76b140 fffffdff b3e36600 000003ff 00005330 00000000 0000001e 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ca0: fc76248c fffffdff fc762488 fffffdff bd963d00 fffffe00 fc4fc108 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3cc0: 016e0000 fffffc00 00000004 00000000 bd963ed0 fffffe00 bd960000 fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ce0: 016e0000 fffffc00 fc7537d0 fffffdff b8ba6f28 fffffe00 fc753750 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d00: bd963d50 fffffe00 fc4fd7f8 fffffdff 016e0000 fffffc00 016e1ba8 fffffc00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d20: 00000005 00000000 fc790000 fffffdff 016e1000 fffffc00 00f42000 fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d40: 016e1fb8 fffffc00 fc753750 fffffdff bd963dc0 fffffe00 fc4fde4c fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d60: fc790000 fffffdff fc5c4a68 fffffdff bd963ed0 fffffe00 fc790ab8 fffffdff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d80: 73343777 00000006 d583b4a0 fffffe01 d583b420 fffffe01 00000001 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3da0: 00b5cf38 fffffe00 bd960000 fffffe00 016e1fb8 fffffc00 fff60a00 fffffe01
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3dc0: bd963e10 fffffe00 001a2af4 fffffe00 fc5c4a68 fffffdff b3e6d68c 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3de0: 00000000 00000000 bd963ed0 fffffe00 d583b400 fffffe01 001a2abc fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e00: bc2c0000 fffffe00 016e0000 fffffc00 bd963eb0 fffffe00 000972f0 fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e20: 00000610 00000000 bd963ed0 fffffe00 ffffffff ffffffff b3e6d68c 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e40: 80000000 00000000 0000003c 00000000 f2000008 00000000 00000000 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e60: dafddb80 000003ff bd960000 fffffe00 00000000 00000000 00b5cff0 fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e80: 00000000 00000000 00000001 00000000 00000000 00000003 00000003 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ea0: dafdda00 000003ff 000939d8 fffffe00 dafdd3a0 000003ff 0009388c fffffe00
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ec0: 00000600 00000000 00000030 00000000 0e8146e0 00000000 0040cdc0 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ee0: 0e8148b4 00000000 0e8148b4 00000000 68676972 43282074 b3e74fbc 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f00: b3e74fac 000003ff ffffffff ffffffff 25252525 25252525 dafddcf8 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f20: ffffffff 00000000 0e814458 00000000 0000002d 00000000 dafddde0 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f40: cccb7300 d3140edb 005dff60 00000000 b3e6d68c 000003ff 00675f50 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f60: dafdd170 000003ff dafddb80 000003ff 00000030 00000000 00000000 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f80: 0e8146e0 00000000 0e814850 00000000 00000064 00000000 0000012c 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fa0: 00000000 00000000 dafddb80 000003ff 00000000 00000000 dafdd3a0 000003ff
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fc0: b3e67680 000003ff dafdd3a0 000003ff b3e6d68c 000003ff 80000000 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fe0: 00000003 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Code: d2804501 9b017c42 91023001 8b011301 (a9401c26)
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: ---[ end trace 98f66aa70f9b298b ]---
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-19 15:15 last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace() William Cohen
@ 2015-06-19 16:29 ` William Cohen
2015-06-19 16:38 ` Frank Ch. Eigler
1 sibling, 0 replies; 10+ messages in thread
From: William Cohen @ 2015-06-19 16:29 UTC (permalink / raw)
To: systemtap, Mark Wielaard, Pratyush Anand, Dave Long
On 06/19/2015 11:15 AM, William Cohen wrote:
> Pratyush has been working on uprobes support for aarch64 and I have been exercising it with systemtap. When running the tests on a locally built checkout of systemtap sometimes the last_100_frees.stp example causes the machine to crash. It looks like this is being triggered by the scrip doing the userspace backtrace. Any suggestions on how to further diagnose this? Other nuggets of information available in the output below?
>
> -Will
>
>
>
> Jun 19 10:36:45 apm-mustang-ev3-11 kernel: Unable to handle kernel paging request at virtual address fffffc0001707e90
> Jun 19 10:36:45 apm-mustang-ev3-11 kernel: pgd = fffffe01ddab0000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [fffffc0001707e90] *pgd=00000041ddd20003, *pud=00000041ddd20003, *pmd=00000041ddd20003, *pte=00000000000000\
> 00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Internal error: Oops: 96000007 [#1] SMP
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Modules linked in: stap_30b4cb5617d66b47c47d1ba687c18f92_2825(O) vfat fat xfs libcrc32c realtek [last unloa\
> ded: stap_30b4cb5617d66b47c47d1ba687c18f92_2814]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: CPU: 6 PID: 2826 Comm: stap Tainted: G W O 4.1.0-rc3+ #1
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Apr 22 2015
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: task: fffffe01dda97300 ti: fffffe00bd960000 task.ti: fffffe00bd960000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: PC is at processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: LR is at processCFI.constprop.119+0x75c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: pc : [<fffffdfffc4fa2e0>] lr : [<fffffdfffc4fa2c0>] pstate: 20000145
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: sp : fffffe00bd963b30
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x29: fffffe00bd963b30 x28: 0000000000005330
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x27: fffffdfffc501e88 x26: fffffdfffc76b119
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x25: 000003ffb3e36d1b x24: fffffc00016e04e0
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x23: fffffdfffc5034e8 x22: 0000000000000001
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x21: 000000000000001b x20: fffffdfffc501e8c
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x19: fffffdfffc76b140 x18: 000003ffdafdd170
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x17: 0000000000675f50 x16: 000003ffb3e6d68c
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x15: 0000000000000000 x14: fffffffffffff928
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x13: 0000000000000002 x12: fffffffffffff920
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x11: 0000000000000002 x10: fffffffffffff978
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x9 : 0000000000000040 x8 : 0000000000000039
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x7 : 000000000000004f x6 : 000000000000004f
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x5 : 000000000000000e x4 : 000000000000004f
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x3 : fffffdfffc76b11a x2 : 0000000000000228
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x1 : fffffc0001707e90 x0 : 000000000000270f
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel:
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Process stap (pid: 2826, stack limit = 0xfffffe00bd960020)
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Stack: (0xfffffe00bd963b30 to 0xfffffe00bd964000)
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b20: bd963bd0 fffffe00 fc4fb3f8 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b40: fc502000 fffffdff 016e03b8 fffffc00 00000001 00000000 fc502510 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b60: 00000228 00000000 fc7537d0 fffffdff 016e04e0 fffffc00 b3e3b930 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b80: fc761820 fffffdff 00005330 00000000 fc766969 fffffdff fc76b11a fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ba0: bd963c68 fffffe00 bd963c70 fffffe00 016e04e8 fffffc00 016e04f0 fffffc00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3bc0: bd963c98 fffffe00 bd963c64 fffffe00 bd963cb0 fffffe00 fc4fbe84 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3be0: fc753750 fffffdff 016e03b8 fffffc00 00000001 00000000 b3e36d1b 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c00: 016e03b8 fffffc00 00000000 00000400 00000004 00000000 bd960000 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c20: 00ef0a00 fffffe00 ff070000 00000001 fc766954 fffffdff 00134c38 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c40: 00000004 00000000 00000004 00000008 fc766958 fffffdff b3e36d1b 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c60: 0000001b 00000001 fc766969 fffffdff fc766970 fffffdff fc76b0c1 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c80: fc76b140 fffffdff b3e36600 000003ff 00005330 00000000 0000001e 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ca0: fc76248c fffffdff fc762488 fffffdff bd963d00 fffffe00 fc4fc108 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3cc0: 016e0000 fffffc00 00000004 00000000 bd963ed0 fffffe00 bd960000 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ce0: 016e0000 fffffc00 fc7537d0 fffffdff b8ba6f28 fffffe00 fc753750 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d00: bd963d50 fffffe00 fc4fd7f8 fffffdff 016e0000 fffffc00 016e1ba8 fffffc00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d20: 00000005 00000000 fc790000 fffffdff 016e1000 fffffc00 00f42000 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d40: 016e1fb8 fffffc00 fc753750 fffffdff bd963dc0 fffffe00 fc4fde4c fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d60: fc790000 fffffdff fc5c4a68 fffffdff bd963ed0 fffffe00 fc790ab8 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d80: 73343777 00000006 d583b4a0 fffffe01 d583b420 fffffe01 00000001 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3da0: 00b5cf38 fffffe00 bd960000 fffffe00 016e1fb8 fffffc00 fff60a00 fffffe01
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3dc0: bd963e10 fffffe00 001a2af4 fffffe00 fc5c4a68 fffffdff b3e6d68c 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3de0: 00000000 00000000 bd963ed0 fffffe00 d583b400 fffffe01 001a2abc fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e00: bc2c0000 fffffe00 016e0000 fffffc00 bd963eb0 fffffe00 000972f0 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e20: 00000610 00000000 bd963ed0 fffffe00 ffffffff ffffffff b3e6d68c 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e40: 80000000 00000000 0000003c 00000000 f2000008 00000000 00000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e60: dafddb80 000003ff bd960000 fffffe00 00000000 00000000 00b5cff0 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e80: 00000000 00000000 00000001 00000000 00000000 00000003 00000003 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ea0: dafdda00 000003ff 000939d8 fffffe00 dafdd3a0 000003ff 0009388c fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ec0: 00000600 00000000 00000030 00000000 0e8146e0 00000000 0040cdc0 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ee0: 0e8148b4 00000000 0e8148b4 00000000 68676972 43282074 b3e74fbc 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f00: b3e74fac 000003ff ffffffff ffffffff 25252525 25252525 dafddcf8 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f20: ffffffff 00000000 0e814458 00000000 0000002d 00000000 dafddde0 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f40: cccb7300 d3140edb 005dff60 00000000 b3e6d68c 000003ff 00675f50 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f60: dafdd170 000003ff dafddb80 000003ff 00000030 00000000 00000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f80: 0e8146e0 00000000 0e814850 00000000 00000064 00000000 0000012c 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fa0: 00000000 00000000 dafddb80 000003ff 00000000 00000000 dafdd3a0 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fc0: b3e67680 000003ff dafdd3a0 000003ff b3e6d68c 000003ff 80000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fe0: 00000003 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Code: d2804501 9b017c42 91023001 8b011301 (a9401c26)
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: ---[ end trace 98f66aa70f9b298b ]---
>
Taking a look at the LR and PC values the snippet of the systemtap where things go wrong is in processCFI.constprop.119. The code has called and returned from get_uleb128 at a2bc. The processor continues on to a2e0. The ldp attempts to load the register pair from fffffc0001707e90 which causes the panic.
x24 is used as a base address in both a2dc and a2c0. The instruction at a2c0 only has an offset of 24 off of fffffc00016e04e0. x1 at a2e0 has offset of 279b0 (162,224 bytes) away. This is likely on another page.
0000000000009b64 <processCFI.constprop.119>: ...
a2b4: aa1303e1 mov x1, x19
a2b8: 9101a3a0 add x0, x29, #0x68
a2bc: 97ffd7df bl 238 <get_uleb128>
a2c0: 39406302 ldrb w2, [x24,#24]
a2c4: d284e1e1 mov x1, #0x270f // #9999
a2c8: f100801f cmp x0, #0x20
a2cc: 9a813000 csel x0, x0, x1, cc
a2d0: d2804501 mov x1, #0x228 // #552
a2d4: 9b017c42 mul x2, x2, x1
a2d8: 91023001 add x1, x0, #0x8c
a2dc: 8b011301 add x1, x24, x1, lsl #4
a2e0: a9401c26 ldp x6, x7, [x1]
a2e4: 8b001040 add x0, x2, x0, lsl #4
a2e8: 8b000300 add x0, x24, x0
a2ec: a9031c06 stp x6, x7, [x0,#48]
a2f0: f94037a6 ldr x6, [x29,#104]
a2f4: 52800021 mov w1, #0x1 // #1
-Will
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-19 15:15 last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace() William Cohen
2015-06-19 16:29 ` William Cohen
@ 2015-06-19 16:38 ` Frank Ch. Eigler
2015-06-19 21:01 ` William Cohen
1 sibling, 1 reply; 10+ messages in thread
From: Frank Ch. Eigler @ 2015-06-19 16:38 UTC (permalink / raw)
To: William Cohen; +Cc: systemtap, Mark Wielaard, Pratyush Anand, Dave Long
William Cohen <wcohen@redhat.com> writes:
> [...] Any suggestions on how to further diagnose this? Other
> nuggets of information available in the output below?
Certainly. One would start from retaining a copy of the systemtap
module, built with
# stap -p4 -k -DDEBUG_UNWIND=2 -G CONFIG_DEBUGINFO=y ...
so one can apply normal post-mortem debugging techniques to the module
(disassemble at the trapping address, deduce whether data or code is
more suspect, etc.). dwarf-dump the target process to check the dwarf
data by hand. Attempt run; collect vmcore and any panicy messages.
runtime/unwind.c:processCFI() works its way through the dwarf data
with some protection against e.g. bad data, but it may be
insufficient.
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Apr 22 2015
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: task: fffffe01dda97300 ti: fffffe00bd960000 task.ti: fffffe00bd960000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: PC is at processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: LR is at processCFI.constprop.119+0x75c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: pc : [<fffffdfffc4fa2e0>] lr : [<fffffdfffc4fa2c0>] pstate: 20000145
> [...]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> [...]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-19 16:38 ` Frank Ch. Eigler
@ 2015-06-19 21:01 ` William Cohen
2015-06-22 9:00 ` Mark Wielaard
0 siblings, 1 reply; 10+ messages in thread
From: William Cohen @ 2015-06-19 21:01 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: systemtap, Mark Wielaard, Pratyush Anand, Dave Long
On 06/19/2015 12:38 PM, Frank Ch. Eigler wrote:
> William Cohen <wcohen@redhat.com> writes:
>
>> [...] Any suggestions on how to further diagnose this? Other
>> nuggets of information available in the output below?
>
> Certainly. One would start from retaining a copy of the systemtap
> module, built with
>
> # stap -p4 -k -DDEBUG_UNWIND=2 -G CONFIG_DEBUGINFO=y ...
Hi Frank,
I did use -p4 -k for the stap modules and took a look at output earlier. The -DDEBUG_UNWIND=2 -G CONFIG_DEBUG_INFO=y are good additional suggestions to get a better understanding. The -DEBUG_UNWIND=2 looks like it might produce a bit too much output for this particular case. The -G CONFIG_DEBUGINFO=y doesn't have any effect. https://lwn.net/Articles/502773/ suggests -B CONFIG_DEBUG_INFO=y because of:
commit 150f207589149d410687e914f27eeb3b46d68b3d
Author: Frank Ch. Eigler <fche@redhat.com>
Date: Wed Mar 14 13:10:38 2012 -0400
PR13847: don't build debuginfo for stap modules
Suppressing the -g (by default) for stap_*.ko reduces compile time
and greatly reduces resulting .ko file size. Since stap modules
are not usually probed themselves, this is a good default.
* buildrun.cxx (make_any_make_cmd): Prefix kbuildflags with CONFIG_DEBUG_INFO=.
However, I can't seem to get the stap module to be built with debuginfo even with what seem like appropriate command line options.
>
> so one can apply normal post-mortem debugging techniques to the module
> (disassemble at the trapping address, deduce whether data or code is
> more suspect, etc.). dwarf-dump the target process to check the dwarf
> data by hand. Attempt run; collect vmcore and any panicy messages.
>
> runtime/unwind.c:processCFI() works its way through the dwarf data
> with some protection against e.g. bad data, but it may be
> insufficient.
I recall Mark making some changes to make elf handling more robust in elfutils. I suspect similar robustification is missing somewhere in the systemtap unwinder code.
-Will
>
>
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Apr 22 2015
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: task: fffffe01dda97300 ti: fffffe00bd960000 task.ti: fffffe00bd960000
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: PC is at processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: LR is at processCFI.constprop.119+0x75c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: pc : [<fffffdfffc4fa2e0>] lr : [<fffffdfffc4fa2c0>] pstate: 20000145
>> [...]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> [...]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-19 21:01 ` William Cohen
@ 2015-06-22 9:00 ` Mark Wielaard
2015-06-22 13:53 ` William Cohen
0 siblings, 1 reply; 10+ messages in thread
From: Mark Wielaard @ 2015-06-22 9:00 UTC (permalink / raw)
To: William Cohen; +Cc: Frank Ch. Eigler, systemtap, Pratyush Anand, Dave Long
On Fri, 2015-06-19 at 17:01 -0400, William Cohen wrote:
> However, I can't seem to get the stap module to be built with
> debuginfo even with what seem like appropriate command line options.
Are you sure? The following should generate debuginfo for the stap.ko
you are creating:
stap -k -p4 -vv -B CONFIG_DEBUG_INFO=y -e 'probe begin {log("hello"); exit(); }'
If not, please post the -vv output so we can analyze what is going
wrong.
> > runtime/unwind.c:processCFI() works its way through the dwarf data
> > with some protection against e.g. bad data, but it may be
> > insufficient.
>
> I recall Mark making some changes to make elf handling more robust in
> elfutils. I suspect similar robustification is missing somewhere in
> the systemtap unwinder code.
Maybe indeed. But I don't yet understand what is going in. Hopefully we
can get a module with debuginfo to better analyse.
Thanks,
Mark
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-22 9:00 ` Mark Wielaard
@ 2015-06-22 13:53 ` William Cohen
2015-06-22 14:15 ` Mark Wielaard
2015-06-22 17:07 ` Josh Stone
0 siblings, 2 replies; 10+ messages in thread
From: William Cohen @ 2015-06-22 13:53 UTC (permalink / raw)
To: Mark Wielaard; +Cc: Frank Ch. Eigler, systemtap, Pratyush Anand, Dave Long
On 06/22/2015 05:00 AM, Mark Wielaard wrote:
> On Fri, 2015-06-19 at 17:01 -0400, William Cohen wrote:
>> However, I can't seem to get the stap module to be built with
>> debuginfo even with what seem like appropriate command line options.
>
> Are you sure? The following should generate debuginfo for the stap.ko
> you are creating:
>
> stap -k -p4 -vv -B CONFIG_DEBUG_INFO=y -e 'probe begin {log("hello"); exit(); }'
>
> If not, please post the -vv output so we can analyze what is going
> wrong.
>
>>> runtime/unwind.c:processCFI() works its way through the dwarf data
>>> with some protection against e.g. bad data, but it may be
>>> insufficient.
>>
>> I recall Mark making some changes to make elf handling more robust in
>> elfutils. I suspect similar robustification is missing somewhere in
>> the systemtap unwinder code.
>
> Maybe indeed. But I don't yet understand what is going in. Hopefully we
> can get a module with debuginfo to better analyse.
>
> Thanks,
>
> Mark
>
Hi Mark,
One of the things that I thought that might be tripping things up is that there might be previous builds that have been done without the -B CONFIG_DEBUG_INFO=y and the caching might be using. I tried building the module with:
/root/systemtap_write/install/bin/stap -B CONFIG_DEBUG_INFO=y -m l100 -p4 -vv -k systemtap.examples/memory/last_100_frees.stp -c "/root/systemtap_write/install/bin/stap -V" -d /root/systemtap_write/install/bin/stap --ldd >& l100.log
Below is "eu-readelf -S" output for the module which seems to have debuginfo in it:
$eu-readelf -S l100.ko
There are 44 section headers, starting at offset 0x301c60:
Section Headers:
[Nr] Name Type Addr Off Size ES Flags Lk Inf Al
[ 0] NULL 0000000000000000 00000000 00000000 0 0 0 0
[ 1] .text PROGBITS 0000000000000000 00000040 00011c7c 0 AX 0 0 4
[ 2] .rela.text RELA 0000000000000000 00302760 0000d1b8 24 42 1 8
[ 3] .fixup PROGBITS 0000000000000000 00011cbc 0000009c 0 AX 0 0 4
[ 4] .rela.fixup RELA 0000000000000000 0030f918 00000138 24 42 3 8
[ 5] .text.unlikely PROGBITS 0000000000000000 00011d58 00000088 0 AX 0 0 4
[ 6] .rela.text.unlikely RELA 0000000000000000 0030fa50 00000030 24 42 5 8
[ 7] .rodata PROGBITS 0000000000000000 00011de0 00000980 0 A 0 0 8
[ 8] .rela.rodata RELA 0000000000000000 0030fa80 000003a8 24 42 7 8
[ 9] __ex_table PROGBITS 0000000000000000 00012760 000000d0 0 A 0 0 8
[10] .rela__ex_table RELA 0000000000000000 0030fe28 00000270 24 42 9 8
[11] .modinfo PROGBITS 0000000000000000 00012830 000000b0 0 A 0 0 8
[12] __param PROGBITS 0000000000000000 000128e0 00000020 0 A 0 0 8
[13] .rela__param RELA 0000000000000000 00310098 00000048 24 42 12 8
[14] .rodata.str1.8 PROGBITS 0000000000000000 00012900 000bf218 1 AMS 0 0 8
[15] __mcount_loc PROGBITS 0000000000000000 000d1b18 00000570 0 A 0 0 8
[16] .rela__mcount_loc RELA 0000000000000000 003100e0 00001050 24 42 15 8
[17] .eh_frame PROGBITS 0000000000000000 000d2088 00002744 0 WA 0 0 8
[18] .rela.eh_frame RELA 0000000000000000 00311130 00001050 24 42 17 8
[19] .data PROGBITS 0000000000000000 000d47d0 001cc160 0 WA 0 0 8
[20] .rela.data RELA 0000000000000000 00312180 000ad460 24 42 19 8
[21] .data.unlikely PROGBITS 0000000000000000 002a0930 00000001 0 WA 0 0 1
[22] .stap_privilege PROGBITS 0000000000000000 002a0934 00000004 0 WA 0 0 4
[23] .gnu.linkonce.this_module PROGBITS 0000000000000000 002a0938 00000248 0 WA 0 0 8
[24] .rela.gnu.linkonce.this_module RELA 0000000000000000 003bf5e0 00000030 24 42 23 8
[25] .note.gnu.build-id NOTE 0000000000000000 002a0b80 00000024 0 A 0 0 4
[26] .bss NOBITS 0000000000000000 002a0ba8 00006798 0 WA 0 0 8
[27] .comment PROGBITS 0000000000000000 002a0ba8 00000087 1 MS 0 0 1
[28] .note.GNU-stack PROGBITS 0000000000000000 002a0c2f 00000000 0 0 0 1
[29] .debug_aranges PROGBITS 0000000000000000 002a0c2f 00000080 0 0 0 1
[30] .rela.debug_aranges RELA 0000000000000000 003bf610 00000078 24 42 29 8
[31] .debug_info PROGBITS 0000000000000000 002a0caf 0002ca16 0 0 0 1
[32] .rela.debug_info RELA 0000000000000000 003bf688 000404b8 24 42 31 8
[33] .debug_abbrev PROGBITS 0000000000000000 002cd6c5 00000b67 0 0 0 1
[34] .debug_line PROGBITS 0000000000000000 002ce22c 000063b3 0 0 0 1
[35] .rela.debug_line RELA 0000000000000000 003ffb40 00000030 24 42 34 8
[36] .debug_str PROGBITS 0000000000000000 002d45df 0000dea2 1 MS 0 0 1
[37] .debug_loc PROGBITS 0000000000000000 002e2481 00017ec5 0 0 0 1
[38] .rela.debug_loc RELA 0000000000000000 003ffb70 00030c60 24 42 37 8
[39] .debug_ranges PROGBITS 0000000000000000 002fa346 00007780 0 0 0 1
[40] .rela.debug_ranges RELA 0000000000000000 004307d0 00010440 24 42 39 8
[41] .shstrtab STRTAB 0000000000000000 00301ac6 00000195 0 0 0 1
[42] .symtab SYMTAB 0000000000000000 00440c10 00004188 24 43 552 8
[43] .strtab STRTAB 0000000000000000 00444d98 00003396 0 0 0 1
The original traceback was:
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
Adjusting the addresses for the module by itself and Using addr2line the call trace maps back to:
a2e0
/root/systemtap_write/install/share/systemtap/runtime/unwind.c:429
b3f4
/root/systemtap_write/install/share/systemtap/runtime/unwind.c:1287
be80
/root/systemtap_write/install/share/systemtap/runtime/unwind.c:1518
c104
/root/systemtap_write/install/share/systemtap/runtime/stack.c:491
d7f4
/root/systemtap_write/install/share/systemtap/runtime/stack.c:591
de48
/tmp/stapUGCcwL/l100_src.c:867
I hope that provides a bit more insight into what is going wrong in the systemtap unwinder runtime.
-Will
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-22 13:53 ` William Cohen
@ 2015-06-22 14:15 ` Mark Wielaard
2015-06-22 14:47 ` William Cohen
2015-06-22 17:07 ` Josh Stone
1 sibling, 1 reply; 10+ messages in thread
From: Mark Wielaard @ 2015-06-22 14:15 UTC (permalink / raw)
To: William Cohen; +Cc: Frank Ch. Eigler, systemtap, Pratyush Anand, Dave Long
On Mon, 2015-06-22 at 09:53 -0400, William Cohen wrote:
> The original traceback was:
>
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
>
> Adjusting the addresses for the module by itself and Using addr2line the call trace maps back to:
> a2e0
> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:429
> b3f4
> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:1287
> be80
> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:1518
> c104
> /root/systemtap_write/install/share/systemtap/runtime/stack.c:491
> d7f4
> /root/systemtap_write/install/share/systemtap/runtime/stack.c:591
> de48
> /tmp/stapUGCcwL/l100_src.c:867
>
> I hope that provides a bit more insight into what is going wrong in the systemtap unwinder runtime.
Most certainly. Assuming that maps to current git sources the crash is
at the end of the following case statement, at the memcpy:
case DW_CFA_restore_extended:
value = get_uleb128(&ptr.p8, end);
if (compat_task) {
dbug_unwind(1, "map DW_CFA_restore_extended value %ld to reg_info idx %ld\n",
value, COMPAT_REG_MAP(DWARF_REG_MAP(value)));
value = COMPAT_REG_MAP(DWARF_REG_MAP(value));
} else {
dbug_unwind(1, "map DW_CFA_restore_extended value %ld to reg_info idx %ld\n",
value, DWARF_REG_MAP(value));
value = DWARF_REG_MAP(value);
}
memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
break;
Note how value comes from DWARF_REG_MAP. It maps the DWARF register
number to the number we use in the arrays. It is defined as follows in
runtime/unwind/arm64.h
#define DWARF_REG_MAP(r) \
((r >= 0 && r <= 31) ? r /* regs[0-30] + sp */ \
: 9999)
So, if it doesn't recognize (or more accurately, isn't interested in)
the DWARF register number it will return a ridiculously large number.
Which we will use directly to index the register array. Oops...
In all other cases, except this and DW_CFA_restore, an helper function
is called (set_*_rule) which will only use the reg value if (reg <
ARRAY_SIZE(REG_STATE.regs)).
So we will need something like:
diff --git a/runtime/unwind.c b/runtime/unwind.c
index d38363b..4dbab33 100644
--- a/runtime/unwind.c
+++ b/runtime/unwind.c
@@ -426,7 +426,8 @@ static int processCFI(const u8 *start, const u8 *end, unsigned long targetLoc,
value, DWARF_REG_MAP(value));
value = DWARF_REG_MAP(value);
}
- memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
+ if (value < ARRAY_SIZE(REG_STATE.regs))
+ memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
break;
case DW_CFA_undefined:
value = get_uleb128(&ptr.p8, end);
@@ -641,7 +642,8 @@ static int processCFI(const u8 *start, const u8 *end, unsigned long targetLoc,
value, DWARF_REG_MAP(value));
value = DWARF_REG_MAP(value);
}
- memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
+ if (value < ARRAY_SIZE(REG_STATE.regs))
+ memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
break;
}
dbug_unwind(1, "targetLoc=%lx state->loc=%lx\n", targetLoc, state->loc);
Does that help in your case?
Thanks,
Mark
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-22 14:15 ` Mark Wielaard
@ 2015-06-22 14:47 ` William Cohen
2015-06-22 15:07 ` Mark Wielaard
0 siblings, 1 reply; 10+ messages in thread
From: William Cohen @ 2015-06-22 14:47 UTC (permalink / raw)
To: Mark Wielaard; +Cc: Frank Ch. Eigler, systemtap, Pratyush Anand, Dave Long
On 06/22/2015 10:15 AM, Mark Wielaard wrote:
> On Mon, 2015-06-22 at 09:53 -0400, William Cohen wrote:
>> The original traceback was:
>>
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
>>
>> Adjusting the addresses for the module by itself and Using addr2line the call trace maps back to:
>> a2e0
>> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:429
>> b3f4
>> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:1287
>> be80
>> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:1518
>> c104
>> /root/systemtap_write/install/share/systemtap/runtime/stack.c:491
>> d7f4
>> /root/systemtap_write/install/share/systemtap/runtime/stack.c:591
>> de48
>> /tmp/stapUGCcwL/l100_src.c:867
>>
>> I hope that provides a bit more insight into what is going wrong in the systemtap unwinder runtime.
>
> Most certainly. Assuming that maps to current git sources the crash is
> at the end of the following case statement, at the memcpy:
Yes, this for a systemtap built from the current git repository the last commit in there is:
commit 7e05b29acbc5bf84b860499ab1e3d1fa648ad086
Author: David Smith <dsmith@redhat.com>
Date: Fri Jun 19 13:26:41 2015 -0500
Fixed PR18563 by updating mbrwatch.meta.
* testsuite/systemtap.examples/io/mbrwatch.meta: Update to ignore any
devices called 'sr[0-0]', which on ppc64 aren't disk drives.
> case DW_CFA_restore_extended:
> value = get_uleb128(&ptr.p8, end);
> if (compat_task) {
> dbug_unwind(1, "map DW_CFA_restore_extended value %ld to reg_info idx %ld\n",
> value, COMPAT_REG_MAP(DWARF_REG_MAP(value)));
> value = COMPAT_REG_MAP(DWARF_REG_MAP(value));
> } else {
> dbug_unwind(1, "map DW_CFA_restore_extended value %ld to reg_info idx %ld\n",
> value, DWARF_REG_MAP(value));
> value = DWARF_REG_MAP(value);
> }
> memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> break;
>
> Note how value comes from DWARF_REG_MAP. It maps the DWARF register
> number to the number we use in the arrays. It is defined as follows in
> runtime/unwind/arm64.h
>
> #define DWARF_REG_MAP(r) \
> ((r >= 0 && r <= 31) ? r /* regs[0-30] + sp */ \
> : 9999)
>
> So, if it doesn't recognize (or more accurately, isn't interested in)
> the DWARF register number it will return a ridiculously large number.
> Which we will use directly to index the register array. Oops...
>
> In all other cases, except this and DW_CFA_restore, an helper function
> is called (set_*_rule) which will only use the reg value if (reg <
> ARRAY_SIZE(REG_STATE.regs)).
>
I have built the systemtap code with the patch below and exercising with:
while [ 1 ]; do /root/systemtap_write/install/bin/stap systemtap.examples/memory/last_100_frees.stp -B CONFIG_DEBUG_INFO=y -c "/root/systemtap_write/install/bin/stap -V" -d /root/systemtap_write/install/bin/stap --ldd;done
Thanks again for the fix Mark.
-Will
> So we will need something like:
>
> diff --git a/runtime/unwind.c b/runtime/unwind.c
> index d38363b..4dbab33 100644
> --- a/runtime/unwind.c
> +++ b/runtime/unwind.c
> @@ -426,7 +426,8 @@ static int processCFI(const u8 *start, const u8 *end, unsigned long targetLoc,
> value, DWARF_REG_MAP(value));
> value = DWARF_REG_MAP(value);
> }
> - memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> + if (value < ARRAY_SIZE(REG_STATE.regs))
> + memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> break;
> case DW_CFA_undefined:
> value = get_uleb128(&ptr.p8, end);
> @@ -641,7 +642,8 @@ static int processCFI(const u8 *start, const u8 *end, unsigned long targetLoc,
> value, DWARF_REG_MAP(value));
> value = DWARF_REG_MAP(value);
> }
> - memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> + if (value < ARRAY_SIZE(REG_STATE.regs))
> + memcpy(®_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> break;
> }
> dbug_unwind(1, "targetLoc=%lx state->loc=%lx\n", targetLoc, state->loc);
>
> Does that help in your case?
>
> Thanks,
>
> Mark
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-22 14:47 ` William Cohen
@ 2015-06-22 15:07 ` Mark Wielaard
0 siblings, 0 replies; 10+ messages in thread
From: Mark Wielaard @ 2015-06-22 15:07 UTC (permalink / raw)
To: William Cohen; +Cc: Frank Ch. Eigler, systemtap, Pratyush Anand, Dave Long
On Mon, 2015-06-22 at 10:47 -0400, William Cohen wrote:
> I have built the systemtap code with the patch below and exercising with:
>
> while [ 1 ]; do /root/systemtap_write/install/bin/stap
> systemtap.examples/memory/last_100_frees.stp -B CONFIG_DEBUG_INFO=y -c
> "/root/systemtap_write/install/bin/stap -V"
> -d /root/systemtap_write/install/bin/stap --ldd;done
Thanks for testing. I also tested locally (on x86_64) that exelib.exp,
which uses the unwinder extensively, works as expected. Patch committed
as:
commit 81bde8f873216c116988a98a0804dd79009b3d40
Author: Mark Wielaard <mjw@redhat.com>
Date: Mon Jun 22 16:57:59 2015 +0200
runtime/unwind.c: Also sanity check DWARF regno for DW_CFA_restore[_extended].
When processCFI wanted to restore a register state to its initial value it
wasn't checking whether the register was actually interesting (or existing).
DWARF_REG_MAP might return a marker (9999) that we don't know or don't care
about this register. This was checked in all the set_*_rule functions, but
not in the case we reset the rule of the register. Add this check also for
DW_CFA_restore[_extended].
Thanks,
Mark
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
2015-06-22 13:53 ` William Cohen
2015-06-22 14:15 ` Mark Wielaard
@ 2015-06-22 17:07 ` Josh Stone
1 sibling, 0 replies; 10+ messages in thread
From: Josh Stone @ 2015-06-22 17:07 UTC (permalink / raw)
To: systemtap
On 06/22/2015 06:53 AM, William Cohen wrote:
> One of the things that I thought that might be tripping things up is
> that there might be previous builds that have been done without the
> -B CONFIG_DEBUG_INFO=y and the caching might be using.
That shouldn't matter -- find_script_hash() adds all the -B options to
the hashed content, so they should be cached distinctly.
But you can also use --poison-cache if you don't trust it.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-06-22 17:07 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-19 15:15 last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace() William Cohen
2015-06-19 16:29 ` William Cohen
2015-06-19 16:38 ` Frank Ch. Eigler
2015-06-19 21:01 ` William Cohen
2015-06-22 9:00 ` Mark Wielaard
2015-06-22 13:53 ` William Cohen
2015-06-22 14:15 ` Mark Wielaard
2015-06-22 14:47 ` William Cohen
2015-06-22 15:07 ` Mark Wielaard
2015-06-22 17:07 ` Josh Stone
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).