[Bug runtime/17862] New: Kernel crash during on module insertion: kernel tried to execute NX-protected page - exploit attempt

[Bug runtime/17862] New: Kernel crash during on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

            Bug ID: 17862
           Summary: Kernel crash during on module insertion: kernel tried
                    to execute NX-protected page - exploit attempt
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: izi at guardicore dot com

Getting an error of "kernel tried to execute NX-protected page - exploit
attempt" in syslog during module insertion. Problem is not reproduced on every
run, possibly due to races with other modules which are loaded at the same
time.

Attached is the failed script, which is precompiled into a .ko, and the
stap-report data.

Jan 21 02:59:15 ldsm kernel: [   13.454242] g_2475: systemtap: 2.6/0.157, base:
ffffffffa02d2000, memory: 221data/56text/64ctx/2058net/9alloc kb, probes: 7
Jan 21 02:59:15 ldsm kernel: [   13.489567] g_2471: systemtap: 2.6/0.157, base:
ffffffffa024d000, memory: 411data/88text/4417ctx/2058net/649alloc kb, probes:
10
Jan 21 02:59:15 ldsm kernel: [   13.542182] gc_2480: systemtap: 2.6/0.157,
base: ffffffffa031f000, memory: 195data/52text/960ctx/2058net/9alloc kb,
probes: 2
Jan 21 02:59:15 ldsm kernel: [   13.562902] g_2486: systemtap: 2.6/0.157, base:
ffffffffa035e000, memory: 191data/48text/448ctx/2058net/9alloc kb, probes: 2
Jan 21 02:59:15 ldsm kernel: [   13.580491] kernel tried to execute
NX-protected page - exploit attempt? (uid: 0)
Jan 21 02:59:15 ldsm kernel: [   13.580673] BUG: unable to handle kernel paging
request at ffff88003b22c0e1
Jan 21 02:59:15 ldsm kernel: [   13.580841] IP: [<ffff88003b22c0e1>]
0xffff88003b22c0e0
Jan 21 02:59:15 ldsm kernel: [   13.581018] PGD 1fd1067 PUD 1fd2067 PMD
3bd95063 PTE 800000003b22c163
Jan 21 02:59:15 ldsm kernel: [   13.581190] Oops: 0011 [#1] SMP
Jan 21 02:59:15 ldsm kernel: [   13.581346] Modules linked in: gc__2489(OF)
g_2486(OF) gc_2480(OF) g_2475(OF) g_2471(OF) veth(F) arc4(F) md4(F) nls_utf8
cifs(F) fscache(F) openvswitch gre(F) snd_hda_intel cirrus snd_hda_codec
snd_hwdep(F) microcode(F) ttm drm_kms_helper snd_pcm(F) snd_page_alloc(F)
snd_timer(F) psmouse(F) snd(F) serio_raw(F) virtio_balloon(F) soundcore(F) drm
syscopyarea(F) sysfillrect(F) sysimgblt(F) i2c_piix4 mac_hid lp(F) parport(F)
ext2(F) 8139too(F) 8139cp(F) mii(F) floppy(F)
Jan 21 02:59:15 ldsm kernel: [   13.582014] CPU: 0 PID: 2496 Comm: ntpd
Tainted: GF          O 3.11.0-12-generic #19-Ubuntu
Jan 21 02:59:15 ldsm kernel: [   13.582183] Hardware name: QEMU Standard PC
(i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Jan 21 02:59:15 ldsm kernel: [   13.582352] task: ffff8800299bc650 ti:
ffff88002977e000 task.ti: ffff88002977e000
Jan 21 02:59:15 ldsm kernel: [   13.582515] RIP: 0010:[<ffff88003b22c0e1>] 
[<ffff88003b22c0e1>] 0xffff88003b22c0e0
Jan 21 02:59:15 ldsm kernel: [   13.582687] RSP: 0018:ffff88002977ff20  EFLAGS:
00010286
Jan 21 02:59:15 ldsm kernel: [   13.582843] RAX: ffff88003b22c0e1 RBX:
ffff88002977ff58 RCX: 0000000000000003
Jan 21 02:59:15 ldsm kernel: [   13.583005] RDX: 0000000000000000 RSI:
ffff88002977ff58 RDI: ffff880036c617a0
Jan 21 02:59:15 ldsm kernel: [   13.583168] RBP: ffff88002977ff40 R08:
000000000155629b R09: 0000000000000001
Jan 21 02:59:15 ldsm kernel: [   13.583333] R10: ffffea0000f33600 R11:
ffffffffa02dcc5c R12: ffff88003cc4c430
Jan 21 02:59:15 ldsm kernel: [   13.583496] R13: 0000000000000000 R14:
0000000000000000 R15: 0000000000000000
Jan 21 02:59:15 ldsm kernel: [   13.583659] FS:  0000000000000000(0000)
GS:ffff88003fc00000(0000) knlGS:0000000000000000
Jan 21 02:59:15 ldsm kernel: [   13.583826] CS:  0010 DS: 0000 ES: 0000 CR0:
000000008005003b
Jan 21 02:59:15 ldsm kernel: [   13.583984] CR2: ffff88003b22c0e1 CR3:
000000002979f000 CR4: 00000000000006f0
Jan 21 02:59:15 ldsm kernel: [   13.584020] Stack:
Jan 21 02:59:15 ldsm kernel: [   13.584020]  ffffffff8101fb17 0000000000000000
0000000000000000 0000000000000000
Jan 21 02:59:15 ldsm kernel: [   13.584020]  0000000000000000 ffffffff816f54bf
000000001008feff 0000000000000000
Jan 21 02:59:15 ldsm kernel: [   13.584020]  0000000000000000 0000000000000000
0000000000000000 0000000000000000
Jan 21 02:59:15 ldsm kernel: [   13.584020] Call Trace:
Jan 21 02:59:15 ldsm kernel: [   13.584020]  [<ffffffff8101fb17>] ?
syscall_trace_leave+0xd7/0xf0
Jan 21 02:59:15 ldsm kernel: [   13.584020]  [<ffffffff816f54bf>]
int_check_syscall_exit_work+0x34/0x3d
Jan 21 02:59:15 ldsm kernel: [   13.584020] Code: 00 00 07 00 00 00 00 00 00 00
58 99 c0 3c 00 88 ff ff 00 68 4d 3b 00 88 ff ff d0 99 c0 3c 00 88 ff ff d0 99
c0 3c 00 88 ff ff 60 <db> 25 3b 00 88 ff ff e0 c4 c4 3c 00 88 ff ff 20 c4 c4 3c
00 88
Jan 21 02:59:15 ldsm kernel: [   13.584020] RIP  [<ffff88003b22c0e1>]
0xffff88003b22c0e0
Jan 21 02:59:15 ldsm kernel: [   13.584020]  RSP <ffff88002977ff20>
Jan 21 02:59:15 ldsm kernel: [   13.584020] CR2: ffff88003b22c0e1
Jan 21 02:59:15 ldsm kernel: [   13.584020] ---[ end trace e1a4d67e626da1fa
]---

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash during on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

--- Comment #2 from izi at guardicore dot com ---
Created attachment 8073
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8073&action=edit
stap-report

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash during on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

--- Comment #1 from izi at guardicore dot com ---
Created attachment 8072
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8072&action=edit
Failed script

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

izi at guardicore dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Kernel crash during on      |Kernel crash on module
                   |module insertion: kernel    |insertion: kernel tried to
                   |tried to execute            |execute NX-protected page -
                   |NX-protected page - exploit |exploit attempt
                   |attempt                     |

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

izi at guardicore dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #8072|0                           |1
        is obsolete|                            |

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

--- Comment #3 from izi at guardicore dot com ---
Created attachment 8074
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8074&action=edit
Failed script

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash on module insertion: kernel tried to execute NX-protected page - exploit attempt

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #4 from David Smith <dsmith at redhat dot com> ---
I looked at your script and didn't see anything too strange - 5 uprobes and a
timer probe. My best guess would be that systemtap is probing an incorrect
address.

Can you try running the script again, enabling 1 probe at a time?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash on module insertion: kernel tried to execute NX-protected page - exploit attempt

[jistone at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #5 from Josh Stone <jistone at redhat dot com> ---
My first guess is that this is the XOL page, execute-out-of-line for the
instruction replaced by a breakpoint.  If the kernel uprobes didn't set the
permissions on that page properly, it would obviously fail.

Or even if the XOL page is correct, it could be that uprobes wrongly tried to
step through some branching instruction, jmp/call/ret, sending the RIP off into
the weeds.

Can you try to reproduce the same probes with perf?  Something like:

  perf probe -x /usr/sbin/ntpd receive
  perf probe -x /usr/sbin/ntpd receive%return
  perf probe -x /usr/sbin/ntpd configure
  [etc.]
  perf trace -e 'probe_ntpd:*'

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17862] Kernel crash on module insertion: kernel tried to execute NX-protected page - exploit attempt

[izi at guardicore dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17862

--- Comment #6 from izi at guardicore dot com ---
I'm loading several systemtap modules concurrently, so I'm guessing there is a
race here. The other modules also include a few uprobes and a timer probe for
each one. The module insertion usually works fine in 9 out of 10 runs and I see
the printfs later on when the probed functions are called. So it probably does
successfully install the probes in the correct place, unless a race occurs.

Additionally, I see that the crash doesn't necessarily occurs in the same
place. This could be the same problem or a separate one. For instance, one of
them:

Jan 18 05:37:36 ldsm kernel: [   17.113464] WARNING: CPU: 0 PID: 2759 at
/build/buildd/linux-3.11.0/kernel/trace/ftrace.c:1701 ftrace_bug+0x206/0x270()
Jan 18 05:37:36 ldsm kernel: [   17.113465] Modules linked in: gc__2757(OF+)
g_2759(OF+) gc_2751(OF) g_2745(OF) g_2742(OF) veth(F) arc4(F) md4(F) nls_utf8
cifs(F) fscache(F) openvswitch gre(F) snd_hda_intel cirrus snd_hda_codec ttm
drm_kms_helper microcode(F) snd_hwdep(F) psmouse(F) snd_pcm(F) serio_raw(F)
snd_page_alloc(F) drm virtio_balloon(F) snd_timer(F) snd(F) soundcore(F)
syscopyarea(F) sysfillrect(F) sysimgblt(F) i2c_piix4 mac_hid lp(F) parport(F)
ext2(F) 8139too(F) 8139cp(F) mii(F) floppy(F)
Jan 18 05:37:36 ldsm kernel: [   17.113498] CPU: 0 PID: 2759 Comm: staprun
Tainted: GF          O 3.11.0-12-generic #19-Ubuntu
Jan 18 05:37:36 ldsm kernel: [   17.113500] Hardware name: QEMU Standard PC
(i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Jan 18 05:37:36 ldsm kernel: [   17.113501]  0000000000000009 ffff88006a99bc30
ffffffff816e547a 0000000000000000
Jan 18 05:37:36 ldsm kernel: [   17.113504]  ffff88006a99bc68 ffffffff81061dbd
0000000000000000 ffffffffa0358000
Jan 18 05:37:36 ldsm kernel: [   17.113506]  ffff88007b735b80 0000000000000000
ffff880069d85000 ffff88006a99bc78
Jan 18 05:37:36 ldsm kernel: [   17.113508] Call Trace:
Jan 18 05:37:36 ldsm kernel: [   17.113514]  [<ffffffff816e547a>]
dump_stack+0x45/0x56
Jan 18 05:37:36 ldsm kernel: [   17.113517]  [<ffffffff81061dbd>]
warn_slowpath_common+0x7d/0xa0
Jan 18 05:37:36 ldsm kernel: [   17.113520]  [<ffffffffa0358000>] ?
0xffffffffa0357fff
Jan 18 05:37:36 ldsm kernel: [   17.113522]  [<ffffffff81061e9a>]
warn_slowpath_null+0x1a/0x20
Jan 18 05:37:36 ldsm kernel: [   17.113525]  [<ffffffff81108566>]
ftrace_bug+0x206/0x270
Jan 18 05:37:36 ldsm kernel: [   17.113527]  [<ffffffffa0358000>] ?
0xffffffffa0357fff
Jan 18 05:37:36 ldsm kernel: [   17.113529]  [<ffffffff811088da>]
ftrace_process_locs+0x30a/0x640
Jan 18 05:37:36 ldsm kernel: [   17.113532]  [<ffffffff81108c4c>]
ftrace_module_notify_enter+0x3c/0x40
Jan 18 05:37:36 ldsm kernel: [   17.113535]  [<ffffffff816f0a7c>]
notifier_call_chain+0x4c/0x70
Jan 18 05:37:36 ldsm kernel: [   17.113539]  [<ffffffff8108a1dd>]
__blocking_notifier_call_chain+0x4d/0x70
Jan 18 05:37:36 ldsm kernel: [   17.113541]  [<ffffffff8108a216>]
blocking_notifier_call_chain+0x16/0x20
Jan 18 05:37:36 ldsm kernel: [   17.113544]  [<ffffffff810cbd3f>]
load_module+0x125f/0x1b80
Jan 18 05:37:36 ldsm kernel: [   17.113546]  [<ffffffff810c7c60>] ?
store_uevent+0x40/0x40
Jan 18 05:37:36 ldsm kernel: [   17.113550]  [<ffffffff810cc702>]
SyS_init_module+0xa2/0xf0
Jan 18 05:37:36 ldsm kernel: [   17.113552]  [<ffffffff816f542f>]
tracesys+0xe1/0xe6
Jan 18 05:37:36 ldsm kernel: [   17.113554] ---[ end trace 41fb784a51ea714c
]---
Jan 18 05:37:36 ldsm kernel: [   17.113555] ftrace faulted on writing
[<ffffffffa0358000>] stp_task_work_cancel+0x0/0x20 [g_2759]
Jan 18 05:37:36 ldsm kernel: [   17.121994] gc_2751: systemtap: 2.6/0.157,
base: ffffffffa0319000, memory: 195data/52text/960ctx/2058net/9alloc kb,
probes: 2
Jan 18 05:37:36 ldsm kernel: [   17.183226] g_2759: systemtap: 2.6/0.157, base:
ffffffffa0358000, memory: 191data/48text/448ctx/2058net/9alloc kb, probes: 2

But it also crashes in other places.

-- 
You are receiving this mail because:
You are the assignee for the bug.