From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 127649 invoked by alias); 20 Feb 2020 22:55:11 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 127610 invoked by uid 48); 20 Feb 2020 22:55:06 -0000 From: "dale.hamel at srvthe dot net" To: systemtap@sourceware.org Subject: [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable Date: Thu, 20 Feb 2020 22:55:00 -0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: systemtap X-Bugzilla-Component: releng X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: enhancement X-Bugzilla-Who: dale.hamel at srvthe dot net X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: systemtap at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-SW-Source: 2020-q1/txt/msg00041.txt https://sourceware.org/bugzilla/show_bug.cgi?id=3D25581 --- Comment #2 from Dale Hamel --- This minimal proof of concept shows the main concept employed by this patch: ```c #include #include void _check(); #define MY_ASM \ do { \ __asm__ __volatile__ ("_check:\n990: nop");\ } while(0) int main(int argc, char **argv) { while(1) { MY_ASM; printf("%08X\n", (*(char *)_check) & 0x90); sleep(1); } } ``` When I run the program, I see the output: ``` ... 00000090 00000090 00000090 ... ``` Now If I attach to it with bpftrace: ``` bpftrace -e 'uprobe:./a.out:0x1164 { printf("here\n") }' ``` Note that 0x1164 is from: ``` $readelf -s ./a.out| grep _check 35: 0000000000001164 0 NOTYPE LOCAL DEFAULT 13 _check ``` I can see that the value changes in the program: ``` 00000090 00000090 00000080 00000080 ``` Though I was expecting 0xCC, not 0x80. It seems like 0x80 is for a syscall? Unsure why, but it toggles as I attach/detach with bpftrace. I compiled ruby with `--enable-dtrace` with these sys/sdt and dtrace.py, and performed these tests: Given the ruby process: ``` ruby -e 'TracePoint.new{}.enable; def foo; puts "hi"; sleep 1; end; while t= rue do; foo;end' ``` Note that `TracePoint.new{}.enable` is needed post ruby 2.5, aside from tha= t we are just sleeping and calling `foo`. To start with, I was perplexed when I saw there were no sdt notes on /proc/RUBYPID/exe, but on checking /proc/RUBYPID/maps, I see that this is b= uilt with libruby.so, and the probes are in there. I check the elf notes on that, and find `method__entry`: ``` $ readelf --notes /proc/168586/root/usr/local/lib/libruby.so | grep method__entry -A1 Name: cmethod__entry Location: 0x0000000000237e5a, Base: 0x00000000002f7305, Semaphore: 0x0000000000000000 -- Name: method__entry Location: 0x000000000023819e, Base: 0x00000000002f7305, Semaphore: 0x0000000000000000 -- Name: method__entry Location: 0x0000000000243ec3, Base: 0x00000000002f7305, Semaphore: 0x0000000000000000 -- Name: cmethod__entry Location: 0x00000000002462e4, Base: 0x00000000002f7305, Semaphore: 0x0000000000000000 ``` There are multiple addresses for this probe that we might attach to, but on= ly one of them will actually be checked in the source. To determine this, we h= ave to find the hack function: ``` $ readelf -s /proc/168586/root/usr/local/lib/libruby.so | grep ruby_method__entry_check 3922: 000000000023819e 0 NOTYPE LOCAL DEFAULT 12 ruby_method__entry_check ``` So it is the one at the address `0x000000000023819e`. To translate this to = the vmaddr, we need the base of libruby.so, which we can read from /proc/RUBYPID/maps: ``` cat /proc/168586/maps | grep libruby 7f4f035f4000-7f4f0361f000 r--p 00000000 08:01 2637891=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 /usr/local/lib/libruby.so.2.6.5 7f4f0361f000-7f4f03852000 r-xp 0002b000 08:01 2637891=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 /usr/local/lib/libruby.so.2.6.5 7f4f03852000-7f4f03938000 r--p 0025e000 08:01 2637891=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 /usr/local/lib/libruby.so.2.6.5 7f4f03938000-7f4f0393e000 r--p 00343000 08:01 2637891=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 /usr/local/lib/libruby.so.2.6.5 7f4f0393e000-7f4f03941000 rw-p 00349000 08:01 2637891=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 /usr/local/lib/libruby.so.2.6.5 ``` So 0x7f4f035f4000 + 0x000000000023819e =3D 0x7f4f0382c19e=20 Now lets just check the data in memory at that address without anything attached: ``` $dd if=3D/proc/168586/mem count=3D1 bs=3D1 skip=3D$(( 0x7F4F0382C19E )) 2> = /dev/null | xxd 00000000: 90 . ``` It's the NOP exactly as expected. Now for the real magic, we attach bpftrac= e in one terminal: ``` $ bpftrace -e 'usdt::ruby:method__entry {printf("%s\n", str(arg1))}' -p 168= 586 ``` And immediately see it is printing `foo`, a good sign. But my `ENABLED` che= ck could still be broken right? So lets check the memory now that the probe is enabled: ``` dd if=3D/proc/168586/mem count=3D1 bs=3D1 skip=3D$(( 0x7F4F0382C19E )) 2> /= dev/null | xxd 00000000: cc . ``` Exactly as expected, the kernel has overwritten the NOP (0x90) with INT3 (0xCC). The macro to check this determines that 0x90 !=3D 0xCC, and returns true - = the probe is enabled by the uprobe itself. --=20 You are receiving this mail because: You are the assignee for the bug.