* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
@ 2020-02-20 2:02 ` dale.hamel at srvthe dot net
2020-02-20 2:03 ` dale.hamel at srvthe dot net
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-20 2:02 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |dale.hamel at srvthe dot net
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
2020-02-20 2:02 ` [Bug releng/25581] " dale.hamel at srvthe dot net
@ 2020-02-20 2:03 ` dale.hamel at srvthe dot net
2020-02-20 2:06 ` dale.hamel at srvthe dot net
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-20 2:03 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Target| |Kernel 4.14
Host| |Linux X86 Server
Build| |Google Container OS,
| |Milestone 62, build ID
| |10895
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
2020-02-20 2:02 ` [Bug releng/25581] " dale.hamel at srvthe dot net
2020-02-20 2:03 ` dale.hamel at srvthe dot net
@ 2020-02-20 2:06 ` dale.hamel at srvthe dot net
2020-02-20 22:55 ` dale.hamel at srvthe dot net
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-20 2:06 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
--- Comment #1 from Dale Hamel <dale.hamel at srvthe dot net> ---
This is the first bug I have created on here, hopefully I have submitted it
correctly. If I am out of order at all, please direct me to the right way to
submit this issue / proposal for enhancement.
This patch has been tested in production and works around the dependency on a
semaphore for popular programs like ruby, memcached, mysql, and redis, and for
using USDT probes without impacting performance when not attached.
Thanks for taking the time to review this bug, I look forward to hearing your
feedback and suggestions,
-Dale
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (2 preceding siblings ...)
2020-02-20 2:06 ` dale.hamel at srvthe dot net
@ 2020-02-20 22:55 ` dale.hamel at srvthe dot net
2020-02-21 0:18 ` dale.hamel at srvthe dot net
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-20 22:55 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
--- Comment #2 from Dale Hamel <dale.hamel at srvthe dot net> ---
This minimal proof of concept shows the main concept employed by this patch:
```c
#include <stdio.h>
#include <unistd.h>
void _check();
#define MY_ASM \
do { \
__asm__ __volatile__ ("_check:\n990: nop");\
} while(0)
int main(int argc, char **argv)
{
while(1) {
MY_ASM;
printf("%08X\n", (*(char *)_check) & 0x90);
sleep(1);
}
}
```
When I run the program, I see the output:
```
...
00000090
00000090
00000090
...
```
Now If I attach to it with bpftrace:
```
bpftrace -e 'uprobe:./a.out:0x1164 { printf("here\n") }'
```
Note that 0x1164 is from:
```
$readelf -s ./a.out| grep _check
35: 0000000000001164 0 NOTYPE LOCAL DEFAULT 13 _check
```
I can see that the value changes in the program:
```
00000090
00000090
00000080
00000080
```
Though I was expecting 0xCC, not 0x80. It seems like 0x80 is for a syscall?
Unsure why, but it toggles as I attach/detach with bpftrace.
I compiled ruby with `--enable-dtrace` with these sys/sdt and dtrace.py, and
performed these tests:
Given the ruby process:
```
ruby -e 'TracePoint.new{}.enable; def foo; puts "hi"; sleep 1; end; while true
do; foo;end'
```
Note that `TracePoint.new{}.enable` is needed post ruby 2.5, aside from that we
are just sleeping and calling `foo`.
To start with, I was perplexed when I saw there were no sdt notes on
/proc/RUBYPID/exe, but on checking /proc/RUBYPID/maps, I see that this is built
with libruby.so, and the probes are in there. I check the elf notes on that,
and find `method__entry`:
```
$ readelf --notes /proc/168586/root/usr/local/lib/libruby.so | grep
method__entry -A1
Name: cmethod__entry
Location: 0x0000000000237e5a, Base: 0x00000000002f7305, Semaphore:
0x0000000000000000
--
Name: method__entry
Location: 0x000000000023819e, Base: 0x00000000002f7305, Semaphore:
0x0000000000000000
--
Name: method__entry
Location: 0x0000000000243ec3, Base: 0x00000000002f7305, Semaphore:
0x0000000000000000
--
Name: cmethod__entry
Location: 0x00000000002462e4, Base: 0x00000000002f7305, Semaphore:
0x0000000000000000
```
There are multiple addresses for this probe that we might attach to, but only
one of them will actually be checked in the source. To determine this, we have
to find the hack function:
```
$ readelf -s /proc/168586/root/usr/local/lib/libruby.so | grep
ruby_method__entry_check
3922: 000000000023819e 0 NOTYPE LOCAL DEFAULT 12
ruby_method__entry_check
```
So it is the one at the address `0x000000000023819e`. To translate this to the
vmaddr, we need the base of libruby.so, which we can read from
/proc/RUBYPID/maps:
```
cat /proc/168586/maps | grep libruby
7f4f035f4000-7f4f0361f000 r--p 00000000 08:01 2637891
/usr/local/lib/libruby.so.2.6.5
7f4f0361f000-7f4f03852000 r-xp 0002b000 08:01 2637891
/usr/local/lib/libruby.so.2.6.5
7f4f03852000-7f4f03938000 r--p 0025e000 08:01 2637891
/usr/local/lib/libruby.so.2.6.5
7f4f03938000-7f4f0393e000 r--p 00343000 08:01 2637891
/usr/local/lib/libruby.so.2.6.5
7f4f0393e000-7f4f03941000 rw-p 00349000 08:01 2637891
/usr/local/lib/libruby.so.2.6.5
```
So 0x7f4f035f4000 + 0x000000000023819e = 0x7f4f0382c19e
Now lets just check the data in memory at that address without anything
attached:
```
$dd if=/proc/168586/mem count=1 bs=1 skip=$(( 0x7F4F0382C19E )) 2> /dev/null |
xxd
00000000: 90 .
```
It's the NOP exactly as expected. Now for the real magic, we attach bpftrace in
one terminal:
```
$ bpftrace -e 'usdt::ruby:method__entry {printf("%s\n", str(arg1))}' -p 168586
```
And immediately see it is printing `foo`, a good sign. But my `ENABLED` check
could still be broken right? So lets check the memory now that the probe is
enabled:
```
dd if=/proc/168586/mem count=1 bs=1 skip=$(( 0x7F4F0382C19E )) 2> /dev/null |
xxd
00000000: cc .
```
Exactly as expected, the kernel has overwritten the NOP (0x90) with INT3
(0xCC).
The macro to check this determines that 0x90 != 0xCC, and returns true - the
probe is enabled by the uprobe itself.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (3 preceding siblings ...)
2020-02-20 22:55 ` dale.hamel at srvthe dot net
@ 2020-02-21 0:18 ` dale.hamel at srvthe dot net
2020-02-21 3:35 ` dale.hamel at srvthe dot net
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 0:18 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #12303|0 |1
is obsolete| |
--- Comment #3 from Dale Hamel <dale.hamel at srvthe dot net> ---
Created attachment 12305
--> https://sourceware.org/bugzilla/attachment.cgi?id=12305&action=edit
Improved patch for style, make semaphore use optional
Cleaned up version of my original hard-coded patch.
I haven't tested this as extensively, but should be better to review
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (4 preceding siblings ...)
2020-02-21 0:18 ` dale.hamel at srvthe dot net
@ 2020-02-21 3:35 ` dale.hamel at srvthe dot net
2020-02-21 3:35 ` dale.hamel at srvthe dot net
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 3:35 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
--- Comment #4 from Dale Hamel <dale.hamel at srvthe dot net> ---
Created attachment 12306
--> https://sourceware.org/bugzilla/attachment.cgi?id=12306&action=edit
Additional fixes on cleaned-up patch
Found a couple of issues, will continue to tidy if i find more
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (5 preceding siblings ...)
2020-02-21 3:35 ` dale.hamel at srvthe dot net
@ 2020-02-21 3:35 ` dale.hamel at srvthe dot net
2020-02-21 3:48 ` dale.hamel at srvthe dot net
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 3:35 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #12305|0 |1
is obsolete| |
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (6 preceding siblings ...)
2020-02-21 3:35 ` dale.hamel at srvthe dot net
@ 2020-02-21 3:48 ` dale.hamel at srvthe dot net
2020-02-21 6:08 ` dale.hamel at srvthe dot net
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 3:48 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #12306|0 |1
is obsolete| |
--- Comment #5 from Dale Hamel <dale.hamel at srvthe dot net> ---
Created attachment 12307
--> https://sourceware.org/bugzilla/attachment.cgi?id=12307&action=edit
Additional fixes
Ran it and double checked the generated headers.
Sample without -S flag:
```
/* RUBY_METHOD_ENTRY ( const char *classname, const char *methodname, const
char *filename, int lineno ) */
#if defined STAP_SDT_V1
#define ruby_method__entry_semaphore method__entry_semaphore
#else
#define RUBY_METHOD_ENTRY_ENABLED() __builtin_expect
(ruby_method__entry_semaphore, 0)
#endif
__extension__ extern unsigned short ruby_method__entry_semaphore __attribute__
((unused)) __attribute__ ((section (".probes")));
#define RUBY_METHOD_ENTRY(arg1, arg2, arg3, arg4) \
DTRACE_PROBE4 (ruby, method__entry, arg1, arg2, arg3, arg4)
```
With the -S flag:
```
#define _SDT_HAS_SEMAPHORES 0
#define STAP_HAS_SEMAPHORES 0 /* deprecated */
#define __PLATFORM_UPROBE_ENABLED(provider, name)\
provider##_##name##_check != 0 && \
((*(char *) __##provider##_##name##_asm_check) & 0x90) != 0x90
#include <sys/sdt.h>
/* RUBY_METHOD_ENTRY ( const char *classname, const char *methodname, const
char *filename, int lineno ) */
#define RUBY_METHOD_ENTRY_ENABLED() __PLATFORM_UPROBE_ENABLED(ruby,
method__entry)
#define RUBY_METHOD_ENTRY(arg1, arg2, arg3, arg4) \
DTRACE_PROBE4 (ruby, method__entry, arg1, arg2, arg3, arg4)
void __ruby_method__entry_asm_check();
```
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (7 preceding siblings ...)
2020-02-21 3:48 ` dale.hamel at srvthe dot net
@ 2020-02-21 6:08 ` dale.hamel at srvthe dot net
2020-02-21 6:38 ` dale.hamel at srvthe dot net
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 6:08 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #12307|0 |1
is obsolete| |
--- Comment #6 from Dale Hamel <dale.hamel at srvthe dot net> ---
Created attachment 12308
--> https://sourceware.org/bugzilla/attachment.cgi?id=12308&action=edit
Additional testing and improvements
Verified this works end-to-end as my original patch, but more modular.
Behavior is preserved without -S.
If I create a wrapper for dtrace like:
```
#!/bin/bash
dtrace -S $@
```
I can use the uprobe approach to probe on a system where /proc/[pid]/mem is not
writeable.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (8 preceding siblings ...)
2020-02-21 6:08 ` dale.hamel at srvthe dot net
@ 2020-02-21 6:38 ` dale.hamel at srvthe dot net
2020-02-21 15:46 ` dale.hamel at srvthe dot net
2020-02-21 19:02 ` dale.hamel at srvthe dot net
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 6:38 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #12308|0 |1
is obsolete| |
--- Comment #7 from Dale Hamel <dale.hamel at srvthe dot net> ---
Created attachment 12309
--> https://sourceware.org/bugzilla/attachment.cgi?id=12309&action=edit
Fixes typo
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (9 preceding siblings ...)
2020-02-21 6:38 ` dale.hamel at srvthe dot net
@ 2020-02-21 15:46 ` dale.hamel at srvthe dot net
2020-02-21 19:02 ` dale.hamel at srvthe dot net
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 15:46 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
--- Comment #8 from Dale Hamel <dale.hamel at srvthe dot net> ---
I may have introduced a regression with this patch, as when I try to build ruby
with it with semaphores enabled it doesn't generate the necessary object file
anymore. I think that I may need to use a different value for falling back than
the current define of _HAS_SDT_SEMAPHORE
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug releng/25581] USDT probes when /proc/[pid]/mem not writeable
2020-02-20 2:00 [Bug releng/25581] New: USDT probes when /proc/[pid]/mem not writeable dale.hamel at srvthe dot net
` (10 preceding siblings ...)
2020-02-21 15:46 ` dale.hamel at srvthe dot net
@ 2020-02-21 19:02 ` dale.hamel at srvthe dot net
11 siblings, 0 replies; 13+ messages in thread
From: dale.hamel at srvthe dot net @ 2020-02-21 19:02 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=25581
Dale Hamel <dale.hamel at srvthe dot net> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #12309|0 |1
is obsolete| |
--- Comment #9 from Dale Hamel <dale.hamel at srvthe dot net> ---
Created attachment 12311
--> https://sourceware.org/bugzilla/attachment.cgi?id=12311&action=edit
Fixes regression where semaphores need shared object
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 13+ messages in thread