* [Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x
2011-04-06 20:08 [Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x dsmith at redhat dot com
@ 2011-04-18 19:21 ` dsmith at redhat dot com
2011-04-18 21:22 ` dsmith at redhat dot com
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: dsmith at redhat dot com @ 2011-04-18 19:21 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=12645
--- Comment #1 from David Smith <dsmith at redhat dot com> 2011-04-18 19:20:44 UTC ---
From some additional debugging, here's how to duplicate the first failure
without running the entire testcase:
# gcc /root/src/testsuite/systemtap.exelib/uprobes_lib.c
-isystem/root/src/testsuite -isystem/usr/local/include -g -O3 -fPIC -shared
-lm -o ./libuprobeslibgcc-O3default-prelink-debug.so
# /usr/sbin/prelink -vfNR -r 0x6400000
./libuprobeslibgcc-O3default-prelink-debug.so
# gcc /root/src/testsuite/systemtap.exelib/uprobes_exe.c
-isystem/root/src/testsuite -isystem/usr/local/include -g -O3 -L.
-luprobeslibgcc-O3default-prelink-debug -Wl,-rpath,. -lm -o
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
# stap -w --ldd /root/src/testsuite/systemtap.exelib/libmarkunamestack.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Here's the output you get when running the above stap command:
====
main
exe: main=main
main_func: 3
exe: main_func=main_func
print_ubacktrace exe 0
0x8000067c : main_func+0x0/0x60
[...cc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe]
main_count: 3
main_func: 2
exe: main_func=main_func
print_ustack exe 1
0x8000067c : main_func+0x0/0x60
[...cc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe]
main_count: 2
main_func: 1
exe: main_func=main_func
main_count: 1
lib_main
lib: lib_main=lib_main
lib_func: 3
lib: lib_func=lib_func
print_ubacktrace lib 2
0x6400608 : lib_func+0x0/0x48
[/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so]
====
At that point stap hangs. You can kill it, but stapio still runs and the
module remains loaded.
Note that without the prelink step (the 2nd command above), everything works
correctly.
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x
2011-04-06 20:08 [Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x dsmith at redhat dot com
2011-04-18 19:21 ` [Bug runtime/12645] " dsmith at redhat dot com
@ 2011-04-18 21:22 ` dsmith at redhat dot com
2011-04-18 22:12 ` dsmith at redhat dot com
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: dsmith at redhat dot com @ 2011-04-18 21:22 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=12645
--- Comment #2 from David Smith <dsmith at redhat dot com> 2011-04-18 21:21:48 UTC ---
Here's a further update. Normally, the exelib.exp test uses the
libmarkunamestack.stp script which tests several things at once. Optionally,
you can run lib.stp, mark.stp, uname.stp, or ustack.stp to test things
individually. The hang doesn't occur when using lib.stp, uname.stp, or
ustack.stp - only when using mark.stp. The mark.stp script tests userspace
probes being hit in the main program and the shared library.
# stap -w --ldd /root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
main_count: 3
main_count: 2
main_count: 1
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x
2011-04-06 20:08 [Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x dsmith at redhat dot com
2011-04-18 19:21 ` [Bug runtime/12645] " dsmith at redhat dot com
2011-04-18 21:22 ` dsmith at redhat dot com
@ 2011-04-18 22:12 ` dsmith at redhat dot com
2011-04-18 22:32 ` dsmith at redhat dot com
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: dsmith at redhat dot com @ 2011-04-18 22:12 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=12645
--- Comment #3 from David Smith <dsmith at redhat dot com> 2011-04-18 22:11:55 UTC ---
As mentioned in comment #1, without prelink the test passes. Here's the output
from the run without prelink (with DEBUG_UPROBES added):
=========
# stap -v -w -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 190usr/10sys/412real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 0usr/0sys/6real ms.
Pass 3: using cached
/root/.systemtap/cache/0e/stap_0e0c9728ef76b1d488292965fb7009de_1911.c
Pass 4: using cached
/root/.systemtap/cache/0e/stap_0e0c9728ef76b1d488292965fb7009de_1911.ko
Pass 5: starting run.
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[9900]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[9900]
addr 0000020000005614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 2 process uprobesgcc-O3de[9900]
addr 0000020000006614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
main_count: 3
main_count: 2
main_count: 1
func_count: 3
func_count: 2
func_count: 1
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 9900 addr
0000000080000688
systemtap_module_exit:719: -uprobe spec 1 index 1 pid 9900 addr
0000020000005614
systemtap_module_exit:719: -uprobe spec 1 index 2 pid 9900 addr
0000020000006614
Pass 5: run completed in 0usr/40sys/152real ms.
=========
Here's the failing run after the library has been prelinked:
=========
# stap -v -w -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 190usr/10sys/426real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/10sys/24real ms.
Pass 3: translated to C into
"/tmp/stapL0D0vy/stap_544bd75ae2d773f58b6dc536e78c9618_1919.c" using
200076virt/23500res/3584shr kb, in 90usr/50sys/275real ms.
Pass 4: compiled C into "stap_544bd75ae2d773f58b6dc536e78c9618_1919.ko" in
3060usr/560sys/16876real ms.
Pass 5: starting run.
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[10342]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[10342]
addr 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[10342]
addr 0000000006401614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
main_count: 3
main_count: 2
main_count: 1
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 10342 addr
0000000080000688
=========
The marker in the library isn't being hit in the prelinked case, which suggests
systemtap is calculating the wrong address.
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x
2011-04-06 20:08 [Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x dsmith at redhat dot com
` (2 preceding siblings ...)
2011-04-18 22:12 ` dsmith at redhat dot com
@ 2011-04-18 22:32 ` dsmith at redhat dot com
2011-04-18 23:49 ` jistone at redhat dot com
2011-04-20 19:12 ` dsmith at redhat dot com
5 siblings, 0 replies; 7+ messages in thread
From: dsmith at redhat dot com @ 2011-04-18 22:32 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=12645
--- Comment #4 from David Smith <dsmith at redhat dot com> 2011-04-18 22:31:21 UTC ---
Here's a run (with a non-prelinked .so) adding DEBUG_TASK_FINDER_VMA:
============
# stap -v -w -DDEBUG_TASK_FINDER_VMA -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 190usr/20sys/805real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/0sys/41real ms.
Pass 3: translated to C into
"/tmp/stapBCJPs1/stap_21b14b43fda22050ae3798a72081e5a8_1932.c" using
200076virt/23500res/3584shr kb, in 100usr/210sys/5099real ms.
Pass 4: compiled C into "stap_21b14b43fda22050ae3798a72081e5a8_1932.ko" in
3110usr/510sys/9289real ms.
Pass 5: starting run.
stap_uprobe_process_munmap:334: +proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x80000000 0xb000 0x0
r-xp /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x8000b000 0x1000
0xb000 rwxp /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad4ef000 0x21000
0x0 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad510000 0x1000
0x20000 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad511000 0x1000
0x21000 rwxp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad519000 0x1ad000
0x0 r-xp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6c6000 0x4000
0x1ac000 r-xp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6ca000 0x1000
0x1b0000 rwxp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6d9000 0x1c000
0x0 r-xp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6f5000 0x1000
0x1c000 r-xp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6f6000 0x1000
0x1d000 rwxp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000014000 0x5e91000
0x0 r-xp /usr/lib/locale/locale-archive
stap_uprobe_process_munmap:334: -proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_munmap:334: +proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x80000000 0x1000 0x0
r-xp
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x80001000 0x1000 0x0
rwxp
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad4ef000 0x21000
0x0 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad510000 0x2000
0x20000 rwxp /lib64/ld-2.12.so
stap_uprobe_process_found:257: +proc pid 2227 stf 000003c001889528
000003c001889528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[2227]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000002000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000003000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000005000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000005000 0x2000
0x0 r-xp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2227 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000005000
length 8192 offset (null) stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2227]
addr 0000020000005614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000006000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000006000 0x1000
0x0 rwxp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2227 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000
length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 2 process uprobesgcc-O3de[2227]
addr 0000020000006614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
stap_uprobe_mmap_found:307: +mmap W pid 2227 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000
length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000007000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000007000 0xd000
0x0 r-xp /etc/ld.so.cache
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad6fd000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6fd000 0xa6000
0x0 r-xp /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad7a1000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad7a1000 0x2000
0xa3000 rwxp /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad519000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad519000 0x1b7000
0x0 r-xp /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad6c6000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6c6000 0x5000
0x1ac000 rwxp /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad6cb000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000014000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000015000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found
munmap(0x20000007000), returned 0x0
stap_uprobe_munmap_found:322: -mmap pid 2227 addr 0000020000007000 length 49422
stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
main_count: 3
main_count: 2
main_count: 1
func_count: 3
func_count: 2
func_count: 1
stap_uprobe_process_munmap:334: -proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_found:257: -proc pid 2227 stf 000003c001889528
000003c001889528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 2227 addr
0000000080000688
systemtap_module_exit:719: -uprobe spec 1 index 1 pid 2227 addr
0000020000005614
systemtap_module_exit:719: -uprobe spec 1 index 2 pid 2227 addr
0000020000006614
Pass 5: run completed in 10usr/60sys/225real ms.
============
Here's a failing run after prelinking:
============
# stap -v -w -DDEBUG_TASK_FINDER_VMA -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 180usr/10sys/253real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/0sys/7real ms.
Pass 3: translated to C into
"/tmp/stapiM5Oh7/stap_60fe9b07c681cb8fbcbbaa2de526f0d3_1940.c" using
200076virt/23504res/3588shr kb, in 90usr/60sys/155real ms.
Pass 4: compiled C into "stap_60fe9b07c681cb8fbcbbaa2de526f0d3_1940.ko" in
3070usr/450sys/3742real ms.
Pass 5: starting run.
stap_uprobe_process_munmap:334: +proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x80000000 0xb000 0x0
r-xp /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x8000b000 0x1000
0xb000 rwxp /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad4ef000 0x21000
0x0 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad510000 0x1000
0x20000 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad511000 0x1000
0x21000 rwxp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad519000 0x1ad000
0x0 r-xp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6c6000 0x4000
0x1ac000 r-xp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6ca000 0x1000
0x1b0000 rwxp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6d9000 0x1c000
0x0 r-xp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6f5000 0x1000
0x1c000 r-xp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6f6000 0x1000
0x1d000 rwxp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x20000014000 0x5e91000
0x0 r-xp /usr/lib/locale/locale-archive
stap_uprobe_process_munmap:334: -proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_munmap:334: +proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x80000000 0x1000 0x0
r-xp
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x80001000 0x1000 0x0
rwxp
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad4ef000 0x21000
0x0 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad510000 0x2000
0x20000 rwxp /lib64/ld-2.12.so
stap_uprobe_process_found:257: +proc pid 2670 stf 000003c001bae528
000003c001bae528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[2670]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000002000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000003000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x6400000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x6400000 0x2000 0x0
r-xp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2670 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006400000
length 8192 offset (null) stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2670]
addr 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x6401000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x6401000 0x1000 0x0
rwxp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2670 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006401000
length 4096 offset (null) stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2670]
addr 0000000006401614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
stap_uprobe_mmap_found:307: +mmap W pid 2670 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006401000
length 4096 offset (null) stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000005000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x20000005000 0xd000
0x0 r-xp /etc/ld.so.cache
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad6fd000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6fd000 0xa6000
0x0 r-xp /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad7a1000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad7a1000 0x2000
0xa3000 rwxp /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad519000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad519000 0x1b7000
0x0 r-xp /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad6c6000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6c6000 0x5000
0x1ac000 rwxp /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad6cb000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000012000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000013000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found
munmap(0x20000005000), returned 0x0
stap_uprobe_munmap_found:322: -mmap pid 2670 addr 0000020000005000 length 49422
stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
main_count: 3
main_count: 2
main_count: 1
stap_uprobe_process_munmap:334: -proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_found:257: -proc pid 2670 stf 000003c001bae528
000003c001bae528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 2670 addr
0000000080000688
============
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x
2011-04-06 20:08 [Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x dsmith at redhat dot com
` (3 preceding siblings ...)
2011-04-18 22:32 ` dsmith at redhat dot com
@ 2011-04-18 23:49 ` jistone at redhat dot com
2011-04-20 19:12 ` dsmith at redhat dot com
5 siblings, 0 replies; 7+ messages in thread
From: jistone at redhat dot com @ 2011-04-18 23:49 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=12645
Josh Stone <jistone at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jistone at redhat dot com
--- Comment #5 from Josh Stone <jistone at redhat dot com> 2011-04-18 23:48:36 UTC ---
The prelink may be shifting things in a particularly bad way, but I suspect
that even the non-prelinked mode is still wrong.
(In reply to comment #4)
> Here's a run (with a non-prelinked .so) adding DEBUG_TASK_FINDER_VMA:
> [...]
> __stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0), returned 0x20000005000
> __stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000005000 0x2000 0x0 r-xp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_mmap_found:292: +mmap X pid 2227 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000005000 length 8192 offset (null) stf 000003c0018895f0 000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2227] addr 0000020000005614 pp process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
> __stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0), returned 0x20000006000
> __stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000006000 0x1000 0x0 rwxp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_mmap_found:292: +mmap X pid 2227 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000 length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_change_plus:67: +uprobe spec 1 idx 2 process uprobesgcc-O3de[2227] addr 0000020000006614 pp process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
> stap_uprobe_mmap_found:307: +mmap W pid 2227 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000 length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
Note how the .so is being mapped twice, once at 0x20000005000 r-xp, then at
0x20000006000 rwxp. This corresponds to the text and data mappings for the
shared object. So we'd expect to see +uprobes only on the first mapping, then
+semaphore on the second.
The first thing I noticed in the code is that stap_uprobe_change_plus() doesn't
consider the given offset much, except to compute the relocated
sdt_sem_address. But here that doesn't even matter, as both mappings have
offset zero. The LOAD headers I got from the .so are:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg
Align
LOAD 0x000000 0x0000000006400000 0x0000000006400000 0x00079c 0x00079c R E
0x1000
LOAD 0x0007a0 0x00000000064017a0 0x00000000064017a0 0x000200 0x000218 RW
0x1000
Since the offsets are both in the same 4k-aligned region, they both get mmap
offset 0, though the VirtAddr is padded to compensate.
So, I think stap_uprobe_change_plus() really ought to be considering the offset
in its address computation, and we should also not even attempt uprobes for
writeable mappings.
Beyond that, I think there may still be some oddness with the semaphores, as
-p3 shows the .so sdt_sem_offset=0x6400000, which looks more like a relocation
address. And it would certainly be nice to better understand why failure here
ends up locking up stapio and the module shutdown path.
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x
2011-04-06 20:08 [Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x dsmith at redhat dot com
` (4 preceding siblings ...)
2011-04-18 23:49 ` jistone at redhat dot com
@ 2011-04-20 19:12 ` dsmith at redhat dot com
5 siblings, 0 replies; 7+ messages in thread
From: dsmith at redhat dot com @ 2011-04-20 19:12 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=12645
David Smith <dsmith at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #6 from David Smith <dsmith at redhat dot com> 2011-04-20 19:11:47 UTC ---
As Josh mentioned in comment #5, there are really 2 problems here. The
original failure and the problem that ends up locking up stapio and the module
shutdown path.
There was an early exit in the generated
enter_uprobe_probe()/enter_uretprobe_probe() functions. When the error was
hit, the function would exit without properly unlocking the context 'busy'
field. This causes the endlessly looping module shutdown. Fixed in the
following commits:
0d5561a599cacbfaa91bb8ffa23c40122c09aacc
6e8950296ae2019e5002c0ff688bcb73e30289a8
0e090c748f667c810a5737104a25296040b9a34c
The original failure was in attempting to put a uprobes in a writable
executable mapping. This was fixed in the following commit:
d2696bbab3b755a17098b756eb7238267339dea4
With these fixes, exelib.exp no longer leaves behind "stuck" stapio process and
modules. Note that since backtraces don't work too well on s390x the test
still doesn't pass.
Here's a run with the new code on the prelinked library:
============
# stap -kv -w -DDEBUG_TASK_FINDER_VMA -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21752res/2656shr kb, in 180usr/20sys/301real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/0sys/74real ms.
Pass 3: translated to C into "/tmp/stap77evIc/stap_41853.c" using
200068virt/23428res/3516shr kb, in 100usr/100sys/7355real ms.
Pass 4: compiled C into "stap_41853.ko" in 6140usr/990sys/10083real ms.
Pass 5: starting run.
stap_uprobe_process_munmap:334: +proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x80000000 0xb000 0x0
r-xp /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x8000b000 0x1000
0xb000 rwxp /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad4ef000 0x21000
0x0 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad510000 0x1000
0x20000 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad511000 0x1000
0x21000 rwxp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad519000 0x1ad000
0x0 r-xp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6c6000 0x4000
0x1ac000 r-xp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6ca000 0x1000
0x1b0000 rwxp /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6d9000 0x1c000
0x0 r-xp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6f5000 0x1000
0x1c000 r-xp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6f6000 0x1000
0x1d000 rwxp /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x20000014000 0x5e91000
0x0 r-xp /usr/lib/locale/locale-archive
stap_uprobe_process_munmap:334: -proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_munmap:334: +proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x80000000 0x1000 0x0
r-xp
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x80001000 0x1000 0x0
rwxp
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad4ef000 0x21000
0x0 r-xp /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad510000 0x2000
0x20000 rwxp /lib64/ld-2.12.so
stap_uprobe_process_found:257: +proc pid 42392 stf 000003c0005ea620
000003c0005ea620 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[42392]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000002000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000003000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x6400000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x6400000 0x2000 0x0
r-xp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 42392 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006400000
length 8192 offset (null) stf 000003c0005ea6e8 000003c0005ea6e8 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[42392]
addr 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x6401000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x6401000 0x1000 0x0
rwxp /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:307: +mmap W pid 42392 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006401000
length 4096 offset (null) stf 000003c0005ea6e8 000003c0005ea6e8 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000005000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x20000005000 0xd000
0x0 r-xp /etc/ld.so.cache
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad6fd000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6fd000 0xa6000
0x0 r-xp /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad7a1000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad7a1000 0x2000
0xa3000 rwxp /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad519000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad519000 0x1b7000
0x0 r-xp /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad6c6000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6c6000 0x5000
0x1ac000 rwxp /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad6cb000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000012000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000013000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found
munmap(0x20000005000), returned 0x0
stap_uprobe_munmap_found:322: -mmap pid 42392 addr 0000020000005000 length
49422 stf 000003c0005ea6e8 000003c0005ea6e8 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
main_count: 3
main_count: 2
main_count: 1
func_count: 3
func_count: 2
func_count: 1
stap_uprobe_process_munmap:334: -proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_minus:229: -uprobe spec 1 idx 1 process
uprobesgcc-O3de[42392] reloc 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
stap_uprobe_process_found:257: -proc pid 42392 stf 000003c0005ea620
000003c0005ea620 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
systemtap_module_exit:729: -uprobe spec 0 index 0 pid 42392 addr
0000000080000688
Pass 5: run completed in 20usr/60sys/809real ms.
Keeping temporary directory "/tmp/stap77evIc"
============
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread