[Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x

[Bug runtime/12645] New: exelib.exp failures (and hung stapio's) on s390x

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

           Summary: exelib.exp failures (and hung stapio's) on s390x
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
        AssignedTo: systemtap@sourceware.org
        ReportedBy: dsmith@redhat.com


kernel: 2.6.32-125.el6.s390x

On s390x, the exelib.exp testcase gets 72 passes and 124 unexpected failures. 
Even though commit 273e7bf updated that test to kill stap when the test fails,
9 stapio processes get left behind:

# ps x | fgrep stapio
10937 ?        Rl     8:23 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe -u
/tmp/stapz81ubu/stap_3124cb608964f80ab49b78d36c0d613e_10311.ko
11387 ?        Rl     7:19 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-sep-debug-uprobeslibgcc-O3default-prelink-debug_exe -u
/tmp/stapbnXVPL/stap_e0453cf16865566fe0db1126fc5ba817_10371.ko
11851 ?        Rl     6:25 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-pie-debug-uprobeslibgcc-O3default-prelink-debug_exe -u
/tmp/stapuhQmTJ/stap_2b7361788bf0f1ab00a7ef97d97882f7_10325.ko
12324 ?        Rl     5:35 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-pie-sep-debug-uprobeslibgcc-O3default-prelink-debug_exe
-u /tmp/stapsFzAmD/stap_632f491261ce1ffd759ac0b8972667fd_10385.ko
14581 ?        Rl     4:08 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-sep-debug-after_exe
-u /tmp/stapMqlmmN/stap_23d8e70dafb27f52d8aebd7623048c1d_10611.ko
15030 ?        Rl     3:39 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-sep-debug-uprobeslibgcc-O3default-prelink-sep-debug-after_exe
-u /tmp/stapz9fPix/stap_aaca3cde249673e0a0f3f3a5f00d5004_10671.ko
15480 ?        Rl     3:07 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-pie-debug-uprobeslibgcc-O3default-prelink-sep-debug-after_exe
-u /tmp/stap1Hv065/stap_bda7ff911f65c3a752b4db9772267f9b_10625.ko
15931 ?        Rl     2:41 /usr/local/libexec/systemtap/stapio -w -c
./uprobesgcc-O3default-pie-sep-debug-uprobeslibgcc-O3default-prelink-sep-debug-after_exe
-u /tmp/stap53sDwo/stap_1753fc2af1aad9dd913c37c6957f6bf0_10685.ko

These stapio processes cannot be killed (even a 'kill -9' fails).  The modules
left behind are all "in use", so they cannot be removed.  The stapio processes
stay in the running state.

Even if this testcase (and related user-space probing) fail, stap should exit
cleanly and not leave behind these stuck stapio process and modules.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

--- Comment #1 from David Smith <dsmith at redhat dot com> 2011-04-18 19:20:44 UTC ---
From some additional debugging, here's how to duplicate the first failure
without running the entire testcase:

# gcc /root/src/testsuite/systemtap.exelib/uprobes_lib.c 
-isystem/root/src/testsuite -isystem/usr/local/include -g -O3 -fPIC -shared 
-lm   -o ./libuprobeslibgcc-O3default-prelink-debug.so
# /usr/sbin/prelink -vfNR -r 0x6400000
./libuprobeslibgcc-O3default-prelink-debug.so
# gcc /root/src/testsuite/systemtap.exelib/uprobes_exe.c 
-isystem/root/src/testsuite -isystem/usr/local/include -g -O3 -L.
-luprobeslibgcc-O3default-prelink-debug -Wl,-rpath,.  -lm   -o
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
# stap -w --ldd /root/src/testsuite/systemtap.exelib/libmarkunamestack.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe

Here's the output you get when running the above stap command:

====
main
exe: main=main
main_func: 3
exe: main_func=main_func
print_ubacktrace exe 0
 0x8000067c : main_func+0x0/0x60
[...cc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe]
main_count: 3
main_func: 2
exe: main_func=main_func
print_ustack exe 1
 0x8000067c : main_func+0x0/0x60
[...cc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe]
main_count: 2
main_func: 1
exe: main_func=main_func
main_count: 1
lib_main
lib: lib_main=lib_main
lib_func: 3
lib: lib_func=lib_func
print_ubacktrace lib 2
 0x6400608 : lib_func+0x0/0x48
[/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so]
====

At that point stap hangs.  You can kill it, but stapio still runs and the
module remains loaded.

Note that without the prelink step (the 2nd command above), everything works
correctly.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

--- Comment #2 from David Smith <dsmith at redhat dot com> 2011-04-18 21:21:48 UTC ---
Here's a further update.  Normally, the exelib.exp test uses the
libmarkunamestack.stp script which tests several things at once.  Optionally,
you can run lib.stp, mark.stp, uname.stp, or ustack.stp to test things
individually.  The hang doesn't occur when using lib.stp, uname.stp, or
ustack.stp - only when using mark.stp.  The mark.stp script tests userspace
probes being hit in the main program and the shared library.

# stap -w --ldd /root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe 
main_count: 3
main_count: 2
main_count: 1

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

--- Comment #3 from David Smith <dsmith at redhat dot com> 2011-04-18 22:11:55 UTC ---
As mentioned in comment #1, without prelink the test passes.  Here's the output
from the run without prelink (with DEBUG_UPROBES added):

=========
# stap -v -w -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 190usr/10sys/412real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 0usr/0sys/6real ms.
Pass 3: using cached
/root/.systemtap/cache/0e/stap_0e0c9728ef76b1d488292965fb7009de_1911.c
Pass 4: using cached
/root/.systemtap/cache/0e/stap_0e0c9728ef76b1d488292965fb7009de_1911.ko
Pass 5: starting run.
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[9900]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[9900]
addr 0000020000005614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 2 process uprobesgcc-O3de[9900]
addr 0000020000006614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
main_count: 3
main_count: 2
main_count: 1
func_count: 3
func_count: 2
func_count: 1
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 9900 addr
0000000080000688
systemtap_module_exit:719: -uprobe spec 1 index 1 pid 9900 addr
0000020000005614
systemtap_module_exit:719: -uprobe spec 1 index 2 pid 9900 addr
0000020000006614
Pass 5: run completed in 0usr/40sys/152real ms.
=========

Here's the failing run after the library has been prelinked:

=========
# stap -v -w -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 190usr/10sys/426real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/10sys/24real ms.
Pass 3: translated to C into
"/tmp/stapL0D0vy/stap_544bd75ae2d773f58b6dc536e78c9618_1919.c" using
200076virt/23500res/3584shr kb, in 90usr/50sys/275real ms.
Pass 4: compiled C into "stap_544bd75ae2d773f58b6dc536e78c9618_1919.ko" in
3060usr/560sys/16876real ms.
Pass 5: starting run.
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[10342]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[10342]
addr 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[10342]
addr 0000000006401614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
main_count: 3
main_count: 2
main_count: 1
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 10342 addr
0000000080000688
=========

The marker in the library isn't being hit in the prelinked case, which suggests
systemtap is calculating the wrong address.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

--- Comment #4 from David Smith <dsmith at redhat dot com> 2011-04-18 22:31:21 UTC ---
Here's a run (with a non-prelinked .so) adding DEBUG_TASK_FINDER_VMA:

============
# stap -v -w -DDEBUG_TASK_FINDER_VMA -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 190usr/20sys/805real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/0sys/41real ms.
Pass 3: translated to C into
"/tmp/stapBCJPs1/stap_21b14b43fda22050ae3798a72081e5a8_1932.c" using
200076virt/23500res/3584shr kb, in 100usr/210sys/5099real ms.
Pass 4: compiled C into "stap_21b14b43fda22050ae3798a72081e5a8_1932.ko" in
3110usr/510sys/9289real ms.
Pass 5: starting run.
stap_uprobe_process_munmap:334: +proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x80000000  0xb000  0x0 
r-xp  /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x8000b000  0x1000 
0xb000  rwxp  /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad4ef000  0x21000 
0x0  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad510000  0x1000 
0x20000  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad511000  0x1000 
0x21000  rwxp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad519000  0x1ad000 
0x0  r-xp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6c6000  0x4000 
0x1ac000  r-xp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6ca000  0x1000 
0x1b0000  rwxp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6d9000  0x1c000 
0x0  r-xp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6f5000  0x1000 
0x1c000  r-xp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6f6000  0x1000 
0x1d000  rwxp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000014000  0x5e91000 
0x0  r-xp  /usr/lib/locale/locale-archive
stap_uprobe_process_munmap:334: -proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_munmap:334: +proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x80000000  0x1000  0x0 
r-xp 
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x80001000  0x1000  0x0 
rwxp 
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad4ef000  0x21000 
0x0  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad510000  0x2000 
0x20000  rwxp  /lib64/ld-2.12.so
stap_uprobe_process_found:257: +proc pid 2227 stf 000003c001889528
000003c001889528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[2227]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000002000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000003000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000005000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000005000  0x2000 
0x0  r-xp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2227 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000005000
length 8192 offset (null) stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2227]
addr 0000020000005614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000006000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000006000  0x1000 
0x0  rwxp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2227 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000
length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 2 process uprobesgcc-O3de[2227]
addr 0000020000006614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
stap_uprobe_mmap_found:307: +mmap W pid 2227 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000
length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000007000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000007000  0xd000 
0x0  r-xp  /etc/ld.so.cache
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad6fd000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6fd000  0xa6000 
0x0  r-xp  /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad7a1000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad7a1000  0x2000 
0xa3000  rwxp  /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad519000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad519000  0x1b7000 
0x0  r-xp  /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad6c6000
__stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x4aad6c6000  0x5000 
0x1ac000  rwxp  /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x4aad6cb000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000014000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0),
returned 0x20000015000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found
munmap(0x20000007000), returned 0x0
stap_uprobe_munmap_found:322: -mmap pid 2227 addr 0000020000007000 length 49422
stf 000003c0018895f0 000003c0018895f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
main_count: 3
main_count: 2
main_count: 1
func_count: 3
func_count: 2
func_count: 1
stap_uprobe_process_munmap:334: -proc pid 2227 stf 000003c0018895f0
000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_found:257: -proc pid 2227 stf 000003c001889528
000003c001889528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 2227 addr
0000000080000688
systemtap_module_exit:719: -uprobe spec 1 index 1 pid 2227 addr
0000020000005614
systemtap_module_exit:719: -uprobe spec 1 index 2 pid 2227 addr
0000020000006614
Pass 5: run completed in 10usr/60sys/225real ms.
============

Here's a failing run after prelinking:

============
# stap -v -w -DDEBUG_TASK_FINDER_VMA -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
Pass 1: parsed user script and 76 library script(s) using
125672virt/21744res/2648shr kb, in 180usr/10sys/253real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/0sys/7real ms.
Pass 3: translated to C into
"/tmp/stapiM5Oh7/stap_60fe9b07c681cb8fbcbbaa2de526f0d3_1940.c" using
200076virt/23504res/3588shr kb, in 90usr/60sys/155real ms.
Pass 4: compiled C into "stap_60fe9b07c681cb8fbcbbaa2de526f0d3_1940.ko" in
3070usr/450sys/3742real ms.
Pass 5: starting run.
stap_uprobe_process_munmap:334: +proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x80000000  0xb000  0x0 
r-xp  /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x8000b000  0x1000 
0xb000  rwxp  /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad4ef000  0x21000 
0x0  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad510000  0x1000 
0x20000  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad511000  0x1000 
0x21000  rwxp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad519000  0x1ad000 
0x0  r-xp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6c6000  0x4000 
0x1ac000  r-xp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6ca000  0x1000 
0x1b0000  rwxp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6d9000  0x1c000 
0x0  r-xp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6f5000  0x1000 
0x1c000  r-xp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6f6000  0x1000 
0x1d000  rwxp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x20000014000  0x5e91000 
0x0  r-xp  /usr/lib/locale/locale-archive
stap_uprobe_process_munmap:334: -proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_munmap:334: +proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x80000000  0x1000  0x0 
r-xp 
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x80001000  0x1000  0x0 
rwxp 
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad4ef000  0x21000 
0x0  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad510000  0x2000 
0x20000  rwxp  /lib64/ld-2.12.so
stap_uprobe_process_found:257: +proc pid 2670 stf 000003c001bae528
000003c001bae528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[2670]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000002000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000003000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x6400000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x6400000  0x2000  0x0 
r-xp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2670 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006400000
length 8192 offset (null) stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2670]
addr 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x6401000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x6401000  0x1000  0x0 
rwxp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 2670 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006401000
length 4096 offset (null) stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2670]
addr 0000000006401614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
stap_uprobe_mmap_found:307: +mmap W pid 2670 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006401000
length 4096 offset (null) stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000005000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x20000005000  0xd000 
0x0  r-xp  /etc/ld.so.cache
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad6fd000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6fd000  0xa6000 
0x0  r-xp  /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad7a1000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad7a1000  0x2000 
0xa3000  rwxp  /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad519000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad519000  0x1b7000 
0x0  r-xp  /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad6c6000
__stp_call_mmap_callbacks:611: pid 2670, a/l/o/p/path 0x4aad6c6000  0x5000 
0x1ac000  rwxp  /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x4aad6cb000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000012000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found mmap(0x0),
returned 0x20000013000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 2670 found
munmap(0x20000005000), returned 0x0
stap_uprobe_munmap_found:322: -mmap pid 2670 addr 0000020000005000 length 49422
stf 000003c001bae5f0 000003c001bae5f0 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
main_count: 3
main_count: 2
main_count: 1
stap_uprobe_process_munmap:334: -proc pid 2670 stf 000003c001bae5f0
000003c001bae5f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_found:257: -proc pid 2670 stf 000003c001bae528
000003c001bae528 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
systemtap_module_exit:719: -uprobe spec 0 index 0 pid 2670 addr
0000000080000688
============

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #5 from Josh Stone <jistone at redhat dot com> 2011-04-18 23:48:36 UTC ---
The prelink may be shifting things in a particularly bad way, but I suspect
that even the non-prelinked mode is still wrong.

(In reply to comment #4)
> Here's a run (with a non-prelinked .so) adding DEBUG_TASK_FINDER_VMA:
> [...]
> __stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0), returned 0x20000005000
> __stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000005000  0x2000  0x0  r-xp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_mmap_found:292: +mmap X pid 2227 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000005000 length 8192 offset (null) stf 000003c0018895f0 000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[2227] addr 0000020000005614 pp process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
> __stp_utrace_task_finder_target_syscall_exit:1435: tsk 2227 found mmap(0x0), returned 0x20000006000
> __stp_call_mmap_callbacks:611: pid 2227, a/l/o/p/path 0x20000006000  0x1000  0x0  rwxp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_mmap_found:292: +mmap X pid 2227 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000 length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
> stap_uprobe_change_plus:67: +uprobe spec 1 idx 2 process uprobesgcc-O3de[2227] addr 0000020000006614 pp process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x614)
> stap_uprobe_mmap_found:307: +mmap W pid 2227 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000020000006000 length 4096 offset (null) stf 000003c0018895f0 000003c0018895f0 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so

Note how the .so is being mapped twice, once at 0x20000005000 r-xp, then at
0x20000006000 rwxp.  This corresponds to the text and data mappings for the
shared object.  So we'd expect to see +uprobes only on the first mapping, then
+semaphore on the second.

The first thing I noticed in the code is that stap_uprobe_change_plus() doesn't
consider the given offset much, except to compute the relocated
sdt_sem_address.  But here that doesn't even matter, as both mappings have
offset zero.  The LOAD headers I got from the .so are:

  Type Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg
Align
  LOAD 0x000000 0x0000000006400000 0x0000000006400000 0x00079c 0x00079c R E
0x1000
  LOAD 0x0007a0 0x00000000064017a0 0x00000000064017a0 0x000200 0x000218 RW 
0x1000

Since the offsets are both in the same 4k-aligned region, they both get mmap
offset 0, though the VirtAddr is padded to compensate.

So, I think stap_uprobe_change_plus() really ought to be considering the offset
in its address computation, and we should also not even attempt uprobes for
writeable mappings.

Beyond that, I think there may still be some oddness with the semaphores, as
-p3 shows the .so sdt_sem_offset=0x6400000, which looks more like a relocation
address.  And it would certainly be nice to better understand why failure here
ends up locking up stapio and the module shutdown path.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12645] exelib.exp failures (and hung stapio's) on s390x

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12645

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #6 from David Smith <dsmith at redhat dot com> 2011-04-20 19:11:47 UTC ---
As Josh mentioned in comment #5, there are really 2 problems here.  The
original failure and the problem that ends up locking up stapio and the module
shutdown path.

There was an early exit in the generated
enter_uprobe_probe()/enter_uretprobe_probe() functions.  When the error was
hit, the function would exit without properly unlocking the context 'busy'
field.  This causes the endlessly looping module shutdown.  Fixed in the
following commits:

0d5561a599cacbfaa91bb8ffa23c40122c09aacc
6e8950296ae2019e5002c0ff688bcb73e30289a8
0e090c748f667c810a5737104a25296040b9a34c

The original failure was in attempting to put a uprobes in a writable
executable mapping.  This was fixed in the following commit:

d2696bbab3b755a17098b756eb7238267339dea4

With these fixes, exelib.exp no longer leaves behind "stuck" stapio process and
modules.  Note that since backtraces don't work too well on s390x the test
still doesn't pass.

Here's a run with the new code on the prelinked library:

============
# stap -kv -w -DDEBUG_TASK_FINDER_VMA -DDEBUG_UPROBES --ldd
/root/src/testsuite/systemtap.exelib/mark.stp
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
./libuprobeslibgcc-O3default-prelink-debug.so -c
./uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe 
Pass 1: parsed user script and 76 library script(s) using
125672virt/21752res/2656shr kb, in 180usr/20sys/301real ms.
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 126220virt/22884res/3224shr kb, in 10usr/0sys/74real ms.
Pass 3: translated to C into "/tmp/stap77evIc/stap_41853.c" using
200068virt/23428res/3516shr kb, in 100usr/100sys/7355real ms.
Pass 4: compiled C into "stap_41853.ko" in 6140usr/990sys/10083real ms.
Pass 5: starting run.
stap_uprobe_process_munmap:334: +proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x80000000  0xb000  0x0 
r-xp  /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x8000b000  0x1000 
0xb000  rwxp  /usr/local/libexec/systemtap/stapio
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad4ef000  0x21000 
0x0  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad510000  0x1000 
0x20000  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad511000  0x1000 
0x21000  rwxp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad519000  0x1ad000 
0x0  r-xp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6c6000  0x4000 
0x1ac000  r-xp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6ca000  0x1000 
0x1b0000  rwxp  /lib64/libc-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6d9000  0x1c000 
0x0  r-xp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6f5000  0x1000 
0x1c000  r-xp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6f6000  0x1000 
0x1d000  rwxp  /lib64/libpthread-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x20000014000  0x5e91000
 0x0  r-xp  /usr/lib/locale/locale-archive
stap_uprobe_process_munmap:334: -proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_process_munmap:334: +proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x80000000  0x1000  0x0 
r-xp 
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x80001000  0x1000  0x0 
rwxp 
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad4ef000  0x21000 
0x0  r-xp  /lib64/ld-2.12.so
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad510000  0x2000 
0x20000  rwxp  /lib64/ld-2.12.so
stap_uprobe_process_found:257: +proc pid 42392 stf 000003c0005ea620
000003c0005ea620 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobesgcc-O3de[42392]
addr 0000000080000688 pp
process("/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe").statement(0x80000688)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000002000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000003000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x6400000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x6400000  0x2000  0x0 
r-xp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:292: +mmap X pid 42392 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006400000
length 8192 offset (null) stf 000003c0005ea6e8 000003c0005ea6e8 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_plus:67: +uprobe spec 1 idx 1 process uprobesgcc-O3de[42392]
addr 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x6401000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x6401000  0x1000  0x0 
rwxp  /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_mmap_found:307: +mmap W pid 42392 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so addr 0000000006401000
length 4096 offset (null) stf 000003c0005ea6e8 000003c0005ea6e8 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000005000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x20000005000  0xd000 
0x0  r-xp  /etc/ld.so.cache
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad6fd000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6fd000  0xa6000 
0x0  r-xp  /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad7a1000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad7a1000  0x2000 
0xa3000  rwxp  /lib64/libm-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad519000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad519000  0x1b7000 
0x0  r-xp  /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad6c6000
__stp_call_mmap_callbacks:611: pid 42392, a/l/o/p/path 0x4aad6c6000  0x5000 
0x1ac000  rwxp  /lib64/libc-2.12.so
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x4aad6cb000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000012000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found mmap(0x0),
returned 0x20000013000
__stp_utrace_task_finder_target_syscall_exit:1435: tsk 42392 found
munmap(0x20000005000), returned 0x0
stap_uprobe_munmap_found:322: -mmap pid 42392 addr 0000020000005000 length
49422 stf 000003c0005ea6e8 000003c0005ea6e8 path
/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
main_count: 3
main_count: 2
main_count: 1
func_count: 3
func_count: 2
func_count: 1
stap_uprobe_process_munmap:334: -proc pid 42392 stf 000003c0005ea6e8
000003c0005ea6e8 path /root/s390x/libuprobeslibgcc-O3default-prelink-debug.so
stap_uprobe_change_minus:229: -uprobe spec 1 idx 1 process
uprobesgcc-O3de[42392] reloc 0000000006400614 pp
process("/root/s390x/libuprobeslibgcc-O3default-prelink-debug.so").statement(0x6400614)
stap_uprobe_process_found:257: -proc pid 42392 stf 000003c0005ea620
000003c0005ea620 path
/root/s390x/uprobesgcc-O3default-debug-uprobeslibgcc-O3default-prelink-debug_exe
systemtap_module_exit:729: -uprobe spec 0 index 0 pid 42392 addr
0000000080000688
Pass 5: run completed in 20usr/60sys/809real ms.
Keeping temporary directory "/tmp/stap77evIc"
============

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.