[Bug dyninst/16795] New: utrace_p5.exp leaves stapdyn processes running

[Bug dyninst/16795] New: utrace_p5.exp leaves stapdyn processes running

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16795

            Bug ID: 16795
           Summary: utrace_p5.exp leaves stapdyn processes running
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: dyninst
          Assignee: systemtap at sourceware dot org
          Reporter: jlebon at redhat dot com

During some runs of utrace_p5.exp, this failure can occur:

FAIL: UTRACE_P5_06_cmd_dyninst shutdown (timeout)

stap:[vm systemtap ((daa4b33...))]$ ps aux | grep utrace
root     31609  0.0  5.1 328188 105456 ?       Sl   18:35   0:00
/home/vm/codebase/systemtap/install/bin/stapdyn -c
/home/vm/codebase/systemtap/build/testsuite/utrace_p5_multi_31233
/tmp/stap67S2bK/stap_682284b48a759123a41de08383dfcf7f_1449.so
root     31613  0.0  0.0  46120   776 ?        tl   18:35   0:00
/home/vm/codebase/systemtap/build/testsuite/utrace_p5_multi_31233
root     32053  0.0  5.0 328188 103408 ?       Sl   18:36   0:01
/home/vm/codebase/systemtap/install/bin/stapdyn -c
/home/vm/codebase/systemtap/build/testsuite/utrace_p5_multi_31677
/tmp/stapflya1s/stap_e21127b193a834855e49e99e76eaf0a1_1449.so
root     32057  0.0  0.0  46120   772 ?        tl   18:36   0:00
/home/vm/codebase/systemtap/build/testsuite/utrace_p5_multi_31677
vm       32717  0.0  0.0 112632   936 pts/1    S+   18:56   0:00 grep
--color=auto utrace
stap:[vm systemtap ((daa4b33...))]$ sudo gdb -p 31609
GNU gdb (GDB) Fedora 7.6.1-46.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 31609
Reading symbols from /home/vm/codebase/systemtap/install/bin/stapdyn...done.
...
Reading symbols from /lib64/libselinux.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libselinux.so.1.debug...done.
Loaded symbols for /lib64/libthread_db.so
0x0000003accceb8bf in __GI_ppoll (fds=fds@entry=0x7fff81a23840,
nfds=nfds@entry=1,
    timeout=<optimized out>, timeout@entry=0x7fff81a23850,
sigmask=sigmask@entry=0x7fff81a23860)
    at ../sysdeps/unix/sysv/linux/ppoll.c:56
56            result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask,
Missing separate debuginfos, use: debuginfo-install dyninst-8.1.2-1.fc19.x86_64
elfutils-libelf-0.156-5.fc19.x86_64 libdwarf-20130207-1.fc19.x86_64
pcre-8.32-8.fc19.x86_64
(gdb) bt
#0  0x0000003accceb8bf in __GI_ppoll (fds=fds@entry=0x7fff81a23840,
nfds=nfds@entry=1,
    timeout=<optimized out>, timeout@entry=0x7fff81a23850,
sigmask=sigmask@entry=0x7fff81a23860)
    at ../sysdeps/unix/sysv/linux/ppoll.c:56
#1  0x000000000040db72 in ppoll (__ss=0x7fff81a23860, __timeout=0x7fff81a23850,
__nfds=1,
    __fds=0x7fff81a23840) at /usr/include/bits/poll2.h:77
#2  mutator::run (this=0x25bfe80) at ../../systemtap/stapdyn/mutator.cxx:586
#3  0x00000000004075b7 in main (argc=<optimized out>, argv=<optimized out>)
    at ../../systemtap/stapdyn/stapdyn.cxx:157
(gdb) thread apply all backtrace

Thread 5 (Thread 0x7faf84a94700 (LWP 31610)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007faf85c7a03d in CondVar::wait() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#2  0x00007faf85cc01c8 in MTManager::evhandler_main() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#3  0x00007faf85c79c7c in thread_init(void*) () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#4  0x0000003acd807c53 in start_thread (arg=0x7faf84a94700) at
pthread_create.c:308
#5  0x0000003acccf5dbd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7faf7ffff700 (LWP 31611)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007faf85c7a03d in CondVar::wait() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#2  0x00007faf85c63e18 in LinuxPtrace::main() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#3  0x00007faf85c79c7c in thread_init(void*) () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#4  0x0000003acd807c53 in start_thread (arg=0x7faf7ffff700) at
pthread_create.c:308
#5  0x0000003acccf5dbd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7faf7f5f6700 (LWP 31612)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007faf85c7a03d in CondVar::wait() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#2  0x00007faf85c85a5f in
Dyninst::ProcControlAPI::GeneratorMT::processWait(bool) ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#3  0x00007faf85c8624e in
Dyninst::ProcControlAPI::Generator::getAndQueueEventInt(bool) ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#4  0x00007faf85c640d2 in Dyninst::ProcControlAPI::GeneratorMT::main() ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#5  0x00007faf85c868d5 in Dyninst::ProcControlAPI::GeneratorMT::start() ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#6  0x00007faf85c79c7c in thread_init(void*) () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#7  0x0000003acd807c53 in start_thread (arg=0x7faf7f5f6700) at
pthread_create.c:308
#8  0x0000003acccf5dbd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 2 (Thread 0x7faf7e4dc700 (LWP 31614)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007faf8408c96b in _stp_dyninst_transport_thread_func ()
   from /tmp/stap67S2bK/stap_682284b48a759123a41de08383dfcf7f_1449.so
#2  0x0000003acd807c53 in start_thread (arg=0x7faf7e4dc700) at
pthread_create.c:308
#3  0x0000003acccf5dbd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x7faf84a96840 (LWP 31609)):
#0  0x0000003accceb8bf in __GI_ppoll (fds=fds@entry=0x7fff81a23840,
nfds=nfds@entry=1,
    timeout=<optimized out>, timeout@entry=0x7fff81a23850,
sigmask=sigmask@entry=0x7fff81a23860)
    at ../sysdeps/unix/sysv/linux/ppoll.c:56
#1  0x000000000040db72 in ppoll (__ss=0x7fff81a23860, __timeout=0x7fff81a23850,
__nfds=1,
    __fds=0x7fff81a23840) at /usr/include/bits/poll2.h:77
#2  mutator::run (this=0x25bfe80) at ../../systemtap/stapdyn/mutator.cxx:586
#3  0x00000000004075b7 in main (argc=<optimized out>, argv=<optimized out>)
    at ../../systemtap/stapdyn/stapdyn.cxx:157
(gdb)

Also can't seem to kill the target command:

stap:[vm systemtap ((daa4b33...))]$ ps 31613
  PID TTY      STAT   TIME COMMAND
31613 ?        tl     0:00
/home/vm/codebase/systemtap/build/testsuite/utrace_p5_multi_31233
stap:[vm systemtap ((daa4b33...))]$ sudo kill -KILL 31613
stap:[vm systemtap ((daa4b33...))]$ ps 31613
  PID TTY      STAT   TIME COMMAND
31613 ?        tl     0:00
/home/vm/codebase/systemtap/build/testsuite/utrace_p5_multi_31233
stap:[vm systemtap ((daa4b33...))]$

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/16795] utrace_p5.exp leaves stapdyn processes running

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16795

Jonathan Lebon <jlebon at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/16795] utrace_p5.exp leaves stapdyn processes running

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16795

--- Comment #1 from Jonathan Lebon <jlebon at redhat dot com> ---
A similar issue also occurs with at_var_mark.exp, which causes the testsuite to
hang there. [Un]fortunately, it doesn't happen very often, so it's hard to
reproduce. I've found that it's more likely to happen after doing a clean
configure, make, make install (might have to do with caching perhaps?).

$ ps aux | grep stap
root     26910  0.2  0.2  85084  8108 pts/1    Ss+  13:58   0:00 stap -v
--runtime=dyninst ../../systemtap/testsuite/systemtap.base/at_var_mark.stp -c
/notnfs/jlebon/codebase/systemtap/install/bin/stap -e 'probe begin { exit() }'
root     27459  7.5  7.8 610172 306308 pts/1   Sl+  13:58   0:04
/notnfs/jlebon/codebase/systemtap/install/bin/stapdyn -c
/notnfs/jlebon/codebase/systemtap/install/bin/stap -e 'probe begin { exit() }'
/tmp/stapM3GKb1/stap_f8c1407b5661cce21695e03ffa4fc8ff_13290.so
root     27463  0.3  0.8 133200 32516 pts/1    t+   13:58   0:00
/notnfs/jlebon/codebase/systemtap/install/bin/stap -e probe begin { exit() }
root     27466  0.0  0.0      0     0 pts/1    Z+   13:58   0:00 [staprun]
<defunct>
[jlebon systemtap.base (master)]$ sudo gdb -p 27459
(gdb) bt
#0  0x00007f5e13ad4ecf in __GI_ppoll (fds=fds@entry=0x7ffffe6b10f0,
nfds=nfds@entry=1,
    timeout=<optimized out>, timeout@entry=0x7ffffe6b1100,
sigmask=sigmask@entry=0x7ffffe6b1110)
    at ../sysdeps/unix/sysv/linux/ppoll.c:56
#1  0x000000000040db72 in ppoll (__ss=0x7ffffe6b1110, __timeout=0x7ffffe6b1100,
__nfds=1,
    __fds=0x7ffffe6b10f0) at /usr/include/bits/poll2.h:77
#2  mutator::run (this=0x1ed0ed0) at ../../systemtap/stapdyn/mutator.cxx:586
#3  0x00000000004075b7 in main (argc=<optimized out>, argv=<optimized out>)
    at ../../systemtap/stapdyn/stapdyn.cxx:157
(gdb) thread apply all backtrace

Thread 5 (Thread 0x7f5e11963700 (LWP 27460)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f5e12d6243d in CondVar::wait() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#2  0x00007f5e12dacfe8 in MTManager::evhandler_main() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#3  0x00007f5e12d6207c in thread_init(void*) () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#4  0x00007f5e145d0df3 in start_thread (arg=0x7f5e11963700) at
pthread_create.c:308
#5  0x00007f5e13adf3dd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7f5e10d49700 (LWP 27461)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f5e12d6243d in CondVar::wait() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#2  0x00007f5e12d4b118 in LinuxPtrace::main() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#3  0x00007f5e12d6207c in thread_init(void*) () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#4  0x00007f5e145d0df3 in start_thread (arg=0x7f5e10d49700) at
pthread_create.c:308
#5  0x00007f5e13adf3dd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f5e0843f700 (LWP 27462)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f5e12d6243d in CondVar::wait() () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#2  0x00007f5e12d6f82f in
Dyninst::ProcControlAPI::GeneratorMT::processWait(bool) ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#3  0x00007f5e12d7004e in
Dyninst::ProcControlAPI::Generator::getAndQueueEventInt(bool) ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#4  0x00007f5e12d4b3d2 in Dyninst::ProcControlAPI::GeneratorMT::main() ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#5  0x00007f5e12d706d5 in Dyninst::ProcControlAPI::GeneratorMT::start() ()
   from /usr/lib64/dyninst/libpcontrol.so.8.1
#6  0x00007f5e12d6207c in thread_init(void*) () from
/usr/lib64/dyninst/libpcontrol.so.8.1
#7  0x00007f5e145d0df3 in start_thread (arg=0x7f5e0843f700) at
pthread_create.c:308
#8  0x00007f5e13adf3dd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 2 (Thread 0x7f5e04bc5700 (LWP 27464)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f5e10f55eeb in _stp_dyninst_transport_thread_func ()
   from /tmp/stapM3GKb1/stap_f8c1407b5661cce21695e03ffa4fc8ff_13290.so
#2  0x00007f5e145d0df3 in start_thread (arg=0x7f5e04bc5700) at
pthread_create.c:308
#3  0x00007f5e13adf3dd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x7f5e155bd840 (LWP 27459)):
#0  0x00007f5e13ad4ecf in __GI_ppoll (fds=fds@entry=0x7ffffe6b10f0,
nfds=nfds@entry=1,
    timeout=<optimized out>, timeout@entry=0x7ffffe6b1100,
sigmask=sigmask@entry=0x7ffffe6b1110)
    at ../sysdeps/unix/sysv/linux/ppoll.c:56
#1  0x000000000040db72 in ppoll (__ss=0x7ffffe6b1110, __timeout=0x7ffffe6b1100,
__nfds=1,
    __fds=0x7ffffe6b10f0) at /usr/include/bits/poll2.h:77
#2  mutator::run (this=0x1ed0ed0) at ../../systemtap/stapdyn/mutator.cxx:586
#3  0x00000000004075b7 in main (argc=<optimized out>, argv=<optimized out>)
    at ../../systemtap/stapdyn/stapdyn.cxx:157
(gdb)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/16795] utrace_p5.exp leaves stapdyn processes running

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16795

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #2 from David Smith <dsmith at redhat dot com> ---
On f20 (3.15.4-200.fc20.x86_64), I'm seeing a problem with the
pthreadprobes.exp testcase and dyninst. One of the dyninst tests causes the
test executable to get stuck. In order to get the test to proceed I have to
manually kill the test executable. This may (or may not) be related to the
problems mentioned in this bug report.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/16795] utrace_p5.exp leaves stapdyn processes running

[scox at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16795

Stan Cox <scox at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |scox at redhat dot com
         Resolution|---                         |FIXED

--- Comment #3 from Stan Cox <scox at redhat dot com> ---
Tested with dyninst-9.3.1-1.fc25.x86_64 via induction by running each of: 
 pthreadprobes.exp
 utrace_p5.exp
 at_var_mark.exp
multiple times with a cache cleaning between each run.

-- 
You are receiving this mail because:
You are the assignee for the bug.