[Bug dyninst/23513] New: Dyninst runtime kills the target process when attaching to the target process for a second time

[Bug dyninst/23513] New: Dyninst runtime kills the target process when attaching to the target process for a second time

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

            Bug ID: 23513
           Summary: Dyninst runtime kills the target process when
                    attaching to the target process for a second time
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: dyninst
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com
  Target Milestone: ---

Created attachment 11178
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11178&action=edit
stap-report output on the same box

Consider the following simple forever-running C program:

```C
#include <unistd.h>

int main(void) {
    while (1) {
        usleep (1000);
    }
    return 0;
}
```

Compile it and generate `./a.out`:

```
gcc -g -Wall a.c
```

Then start it in one terminal (let's call it terminal A):

```
$ ./a.out
```

It hangs as expected.

Then try running the stap dyninst command like below in another terminal (let's
say, terminal B):

```
$ /opt/stap/bin/stap -e 'probe process("/lib64/libc.so.6").function("usleep") {
println("hi") exit() }' -x `pgrep -f './a.out'` --dyninst
hi
```

It works as expected. And in terminal A, the `./a.out` program is still
running. So far so good.

But now as soon as we run the `stap` command above again, we got the following
error on terminal B:

```
$ /opt/stap/bin/stap -e 'probe process("/lib64/libc.so.6").function("usleep") {
println("hi") exit() }' -x `pgrep -f './a.out'` --dyninst
--FATAL-- #68: Dyninst was unable to attach to the specified process
--FATAL-- #68: BPatch.C[1047]: no process 40758 defined in procsByPid

stapdyn: ERROR: Couldn't attach to the target process
WARNING: /opt/stap/bin/stapdyn exited with status: 1
Pass 5: run failed.  [man error::pass5]
```

And in terminal A, the `./a.out` process is already killed too:

```
agentzh@work2 ~ $ ./a.out
Killed
```

This can also be reproduced by a `probe begin` one-liner:

```
agentzh@work2 ~ $ /opt/stap/bin/stap -e 'probe begin { println("hi") exit() }'
-x `pgrep -f './a.out'` --dyninst
hi

agentzh@work2 ~ $ /opt/stap/bin/stap -e 'probe begin { println("hi") exit() }'
-x `pgrep -f './a.out'` --dyninst
--FATAL-- #68: Dyninst was unable to attach to the specified process
--FATAL-- #68: BPatch.C[1047]: no process 40816 defined in procsByPid

stapdyn: ERROR: Couldn't attach to the target process
WARNING: /opt/stap/bin/stapdyn exited with status: 1
Pass 5: run failed.  [man error::pass5]
```

I'm using the 3.3 release tag of systemtap on Fedora 27 x86_64:

```
$ /opt/stap/bin/stap --version
Systemtap translator/driver (version 3.3/0.173, commit
release-3.3-0-g48867d1cface)
Copyright (C) 2005-2018 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.18-rc0
enabled features: BOOST_STRING_REF DYNINST BPF NLS NSS
```

Also attached the stap-report output for the same box.

Am I doing anything wrong here?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/23513] Dyninst runtime kills the target process when attaching to the target process for a second time

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
The second dyninst-based probe's failure is expected: there can be only one
concurrent "mutator" for a given target process.  It is not right that the
process should be killed though, that sounds like a bug.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/23513] Dyninst runtime kills the target process when attaching to the target process for a second time

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

--- Comment #2 from agentzh <agentzh at gmail dot com> ---
But the first `stap` command run already exits before the second one starts
(please note the `exit()` function call in the probe handler). So there is
still only *one* concurrent mutator here.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/23513] Dyninst runtime kills the target process when attaching to the target process for a second time

[scox at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

Stan Cox <scox at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |scox at redhat dot com

--- Comment #3 from Stan Cox <scox at redhat dot com> ---
I notice if the first dyninst run is via stap:
% stap -e 'probe process("/lib64/libc.so.6").function("usleep") { println("hi")
exit() }' -x `pgrep -f tst.x` --dyninst -k -vv

and if subsequent runs were directly via stapdyn then it worked okay:
% stapdyn -v -x 8955 /tmp/staprclqlD/stap_8970.so
stapdyn: found target "/usr/lib64/libc-2.28.so" in pid 8955, inserting 1 probes
hi

% stapdyn -v -x 8955 /tmp/staprclqlD/stap_8970.so
stapdyn: found target "/usr/lib64/libc-2.28.so" in pid 8955, inserting 1 probes
hi

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/23513] Dyninst runtime kills the target process when attaching to the target process for a second time

[scox at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

--- Comment #4 from Stan Cox <scox at redhat dot com> ---
Running with DYNINST_DEBUG_STARTUP=1 
it seems the problem is dyninst fails to parse the stap so (not sure why) and
intentionally terminates the process.

[140306531820032]image.C[1461]:  opening file
/tmp/stapSmTcdi/stap_2ecef162cf9d3c769f9c52398beab57b_1423.so
[140306531820032]image.C[1226]:  created image
[140306531820032]image.C[1229]: processing shared object
[140306531820032]image.C[1247]: error in processing, deleting image and
returning
[140306531820032]mapped_object.C[157]:  failed to parseImage
[140306531820032]dynProcess.C[618]: failed to create mapped object for library
/tmp/stapSmTcdi/stap_2ecef162cf9d3c769f9c52398beab57b_1423.so
[140306531820032]dynProcess.C[361]: bootstrap failed while creating mapped
objects
Failed to bootstrap process 9255: terminating...

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/23513] Dyninst runtime kills the target process when attaching to the target process for a second time

[scox at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

Stan Cox <scox at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2019-04-10
     Ever confirmed|0                           |1

--- Comment #5 from Stan Cox <scox at redhat dot com> ---
The problem seems intermittant but I do see it and the culprit is that the stap
shared object is not always detached when stapdyn completes.  (gdb session
after failing stapdyn)

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007f3bd7f64670  0x00007f3bd80afc3f  Yes (*)     /lib64/libc.so.6
0x00007f3bd8139110  0x00007f3bd81582b4  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00007f3bd6eb2320  0x00007f3bd6eb4845  Yes (*)    
/usr/lib64/dyninst/libdyninstAPI_RT.so
0x00007f3bd6eab270  0x00007f3bd6eac039  Yes (*)     /lib64/libdl.so.2
                                        No         
/tmp/stapUaA2KQ/stap_4e52f2023ce6edaabf143f6d5214b684_1361.so
0x00007f3bd8124710  0x00007f3bd8127a80  Yes (*)     /lib64/librt.so.1
0x00007f3bd6e60b50  0x00007f3bd6e6f025  Yes (*)     /lib64/libpthread.so.0
(*): Shared library is missing debugging information.

So whenever stapdyn subsequently runs, dyninst tries to load that nonexistant
library: /tmp/stapUaA2KQ/stap_4e52f2023ce6edaabf143f6d5214b684_1361.so

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/23513] Dyninst runtime kills the target process when attaching to the target process for a second time

[avi@cloudius-systems.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23513

Avi Kivity <avi@cloudius-systems.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |avi@cloudius-systems.com

-- 
You are receiving this mail because:
You are the assignee for the bug.