[Bug dyninst/15566] New: Support multiple stap --dyninst sessions on the same processes

[Bug dyninst/15566] New: Support multiple stap --dyninst sessions on the same processes

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15566

            Bug ID: 15566
           Summary: Support multiple stap --dyninst sessions on the same
                    processes
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: dyninst
          Assignee: systemtap at sourceware dot org
          Reporter: jistone at redhat dot com

As it currently stands, stap --runtime=dyninst (via stapdyn) can either launch
new target processes or attach to existing processes.  However, this is an
exclusive action, so no other stapdyn scripts are able to attach to the same
process.  This is ultimately a limitation of ptrace.

So, it might be nice to have some sort of multiplexing layer to allow multiple
stap sessions on overlapping sets of processes.  This probably can't be all the
way down at the ptrace layer though, since Dyninst will expect to know about
all transformations in the process.  But it could be multiplexed at the Dyninst
layer, such that multiple stap modules are loaded at once in stapdyn, with
their respective instrumentation routed appropriately.

There is a slight notion of this in stapdyn's mutator.cxx, with the g_mutators
vector distributing BPatch callbacks.  But in reality this is always a
single-member vector right now.  All of the code will have to learn not to
expect a single mutator.  We'll also have to invent a mechanism for dynamically
adding and removing stap modules in the mix, from different stap --dyninst
invocations.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/15566] Support multiple stap --dyninst sessions on the same processes

[serhei.public at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=15566

Serhei Makarov <serhei.public at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |serhei.public at gmail dot com

--- Comment #1 from Serhei Makarov <serhei.public at gmail dot com> ---
Going to resume work on this feature. I have a basic (highly deficient)
implementation of the functionality, based off sometime in-between release-2.3
and 2.4, in branch serhei/tpdd. Based on a cursory test run (as suggested by
demo.txt), the glaring missing functionality / bugs for the existing multidyn
demo (before moving on to the IPC server which would be the proper embodiment
of the feature) seem to be as follows:

(1) Anything having to do with detaching cleanly from a target process so that
subsequent modules can attach to it safely. Try the echo test -- see that it
works. Then abort the modules (with ^C) and try the echo test again on the same
target process. Watch the target process get killed (!! very bad !!) with a
message like this:

--SERIOUS-- #0: Dyninst was unable to load the dyninst runtime library into the
application.  This may be cause by statically linked executables, or by having
dyninst linked against a different version of libelf than it was built with.
--FATAL-- #68: Dyninst was unable to attach to the specified process
--FATAL-- #68: BPatch.C[1040]: no process 28790 defined in procsByPid

stapdyn: ERROR: Couldn't attach to the target process

Yuck :-P

Now if I use the plain version of Systemtap in RPM (with none of my tricksy
modifications of the stapdyn loader) and try something like:
$ /usr/bin/stap --dyninst -x 28741 -e 'probe process("./echo").function("echo")
{ println("echo") }'
^C
$ <same exact command again>
--FATAL-- #68: Dyninst was unable to attach to the specified process
--FATAL-- #68: BPatch.C[1040]: no process 28790 defined in procsByPid

stapdyn: ERROR: Couldn't attach to the target process

... it crashes the target process in the same fashion. (echo prints "Killed" on
its standard output and quits when this error happens.)

So this may be a deficiency in standard Stapdyn (or something weird with my
Dyninst setup, though I'm using the RPM for that as well), which may or may not
have been fixed in another PR since stap2.3. I'm not seeing any record of this
issue in our Bugzilla history, so I'll compile the latest stap to check if it's
still the case. If it happens for me with 2.5 and/or git HEAD, I'll have to
file this as a separate issue.

(2) Running multiple scripts that launch their target processes with -c is
weird. The processes launch one after the other, with one process starting
after the next one finishes. This is a blatant stupidity in my own multidyn
code, but one which should be possible to fix.

I plan to wrap the tests in a tiny expect testsuite, hopefully tomorrow, and
then have a look at forward-porting the implementation onto Systemtap 2.5
(still in a separate branch) so I can keep working on it while incorporating
the latest Stapdyn improvements.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/15566] Support multiple stap --dyninst sessions on the same processes

[serhei.public at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=15566

--- Comment #2 from Serhei Makarov <serhei.public at gmail dot com> ---
NB: this work also has relevance to PR14706, since down the line we're
effectively building a stapdyn-server process which handles multiple target
processes for multiple systemtap modules.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/15566] Support multiple stap --dyninst sessions on the same processes

[serhei.public at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=15566

--- Comment #3 from Serhei Makarov <serhei.public at gmail dot com> ---
Okay, I got Dyninst 8.2 set up from git, and the new version indeed allows
detaching and re-instrumenting the target multiple times, although there is a
weird buffering delay on stapdyn that will need to be investigated (script
output only appears after hitting ^C, even on the first time I run a plain
stapdyn session). I'll see if it also happens on the newer Systemtap; but in
any case, I think I will now go on with integrating newer stapdyn functionality
into the serhei/tpdd branch.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/15566] Support multiple stap --dyninst sessions on the same processes

[serhei.public at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=15566

--- Comment #4 from Serhei Makarov <serhei.public at gmail dot com> ---
I was working a while ago on getting several Stapdyn modules to run in the same
`stapdyn` loader process. I've put up a forward-ported implementation of my
earlier code, rebased onto a fairly recent instance of the master branch, at
https://github.com/serhei/systemtap/commits/serhei/multiplexing

The overall architecture within the loader is explained by a comment near the
top of
https://github.com/serhei/systemtap/blob/serhei/multiplexing/stapdyn/mutator.h

There are no regressions with the single-module case, but multiple modules have
a range of problems and limitations. In increasing order of difficulty:
- There needs to be a proper naming scheme implemented for distinguishing
shared memory belonging to different script modules. This could be the PID of
the stapdyn command that launched the module (so the full SHM name would be
something like "stapdyn.<PID of stapdyn command>.<PID of server>").
- dlopen()ing multiple instances of the same module, currently fails
completely. This is a regression relative to the earlier version.
- A script which calls exit(), exits all other scripts running in the same
`stapdyn` process. There are similar further things to add, in terms of
adequately separating the effects of script errors, directing the output of
different scripts to different places, etc.

I think I will have to admit defeat on getting this functionality into
Systemtap for the time being. I had some work-in-progress on sharing the
Dyninst session via a server-daemon rather than via a toy 'stapdyn -M' option,
but it does not seem useful to put more effort into it until the basic
multiplexing functionality is working more reliably.

-- 
You are receiving this mail because:
You are the assignee for the bug.