[Bug uprobes/27975] New: could systemtap probe user process running in docker(overlay2 fs)?

[Bug uprobes/27975] New: could systemtap probe user process running in docker(overlay2 fs)?

[q1875486458 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27975

            Bug ID: 27975
           Summary: could systemtap probe user process running in
                    docker(overlay2 fs)?
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: uprobes
          Assignee: systemtap at sourceware dot org
          Reporter: q1875486458 at gmail dot com
  Target Milestone: ---

## version info
systemtap version: 4.0/0.176
kernel version: 3.10.0-1160.25.1.el7.x86_64
os: CentOS Linux 7 (Core)

```bash
stap -V
Systemtap translator/driver (version 4.0/0.176, rpm 4.0-13.el7)
enabled features: AVAHI BOOST_STRING_REF DYNINST BPF JAVA PYTHON2 LIBRPM
LIBSQLITE3 LIBVIRT LIBXML2 NLS NSS READLINE
```
## what i want 
i'm try to probe nginx(has debuginfo) which running in docker.

## what i have do
since nginx running in docker,so i just find the hostpath of the nginx in
docker
and run 

`stap -x 27368  -e 'probe process.syscall {printf("%s %d %s\n", execname(),
pid(), pp())} probe
process("/var/lib/docker/overlay2/{a-long-long-string}/merged/usr/local/openresty/nginx/sbin/nginx").function("*")
{printf("%s",probefunc())}' 

`process.syscall` works find,but nginx probe just nothing happen.

and 
`stap -x 27368 -e 'probe process.syscall {printf("%s %d %s\n", execname(),
pid(), pp())} probe
process("/usr/local/openresty/nginx/sbin/nginx").function("*")
{printf("%s",probefunc())}'
--sysroot=/var/lib/docker/overlay2/{a-long-long-string}/merged/
` 
report err like 

`WARNING: probe
process("/usr/local/openresty/nginx/sbin/nginx").function("ngx_write_fd@src/os/unix/ngx_files.h:145")
inode-offset 0000000000041e55 registration error (rc -22)`

## what i searched
i find some says [systemtap does not support overlays/union
filesystems](https://stackoverflow.com/questions/57487292/systemtap-failed-to-probe-the-functions-registration-error)

there is samiler question asked in stackoverflow,but have to meanning answer
[smailer question but no
answer](https://stackoverflow.com/questions/29314665/systemtapwhile-resolving-probe-point-identifier-process-at-source-probe-pro)

[SystemTap_Beginners_Guide chapter 4
userspace-probing](https://www.sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html)
just say you to point the exepath of the process.
## what i want to know
is that true? how should i probe user process which run in docker? 
what should i know to fix this question?
how to know what happen when uprobe just echo nothing?

## other info
```bash
 docker top a0bb57
UID                 PID                 PPID                C                  
STIME               TTY                 TIME                CMD
root                20807               20755               0                  
11:19               ?                   00:00:00            /sbin/tini --
/run.sh
root                20852               20807               0                  
11:19               ?                   00:00:00            sh /run.sh
root                27368               20807               0                  
11:20               ?                   00:00:00            nginx: master
process /usr/local/openresty/nginx/sbin/nginx -c
/usr/local/openresty/nginx/conf/nginx.conf
root                27369               27368               4                  
11:20               ?                   00:10:00            nginx: worker
process
root                27370               27368               0                  
11:20               ?                   00:01:46            nginx: worker
process
```

i have copy /lib/modules/3.10.0-1160.25.1.el7.x86_64/ to
/var/lib/docker/overlay2/{a-long-long-string}/merged/

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/27975] could systemtap probe user process running in docker(overlay2 fs)?

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27975

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-06-10
             Status|UNCONFIRMED                 |WAITING
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Consider trying the probe-by-buildid mechanism added recently:

    probe process("deadbeefdeadbeef").FOO { }
instead of
    probe process("/path/to/binary").FOO { }

... where deadbeef is simply the hexadecimal buildid for the target binary, as
printed by "file" or "readelf -n".

Your RHEL7 version of stap is much too old for that, but maybe you can get hold
of RH DTS builds, or build your own from upstream sources / RPMs, this should
work.

Otherwise, yeah, path name based matching is known to be difficult when there
are many names for the same binary, and from different containers' points of
view.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/27975] could systemtap probe user process running in docker(overlay2 fs)?

[q1875486458 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27975

--- Comment #2 from cong <q1875486458 at gmail dot com> ---
is there any way to distinguish between
1. systemtap does not probe to correct process (which should not happen because
i set pid explicitly)
2. systemtap could not find those function which should probe
3. there is no event came out

since the method 1 i use in before code does not report any wran/erorr ,it
seems that it probe to correct process and find the debuginfo and find the
function i want to probe.  

a correspond conjecture is that: "is just no event came out".

did this logic chain make sense? or i just missing some point?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/27975] could systemtap probe user process running in docker(overlay2 fs)?

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27975

--- Comment #3 from Frank Ch. Eigler <fche at redhat dot com> ---
Running a stap module built with super verbosity like

stap -DDEBUG_TASK_FINDER -DDEBUG_TASK_FINDER_VMA -DDEBUG_SYMBOLS
-DDEBUG_UPROBES [...]

may let you see the difference.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/27975] could systemtap probe user process running in docker(overlay2 fs)?

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27975

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |WORKSFORME

-- 
You are receiving this mail because:
You are the assignee for the bug.