Systemtap cannot probe user programs in docker

Systemtap cannot probe user programs in docker

[lifeibiren]

HI,

     I try to probe user program in docker but it doesn't return 
anything at exit. But it works in the host which runs the docker 
container. When I probe kernel in docker, it works fine.

     I google for half a day and got nothing. There are also some other 
guys who meet the same problems and can't resolve it. Can anybody help? 
Thanks.


----------------------------Execution----------------------------------
Run docker by: 'docker run -it --privileged fa57ca962f13 /bin/bash'

$ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
WARNING: function _start return probe is blacklisted: keyword at 
para-callgraph.stp:24:1
  source: probe $1.return { trace(-1, $$return) }


While in host:
$ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
WARNING: function _start return probe is blacklisted: keyword at 
para-callgraph.stp:24:1
  source: probe $1.return { trace(-1, $$return) }
          ^
      0 test2(22954):->_start
     15 test2(22954): ->__libc_csu_init
     19 test2(22954):  ->_init
     22 test2(22954):  <-_init
     24 test2(22954):  ->frame_dummy
     28 test2(22954):   ->register_tm_clones
     31 test2(22954):   <-register_tm_clones
     32 test2(22954):  <-frame_dummy
     34 test2(22954): <-__libc_csu_init
     36 test2(22954): ->main
    102 test2(22954):  ->dummy
    105 test2(22954):  <-dummy
    107 test2(22954): <-main
    111 test2(22954): ->__do_global_dtors_aux
    114 test2(22954):  ->deregister_tm_clones
    117 test2(22954):  <-deregister_tm_clones
    119 test2(22954): <-__do_global_dtors_aux
    121 test2(22954): ->_fini
    123 test2(22954): <-_fini

-----------------------------Source-------------------------------
$ cat para-callgraph.stp
#! /usr/bin/env stap
function trace(entry_p, extra) {
   %( $# > 1 %? if (tid() in trace) %)
   printf("%s%s%s %s\n",
          thread_indent (entry_p),
          (entry_p>0?"->":"<-"),
          ppfunc (),
          extra)
}
%( $# > 1 %?
global trace
probe $2.call {
   trace[tid()] = 1
}
probe $2.return {
   delete trace[tid()]
}
%)
probe $1.call   { trace(1, $$parms) }
probe $1.return { trace(-1, $$return) }


---------------------------Enviroment-------------------------
Host OS: Ubuntu 16.04 LTS

$ uname -r
4.4.0-98-generic

$ stap -V
Systemtap translator/driver (version 2.9/0.165, Debian version 
2.9-2ubuntu2 (xenial))
Copyright (C) 2005-2015 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP

$ docker version
Client:
  Version:      17.09.0-ce
  API version:  1.32
  Go version:   go1.8.3
  Git commit:   afdb6d4
  Built:        Tue Sep 26 22:42:18 2017
  OS/Arch:      linux/amd64

Server:
  Version:      17.09.0-ce
  API version:  1.32 (minimum version 1.12)
  Go version:   go1.8.3
  Git commit:   afdb6d4
  Built:        Tue Sep 26 22:40:56 2017
  OS/Arch:      linux/amd64
  Experimental: false

Re: Systemtap cannot probe user programs in docker

[Arkady]

Containers run in different namespaces. For example, inside of a
Docker container PIDs start from 1.
I would suggest to hook the Docker kernel module.


On Sat, Dec 23, 2017 at 12:48 PM, lifeibiren@gmail.com
<lifeibiren@gmail.com> wrote:
> HI,
>
>     I try to probe user program in docker but it doesn't return anything at
> exit. But it works in the host which runs the docker container. When I probe
> kernel in docker, it works fine.
>
>     I google for half a day and got nothing. There are also some other guys
> who meet the same problems and can't resolve it. Can anybody help? Thanks.
>
>
> ----------------------------Execution----------------------------------
> Run docker by: 'docker run -it --privileged fa57ca962f13 /bin/bash'
>
> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
> WARNING: function _start return probe is blacklisted: keyword at
> para-callgraph.stp:24:1
>  source: probe $1.return { trace(-1, $$return) }
>
>
> While in host:
> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
> WARNING: function _start return probe is blacklisted: keyword at
> para-callgraph.stp:24:1
>  source: probe $1.return { trace(-1, $$return) }
>          ^
>      0 test2(22954):->_start
>     15 test2(22954): ->__libc_csu_init
>     19 test2(22954):  ->_init
>     22 test2(22954):  <-_init
>     24 test2(22954):  ->frame_dummy
>     28 test2(22954):   ->register_tm_clones
>     31 test2(22954):   <-register_tm_clones
>     32 test2(22954):  <-frame_dummy
>     34 test2(22954): <-__libc_csu_init
>     36 test2(22954): ->main
>    102 test2(22954):  ->dummy
>    105 test2(22954):  <-dummy
>    107 test2(22954): <-main
>    111 test2(22954): ->__do_global_dtors_aux
>    114 test2(22954):  ->deregister_tm_clones
>    117 test2(22954):  <-deregister_tm_clones
>    119 test2(22954): <-__do_global_dtors_aux
>    121 test2(22954): ->_fini
>    123 test2(22954): <-_fini
>
> -----------------------------Source-------------------------------
> $ cat para-callgraph.stp
> #! /usr/bin/env stap
> function trace(entry_p, extra) {
>   %( $# > 1 %? if (tid() in trace) %)
>   printf("%s%s%s %s\n",
>          thread_indent (entry_p),
>          (entry_p>0?"->":"<-"),
>          ppfunc (),
>          extra)
> }
> %( $# > 1 %?
> global trace
> probe $2.call {
>   trace[tid()] = 1
> }
> probe $2.return {
>   delete trace[tid()]
> }
> %)
> probe $1.call   { trace(1, $$parms) }
> probe $1.return { trace(-1, $$return) }
>
>
> ---------------------------Enviroment-------------------------
> Host OS: Ubuntu 16.04 LTS
>
> $ uname -r
> 4.4.0-98-generic
>
> $ stap -V
> Systemtap translator/driver (version 2.9/0.165, Debian version 2.9-2ubuntu2
> (xenial))
> Copyright (C) 2005-2015 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> enabled features: AVAHI LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>
> $ docker version
> Client:
>  Version:      17.09.0-ce
>  API version:  1.32
>  Go version:   go1.8.3
>  Git commit:   afdb6d4
>  Built:        Tue Sep 26 22:42:18 2017
>  OS/Arch:      linux/amd64
>
> Server:
>  Version:      17.09.0-ce
>  API version:  1.32 (minimum version 1.12)
>  Go version:   go1.8.3
>  Git commit:   afdb6d4
>  Built:        Tue Sep 26 22:40:56 2017
>  OS/Arch:      linux/amd64
>  Experimental: false

Re: Systemtap cannot probe user programs in docker

[lifeibiren]

     Thanks for your ripley.

     I just looked into it today.And it turns out that docker's storage 
driver might be the problem.
     In my test case, programs put in aufs or overlay2 filesystem can't 
be probed with uprobe, even if I run it from host instead of docker 
container. I modify systemtap's runtime to print inode number and it's 
correct. But if I mount a host's directory to docker containers, such 
problems are gone. So I guess it has nothing to do with systemtap.

     Then I change docker's driver and test again and again, finding 
that it only works well with vfs driver. (only tested aufs, overlay2 and 
vfs.)


On 12/23/2017 08:33 PM, Arkady wrote:
> Containers run in different namespaces. For example, inside of a
> Docker container PIDs start from 1.
> I would suggest to hook the Docker kernel module.
> 
> 
> On Sat, Dec 23, 2017 at 12:48 PM, lifeibiren@gmail.com
> <lifeibiren@gmail.com> wrote:
>> HI,
>>
>>      I try to probe user program in docker but it doesn't return anything at
>> exit. But it works in the host which runs the docker container. When I probe
>> kernel in docker, it works fine.
>>
>>      I google for half a day and got nothing. There are also some other guys
>> who meet the same problems and can't resolve it. Can anybody help? Thanks.
>>
>>
>> ----------------------------Execution----------------------------------
>> Run docker by: 'docker run -it --privileged fa57ca962f13 /bin/bash'
>>
>> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
>> WARNING: function _start return probe is blacklisted: keyword at
>> para-callgraph.stp:24:1
>>   source: probe $1.return { trace(-1, $$return) }
>>
>>
>> While in host:
>> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
>> WARNING: function _start return probe is blacklisted: keyword at
>> para-callgraph.stp:24:1
>>   source: probe $1.return { trace(-1, $$return) }
>>           ^
>>       0 test2(22954):->_start
>>      15 test2(22954): ->__libc_csu_init
>>      19 test2(22954):  ->_init
>>      22 test2(22954):  <-_init
>>      24 test2(22954):  ->frame_dummy
>>      28 test2(22954):   ->register_tm_clones
>>      31 test2(22954):   <-register_tm_clones
>>      32 test2(22954):  <-frame_dummy
>>      34 test2(22954): <-__libc_csu_init
>>      36 test2(22954): ->main
>>     102 test2(22954):  ->dummy
>>     105 test2(22954):  <-dummy
>>     107 test2(22954): <-main
>>     111 test2(22954): ->__do_global_dtors_aux
>>     114 test2(22954):  ->deregister_tm_clones
>>     117 test2(22954):  <-deregister_tm_clones
>>     119 test2(22954): <-__do_global_dtors_aux
>>     121 test2(22954): ->_fini
>>     123 test2(22954): <-_fini
>>
>> -----------------------------Source-------------------------------
>> $ cat para-callgraph.stp
>> #! /usr/bin/env stap
>> function trace(entry_p, extra) {
>>    %( $# > 1 %? if (tid() in trace) %)
>>    printf("%s%s%s %s\n",
>>           thread_indent (entry_p),
>>           (entry_p>0?"->":"<-"),
>>           ppfunc (),
>>           extra)
>> }
>> %( $# > 1 %?
>> global trace
>> probe $2.call {
>>    trace[tid()] = 1
>> }
>> probe $2.return {
>>    delete trace[tid()]
>> }
>> %)
>> probe $1.call   { trace(1, $$parms) }
>> probe $1.return { trace(-1, $$return) }
>>
>>
>> ---------------------------Enviroment-------------------------
>> Host OS: Ubuntu 16.04 LTS
>>
>> $ uname -r
>> 4.4.0-98-generic
>>
>> $ stap -V
>> Systemtap translator/driver (version 2.9/0.165, Debian version 2.9-2ubuntu2
>> (xenial))
>> Copyright (C) 2005-2015 Red Hat, Inc. and others
>> This is free software; see the source for copying conditions.
>> enabled features: AVAHI LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>
>> $ docker version
>> Client:
>>   Version:      17.09.0-ce
>>   API version:  1.32
>>   Go version:   go1.8.3
>>   Git commit:   afdb6d4
>>   Built:        Tue Sep 26 22:42:18 2017
>>   OS/Arch:      linux/amd64
>>
>> Server:
>>   Version:      17.09.0-ce
>>   API version:  1.32 (minimum version 1.12)
>>   Go version:   go1.8.3
>>   Git commit:   afdb6d4
>>   Built:        Tue Sep 26 22:40:56 2017
>>   OS/Arch:      linux/amd64
>>   Experimental: false