user-space tracing on Debian... or..?

user-space tracing on Debian... or..?

[corpaul]

Hi, 

So for the last days I have been trying to get user-space tracing working on
Debian.
The first problem was that utrace is not supported, so we tried to compile a
custom kernel with uprobes support.
This worked, however, process probes are not supported.


To my understanding these are a fundamental part of user-space tracing...
(if not, please explain how I can avoid using them :) )

So it seems that user-space tracing is not supported on Debian.. what do you
advise?
We tried the latest Fedora core but  as this uses uprobes as well, the same
problem arises.

Any help is appreciated.

Thanks, CP



--
View this message in context: http://sourceware-org.1504.n7.nabble.com/user-space-tracing-on-Debian-or-tp225167.html
Sent from the Sourceware - systemtap mailing list archive at Nabble.com.

Re: user-space tracing on Debian... or..?

[Frank Ch. Eigler]

Hi -

c.bezemer wrote:

> [...] we tried to compile a custom kernel with uprobes support.
> This worked, however, process probes are not supported.

Please follow the instructions in "man 7stap error::reporting",
or http://sourceware.org/systemtap/wiki/HowToReportBugs to supply
us with enough information to help you.

- FChE

Re: user-space tracing on Debian... or..?

[Timo Juhani Lindfors]

corpaul <c.bezemer@tudelft.nl> writes:
> So for the last days I have been trying to get user-space tracing working on
> Debian.
> The first problem was that utrace is not supported, so we tried to compile a
> custom kernel with uprobes support.
> This worked, however, process probes are not supported.

How exactly did you build your custom kernel? Which kernel version did
you use? I tested this with squeeze two years ago and I think it
worked. The kernel I built is still at

http://lindi.iki.fi/lindi/systemtap/linux/

and includes the source (the *.dsc file) that describes exactly the
patches that were applied.

Re: user-space tracing on Debian... or..?

[corpaul]

I tried with a custom 3.8.x. kernel.
Also I tried with the latest Fedora which uses a 3.6.x kernel.
Both use Systemtap 2.1.

For both kernels I get an error when I try to run the probe:
probe process("./test").function("main").return
{
        printf("Main function exited\n");
}

Error:
Pass 1: parsed user script and 97 library script(s) using
216852virt/35972res/2972shr/33668data kb, in 230usr/50sys/507real ms.
semantic error: while resolving probe point: identifier 'process' at
test.stp:2:7
        source: probe process("./test").function("main").return
                      ^

semantic error: process return probes not available [man
error::inode-uprobes]
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s)
using 217908virt/37272res/3116shr/34724data kb, in 10usr/10sys/27real ms.
Pass 2: analysis failed.  [man error::pass2]

From what I understand is that it has something to do with the fact that
Systemtap uses Uprobes instead of Utrace in newer kernels. Is there any way
I can run probes of this type? Or is there a workaround to be able to run a
probe with the same effect?

Thanks! CP



--
View this message in context: http://sourceware-org.1504.n7.nabble.com/user-space-tracing-on-Debian-or-tp225167p225252.html
Sent from the Sourceware - systemtap mailing list archive at Nabble.com.

Re: user-space tracing on Debian... or..?

[Lukas Berk]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]

Hey,

[...]
> Error:
> Pass 1: parsed user script and 97 library script(s) using
> 216852virt/35972res/2972shr/33668data kb, in 230usr/50sys/507real ms.
> semantic error: while resolving probe point: identifier 'process' at
> test.stp:2:7
>         source: probe process("./test").function("main").return
>                       ^
> 
> semantic error: process return probes not available [man
> error::inode-uprobes]
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s)
> using 217908virt/37272res/3116shr/34724data kb, in 10usr/10sys/27real ms.
> Pass 2: analysis failed.  [man error::pass2]
> 
> From what I understand is that it has something to do with the fact that
> Systemtap uses Uprobes instead of Utrace in newer kernels. Is there any way
> I can run probes of this type? Or is there a workaround to be able to run a
> probe with the same effect?

Taken from the error::inode-uprobes man page;
function.return probes
  process.function().return probes require "return-probes" or "uretprobes"
functionality, that has not been implemented yet in the kernel.  Until  it
is re-added, consider using process.statement() probes placed on source
line numbers at the function's return statements.

It may also be possible to use the --runtime=dyninst option to use
function.return style probes, however I'm not sure on the state of
dyninst on debian.  So for script portability it might be worth using the
process.statement approach.

Please let us know if any of the error:: man page entries are unclear
and we'll do our best to make them more precise and/or readable.

Cheers,

Lukas

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: user-space tracing on Debian... or..?

[Timo Juhani Lindfors]

Lukas Berk <lberk@redhat.com> writes:
> It may also be possible to use the --runtime=dyninst option to use
> function.return style probes, however I'm not sure on the state of
> dyninst on debian.  So for script portability it might be worth using the
> process.statement approach.

systemtap in Debian has not been built with dyninst support since
dyninst is not yet part of Debian. I can consider packaging it if it
offers real advantages.

Re: user-space tracing on Debian... or..?

[corpaul]

Thanks, this has gotten me a little bit further.. however, setting probes at
line numbers is very inconvenient for large source files (unless I
misunderstood something :) ). Also, this does not seem to be very evolvable
to me - I should change the line numbers everytime the source file changes..

So is there another way to set probes on all function returns, rather than
creating a new probe explicitly for every return using the line number?




--
View this message in context: http://sourceware-org.1504.n7.nabble.com/user-space-tracing-on-Debian-or-tp225167p225525.html
Sent from the Sourceware - systemtap mailing list archive at Nabble.com.

Re: user-space tracing on Debian... or..?

[Frank Ch. Eigler]

c.bezemer wrote:

> [...]  So is there another way to set probes on all function
> returns, rather than creating a new probe explicitly for every
> return using the line number?  [...]

With a basic facility missing in the kernel, systemtap users are
forced to wait for "uretprobes" to be finished / merged.

An alternative is the stap --runtime=dyninst backend, which would have
to be enabled in your systemtap build/distro, and is still relatively
young code.

(Another alternative is to use an older Fedora or any RHEL[56] distro,
where the kernel utrace patches were applied, and thus systemtap users
have full function-return-probing powers.)

- FChE

Re: user-space tracing on Debian... or..?

[corpaul]

Unfortunately, using a RHEL distr is not possible for me as the application I
want to probe does not work well on RHEL distros.

I have tried to build Systemtap 2.1.1 with dyninst 8.0.0 (the newest dyninst
gives compilation problems) the following way:
./configure  --with-dyninst=/home/corpaul/dyninst/
--prefix=/home/corpaul/systemtap-2.1
--with-elfutils=/home/corpaul/Play/systemtap2.1/systemtap-2.1/elfutils/elfutils-0.153 

make results in the following errors (last part of output):
touch stamp-elfutils
make  all-recursive
make[1]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
Making all in .
make[2]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
  CXX    stap-main.o
  CXX    stap-session.o
  CXX    stap-parse.o
  CXX    stap-staptree.o
  CXX    stap-elaborate.o
  CXX    stap-translate.o
  CXX    stap-tapsets.o
  CXX    stap-buildrun.o
  CC     stap-loc2c.o
  CXX    stap-hash.o
  CXX    stap-cache.o
  CXX    stap-util.o
  CXX    stap-coveragedb.o
  CXX    stap-dwarf_wrappers.o
  CXX    stap-tapset-been.o
  CXX    stap-tapset-procfs.o
  CXX    stap-tapset-timers.o
  CXX    stap-tapset-netfilter.o
  CXX    stap-tapset-perfmon.o
  CXX    stap-tapset-mark.o
  CXX    stap-tapset-itrace.o
  CXX    stap-tapset-utrace.o
  CXX    stap-task_finder.o
  CXX    stap-dwflpp.o
  CXX    stap-rpm_finder.o
  CXX    stap-setupdwfl.o
  CXX    stap-remote.o
  CXX    stap-tapset-dynprobe.o
  CXX    stap-stapregex.o
  CXX    stap-csclient.o
  CXXLD  stap
  CC     loc2c_test-loc2c-test.o
  CC     loc2c_test-loc2c.o
  CCLD   loc2c-test
  CXX    stap_sign_module-util.o
  CXXLD  stap-sign-module
  CXX    stap_authorize_cert-util.o
  CXXLD  stap-authorize-cert
  CXX    stap_serverd-util.o
  CXXLD  stap-serverd
  CXX    stap_gen_cert-util.o
  CXXLD  stap-gen-cert
make[2]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
Making all in stapdyn
make[2]: Entering directory
`/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
make -C .. git_version.stamp
make[3]: Entering directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
/bin/bash ./git_version.sh -k -s . -o git_version.h
make[3]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
make  all-am
make[3]: Entering directory
`/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
  CXX    stapdyn-stapdyn.o
  CXX    stapdyn-util.o
  CXXLD  stapdyn
/usr/lib/libdynDwarf.so.8.0: undefined reference to
`dwarf_get_fde_info_for_cfa_reg3'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_errmsg'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dieoffset'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lowpc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to
`dwarf_get_fde_augmentation_data'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_next_cu_header'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_whatform'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formref'
/usr/lib/libdynDwarf.so.8.0: undefined reference to
`dwarf_loclist_from_expr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_linesrc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formsdata'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formudata'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formaddr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_hasattr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to
`dwarf_loclist_from_expr_a'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineaddr'
/usr/lib/libdynDwarf.so.8.0: undefined reference to
`dwarf_get_fde_info_for_reg3'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_siblingof'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_child'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_elf_init'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_cie_info'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formexprloc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_die_CU_offset'
/usr/lib/libstackwalk.so.8.0: undefined reference to `dwarf_errno'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_get_fde_at_pc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_diename'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_n'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_srclines'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_form_class'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_tag'
/usr/lib/libdynDwarf.so.8.0: undefined reference to
`dwarf_set_frame_cfa_value'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineno'
/usr/lib/libdynDwarf.so.8.0: undefined reference to
`dwarf_fde_cie_list_dealloc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_finish'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_range'
/usr/lib/libdynDwarf.so.8.0: undefined reference to `dwarf_get_fde_list'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_offdie'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_loclist_n'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to
`dwarf_get_cie_augmentation_data'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_fde_list_eh'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_formstring'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_srcfiles'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_get_cie_of_fde'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dealloc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineendsequence'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_attr'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_highpc'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_global_formref'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to
`dwarf_next_cu_header_c'
/usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lineoff'
collect2: error: ld returned 1 exit status
make[3]: *** [stapdyn] Error 1
make[3]: Leaving directory
`/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
make[2]: *** [all] Error 2
make[2]: Leaving directory
`/home/corpaul/Play/systemtap2.1/systemtap-2.1/stapdyn'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/corpaul/Play/systemtap2.1/systemtap-2.1'
make: *** [all] Error 2



Any suggestions on this?
I'm about to give up on user-space tracing :')

Thanks.

--CP




--
View this message in context: http://sourceware-org.1504.n7.nabble.com/user-space-tracing-on-Debian-or-tp225167p226775.html
Sent from the Sourceware - systemtap mailing list archive at Nabble.com.

Re: user-space tracing on Debian... or..?

[Frank Ch. Eigler]

corpaul <c.bezemer@tudelft.nl> writes:

> Unfortunately, using a RHEL distr is not possible for me as the application I
> want to probe does not work well on RHEL distros.
> [...]
> /usr/lib/libdynDwarf.so.8.0: undefined reference to
> `dwarf_get_fde_info_for_cfa_reg3'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_errmsg'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dieoffset'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lowpc'
> [...]

That suggests to me that dyninst was not linked quite right, it's
missing -ldwarf.  See "ldd /usr/lib/libsymtabAPI.so.8.0" - it should
list some libdwarf.so file.

To fix that, could try rebuilding dyninst more correctly, or adding
-ldwarf to the systemtap stapdyn/Makefile stapdyn_LDADD line, or
perhaps using 
% env LD_PRELOAD=/usr/lib/libdwarf.so.something stap


- FChE

Re: user-space tracing on Debian... or..?

[Josh Stone]

On 03/28/2013 03:26 AM, corpaul wrote:
> Unfortunately, using a RHEL distr is not possible for me as the application I
> want to probe does not work well on RHEL distros.
> 
> I have tried to build Systemtap 2.1.1 with dyninst 8.0.0 (the newest dyninst
> gives compilation problems) the following way:
[...]
>   CXXLD  stapdyn
> /usr/lib/libdynDwarf.so.8.0: undefined reference to
> `dwarf_get_fde_info_for_cfa_reg3'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_errmsg'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_dieoffset'
> /usr/lib/libsymtabAPI.so.8.0: undefined reference to `dwarf_lowpc'
[...]

Is your libdwarf static?  There is a thread discussing issues with this
on the dyninst-api list:
https://lists.cs.wisc.edu/archive/dyninst-api/2013/msg00026.shtml

Re: user-space tracing on Debian... or..?

[Josh Stone]

On 03/28/2013 09:25 AM, Josh Stone wrote:
> Is your libdwarf static?  There is a thread discussing issues with this
> on the dyninst-api list:
> https://lists.cs.wisc.edu/archive/dyninst-api/2013/msg00026.shtml

Now I feel I should suggest some actual solution too. :)

We could probably add a link test during configure, trying a small test
first with just -ldyninstAPI, and if that fails we can try again with
the whole-archive -ldwarf options added.

Re: user-space tracing on Debian... or..?

[corpaul]

I was finally able to get user-space probes working in Debian. 

What I did was (thanks to Andrew Bernat from Dyninst for providing excellent
support):

1 - Install libdwarf as shared library
Download libdwarf
cd dwarf-2013-02-07/libdwarf
./configure --enable-shared
make
mkdir -p /where/you/want/dwarf/include/files
mkdir -p /where/you/want/dwarf/lib/files
cp libdwarf.h dwarf.h /where/you/want/dwarf/include/files
cp libdwarf.so /where/you/want/dwarf/lib/files
Add /where/you/want/dwarf/lib/files to LD_LIBRARY_PATH
add dwarf.conf to /etc/ld.so.conf.d/ with /where/you/want/dwarf/lib/files as
contents

(not sure all these steps were mandatory but at least they worked for me)

2 - Build Dyninst 8.1.1
Configure Dyninst with
--with-libdwarf-incdir=/where/you/want/dwarf/include/files and
--with-libdwarf-libdir=/where/you/want/dwarf/lib/files
--prefix=/where/you/want/dyninst

3 - Build Systemtap 2.2
Configure with: ./configure --with-dyninst=/where/you/want/dyninst
--prefix=/where/you/want/systemtap --with-elfutils=elfutils/elfutils-0.153  
and make the source code

and the probe ./stap -e 'probe process.function("main") { print(pn()); }'
--runtime=dyninst -c './stap -V'
should work now.



Now... next step is getting python user-space probing to work :')



--
View this message in context: http://sourceware-org.1504.n7.nabble.com/user-space-tracing-on-Debian-or-tp225167p227467.html
Sent from the Sourceware - systemtap mailing list archive at Nabble.com.

Re: user-space tracing on Debian... or..?

[Timo Juhani Lindfors]

corpaul <c.bezemer@tudelft.nl> writes:
> I was finally able to get user-space probes working in Debian. 

Thanks for the info. So this was on amd64 and in debian unstable?

Do you have any interest in Debian packaging? I could consider adding
dyninst to Debian but I really lack the time to extensively test it.

RE: user-space tracing on Debian... or..?

[Cor-paul Bezemer - EWI]

This was on amd64 and on Debian squeeze. 

Somewhere in the following months I will have to package something to make installation on a test environment easier (e.g. VMs). I will keep you posted on the progress we make.

--CP



________________________________________
From: Timo Juhani Lindfors [timo.lindfors@iki.fi]
Sent: Friday, April 05, 2013 3:58 PM
To: Cor-paul Bezemer - EWI
Cc: systemtap@sourceware.org
Subject: Re: user-space tracing on Debian... or..?

corpaul <c.bezemer@tudelft.nl> writes:
> I was finally able to get user-space probes working in Debian.

Thanks for the info. So this was on amd64 and in debian unstable?

Do you have any interest in Debian packaging? I could consider adding
dyninst to Debian but I really lack the time to extensively test it.

Re: user-space tracing on Debian... or..?

[Timo Juhani Lindfors]

Hi,

Cor-paul Bezemer - EWI <C.Bezemer@tudelft.nl> writes:
> Somewhere in the following months I will have to package something to
> make installation on a test environment easier (e.g. VMs). I will keep
> you posted on the progress we make.

ok, do you have previous packaging experience?

-Timo

RE: user-space tracing on Debian... or..?

[Cor-paul Bezemer - EWI]

No, but a colleague of mine who is managing the test environment has. 

--CP
________________________________________
From: Timo Juhani Lindfors [timo.lindfors@iki.fi]
Sent: Friday, April 05, 2013 4:06 PM
To: Cor-paul Bezemer - EWI
Cc: systemtap@sourceware.org
Subject: Re: user-space tracing on Debian... or..?

Hi,

Cor-paul Bezemer - EWI <C.Bezemer@tudelft.nl> writes:
> Somewhere in the following months I will have to package something to
> make installation on a test environment easier (e.g. VMs). I will keep
> you posted on the progress we make.

ok, do you have previous packaging experience?

-Timo