How do I trap the return of a function in a user space process?

How do I trap the return of a function in a user space process?

[Martin Martin]

The following systemtap script fails to compile:

probe process("myexe").function("myFun").return { print("hi mom\n") }

It says:

semantic error: while resolving probe point: identifier 'process' at
proxy.stp:6:7 source: probe
process("/home/martin/experiments/systemtap/a.out").function("doSomething").return
^

semantic error: process return probes not available [man
error::inode-uprobes] Pass 2: analysis failed. [man error::pass2] Tip:
/usr/share/doc/systemtap/README.Debian should help you get started.

I want to trap when the function exits, not the process. If I remove
the .return it compiles & runs fine. How do I trap the function exit?

I'm on Ubuntu 13.04, Linux 3.8.0-29-generic, system tap 2.1/0.153.

Thanks,
Martin

Re: How do I trap the return of a function in a user space process?

[Yichun Zhang (agentzh)]

Hello!

On Thu, Sep 5, 2013 at 1:36 PM, Martin Martin wrote:
> The following systemtap script fails to compile:
>
> probe process("myexe").function("myFun").return { print("hi mom\n") }
>
[...]
>
> I'm on Ubuntu 13.04, Linux 3.8.0-29-generic, system tap 2.1/0.153.
>

The Linux kernel 3.8 does not support uretprobes. I think you need to
use kernel 3.10+ for that or downgrade to older kernels with the
utrace patch applied.

Regards,
-agentzh

Re: How do I trap the return of a function in a user space process?

[Martin Martin]

Hi,

Thanks a lot for your response.  I created a virtual machine running
the latest beta of Ubuntu 13.10, which is Linux 3.11.0, and I can now
probe returns!

However, I only get events (function enter or exit) if I start the
process from stap using -c.  If I just run the process from the shell,
then run stap, I don't get any events. Any idea what I could be doing
wrong?

Thanks,
Martin

On Thu, Sep 5, 2013 at 4:42 PM, Yichun Zhang (agentzh)
<agentzh@gmail.com> wrote:
> Hello!
>
> On Thu, Sep 5, 2013 at 1:36 PM, Martin Martin wrote:
>> The following systemtap script fails to compile:
>>
>> probe process("myexe").function("myFun").return { print("hi mom\n") }
>>
> [...]
>>
>> I'm on Ubuntu 13.04, Linux 3.8.0-29-generic, system tap 2.1/0.153.
>>
>
> The Linux kernel 3.8 does not support uretprobes. I think you need to
> use kernel 3.10+ for that or downgrade to older kernels with the
> utrace patch applied.
>
> Regards,
> -agentzh

Re: How do I trap the return of a function in a user space process?

[Martin Martin]

My next problem is that all of the C++ methods in the class I'm
interested in don't show up when I do:

sudo stap -l 'process("myexe").function("*")'

Although static methods defined in the same .cpp file do show up.  The
mangled names don't show up in the output of .function("_Z*"),
although nm shows them just fine.  How does systemtap decide what to
filter out and what to keep?

Thanks again,
Martin


On Fri, Sep 6, 2013 at 10:17 AM, Martin Martin <martin@infinio.com> wrote:
> Hi,
>
> Thanks a lot for your response.  I created a virtual machine running
> the latest beta of Ubuntu 13.10, which is Linux 3.11.0, and I can now
> probe returns!
>
> However, I only get events (function enter or exit) if I start the
> process from stap using -c.  If I just run the process from the shell,
> then run stap, I don't get any events. Any idea what I could be doing
> wrong?
>
> Thanks,
> Martin
>
> On Thu, Sep 5, 2013 at 4:42 PM, Yichun Zhang (agentzh)
> <agentzh@gmail.com> wrote:
>> Hello!
>>
>> On Thu, Sep 5, 2013 at 1:36 PM, Martin Martin wrote:
>>> The following systemtap script fails to compile:
>>>
>>> probe process("myexe").function("myFun").return { print("hi mom\n") }
>>>
>> [...]
>>>
>>> I'm on Ubuntu 13.04, Linux 3.8.0-29-generic, system tap 2.1/0.153.
>>>
>>
>> The Linux kernel 3.8 does not support uretprobes. I think you need to
>> use kernel 3.10+ for that or downgrade to older kernels with the
>> utrace patch applied.
>>
>> Regards,
>> -agentzh

Re: How do I trap the return of a function in a user space process?

[Martin Martin]

I manged to solve this problem: I was compiling with clang++.
Switching to g++ caused all my functions to show up.

Thanks,
Martin

On Fri, Sep 6, 2013 at 11:57 AM, Martin Martin <martin@infinio.com> wrote:
> My next problem is that all of the C++ methods in the class I'm
> interested in don't show up when I do:
>
> sudo stap -l 'process("myexe").function("*")'
>
> Although static methods defined in the same .cpp file do show up.  The
> mangled names don't show up in the output of .function("_Z*"),
> although nm shows them just fine.  How does systemtap decide what to
> filter out and what to keep?
>
> Thanks again,
> Martin
>
>
> On Fri, Sep 6, 2013 at 10:17 AM, Martin Martin <martin@infinio.com> wrote:
>> Hi,
>>
>> Thanks a lot for your response.  I created a virtual machine running
>> the latest beta of Ubuntu 13.10, which is Linux 3.11.0, and I can now
>> probe returns!
>>
>> However, I only get events (function enter or exit) if I start the
>> process from stap using -c.  If I just run the process from the shell,
>> then run stap, I don't get any events. Any idea what I could be doing
>> wrong?
>>
>> Thanks,
>> Martin
>>
>> On Thu, Sep 5, 2013 at 4:42 PM, Yichun Zhang (agentzh)
>> <agentzh@gmail.com> wrote:
>>> Hello!
>>>
>>> On Thu, Sep 5, 2013 at 1:36 PM, Martin Martin wrote:
>>>> The following systemtap script fails to compile:
>>>>
>>>> probe process("myexe").function("myFun").return { print("hi mom\n") }
>>>>
>>> [...]
>>>>
>>>> I'm on Ubuntu 13.04, Linux 3.8.0-29-generic, system tap 2.1/0.153.
>>>>
>>>
>>> The Linux kernel 3.8 does not support uretprobes. I think you need to
>>> use kernel 3.10+ for that or downgrade to older kernels with the
>>> utrace patch applied.
>>>
>>> Regards,
>>> -agentzh

Re: How do I trap the return of a function in a user space process?

[Frank Ch. Eigler]

Martin Martin <martin@infinio.com> writes:

> I manged to solve this problem: I was compiling with clang++.
> Switching to g++ caused all my functions to show up.

Can clang++ generate DWARF data?  It'd be interesting to
see a sample binary that stap cannot deal with.

- FChE

Re: How do I trap the return of a function in a user space process?

[Martin Martin]

Here's a simple one.  If you remove the namespace, stap -l shows fun,
but as below, it doesn't:

namespace yummy {
class Foo {
    int fun();
};

int Foo::fun() { return 23; }
}


int main()
{
}

A quick google search shows Clang doesn't pass the GDB 7.5 test suite,
so it seems there are known errors with DWARF generation, but
presumably something this basic should work?

http://llvm.org/bugs/show_bug.cgi?id=14330

Best,
Martin


On Fri, Sep 6, 2013 at 8:00 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Martin Martin <martin@infinio.com> writes:
>
>> I manged to solve this problem: I was compiling with clang++.
>> Switching to g++ caused all my functions to show up.
>
> Can clang++ generate DWARF data?  It'd be interesting to
> see a sample binary that stap cannot deal with.
>
> - FChE

Re: How do I trap the return of a function in a user space process?

[Mark Wielaard]

On Mon, Sep 09, 2013 at 08:23:11AM -0400, Martin Martin wrote:
> Here's a simple one.  If you remove the namespace, stap -l shows fun,
> but as below, it doesn't:
> 
> namespace yummy {
> class Foo {
>     int fun();
> };
> 
> int Foo::fun() { return 23; }
> }
> 
> 
> int main()
> {
> }
> 
> A quick google search shows Clang doesn't pass the GDB 7.5 test suite,
> so it seems there are known errors with DWARF generation, but
> presumably something this basic should work?

The problem is that clang generates both the foo declaration TAG and
the actual foo subprogram TAG under the namespace and class_type TAG.
That isn't technically invalid DWARF, but it is somewhat cumbersome
for a DWARF consumer because it mixes the type entries and the code
entries hierarchy. That makes it much harder for the DWARF consumer
to find the actual code entry TAGs.

stap (dwflcpp.cxx) uses elfutils libdw dwarf_funcs which finds all
subprograms for a DIE tree that are direct children of the root CU die.
dwarf_funcs expects all functions of a compile unit to be represented by 
subprogram DIEs that are direct children of the compile unit.

We could support the DWARF that clang outputs in this case, but that
does mean a somewhat expensive walk of the whole DIE tree. We might get
away with only looking for (nested) DW_TAG_namespace, DW_TAG_class_type
and DW_TAG_structure_type DIEs. But if so we probably also need to update
dwarf_getfuncs to handle imported_units. Which currently aren't an
issue since dwz will never place the top-level DW_TAG_subprograms
in a partial unit, but might do so when DW_TAG_subprograms are nested
in the type hierarchy like here.

Cheers,

Mark

Re: How do I trap the return of a function in a user space process?

[Josh Stone]

On 09/09/2013 11:13 AM, Mark Wielaard wrote:
> The problem is that clang generates both the foo declaration TAG and
> the actual foo subprogram TAG under the namespace and class_type TAG.

FWIW, on my system (with clang-3.3-0.6.rc3.fc19.x86_64) I get the second
subprogram within just the namespace, not the class.  Don't know if
yours is different, or maybe you just misread the indentation.  It would
be not quite so ugly if dwarf_getfuncs only had to descend into
namespace DIEs.

BTW, there is a similar problem with GCC nested functions, e.g. stap
can't see the deepen() functions in stap's own loc2c.c.  Iterating
namespaces wouldn't help here, not even adding class/struct, as these
DIEs are nested within other functions.

Re: How do I trap the return of a function in a user space process?

[Mark Wielaard]

On Mon, Sep 09, 2013 at 11:32:16AM -0700, Josh Stone wrote:
> On 09/09/2013 11:13 AM, Mark Wielaard wrote:
> > The problem is that clang generates both the foo declaration TAG and
> > the actual foo subprogram TAG under the namespace and class_type TAG.
> 
> FWIW, on my system (with clang-3.3-0.6.rc3.fc19.x86_64) I get the second
> subprogram within just the namespace, not the class.  Don't know if
> yours is different, or maybe you just misread the indentation.  It would
> be not quite so ugly if dwarf_getfuncs only had to descend into
> namespace DIEs.

Your eyes are clearly better than mine :) Indeed the declaration is
under the class_type, but the instance is just under the namespace.
 
> BTW, there is a similar problem with GCC nested functions, e.g. stap
> can't see the deepen() functions in stap's own loc2c.c.  Iterating
> namespaces wouldn't help here, not even adding class/struct, as these
> DIEs are nested within other functions.

Hmmm. You are right. And it is a fun example because there are actually
two "deepen" functions nested in two different other functions.

Seems it is time to create some example binaries for elfutils of
where dwarf_getfuncs() doesn't find some functions/subprograms
with the current approach and see if we can fix it or whether we
need to have a totally different function/approach.

Thanks,

Mark