process().statement() doesn't seem to work

process().statement() doesn't seem to work

[Zoltan Kiss]

Hi,

I want to use systemtap to figure out the local variables in a function 
where something goes fishy. The function's code is here:

http://dpdk.org/browse/dpdk/tree/lib/librte_pmd_ixgbe/ixgbe_rxtx.c?id=v1.7.1#n1313

It's a receive function of DPDK's userspace poll mode driver, so it's 
called in an infinite loop. When I try this, it appers to work:

probe 
process("/usr/sbin/ovs-vswitchd").function("ixgbe_recv_scattered_pkts").return 
{
   log($$locals)
   exit();
}

The output is:

rxq=0x7fff852ee000 rx_ring=? rxdp=? sw_ring=? rxe=? first_seg=? 
last_seg=? rxm=? nmb=? rxd={...} dma=? staterr=? hlen_type_rss=? rx_id=? 
nb_rx=0x0 nb_hold=0x0 data_len=? pkt_flags=?

But it doesn't show most of the variables I need. It returns nb_rx=0, so 
I know where are the two points where things can go wrong, but I can't 
see the output of those variables (staterr and nmb)

When I try to define the probe as this:

probe 
process("/usr/sbin/ovs-vswitchd").statement("*@lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1359")

Or with any line number inside that function, the probe is never 
reached. I'm using gcc 4.8.4, DPDK is statically linked to my application.
Does anyone have an idea what might went wrong?

Regards,

Zoltan Kiss

Re: process().statement() doesn't seem to work

[Josh Stone]

On 06/26/2015 11:58 AM, Zoltan Kiss wrote:
> Hi,
> 
> I want to use systemtap to figure out the local variables in a function 
> where something goes fishy. The function's code is here:
> 
> http://dpdk.org/browse/dpdk/tree/lib/librte_pmd_ixgbe/ixgbe_rxtx.c?id=v1.7.1#n1313
> 
> It's a receive function of DPDK's userspace poll mode driver, so it's 
> called in an infinite loop. When I try this, it appers to work:
> 
> probe 
> process("/usr/sbin/ovs-vswitchd").function("ixgbe_recv_scattered_pkts").return 
> {
>    log($$locals)
>    exit();
> }
> 
> The output is:
> 
> rxq=0x7fff852ee000 rx_ring=? rxdp=? sw_ring=? rxe=? first_seg=? 
> last_seg=? rxm=? nmb=? rxd={...} dma=? staterr=? hlen_type_rss=? rx_id=? 
> nb_rx=0x0 nb_hold=0x0 data_len=? pkt_flags=?
> 
> But it doesn't show most of the variables I need. It returns nb_rx=0, so 
> I know where are the two points where things can go wrong, but I can't 
> see the output of those variables (staterr and nmb)

The behavior of .return variables is often suprising to people.  By the
time a .return probe is reached, the program actually will have already
left that stack frame.  Technically, that means only the $return value
is available to read.

But we cheat a little and also save values from the *entry* of the
function for you.  That's most useful for the $$parms string or any of
the individual parameters.

Sometimes you can also get a glimpse of $$locals, if the debuginfo tells
us how, but again this is only from the very beginning of the function.
 In your case, the three visible values all happen to have trivial
initialization: nb_rx = 0; nb_hold = 0; rxq = rx_queue;

So you can't trust that these were still the values after the function
returned.  For that, you really need to probe the statement just before
it actually returns.  Even on the return statement itself is ok.  Since
your function has just one return, it looks like line 1574 is best.
(but depending on optimizations, some locals might be missing there.)

> When I try to define the probe as this:
> 
> probe 
> process("/usr/sbin/ovs-vswitchd").statement("*@lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1359")
> 
> Or with any line number inside that function, the probe is never 
> reached. I'm using gcc 4.8.4, DPDK is statically linked to my application.
> Does anyone have an idea what might went wrong?

It may be that you're really not reaching that line, e.g. if nb_pkts==0.

You can see all the lines that stap can probe like this:

stap -l 'process("/usr/sbin/ovs-vswitchd")
         .statement("ixgbe_recv_scattered_pkts@*:*")'

Or change -l to -L to also show all readable variables at each point.


You might like the varwatch example script:

https://sourceware.org/systemtap/examples/#general/varwatch.stp

It takes two arguments - first a probe pattern like your
process.statement wildcard, then the variable to watch like $$parms,
$$locals, or $$vars for both.  A single name like $nb_rx is fine too.

Re: process().statement() doesn't seem to work

[Frank Ch. Eigler]

jistone wrote:

> [...]  The behavior of .return variables is often suprising to
> people.  [...]  But we cheat a little and also save values from the
> *entry* of the function for you.  That's most useful for the $$parms
> string or any of the individual parameters. [...]

Should we deprecate this behavior, now that the superceding @entry()
construct is itself quite old?

- FChE

Re: process().statement() doesn't seem to work

[Josh Stone]

On 07/09/2015 07:50 AM, Frank Ch. Eigler wrote:
> 
> jistone wrote:
> 
>> [...]  The behavior of .return variables is often suprising to
>> people.  [...]  But we cheat a little and also save values from the
>> *entry* of the function for you.  That's most useful for the $$parms
>> string or any of the individual parameters. [...]
> 
> Should we deprecate this behavior, now that the superceding @entry()
> construct is itself quite old?

As in, make .return $var an error and require explicit @entry($var) ?

@entry is inferior in one important way: it doesn't know the type of the
thing being saved until later, in the type resolution phase.  We might
be able to shortcut this for special cases, including $var, but a
general expression could end up as a string or a long.

For instance, in dwarf_var_expanding_visitor::visit_entry_op it must
always use gen_mapped_saved_return, with a global map and all the
indexing and locking that entails, rather than the nicer kretprobe data
pouch via gen_kretprobe_saved_return.  Maybe that could be redesigned to
not care about the type so soon, but for now it needs to know up front.

The autocast dwarf types also fail through @entry, PR18579.
https://sourceware.org/bugzilla/show_bug.cgi?id=18579

Re: process().statement() doesn't seem to work

[Zoltan Kiss]

On 30/06/15 00:39, Josh Stone wrote:
> On 06/26/2015 11:58 AM, Zoltan Kiss wrote:
>> Hi,
>>
>> I want to use systemtap to figure out the local variables in a function
>> where something goes fishy. The function's code is here:
>>
>> http://dpdk.org/browse/dpdk/tree/lib/librte_pmd_ixgbe/ixgbe_rxtx.c?id=v1.7.1#n1313
>>
>> It's a receive function of DPDK's userspace poll mode driver, so it's
>> called in an infinite loop. When I try this, it appers to work:
>>
>> probe
>> process("/usr/sbin/ovs-vswitchd").function("ixgbe_recv_scattered_pkts").return
>> {
>>     log($$locals)
>>     exit();
>> }
>>
>> The output is:
>>
>> rxq=0x7fff852ee000 rx_ring=? rxdp=? sw_ring=? rxe=? first_seg=?
>> last_seg=? rxm=? nmb=? rxd={...} dma=? staterr=? hlen_type_rss=? rx_id=?
>> nb_rx=0x0 nb_hold=0x0 data_len=? pkt_flags=?
>>
>> But it doesn't show most of the variables I need. It returns nb_rx=0, so
>> I know where are the two points where things can go wrong, but I can't
>> see the output of those variables (staterr and nmb)
>
> The behavior of .return variables is often suprising to people.  By the
> time a .return probe is reached, the program actually will have already
> left that stack frame.  Technically, that means only the $return value
> is available to read.
>
> But we cheat a little and also save values from the *entry* of the
> function for you.  That's most useful for the $$parms string or any of
> the individual parameters.
>
> Sometimes you can also get a glimpse of $$locals, if the debuginfo tells
> us how, but again this is only from the very beginning of the function.
>   In your case, the three visible values all happen to have trivial
> initialization: nb_rx = 0; nb_hold = 0; rxq = rx_queue;
>
> So you can't trust that these were still the values after the function
> returned.  For that, you really need to probe the statement just before
> it actually returns.  Even on the return statement itself is ok.  Since
> your function has just one return, it looks like line 1574 is best.
> (but depending on optimizations, some locals might be missing there.)
>
>> When I try to define the probe as this:
>>
>> probe
>> process("/usr/sbin/ovs-vswitchd").statement("*@lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1359")
>>
>> Or with any line number inside that function, the probe is never
>> reached. I'm using gcc 4.8.4, DPDK is statically linked to my application.
>> Does anyone have an idea what might went wrong?
>
> It may be that you're really not reaching that line, e.g. if nb_pkts==0.
>
> You can see all the lines that stap can probe like this:
>
> stap -l 'process("/usr/sbin/ovs-vswitchd")
>           .statement("ixgbe_recv_scattered_pkts@*:*")'
>
> Or change -l to -L to also show all readable variables at each point.

Thanks, I've figured out what I want in other ways. I'm still facing 
problems with stap, but let's start a different thread for that

>
>
> You might like the varwatch example script:
>
> https://sourceware.org/systemtap/examples/#general/varwatch.stp
>
> It takes two arguments - first a probe pattern like your
> process.statement wildcard, then the variable to watch like $$parms,
> $$locals, or $$vars for both.  A single name like $nb_rx is fine too.
>