$return in wildcarded void functions

$return in wildcarded void functions

[Mike Mason]

A co-worker posed this problem... He's debugging code he's not familiar with.  He's using wildcarding to probe the entries and exits of all functions in a specific module. The return probe looks something like this:

probe module("uhci_hcd").function("*").return
{
	printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), errno_str($return))
}

The return probe attempts to access $return.  For functions defined as void, obviously there is no return value, which causes pass 2 to fail with the following errors: 

semantic error: function has no return value: identifier '$return' at usb2:11:85
semantic error: function has no return value: identifier '$return' at usb2:23:85

Is there any way around this problem short of writing a return probe for each function?  Is there any way to treat a variable as conditional?  That would be a useful feature, especially for wildcarded return probes.

Mike

Re: $return in wildcarded void functions

[Vara Prasad]

Mike Mason wrote:

> [snip]
>
> probe module("uhci_hcd").function("*").return
> {
>     printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), 
> errno_str($return))
> }
>
My interpretation of the above statement is put a return probe in all 
the functions that have return value in the "uhci_hcd" module, with that 
interpretation we shouldn't even try to put return probes on the 
functions with a return type of void.

If the user how ever specifically says put  a probe in a function that 
has return type of void i think we should give "no return value" error.

Looks like based on dwarf info we can detect void functions. I guess the 
issue is how easy it is to carry forward the information from wild card 
expansion stage to elaboration stage so we can differentiate between the 
functions that were specified by user explicitly vs wild card expanded 
ones. I have not spent enough time to figure that out, i will let Frank 
comment on that.

> The return probe attempts to access $return.  For functions defined as 
> void, obviously there is no return value, which causes pass 2 to fail 
> with the following errors:
> semantic error: function has no return value: identifier '$return' at 
> usb2:11:85
> semantic error: function has no return value: identifier '$return' at 
> usb2:23:85
>
> Is there any way around this problem short of writing a return probe 
> for each function?  Is there any way to treat a variable as 
> conditional?  That would be a useful feature, especially for 
> wildcarded return probes.
>
> Mike
>
>

Re: $return in wildcarded void functions

[Jim Keniston]

On Mon, 2007-03-05 at 18:21 -0800, Vara Prasad wrote:
> Mike Mason wrote:
> 
> > [snip]
> >
> > probe module("uhci_hcd").function("*").return
> > {
> >     printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), 
> > errno_str($return))
> > }
> >
> My interpretation of the above statement is put a return probe in all 
> the functions that have return value in the "uhci_hcd" module, with that 
> interpretation we shouldn't even try to put return probes on the 
> functions with a return type of void.

Sounds like you're saying SystemTap should refuse to do return probes
for void functions.  This is plainly not the case.  E.g., in the example
shown, the user wants to know when the function returns (to match up
calls and returns), even if it's a void function.

> 
> If the user how ever specifically says put  a probe in a function that 
> has return type of void i think we should give "no return value" error.
> 
> Looks like based on dwarf info we can detect void functions. I guess the 
> issue is how easy it is to carry forward the information from wild card 
> expansion stage to elaboration stage so we can differentiate between the 
> functions that were specified by user explicitly vs wild card expanded 
> ones.

Yeah, that'd be nice.

Other possibilities:
1) Redefine $return to evaluate to 0 for void functions.
2) Define a new variable $vreturn (or some such) that does (1).
3) Change the error to a warning, perhaps considering wildcarding as
Vara proposed.

Seems like all these would be easier than a whole new $?var language
construct (though $?var would certainly help us smooth over the
occasional deficiency in the dawrf info).

Jim
 
[snip]

RE: $return in wildcarded void functions

[John Liang]

Sorry guys. I am new to this mailinglist. Could you please
tell me which email address I should use to send an email
to if I want to ask a question. 

I really appreciate your help. 

Thanks,
--
John Liang 

-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org]
On Behalf Of Jim Keniston
Sent: Tuesday, March 06, 2007 12:06 PM
To: Vara Prasad
Cc: Mike Mason; systemtap@sources.redhat.com
Subject: Re: $return in wildcarded void functions

On Mon, 2007-03-05 at 18:21 -0800, Vara Prasad wrote:
> Mike Mason wrote:
> 
> > [snip]
> >
> > probe module("uhci_hcd").function("*").return
> > {
> >     printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), 
> > errno_str($return))
> > }
> >
> My interpretation of the above statement is put a return probe in all 
> the functions that have return value in the "uhci_hcd" module, with that 
> interpretation we shouldn't even try to put return probes on the 
> functions with a return type of void.

Sounds like you're saying SystemTap should refuse to do return probes
for void functions.  This is plainly not the case.  E.g., in the example
shown, the user wants to know when the function returns (to match up
calls and returns), even if it's a void function.

> 
> If the user how ever specifically says put  a probe in a function that 
> has return type of void i think we should give "no return value" error.
> 
> Looks like based on dwarf info we can detect void functions. I guess the 
> issue is how easy it is to carry forward the information from wild card 
> expansion stage to elaboration stage so we can differentiate between the 
> functions that were specified by user explicitly vs wild card expanded 
> ones.

Yeah, that'd be nice.

Other possibilities:
1) Redefine $return to evaluate to 0 for void functions.
2) Define a new variable $vreturn (or some such) that does (1).
3) Change the error to a warning, perhaps considering wildcarding as
Vara proposed.

Seems like all these would be easier than a whole new $?var language
construct (though $?var would certainly help us smooth over the
occasional deficiency in the dawrf info).

Jim
 
[snip]

Re: $return in wildcarded void functions

[Frank Ch. Eigler]

Mike Mason <mmlnx@us.ibm.com> writes:

> [...] Is there any way around this problem short of writing a return
> probe for each function?

Not at present.  However, one could overload the probe optional flag

    probe foo.bar("*") ?   { $zoo }

to skip probes where complications such as a bad $zoo cause a probe
to be deemed erroneous.  Nonexistent functions are handled like that;
perhaps nonexistent variables could be too.

> Is there any way to treat a variable as conditional?  That would be
> a useful feature, especially for wildcarded return probes.

Not at present.  One might imagine a "$?variable" syntax for optional
values.  (This would be an independent facility from the optional
probe widgetry above.)

- FChE

Re: $return in wildcarded void functions

[Mike Mason]

I'd prefer something like the current preprocessing conditionals, only for variables (see example below).  This way no probes would be skipped, but the probe body could be different based on whether the variable is present.  This could be used in a variety of ways beyond just the $return variable.  I have no idea how difficult this would be to implement.  I assume it isn't a simple matter of extending the preprocessing code since preprocessing is done before accessing the dwarf info.

Mike

probe module("uhci_hcd").function("*").return
{
%( $return exists %?
    printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), errno_str($return))
%:
    printf ("%s <- %s: void\n", thread_indent(-2), probefunc())
%)
}





Frank Ch. Eigler wrote:
> Mike Mason <mmlnx@us.ibm.com> writes:
> 
>> [...] Is there any way around this problem short of writing a return
>> probe for each function?
> 
> Not at present.  However, one could overload the probe optional flag
> 
>     probe foo.bar("*") ?   { $zoo }
> 
> to skip probes where complications such as a bad $zoo cause a probe
> to be deemed erroneous.  Nonexistent functions are handled like that;
> perhaps nonexistent variables could be too.
> 
>> Is there any way to treat a variable as conditional?  That would be
>> a useful feature, especially for wildcarded return probes.
> 
> Not at present.  One might imagine a "$?variable" syntax for optional
> values.  (This would be an independent facility from the optional
> probe widgetry above.)
> 
> - FChE

Re: $return in wildcarded void functions

[Frank Ch. Eigler]

Mike Mason <mmlnx@us.ibm.com> writes:

> I'd prefer something like the current preprocessing conditionals
> [...] I assume it isn't a simple matter of extending the
> preprocessing code since preprocessing is done before accessing the
> dwarf info.

Indeed.  Plus preprocessor conditionals don't nest within the other
nonterminals - they just enclose arbitrary token lists.

I've thought about it a while, and it seems easiest as well as fairly
general to support markup of the $variable use itself.  Since more
elaborate script code may want to substitute a missing $value, and
also know that such a substitution has been done, we'd probably need
two variables:

# probe module("uhci_hcd").function("*").return
# {
# if ($?return)
#   printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), errno_str($!return))
# else
#     printf ("%s <- %s: void\n", thread_indent(-2), probefunc())
# }

So $?var would return true or false if the given expression was
defined.  And $!var would return the value, if defined, or else fall
back to 0.  (This could also cover a fault being encountered while
$!var->field->array[99] pointer chains were gbeing followed.)

So:
   $?var==0 implies $!var==0
   $?var==0 implies $var would have caused compile-time error
   $?var==1 implies $!var equals what $var would have

While $?var would be a compile-time constant, by also providing the
"faultless" $!var variant, there is no need to be clever within the
translator.
  
- FChE

Re: $return in wildcarded void functions

[Mike Mason]

Frank Ch. Eigler wrote:
> I've thought about it a while, and it seems easiest as well as fairly
> general to support markup of the $variable use itself.  Since more
> elaborate script code may want to substitute a missing $value, and
> also know that such a substitution has been done, we'd probably need
> two variables:
> 
> # probe module("uhci_hcd").function("*").return
> # {
> # if ($?return)
> #   printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), errno_str($!return))
> # else
> #     printf ("%s <- %s: void\n", thread_indent(-2), probefunc())
> # }
> 
> So $?var would return true or false if the given expression was
> defined.  And $!var would return the value, if defined, or else fall
> back to 0.  (This could also cover a fault being encountered while
> $!var->field->array[99] pointer chains were gbeing followed.)
> 
> So:
>    $?var==0 implies $!var==0
>    $?var==0 implies $var would have caused compile-time error
>    $?var==1 implies $!var equals what $var would have

I like this approach.  Could it also work for variables defined locally in probes (e.g. ?var and !var, no $)?  What about global variables?

I'm not crazy about returning 0 for a non-existent variable, but at least this method provides a way to avoid that.

Does $?var==1 imply that $var and $!var exist ?

Mike

Re: $return in wildcarded void functions

[Frank Ch. Eigler]

Hi -

On Tue, Mar 06, 2007 at 02:05:39PM -0800, Mike Mason wrote:
> [...]
> >So:
> >   $?var==0 implies $!var==0
> >   $?var==0 implies $var would have caused compile-time error
> >   $?var==1 implies $!var equals what $var would have
> 
> I like this approach.  Could it also work for variables defined locally in 
> probes (e.g. ?var and !var, no $)?  What about global variables?

I can't think of any reason to apply it to normal script variables.
The point here is to adapt to vagaries of the probed software, not the
probe script itself.

> I'm not crazy about returning 0 for a non-existent variable, but at
> least this method provides a way to avoid that.

Yes.

> Does $?var==1 imply that $var and $!var exist ?

$?var and $!var would exist for any $var and $var->field->array[43]
expression.  $var would exist only when it "should" and cause a
translation-time error otherwise.

- FChE

Re: $return in wildcarded void functions

[Mike Mason]

Frank Ch. Eigler wrote:
> Hi -
> 
> On Tue, Mar 06, 2007 at 02:05:39PM -0800, Mike Mason wrote:
>> [...]
>>> So:
>>>   $?var==0 implies $!var==0
>>>   $?var==0 implies $var would have caused compile-time error
>>>   $?var==1 implies $!var equals what $var would have
>> I like this approach.  Could it also work for variables defined locally in 
>> probes (e.g. ?var and !var, no $)?  What about global variables?
> 
> I can't think of any reason to apply it to normal script variables.
> The point here is to adapt to vagaries of the probed software, not the
> probe script itself.

I should have said "locally in *tapset* probes".  Tapsets themselves may change just like the probed software, though we have more control over that.  Supporting this approach for tapset variables would help us to write user scripts that work with multiple versions of systemtap, even if variables are missing in some versions.

Mike

Re: $return in wildcarded void functions

[David Smith]

Frank Ch. Eigler wrote:
> Mike Mason <mmlnx@us.ibm.com> writes:
> 
>> I'd prefer something like the current preprocessing conditionals
>> [...] I assume it isn't a simple matter of extending the
>> preprocessing code since preprocessing is done before accessing the
>> dwarf info.
> 
> Indeed.  Plus preprocessor conditionals don't nest within the other
> nonterminals - they just enclose arbitrary token lists.
> 
> I've thought about it a while, and it seems easiest as well as fairly
> general to support markup of the $variable use itself.  Since more
> elaborate script code may want to substitute a missing $value, and
> also know that such a substitution has been done, we'd probably need
> two variables:
> 
> # probe module("uhci_hcd").function("*").return
> # {
> # if ($?return)
> #   printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(), errno_str($!return))
> # else
> #     printf ("%s <- %s: void\n", thread_indent(-2), probefunc())
> # }
> 
> So $?var would return true or false if the given expression was
> defined.  And $!var would return the value, if defined, or else fall
> back to 0.  (This could also cover a fault being encountered while
> $!var->field->array[99] pointer chains were gbeing followed.)
> 
> So:
>    $?var==0 implies $!var==0
>    $?var==0 implies $var would have caused compile-time error
>    $?var==1 implies $!var equals what $var would have
> 
> While $?var would be a compile-time constant, by also providing the
> "faultless" $!var variant, there is no need to be clever within the
> translator.

Hmm, I'm missing something.  Why do we need $?var *and* $!var?  It seems 
like $?var would suffice.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: $return in wildcarded void functions

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

On Wed, Mar 07, 2007 at 11:05:12AM -0600, David Smith wrote:
(please trim!)

> [...]
> Hmm, I'm missing something.  Why do we need $?var *and* $!var?  It seems 
> like $?var would suffice.

The reason why it doesn't becomes clear with an example, and with the
realization that $var would still trigger a *translate-time* error if
it did not exist in context.

   if ($?var) { use($!var) } // works even if $var would fail to translate

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: $return in wildcarded void functions

[David Smith]

Frank Ch. Eigler wrote:
> On Wed, Mar 07, 2007 at 11:05:12AM -0600, David Smith wrote:
>> Hmm, I'm missing something.  Why do we need $?var *and* $!var?  It seems 
>> like $?var would suffice.
> 
> The reason why it doesn't becomes clear with an example, and with the
> realization that $var would still trigger a *translate-time* error if
> it did not exist in context.
> 
>    if ($?var) { use($!var) } // works even if $var would fail to translate

Ah, the idea makes sense now.  But, how would the above translate? 
Would the entire statement get deleted (if $var didn't exist)?

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: $return in wildcarded void functions

[Frank Ch. Eigler]

dsmith wrote:

> > [...]
> >    if ($?var) { use($!var) } // works even if $var would fail to translate
> 
> Ah, the idea makes sense now.  But, how would the above translate?

$?var - to a literal constant
$!var - to literal 0 if $?var was 0, or else to whatever $var would have

> Would the entire statement get deleted (if $var didn't exist)?

Could be in the future, as a special case of dead code elimination.

- FChE

RE: $return in wildcarded void functions

[Stone, Joshua I]

Frank Ch. Eigler wrote:
> dsmith wrote:
>> Would the entire statement get deleted (if $var didn't exist)?
> 
> Could be in the future, as a special case of dead code elimination.

If you start with that elimination as a requirement, then there is no
explicit need for the $!var construct.  At that point you could
synthesize the same behavior with ($?var ? $var : 0), and $!var is
perhaps just a convenient shorthand.

Josh

NFS probes with redhat 4 update 3/4

[John Liang]

HI There,
 
If I want to track which clients are making NFS requests to my
server, which probes can I use for tracking the client IP addresses?


It looks there is no nfs related functions in kernel as following:
[root@perf8 nfs]#  stap -p2 -e 'probe kernel.function("*") {}' | sort | uniq
| grep -i nfs

kernel.function("compat_nfs_clnt_trans@fs/compat.c:1799"),
kernel.function("compat_sys_nfsservctl@fs/compat.c:1900"),
kernel.function("security_genfs_sid@security/selinux/ss/services.c:1422"),
kernel.function("sys_nfsservctl@fs/nfsctl.c:90"),

When I track the module of nfs, I run into the "No DWARF info" problem:
[root@  nfs]# lsmod | grep -i nfs
nfsd                  277217  107 
exportfs                7745  1 nfsd
nfs                   243825  1 
lockd                  77809  3 nfsd,nfs
nfs_acl                 5185  2 nfsd,nfs
sunrpc                174905  19 nfsd,ibrix,nfs,lockd,nfs_acl

[root@ nfs]# stap -p2 -e 'probe module("nfs").function("*") {}' | sort |
uniq
semantic error: cannot find module nfs debuginfo: No DWARF information found
semantic error: no match for probe point
         while: resolving probe point module("nfs").function("*")
Pass 2: analysis failed.  Try again with '-v' (verbose) option.

Any comments?


Thanks,
--
John Liang

Re: NFS probes with redhat 4 update 3/4

[David Smith]

John Liang wrote:
> [root@ nfs]# stap -p2 -e 'probe module("nfs").function("*") {}' | sort |
> uniq
> semantic error: cannot find module nfs debuginfo: No DWARF information found
> semantic error: no match for probe point
>          while: resolving probe point module("nfs").function("*")
> Pass 2: analysis failed.  Try again with '-v' (verbose) option.
> 
> Any comments?

On FC6 (kernel 2.6.19-1.2911.fc6), here's what I get:

#  stap -p2 -v -e 'probe module("nfs").function("*") { }' | grep -v 
'[{}]' | wc -l
Pass 1: parsed user script and 54 library script(s) in 
680usr/30sys/716real ms.
Pass 2: analyzed script: 638 probe(s), 0 function(s), 0 embed(s), 0 
global(s) in 260usr/210sys/481real ms.
639

So, systemtap sees 638 (1 line is '# probes') different functions in the 
nfs module.  On FC6, you need to have the following kernel debug 
information installed for the above to work:

kernel-debuginfo-2.6.19-1.2911.fc6
kernel-debuginfo-common-2.6.19-1.2911.fc6

It looks like either the debug information for the module isn't 
installed or systemtap can't find it.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: NFS probes with redhat 4 update 3/4

[Tom Burns]

Hi John, list;

On 3/9/07, David Smith <dsmith@redhat.com> wrote:
> John Liang wrote:
> > [root@ nfs]# stap -p2 -e 'probe module("nfs").function("*") {}' | sort |
> > uniq
> > semantic error: cannot find module nfs debuginfo: No DWARF information found
> .. snip ...
>
> It looks like either the debug information for the module isn't
> installed or systemtap can't find it.

Just FYI John, you can tell if your module has the requisite DWARF
debugging symbols by executing "objdump -W module.ko".  If it does you
will see a lot of information.  A lot of distro's release their kernel
module debugging symbols as a seperate package (ie what David says
about FC6), in which case the DWARF symbols are in a different file
(try /usr/lib/debug/lib/... in case of FC anyways).

however if you have the debugging info in neither place then you will
need to get it somehow to probe that module.

Cheers,
Tom Burns
Software Developer
International Datacasting

backwards compability of systemtap

[John Liang]

HI there,

It looks array syntax like "foobar["string",1]" is not supported 
in stap version 0.5.4 which comes with redhat 4 update 3. But it is 
definitely supported in stap Version 0.5.8 of redhat 4 update 4. 

I am wondering if there is a way to run the compiled the module/C code
of version 0.5.8 on redhat 4 update 3. If yes, then it means I can 
use the most recent features of systemtap to old version of redhat
releases. 

When I try this, I got the following message from /var/log/messages:

Mar 12 09:51:19 perf8 kernel: stap_7948: disagrees about version of symbol
register_kprobe
Mar 12 09:51:19 perf8 kernel: stap_7948: Unknown symbol register_kprobe

Any comments?

Thanks,
--
John Liang

Re: backwards compability of systemtap

[Frank Ch. Eigler]

"John Liang" <jliang@ibrix.com> writes:

> [...]  I am wondering if there is a way to run the compiled the
> module/C code of version 0.5.8 on redhat 4 update 3. If yes, then it
> means I can use the most recent features of systemtap to old version
> of redhat releases.

We haven't tested this configuration.  In theory, it could work.  It's
equivalent to simply building a modern systemtap natively on your
RHEL4U3 machine.

> When I try this, I got the following message from /var/log/messages:
> Mar 12 09:51:19 perf8 kernel: stap_7948: disagrees about version of symbol
> register_kprobe
> Mar 12 09:51:19 perf8 kernel: stap_7948: Unknown symbol register_kprobe

However, sometimes kernel API changes pop up.  If someone provides a
patch to let systemtap (again) create modules against this older API,
we could merge it.

- FChE

Re: backwards compability of systemtap

[David Smith]

John Liang wrote:
> HI there,
> 
> It looks array syntax like "foobar["string",1]" is not supported 
> in stap version 0.5.4 which comes with redhat 4 update 3. But it is 
> definitely supported in stap Version 0.5.8 of redhat 4 update 4. 
> 
> I am wondering if there is a way to run the compiled the module/C code
> of version 0.5.8 on redhat 4 update 3. If yes, then it means I can 
> use the most recent features of systemtap to old version of redhat
> releases. 
> 
> When I try this, I got the following message from /var/log/messages:
> 
> Mar 12 09:51:19 perf8 kernel: stap_7948: disagrees about version of symbol
> register_kprobe
> Mar 12 09:51:19 perf8 kernel: stap_7948: Unknown symbol register_kprobe

You've got 2 problems here:

1) Kernel version differences (which is probably what you see above). 
If you take a kernel module compiled against the U4 kernel and try to 
run it against the U3 kernel, you are going to have problems.  It would 
be possible to install the U3 kernel on your U4 system (but not actually 
run the U3 kernel), compile the systemtap module against the U3 kernel, 
then copy it to your U3 system.

But, then you'll come up against:

2) Possible backend differences.  I don't deal much with the backend, 
but there are almost certainly differences between the 0.5.4 and 0.5.8 
versions of staprun.  The question is can the 0.5.4 staprun handle an 
0.5.8 module?  I don't believe we've ever tested this (or have ever made 
any promises that it would/could/should work).  Martin Hunt would know 
more here.

You safest bet would be to compile the 0.5.8 version of systemtap on 
your U3 system.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

systemtap crashed the kernel

[John Liang]

Folks,

I consistently run into the following coredump while I have
my script running to monitor NFS traffic. The core stack is 
as following:

crash> bt
PID: 0      TASK: ffffffff803d4300  CPU: 0   COMMAND: "swapper"
 #0 [ffffffff80456ba0] start_disk_dump at ffffffffa07a036d
 #1 [ffffffff80456bd0] try_crashdump at ffffffff8014bd01
 #2 [ffffffff80456be0] do_page_fault at ffffffff80124143
 #3 [ffffffff80456c60] ip_local_deliver_finish at ffffffff802c8f28
 #4 [ffffffff80456c80] nf_hook_slow at ffffffff802b92fe
 #5 [ffffffff80456cc0] error_exit at ffffffff80110d91
    [exception RIP: bnx2_poll+240]
    RIP: ffffffffa00e4b28  RSP: ffffffff80456d78  RFLAGS: 00010206
    RAX: 0000000000007baa  RBX: 00000101265f7a60  RCX: 0000010122294000
    RDX: ffffffff804e5080  RSI: 0000000000007ba9  RDI: 00000000000000a6
    RBP: 0000000000000000   R8: 0000010005f6c270   R9: 0000000000000040
    R10: 0000000000000040  R11: 0000000000000088  R12: 0000010129179b80
    R13: 0000000000007ba6  R14: 0000000000000120  R15: 0000000000000012
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #6 [ffffffff80456df0] bnx2_start_xmit at ffffffffa00e81c8
 #7 [ffffffff80456e90] net_rx_action at ffffffff802b0884
 #8 [ffffffff80456ec0] __do_softirq at ffffffff8013c738
 #9 [ffffffff80456ef0] do_softirq at ffffffff8013c7e1
#10 [ffffffff80456f00] do_IRQ at ffffffff80113247
--- <IRQ stack> ---
#11 [ffffffff804e9f08] ret_from_intr at ffffffff80110833
    [exception RIP: mwait_idle+86]
    RIP: ffffffff8010e84c  RSP: ffffffff804e9fb8  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffffffff803d4300  RDI: 00000100010427e0
    RBP: 0000000000000000   R8: ffffffff804e8000   R9: 0000000000000008
    R10: 0000000000000040  R11: 0000000000000088  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffd9  CS: 0010  SS: 0018
#12 [ffffffff804e9fb8] cpu_idle at ffffffff8010e7dc

I can reproduce this with "version 0.5.8 built 2006-07-14" and earlier
version that comes with redhat 4 update 3. Is this a known issue? Any
comments?

Thanks,
--
John Liang

Re: systemtap crashed the kernel

[Frank Ch. Eigler]

"John Liang" <jliang@ibrix.com> writes:

> I consistently run into the following coredump while I have
> my script running to monitor NFS traffic. The core stack is 
> as following [...]

It seems not to include any systemtap-related code, interesting.
If you could gather other information requested by the
http://sources.redhat.com/systemtap/wiki/HowToReportBugs page,
it would help.

> I can reproduce this with "version 0.5.8 built 2006-07-14" and
> earlier version that comes with redhat 4 update 3. [...]

For what it's worth, a version labeled 0.5.10 was released for RHEL4U4.

- FChE

RE: systemtap crashed the kernel

[John Liang]

FChE,

This issue turns to be a bnx2 driver issue and has
nothing to do with systemtap. 

But while I was working on the issue, I wonder how
systemtap protects global variables by concurrent multiple 
threads with multiple CPUs. For example, one probe would
call to "delete an array" while another or the same probe is trying
to read an element of this array at the same time. 

Any comments?

Thanks,
--
John Liang

-----Original Message-----
From: fche@redhat.com [mailto:fche@redhat.com] 
Sent: Tuesday, March 20, 2007 10:49 AM
To: John Liang
Cc: systemtap@sources.redhat.com
Subject: Re: systemtap crashed the kernel

"John Liang" <jliang@ibrix.com> writes:

> I consistently run into the following coredump while I have
> my script running to monitor NFS traffic. The core stack is 
> as following [...]

It seems not to include any systemtap-related code, interesting.
If you could gather other information requested by the
http://sources.redhat.com/systemtap/wiki/HowToReportBugs page,
it would help.

> I can reproduce this with "version 0.5.8 built 2006-07-14" and
> earlier version that comes with redhat 4 update 3. [...]

For what it's worth, a version labeled 0.5.10 was released for RHEL4U4.

- FChE

Re: systemtap crashed the kernel

[Frank Ch. Eigler]

Hi -

On Wed, Mar 21, 2007 at 04:21:51PM -0400, John Liang wrote:
> This issue turns to be a bnx2 driver issue and has
> nothing to do with systemtap. 

Thanks for the update.


> But while I was working on the issue, I wonder how systemtap
> protects global variables by concurrent multiple threads with
> multiple CPUs. For example, one probe would call to "delete an
> array" while another or the same probe is trying to read an element
> of this array at the same time.

We use rwlocks for script-level global variables.  If those two probes
run at the same time on a different processor, one of them will be
delayed very briefly, waiting for the other probe finish and release
the lock.  If this does not happen "soon" (as defined by configurable
parameters), the loser probe is skipped entirely.  (After a
configurable number of skipped probes, the entire script aborts.)

Write some test scripts, and look at the generated C code with "stap -p3".

- FChE

RE: NFS probes with redhat 4 update 3/4

[John Liang]

David,

After I install kernel-debuginfo rpm, it works like a charm. 

Thanks,
--
John Liang

-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org]
On Behalf Of David Smith
Sent: Friday, March 09, 2007 11:42 AM
To: John Liang
Cc: systemtap@sources.redhat.com
Subject: Re: NFS probes with redhat 4 update 3/4

John Liang wrote:
> [root@ nfs]# stap -p2 -e 'probe module("nfs").function("*") {}' | sort |
> uniq
> semantic error: cannot find module nfs debuginfo: No DWARF information
found
> semantic error: no match for probe point
>          while: resolving probe point module("nfs").function("*")
> Pass 2: analysis failed.  Try again with '-v' (verbose) option.
> 
> Any comments?

On FC6 (kernel 2.6.19-1.2911.fc6), here's what I get:

#  stap -p2 -v -e 'probe module("nfs").function("*") { }' | grep -v 
'[{}]' | wc -l
Pass 1: parsed user script and 54 library script(s) in 
680usr/30sys/716real ms.
Pass 2: analyzed script: 638 probe(s), 0 function(s), 0 embed(s), 0 
global(s) in 260usr/210sys/481real ms.
639

So, systemtap sees 638 (1 line is '# probes') different functions in the 
nfs module.  On FC6, you need to have the following kernel debug 
information installed for the above to work:

kernel-debuginfo-2.6.19-1.2911.fc6
kernel-debuginfo-common-2.6.19-1.2911.fc6

It looks like either the debug information for the module isn't 
installed or systemtap can't find it.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

RE: $return in wildcarded void functions

[John Liang]

HI There,

I tried to use $return with systemtap 0.5.4. It seems 
$return is not supported or I did something wrong here?

[]# stap -V
SystemTap translator/driver (version 0.5.4 built 2006-02-02)
Copyright (C) 2005-2006 Red Hat, Inc. and others
This is free software; see the source for copying conditions

probe kernel.function("d_lookup").return
{
       if($return)        
        missedcnt++ 
}

[]# ./cache.stp 
semantic error: target variables not available to .return probes
semantic error: no match for probe point
         while: resolving probe point kernel.function("d_lookup").return
Pass 2: analysis failed.  Try again with '-v' (verbose) option.

Thanks,
--
John Liang

-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org]
On Behalf Of Mike Mason
Sent: Monday, March 05, 2007 8:31 PM
To: systemtap@sources.redhat.com
Subject: $return in wildcarded void functions

A co-worker posed this problem... He's debugging code he's not familiar
with.  He's using wildcarding to probe the entries and exits of all
functions in a specific module. The return probe looks something like this:

probe module("uhci_hcd").function("*").return
{
	printf ("%s <- %s: %s\n", thread_indent(-2), probefunc(),
errno_str($return))
}

The return probe attempts to access $return.  For functions defined as void,
obviously there is no return value, which causes pass 2 to fail with the
following errors: 

semantic error: function has no return value: identifier '$return' at
usb2:11:85
semantic error: function has no return value: identifier '$return' at
usb2:23:85

Is there any way around this problem short of writing a return probe for
each function?  Is there any way to treat a variable as conditional?  That
would be a useful feature, especially for wildcarded return probes.

Mike

Re: $return in wildcarded void functions

[David Smith]

John Liang wrote:
> HI There,
> 
> I tried to use $return with systemtap 0.5.4. It seems 
> $return is not supported or I did something wrong here?
> 
> []# stap -V
> SystemTap translator/driver (version 0.5.4 built 2006-02-02)
> Copyright (C) 2005-2006 Red Hat, Inc. and others
> This is free software; see the source for copying conditions
> 

Hmm, I'm a bit confused by your version number, since we haven't 
released a 0.5.4 version yet...  We're on 0.5.13.  What distro is this on?

However, I added the $return functionality on 2006-06-05, which is 
certainly past your build date.

I'd upgrade to a newer version of systemtap.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)