[PATCH 2/3] io_submit.stp: use an accumulator for traces

[PATCH 2/3] io_submit.stp: use an accumulator for traces

[Jeff Moyer]

On very large systems, we get a lot of skipped probes due to lock
contention on the traces array.  The end result is that we don't
get any data for such systems.  Simply converting the traces array
to an accumulator resolves this issue in testing (on a highly-
loaded 288 cpu system).

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
 testsuite/systemtap.examples/io/io_submit.stp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testsuite/systemtap.examples/io/io_submit.stp b/testsuite/systemtap.examples/io/io_submit.stp
index c5edd66..62d15c9 100755
--- a/testsuite/systemtap.examples/io/io_submit.stp
+++ b/testsuite/systemtap.examples/io/io_submit.stp
@@ -37,7 +37,7 @@ probe syscall.io_submit.return {
  */
 probe kernel.function("schedule") {
   if (tid() in in_iosubmit) {
-    traces[backtrace()]++
+    traces[backtrace()] <<< 1
 
     /*
      * change this to if (1) if you want a backtrace every time
@@ -59,7 +59,7 @@ probe kernel.function("schedule") {
  */
 probe end {
   foreach (stack in traces- limit 30) {
-    printf("%d:", traces[stack])
+    printf("%d:", @count(traces[stack]))
     print_syms(stack);
   }
 }
-- 
2.8.2.335.g4bb51ae

[PATCH 1/3] io_submit.stp: fix array membership test

[Jeff Moyer]

io_submit.stp keeps track of threads which are currently executing
in the io_submit system call like so:

probe syscall.io_submit {
          in_iosubmit[tid()] = 1
}

probe syscall.io_submit.return {
          /* this assumes a given proc will do lots of io_submit calls, and
           * so doesn't do the more expensive "delete in_iosubmit[p]".  If
           * there are lots of procs doing small number of io_submit calls,
           * the hash may grow pretty big, so using delete may be better
           */
          in_iosubmit[tid()] = 0
}

However, the test to see if a thread is currently executing in io_submit
is performed using the membership operator 'in':

  if (tid() in in_iosubmit)

This is obviously wrong.  We can do one of two things:
1) change the test to if (in_iosubmit[tid()] == 1) or
2) just perform the delete in the return probe

While I agree that we typically have a small number of threads performing
io_submit, I don't believe there is substance to the performance claims
for the delete operator.  So, I've opted for solution 2.

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
 testsuite/systemtap.examples/io/io_submit.stp | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/testsuite/systemtap.examples/io/io_submit.stp b/testsuite/systemtap.examples/io/io_submit.stp
index 74ab639..c5edd66 100755
--- a/testsuite/systemtap.examples/io/io_submit.stp
+++ b/testsuite/systemtap.examples/io/io_submit.stp
@@ -28,12 +28,7 @@ probe syscall.io_submit {
  * when we return from sys_io_submit, record that we're no longer there
  */
 probe syscall.io_submit.return {
-  /* this assumes a given proc will do lots of io_submit calls, and
-   * so doesn't do the more expensive "delete in_iosubmit[p]".  If
-   * there are lots of procs doing small number of io_submit calls,
-   * the hash may grow pretty big, so using delete may be better
-   */
-  in_iosubmit[tid()] = 0
+  delete in_iosubmit[tid()]
 }
 
 /*
-- 
2.8.2.335.g4bb51ae

[PATCH 3/3] io_submit.stp: let the user know when the script is loaded

[Jeff Moyer]

I often find myself checking lsmod to see when the script is finally
ready to collect data.  Just print a message from the begin probe to
make it obvious when the script is ready.

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
 testsuite/systemtap.examples/io/io_submit.stp | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/testsuite/systemtap.examples/io/io_submit.stp b/testsuite/systemtap.examples/io/io_submit.stp
index 62d15c9..7e8c8d4 100755
--- a/testsuite/systemtap.examples/io/io_submit.stp
+++ b/testsuite/systemtap.examples/io/io_submit.stp
@@ -53,6 +53,10 @@ probe kernel.function("schedule") {
   }
 }
 
+probe begin {
+  printf("Ready!\n")
+}
+
 /*
  * when stap is done (via ctrl-c) go through the record of all the
  * trace paths and print the 30 most common.
-- 
2.8.2.335.g4bb51ae

io_submit.stp: various fixes

[Jeff Moyer]

These fixes are pretty important for making the data gathered from
io_submit.stp useful.  Patches 1 and 3 I've been using for a while.
Patch 3 was developed recently due to issues collecting data on a
large cpu count system.

Cheers,
Jeff

[PATCH 1/3] io_submit.stp: fix array membership test
[PATCH 2/3] io_submit.stp: use an accumulator for traces
[PATCH 3/3] io_submit.stp: let the user know when the script is

Re: [PATCH 3/3] io_submit.stp: let the user know when the script is loaded

[Frank Ch. Eigler]

jmoyer wrote:

> I often find myself checking lsmod to see when the script is finally
> ready to collect data.  Just print a message from the begin probe to
> make it obvious when the script is ready.
> [...]
> +probe begin {
> +  printf("Ready!\n")
> +}

I merged this and the other patches (thanks!).  For the record though,
one may prefer a solution like

cat >> $HOME/.systemtap/rc
-E 'probe begin { printf("Ready!\n") }'
^D

so that every stap script you use automagically gets that extra probe
added on.


- FChE

Re: [PATCH 3/3] io_submit.stp: let the user know when the script is loaded

[Jeff Moyer]

fche@redhat.com (Frank Ch. Eigler) writes:

> jmoyer wrote:
>
>> I often find myself checking lsmod to see when the script is finally
>> ready to collect data.  Just print a message from the begin probe to
>> make it obvious when the script is ready.
>> [...]
>> +probe begin {
>> +  printf("Ready!\n")
>> +}
>
> I merged this and the other patches (thanks!).  For the record though,
> one may prefer a solution like
>
> cat >> $HOME/.systemtap/rc
> -E 'probe begin { printf("Ready!\n") }'
> ^D
>
> so that every stap script you use automagically gets that extra probe
> added on.

I didn't know I could do that.  Thanks!

-Jeff