workqueue testsuite - optional probes

workqueue testsuite - optional probes

[Turgis, Frederic]

Hi,

In previous thread solved by Josh stone (http://sourceware.org/ml/systemtap/2011-q3/msg00413.html), I had promised to share an example script to correlate workqueue execution, process name, function executed and workqueue name. It is shared below and tested on a x86 Ubuntu 11.04 and Android Gingerbread ARM (kernel 3.0)

However, I am a bit late because I recently realized that we have a tapset for that in irq.stp, "probe workqueue.execute" relying on kernel.trace probe. I don't know why I didn't look for it first, as I usually do !

- the example can eventually still be of some interest for the workqueue name (which we don't need in the tapset, I do that for my own convenience). I let you judge that. But I would rewrite it using the tapset probe

- I met some issues with workqueue tapset:
   * tracepoints are too old but test testsuite/buildok/irq-detailed.stp succeeds:
      + testsuite uses "probe workqueue.execute ?" so optional probe. Naïve question: is it really expected ? Not sure we are then testing much
      + tapset also uses "?" in definition of the probe. But if I remove the "?" in the testsuite, test fails. And the tapset probe itself in my script fails. It is optional, it shall not fail ? I didn't find known issue in bug list

   * there are now tracepoints for execution start/end, activation and insertion. There is then no equivalent for creation and destruction. I guess we port then to functions, not tracepoints ?

   * trace point "trace_workqueue_execute_start(work)" seems to be enabled correctly (I compared with irq_handler_entry for generated functions in System.map). Still I have no match while resolving the probe. Log states:
Pass 2: getting a tracepoint query for 1 headers:
  xxx/omap//include/trace/events/workqueue.h
...
include/trace/events/workqueue.h:37: error: 'struct cpu_workqueue_struct' declared inside parameter list
include/trace/events/workqueue.h:37: error: its scope is only this definition or declaration, which is probably not what you want

I digged a bit further and found that trace/events/workqueue.h uses "struct cpu_workqueue_struct", which is defined in workqueue.c (oups !). Kernel solves that by including trace/events/workqueue.h in workqueue.c at line 263 and not at start. Confirmed by commit 97bd234701b2b39a0e749c1fe0e44f1d14c94292.

Any thought ? Wait for clean tracepoints ? Temporarily port to function probing ?




#! /usr/bin/env stap

# Copyright (C) 2011 Texas Instruments
# by Frederic Turgis <f-turgis@ti.com>
#
# c34056a3fdde777c079cc8a70785c2602f2586cb renames all workqueue workers
# as kworker/CPUID:WORKERID. This prepares 7e11629d0efec829cbf62366143ba1081944a70e,
# which pools all workers per cpu. As there is no longer strict association
# between a cwq and its worker, it is interesting to trace function called
# and workqueue name when a work is executed.
# Note that a single kworker execution can even execute several works from
# different workqueues !
# It is also interesting to mix this script with any kind of scheduler
# monitoring script, add timestamp, ...

probe kernel.function("process_one_work") {
        /* we don't test cpu_workqueue==NULL because kernel code does not */
        cpu_workqueue = atomic_long_read(& $work->data)
        cpu_workqueue &= %{ WORK_STRUCT_WQ_DATA_MASK %};
        name = @cast(cpu_workqueue, "struct cpu_workqueue_struct")->wq->name;
        printf("Workqueue %s calls %p in %s:%u\n", kernel_string(name), $work->func, execname(), tid());
}

probe kernel.function("process_one_work").return {
        printf("Workqueue in %s:%u ends\n", execname(), tid());
}



Regards
Fred

Frederic Turgis
OMAP Platform Business Unit - OMAP System Engineering - Platform Enablement - System Multimedia


Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920

Re: workqueue testsuite - optional probes

[Frank Ch. Eigler]

Hi, Frederic -

> - I met some issues with workqueue tapset:
>    * tracepoints are too old but test testsuite/buildok/irq-detailed.stp succeeds:
>       + testsuite uses "probe workqueue.execute ?" so optional probe.  [...]

We'd need a dejagnu .exp-time divination to see whether the probe should exist,
and skip the test with "untested $test" otherwise.

>       + tapset also uses "?" in definition of the probe. [...]

We mark way too many tapset (and synthetic) probes with ?.  Optionality should be
decided by the end user script, not the tapset.

>    * there are now tracepoints for execution start/end, activation
>    and insertion. There is then no equivalent for creation and
>    destruction. I guess we port then to functions, not tracepoints ?

Could be -- or lament to LKML to get those tracepoints created.
The tapset can define the abstract probe point thusly:

 probe worksomething.something = kernel.trace("something") !,
                                 kernel.function("foo_something") { ... }

Or sometimes in two steps (in case $params are too different to be pulled out
with a common
   if (@defined($par)) par = $par
type code.


>    * trace point "trace_workqueue_execute_start(work)" [...]
> Pass 2: getting a tracepoint query for 1 headers:
>   xxx/omap//include/trace/events/workqueue.h
> ...
> include/trace/events/workqueue.h:37: error: 'struct cpu_workqueue_struct' declared inside parameter list
> include/trace/events/workqueue.h:37: error: its scope is only this definition or declaration, which is probably not what you want

See tapsets.cxx:tracepoint_extra_decls() for hacks sometimes needed
for incomplete kernel headers.

- FChE

RE: workqueue testsuite - optional probes

[Turgis, Frederic]

thanks

>We'd need a dejagnu .exp-time divination to see whether the
>probe should exist, and skip the test with "untested $test" otherwise.
Complex topic then. I thought tests shall simply delegate choice to tapset but even that is not obvious, as you suggest. Scheduler tick is considered a not optional feature, workqueue is considered optional so workqueue issue would still not have been detected

Then I will also do an update of tapset if I find relevant functions (and we can see if we really want to update the tapset) and let test as it is

>See tapsets.cxx:tracepoint_extra_decls() for hacks sometimes
>needed for incomplete kernel headers.
>
>- FChE
>
Issue has been solved in v1.6. Time to deploy a new version inside TI

Regards
Fred

Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920