RE: Stress testing - all functions

RE: Stress testing - all functions

[Stone, Joshua I]

Frank Ch. Eigler wrote:
>> d*: system reboots - there was no OOPS or BUG or anything printed to
>> the console.
> 
> Given time, one can try d[a-m]* and d[n-z]* to recursively subdivide
> the namespace.  It would be great if this were done automatically.  (A
> full search would of course take lots of time due to reboots.)

Don't forget [A-Z0-9_] in legal identifiers as well.  I had forgotten
the capitals at first, so I will update the test accordingly.

What you suggest is somewhat painful to implement since stap uses glob
matching, not regex.  However, given that testing success is cheap and
failure is expensive (reboot), a linear sweep of da*, db*, etc.
suffices.  In this way you isolate one more letter for each reboot you
endure.

I narrowed this crash down to 'do_*', and then I was fortunate to get a
stack trace on 'do_d*'.  The trace shows a cycle of
do_debug->do_page_fault->error_code->do_debug->(loop).  I've been unable
to reproduce this with probing either do_debug or 'do_d*'.  Probing
'do_*' always reboots immediately though, completely reproducible.

>> The two that failed in a* and c* are both functions that are
>> decorated with __exit.  Perhaps the translator needs to disallow
>> __exit functions, just as it disallows __init?
> 
> That's right, for routines that are linked into vmlinux (not in a
> module).

In all, fourteen letters of the alphabet fail on inserting probes on
__exit functions, so this is a blocking issue for doing more complete
tests.  I will file a bugzilla on this, and if it's not difficult I'll
go ahead and try to fix it.


Josh


(BTW, these results are all from i386 RHEL4 U3, w/ Anil's backported
patches)

Re: Stress testing - all functions

[Frank Ch. Eigler]

joshua.i.stone wrote:

> I just added a new stress test that tries to run probes on all kernel
> functions that start with the same letter.  The test script is in:
>     tests/testsuite/systemtap.stress/all_kernel_functions.stp

Good stuff.

> [...]
> d*: system reboots - there was no OOPS or BUG or anything printed to the
> console.

Given time, one can try d[a-m]* and d[n-z]* to recursively subdivide
the namespace.  It would be great if this were done automatically.  (A
full search would of course take lots of time due to reboots.)

> The two that failed in a* and c* are both functions that are
> decorated with __exit.  Perhaps the translator needs to disallow
> __exit functions, just as it disallows __init?

That's right, for routines that are linked into vmlinux (not in a
module).

- FChE

Stress testing - all functions

[Stone, Joshua I]

I just added a new stress test that tries to run probes on all kernel
functions that start with the same letter.  The test script is in:
    tests/testsuite/systemtap.stress/all_kernel_functions.stp

As you might expect this does cause problems, so what I checked into CVS
is disabled with an 'if 0'.  Just change it to 'if 1' if you want to try
it.  Here are my initial test results:

a*: ERROR: probe 1 registration failed, rc=1,
kernel.function("acpi_cpufreq_exit@arch/i386/kernel/cpu/cpufreq/acpi.c:5
27")

b*: OK!

c*: probe 1 registration failed, rc=1,
kernel.function("centrino_exit@arch/i386/kernel/cpu/cpufreq/speedstep-ce
ntrino.c:692")

d*: system reboots - there was no OOPS or BUG or anything printed to the
console.


So d* was as far as I got - later I will try removing d to see what else
fails.  And of course, just because b* passed this test doesn't mean
that all b* functions are necessarily safe.

The two that failed in a* and c* are both functions that are decorated
with __exit.  Perhaps the translator needs to disallow __exit functions,
just as it disallows __init?


Josh