Tests that require 'modprobe <module>'

Tests that require 'modprobe <module>'

[Mike Mason]

I'd like to add some tests that require certain modules be inserted (specifically nfs & rpc).  Where's the best place in the test framework to do modprobe?

Mike

RE: Tests that require 'modprobe <module>'

[Stone, Joshua I]

Mike Mason wrote:
> I'd like to add some tests that require certain modules be inserted
> (specifically nfs & rpc).  Where's the best place in the test
> framework to do modprobe?

First, using 'modprobe' requires root access.  Most of 'make
installcheck' only needs the stapdev group, with the exception of a few
tests that use sudo.  I would like to see us reduce the root requirement
more...

But anyway, a test requiring specific modules is bound to be fragile.
What happens if that module was instead built into the kernel, or not
built at all?  Or if it is a module and you successfully load it, it
clearly wasn't in use, so how can you test any probes within?

Can you instead write your test to gracefully skip if the module isn't
present?  (It shouldn't record a failure...)


Josh

Re: Tests that require 'modprobe <module>'

[Mike Mason]

Stone, Joshua I wrote:
> Mike Mason wrote:
>> I'd like to add some tests that require certain modules be inserted
>> (specifically nfs & rpc).  Where's the best place in the test
>> framework to do modprobe?
> 
> First, using 'modprobe' requires root access.  Most of 'make
> installcheck' only needs the stapdev group, with the exception of a few
> tests that use sudo.  I would like to see us reduce the root requirement
> more...

I understand the desire to reduce the root requirement for users, but for
complete testing that may be unrealistic.

> 
> But anyway, a test requiring specific modules is bound to be fragile.
> What happens if that module was instead built into the kernel, or not
> built at all?  Or if it is a module and you successfully load it, it
> clearly wasn't in use, so how can you test any probes within?

I was thinking the modprobe would simply fail if the code is built 
into the kernel instead.  There's no harm silently ignoring that
as far as I can tell.  If the code isn't in the kernel and isn't built 
as a module, the tests for those probe points should fail.  I know, the
test didn't actually run and fail, we just failed to test.  To me, that's 
still a failure.

As for whether the module's in use, in some cases that doesn't matter.
For example, you can still load the nfs, nfsd and sunrpc modules and
run build tests against them even though they're not being used.  This
may not be true for all modules.

> 
> Can you instead write your test to gracefully skip if the module isn't
> present?  (It shouldn't record a failure...)

I don't think untested tapsets should be silently ignored.  If we don't
figure out how to test *everything*, even if it's hard, how will we
know when some change in the kernel breaks a tapset?  When a 
systemtap user stumbles on the problem?  IMHO, that's
unacceptable.

Mike

> 
> 
> Josh

RE: Tests that require 'modprobe <module>'

[Stone, Joshua I]

Mike Mason wrote:
>  I understand the desire to reduce the root requirement for users,
> but for complete testing that may be unrealistic.

It's not unrealistic if you make it a goal.  Grep'ing for sudo in the
testsuite right now, I see only two ways that sudo is used:

systemtap.printf/*b.exp -- calls 'sudo rm' to cleanup temp files.  This
shouldn't be necessary, since the files are owned by the user running
the test.

systemtap.context/context.exp -- builds and loads a test kernel module.
Clever use of embedded-C in a stap-script could generate and load the
test modules without needing an explicit insmod.  There's still an issue
with resolving symbols for this test to work, but bug #2151 would help.

> If the code isn't in the kernel and isn't built as a module, the
> tests for those probe points should fail.  I know, the test didn't
> actually run and fail, we just failed to test.  To me, that's still
> a failure.

But it's a failure of a different sort.  That's what skipped &
unsupported result codes are for.

> As for whether the module's in use, in some cases that doesn't matter.
> For example, you can still load the nfs, nfsd and sunrpc modules and
> run build tests against them even though they're not being used.  This
> may not be true for all modules.

I suppose I can accept that, as long as you can restore the system when
you're done.  IIRC, some modules may trigger loading other modules
though, so cleanup may be difficult.

>>  Can you instead write your test to gracefully skip if the module
>> isn't present?  (It shouldn't record a failure...)
>  I don't think untested tapsets should be silently ignored.  If we
> don't figure out how to test *everything*, even if it's hard, how
> will we know when some change in the kernel breaks a tapset?  When a
> systemtap user stumbles on the problem?  IMHO, that's unacceptable.

I don't mean to call it a 'pass' -- again, that's what skipped &
unsupported are for.  It's important to distinguish between "I can't
test that on this system" and "something is really broken".

Josh

Re: Tests that require 'modprobe <module>'

[Mike Mason]

Stone, Joshua I wrote:
> Mike Mason wrote:
>>  I understand the desire to reduce the root requirement for users,
>> but for complete testing that may be unrealistic.
> 
> It's not unrealistic if you make it a goal.  Grep'ing for sudo in the
> testsuite right now, I see only two ways that sudo is used:

I agree with the goal, but it shouldn't be a higher priority than making 
sure every tapset is tested.

> 
> systemtap.printf/*b.exp -- calls 'sudo rm' to cleanup temp files.  This
> shouldn't be necessary, since the files are owned by the user running
> the test.
> 
> systemtap.context/context.exp -- builds and loads a test kernel module.
> Clever use of embedded-C in a stap-script could generate and load the
> test modules without needing an explicit insmod.  There's still an issue
> with resolving symbols for this test to work, but bug #2151 would help.
> 
>> If the code isn't in the kernel and isn't built as a module, the
>> tests for those probe points should fail.  I know, the test didn't
>> actually run and fail, we just failed to test.  To me, that's still
>> a failure.
> 
> But it's a failure of a different sort.  That's what skipped &
> unsupported result codes are for.

I agree, it's not the same type of failure.  I'm just concerned we don't 
track skipped and unsupported results and make sure they're tested at some 
point.

> 
>> As for whether the module's in use, in some cases that doesn't matter.
>> For example, you can still load the nfs, nfsd and sunrpc modules and
>> run build tests against them even though they're not being used.  This
>> may not be true for all modules.
> 
> I suppose I can accept that, as long as you can restore the system when
> you're done.  IIRC, some modules may trigger loading other modules
> though, so cleanup may be difficult.

Restoring the system is a good goal, but again I don't think it should be a 
higher priority than testing everything.

> 
>>>  Can you instead write your test to gracefully skip if the module
>>> isn't present?  (It shouldn't record a failure...)
>>  I don't think untested tapsets should be silently ignored.  If we
>> don't figure out how to test *everything*, even if it's hard, how
>> will we know when some change in the kernel breaks a tapset?  When a
>> systemtap user stumbles on the problem?  IMHO, that's unacceptable.
> 
> I don't mean to call it a 'pass' -- again, that's what skipped &
> unsupported are for.  It's important to distinguish between "I can't
> test that on this system" and "something is really broken".

The bottom line for me is how do we make sure every tapset can at least be 
built, including those that require that modules be loaded?  I realize 
triggering the probe points is impossible in some environments, but 
resolving the probe points and making sure the resulting code builds is 
doable.  I'm open to suggestions.

Mike

> 
> Josh

Re: Tests that require 'modprobe <module>'

[Frank Ch. Eigler]

Mike Mason <mmlnx@us.ibm.com> writes:

> I'd like to add some tests that require certain modules be inserted
> (specifically nfs & rpc).  Where's the best place in the test
> framework to do modprobe?

What to these particular tests do differently if the probed modules
are loaded?  buildok tests work just as well.

- FChE

Re: Tests that require 'modprobe <module>'

[Mike Mason]

Frank Ch. Eigler wrote:
> Mike Mason <mmlnx@us.ibm.com> writes:
> 
>> I'd like to add some tests that require certain modules be inserted
>> (specifically nfs & rpc).  Where's the best place in the test
>> framework to do modprobe?
> 
> What to these particular tests do differently if the probed modules
> are loaded?  buildok tests work just as well.

Hmm... you're right.  I was thinking the module had to be loaded to do any 
kind of test.  Obviously that's wrong.  Never mind :-)

If we ever want to test the insertion of a kprobe into a particular module, 
that module would obviously have to be loaded, but I'm just trying to do 
build testing.

Mike

> 
> - FChE