KFAIL tests in testsuite

KFAIL tests in testsuite

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

I am looking through and appropriately markiung tests that fail due to known 
reasons as KFAIL. So something like the attached patch.

The one concern that I have about marking some tests as KFAIL is that failure 
may vary on the environment. Some versions of supporting software or kernels may 
cause a test to fail. However, the test doesn't uniformly fail. It may fail for 
other reasons on some environments and KFAIL may obscure that. The KFAIL marking 
seems a bit coarse grained. Any comments about the KFAIL marking or the propsed 
patch?

-Will


2008-01-30  Will Cohen <wcohen@redhat.com>

	* systemtap.pass1-4/buildok.exp: Add some kfails.

[-- Attachment #2: kfail_buildok_seventeen.diff --]
[-- Type: text/x-patch, Size: 684 bytes --]

? testsuite/systemtap.syscall/hidden
Index: testsuite/systemtap.pass1-4/buildok.exp
===================================================================
RCS file: /cvs/systemtap/src/testsuite/systemtap.pass1-4/buildok.exp,v
retrieving revision 1.8
diff -U2 -u -r1.8 buildok.exp
--- testsuite/systemtap.pass1-4/buildok.exp	27 Aug 2007 14:18:20 -0000	1.8
+++ testsuite/systemtap.pass1-4/buildok.exp	30 Jan 2008 16:45:44 -0000
@@ -7,4 +7,5 @@
     switch $test {
         buildok/perfmon01.stp {setup_kfail 909 *-*-*}
+        buildok/seventeen.stp {setup_kfail 4393 *-*-*}
         buildok/twentysix.stp {setup_kfail 4105 *-*-*}
         buildok/twentyseven.stp {setup_kfail 4166 *-*-*}

Re: KFAIL tests in testsuite

[Frank Ch. Eigler]

William Cohen <wcohen@redhat.com> writes:

> I am looking through and appropriately markiung tests that fail due to
> known reasons as KFAIL. So something like the attached patch.

This one is fine.

> The one concern that I have about marking some tests as KFAIL is
> that failure may vary on the environment. Some versions of
> supporting software or kernels may cause a test to fail. However,
> the test doesn't uniformly fail. [...]

Right, though if this becomes a problem (so one can't tell whether
one's dealing with something new), then one can conditionalize the
setup_kfail clauses further, based on architecture, versions, and
maybe even a bit of whatnot(tm).

- FChE