Re: Added frysk.junit.TestCase.brokenIfKernel(int bug, String[] kernels)

[Andrew Cagney <cagney@redhat.com>]

Mark,

Obviously, there are two requirements here:

- the need to ensure that all tests are run, and thus recognize regressions
Without this regressions can be unintentionally introduced.

- have a clear baseline - the zero failures - from which to work from
Without this it is never possible for a developer to know if their 
change has caused a regression on a specific system

While test-first dogma assumes these are complementary, the realities of 
kernel, compiler, and other bugs means this isn't achievable.  Some 
tests can't pass on some systems and consequently full tests runs are 
not normally possible.

The compromise is at least to mark up the tests that can't pass on a 
specific system with that information - the brokenXXX flag.  Doing this 
ensures that:

- zero failures continues as the only acceptable baseline
- no time is lost re-analyzing failures you or I have previously 
analyzed or one of our new tests has introduced

Can we work with this?

Andrew

Mark Wielaard wrote:
> Hi Andrew,
>
> On Fri, 2007-02-09 at 19:31 -0500, Andrew Cagney wrote:
>   
>> Where a failure is occurring on a specific kernel, try to 
>> fix it, or at least mark it up as a known breakage.  By doing this, 
>> people always have access to a failure free starting point, and with out 
>> it, no one is ever sure if their change or existing breakage is causing 
>> a failure.
>>     
>
> Although I don't mind people blacklisting some tests on their systems
> because of buggy kernel, compiler, libaudit, etc. to make some things
> more convenient for them. I am not sure how maintainable that is. We do
> have explicit tests for broken systems in frysk-import/tests. If a
> developer is working on a system that is know to be broken and buggy
> (because they have frysk-import/tests fail one or more checks) it seems
> that they better make sure to fix the bug in the dependencies and/or
> migrate to a non-broken system. Otherwise they will be running into
> trouble sooner or later anyway. And the skipped tests only paper over
> the problem that will come back and bite them later on during
> development.
>
> Cheers,
>
> Mark
>
>