From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 22109 invoked by alias); 12 Feb 2007 17:54:15 -0000 Received: (qmail 22101 invoked by uid 22791); 12 Feb 2007 17:54:14 -0000 X-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.31) with ESMTP; Mon, 12 Feb 2007 17:54:10 +0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l1CHs7Vm023983; Mon, 12 Feb 2007 12:54:07 -0500 Received: from pobox.corp.redhat.com (pobox.corp.redhat.com [10.11.255.20]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l1CHs7rZ000974; Mon, 12 Feb 2007 12:54:07 -0500 Received: from [127.0.0.1] (sebastian-int.corp.redhat.com [172.16.52.221]) by pobox.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l1CHs4aV021644; Mon, 12 Feb 2007 12:54:06 -0500 Message-ID: <45D0A9B6.8020108@redhat.com> Date: Mon, 12 Feb 2007 17:54:00 -0000 From: Andrew Cagney User-Agent: Thunderbird 1.5.0.9 (X11/20070102) MIME-Version: 1.0 To: Mark Wielaard CC: frysk@sourceware.org Subject: Re: Added frysk.junit.TestCase.brokenIfKernel(int bug, String[] kernels) References: <45CD125C.7060803@redhat.com> <1171282158.3010.5.camel@hermans.wildebeest.org> In-Reply-To: <1171282158.3010.5.camel@hermans.wildebeest.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact frysk-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: frysk-owner@sourceware.org X-SW-Source: 2007-q1/txt/msg00106.txt.bz2 Mark, Obviously, there are two requirements here: - the need to ensure that all tests are run, and thus recognize regressions Without this regressions can be unintentionally introduced. - have a clear baseline - the zero failures - from which to work from Without this it is never possible for a developer to know if their change has caused a regression on a specific system While test-first dogma assumes these are complementary, the realities of kernel, compiler, and other bugs means this isn't achievable. Some tests can't pass on some systems and consequently full tests runs are not normally possible. The compromise is at least to mark up the tests that can't pass on a specific system with that information - the brokenXXX flag. Doing this ensures that: - zero failures continues as the only acceptable baseline - no time is lost re-analyzing failures you or I have previously analyzed or one of our new tests has introduced Can we work with this? Andrew Mark Wielaard wrote: > Hi Andrew, > > On Fri, 2007-02-09 at 19:31 -0500, Andrew Cagney wrote: > >> Where a failure is occurring on a specific kernel, try to >> fix it, or at least mark it up as a known breakage. By doing this, >> people always have access to a failure free starting point, and with out >> it, no one is ever sure if their change or existing breakage is causing >> a failure. >> > > Although I don't mind people blacklisting some tests on their systems > because of buggy kernel, compiler, libaudit, etc. to make some things > more convenient for them. I am not sure how maintainable that is. We do > have explicit tests for broken systems in frysk-import/tests. If a > developer is working on a system that is know to be broken and buggy > (because they have frysk-import/tests fail one or more checks) it seems > that they better make sure to fix the bug in the dependencies and/or > migrate to a non-broken system. Otherwise they will be running into > trouble sooner or later anyway. And the skipped tests only paper over > the problem that will come back and bite them later on during > development. > > Cheers, > > Mark > >