From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 10990 invoked by alias); 13 Sep 2007 12:34:03 -0000 Received: (qmail 10980 invoked by uid 22791); 13 Sep 2007 12:34:02 -0000 X-Spam-Status: No, hits=-0.1 required=5.0 tests=AWL,BAYES_00,DK_POLICY_SIGNSOME,UNPARSEABLE_RELAY X-Spam-Check-By: sourceware.org Received: from agminet01.oracle.com (HELO agminet01.oracle.com) (141.146.126.228) by sourceware.org (qpsmtpd/0.31) with ESMTP; Thu, 13 Sep 2007 12:33:49 +0000 Received: from agmgw2.us.oracle.com (agmgw2.us.oracle.com [152.68.180.213]) by agminet01.oracle.com (Switch-3.2.4/Switch-3.1.7) with ESMTP id l8DCXiNp016024 for ; Thu, 13 Sep 2007 07:33:44 -0500 Received: from acsmt351.oracle.com (acsmt351.oracle.com [141.146.40.151]) by agmgw2.us.oracle.com (Switch-3.2.0/Switch-3.2.0) with ESMTP id l8D5DgsF027667 for ; Thu, 13 Sep 2007 06:33:44 -0600 Received: from alchar.org by acsmt352.oracle.com with ESMTP id 3209444031189686724; Thu, 13 Sep 2007 05:32:04 -0700 Date: Thu, 13 Sep 2007 12:34:00 -0000 From: Kris Van Hees To: frysk@sourceware.org Cc: Kris Van Hees Subject: Re: Patch for TearDownProcess Message-ID: <20070913123203.GD21435@oracle.com> References: <20070907022100.GK22263@oracle.com> <46E15274.5020400@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46E15274.5020400@redhat.com> User-Agent: Mutt/1.5.16 (2007-06-09) X-Brightmail-Tracker: AAAAAQAAAAI= X-Brightmail-Tracker: AAAAAQAAAAI= X-Whitelist: TRUE X-Whitelist: TRUE X-IsSubscribed: yes Mailing-List: contact frysk-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: frysk-owner@sourceware.org X-SW-Source: 2007-q3/txt/msg00394.txt.bz2 In view of the quote below, and history over the past couple of months, I'd like to pose an important question to the overall developer community behind Frysk: Should we strive to have a consistent, dependable testsuite? The reason to ask is that on one hand, it seems some people prefer to look at tests as individual units, regardless of their side-effects on later tests in the testsuite (see the past discussions about the file descriptor leak testing and its effects on the testsuite), whereas others prefer to see the testsuite as a collection of tests that needs to be capable of producing dependable results regardless of the outcome of one or more individual tests. Right now it seems that not everyone is even on a specific side of the issue, preferring one behaviour for some test cases, and the other for other test cases. The end result is that the Frysk testsuite is not producing results that can be considered a valid indicator in terms of correctness. In addition, just yesterday we encountered a case where some build-time testing is conditional, causing a commit to go through (tested on F7 prior to commit) that broke the build on other platforms (FC6) because on the latter platform the test was enabled. That kind of thing obviously gives developers a wrong impression on whether their patch is going to be break builds or not. Today I am seeing the following (first time I say this was July 3rd, and it was reported in bugzilla): | Running testDataFuncPeekBytes(frysk.sys.TestPtrace) ...PASS | Running testDataFuncPeekBytes(frysk.sys.TestPtrace) ...ERROR | close: Bad file descriptor (fd 0) | Running testLengthUnderBound(frysk.sys.TestPtrace) ...PASS | Running testLengthUnderBound(frysk.sys.TestPtrace) ...ERROR | close: Bad file descriptor (fd 0) | Running testOffsetUnderBound(frysk.sys.TestPtrace) ...PASS | Running testOffsetUnderBound(frysk.sys.TestPtrace) ...ERROR | close: Bad file descriptor (fd 0) | Running testLengthOverBound(frysk.sys.TestPtrace) ...PASS | Running testLengthOverBound(frysk.sys.TestPtrace) ...ERROR | close: Bad file descriptor (fd 0) | Running testOffsetOverBound(frysk.sys.TestPtrace) ...PASS | Running testOffsetOverBound(frysk.sys.TestPtrace) ...PASS | Running testLengthOnBound(frysk.sys.TestPtrace) ...ERROR | close: Bad file descriptor (fd 0) | Running testOffsetOnBound(frysk.sys.TestPtrace) ...PASS | Running testOffsetOnBound(frysk.sys.TestPtrace) ...PASS While it can be argued that a test reporting its result twice is hardly an issue, I would think that a single test reporting both a PASS *and* an ERROR result would be more reason for concern... In all, most development methodologies put a lot of value in testing. It is also my experience that even without a formal methodology, testing tends to be considered rather important. I do not believe that the Frysk testsuite can be taken seriously as long as we cannot have a consistent policy in terms of test quality. Either test we allow failing tests to potentially corrupt the validity of following tests, or we prohibit failing tests from corrupting the validity of following tests. Having it one way in some cases, and the other way in some other cases simply leads to not being able to trust the results of the testsuite at all. Cheers, Kris On Fri, Sep 07, 2007 at 09:30:28AM -0400, Andrew Cagney wrote: > Given the choices between a potential test-suite hang, and tear-down > leaving waitpid events around, the decision was made in favor of the > latter. That decision hasn't changed. Having the test run hang, is a > lesser evil then the test-suite continuing but producing bogus results.. > I restored the old behavior; and then added a timeout. It currently logs a > message, I suspect it should abandon the test run, since the problem state > hasn't gone away. > > Andrew >