From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 6964 invoked by alias); 13 May 2003 18:33:58 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 22056 invoked from network); 13 May 2003 18:30:51 -0000 Received: from unknown (HELO e31.co.us.ibm.com) (32.97.110.129) by sources.redhat.com with SMTP; 13 May 2003 18:30:51 -0000 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e31.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h4DIUGQs222394; Tue, 13 May 2003 14:30:16 -0400 Received: from unknown.host (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h4DIUAGj018362; Tue, 13 May 2003 12:30:15 -0600 Received: (from janis@localhost) by unknown.host (8.9.3/8.9.3) id LAA02318; Tue, 13 May 2003 11:29:08 -0700 Date: Tue, 13 May 2003 18:33:00 -0000 From: Janis Johnson To: Volker Reichelt Cc: gcc@gcc.gnu.org, giovannibajo@libero.it, ehrhardt@mathematik.uni-ulm.de Subject: Re: Testcases in Bugzilla Message-ID: <20030513112908.A2292@us.ibm.com> References: <200305131148.h4DBmONZ006303@relay.rwth-aachen.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200305131148.h4DBmONZ006303@relay.rwth-aachen.de>; from reichelt@igpm.rwth-aachen.de on Tue, May 13, 2003 at 01:48:26PM +0200 X-SW-Source: 2003-05/txt/msg01340.txt.bz2 On Tue, May 13, 2003 at 01:48:26PM +0200, Volker Reichelt wrote: > Hi again, > > we still haven't reached a conclusion about the testcases in Bugzilla. No, but we're making progress. > Bugzilla offers us the possibility to mark an attachment as testcase > and a possibility to download them as pointed out in > > http://gcc.gnu.org/ml/gcc/2003-05/msg01212.html > > But how can we actually test the testcases? > The dejagnu testing harness doesn't allow you to specify a branch > or something like that where the test is supposed to pass/fail AFAIK. > And without such a feature the whole testing becomes a nightmare. The testing harness is actually quite flexible and lets you add whatever new functionality you need in the .exp files used by the tests. > There are some resolutions, but they all have their drawbacks: > > a) Create a testcase for each branch. (Way too much work IMHO.) No, although in some cases it might be necessary to have different test cases. > b) Mark all tests as XPASS and supply lists for each branch with the failures. > (Too complicated in practice IMHO. And what about other architectures?) We will want to be able to handle tests that are only expected to fail on some targets, as well as on some branches. > c) Add a couple of lines at the top of each testcase that specify the > compiler versions where we should XFAIL. Yes. > Run them through a little preprocessor script that removes these lines > and adds an XFAIL where appropriate. Hand the result over to dejagnu. The code in the .exp file could handle this directly. > d) We can rely on my testing script which is just a small shell script > which doesn't use dejagnu and therefore has only limited capabilities, > but works for a lot of testcases and can deal with different compiler > versions. (See http://gcc.gnu.org/ml/gcc/2002-12/msg01250.html ) > In that case I'd post a revamped version. I'd prefer to use the existing testing tools. > e) We could modify the script in d) to accomplish c). > > f) Extend dejagnu with versioning. Use DejaGNU functionality to support versioning. > g) ...? > > > I think we should reach a conclusion *before* we start adding the testcases > to bugzilla so that we don't have to modify the testcases over and over again. Yes, that's a good point. I can spend some time on this now. > Any suggestions? > > Regards, > Volker > > PS: To avoid name clashes between testcases, we should agree on a naming > convention (e.g. PR12345A.c). Or PR12345-1.c, in line with the existing convention for multiple tests of the same bug/fix. We'll need some kind of restriction on execute tests to assure people running these tests that they aren't going to do anything antisocial. Janis