From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 24451 invoked by alias); 13 May 2003 11:50:19 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 23493 invoked from network); 13 May 2003 11:50:05 -0000 Received: from unknown (HELO ms-dienst.rz.rwth-aachen.de) (134.130.3.130) by sources.redhat.com with SMTP; 13 May 2003 11:50:05 -0000 Received: from ms-2 (ms-2 [134.130.3.131]) by ms-dienst.rz.rwth-aachen.de (iPlanet Messaging Server 5.2 HotFix 1.12 (built Feb 13 2003)) with ESMTP id <0HET009KCQ4P77@ms-dienst.rz.rwth-aachen.de> for gcc@gcc.gnu.org; Tue, 13 May 2003 13:48:26 +0200 (MEST) Received: from relay.RWTH-Aachen.DE ([134.130.3.1]) by ms-2 (MailMonitor for SMTP v1.2.2 ) ; Tue, 13 May 2003 13:48:25 +0200 (MEST) Received: from numa-vi (numa-vi.igpm.RWTH-Aachen.DE [134.130.161.247]) by relay.rwth-aachen.de (8.12.9/8.12.7-1) with ESMTP id h4DBmONZ006303; Tue, 13 May 2003 13:48:24 +0200 (MEST) Date: Tue, 13 May 2003 11:50:00 -0000 From: Volker Reichelt Subject: Testcases in Bugzilla To: gcc@gcc.gnu.org Cc: giovannibajo@libero.it, reichelt@igpm.rwth-aachen.de, ehrhardt@mathematik.uni-ulm.de Reply-to: Volker Reichelt Message-id: <200305131148.h4DBmONZ006303@relay.rwth-aachen.de> MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=us-ascii Content-transfer-encoding: 7BIT Content-disposition: INLINE X-SW-Source: 2003-05/txt/msg01282.txt.bz2 Hi again, we still haven't reached a conclusion about the testcases in Bugzilla. Bugzilla offers us the possibility to mark an attachment as testcase and a possibility to download them as pointed out in http://gcc.gnu.org/ml/gcc/2003-05/msg01212.html But how can we actually test the testcases? The dejagnu testing harness doesn't allow you to specify a branch or something like that where the test is supposed to pass/fail AFAIK. And without such a feature the whole testing becomes a nightmare. There are some resolutions, but they all have their drawbacks: a) Create a testcase for each branch. (Way too much work IMHO.) b) Mark all tests as XPASS and supply lists for each branch with the failures. (Too complicated in practice IMHO. And what about other architectures?) c) Add a couple of lines at the top of each testcase that specify the compiler versions where we should XFAIL. Run them through a little preprocessor script that removes these lines and adds an XFAIL where appropriate. Hand the result over to dejagnu. d) We can rely on my testing script which is just a small shell script which doesn't use dejagnu and therefore has only limited capabilities, but works for a lot of testcases and can deal with different compiler versions. (See http://gcc.gnu.org/ml/gcc/2002-12/msg01250.html ) In that case I'd post a revamped version. e) We could modify the script in d) to accomplish c). f) Extend dejagnu with versioning. g) ...? I think we should reach a conclusion *before* we start adding the testcases to bugzilla so that we don't have to modify the testcases over and over again. Any suggestions? Regards, Volker PS: To avoid name clashes between testcases, we should agree on a naming convention (e.g. PR12345A.c).