From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) by sourceware.org (Postfix) with ESMTPS id 245873858404 for ; Tue, 26 Sep 2023 15:46:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 245873858404 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-5044dd5b561so9422326e87.1 for ; Tue, 26 Sep 2023 08:46:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1695743169; x=1696347969; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=+arqN3boa2kTOppX6VXOG5Iz+j8u0Br3ARQ5o6UarO8=; b=dktaZQBy+6zxVdSVhGIenNlKhw7hcll1EV/LgPooUHL2XSpL7GGR7nCN7Kfi0jITuZ SNhtzbFTBy0cU+KRQHFydAYtgRpVEmRqId8qEPjCU8yGiYBxnFz7mLrQ9oImFiGWScLR 7crwCjZzkYiqqeTo6fMmfqMDpWCDws3lUPp/7OqVDX4laK+mqfUfX+iNMOR4bAzocB4g AxVccKKNTI556AGXRqnIaIzVs0UTpiuI9ftFovwT+wwxXB1RF2wLFkY0Fwrt4H+PrRsm USZLQoifHfdCbJi6pKPJeIeOaK9pmxHvOkcQvuV4b8+PNPAab/Uchyl7Atie1gDvad6o xDVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695743169; x=1696347969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+arqN3boa2kTOppX6VXOG5Iz+j8u0Br3ARQ5o6UarO8=; b=iYIO+hwtfNZBwVdaxSnxbCD9LUmLCk2Qpw43CCm8rbQRUNqPh2PBY9VSXnafcG01Vy IWFodOpNb1kT3eGl8EzYkgbpRdnWshRsKS6cjckir+xLDkt//Ps7GMoS4ffRXTtjx979 ZCDIBCzPBEj1IIxTiQRQHaHwufoz3FQG2t35H0brOO8+QiBNMWvyrTCMJzzTb1tMe3xb kRHLPV3coLaa8ujimNccq3o0To+EawVFE4sbIR4OawAxUJDDQQYb4ucJA+b5KpQ5V7N4 8aEWq5sznwQ+Vnlovb+VGzR2pmhX3BvvLs4Kqprgqwqr8SR4UwtKdiuloTv0NDaeHO5x DWCQ== X-Gm-Message-State: AOJu0YzABr0ig8pW5t3WgzPagRDRHQlNxAbdnLKuqhB6WF2tYle2JxVj JOHP4/zgqz+DYtUfOov0DB4= X-Google-Smtp-Source: AGHT+IF1dy/7LiquskWfT+UELOWzfZPnbqNs+1FaQiHWN2+NA4j8JctJfesH/rKGbAz08MeUB3NPIw== X-Received: by 2002:a05:6512:3048:b0:503:eae:4896 with SMTP id b8-20020a056512304800b005030eae4896mr10392478lfb.39.1695743168518; Tue, 26 Sep 2023 08:46:08 -0700 (PDT) Received: from nbbrfq.loc (80-110-214-113.static.upcbusiness.at. [80.110.214.113]) by smtp.gmail.com with ESMTPSA id ek10-20020a056402370a00b00533dd4d2947sm3728051edb.74.2023.09.26.08.46.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Sep 2023 08:46:07 -0700 (PDT) Date: Tue, 26 Sep 2023 17:46:04 +0200 From: Bernhard Reutner-Fischer To: Maxim Kuvyrkov via Gcc-patches Cc: rep.dot.nop@gmail.com, Maxim Kuvyrkov , Jeff Law , Diego Novillo , Doug Evans Subject: Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing Message-ID: <20230926174604.4266b7c7@nbbrfq.loc> In-Reply-To: References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> <20230602152052.1874860-2-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi Maxim! On Mon, 5 Jun 2023 18:06:25 +0400 Maxim Kuvyrkov via Gcc-patches wrote: > > On Jun 3, 2023, at 19:17, Jeff Law wrote: > > > > On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: > >> This patch adds tracking of current testsuite "tool" and "exp" > >> to the processing of .sum files. This avoids aliasing between > >> tests from different testsuites with same name+description. > >> E.g., this is necessary for testsuite/c-c++-common, which is ran > >> for both gcc and g++ "tools". > >> This patch changes manifest format from ... > >> > >> FAIL: gcc_test > >> FAIL: g++_test > >> > >> ... to ... > >> > >> === gcc tests === > >> Running gcc/foo.exp ... > >> FAIL: gcc_test > >> === gcc Summary == > >> === g++ tests === > >> Running g++/bar.exp ... > >> FAIL: g++_test > >> === g++ Summary == > >> . > >> The new format uses same formatting as DejaGnu's .sum files > >> to specify which "tool" and "exp" the test belongs to. > > I think the series is fine. You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore. > > > > Thanks, Jeff. I'll wait for a couple of days and will merge if there are no new comments. Maxim, may i ask you to have a look at the following problem, please? ISTM that your exp code does not work as expected for go, maybe you forgot to test the changes with go enabled? Ever since your changes in summer i see the following: gcc-14.mine$ /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py --clean_build ../gcc-14.orig/ Getting actual results from build directory . ./gcc/testsuite/go/go.sum ./gcc/testsuite/gcc/gcc.sum ./gcc/testsuite/objc/objc.sum ./gcc/testsuite/jit/jit.sum ./gcc/testsuite/gdc/gdc.sum ./gcc/testsuite/gnat/gnat.sum ./gcc/testsuite/ada/acats/acats.sum ./gcc/testsuite/g++/g++.sum ./gcc/testsuite/obj-c++/obj-c++.sum ./gcc/testsuite/rust/rust.sum ./gcc/testsuite/gfortran/gfortran.sum ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum ./x86_64-pc-linux-gnu/libgo/libgo.sum ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum ./gotools/gotools.sum .sum file seems to be broken: tool="gotools", exp="None", summary_line="FAIL: TestScript" Traceback (most recent call last): File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 732, in retval = Main(sys.argv) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 721, in Main retval = CompareBuilds() File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 622, in CompareBuilds actual = GetResults(sum_files) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 466, in GetResults build_results.update(ParseSummary(sum_fname)) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 405, in ParseSummary result = result_set.MakeTestResult(line, ordinal) File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 239, in MakeTestResult return TestResult(summary_line, ordinal, File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 151, in __init__ raise RuntimeError: No active exception to reraise The problem seems to be that gotools.sum does not mention any ".exp" files. $ grep "Running " gotools/gotools.sum Running cmd/go Running runtime Running cgo Running carchive Running cmd/vet Running embed $ grep -c "\.exp" gotools/gotools.sum 0 The .sum files looks like this: ---8<--- Test Run By foo on Tue Sep 26 14:46:48 CEST 2023 Native configuration is x86_64-foo-linux-gnu === gotools tests === Running cmd/go UNTESTED: TestAccidentalGitCheckout PASS: TestAlwaysLinkSysoFiles ... UNTESTED: TestParallelTest FAIL: TestScript ... ---8<--- May i ask you to have a look, please? TIA,