From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) by sourceware.org (Postfix) with ESMTPS id 965EA3856DF2 for ; Wed, 27 Sep 2023 14:47:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 965EA3856DF2 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-503178a0d7fso3330406e87.1 for ; Wed, 27 Sep 2023 07:47:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1695826060; x=1696430860; darn=gcc.gnu.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=snfJ4EAyLUmddjWEoncmV2N2xcLxpgZie8znOtmXi2g=; b=xSJ18jHImspVcQ8I8Bpl5P5g9+TskXb7o1TZBWfeVjS8dGwV/WhIxiE0F5UybN780t fZJIgskp3S22+ZW6dor7t3mnB/dr7dQgMuXIKmG0bEm6xCz3Rg1R9Mgr8cai8vgbwVg5 U+krhmHrrtSWIsiszASDggv4KRpVu/ib3EDIgMqKKokOdtJxZjQuhQNd4+VwXFQ5lM/Z 3/d4frhpKpWxPJvFdYkPMHN9Mh7Apw8QEI0C7XGJEdyyHlapnlU2wwuuo8JTeWzwvley tKvWYZVStWy8JC41hFJkXIxHnOr+ZI2yI48Rf2+2br2WJ960Oiw4LPf0CxKe1OYo+Txz xJfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695826060; x=1696430860; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=snfJ4EAyLUmddjWEoncmV2N2xcLxpgZie8znOtmXi2g=; b=DR8U9Zol0rOGr5tbDN1bbwPyVv/gLJrBFLIpO2QXYspO3A7Zcx5jVR41pY6bstsDot pMVp8TBvF5t9bh+zXHBYP6T15trizl3mxmLimTx92clmSUKE6gc75UboY1TCwiGeD3me RolyJ6V3NuqajGlWDQQh8lZiQS6d1dP1giNi1a4ESY3VmBgjNhjywY1UCj1rgZulYIxy IoIt+6ylh9THLc9yE8uRJ9S+0hYhE3wS0Jrk0IdE01M3V2Pj2IsET4J1Up8YvGJFRPWd U9VVqdkfIH+fMzdwxYFue0Whvxn99g1H3Vyz+kw56Ja1VCn+Bsdkim99ampqJ70+B9Vt O/og== X-Gm-Message-State: AOJu0YwhG21bTcHCiq4UxsO7Wqg1NeqIW/K5nE1m3yQKJZoGucoFw8X2 aRbzNC4p3G3VqHzqkeAx8AYp X-Google-Smtp-Source: AGHT+IHnaOxHETVUasPLiP9g3XP+sTKssIa0iDKYd/3FyAmjQLPH0MqsJp/30GzLhi0AEgMJX6QVhg== X-Received: by 2002:ac2:5197:0:b0:503:46c:5577 with SMTP id u23-20020ac25197000000b00503046c5577mr1844920lfi.3.1695826059885; Wed, 27 Sep 2023 07:47:39 -0700 (PDT) Received: from smtpclient.apple ([37.252.81.6]) by smtp.gmail.com with ESMTPSA id g7-20020aa7d1c7000000b0053112d6a40esm8350261edp.82.2023.09.27.07.47.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Sep 2023 07:47:39 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing From: Maxim Kuvyrkov In-Reply-To: <20230926174604.4266b7c7@nbbrfq.loc> Date: Wed, 27 Sep 2023 18:47:27 +0400 Cc: Maxim Kuvyrkov via Gcc-patches , Jeff Law , Diego Novillo , Doug Evans Content-Transfer-Encoding: quoted-printable Message-Id: <5D44DB95-5D01-4D4A-AA36-499079DBA92D@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> <20230602152052.1874860-2-maxim.kuvyrkov@linaro.org> <20230926174604.4266b7c7@nbbrfq.loc> To: Bernhard Reutner-Fischer X-Mailer: Apple Mail (2.3731.700.6) X-Spam-Status: No, score=-5.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi Bernhard, Thanks, I meant to fix this, but forgot. The underlying problem here is that we want to detect which = sub-testsuites had failures. Current regex doesn't match go's case = because there is no "..." at the end: "Running foo" vs "Running foo ..." = . My preferred way of fixing this is to make go's testsuite print out = "..." . We have a similar patch for glibc [1]. [1] https://sourceware.org/pipermail/libc-alpha/2023-June/148702.html -- Maxim Kuvyrkov https://www.linaro.org > On Sep 26, 2023, at 19:46, Bernhard Reutner-Fischer = wrote: >=20 > Hi Maxim! >=20 > On Mon, 5 Jun 2023 18:06:25 +0400 > Maxim Kuvyrkov via Gcc-patches wrote: >=20 >>> On Jun 3, 2023, at 19:17, Jeff Law wrote: >>>=20 >>> On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote: =20 >>>> This patch adds tracking of current testsuite "tool" and "exp" >>>> to the processing of .sum files. This avoids aliasing between >>>> tests from different testsuites with same name+description. >>>> E.g., this is necessary for testsuite/c-c++-common, which is ran >>>> for both gcc and g++ "tools". >>>> This patch changes manifest format from ... >>>> >>>> FAIL: gcc_test >>>> FAIL: g++_test >>>> >>>> ... to ... >>>> >>>> =3D=3D=3D gcc tests =3D=3D=3D >>>> Running gcc/foo.exp ... >>>> FAIL: gcc_test >>>> =3D=3D=3D gcc Summary =3D=3D >>>> =3D=3D=3D g++ tests =3D=3D=3D >>>> Running g++/bar.exp ... >>>> FAIL: g++_test >>>> =3D=3D=3D g++ Summary =3D=3D >>>> . >>>> The new format uses same formatting as DejaGnu's .sum files >>>> to specify which "tool" and "exp" the test belongs to. =20 >>> I think the series is fine. You're not likely to hear from Diego or = Doug I suspect, I don't think either are involved in GNU stuff anymore. >>>=20 >>=20 >> Thanks, Jeff. I'll wait for a couple of days and will merge if there = are no new comments. >=20 > Maxim, may i ask you to have a look at the following problem, please? >=20 > ISTM that your exp code does not work as expected for go, maybe you > forgot to test the changes with go enabled? >=20 > Ever since your changes in summer i see the following: >=20 > gcc-14.mine$ = /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py= --clean_build ../gcc-14.orig/ > Getting actual results from build directory . > ./gcc/testsuite/go/go.sum > ./gcc/testsuite/gcc/gcc.sum > ./gcc/testsuite/objc/objc.sum > ./gcc/testsuite/jit/jit.sum > ./gcc/testsuite/gdc/gdc.sum > ./gcc/testsuite/gnat/gnat.sum > ./gcc/testsuite/ada/acats/acats.sum > ./gcc/testsuite/g++/g++.sum > ./gcc/testsuite/obj-c++/obj-c++.sum > ./gcc/testsuite/rust/rust.sum > ./gcc/testsuite/gfortran/gfortran.sum > ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum > ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum > ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum > ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum > ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum > ./x86_64-pc-linux-gnu/libgo/libgo.sum > ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum > ./gotools/gotools.sum > .sum file seems to be broken: tool=3D"gotools", exp=3D"None", = summary_line=3D"FAIL: TestScript" > Traceback (most recent call last): > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 732, in > retval =3D Main(sys.argv) > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 721, in Main > retval =3D CompareBuilds() > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 622, in CompareBuilds > actual =3D GetResults(sum_files) > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 466, in GetResults > build_results.update(ParseSummary(sum_fname)) > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 405, in ParseSummary > result =3D result_set.MakeTestResult(line, ordinal) > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 239, in MakeTestResult > return TestResult(summary_line, ordinal, > File = "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.p= y", line 151, in __init__ > raise > RuntimeError: No active exception to reraise >=20 >=20 > The problem seems to be that gotools.sum does not mention any ".exp" > files. >=20 > $ grep "Running " gotools/gotools.sum=20 > Running cmd/go > Running runtime > Running cgo > Running carchive > Running cmd/vet > Running embed > $ grep -c "\.exp" gotools/gotools.sum=20 > 0 >=20 > The .sum files looks like this: > ---8<--- > Test Run By foo on Tue Sep 26 14:46:48 CEST 2023 > Native configuration is x86_64-foo-linux-gnu >=20 > =3D=3D=3D gotools tests =3D=3D=3D >=20 > Running cmd/go > UNTESTED: TestAccidentalGitCheckout > PASS: TestAlwaysLinkSysoFiles > ... > UNTESTED: TestParallelTest > FAIL: TestScript > ... > ---8<--- >=20 > May i ask you to have a look, please? >=20 > TIA,