Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing

[Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>]

Hi Bernhard,

Thanks, I meant to fix this, but forgot.

The underlying problem here is that we want to detect which sub-testsuites had failures.  Current regex doesn't match go's case because there is no "..." at the end: "Running foo" vs "Running foo ..." .

My preferred way of fixing this is to make go's testsuite print out "..." .  We have a similar patch for glibc [1].

[1] https://sourceware.org/pipermail/libc-alpha/2023-June/148702.html

--
Maxim Kuvyrkov
https://www.linaro.org

> On Sep 26, 2023, at 19:46, Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> wrote:
> 
> Hi Maxim!
> 
> On Mon, 5 Jun 2023 18:06:25 +0400
> Maxim Kuvyrkov via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
> 
>>> On Jun 3, 2023, at 19:17, Jeff Law <jeffreyalaw@gmail.com> wrote:
>>> 
>>> On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote:  
>>>> This patch adds tracking of current testsuite "tool" and "exp"
>>>> to the processing of .sum files.  This avoids aliasing between
>>>> tests from different testsuites with same name+description.
>>>> E.g., this is necessary for testsuite/c-c++-common, which is ran
>>>> for both gcc and g++ "tools".
>>>> This patch changes manifest format from ...
>>>> <cut>
>>>> FAIL: gcc_test
>>>> FAIL: g++_test
>>>> </cut>
>>>> ... to ...
>>>> <cut>
>>>> === gcc tests ===
>>>> Running gcc/foo.exp ...
>>>> FAIL: gcc_test
>>>> === gcc Summary ==
>>>> === g++ tests ===
>>>> Running g++/bar.exp ...
>>>> FAIL: g++_test
>>>> === g++ Summary ==
>>>> </cut>.
>>>> The new format uses same formatting as DejaGnu's .sum files
>>>> to specify which "tool" and "exp" the test belongs to.  
>>> I think the series is fine.  You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore.
>>> 
>> 
>> Thanks, Jeff.  I'll wait for a couple of days and will merge if there are no new comments.
> 
> Maxim, may i ask you to have a look at the following problem, please?
> 
> ISTM that your exp code does not work as expected for go, maybe you
> forgot to test the changes with go enabled?
> 
> Ever since your changes in summer i see the following:
> 
> gcc-14.mine$ /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py --clean_build ../gcc-14.orig/
> Getting actual results from build directory .
> ./gcc/testsuite/go/go.sum
> ./gcc/testsuite/gcc/gcc.sum
> ./gcc/testsuite/objc/objc.sum
> ./gcc/testsuite/jit/jit.sum
> ./gcc/testsuite/gdc/gdc.sum
> ./gcc/testsuite/gnat/gnat.sum
> ./gcc/testsuite/ada/acats/acats.sum
> ./gcc/testsuite/g++/g++.sum
> ./gcc/testsuite/obj-c++/obj-c++.sum
> ./gcc/testsuite/rust/rust.sum
> ./gcc/testsuite/gfortran/gfortran.sum
> ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum
> ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum
> ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
> ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum
> ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum
> ./x86_64-pc-linux-gnu/libgo/libgo.sum
> ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum
> ./gotools/gotools.sum
> .sum file seems to be broken: tool="gotools", exp="None", summary_line="FAIL: TestScript"
> Traceback (most recent call last):
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 732, in <module>
>    retval = Main(sys.argv)
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 721, in Main
>    retval = CompareBuilds()
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 622, in CompareBuilds
>    actual = GetResults(sum_files)
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 466, in GetResults
>    build_results.update(ParseSummary(sum_fname))
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 405, in ParseSummary
>    result = result_set.MakeTestResult(line, ordinal)
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 239, in MakeTestResult
>    return TestResult(summary_line, ordinal,
>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 151, in __init__
>    raise
> RuntimeError: No active exception to reraise
> 
> 
> The problem seems to be that gotools.sum does not mention any ".exp"
> files.
> 
> $ grep "Running " gotools/gotools.sum 
> Running cmd/go
> Running runtime
> Running cgo
> Running carchive
> Running cmd/vet
> Running embed
> $ grep -c "\.exp" gotools/gotools.sum 
> 0
> 
> The .sum files looks like this:
> ---8<---
> Test Run By foo on Tue Sep 26 14:46:48 CEST 2023
> Native configuration is x86_64-foo-linux-gnu
> 
>                === gotools tests ===
> 
> Running cmd/go
> UNTESTED: TestAccidentalGitCheckout
> PASS: TestAlwaysLinkSysoFiles
> ...
> UNTESTED: TestParallelTest
> FAIL: TestScript
> ...
> ---8<---
> 
> May i ask you to have a look, please?
> 
> TIA,