Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing

[Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>]

Hi Maxim!

On Mon, 5 Jun 2023 18:06:25 +0400
Maxim Kuvyrkov via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:

> > On Jun 3, 2023, at 19:17, Jeff Law <jeffreyalaw@gmail.com> wrote:
> > 
> > On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote:  
> >> This patch adds tracking of current testsuite "tool" and "exp"
> >> to the processing of .sum files.  This avoids aliasing between
> >> tests from different testsuites with same name+description.
> >> E.g., this is necessary for testsuite/c-c++-common, which is ran
> >> for both gcc and g++ "tools".
> >> This patch changes manifest format from ...
> >> <cut>
> >> FAIL: gcc_test
> >> FAIL: g++_test
> >> </cut>
> >> ... to ...
> >> <cut>
> >> === gcc tests ===
> >> Running gcc/foo.exp ...
> >> FAIL: gcc_test
> >> === gcc Summary ==
> >> === g++ tests ===
> >> Running g++/bar.exp ...
> >> FAIL: g++_test
> >> === g++ Summary ==
> >> </cut>.
> >> The new format uses same formatting as DejaGnu's .sum files
> >> to specify which "tool" and "exp" the test belongs to.  
> > I think the series is fine.  You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore.
> >   
> 
> Thanks, Jeff.  I'll wait for a couple of days and will merge if there are no new comments.

Maxim, may i ask you to have a look at the following problem, please?

ISTM that your exp code does not work as expected for go, maybe you
forgot to test the changes with go enabled?

Ever since your changes in summer i see the following:

gcc-14.mine$ /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py --clean_build ../gcc-14.orig/
Getting actual results from build directory .
	./gcc/testsuite/go/go.sum
	./gcc/testsuite/gcc/gcc.sum
	./gcc/testsuite/objc/objc.sum
	./gcc/testsuite/jit/jit.sum
	./gcc/testsuite/gdc/gdc.sum
	./gcc/testsuite/gnat/gnat.sum
	./gcc/testsuite/ada/acats/acats.sum
	./gcc/testsuite/g++/g++.sum
	./gcc/testsuite/obj-c++/obj-c++.sum
	./gcc/testsuite/rust/rust.sum
	./gcc/testsuite/gfortran/gfortran.sum
	./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum
	./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum
	./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
	./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum
	./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum
	./x86_64-pc-linux-gnu/libgo/libgo.sum
	./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum
	./gotools/gotools.sum
.sum file seems to be broken: tool="gotools", exp="None", summary_line="FAIL: TestScript"
Traceback (most recent call last):
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 732, in <module>
    retval = Main(sys.argv)
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 721, in Main
    retval = CompareBuilds()
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 622, in CompareBuilds
    actual = GetResults(sum_files)
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 466, in GetResults
    build_results.update(ParseSummary(sum_fname))
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 405, in ParseSummary
    result = result_set.MakeTestResult(line, ordinal)
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 239, in MakeTestResult
    return TestResult(summary_line, ordinal,
  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 151, in __init__
    raise
RuntimeError: No active exception to reraise


The problem seems to be that gotools.sum does not mention any ".exp"
files.

$ grep "Running " gotools/gotools.sum 
Running cmd/go
Running runtime
Running cgo
Running carchive
Running cmd/vet
Running embed
$ grep -c "\.exp" gotools/gotools.sum 
0

The .sum files looks like this:
---8<---
Test Run By foo on Tue Sep 26 14:46:48 CEST 2023
Native configuration is x86_64-foo-linux-gnu

                === gotools tests ===

Running cmd/go
UNTESTED: TestAccidentalGitCheckout
PASS: TestAlwaysLinkSysoFiles
...
UNTESTED: TestParallelTest
FAIL: TestScript
...
---8<---

May i ask you to have a look, please?

TIA,