Re: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing

[rep.dot.nop@gmail.com]

On 27 September 2023 16:47:27 CEST, Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org> wrote:
>Hi Bernhard,
>
>Thanks, I meant to fix this, but forgot.

np.

>The underlying problem here is that we want to detect which sub-testsuites had failures.  Current regex doesn't match go's case because there is no "..." at the end: "Running foo" vs "Running foo ..." .
>
>My preferred way of fixing this is to make go's testsuite print out "..." .  We have a similar patch for glibc [1].
>
>[1] https://sourceware.org/pipermail/libc-alpha/2023-June/148702.html

Which asks:
---8<---
>> WDYT?
> 
> I looked at the gcc-testresults mailing list, and there appear no
> === … failures === lines at all?  What was the motivation for adding it
> in the first place?

The only motivation is that it looks like a nice header for the following FAILs.  What's your preference for the line -- drop it entirely or print out:

=== glibc failures ===
no unexpected failures

?
---8<---

I'd drop the above entirely if there are no failures, it's pretty superfluous, isn't it.

And concerning gotools and the missing trailing ".exp ...", I guess it's fine to add that to streamline the gotools output to all the other existing sum output.

TIA,


>
>--
>Maxim Kuvyrkov
>https://www.linaro.org
>
>> On Sep 26, 2023, at 19:46, Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> wrote:
>> 
>> Hi Maxim!
>> 
>> On Mon, 5 Jun 2023 18:06:25 +0400
>> Maxim Kuvyrkov via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>> 
>>>> On Jun 3, 2023, at 19:17, Jeff Law <jeffreyalaw@gmail.com> wrote:
>>>> 
>>>> On 6/2/23 09:20, Maxim Kuvyrkov via Gcc-patches wrote:  
>>>>> This patch adds tracking of current testsuite "tool" and "exp"
>>>>> to the processing of .sum files.  This avoids aliasing between
>>>>> tests from different testsuites with same name+description.
>>>>> E.g., this is necessary for testsuite/c-c++-common, which is ran
>>>>> for both gcc and g++ "tools".
>>>>> This patch changes manifest format from ...
>>>>> <cut>
>>>>> FAIL: gcc_test
>>>>> FAIL: g++_test
>>>>> </cut>
>>>>> ... to ...
>>>>> <cut>
>>>>> === gcc tests ===
>>>>> Running gcc/foo.exp ...
>>>>> FAIL: gcc_test
>>>>> === gcc Summary ==
>>>>> === g++ tests ===
>>>>> Running g++/bar.exp ...
>>>>> FAIL: g++_test
>>>>> === g++ Summary ==
>>>>> </cut>.
>>>>> The new format uses same formatting as DejaGnu's .sum files
>>>>> to specify which "tool" and "exp" the test belongs to.  
>>>> I think the series is fine.  You're not likely to hear from Diego or Doug I suspect, I don't think either are involved in GNU stuff anymore.
>>>> 
>>> 
>>> Thanks, Jeff.  I'll wait for a couple of days and will merge if there are no new comments.
>> 
>> Maxim, may i ask you to have a look at the following problem, please?
>> 
>> ISTM that your exp code does not work as expected for go, maybe you
>> forgot to test the changes with go enabled?
>> 
>> Ever since your changes in summer i see the following:
>> 
>> gcc-14.mine$ /scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py --clean_build ../gcc-14.orig/
>> Getting actual results from build directory .
>> ./gcc/testsuite/go/go.sum
>> ./gcc/testsuite/gcc/gcc.sum
>> ./gcc/testsuite/objc/objc.sum
>> ./gcc/testsuite/jit/jit.sum
>> ./gcc/testsuite/gdc/gdc.sum
>> ./gcc/testsuite/gnat/gnat.sum
>> ./gcc/testsuite/ada/acats/acats.sum
>> ./gcc/testsuite/g++/g++.sum
>> ./gcc/testsuite/obj-c++/obj-c++.sum
>> ./gcc/testsuite/rust/rust.sum
>> ./gcc/testsuite/gfortran/gfortran.sum
>> ./x86_64-pc-linux-gnu/libgomp/testsuite/libgomp.sum
>> ./x86_64-pc-linux-gnu/libphobos/testsuite/libphobos.sum
>> ./x86_64-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
>> ./x86_64-pc-linux-gnu/libffi/testsuite/libffi.sum
>> ./x86_64-pc-linux-gnu/libitm/testsuite/libitm.sum
>> ./x86_64-pc-linux-gnu/libgo/libgo.sum
>> ./x86_64-pc-linux-gnu/libatomic/testsuite/libatomic.sum
>> ./gotools/gotools.sum
>> .sum file seems to be broken: tool="gotools", exp="None", summary_line="FAIL: TestScript"
>> Traceback (most recent call last):
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 732, in <module>
>>    retval = Main(sys.argv)
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 721, in Main
>>    retval = CompareBuilds()
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 622, in CompareBuilds
>>    actual = GetResults(sum_files)
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 466, in GetResults
>>    build_results.update(ParseSummary(sum_fname))
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 405, in ParseSummary
>>    result = result_set.MakeTestResult(line, ordinal)
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 239, in MakeTestResult
>>    return TestResult(summary_line, ordinal,
>>  File "/scratch/src/gcc-14.mine/contrib/testsuite-management/validate_failures.py", line 151, in __init__
>>    raise
>> RuntimeError: No active exception to reraise
>> 
>> 
>> The problem seems to be that gotools.sum does not mention any ".exp"
>> files.
>> 
>> $ grep "Running " gotools/gotools.sum 
>> Running cmd/go
>> Running runtime
>> Running cgo
>> Running carchive
>> Running cmd/vet
>> Running embed
>> $ grep -c "\.exp" gotools/gotools.sum 
>> 0
>> 
>> The .sum files looks like this:
>> ---8<---
>> Test Run By foo on Tue Sep 26 14:46:48 CEST 2023
>> Native configuration is x86_64-foo-linux-gnu
>> 
>>                === gotools tests ===
>> 
>> Running cmd/go
>> UNTESTED: TestAccidentalGitCheckout
>> PASS: TestAlwaysLinkSysoFiles
>> ...
>> UNTESTED: TestParallelTest
>> FAIL: TestScript
>> ...
>> ---8<---
>> 
>> May i ask you to have a look, please?
>> 
>> TIA,
>
>