[PATCH] Format test results closer to what DejaGnu does

[PATCH] Format test results closer to what DejaGnu does

[Maxim Kuvyrkov]

The years of dealing with Binutils, GCC and GDB test results
made the community create good tools for comparison and analysis
of DejaGnu test results.  This change allows to use those tools
for Glibc's test results as well.

The motivation for this change is Linaro's pre-commit testers,
which use a modified version of GCC's validate_failures.py
to create test xfail lists with baseline failures and known
flaky tests.  See below links for an example xfails file (only
one link is supposed to work at any given time):
- https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/
- https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/

Specifacally, this patch changes format of glibc's .sum files from ...
<cut>
FAIL: elf/test1
PASS: string/test2
</cut>
... to ...
<cut>
             === glibc tests ===

Running elf ...
FAIL: elf/test1

Running string ...
PASS: string/test2
</cut>.

And output of "make check" from ...
<cut>
FAIL: elf/test1
</cut>
... to ...
<cut>
		=== glibc failures ===
FAIL: elf/test1
		=== Summary of results ===
      1 FAIL
      1 PASS
</cut>.

Signed-off-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
---
 Makefile                      | 12 ++++++++----
 scripts/merge-test-results.sh |  3 +++
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 224c792185..779938f3b7 100644
--- a/Makefile
+++ b/Makefile
@@ -564,11 +564,15 @@ $(objpfx)check-wrapper-headers.out: scripts/check-wrapper-headers.py $(headers)
 	  --generated $(common-generated) > $@; $(evaluate-test)
 endif # $(headers)
 
+# Print test summary for tests in $1 .sum file;
+# $2 is optional test identifier.
+# Fail if there are unexpected failures in the test results.
 define summarize-tests
-@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
-@echo "Summary of test results$2:"
-@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c
-@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1
+@echo "		=== glibc failures ==="
+@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
+@echo "		=== Summary of results$2 ==="
+@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
+@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
 endef
 
 # The intention here is to do ONE install of our build into the
diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
index e4dcc2520a..8f9d81f6eb 100755
--- a/scripts/merge-test-results.sh
+++ b/scripts/merge-test-results.sh
@@ -50,7 +50,10 @@ case $type in
   -t)
     subdir_file_name=$1
     shift
+    echo "		=== glibc tests ==="
     for d in "$@"; do
+      echo
+      echo "Running $d ..."
       if [ -f "$objpfx$d/$subdir_file_name" ]; then
 	cat "$objpfx$d/$subdir_file_name"
       else
-- 
2.34.1

Re: [PATCH] Format test results closer to what DejaGnu does

[Florian Weimer]

* Maxim Kuvyrkov via Libc-alpha:

> +# Print test summary for tests in $1 .sum file;
> +# $2 is optional test identifier.
> +# Fail if there are unexpected failures in the test results.
>  define summarize-tests
> +@echo "		=== glibc failures ==="
> +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
> +@echo "		=== Summary of results$2 ==="
> +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
> +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>  endef

Doesn't this print “=== glibc failures ===” unconditionally even if
there are no failures?  Is this really what DejaGnu does?

Thanks,
Florian

Re: [PATCH] Format test results closer to what DejaGnu does

[Maxim Kuvyrkov]

> On Jun 5, 2023, at 17:35, Florian Weimer <fweimer@redhat.com> wrote:
> 
> * Maxim Kuvyrkov via Libc-alpha:
> 
>> +# Print test summary for tests in $1 .sum file;
>> +# $2 is optional test identifier.
>> +# Fail if there are unexpected failures in the test results.
>> define summarize-tests
>> +@echo " === glibc failures ==="
>> +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
>> +@echo " === Summary of results$2 ==="
>> +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
>> +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>> endef
> 
> Doesn't this print “=== glibc failures ===” unconditionally even if
> there are no failures?  Is this really what DejaGnu does?

Hi Florian,

Good point.  Note that the above output doesn't reach .sum files, the output is only printed out to stdout.

I could either drop "=== glibc failures ===" line entirely, or print out "no unexpected failures", e.g.:
@echo " === glibc failures ==="
@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || echo "no unexpected failures"

WDYT?

Thanks,

--
Maxim Kuvyrkov
https://www.linaro.org

Re: [PATCH] Format test results closer to what DejaGnu does

[Florian Weimer]

* Maxim Kuvyrkov:

>> On Jun 5, 2023, at 17:35, Florian Weimer <fweimer@redhat.com> wrote:
>> 
>> * Maxim Kuvyrkov via Libc-alpha:
>> 
>>> +# Print test summary for tests in $1 .sum file;
>>> +# $2 is optional test identifier.
>>> +# Fail if there are unexpected failures in the test results.
>>> define summarize-tests
>>> +@echo " === glibc failures ==="
>>> +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
>>> +@echo " === Summary of results$2 ==="
>>> +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
>>> +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>>> endef
>> 
>> Doesn't this print “=== glibc failures ===” unconditionally even if
>> there are no failures?  Is this really what DejaGnu does?
>
> Hi Florian,
>
> Good point.  Note that the above output doesn't reach .sum files, the output is only printed out to stdout.
>
> I could either drop "=== glibc failures ===" line entirely, or print out "no unexpected failures", e.g.:
> @echo " === glibc failures ==="
> @grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || echo "no unexpected failures"
>
> WDYT?

I looked at the gcc-testresults mailing list, and there appear no
=== … failures === lines at all?  What was the motivation for adding it
in the first place?

Thanks,
Florian

Re: [PATCH] Format test results closer to what DejaGnu does

[Maxim Kuvyrkov]

> On Jun 19, 2023, at 17:59, Florian Weimer <fweimer@redhat.com> wrote:
> 
> * Maxim Kuvyrkov:
> 
>>> On Jun 5, 2023, at 17:35, Florian Weimer <fweimer@redhat.com> wrote:
>>> 
>>> * Maxim Kuvyrkov via Libc-alpha:
>>> 
>>>> +# Print test summary for tests in $1 .sum file;
>>>> +# $2 is optional test identifier.
>>>> +# Fail if there are unexpected failures in the test results.
>>>> define summarize-tests
>>>> +@echo " === glibc failures ==="
>>>> +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
>>>> +@echo " === Summary of results$2 ==="
>>>> +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
>>>> +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>>>> endef
>>> 
>>> Doesn't this print “=== glibc failures ===” unconditionally even if
>>> there are no failures?  Is this really what DejaGnu does?
>> 
>> Hi Florian,
>> 
>> Good point.  Note that the above output doesn't reach .sum files, the output is only printed out to stdout.
>> 
>> I could either drop "=== glibc failures ===" line entirely, or print out "no unexpected failures", e.g.:
>> @echo " === glibc failures ==="
>> @grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || echo "no unexpected failures"
>> 
>> WDYT?
> 
> I looked at the gcc-testresults mailing list, and there appear no
> === … failures === lines at all?  What was the motivation for adding it
> in the first place?

The only motivation is that it looks like a nice header for the following FAILs.  What's your preference for the line -- drop it entirely or print out:

=== glibc failures ===
no unexpected failures

?
--
Maxim Kuvyrkov
https://www.linaro.org

[PATCH v2] Format test results closer to what DejaGnu does

[Maxim Kuvyrkov]

The only change from v1 is removal of "=== glibc failures ==="
line.

The years of dealing with Binutils, GCC and GDB test results
made the community create good tools for comparison and analysis
of DejaGnu test results.  This change allows to use those tools
for Glibc's test results as well.

The motivation for this change is Linaro's pre-commit testers,
which use a modified version of GCC's validate_failures.py
to create test xfail lists with baseline failures and known
flaky tests.  See below links for an example xfails file (only
one link is supposed to work at any given time):
- https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/
- https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/

Specifacally, this patch changes format of glibc's .sum files from ...
<cut>
FAIL: elf/test1
PASS: string/test2
</cut>
... to ...
<cut>
             === glibc tests ===

Running elf ...
FAIL: elf/test1

Running string ...
PASS: string/test2
</cut>.

And output of "make check" from ...
<cut>
FAIL: elf/test1
</cut>
... to ...
<cut>
FAIL: elf/test1
		=== Summary of results ===
      1 FAIL
      1 PASS
</cut>.

Signed-off-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
---
 Makefile                      | 11 +++++++----
 scripts/merge-test-results.sh |  3 +++
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index b938721166..a4f3378e21 100644
--- a/Makefile
+++ b/Makefile
@@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh
 	$(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \
 	$(evaluate-test)
 
+# Print test summary for tests in $1 .sum file;
+# $2 is optional test identifier.
+# Fail if there are unexpected failures in the test results.
 define summarize-tests
-@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
-@echo "Summary of test results$2:"
-@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c
-@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1
+@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
+@echo "		=== Summary of results$2 ==="
+@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
+@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
 endef
 
 # The intention here is to do ONE install of our build into the
diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
index e4dcc2520a..8f9d81f6eb 100755
--- a/scripts/merge-test-results.sh
+++ b/scripts/merge-test-results.sh
@@ -50,7 +50,10 @@ case $type in
   -t)
     subdir_file_name=$1
     shift
+    echo "		=== glibc tests ==="
     for d in "$@"; do
+      echo
+      echo "Running $d ..."
       if [ -f "$objpfx$d/$subdir_file_name" ]; then
 	cat "$objpfx$d/$subdir_file_name"
       else
-- 
2.34.1

Re: [PATCH v2] Format test results closer to what DejaGnu does

[Adhemerval Zanella Netto]

On 01/11/23 07:28, Maxim Kuvyrkov wrote:
> The only change from v1 is removal of "=== glibc failures ==="
> line.
> 
> The years of dealing with Binutils, GCC and GDB test results
> made the community create good tools for comparison and analysis
> of DejaGnu test results.  This change allows to use those tools
> for Glibc's test results as well.
> 
> The motivation for this change is Linaro's pre-commit testers,
> which use a modified version of GCC's validate_failures.py
> to create test xfail lists with baseline failures and known
> flaky tests.  See below links for an example xfails file (only
> one link is supposed to work at any given time):
> - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/
> - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/
> 
> Specifacally, this patch changes format of glibc's .sum files from ...
> <cut>
> FAIL: elf/test1
> PASS: string/test2
> </cut>
> ... to ...
> <cut>
>              === glibc tests ===
> 
> Running elf ...
> FAIL: elf/test1
> 
> Running string ...
> PASS: string/test2
> </cut>.

I like the idea of splitting tests by subfolder.

> 
> And output of "make check" from ...
> <cut>
> FAIL: elf/test1
> </cut>
> ... to ...
> <cut>
> FAIL: elf/test1
> 		=== Summary of results ===
>       1 FAIL
>       1 PASS
> </cut>.
> 
> Signed-off-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>

LGTM, thanks.

Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>

> ---
>  Makefile                      | 11 +++++++----
>  scripts/merge-test-results.sh |  3 +++
>  2 files changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index b938721166..a4f3378e21 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh
>  	$(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \
>  	$(evaluate-test)
>  
> +# Print test summary for tests in $1 .sum file;
> +# $2 is optional test identifier.
> +# Fail if there are unexpected failures in the test results.
>  define summarize-tests
> -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
> -@echo "Summary of test results$2:"
> -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c
> -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1
> +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
> +@echo "		=== Summary of results$2 ==="
> +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
> +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
>  endef
>  
>  # The intention here is to do ONE install of our build into the
> diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
> index e4dcc2520a..8f9d81f6eb 100755
> --- a/scripts/merge-test-results.sh
> +++ b/scripts/merge-test-results.sh
> @@ -50,7 +50,10 @@ case $type in
>    -t)
>      subdir_file_name=$1
>      shift
> +    echo "		=== glibc tests ==="
>      for d in "$@"; do
> +      echo
> +      echo "Running $d ..."
>        if [ -f "$objpfx$d/$subdir_file_name" ]; then
>  	cat "$objpfx$d/$subdir_file_name"
>        else