From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 1487) id A38683858D28; Fri, 3 Nov 2023 13:00:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A38683858D28 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1699016403; bh=eZFEnccnP20kQiIZvoouvVnoG3jdZhu4ucJUSMRDUT0=; h=From:To:Subject:Date:From; b=O5uaGbwMejN3KwXhaFFHpSRQhjOVgKvYclRT2zMO5uwVs4ANe+Pi4KEDPnUPR2kpS 0KQbq6Y2lVPG7kP83t1GK9sAHzrv4UBys5HW2oe/36moNmpvlnMA807cw5djUAR5IC IRddPxOIq1VxfNcWfTw+DNZzbe3Ershqr/biiRck= Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Maxim Kuvyrkov To: glibc-cvs@sourceware.org Subject: [glibc] Format test results closer to what DejaGnu does X-Act-Checkin: glibc X-Git-Author: Maxim Kuvyrkov X-Git-Refname: refs/heads/master X-Git-Oldrev: 9fd3409842b3e2d31cff5dbd6f96066c430f0aa2 X-Git-Newrev: 02976a4a4b2d01a524b33a508994664ffaf88d79 Message-Id: <20231103130003.A38683858D28@sourceware.org> Date: Fri, 3 Nov 2023 13:00:03 +0000 (GMT) List-Id: https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=02976a4a4b2d01a524b33a508994664ffaf88d79 commit 02976a4a4b2d01a524b33a508994664ffaf88d79 Author: Maxim Kuvyrkov Date: Fri May 19 08:28:21 2023 +0000 Format test results closer to what DejaGnu does The years of dealing with Binutils, GCC and GDB test results made the community create good tools for comparison and analysis of DejaGnu test results. This change allows to use those tools for Glibc's test results as well. The motivation for this change is Linaro's pre-commit testers, which use a modified version of GCC's validate_failures.py to create test xfail lists with baseline failures and known flaky tests. See below links for an example xfails file (only one link is supposed to work at any given time): - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/ - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/ Specifacally, this patch changes format of glibc's .sum files from ... FAIL: elf/test1 PASS: string/test2 ... to ... === glibc tests === Running elf ... FAIL: elf/test1 Running string ... PASS: string/test2 . And output of "make check" from ... FAIL: elf/test1 ... to ... FAIL: elf/test1 === Summary of results === 1 FAIL 1 PASS . Signed-off-by: Maxim Kuvyrkov Reviewed-by: Adhemerval Zanella Diff: --- Makefile | 11 +++++++---- scripts/merge-test-results.sh | 3 +++ 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index b938721166..a4f3378e21 100644 --- a/Makefile +++ b/Makefile @@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh $(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \ $(evaluate-test) +# Print test summary for tests in $1 .sum file; +# $2 is optional test identifier. +# Fail if there are unexpected failures in the test results. define summarize-tests -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true -@echo "Summary of test results$2:" -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1 +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true +@echo " === Summary of results$2 ===" +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' endef # The intention here is to do ONE install of our build into the diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh index e4dcc2520a..8f9d81f6eb 100755 --- a/scripts/merge-test-results.sh +++ b/scripts/merge-test-results.sh @@ -50,7 +50,10 @@ case $type in -t) subdir_file_name=$1 shift + echo " === glibc tests ===" for d in "$@"; do + echo + echo "Running $d ..." if [ -f "$objpfx$d/$subdir_file_name" ]; then cat "$objpfx$d/$subdir_file_name" else