From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x1129.google.com (mail-yw1-x1129.google.com [IPv6:2607:f8b0:4864:20::1129]) by sourceware.org (Postfix) with ESMTPS id 570383858D1E for ; Wed, 1 Nov 2023 13:59:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 570383858D1E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 570383858D1E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1129 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698847151; cv=none; b=mE6OdnCqc3MkfQhFBD5xOaXvTSi9MXcFGGkXpMLleHvseOj3bW/Z1QLZor7ygJMWjClyiEXbmO30mvKsI95R0G6r+QOSjItXhTGIQ/vjacyTjQnyjciDjTAxV0DHfvk+G4gZpOncZ1SbrTzredJNx5M57lWvuw8sddy0uIolxVc= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698847151; c=relaxed/simple; bh=AHM5krEc4qwLNn+1/JPtqBxyZ88BTXaOw2nL5weWqv0=; h=DKIM-Signature:Message-ID:Date:MIME-Version:Subject:To:From; b=POZfzaafobCBtONE8vVpvZv4zQknkCdctIP5rZeIyLJ9dSWVbd5CGY9uYz7mrj993BF/pos/qx+HxoMGCN+AiiuqKtlj5rp2bKkGqdrWfwr7r6bE4V+tblJUBwcmYWLLosjTpQHzhBx8FKJMb8qNl7D54oMv5mGUTBQaBgjMVJs= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-5a8ada42c2aso69634897b3.3 for ; Wed, 01 Nov 2023 06:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1698847140; x=1699451940; darn=sourceware.org; h=content-transfer-encoding:in-reply-to:organization:from:references :to:content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=vlraQYIlkJZ3fumWaeMNNINCrmU12c7ZUjlVPym4/YM=; b=h6turV/fokhyePhY+OP2uUORV1D0evbYL+UNhM+c73WcjCNMWO8L7Fipnt5oTomOtZ 3Bv/Ou72luxj4FMzJsYexXrzw8LP+xl96W8K8TMwrt9iHKUHqabn2aONzJ7nZ93UjR3/ X5CR6OJDkOYOjjeUTDf583Ph3PsV1GPQnkDY1Adl/1hjnLpm+hVN+5Xd6QQ0sSAHEb3e Bk+f+bUPQoalQEvb1li8SGM88Vbp8/wkuD3f68B0APgz1yjNXNmE1ygEyMnUFTrK2Vho D3nON+kIuIO/T6ntDIidsZLMEk3dUNP7oImJrqj5su14KGYWJe+lAWoQsH7ZgX5pBCyq PDhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698847140; x=1699451940; h=content-transfer-encoding:in-reply-to:organization:from:references :to:content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vlraQYIlkJZ3fumWaeMNNINCrmU12c7ZUjlVPym4/YM=; b=FOldLTFtBkJ8SyXq/Xv5EjNKa6cpJy9Y901RkHLc1q91nHImZDIDEsUX6K/vt/AJMo pMefhszNSFrrIHz9F7zCWzNfDwVT/Hz8NlyAP28Qif2v+e0Bapk96WZTKRrEbJX//GTD TbcDWiw9r2vARvCeVDrHlFrceZYSX6MJxmY0kHsDOyou6MsbE2ZjT9lVd5ZrRnPda16G yZPtQ9Wmg7PyFCRK6bJogg+IeCsKd2xP6aA/FejUEOQ3t88L0lp2j+vuD4X4fjLscpTq 1tCHbYgDe04wUl4pwm0RO1BjgKrU2EEc3HSjdVCkm0/CqGLsmVocXdoDvJdORrdtzgFx yI8w== X-Gm-Message-State: AOJu0YzFobkzcIzBkCanEe40VwAV9PAeDtpubVR3uMk7xvZnyAnRTlc8 b2W0oljxIi4k+YVI16qiUVkzEAy0GKHG4/rc0uR+rw== X-Google-Smtp-Source: AGHT+IGsNn3B8jo9ySk1KwQlPooa9+hyWFKoTGbAw1bG4uqhta9YYZmAFTbyqJe6KE6dikG+5I2u6g== X-Received: by 2002:a81:99c3:0:b0:5a8:1aa0:6253 with SMTP id q186-20020a8199c3000000b005a81aa06253mr14023623ywg.7.1698847140024; Wed, 01 Nov 2023 06:59:00 -0700 (PDT) Received: from ?IPV6:2804:1b3:a7c0:a715:846f:dda1:e5d1:3146? ([2804:1b3:a7c0:a715:846f:dda1:e5d1:3146]) by smtp.gmail.com with ESMTPSA id a9-20020a0dd809000000b005b3f6c1b5edsm910334ywe.80.2023.11.01.06.58.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 01 Nov 2023 06:58:59 -0700 (PDT) Message-ID: <0f3951da-785a-4d97-9990-bc16b2d5eaf0@linaro.org> Date: Wed, 1 Nov 2023 10:58:56 -0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] Format test results closer to what DejaGnu does Content-Language: en-US To: libc-alpha@sourceware.org, Maxim Kuvyrkov References: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> <20231101102828.3207231-1-maxim.kuvyrkov@linaro.org> From: Adhemerval Zanella Netto Organization: Linaro In-Reply-To: <20231101102828.3207231-1-maxim.kuvyrkov@linaro.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 01/11/23 07:28, Maxim Kuvyrkov wrote: > The only change from v1 is removal of "=== glibc failures ===" > line. > > The years of dealing with Binutils, GCC and GDB test results > made the community create good tools for comparison and analysis > of DejaGnu test results. This change allows to use those tools > for Glibc's test results as well. > > The motivation for this change is Linaro's pre-commit testers, > which use a modified version of GCC's validate_failures.py > to create test xfail lists with baseline failures and known > flaky tests. See below links for an example xfails file (only > one link is supposed to work at any given time): > - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/ > - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/ > > Specifacally, this patch changes format of glibc's .sum files from ... > > FAIL: elf/test1 > PASS: string/test2 > > ... to ... > > === glibc tests === > > Running elf ... > FAIL: elf/test1 > > Running string ... > PASS: string/test2 > . I like the idea of splitting tests by subfolder. > > And output of "make check" from ... > > FAIL: elf/test1 > > ... to ... > > FAIL: elf/test1 > === Summary of results === > 1 FAIL > 1 PASS > . > > Signed-off-by: Maxim Kuvyrkov LGTM, thanks. Reviewed-by: Adhemerval Zanella > --- > Makefile | 11 +++++++---- > scripts/merge-test-results.sh | 3 +++ > 2 files changed, 10 insertions(+), 4 deletions(-) > > diff --git a/Makefile b/Makefile > index b938721166..a4f3378e21 100644 > --- a/Makefile > +++ b/Makefile > @@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh > $(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \ > $(evaluate-test) > > +# Print test summary for tests in $1 .sum file; > +# $2 is optional test identifier. > +# Fail if there are unexpected failures in the test results. > define summarize-tests > -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true > -@echo "Summary of test results$2:" > -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c > -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1 > +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true > +@echo " === Summary of results$2 ===" > +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c > +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' > endef > > # The intention here is to do ONE install of our build into the > diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh > index e4dcc2520a..8f9d81f6eb 100755 > --- a/scripts/merge-test-results.sh > +++ b/scripts/merge-test-results.sh > @@ -50,7 +50,10 @@ case $type in > -t) > subdir_file_name=$1 > shift > + echo " === glibc tests ===" > for d in "$@"; do > + echo > + echo "Running $d ..." > if [ -f "$objpfx$d/$subdir_file_name" ]; then > cat "$objpfx$d/$subdir_file_name" > else