From: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
To: gcc-patches@gcc.gnu.org
Cc: Diego Novillo <dnovillo@google.com>, Doug Evans <dje@google.com>
Subject: [contrib] Extend and improve validate_failures.py
Date: Fri, 2 Jun 2023 15:20:40 +0000 [thread overview]
Message-ID: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> (raw)
This patch series extends and improves validate_failures.py script
to provide a powerful tool to handle DejaGnu test results in automated
CI environment.
Linaro TCWG uses validate_failures.py to ...
- compare test results without human oversight,
- detect unexpected FAILs vs baseline,
- detect unexpected PASSes vs baseline,
- automatically detect flaky tests,
- create lists of expected failures and flaky tests, see [1].
[1] https://ci.linaro.org/job/tcwg_gcc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/
next reply other threads:[~2023-06-02 15:21 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-02 15:20 Maxim Kuvyrkov [this message]
2023-06-02 15:20 ` [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing Maxim Kuvyrkov
2023-06-03 15:17 ` Jeff Law
2023-06-05 14:06 ` Maxim Kuvyrkov
2023-09-26 15:46 ` Bernhard Reutner-Fischer
2023-09-27 14:47 ` Maxim Kuvyrkov
2023-10-03 10:37 ` rep.dot.nop
2023-11-02 13:31 ` Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 02/12] [contrib] validate_failures.py: Support expiry attributes in manifests Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 03/12] [contrib] validate_failures.py: Read in manifest when comparing build dirs Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 04/12] [contrib] validate_failures.py: Simplify GetManifestPath() Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 05/12] [contrib] validate_failures.py: Add more verbosity levels Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 06/12] [contrib] validate_failures.py: Be more stringent in parsing result lines Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 07/12] [contrib] validate_failures.py: Use exit code "2" to indicate regression Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 08/12] [contrib] validate_failures.py: Support "$tool:" prefix in exp names Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 09/12] [contrib] validate_failures.py: Improve error output Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 10/12] [contrib] validate_failures.py: Add new option --invert_match Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 11/12] [contrib] validate_failures.py: Add "--expiry_date YYYYMMDD" option Maxim Kuvyrkov
2023-06-02 15:20 ` [PATCH 12/12] [contrib] validate_failures.py: Ignore stray filesystem paths in results Maxim Kuvyrkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230602152052.1874860-1-maxim.kuvyrkov@linaro.org \
--to=maxim.kuvyrkov@linaro.org \
--cc=dje@google.com \
--cc=dnovillo@google.com \
--cc=gcc-patches@gcc.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).