public inbox for bunsen@sourceware.org
 help / color / mirror / Atom feed
From: Keith Seitz <keiths@redhat.com>
To: bunsen@sourceware.org
Subject: [RFC PATCH] New script `summarize'
Date: Thu, 24 Sep 2020 11:54:24 -0700	[thread overview]
Message-ID: <20200924185424.1496842-1-keiths@redhat.com> (raw)

This patch proposes to add a new `summarize' script which attempts
to reproduce the summary which appears at the end of DejaGNU .sum files.

While this is largely only useful for test case verification, perhaps
others might find it useful, too.

The script may take an optional argument to limit the output.
This is a comma-separated list of glob expressions, modeled on GDB's
TESTS="..." argument to "make check".

Example (this is with consolidate_pass=False):

$ ./bunsen.py +summarize 969d2e7
Using branch index, checkout name wd-summarize
Running summarize at PATH/bunsen/.bunsen/wd-summarize from PATH/bunsen/scripts-master/summarize.py with ['969d2e7']
===
Summary for commit 969d2e7 of gdb version 10.0.50.20200826-git
from branch master on x86_64 using <unknown board>

\# of expected passes            67853
\# of unexpected failures        181
\# of expected failures          78
\# of known failures             111
\# of untested testcases         8
\# of unsupported tests          106

To facilitate these printing-type operations of Testruns, I've added
a helper method to Testrun to gather common properties into an "info"
dictionary that can be used by callers.

This dictionary contains a new "board" property for recording the
DejaGNU test board. This property is currently not used and will always
be "<unknown board>". Patch coming.

Comments?

Keith
---
 bunsen.py                   | 28 +++++++++++++++
 scripts-master/summarize.py | 68 +++++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100755 scripts-master/summarize.py

diff --git a/bunsen.py b/bunsen.py
index b089d80..99c4cc0 100755
--- a/bunsen.py
+++ b/bunsen.py
@@ -622,6 +622,34 @@ class Testrun(dict):
             # XXX Set summary=False if JSON was missing testcases.
             self.summary = self.summary and 'testcases' in json_data
 
+    # Return properties of this Testrun as printable strings, or
+    # "<unknown PROPERTY>" if unknown.  Returns a dictionary containing
+    # keys for architecture, board, branch, version.
+
+    def get_info_strings(self):
+        info = dict()
+        if 'arch' in self:
+            info['architecture'] = self.arch
+        else:
+            info['architecture'] = '<unknown arch>'
+
+        if 'board' in self:
+            info['board'] = self.board
+        else:
+            info['board'] = '<unknown board>'
+
+        if 'source_branch' in self:
+            info['branch'] = self.source_branch
+        else:
+            info['branch'] = '<unknown branch>'
+
+        if 'version' in self:
+            info['version'] = self.version
+        else:
+            info['version'] = '<unknown version>'
+
+        return info
+
     def add_testcase(self, name, outcome, **kwargs):
         '''
         Append a testcase result to the Testrun data.
diff --git a/scripts-master/summarize.py b/scripts-master/summarize.py
new file mode 100755
index 0000000..e46db18
--- /dev/null
+++ b/scripts-master/summarize.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+
+# Display a DejaGNU-like test summary, given the bunsen commit
+# of the desired test run. Optionally also takes a comma-separated
+# list of glob expressions to limit results.
+
+info = "summarize.py <bunsen_commit> [tests]"
+cmdline_args = [
+    ('commit', None, '<bunsen_commit>',
+     "commit to fetch results for"),
+    ('tests', None, '<test_globs>',
+     "comma-separated list of glob expressions of tests to summarize")
+]
+
+import sys
+import bunsen
+from collections import Counter
+from pathlib import PurePath
+
+# A list of test outcomes in output order.
+outcome_labels = {
+    'PASS' : 'expected passes',
+    'FAIL' : 'unexpected failures',
+    'XPASS' : 'unexpected successes',
+    'XFAIL' : 'expected failures',
+    'KPASS' : 'unknown successes',
+    'KFAIL' : 'known failures',
+    'UNTESTED' : 'untested testcases',
+    'UNRESOLVED' : 'unresolved testcases',
+    'UNSUPPORTED' : 'unsupported tests',
+    'ERROR' : 'errors',
+    'WARNING' : 'warnings'
+}
+
+if __name__ == '__main__':
+    b = bunsen.Bunsen()
+    opts = b.cmdline_args(sys.argv, info=info, args=cmdline_args,
+                          required_args=['commit'], optional_args=['tests'])
+
+    testrun = b.testrun(opts.commit)
+    all_tests = testrun.testcases
+    found_tests = []
+    if opts.tests is not None:
+        for glob in opts.tests.split(','):
+            found_tests.extend([t for t in all_tests if PurePath(t['name']).match(glob)])
+    else:
+        found_tests = all_tests
+
+    if found_tests:
+        info = testrun.get_info_strings()
+
+        project = b.tags[0] if len(b.tags) == 1 else '<multiple projects>'
+        print(f'Summary for commit {opts.commit} of {project} version {info["version"]}')
+        print(f'from branch {info["branch"]} on {info["architecture"]} using {info["board"]}')
+        if opts.tests is not None:
+            print(f'limiting results to tests matching: {opts.tests}')
+        print()
+
+        # Collate results for outcomes
+        c = Counter(t['outcome'] for t in found_tests)
+
+        # We could simply loop over the keys of the Counter, but that would not necessarily give
+        # us the same output order as DejaGNU itself.
+        for l in outcome_labels:
+            if c[l] != 0:
+                print('# of %-26s %d' % (outcome_labels[l], c[l]))
+    else:
+        print(f'found no tests matching \"{opts.tests}\"')
-- 
2.26.2


             reply	other threads:[~2020-09-24 18:54 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-24 18:54 Keith Seitz [this message]
2020-09-25 15:07 ` Serhei Makarov
2020-09-25 17:02   ` Keith Seitz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200924185424.1496842-1-keiths@redhat.com \
    --to=keiths@redhat.com \
    --cc=bunsen@sourceware.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).