[RFC PATCH] New script `summarize'

[RFC PATCH] New script `summarize'

[Keith Seitz]

This patch proposes to add a new `summarize' script which attempts
to reproduce the summary which appears at the end of DejaGNU .sum files.

While this is largely only useful for test case verification, perhaps
others might find it useful, too.

The script may take an optional argument to limit the output.
This is a comma-separated list of glob expressions, modeled on GDB's
TESTS="..." argument to "make check".

Example (this is with consolidate_pass=False):

$ ./bunsen.py +summarize 969d2e7
Using branch index, checkout name wd-summarize
Running summarize at PATH/bunsen/.bunsen/wd-summarize from PATH/bunsen/scripts-master/summarize.py with ['969d2e7']
===
Summary for commit 969d2e7 of gdb version 10.0.50.20200826-git
from branch master on x86_64 using <unknown board>

\# of expected passes            67853
\# of unexpected failures        181
\# of expected failures          78
\# of known failures             111
\# of untested testcases         8
\# of unsupported tests          106

To facilitate these printing-type operations of Testruns, I've added
a helper method to Testrun to gather common properties into an "info"
dictionary that can be used by callers.

This dictionary contains a new "board" property for recording the
DejaGNU test board. This property is currently not used and will always
be "<unknown board>". Patch coming.

Comments?

Keith
---
 bunsen.py                   | 28 +++++++++++++++
 scripts-master/summarize.py | 68 +++++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100755 scripts-master/summarize.py

diff --git a/bunsen.py b/bunsen.py
index b089d80..99c4cc0 100755
--- a/bunsen.py
+++ b/bunsen.py
@@ -622,6 +622,34 @@ class Testrun(dict):
             # XXX Set summary=False if JSON was missing testcases.
             self.summary = self.summary and 'testcases' in json_data
 
+    # Return properties of this Testrun as printable strings, or
+    # "<unknown PROPERTY>" if unknown.  Returns a dictionary containing
+    # keys for architecture, board, branch, version.
+
+    def get_info_strings(self):
+        info = dict()
+        if 'arch' in self:
+            info['architecture'] = self.arch
+        else:
+            info['architecture'] = '<unknown arch>'
+
+        if 'board' in self:
+            info['board'] = self.board
+        else:
+            info['board'] = '<unknown board>'
+
+        if 'source_branch' in self:
+            info['branch'] = self.source_branch
+        else:
+            info['branch'] = '<unknown branch>'
+
+        if 'version' in self:
+            info['version'] = self.version
+        else:
+            info['version'] = '<unknown version>'
+
+        return info
+
     def add_testcase(self, name, outcome, **kwargs):
         '''
         Append a testcase result to the Testrun data.
diff --git a/scripts-master/summarize.py b/scripts-master/summarize.py
new file mode 100755
index 0000000..e46db18
--- /dev/null
+++ b/scripts-master/summarize.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+
+# Display a DejaGNU-like test summary, given the bunsen commit
+# of the desired test run. Optionally also takes a comma-separated
+# list of glob expressions to limit results.
+
+info = "summarize.py <bunsen_commit> [tests]"
+cmdline_args = [
+    ('commit', None, '<bunsen_commit>',
+     "commit to fetch results for"),
+    ('tests', None, '<test_globs>',
+     "comma-separated list of glob expressions of tests to summarize")
+]
+
+import sys
+import bunsen
+from collections import Counter
+from pathlib import PurePath
+
+# A list of test outcomes in output order.
+outcome_labels = {
+    'PASS' : 'expected passes',
+    'FAIL' : 'unexpected failures',
+    'XPASS' : 'unexpected successes',
+    'XFAIL' : 'expected failures',
+    'KPASS' : 'unknown successes',
+    'KFAIL' : 'known failures',
+    'UNTESTED' : 'untested testcases',
+    'UNRESOLVED' : 'unresolved testcases',
+    'UNSUPPORTED' : 'unsupported tests',
+    'ERROR' : 'errors',
+    'WARNING' : 'warnings'
+}
+
+if __name__ == '__main__':
+    b = bunsen.Bunsen()
+    opts = b.cmdline_args(sys.argv, info=info, args=cmdline_args,
+                          required_args=['commit'], optional_args=['tests'])
+
+    testrun = b.testrun(opts.commit)
+    all_tests = testrun.testcases
+    found_tests = []
+    if opts.tests is not None:
+        for glob in opts.tests.split(','):
+            found_tests.extend([t for t in all_tests if PurePath(t['name']).match(glob)])
+    else:
+        found_tests = all_tests
+
+    if found_tests:
+        info = testrun.get_info_strings()
+
+        project = b.tags[0] if len(b.tags) == 1 else '<multiple projects>'
+        print(f'Summary for commit {opts.commit} of {project} version {info["version"]}')
+        print(f'from branch {info["branch"]} on {info["architecture"]} using {info["board"]}')
+        if opts.tests is not None:
+            print(f'limiting results to tests matching: {opts.tests}')
+        print()
+
+        # Collate results for outcomes
+        c = Counter(t['outcome'] for t in found_tests)
+
+        # We could simply loop over the keys of the Counter, but that would not necessarily give
+        # us the same output order as DejaGNU itself.
+        for l in outcome_labels:
+            if c[l] != 0:
+                print('# of %-26s %d' % (outcome_labels[l], c[l]))
+    else:
+        print(f'found no tests matching \"{opts.tests}\"')
-- 
2.26.2

Re: [RFC PATCH] New script `summarize'

[Serhei Makarov]

> diff --git a/bunsen.py b/bunsen.py
> index b089d80..99c4cc0 100755
> --- a/bunsen.py
> +++ b/bunsen.py
> @@ -622,6 +622,34 @@ class Testrun(dict):
>              # XXX Set summary=False if JSON was missing testcases.
>              self.summary = self.summary and 'testcases' in json_data
>  
> +    # Return properties of this Testrun as printable strings, or
Suggest "Return configuration properties"
for consistency with other code/comments that use the term 'configuration'
for these properties.

> +    # "<unknown PROPERTY>" if unknown.  Returns a dictionary containing
> +    # keys for architecture, board, branch, version.
> +
> +    def get_info_strings(self):
> +        info = dict()
> +        if 'arch' in self:
> +            info['architecture'] = self.arch
> +        else:
> +            info['architecture'] = '<unknown arch>'
> +
> +        if 'board' in self:
> +            info['board'] = self.board
> +        else:
> +            info['board'] = '<unknown board>'
> +
> +        if 'source_branch' in self:
> +            info['branch'] = self.source_branch
> +        else:
> +            info['branch'] = '<unknown branch>'
> +
> +        if 'version' in self:
> +            info['version'] = self.version
> +        else:
> +            info['version'] = '<unknown version>'
> +
> +        return info
> +
Looks OK to me, I may simplify or generalize this code with a later patch.
(e.g. by taking a dictionary of field name -> display name as an optional argument,
or having that dictionary predefined somewhere and defaulting to return
all the keys/values that are *not* specific to the Bunsen storage (such as bunsen_commit_id)
 -- needs some thinking on my part, so please commit as-is for now).

It's worth noting that the set of configuration keys will vary by project,
e.g. for SystemTap 'osver' (the Linux distribution) is a more important field.

> diff --git a/scripts-master/summarize.py b/scripts-master/summarize.py
> new file mode 100755
> index 0000000..e46db18
> --- /dev/null
> +++ b/scripts-master/summarize.py
> @@ -0,0 +1,68 @@
> +#!/usr/bin/env python3
> +
> +# Display a DejaGNU-like test summary, given the bunsen commit
> +# of the desired test run. Optionally also takes a comma-separated
> +# list of glob expressions to limit results.
> +
> +info = "summarize.py <bunsen_commit> [tests]"
> +cmdline_args = [
> +    ('commit', None, '<bunsen_commit>',
> +     "commit to fetch results for"),
> +    ('tests', None, '<test_globs>',
> +     "comma-separated list of glob expressions of tests to summarize")
> +]
> +
> +import sys
> +import bunsen
> +from collections import Counter
> +from pathlib import PurePath
> +
> +# A list of test outcomes in output order.
> +outcome_labels = {
> +    'PASS' : 'expected passes',
> +    'FAIL' : 'unexpected failures',
> +    'XPASS' : 'unexpected successes',
> +    'XFAIL' : 'expected failures',
> +    'KPASS' : 'unknown successes',
> +    'KFAIL' : 'known failures',
> +    'UNTESTED' : 'untested testcases',
> +    'UNRESOLVED' : 'unresolved testcases',
> +    'UNSUPPORTED' : 'unsupported tests',
> +    'ERROR' : 'errors',
> +    'WARNING' : 'warnings'
> +}
> +
> +if __name__ == '__main__':
> +    b = bunsen.Bunsen()
> +    opts = b.cmdline_args(sys.argv, info=info, args=cmdline_args,
> +                          required_args=['commit'], 
> optional_args=['tests'])
> +
> +    testrun = b.testrun(opts.commit)
> +    all_tests = testrun.testcases
> +    found_tests = []
> +    if opts.tests is not None:
> +        for glob in opts.tests.split(','):
> +            found_tests.extend([t for t in all_tests if 
> PurePath(t['name']).match(glob)])
> +    else:
> +        found_tests = all_tests
> +
> +    if found_tests:
> +        info = testrun.get_info_strings()
> +
> +        project = b.tags[0] if len(b.tags) == 1 else '<multiple 
> projects>'
> +        print(f'Summary for commit {opts.commit} of {project} version 
> {info["version"]}')
> +        print(f'from branch {info["branch"]} on {info["architecture"]} 
> using {info["board"]}')
> +        if opts.tests is not None:
> +            print(f'limiting results to tests matching: {opts.tests}')
> +        print()
> +
> +        # Collate results for outcomes
> +        c = Counter(t['outcome'] for t in found_tests)
> +
> +        # We could simply loop over the keys of the Counter, but that 
> would not necessarily give
> +        # us the same output order as DejaGNU itself.
> +        for l in outcome_labels:
> +            if c[l] != 0:
> +                print('# of %-26s %d' % (outcome_labels[l], c[l]))
> +    else:
> +        print(f'found no tests matching \"{opts.tests}\"')
LGTM.

Re: [RFC PATCH] New script `summarize'

[Keith Seitz]

On 9/25/20 8:07 AM, Serhei Makarov wrote:
>> diff --git a/bunsen.py b/bunsen.py
>> index b089d80..99c4cc0 100755
>> --- a/bunsen.py
>> +++ b/bunsen.py
>> @@ -622,6 +622,34 @@ class Testrun(dict):
>>              # XXX Set summary=False if JSON was missing testcases.
>>              self.summary = self.summary and 'testcases' in json_data
>>  
>> +    # Return properties of this Testrun as printable strings, or
> Suggest "Return configuration properties"
> for consistency with other code/comments that use the term 'configuration'
> for these properties.

Changed.

>> +    # "<unknown PROPERTY>" if unknown.  Returns a dictionary containing
>> +    # keys for architecture, board, branch, version.
>> +
>> +    def get_info_strings(self):
>> +        info = dict()
>> +        if 'arch' in self:
>> +            info['architecture'] = self.arch
>> +        else:
>> +            info['architecture'] = '<unknown arch>'
>> +
>> +        if 'board' in self:
>> +            info['board'] = self.board
>> +        else:
>> +            info['board'] = '<unknown board>'
>> +
>> +        if 'source_branch' in self:
>> +            info['branch'] = self.source_branch
>> +        else:
>> +            info['branch'] = '<unknown branch>'
>> +
>> +        if 'version' in self:
>> +            info['version'] = self.version
>> +        else:
>> +            info['version'] = '<unknown version>'
>> +
>> +        return info
>> +
> Looks OK to me, I may simplify or generalize this code with a later
> patch. (e.g. by taking a dictionary of field name -> display name as
> an optional argument, or having that dictionary predefined somewhere
> and defaulting to return all the keys/values that are *not* specific
> to the Bunsen storage (such as bunsen_commit_id) -- needs some
> thinking on my part, so please commit as-is for now).> 
> It's worth noting that the set of configuration keys will vary by project,
> e.g. for SystemTap 'osver' (the Linux distribution) is a more important field.
> 

Yes, please do whatever is necessary/efficient/clean. These are currently
the properties that I "need" in order to rule the wo^W^W^W satisfy my
use case.

I'm still getting my proverbial legs with python, so there are almost certainly
better, more pythony ways to do this, I am sure.

> LGTM.

Pushed with your suggested changes. Thank you!

Keith