From: Keith Seitz <keiths@redhat.com>
To: bunsen@sourceware.org
Subject: [PATCH] Add subtest log diff script
Date: Tue, 26 Jul 2022 09:38:52 -0700 [thread overview]
Message-ID: <20220726163852.3627231-1-keiths@redhat.com> (raw)
This patch introduces a new script, r-dejagnu-diff-logs, which will output
various styles of diffs for a given subtest between two commitishes.
I use this when doing regression testing analysis to quickly see how results
between test runs has changed. It is much faster and more convenient than
opening logfiles in a pager, for example.
Supported output templates are:
1) text
$ r-dejagnu-diff-logs unpatched patched 'gdb.base/included.exp: list integer'
commitish: unpatched: gdb.base/included.exp: list integer (FAIL)
< info source
< Current source file is ../../../src/gdb/testsuite/gdb.base/included.c
< Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite
< Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c
< Contains 24 lines.
< Source language is c.
< Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).
< Compiled with DWARF 5 debugging format.
< Includes preprocessor macro info.
< (gdb) list integer
< 18 #include "included.h"
< (gdb) FAIL: gdb.base/included.exp: list integer
commitish: patched: gdb.base/included.exp: list integer (PASS)
> info source
> Current source file is ../../../src/gdb/testsuite/gdb.base/included.c
> Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite
> Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c
> Contains 24 lines.
> Source language is c.
> Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).
> Compiled with DWARF 5 debugging format.
> Includes preprocessor macro info.
> (gdb) list integer
> 18 int integer;
> (gdb) PASS: gdb.base/included.exp: list integer
2) json
$ r-dejagnu-diff-logs --template json unpatched patched \
'gdb.base/included.exp: list integer' | jq
[
{
"commits": [
{
"commitish": "unpatched",
"lines": [
"info source",
"Current source file is ../../../src/gdb/testsuite/gdb.base/included.c",
"Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite",
"Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c",
"Contains 24 lines.",
"Source language is c.",
"Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).",
"Compiled with DWARF 5 debugging format.",
"Includes preprocessor macro info.",
"(gdb) list integer",
"18\t#include \"included.h\"",
"(gdb) FAIL: gdb.base/included.exp: list integer"
],
"outcome": "FAIL"
},
{
"commitish": "patched",
"lines": [
"info source",
"Current source file is ../../../src/gdb/testsuite/gdb.base/included.c",
"Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite",
"Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c",
"Contains 24 lines.",
"Source language is c.",
"Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).",
"Compiled with DWARF 5 debugging format.",
"Includes preprocessor macro info.",
"(gdb) list integer",
"18\tint integer;",
"(gdb) PASS: gdb.base/included.exp: list integer"
],
"outcome": "PASS"
}
],
"name": "gdb.base/included.exp: list integer"
}
]
3) diff
$ r-dejagnu-diff-logs --template diff unpatched patched \
'gdb.base/included.exp: list integer'
*** unpatched: gdb.base/included.exp: list integer
--- patched: gdb.base/included.exp: list integer
***************
*** 8,12 ****
Compiled with DWARF 5 debugging format.
Includes preprocessor macro info.
(gdb) list integer
! 18 #include "included.h"
! (gdb) FAIL: gdb.base/included.exp: list integer
--- 8,12 ----
Compiled with DWARF 5 debugging format.
Includes preprocessor macro info.
(gdb) list integer
! 18 int integer;
! (gdb) PASS: gdb.base/included.exp: list integer
---
bin/r-dejagnu-diff-logs | 251 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 251 insertions(+)
create mode 100755 bin/r-dejagnu-diff-logs
diff --git a/bin/r-dejagnu-diff-logs b/bin/r-dejagnu-diff-logs
new file mode 100755
index 0000000..748c8d8
--- /dev/null
+++ b/bin/r-dejagnu-diff-logs
@@ -0,0 +1,251 @@
+#! /usr/bin/python3
+
+# This script reads in results from two Bunsen commits and displays
+# the log files for a given subtest result for each testrun.
+
+import argparse
+import git
+import os
+import sqlite3 as lite
+import jinja2
+import re
+import io
+import difflib
+import sys
+
+# Query for the database for .exp filename, subtest name, result, logfile,
+# and cursor position for the given testrun ID and expfile.
+query = '''
+SELECT
+ exs.name AS expfile,
+ sts.name AS subtest,
+ rs.name AS result,
+ logfile,
+ logcursor
+FROM
+ dejagnu_testcase tc,
+ dejagnu_testsuite ts
+JOIN
+ dejagnu_string exs ON (expfile = exs.id)
+JOIN
+ dejagnu_string sts ON (subtest = sts.id)
+JOIN
+ dejagnu_string rs ON (result = rs.id)
+WHERE
+ ts.tr IN (?)
+ AND tc.testsuite = ts.id
+ AND exs.name IN (?)
+'''
+
+# Jinja output templates.
+jinja_templates = {
+ 'diff': '''{% for subtest in results %}{% for line in subtest|diff_subtest_lines %}{{ line }}{% endfor %}{% endfor %}''',
+ 'text': '''{% set n = namespace(l = false) %}{% for subtest in results %}{% set n.l = false %}{% for c in subtest.commits %}commitish: {{c.commitish}}: {{subtest.name}} ({{c.outcome}}){% set n.l = not n.l %}
+{% for line in c.lines %}{% if n.l %}<{% else %}>{% endif %} {{line}}\n{% endfor %}{% endfor %}{% endfor %}''',
+ 'json' : '''{{ results|tojson|safe }}'''
+}
+
+# Filter to output diffs.
+def diff_subtest_lines(subtest):
+ lines1 = [t + '\n' if t[-1] != '\n' else t for t in subtest['commits'][0]['lines']]
+ from_commit = subtest['commits'][0]['commitish']
+ if len(subtest['commits']) > 1:
+ lines2 = [t + '\n' if t[-1] != '\n' else t for t in subtest['commits'][1]['lines']]
+ to_commit = subtest['commits'][1]['commitish']
+ else:
+ lines2 = []
+ to_commit = '<none>'
+
+ sio = io.StringIO()
+ if lines1 == lines2:
+ c1 = subtest['commits'][0]['commitish']
+ c2 = subtest['commits'][1]['commitish']
+ sio.write('Commits ')
+ sio.write(c1)
+ sio.write(' and ')
+ sio.write(c2)
+ sio.write(' are identical\n')
+ else:
+ ff = from_commit + ': ' + subtest['name']
+ tf = to_commit + ': ' + subtest['name']
+ sio.writelines(difflib.context_diff(lines1, lines2, fromfile=ff, tofile=tf))
+ return sio.getvalue()
+
+# Create result structure for jinja.
+def create_subtest(upc, ups, upl, pc, ps, pl):
+ subtest = dict()
+ subtest['name'] = ups['subtest']
+ subtest['commits'] = []
+ if ups is not None and upl is not None:
+ subtest['commits'].append({
+ 'commitish': upc,
+ 'outcome': ups['outcome'],
+ 'lines': upl
+ })
+
+ if ps is not None and pl is not None:
+ subtest['commits'].append({
+ 'commitish': pc,
+ 'outcome': ps['outcome'],
+ 'lines': pl
+ })
+ return subtest
+
+# Covert the given unpatched/patched commitishs into commits in TESTRUNS.
+# Returns a tuple of (unpatched_commit, patched_commit).
+def get_commits(testruns, args):
+ unpatched_commit = None
+ patched_commit = None
+ try:
+ unpatched_commit = testruns.commit(args.patched)
+ patched_commit = testruns.commit(args.unpatched)
+ except Exception as e:
+ pass
+ return (unpatched_commit, patched_commit)
+
+# Get the testrun IDs for the given unpatched and patched commits.
+# Returns a tuple of (unpatched_trid, patched_trid).
+def get_trids(con, up_commit, p_commit):
+ unpatched = con.execute('SELECT id FROM testrun WHERE gitcommit = ?',
+ (up_commit.hexsha,)).fetchone()[0]
+ patched = con.execute('SELECT id FROM testrun WHERE gitcommit = ?',
+ (p_commit.hexsha,)).fetchone()[0]
+ return (unpatched, patched)
+
+# Get all test data for the given unpatched and patched commits in the given
+# database.
+# Returns a tuple of (unpatched_results, patched_results). Each result is
+# a list of tuples of (expfile, subtest, outcome).
+def get_results(db, up_commit, p_commit, expfile):
+ con = lite.connect(db, uri=True)
+ up_trid, p_trid = get_trids(con, up_commit, p_commit)
+ unpatched = con.execute(query, (up_trid, expfile,)).fetchall()
+ patched = con.execute(query, (p_trid, expfile,)).fetchall()
+ return (unpatched, patched)
+
+
+# Get the lower bound of lines to display for the result line with cursor position
+# LINE. LINETABLE is a sorted list of all cursor positions in a expfile.
+def get_lower_bound(linetable, line):
+ # Get the index for our LINE.
+ idx = linetable.index(line)
+
+ # The lower bound will be the previous result's cursor, or simply the
+ # previous ten lines if this is the first result in the file.
+ return linetable[idx - 1] if idx > 0 else line - 10
+
+# Returns a list of lines from the log file of COMMIT around where the given
+# SUBTEST result is. RESULTS is a list of all expfile results and is
+# used to determine the bounds of lines to return.
+def get_log_lines(commit, results, subtests):
+ all_lines = []
+ linetable = sorted([t[4] for t in results])
+ for t in subtests:
+ logfile = t['logfile']
+ cursor = t['cursor']
+ ds = commit.tree[logfile].data_stream
+ bytes = ds.read()
+ try:
+ contents = bytes.decode('utf-8')
+ except:
+ contents = bytes.decode('iso-8859-1')
+
+ sio = io.StringIO(contents)
+ lines = []
+ bound = get_lower_bound(linetable, cursor)
+ for idx in range(cursor):
+ line = sio.readline()
+ if idx > cursor:
+ break
+ if idx >= bound:
+ lines.append(line.rstrip())
+ all_lines.append(lines)
+ return all_lines
+
+def main():
+ # Create argument parser and add our options.
+ parser = argparse.ArgumentParser(
+ description='Summarize results of DejaGNU tests',
+ formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+ parser.add_argument('--git', type=str, help='git testrun archive',
+ default='.')
+ parser.add_argument('--db', type=str, help='sqlite database file',
+ default='bunsen.sqlite3')
+ parser.add_argument('--template', type=str, help='jinja template',
+ default='text')
+ parser.add_argument('unpatched', metavar='UNPATCHED', type=str,
+ help='testrun commit or tag of unpatched/before run')
+ parser.add_argument('patched', metavar='PATCHED', type=str,
+ help='testrun commit or tag of patched/after run')
+ parser.add_argument('subtest', metavar='SUBTEST', type=str,
+ help='subtest results to display')
+
+ # Parse and verify the command line arguments.
+ global args
+ args = parser.parse_args()
+
+ # Verify output template.
+ if args.template not in jinja_templates:
+ parser.error(f'Unknown output template `{args.template}\'. Must be one of: {", ".join(jinja_templates.keys())})')
+
+ # Open GIT repo and get the COMMITISH of desired commits.
+ try:
+ testruns = git.Repo(args.git)
+ except Exception as e:
+ parser.error(f'Invalid GIT repository `{args.git}\'. Need --git?')
+
+ unpatched_commit, patched_commit = get_commits(testruns, args)
+ if unpatched_commit is None:
+ parser.error(f'Unknown unpatched commitish `{args.unpatched}\' in git repo `{args.git}\'')
+ if patched_commit is None:
+ parser.error(f'Unknown patched commitish `{args.patched}\' in git repo `{args.git}\'')
+
+ # Connect to the database and read in the data from the Bunsen database.
+ if not os.path.isfile(args.db):
+ parser.error(f'Database `{args.db}\' not found. Need --db?')
+
+ # Test file for which we want results.
+ expfile = args.subtest[:args.subtest.find(':')]
+
+ # Grab results and print them out. RESULTS is a list of tuples: (expfile, subtest, outcome)
+ global unpatched_results, patched_results, unpatched_subtests, patched_subtests
+ unpatched_results, patched_results = get_results(args.db,
+ unpatched_commit,
+ patched_commit,
+ expfile)
+
+ unpatched_subtests_list = [t for t in unpatched_results if t[1] == args.subtest]
+ if len(unpatched_results) == 0:
+ print(f'commitish `{args.unpatched}\' contains no test file named `{expfile}\'', file=sys.stderr)
+ keys = ['expfile', 'subtest', 'outcome', 'logfile', 'cursor']
+ unpatched_subtests = [dict(zip(iter(keys), iter(t))) for t in unpatched_subtests_list]
+
+ patched_subtests_list = [t for t in patched_results if t[1] == args.subtest]
+ if len(patched_results) == 0:
+ print(f'commitish `{args.patched}\' contains no test file named `{expfile}\'', file=sys.stderr)
+ patched_subtests = [dict(zip(iter(keys), iter(t))) for t in patched_subtests_list]
+
+ unpatched_lines = get_log_lines(unpatched_commit, unpatched_results, unpatched_subtests)
+ patched_lines = get_log_lines(patched_commit, patched_results, patched_subtests)
+
+ results = []
+ for s in range(max(len(unpatched_subtests), len(patched_subtests))):
+ up = unpatched_subtests[s] if s < len(unpatched_subtests) else None
+ upl = unpatched_lines[s] if s < len(unpatched_lines) else None
+ p = patched_subtests[s] if s < len(patched_subtests) else None
+ pl = patched_lines[s] if s < len(patched_lines) else None
+ results.append(create_subtest(args.unpatched, up, upl,
+ args.patched, p, pl))
+
+ jinja_params = { 'results' : results }
+ env = jinja2.Environment(loader=jinja2.DictLoader(jinja_templates))
+ env.filters['diff_subtest_lines'] = diff_subtest_lines
+ template = env.get_template(args.template)
+ try:
+ print(template.render(jinja_params), end='')
+ except BrokenPipeError:
+ pass
+
+
+if __name__ == '__main__':
+ main()
--
2.36.1
next reply other threads:[~2022-07-26 16:38 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-26 16:38 Keith Seitz [this message]
2022-07-29 16:59 ` Keith Seitz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220726163852.3627231-1-keiths@redhat.com \
--to=keiths@redhat.com \
--cc=bunsen@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).