public inbox for bunsen@sourceware.org
 help / color / mirror / Atom feed
* [PATCH] Add subtest log diff script
@ 2022-07-26 16:38 Keith Seitz
  2022-07-29 16:59 ` Keith Seitz
  0 siblings, 1 reply; 2+ messages in thread
From: Keith Seitz @ 2022-07-26 16:38 UTC (permalink / raw)
  To: bunsen

This patch introduces a new script, r-dejagnu-diff-logs, which will output
various styles of diffs for a given subtest between two commitishes.

I use this when doing regression testing analysis to quickly see how results
between test runs has changed. It is much faster and more convenient than
opening logfiles in a pager, for example.

Supported output templates are:

1) text
$ r-dejagnu-diff-logs unpatched patched 'gdb.base/included.exp: list integer'
commitish: unpatched: gdb.base/included.exp: list integer (FAIL)
< info source
< Current source file is ../../../src/gdb/testsuite/gdb.base/included.c
< Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite
< Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c
< Contains 24 lines.
< Source language is c.
< Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).
< Compiled with DWARF 5 debugging format.
< Includes preprocessor macro info.
< (gdb) list integer
< 18	#include "included.h"
< (gdb) FAIL: gdb.base/included.exp: list integer
commitish: patched: gdb.base/included.exp: list integer (PASS)
> info source
> Current source file is ../../../src/gdb/testsuite/gdb.base/included.c
> Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite
> Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c
> Contains 24 lines.
> Source language is c.
> Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).
> Compiled with DWARF 5 debugging format.
> Includes preprocessor macro info.
> (gdb) list integer
> 18	int integer;
> (gdb) PASS: gdb.base/included.exp: list integer

2) json
$ r-dejagnu-diff-logs --template json unpatched patched \
   'gdb.base/included.exp: list integer' | jq
[
  {
    "commits": [
      {
        "commitish": "unpatched",
        "lines": [
          "info source",
          "Current source file is ../../../src/gdb/testsuite/gdb.base/included.c",
          "Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite",
          "Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c",
          "Contains 24 lines.",
          "Source language is c.",
          "Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).",
          "Compiled with DWARF 5 debugging format.",
          "Includes preprocessor macro info.",
          "(gdb) list integer",
          "18\t#include \"included.h\"",
          "(gdb) FAIL: gdb.base/included.exp: list integer"
        ],
        "outcome": "FAIL"
      },
      {
        "commitish": "patched",
        "lines": [
          "info source",
          "Current source file is ../../../src/gdb/testsuite/gdb.base/included.c",
          "Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite",
          "Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c",
          "Contains 24 lines.",
          "Source language is c.",
          "Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).",
          "Compiled with DWARF 5 debugging format.",
          "Includes preprocessor macro info.",
          "(gdb) list integer",
          "18\tint integer;",
          "(gdb) PASS: gdb.base/included.exp: list integer"
        ],
        "outcome": "PASS"
      }
    ],
    "name": "gdb.base/included.exp: list integer"
  }
]

3) diff
$ r-dejagnu-diff-logs --template diff unpatched patched \
   'gdb.base/included.exp: list integer'
*** unpatched: gdb.base/included.exp: list integer
--- patched: gdb.base/included.exp: list integer
***************
*** 8,12 ****
  Compiled with DWARF 5 debugging format.
  Includes preprocessor macro info.
  (gdb) list integer
! 18	#include "included.h"
! (gdb) FAIL: gdb.base/included.exp: list integer
--- 8,12 ----
  Compiled with DWARF 5 debugging format.
  Includes preprocessor macro info.
  (gdb) list integer
! 18	int integer;
! (gdb) PASS: gdb.base/included.exp: list integer
---
 bin/r-dejagnu-diff-logs | 251 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 251 insertions(+)
 create mode 100755 bin/r-dejagnu-diff-logs

diff --git a/bin/r-dejagnu-diff-logs b/bin/r-dejagnu-diff-logs
new file mode 100755
index 0000000..748c8d8
--- /dev/null
+++ b/bin/r-dejagnu-diff-logs
@@ -0,0 +1,251 @@
+#! /usr/bin/python3
+
+# This script reads in results from two Bunsen commits and displays
+# the log files for a given subtest result for each testrun.
+
+import argparse
+import git
+import os
+import sqlite3 as lite
+import jinja2
+import re
+import io
+import difflib
+import sys
+
+# Query for the database for .exp filename, subtest name, result, logfile,
+# and cursor position for the given testrun ID and expfile.
+query = '''
+SELECT
+    exs.name AS expfile,
+    sts.name AS subtest,
+    rs.name AS result,
+    logfile,
+    logcursor
+FROM
+    dejagnu_testcase tc,
+    dejagnu_testsuite ts
+JOIN
+    dejagnu_string exs ON (expfile = exs.id)
+JOIN
+    dejagnu_string sts ON (subtest = sts.id)
+JOIN
+    dejagnu_string rs ON (result = rs.id)
+WHERE
+    ts.tr IN (?)
+    AND tc.testsuite = ts.id
+    AND exs.name IN (?)
+'''
+
+# Jinja output templates.
+jinja_templates = {
+    'diff': '''{% for subtest in results %}{% for line in subtest|diff_subtest_lines %}{{ line }}{% endfor %}{% endfor %}''',
+    'text': '''{% set n = namespace(l = false) %}{% for subtest in results %}{% set n.l = false %}{% for c in subtest.commits %}commitish: {{c.commitish}}: {{subtest.name}} ({{c.outcome}}){% set n.l = not n.l %}
+{% for line in c.lines %}{% if n.l %}<{% else %}>{% endif %} {{line}}\n{% endfor %}{% endfor %}{% endfor %}''',
+    'json' : '''{{ results|tojson|safe }}'''
+}
+
+# Filter to output diffs.
+def diff_subtest_lines(subtest):
+    lines1 = [t + '\n' if t[-1] != '\n' else t for t in subtest['commits'][0]['lines']]
+    from_commit = subtest['commits'][0]['commitish']
+    if len(subtest['commits']) > 1:
+        lines2 = [t + '\n' if t[-1] != '\n' else t for t in subtest['commits'][1]['lines']]
+        to_commit = subtest['commits'][1]['commitish']
+    else:
+        lines2 = []
+        to_commit = '<none>'
+
+    sio = io.StringIO()
+    if lines1 == lines2:
+        c1 = subtest['commits'][0]['commitish']
+        c2 = subtest['commits'][1]['commitish']
+        sio.write('Commits ')
+        sio.write(c1)
+        sio.write(' and ')
+        sio.write(c2)
+        sio.write(' are identical\n')
+    else:
+        ff = from_commit + ': ' + subtest['name']
+        tf = to_commit + ': ' + subtest['name']
+        sio.writelines(difflib.context_diff(lines1, lines2, fromfile=ff, tofile=tf))
+    return sio.getvalue()
+
+# Create result structure for jinja.
+def create_subtest(upc, ups, upl, pc, ps, pl):
+    subtest = dict()
+    subtest['name'] = ups['subtest']
+    subtest['commits'] = []
+    if ups is not None and upl is not None:
+        subtest['commits'].append({
+            'commitish': upc,
+            'outcome': ups['outcome'],
+            'lines': upl
+        })
+        
+    if ps is not None and pl is not None:
+        subtest['commits'].append({
+            'commitish': pc,
+            'outcome': ps['outcome'],
+            'lines': pl
+        })
+    return subtest
+    
+# Covert the given unpatched/patched commitishs into commits in TESTRUNS.
+# Returns a tuple of (unpatched_commit, patched_commit).
+def get_commits(testruns, args):
+    unpatched_commit = None
+    patched_commit = None
+    try:
+        unpatched_commit = testruns.commit(args.patched)
+        patched_commit = testruns.commit(args.unpatched)
+    except Exception as e:
+        pass
+    return (unpatched_commit, patched_commit)
+
+# Get the testrun IDs for the given unpatched and patched commits.
+# Returns a tuple of (unpatched_trid, patched_trid).
+def get_trids(con, up_commit, p_commit):
+    unpatched = con.execute('SELECT id FROM testrun WHERE gitcommit = ?',
+        (up_commit.hexsha,)).fetchone()[0]
+    patched = con.execute('SELECT id FROM testrun WHERE gitcommit = ?',
+        (p_commit.hexsha,)).fetchone()[0]
+    return (unpatched, patched)
+
+# Get all test data for the given unpatched and patched commits in the given
+# database.
+# Returns a tuple of (unpatched_results, patched_results).  Each result is
+# a list of tuples of (expfile, subtest, outcome).
+def get_results(db, up_commit, p_commit, expfile):
+    con = lite.connect(db, uri=True)
+    up_trid, p_trid = get_trids(con, up_commit, p_commit)
+    unpatched = con.execute(query, (up_trid, expfile,)).fetchall()
+    patched = con.execute(query, (p_trid, expfile,)).fetchall()
+    return (unpatched, patched)
+
+
+# Get the lower bound of lines to display for the result line with cursor position
+# LINE. LINETABLE is a sorted list of all cursor positions in a expfile.
+def get_lower_bound(linetable, line):
+    # Get the index for our LINE.
+    idx = linetable.index(line)
+    
+    # The lower bound will be the previous result's cursor, or simply the
+    # previous ten lines if this is the first result in the file.
+    return linetable[idx - 1] if idx > 0 else line - 10
+
+# Returns a list of lines from the log file of COMMIT around where the given
+# SUBTEST result is. RESULTS is a list of all expfile results and is
+# used to determine the bounds of lines to return.
+def get_log_lines(commit, results, subtests):
+    all_lines = []
+    linetable = sorted([t[4] for t in results])
+    for t in subtests:
+        logfile = t['logfile']
+        cursor  = t['cursor']
+        ds = commit.tree[logfile].data_stream
+        bytes = ds.read()
+        try:
+            contents = bytes.decode('utf-8')
+        except:
+            contents = bytes.decode('iso-8859-1')
+
+        sio = io.StringIO(contents)
+        lines = []
+        bound = get_lower_bound(linetable, cursor)
+        for idx in range(cursor):
+            line = sio.readline()
+            if idx > cursor:
+                break
+            if idx >= bound:
+                lines.append(line.rstrip())
+        all_lines.append(lines)
+    return all_lines
+
+def main():
+    # Create argument parser and add our options.
+    parser = argparse.ArgumentParser(
+        description='Summarize results of DejaGNU tests',
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument('--git', type=str, help='git testrun archive',
+                        default='.')
+    parser.add_argument('--db', type=str, help='sqlite database file',
+                        default='bunsen.sqlite3')
+    parser.add_argument('--template', type=str, help='jinja template',
+                        default='text')
+    parser.add_argument('unpatched', metavar='UNPATCHED', type=str,
+                        help='testrun commit or tag of unpatched/before run')
+    parser.add_argument('patched', metavar='PATCHED', type=str,
+                        help='testrun commit or tag of patched/after run')
+    parser.add_argument('subtest', metavar='SUBTEST', type=str,
+                        help='subtest results to display')
+
+    # Parse and verify the command line arguments.
+    global args
+    args = parser.parse_args()
+
+    # Verify output template.
+    if args.template not in jinja_templates:
+        parser.error(f'Unknown output template `{args.template}\'. Must be one of: {", ".join(jinja_templates.keys())})')
+
+    # Open GIT repo and get the COMMITISH of desired commits.
+    try:
+        testruns = git.Repo(args.git)
+    except Exception as e:
+        parser.error(f'Invalid GIT repository `{args.git}\'.  Need --git?')
+
+    unpatched_commit, patched_commit = get_commits(testruns, args)
+    if unpatched_commit is None:
+        parser.error(f'Unknown unpatched commitish `{args.unpatched}\' in git repo `{args.git}\'')
+    if patched_commit is None:
+        parser.error(f'Unknown patched commitish `{args.patched}\' in git repo `{args.git}\'')
+
+    # Connect to the database and read in the data from the Bunsen database.
+    if not os.path.isfile(args.db):
+        parser.error(f'Database `{args.db}\' not found.  Need --db?')
+
+    # Test file for which we want results.
+    expfile = args.subtest[:args.subtest.find(':')]
+
+    # Grab results and print them out. RESULTS is a list of tuples: (expfile, subtest, outcome)
+    global unpatched_results, patched_results, unpatched_subtests, patched_subtests
+    unpatched_results, patched_results = get_results(args.db,
+                                                     unpatched_commit,
+                                                     patched_commit,
+                                                     expfile)
+    
+    unpatched_subtests_list = [t for t in unpatched_results if t[1] == args.subtest]
+    if len(unpatched_results) == 0:
+        print(f'commitish `{args.unpatched}\' contains no test file named `{expfile}\'', file=sys.stderr)
+    keys = ['expfile', 'subtest', 'outcome', 'logfile', 'cursor']
+    unpatched_subtests = [dict(zip(iter(keys), iter(t))) for t in unpatched_subtests_list]
+
+    patched_subtests_list = [t for t in patched_results if t[1] == args.subtest]
+    if len(patched_results) == 0:
+        print(f'commitish `{args.patched}\' contains no test file named `{expfile}\'', file=sys.stderr)
+    patched_subtests = [dict(zip(iter(keys), iter(t))) for t in patched_subtests_list]
+
+    unpatched_lines = get_log_lines(unpatched_commit, unpatched_results, unpatched_subtests)
+    patched_lines = get_log_lines(patched_commit, patched_results, patched_subtests)
+
+    results = []
+    for s in range(max(len(unpatched_subtests), len(patched_subtests))):
+        up = unpatched_subtests[s] if s < len(unpatched_subtests) else None
+        upl = unpatched_lines[s] if s < len(unpatched_lines) else None
+        p = patched_subtests[s] if s < len(patched_subtests) else None
+        pl = patched_lines[s] if s < len(patched_lines) else None
+        results.append(create_subtest(args.unpatched, up, upl,
+                                      args.patched, p, pl))
+
+    jinja_params = { 'results' : results }
+    env = jinja2.Environment(loader=jinja2.DictLoader(jinja_templates))
+    env.filters['diff_subtest_lines'] = diff_subtest_lines
+    template = env.get_template(args.template)
+    try:
+        print(template.render(jinja_params), end='')
+    except BrokenPipeError:
+        pass
+
+
+if __name__ == '__main__':
+    main()
-- 
2.36.1


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH] Add subtest log diff script
  2022-07-26 16:38 [PATCH] Add subtest log diff script Keith Seitz
@ 2022-07-29 16:59 ` Keith Seitz
  0 siblings, 0 replies; 2+ messages in thread
From: Keith Seitz @ 2022-07-29 16:59 UTC (permalink / raw)
  To: bunsen

On 7/26/22 09:38, Keith Seitz via Bunsen wrote:
> This patch introduces a new script, r-dejagnu-diff-logs, which will output
> various styles of diffs for a given subtest between two commitishes.
> 

For the record, I've pushed this to Frank's branch after his private
A-OK.

Keith


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2022-07-29 16:59 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-26 16:38 [PATCH] Add subtest log diff script Keith Seitz
2022-07-29 16:59 ` Keith Seitz

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).