From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 72E3938582A6 for ; Tue, 26 Jul 2022 16:38:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 72E3938582A6 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-317-OAdqfdSSNXiEW6-oMOboXw-1; Tue, 26 Jul 2022 12:38:54 -0400 X-MC-Unique: OAdqfdSSNXiEW6-oMOboXw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B6C1A8037AC for ; Tue, 26 Jul 2022 16:38:53 +0000 (UTC) Received: from guittard.uglyboxes.com (unknown [10.2.16.251]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5F3D91415118 for ; Tue, 26 Jul 2022 16:38:53 +0000 (UTC) From: Keith Seitz To: bunsen@sourceware.org Subject: [PATCH] Add subtest log diff script Date: Tue, 26 Jul 2022 09:38:52 -0700 Message-Id: <20220726163852.3627231-1-keiths@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: bunsen@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bunsen mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2022 16:38:59 -0000 This patch introduces a new script, r-dejagnu-diff-logs, which will output various styles of diffs for a given subtest between two commitishes. I use this when doing regression testing analysis to quickly see how results between test runs has changed. It is much faster and more convenient than opening logfiles in a pager, for example. Supported output templates are: 1) text $ r-dejagnu-diff-logs unpatched patched 'gdb.base/included.exp: list integer' commitish: unpatched: gdb.base/included.exp: list integer (FAIL) < info source < Current source file is ../../../src/gdb/testsuite/gdb.base/included.c < Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite < Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c < Contains 24 lines. < Source language is c. < Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36). < Compiled with DWARF 5 debugging format. < Includes preprocessor macro info. < (gdb) list integer < 18 #include "included.h" < (gdb) FAIL: gdb.base/included.exp: list integer commitish: patched: gdb.base/included.exp: list integer (PASS) > info source > Current source file is ../../../src/gdb/testsuite/gdb.base/included.c > Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite > Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c > Contains 24 lines. > Source language is c. > Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36). > Compiled with DWARF 5 debugging format. > Includes preprocessor macro info. > (gdb) list integer > 18 int integer; > (gdb) PASS: gdb.base/included.exp: list integer 2) json $ r-dejagnu-diff-logs --template json unpatched patched \ 'gdb.base/included.exp: list integer' | jq [ { "commits": [ { "commitish": "unpatched", "lines": [ "info source", "Current source file is ../../../src/gdb/testsuite/gdb.base/included.c", "Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite", "Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c", "Contains 24 lines.", "Source language is c.", "Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).", "Compiled with DWARF 5 debugging format.", "Includes preprocessor macro info.", "(gdb) list integer", "18\t#include \"included.h\"", "(gdb) FAIL: gdb.base/included.exp: list integer" ], "outcome": "FAIL" }, { "commitish": "patched", "lines": [ "info source", "Current source file is ../../../src/gdb/testsuite/gdb.base/included.c", "Compilation directory is /home/keiths/work/gdb/branches/amd-v2-may-16/linux/gdb/testsuite", "Located in /home/keiths/work/gdb/branches/amd-v2-may-16/src/gdb/testsuite/gdb.base/included.c", "Contains 24 lines.", "Source language is c.", "Producer is clang version 14.0.0 (Fedora 14.0.0-1.fc36).", "Compiled with DWARF 5 debugging format.", "Includes preprocessor macro info.", "(gdb) list integer", "18\tint integer;", "(gdb) PASS: gdb.base/included.exp: list integer" ], "outcome": "PASS" } ], "name": "gdb.base/included.exp: list integer" } ] 3) diff $ r-dejagnu-diff-logs --template diff unpatched patched \ 'gdb.base/included.exp: list integer' *** unpatched: gdb.base/included.exp: list integer --- patched: gdb.base/included.exp: list integer *************** *** 8,12 **** Compiled with DWARF 5 debugging format. Includes preprocessor macro info. (gdb) list integer ! 18 #include "included.h" ! (gdb) FAIL: gdb.base/included.exp: list integer --- 8,12 ---- Compiled with DWARF 5 debugging format. Includes preprocessor macro info. (gdb) list integer ! 18 int integer; ! (gdb) PASS: gdb.base/included.exp: list integer --- bin/r-dejagnu-diff-logs | 251 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100755 bin/r-dejagnu-diff-logs diff --git a/bin/r-dejagnu-diff-logs b/bin/r-dejagnu-diff-logs new file mode 100755 index 0000000..748c8d8 --- /dev/null +++ b/bin/r-dejagnu-diff-logs @@ -0,0 +1,251 @@ +#! /usr/bin/python3 + +# This script reads in results from two Bunsen commits and displays +# the log files for a given subtest result for each testrun. + +import argparse +import git +import os +import sqlite3 as lite +import jinja2 +import re +import io +import difflib +import sys + +# Query for the database for .exp filename, subtest name, result, logfile, +# and cursor position for the given testrun ID and expfile. +query = ''' +SELECT + exs.name AS expfile, + sts.name AS subtest, + rs.name AS result, + logfile, + logcursor +FROM + dejagnu_testcase tc, + dejagnu_testsuite ts +JOIN + dejagnu_string exs ON (expfile = exs.id) +JOIN + dejagnu_string sts ON (subtest = sts.id) +JOIN + dejagnu_string rs ON (result = rs.id) +WHERE + ts.tr IN (?) + AND tc.testsuite = ts.id + AND exs.name IN (?) +''' + +# Jinja output templates. +jinja_templates = { + 'diff': '''{% for subtest in results %}{% for line in subtest|diff_subtest_lines %}{{ line }}{% endfor %}{% endfor %}''', + 'text': '''{% set n = namespace(l = false) %}{% for subtest in results %}{% set n.l = false %}{% for c in subtest.commits %}commitish: {{c.commitish}}: {{subtest.name}} ({{c.outcome}}){% set n.l = not n.l %} +{% for line in c.lines %}{% if n.l %}<{% else %}>{% endif %} {{line}}\n{% endfor %}{% endfor %}{% endfor %}''', + 'json' : '''{{ results|tojson|safe }}''' +} + +# Filter to output diffs. +def diff_subtest_lines(subtest): + lines1 = [t + '\n' if t[-1] != '\n' else t for t in subtest['commits'][0]['lines']] + from_commit = subtest['commits'][0]['commitish'] + if len(subtest['commits']) > 1: + lines2 = [t + '\n' if t[-1] != '\n' else t for t in subtest['commits'][1]['lines']] + to_commit = subtest['commits'][1]['commitish'] + else: + lines2 = [] + to_commit = '' + + sio = io.StringIO() + if lines1 == lines2: + c1 = subtest['commits'][0]['commitish'] + c2 = subtest['commits'][1]['commitish'] + sio.write('Commits ') + sio.write(c1) + sio.write(' and ') + sio.write(c2) + sio.write(' are identical\n') + else: + ff = from_commit + ': ' + subtest['name'] + tf = to_commit + ': ' + subtest['name'] + sio.writelines(difflib.context_diff(lines1, lines2, fromfile=ff, tofile=tf)) + return sio.getvalue() + +# Create result structure for jinja. +def create_subtest(upc, ups, upl, pc, ps, pl): + subtest = dict() + subtest['name'] = ups['subtest'] + subtest['commits'] = [] + if ups is not None and upl is not None: + subtest['commits'].append({ + 'commitish': upc, + 'outcome': ups['outcome'], + 'lines': upl + }) + + if ps is not None and pl is not None: + subtest['commits'].append({ + 'commitish': pc, + 'outcome': ps['outcome'], + 'lines': pl + }) + return subtest + +# Covert the given unpatched/patched commitishs into commits in TESTRUNS. +# Returns a tuple of (unpatched_commit, patched_commit). +def get_commits(testruns, args): + unpatched_commit = None + patched_commit = None + try: + unpatched_commit = testruns.commit(args.patched) + patched_commit = testruns.commit(args.unpatched) + except Exception as e: + pass + return (unpatched_commit, patched_commit) + +# Get the testrun IDs for the given unpatched and patched commits. +# Returns a tuple of (unpatched_trid, patched_trid). +def get_trids(con, up_commit, p_commit): + unpatched = con.execute('SELECT id FROM testrun WHERE gitcommit = ?', + (up_commit.hexsha,)).fetchone()[0] + patched = con.execute('SELECT id FROM testrun WHERE gitcommit = ?', + (p_commit.hexsha,)).fetchone()[0] + return (unpatched, patched) + +# Get all test data for the given unpatched and patched commits in the given +# database. +# Returns a tuple of (unpatched_results, patched_results). Each result is +# a list of tuples of (expfile, subtest, outcome). +def get_results(db, up_commit, p_commit, expfile): + con = lite.connect(db, uri=True) + up_trid, p_trid = get_trids(con, up_commit, p_commit) + unpatched = con.execute(query, (up_trid, expfile,)).fetchall() + patched = con.execute(query, (p_trid, expfile,)).fetchall() + return (unpatched, patched) + + +# Get the lower bound of lines to display for the result line with cursor position +# LINE. LINETABLE is a sorted list of all cursor positions in a expfile. +def get_lower_bound(linetable, line): + # Get the index for our LINE. + idx = linetable.index(line) + + # The lower bound will be the previous result's cursor, or simply the + # previous ten lines if this is the first result in the file. + return linetable[idx - 1] if idx > 0 else line - 10 + +# Returns a list of lines from the log file of COMMIT around where the given +# SUBTEST result is. RESULTS is a list of all expfile results and is +# used to determine the bounds of lines to return. +def get_log_lines(commit, results, subtests): + all_lines = [] + linetable = sorted([t[4] for t in results]) + for t in subtests: + logfile = t['logfile'] + cursor = t['cursor'] + ds = commit.tree[logfile].data_stream + bytes = ds.read() + try: + contents = bytes.decode('utf-8') + except: + contents = bytes.decode('iso-8859-1') + + sio = io.StringIO(contents) + lines = [] + bound = get_lower_bound(linetable, cursor) + for idx in range(cursor): + line = sio.readline() + if idx > cursor: + break + if idx >= bound: + lines.append(line.rstrip()) + all_lines.append(lines) + return all_lines + +def main(): + # Create argument parser and add our options. + parser = argparse.ArgumentParser( + description='Summarize results of DejaGNU tests', + formatter_class=argparse.ArgumentDefaultsHelpFormatter) + parser.add_argument('--git', type=str, help='git testrun archive', + default='.') + parser.add_argument('--db', type=str, help='sqlite database file', + default='bunsen.sqlite3') + parser.add_argument('--template', type=str, help='jinja template', + default='text') + parser.add_argument('unpatched', metavar='UNPATCHED', type=str, + help='testrun commit or tag of unpatched/before run') + parser.add_argument('patched', metavar='PATCHED', type=str, + help='testrun commit or tag of patched/after run') + parser.add_argument('subtest', metavar='SUBTEST', type=str, + help='subtest results to display') + + # Parse and verify the command line arguments. + global args + args = parser.parse_args() + + # Verify output template. + if args.template not in jinja_templates: + parser.error(f'Unknown output template `{args.template}\'. Must be one of: {", ".join(jinja_templates.keys())})') + + # Open GIT repo and get the COMMITISH of desired commits. + try: + testruns = git.Repo(args.git) + except Exception as e: + parser.error(f'Invalid GIT repository `{args.git}\'. Need --git?') + + unpatched_commit, patched_commit = get_commits(testruns, args) + if unpatched_commit is None: + parser.error(f'Unknown unpatched commitish `{args.unpatched}\' in git repo `{args.git}\'') + if patched_commit is None: + parser.error(f'Unknown patched commitish `{args.patched}\' in git repo `{args.git}\'') + + # Connect to the database and read in the data from the Bunsen database. + if not os.path.isfile(args.db): + parser.error(f'Database `{args.db}\' not found. Need --db?') + + # Test file for which we want results. + expfile = args.subtest[:args.subtest.find(':')] + + # Grab results and print them out. RESULTS is a list of tuples: (expfile, subtest, outcome) + global unpatched_results, patched_results, unpatched_subtests, patched_subtests + unpatched_results, patched_results = get_results(args.db, + unpatched_commit, + patched_commit, + expfile) + + unpatched_subtests_list = [t for t in unpatched_results if t[1] == args.subtest] + if len(unpatched_results) == 0: + print(f'commitish `{args.unpatched}\' contains no test file named `{expfile}\'', file=sys.stderr) + keys = ['expfile', 'subtest', 'outcome', 'logfile', 'cursor'] + unpatched_subtests = [dict(zip(iter(keys), iter(t))) for t in unpatched_subtests_list] + + patched_subtests_list = [t for t in patched_results if t[1] == args.subtest] + if len(patched_results) == 0: + print(f'commitish `{args.patched}\' contains no test file named `{expfile}\'', file=sys.stderr) + patched_subtests = [dict(zip(iter(keys), iter(t))) for t in patched_subtests_list] + + unpatched_lines = get_log_lines(unpatched_commit, unpatched_results, unpatched_subtests) + patched_lines = get_log_lines(patched_commit, patched_results, patched_subtests) + + results = [] + for s in range(max(len(unpatched_subtests), len(patched_subtests))): + up = unpatched_subtests[s] if s < len(unpatched_subtests) else None + upl = unpatched_lines[s] if s < len(unpatched_lines) else None + p = patched_subtests[s] if s < len(patched_subtests) else None + pl = patched_lines[s] if s < len(patched_lines) else None + results.append(create_subtest(args.unpatched, up, upl, + args.patched, p, pl)) + + jinja_params = { 'results' : results } + env = jinja2.Environment(loader=jinja2.DictLoader(jinja_templates)) + env.filters['diff_subtest_lines'] = diff_subtest_lines + template = env.get_template(args.template) + try: + print(template.render(jinja_params), end='') + except BrokenPipeError: + pass + + +if __name__ == '__main__': + main() -- 2.36.1