[PATCH] When importing results from a full GDB test run, bunsen will throw a TypeError:

[PATCH] When importing results from a full GDB test run, bunsen will throw a TypeError:

[Keith Seitz]

$ ./bunsen-add.py <!-- arguments -->
bunsen-upload received a payload:
* gdb.log.xz (1177912 bytes, file)
* gdb.sum.xz (287264 bytes, file)
* README.txt (96 bytes, file)
will decompress /tmp/tmp72zejz8o/gdb.log.xz
will decompress /tmp/tmp72zejz8o/gdb.sum.xz
Traceback (most recent call last):
  File "/home/keiths/work/bunsen/virgin/bunsen/./bunsen-add.py", line 80, in <module>
    commit_id = _commit_logs.commit_logs(b, wd, tar, tarfile=tar,
  File "/home/keiths/work/bunsen/virgin/bunsen/scripts-main/gdb/commit_logs.py", line 224, in commit_logs
    testrun = annotate_dejagnu_log(testrun, gdb_log, all_cases, verbose=False)
  File "/home/keiths/work/bunsen/virgin/bunsen/scripts-main/gdb/parse_dejagnu.py", line 221, in annotate_dejagnu_log
    outcome, expname, subtest = get_expname_subtest(outcome_lines[j])
TypeError: cannot unpack non-iterable NoneType object

This happens in annotate_dejagnu_log, where the code attempts to map
outcome lines:

    for j in range(len(outcome_lines)):
        outcome, expname, subtest = get_expname_subtest(outcome_lines[j])
        if expname not in testcase_line_start:
            testcase_line_start[expname] = j

The problem is that get_expname_subtest can return None whenever it
encounters a line that doesn't fit the "OUTCOME: TESTFILE: TESTNAME"
pattern. In log files, this happens all the time, e.g., "Running TESTFILE..."

This patch simply checks the return result of get_expname_subtest,
skipping any None result.
---
 scripts-main/gdb/parse_dejagnu.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts-main/gdb/parse_dejagnu.py b/scripts-main/gdb/parse_dejagnu.py
index b56ed3c..1c4596a 100755
--- a/scripts-main/gdb/parse_dejagnu.py
+++ b/scripts-main/gdb/parse_dejagnu.py
@@ -218,7 +218,10 @@ def annotate_dejagnu_log(testrun, logfile, outcome_lines=[],
     # (1b) Build a map of outcome_lines:
     testcase_line_start = {} # .exp name -> index of first outcome_line with this name
     for j in range(len(outcome_lines)):
-        outcome, expname, subtest = get_expname_subtest(outcome_lines[j])
+        result = get_expname_subtest(outcome_lines[j])
+        if result is None:
+            continue
+        outcome, expname, subtest = result
         if expname not in testcase_line_start:
             testcase_line_start[expname] = j
 
-- 
2.31.1

Re: [PATCH] When importing results from a full GDB test run, bunsen will throw a TypeError:

[Serhei Makarov]

On Thu, Aug 5, 2021, at 3:03 PM, Keith Seitz via Bunsen wrote:
> $ ./bunsen-add.py <!-- arguments -->
> bunsen-upload received a payload:
> * gdb.log.xz (1177912 bytes, file)
> * gdb.sum.xz (287264 bytes, file)
> * README.txt (96 bytes, file)
> will decompress /tmp/tmp72zejz8o/gdb.log.xz
> will decompress /tmp/tmp72zejz8o/gdb.sum.xz
> Traceback (most recent call last):
>   File "/home/keiths/work/bunsen/virgin/bunsen/./bunsen-add.py", line 
> 80, in <module>
>     commit_id = _commit_logs.commit_logs(b, wd, tar, tarfile=tar,
>   File 
> "/home/keiths/work/bunsen/virgin/bunsen/scripts-main/gdb/commit_logs.py", line 224, in commit_logs
>     testrun = annotate_dejagnu_log(testrun, gdb_log, all_cases, 
> verbose=False)
>   File 
> "/home/keiths/work/bunsen/virgin/bunsen/scripts-main/gdb/parse_dejagnu.py", line 221, in annotate_dejagnu_log
>     outcome, expname, subtest = get_expname_subtest(outcome_lines[j])
> TypeError: cannot unpack non-iterable NoneType object
> 
> This happens in annotate_dejagnu_log, where the code attempts to map
> outcome lines:
> 
>     for j in range(len(outcome_lines)):
>         outcome, expname, subtest = get_expname_subtest(outcome_lines[j])
>         if expname not in testcase_line_start:
>             testcase_line_start[expname] = j
> 
> The problem is that get_expname_subtest can return None whenever it
> encounters a line that doesn't fit the "OUTCOME: TESTFILE: TESTNAME"
> pattern. In log files, this happens all the time, e.g., "Running TESTFILE..."
Hmm, I could have sworn I dealt with this before...
If outcome_lines contains a malformed line, that's actually a bug.
The previous parser function (parse_dejagnu_log) contains a variable
called all_cases, which should accumulate lines of the form 
"OUTCOME: TESTFILE: TESTNAME" only.

In the latest code it's populated like this
https://sourceware.org/git/?p=bunsen.git;a=blob;f=scripts-main/gdb/parse_dejagnu.py;h=b56ed3c3e5cfbfb33e6fa38fa710e5a7096071b0;hb=HEAD#l143

all_cases is then passed as outcome_lines to annotate_dejagnu_log here
https://sourceware.org/git/?p=bunsen.git;a=blob;f=scripts-main/gdb/commit_logs.py;h=a8edb14b7c28975e9b9a8ffa3141ccdbe547e18b;hb=HEAD#l223

Could I have a copy of the input data that was triggering the issue?
It seems like the best way to solve this is with a bit of debugging on my part.

All the best,
      Serhei

Re: [PATCH] When importing results from a full GDB test run, bunsen will throw a TypeError:

[Keith Seitz]

On 8/5/21 12:39 PM, Serhei Makarov wrote:

> Hmm, I could have sworn I dealt with this before...
> If outcome_lines contains a malformed line, that's actually a bug.
> The previous parser function (parse_dejagnu_log) contains a variable
> called all_cases, which should accumulate lines of the form 
> "OUTCOME: TESTFILE: TESTNAME" only.

OMG, I completely overlooked that. Consider this patch (as it is)
withdrawn. I'll dig into it again.

Thank you!

Keith