Hi, While working on fixing up the gdb.sum parser, I've encountered `test_outcome_map', defined in common/parse_dejagnu.py. This is defined: test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS', 'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL', 'ERROR: tcl error sourcing':'ERROR', 'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED', 'UNRESOLVED':'UNRESOLVED'} Can you fill me in on these two unusual keys? IPASS: This is not a standard DejaGNU test result. Is this supposed to be KPASS (which is missing from the map)? ERROR: tcl error sourcing: That's a pretty unusual definition of ERROR. Was this added in order to specifically catch Tcl errors? What about "normal" errors? What's the preferred method for adding members to this map? In GDB, we have a special "DUPLICATE" outcome, and right now my plan is to simply add definitions for this (and any other additions) to our own gdb.parse_dejagnu_sum function, e.g., test_outcome_map['DUPLICATE'] = 'DUPLICATE' I'm not a huge fan of modifying global data like this, but since Bunsen doesn't run over multiple tools at the same time, this shouldn't be an issue. Keith
On Fri, Aug 6, 2021, at 3:35 PM, Keith Seitz via Bunsen wrote: > Hi, > > While working on fixing up the gdb.sum parser, I've encountered > `test_outcome_map', defined in common/parse_dejagnu.py. > > This is defined: > > test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS', > 'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL', > 'ERROR: tcl error sourcing':'ERROR', > 'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED', > 'UNRESOLVED':'UNRESOLVED'} > > Can you fill me in on these two unusual keys? Sorry for the delay, I had to dig up a bit of backstory on this. I thought I saw IPASS in some test results, but I wasn't able to dig up any indication of where, so I may be wrong. It is certainly true that KPASS should be included in addition to KFAIL. I took the liberty of patching accordingly: https://sourceware.org/git/?p=bunsen.git;a=commit;h=763c2037ea61eafdf61df09ef8516278dd1ad12f I also changed 'ERROR: tcl error sourcing':'ERROR' to 'ERROR':'ERROR' although I'm less sure of this part. There may be some SystemTap testcases where the program under observation prints 'ERROR: <stuff>' and that shouldn't be treated as an error outcome for the testcase. But that could be a SystemTap-specific nit and I would solve it by using the method you suggest below to edit test_outcome_map for the SystemTap parser only. > What's the preferred method for adding members to this map? > > In GDB, we have a special "DUPLICATE" outcome, and right now my plan > is to simply add definitions for this (and any other additions) to our > own gdb.parse_dejagnu_sum function, e.g., > > test_outcome_map['DUPLICATE'] = 'DUPLICATE' > > I'm not a huge fan of modifying global data like this, but since > Bunsen doesn't run over multiple tools at the same time, this shouldn't > be an issue. No objections to either method for extending the table. In my patch, I opted to just add 'DUPLICATE' to test_outcome_map since it's extremely unlikely to interfere with other testsuites. -- All the best, Serhei http://serhei.io
On 8/10/21 2:15 PM, Serhei Makarov wrote: > On Fri, Aug 6, 2021, at 3:35 PM, Keith Seitz via Bunsen wrote: >> Hi, >> >> While working on fixing up the gdb.sum parser, I've encountered >> `test_outcome_map', defined in common/parse_dejagnu.py. >> >> This is defined: >> >> test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS', >> 'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL', >> 'ERROR: tcl error sourcing':'ERROR', >> 'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED', >> 'UNRESOLVED':'UNRESOLVED'} >> >> Can you fill me in on these two unusual keys? > Sorry for the delay, I had to dig up a bit of backstory on this. > I thought I saw IPASS in some test results, but I wasn't able to dig up > any indication of where, so I may be wrong. It is certainly true that KPASS should be > included in addition to KFAIL. I took the liberty of patching accordingly: > https://sourceware.org/git/?p=bunsen.git;a=commit;h=763c2037ea61eafdf61df09ef8516278dd1ad12f Thank you! > I also changed 'ERROR: tcl error sourcing':'ERROR' to 'ERROR':'ERROR' > although I'm less sure of this part. There may be some SystemTap testcases > where the program under observation prints 'ERROR: <stuff>' > and that shouldn't be treated as an error outcome for the testcase. > But that could be a SystemTap-specific nit and I would solve it by using > the method you suggest below to edit test_outcome_map for the SystemTap > parser only. GDB has a similar problem (or maybe it is GDB of which you're thinking?) in gdb.guile tests. GDB itself outputs "ERROR:". :-( >> I'm not a huge fan of modifying global data like this, but since >> Bunsen doesn't run over multiple tools at the same time, this shouldn't >> be an issue. > No objections to either method for extending the table. > In my patch, I opted to just add 'DUPLICATE' to test_outcome_map > since it's extremely unlikely to interfere with other testsuites. Great, thank you! Keith