Unusual test_outcome_map keys

Unusual test_outcome_map keys

[Keith Seitz]

Hi,

While working on fixing up the gdb.sum parser, I've encountered
`test_outcome_map', defined in common/parse_dejagnu.py.

This is defined:

test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS',
                    'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL',
                    'ERROR: tcl error sourcing':'ERROR',
                    'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED',
                    'UNRESOLVED':'UNRESOLVED'}

Can you fill me in on these two unusual keys?

IPASS: This is not a standard DejaGNU test result. Is this supposed
  to be KPASS (which is missing from the map)?

ERROR: tcl error sourcing: That's a pretty unusual definition of
  ERROR. Was this added in order to specifically catch Tcl errors?
  What about "normal" errors?

What's the preferred method for adding members to this map?

In GDB, we have a special "DUPLICATE" outcome, and right now my plan
is to simply add definitions for this (and any other additions) to our
own gdb.parse_dejagnu_sum function, e.g.,

    test_outcome_map['DUPLICATE'] = 'DUPLICATE'

I'm not a huge fan of modifying global data like this, but since
Bunsen doesn't run over multiple tools at the same time, this shouldn't
be an issue.

Keith

Re: Unusual test_outcome_map keys

[Serhei Makarov]

On Fri, Aug 6, 2021, at 3:35 PM, Keith Seitz via Bunsen wrote:
> Hi,
> 
> While working on fixing up the gdb.sum parser, I've encountered
> `test_outcome_map', defined in common/parse_dejagnu.py.
> 
> This is defined:
> 
> test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS',
>                     'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL',
>                     'ERROR: tcl error sourcing':'ERROR',
>                     'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED',
>                     'UNRESOLVED':'UNRESOLVED'}
> 
> Can you fill me in on these two unusual keys?
Sorry for the delay, I had to dig up a bit of backstory on this.
I thought I saw IPASS in some test results, but I wasn't able to dig up
any indication of where, so I may be wrong. It is certainly true that KPASS should be
included in addition to KFAIL. I took the liberty of patching accordingly:
https://sourceware.org/git/?p=bunsen.git;a=commit;h=763c2037ea61eafdf61df09ef8516278dd1ad12f

I also changed 'ERROR: tcl error sourcing':'ERROR' to 'ERROR':'ERROR'
although I'm less sure of this part. There may be some SystemTap testcases
where the program under observation prints 'ERROR: <stuff>'
and that shouldn't be treated as an error outcome for the testcase.
But that could be a SystemTap-specific nit and I would solve it by using
the method you suggest below to edit test_outcome_map for the SystemTap
parser only.

> What's the preferred method for adding members to this map?
> 
> In GDB, we have a special "DUPLICATE" outcome, and right now my plan
> is to simply add definitions for this (and any other additions) to our
> own gdb.parse_dejagnu_sum function, e.g.,
> 
>     test_outcome_map['DUPLICATE'] = 'DUPLICATE'
> 
> I'm not a huge fan of modifying global data like this, but since
> Bunsen doesn't run over multiple tools at the same time, this shouldn't
> be an issue.
No objections to either method for extending the table.
In my patch, I opted to just add 'DUPLICATE' to test_outcome_map
since it's extremely unlikely to interfere with other testsuites.

-- 
All the best,
    Serhei
    http://serhei.io

Re: Unusual test_outcome_map keys

[Keith Seitz]

On 8/10/21 2:15 PM, Serhei Makarov wrote:
> On Fri, Aug 6, 2021, at 3:35 PM, Keith Seitz via Bunsen wrote:
>> Hi,
>>
>> While working on fixing up the gdb.sum parser, I've encountered
>> `test_outcome_map', defined in common/parse_dejagnu.py.
>>
>> This is defined:
>>
>> test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS',
>>                     'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL',
>>                     'ERROR: tcl error sourcing':'ERROR',
>>                     'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED',
>>                     'UNRESOLVED':'UNRESOLVED'}
>>
>> Can you fill me in on these two unusual keys?
> Sorry for the delay, I had to dig up a bit of backstory on this.
> I thought I saw IPASS in some test results, but I wasn't able to dig up
> any indication of where, so I may be wrong. It is certainly true that KPASS should be
> included in addition to KFAIL. I took the liberty of patching accordingly:
> https://sourceware.org/git/?p=bunsen.git;a=commit;h=763c2037ea61eafdf61df09ef8516278dd1ad12f

Thank you!

> I also changed 'ERROR: tcl error sourcing':'ERROR' to 'ERROR':'ERROR'
> although I'm less sure of this part. There may be some SystemTap testcases
> where the program under observation prints 'ERROR: <stuff>'
> and that shouldn't be treated as an error outcome for the testcase.
> But that could be a SystemTap-specific nit and I would solve it by using
> the method you suggest below to edit test_outcome_map for the SystemTap
> parser only.

GDB has a similar problem (or maybe it is GDB of which you're thinking?)
in gdb.guile tests. GDB itself outputs "ERROR:". :-(

>> I'm not a huge fan of modifying global data like this, but since
>> Bunsen doesn't run over multiple tools at the same time, this shouldn't
>> be an issue.
> No objections to either method for extending the table.
> In my patch, I opted to just add 'DUPLICATE' to test_outcome_map
> since it's extremely unlikely to interfere with other testsuites.

Great, thank you!

Keith