From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by sourceware.org (Postfix) with ESMTP id 7D56D398B87E for ; Wed, 11 Aug 2021 17:03:13 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7D56D398B87E Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-193-78su7aKuPMK6-BK_v_tdjg-1; Wed, 11 Aug 2021 13:03:11 -0400 X-MC-Unique: 78su7aKuPMK6-BK_v_tdjg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EF6D6108292D; Wed, 11 Aug 2021 17:03:09 +0000 (UTC) Received: from [10.3.115.36] (ovpn-115-36.phx2.redhat.com [10.3.115.36]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8318F69CBA; Wed, 11 Aug 2021 17:03:09 +0000 (UTC) Subject: Re: Unusual test_outcome_map keys To: Serhei Makarov , Bunsen References: From: Keith Seitz Message-ID: <27f68d6e-a412-a8a7-a100-e5fb1549fa41@redhat.com> Date: Wed, 11 Aug 2021 10:03:08 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: bunsen@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bunsen mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2021 17:03:24 -0000 On 8/10/21 2:15 PM, Serhei Makarov wrote: > On Fri, Aug 6, 2021, at 3:35 PM, Keith Seitz via Bunsen wrote: >> Hi, >> >> While working on fixing up the gdb.sum parser, I've encountered >> `test_outcome_map', defined in common/parse_dejagnu.py. >> >> This is defined: >> >> test_outcome_map = {'PASS':'PASS', 'XPASS':'XPASS', 'IPASS':'IPASS', >> 'FAIL':'FAIL', 'KFAIL':'KFAIL', 'XFAIL':'XFAIL', >> 'ERROR: tcl error sourcing':'ERROR', >> 'UNTESTED':'UNTESTED', 'UNSUPPORTED':'UNSUPPORTED', >> 'UNRESOLVED':'UNRESOLVED'} >> >> Can you fill me in on these two unusual keys? > Sorry for the delay, I had to dig up a bit of backstory on this. > I thought I saw IPASS in some test results, but I wasn't able to dig up > any indication of where, so I may be wrong. It is certainly true that KPASS should be > included in addition to KFAIL. I took the liberty of patching accordingly: > https://sourceware.org/git/?p=bunsen.git;a=commit;h=763c2037ea61eafdf61df09ef8516278dd1ad12f Thank you! > I also changed 'ERROR: tcl error sourcing':'ERROR' to 'ERROR':'ERROR' > although I'm less sure of this part. There may be some SystemTap testcases > where the program under observation prints 'ERROR: ' > and that shouldn't be treated as an error outcome for the testcase. > But that could be a SystemTap-specific nit and I would solve it by using > the method you suggest below to edit test_outcome_map for the SystemTap > parser only. GDB has a similar problem (or maybe it is GDB of which you're thinking?) in gdb.guile tests. GDB itself outputs "ERROR:". :-( >> I'm not a huge fan of modifying global data like this, but since >> Bunsen doesn't run over multiple tools at the same time, this shouldn't >> be an issue. > No objections to either method for extending the table. > In my patch, I opted to just add 'DUPLICATE' to test_outcome_map > since it's extremely unlikely to interfere with other testsuites. Great, thank you! Keith