From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by sourceware.org (Postfix) with ESMTP id E189F3858D29 for ; Mon, 14 Dec 2020 20:30:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org E189F3858D29 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-593-jM_sR-F8Px2lLXsFNuKP-A-1; Mon, 14 Dec 2020 15:30:42 -0500 X-MC-Unique: jM_sR-F8Px2lLXsFNuKP-A-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8AAF659; Mon, 14 Dec 2020 20:30:41 +0000 (UTC) Received: from localhost.localdomain (ovpn-113-227.phx2.redhat.com [10.3.113.227]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5404B100164C; Mon, 14 Dec 2020 20:30:41 +0000 (UTC) Subject: Re: Pytest usage in DejaGNU? To: =?UTF-8?Q?Martin_Li=c5=a1ka?= , GCC Development References: From: Jeff Law Message-ID: Date: Mon, 14 Dec 2020 13:30:40 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Spam-Status: No, score=-3.5 required=5.0 tests=BAYES_00, BODY_8BITS, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_ASCII_DIVIDERS, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Dec 2020 20:30:46 -0000 On 12/14/20 9:02 AM, Martin Liška wrote: > Hello. > > GCOV tests suffer from tests that would cover the intermediate format. > It's a JSON format and and I'm attaching an example of its output. > > I would really like to use Python to make more complex tests: > > $ cat test_json.py > import pytest > import json > > def test_gcov_output(): >     data = json.load(open('gcov.json')) >     assert len(data['files']) == 1 >     f0 = data['files'][0] >     assert f0['file'] == 'gcov-lambda.C' >     assert len(f0['functions']) == 3 > >     fns = {} >     for fn in f0['functions']: >         fns[fn['name']] = fn >     lines = f0['lines'] > >     for line in lines: >         lineno = line['line_number'] >         linefn = line['function_name'] >         assert linefn in fns >         fn = fns[linefn] >         assert fn['start_line'] <= lineno and lineno <= fn['end_line'] > > I see it pretty complicated to do the same in DejaGNU. Mainly due the > missing > JSON parser. > > Would it be possible to make optional Python tests in our testsuite? > I can imagine a simple pytest wrapper that will do something like: > > +proc pytest-execute { dgargs } { > +    verbose "dg-pytest-execute: ${dgargs}" 2 > +    set script [lindex $dgargs 0] > +    verbose "  script: ${script}" 2 > + > +    spawn -noecho pytest -rA -s --tb=no $script > + > +    expect { > +      -re "FAILED .*" { > +       fail "pytest $expect_out(0,string)" > +      } > +      -re "PASSED .*" { > +       pass "pytest $expect_out(0,string)" > +      } > +    } > +} > > as Pytest can provide a reasonable close output: > > =========================================================================================================================== > short test summary info > ============================================================================================================================ > PASSED test_json.py::test_gcov_output > PASSED test_json.py::test_gcov_output > PASSED test_json.py::test_gcov_output I thought we already approved using python elsewhere (JIT?  Analyzer?) jeff