From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.CeBiTec.Uni-Bielefeld.DE (smtp.CeBiTec.Uni-Bielefeld.DE [129.70.160.84]) by sourceware.org (Postfix) with ESMTPS id 71911385783D for ; Wed, 13 Jan 2021 13:38:20 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 71911385783D Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=CeBiTec.Uni-Bielefeld.DE Authentication-Results: sourceware.org; spf=none smtp.mailfrom=ro@cebitec.uni-bielefeld.de Received: from localhost (localhost [127.0.0.1]) by smtp.CeBiTec.Uni-Bielefeld.DE (Postfix) with ESMTP id 15548141C; Wed, 13 Jan 2021 14:38:18 +0100 (CET) X-Virus-Scanned: amavisd-new at CeBiTec.Uni-Bielefeld.DE Received: from smtp.CeBiTec.Uni-Bielefeld.DE ([127.0.0.1]) by localhost (smtp.cebitec.uni-bielefeld.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G5fgY_xOx1iq; Wed, 13 Jan 2021 14:38:17 +0100 (CET) Received: from manam.CeBiTec.Uni-Bielefeld.DE (p4fddb15f.dip0.t-ipconnect.de [79.221.177.95]) by smtp.CeBiTec.Uni-Bielefeld.DE (Postfix) with ESMTPSA id 56F3A13B7; Wed, 13 Jan 2021 14:38:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=CeBiTec.Uni-Bielefeld.DE; s=20200306; t=1610545097; bh=Tu1BktXFSWDq4oU2IKnNFU1GXN3FT9O0F1v/pUltY2U=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=P48paq+lzWGd0ysq3oNdMmaheXKtANS+17XhLFNALyzjUyrP/WJwI/VCQfgPmXRZ+ wRx3M/atQtsqYlhwVwTx3zcy9Bomnd6Fdg7pDtDDT/eY4+EZkOtHMPI+LEhEoyst0z SmUAY0dLMgf7uera4V8hBWZ+VotD+IF5dDDXMr2R8KALmhb0QUEPvar4+FWm6La+0P udw1eM/yHh7xTb8GUMGDRfEi2/gRHyPvd1PreDxTR1bwX0HLbTFg0CgbmsDl4pzuB/ wJpLPufyF2/F8ZXn1LmGDnI559euhJma4DjI7O3I7DXo8uBljcBR5qx5qLmVu3cyvt CXzskgiUegyuQ== From: Rainer Orth To: Martin =?utf-8?Q?Li=C5=A1ka?= Cc: Jeff Law , David Malcolm , gcc-patches@gcc.gnu.org Subject: Re: [PATCH] Add pytest for a GCOV test-case References: <05b7e58dc023fbea33a051a7800a214c838feeb4.camel@redhat.com> <9021ed74-22d4-5f2a-c60f-4e0bc63e48b6@redhat.com> Date: Wed, 13 Jan 2021 14:38:16 +0100 In-Reply-To: ("Martin =?utf-8?Q?Li=C5=A1ka=22's?= message of "Thu, 7 Jan 2021 17:14:13 +0100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1.90 (usg-unix-v) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Status: No, score=-3789.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, KAM_ASCII_DIVIDERS, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2021 13:38:26 -0000 Hi Martin, > On 1/6/21 12:36 AM, Jeff Law wrote: >> unresolved "could not find python interpreter $testcase" in >> run-gcov-pytest if you find the right magic in the output of your spawn. > > Achieved that with the updated patch. > > Ready for master? unfortunately, your patch has a large number of problems: * On targets where run-gcov-pytest decides that pytest isn't available (incorrectly in some cases), mail-report.log is cluttered with UNRESOLVED: could not find Python interpreter and (or) pytest module for pr98273.C I fear you've been misled by David and Jeff here: UNRESOLVED isn't appropriate for cases like this. Please read the DejaGnu manual for the semantics of the various test outcomes. If anything (we often just silently skip testcases that cannot be run on some target), use UNSUPPORTED instead. * Besides, the test outcomes are not generic message facilities but are supposed to follow a common format: : [] with the pathname to the test relative to (in this case) gcc/testsuite. In this case, this might be something like UNSUPPORTED: g++.dg/gcov/pr98273.C run-gcov-pytest Currently, you don't have the pathname in run-gcov-pytest, though. * If we now have an (even optional) dependency on python/pytest, this (with the exact versions and use) needs to be documented in install.texi. * Speaking of documenting, the new run-gcov-pytest needs to be documented in sourcebuild.texi. * On to the implementation: your test for the presence of pytest is wrong: set result [remote_exec host "pytest -m pytest --version"] has nothing to do with what you actually use later: on all of Fedora 29, Ubuntu 20.04, and Solaris 11.4 (with a caveat) pytest is Python 2.7 based, but you don't check that. It is well possible that pytest for 2.7 is installed, but pytest for Python 3.x isn't. Besides, while Solaris 11.4 does bundle pytest, they don't deliver pytest, but only py.test due to a conflict with a different pytest from logilab-common, cf. https://github.com/pytest-dev/pytest/issues/1833. This is immaterial, however, since what you actually run is spawn -noecho python3 -m pytest --color=no -rA -s --tb=no $srcdir/$subdir/$pytest_script So you should just run python3 -m pytest --version instead to check for the presence of the version you're going to use. Btw., there's a mess with pytest on Fedora 29: running the above gives [...] pluggy.PluginValidationError: Plugin 'benchmark' could not be loaded: (pytest 3.6.4 (/usr/lib/python3.7/site-packages), Requirement.parse('pytest>=3.8'))! Seems the packagers have broken things there. On top of all this, I wonder why you insist on a particular Python version here: I tried your single testcase and it PASSes just as well with Python 2.7!? One reason I'm asking is that Solaris 11.3 bundles both Python 2.7 and 3.4, but (unlike Linux and Solaris 11.4) don't have /usr/bin/python3, just python (which is 2.7), python2.7, and python3.4. Not that it matters too much, but you should be aware of the issue. When running the test on Solaris 11.4 (with the bundled pytest 4.4.0), I get ============================= test session starts ============================== platform sunos5 -- Python 3.7.9, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 rootdir: /vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov collected 2 items ../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py .. =========================== 2 passed in 0.04 seconds =========================== while 4.6.9 on Linux gives ============================= test session starts ============================== platform linux -- Python 3.8.2, pytest-4.6.9, py-1.8.1, pluggy-0.13.0 rootdir: /vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov collected 2 items ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py .. =========================== short test summary info ============================ PASSED ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_basics PASSED ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_lines =========================== 2 passed in 0.17 seconds =========================== Obviously pytest -rA was introduced only after 4.4.0 and the 'A' is silently ignored. Fortunately, I can just use -rap instead which works with both versions. After this has been processed by gcov.exp, g++.sum contains PASS: ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_basic PASS: ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_line which is again completely wrong in light of what I wrote above on the format of test names: it lacks the testname part completely and contains absolute pathnames which makes it impossible to compare testresults from different systems. Instead, there should be some sort of tag, perhaps patterned after what the various scan-* functions do. Please fix. Rainer -- ----------------------------------------------------------------------------- Rainer Orth, Center for Biotechnology, Bielefeld University