Re: [PATCH] Add pytest for a GCOV test-case

[Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE>]

Hi Martin,

> On 1/6/21 12:36 AM, Jeff Law wrote:
>> unresolved "could not find python interpreter $testcase" in
>> run-gcov-pytest if you find the right magic in the output of your spawn.
>
> Achieved that with the updated patch.
>
> Ready for master?

unfortunately, your patch has a large number of problems:

* On targets where run-gcov-pytest decides that pytest isn't available
  (incorrectly in some cases), mail-report.log is cluttered with

UNRESOLVED: could not find Python interpreter and (or) pytest module for pr98273.C

  I fear you've been misled by David and Jeff here: UNRESOLVED isn't
  appropriate for cases like this.  Please read the DejaGnu manual for
  the semantics of the various test outcomes.  If anything (we often
  just silently skip testcases that cannot be run on some target), use
  UNSUPPORTED instead.

* Besides, the test outcomes are not generic message facilities but are
  supposed to follow a common format:

  <result>: <testname> [<subtest>]

  with <testname> the pathname to the test relative to (in this case)
  gcc/testsuite.  In this case, this might be something like

  UNSUPPORTED: g++.dg/gcov/pr98273.C run-gcov-pytest

  Currently, you don't have the pathname in run-gcov-pytest, though.

* If we now have an (even optional) dependency on python/pytest, this
  (with the exact versions and use) needs to be documented in
  install.texi.

* Speaking of documenting, the new run-gcov-pytest needs to be
  documented in sourcebuild.texi.

* On to the implementation: your test for the presence of pytest is
  wrong:

    set result [remote_exec host "pytest -m pytest --version"]

  has nothing to do with what you actually use later: on all of Fedora
  29, Ubuntu 20.04, and Solaris 11.4 (with a caveat) pytest is Python
  2.7 based, but you don't check that.  It is well possible that pytest
  for 2.7 is installed, but pytest for Python 3.x isn't.

  Besides, while Solaris 11.4 does bundle pytest, they don't deliver
  pytest, but only py.test due to a conflict with a different pytest from
  logilab-common, cf. https://github.com/pytest-dev/pytest/issues/1833.

  This is immaterial, however, since what you actually run is

    spawn -noecho python3 -m pytest --color=no -rA -s --tb=no $srcdir/$subdir/$pytest_script

  So you should just run python3 -m pytest --version instead to check
  for the presence of the version you're going to use.

  Btw., there's a mess with pytest on Fedora 29: running the above gives

[...]
pluggy.PluginValidationError: Plugin 'benchmark' could not be loaded: (pytest 3.6.4 (/usr/lib/python3.7/site-packages), Requirement.parse('pytest>=3.8'))!

  Seems the packagers have broken things there.

  On top of all this, I wonder why you insist on a particular Python
  version here: I tried your single testcase and it PASSes just as well
  with Python 2.7!?  One reason I'm asking is that Solaris 11.3 bundles
  both Python 2.7 and 3.4, but (unlike Linux and Solaris 11.4) don't
  have /usr/bin/python3, just python (which is 2.7), python2.7, and
  python3.4.  Not that it matters too much, but you should be aware of
  the issue.

  When running the test on Solaris 11.4 (with the bundled pytest 4.4.0),
  I get

============================= test session starts ==============================
platform sunos5 -- Python 3.7.9, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
rootdir: /vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov
collected 2 items                                                              

../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py ..

=========================== 2 passed in 0.04 seconds ===========================

while 4.6.9 on Linux gives

============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
rootdir: /vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov
collected 2 items                                                              

../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py ..

=========================== short test summary info ============================
PASSED ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_basics
PASSED ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_lines
=========================== 2 passed in 0.17 seconds ===========================

  Obviously pytest -rA was introduced only after 4.4.0 and the 'A' is
  silently ignored.  Fortunately, I can just use -rap instead which
  works with both versions.

  After this has been processed by gcov.exp, g++.sum contains

PASS:  ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_basic
PASS:  ../../../../../../../../../../vol/gcc/src/hg/master/local/gcc/testsuite/g++.dg/gcov/test-pr98273.py::test_line

  which is again completely wrong in light of what I wrote above on the
  format of test names: it lacks the testname part completely and
  contains absolute pathnames which makes it impossible to compare
  testresults from different systems.  Instead, there should be some
  sort of tag, perhaps patterned after what the various scan-* functions
  do.

Please fix.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University