Pytest usage in DejaGNU?

Pytest usage in DejaGNU?

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 1868 bytes --]

Hello.

GCOV tests suffer from tests that would cover the intermediate format.
It's a JSON format and and I'm attaching an example of its output.

I would really like to use Python to make more complex tests:

$ cat test_json.py
import pytest
import json

def test_gcov_output():
     data = json.load(open('gcov.json'))
     assert len(data['files']) == 1
     f0 = data['files'][0]
     assert f0['file'] == 'gcov-lambda.C'
     assert len(f0['functions']) == 3

     fns = {}
     for fn in f0['functions']:
         fns[fn['name']] = fn
     lines = f0['lines']

     for line in lines:
         lineno = line['line_number']
         linefn = line['function_name']
         assert linefn in fns
         fn = fns[linefn]
         assert fn['start_line'] <= lineno and lineno <= fn['end_line']

I see it pretty complicated to do the same in DejaGNU. Mainly due the missing
JSON parser.

Would it be possible to make optional Python tests in our testsuite?
I can imagine a simple pytest wrapper that will do something like:

+proc pytest-execute { dgargs } {
+    verbose "dg-pytest-execute: ${dgargs}" 2
+    set script [lindex $dgargs 0]
+    verbose "  script: ${script}" 2
+
+    spawn -noecho pytest -rA -s --tb=no $script
+
+    expect {
+      -re "FAILED .*" {
+       fail "pytest $expect_out(0,string)"
+      }
+      -re "PASSED .*" {
+       pass "pytest $expect_out(0,string)"
+      }
+    }
+}

as Pytest can provide a reasonable close output:

=========================================================================================================================== short test summary info ============================================================================================================================
PASSED test_json.py::test_gcov_output
PASSED test_json.py::test_gcov_output
PASSED test_json.py::test_gcov_output

Thoughts?
Martin

[-- Attachment #2: gcov.json --]
[-- Type: application/json, Size: 4817 bytes --]

Re: Pytest usage in DejaGNU?

[Jeff Law]

On 12/14/20 9:02 AM, Martin Liška wrote:
> Hello.
>
> GCOV tests suffer from tests that would cover the intermediate format.
> It's a JSON format and and I'm attaching an example of its output.
>
> I would really like to use Python to make more complex tests:
>
> $ cat test_json.py
> import pytest
> import json
>
> def test_gcov_output():
>     data = json.load(open('gcov.json'))
>     assert len(data['files']) == 1
>     f0 = data['files'][0]
>     assert f0['file'] == 'gcov-lambda.C'
>     assert len(f0['functions']) == 3
>
>     fns = {}
>     for fn in f0['functions']:
>         fns[fn['name']] = fn
>     lines = f0['lines']
>
>     for line in lines:
>         lineno = line['line_number']
>         linefn = line['function_name']
>         assert linefn in fns
>         fn = fns[linefn]
>         assert fn['start_line'] <= lineno and lineno <= fn['end_line']
>
> I see it pretty complicated to do the same in DejaGNU. Mainly due the
> missing
> JSON parser.
>
> Would it be possible to make optional Python tests in our testsuite?
> I can imagine a simple pytest wrapper that will do something like:
>
> +proc pytest-execute { dgargs } {
> +    verbose "dg-pytest-execute: ${dgargs}" 2
> +    set script [lindex $dgargs 0]
> +    verbose "  script: ${script}" 2
> +
> +    spawn -noecho pytest -rA -s --tb=no $script
> +
> +    expect {
> +      -re "FAILED .*" {
> +       fail "pytest $expect_out(0,string)"
> +      }
> +      -re "PASSED .*" {
> +       pass "pytest $expect_out(0,string)"
> +      }
> +    }
> +}
>
> as Pytest can provide a reasonable close output:
>
> ===========================================================================================================================
> short test summary info
> ============================================================================================================================
> PASSED test_json.py::test_gcov_output
> PASSED test_json.py::test_gcov_output
> PASSED test_json.py::test_gcov_output
I thought we already approved using python elsewhere (JIT?  Analyzer?)

jeff

Re: Pytest usage in DejaGNU?

[Joseph Myers]

On Mon, 14 Dec 2020, Martin Liška wrote:

> +    spawn -noecho pytest -rA -s --tb=no $script

"pytest" might not be the right command everywhere.  If I install 
python3-pytest on Ubuntu 20.04 I only get /usr/bin/pytest-3 not 
/usr/bin/pytest.

There is a clear advantage to any usage of Python only depending on the 
standard library and not other Python packages.

> PASSED test_json.py::test_gcov_output
> PASSED test_json.py::test_gcov_output
> PASSED test_json.py::test_gcov_output

The names after "PASS: " or "FAIL: " in DejaGnu output should be unique 
(and not depend on whether the test in question passes or fails, or on 
variables such as the build directory); there should not be three tests 
with the same name.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Pytest usage in DejaGNU?

[Matthias Klose]

On 12/14/20 10:21 PM, Joseph Myers wrote:
> On Mon, 14 Dec 2020, Martin Liška wrote:
> 
>> +    spawn -noecho pytest -rA -s --tb=no $script
> 
> "pytest" might not be the right command everywhere.  If I install 
> python3-pytest on Ubuntu 20.04 I only get /usr/bin/pytest-3 not 
> /usr/bin/pytest.

<python> -m pytest should work everywhere.