Re: [RFC] GDB sanity test

[Keith Seitz <keiths@redhat.com>]

On 4/19/22 13:33, Serhei Makarov wrote:
> On Tue, Apr 19, 2022, at 3:01 PM, Keith Seitz via Bunsen wrote:
>> This RFC is meant to start a discussion about testing. As bunsen
>> starts to mature and more people rely on it, it is imperative that
>> contributions don't randomly break other projects.
>>
>> In that vein, this is a basic sanity test that I've written for GDB
>> which imports a single gdb.{log,sum} from a test fixture (.tar.xz).
>> It double-checks all results and test counts.
> Correct, & I have other ideas for operations to test
> (e.g. duplicate-add, list testruns, update, delete) based on your
> example.
> 
> Any other operations that come to mind as a priority to test?

My priority right now is correctness testing, starting with, e.g,
the summarize script. After that my next step is to test the log
annotation and see if that is working as expected.

>> To do this, I've written a custom gdb test script that outputs every
>> valid test outcome and run this on a fedora 33 VM.
>>
>> I'm only including the actual test file, not the glue code
>> (tests/core.py) necessary to make it work.
> It looks good to me. I'm fine if you commit it and I add some SystemTap variants
> subject to the following caveats/questions:
> 
> - Perhaps the test data is better to commit as plaintext rather than binary .tar.xz?
>    When needed, a tar or tar.xz can be prepared with tar cJf /path/to/sample/data/*

I can do that.

> - We'll want to run the same sequence of test operations on different data sets,
>    so I would need to rework your fixture to allow that (e.g. perhaps
>    expected result counts / etc. could be specified in the fixture and the
>    checking procedure generalized). Similar fixtures could be created
>    for more realistic SystemTap and GDB data (to test the parser's edge cases),
>    and perhaps for other completely-synthetic data (where I can encode
>    some regressions that we would want the downstream analysis to capture).

Yeah, that is something that I've only cursorily thought about as a generalized
solution. This sanity test (which really is sort of a basic gdb parser test) is,
IMO, okay to hardcode the expected results. I agree we will want some infrastructure
to facilitate writing tests.

I guess if we rely on DejaGNU .log/.sum files as test data, we could write up
a function to grok the results at the end of the .sum file to compare with what
the actual bunsen parsers read. That is essentially how/why I wrote summarize.py.

Any other ideas how we might want to do this?

> - Since there can be Bunsen setups operating on private data, ideally
>    I'd like to make sure they can set up their own fixtures to run this
>    testsuite against and check for breakage, the same way that Bunsen
>    itself can invoke local scripts stored in .bunsen/scripts-whatever/.
>    Pretty sure a way to do that can be concocted with these standard Python
>    testing tools.

The one obvious missing bit is being able to easily invoke tests. Right now,
I just rely on "python3 -m pytest tests" to run all tests. Maybe a makefile
or shell script to do this would be appropriate.

> Couple of comments on the code:
> 
>> diff --git a/tests/gdb/test_gdb_sanity.py b/tests/gdb/test_gdb_sanity.py
>> new file mode 100644
>> index 0000000..1ce65be
>> --- /dev/null
>> +++ b/tests/gdb/test_gdb_sanity.py
>> @@ -0,0 +1,82 @@
>> +# A quick sanity test that the ensures that bunsen can import a really
>> +# basic log/sum test run which contains all test outcomes.
>> +
>> +import os.path
>> +from tests.core import setup_bunsen_repo, import_results
> Out of curiousity, is that glue code calling out to shell operations,
> or is it invoking methods in class Bunsen directly?

It is 100% python. Between wrapping python API calls in a shell script for
CLI usage and wrapping shell scripts as API calls for python, I prefer the
former, and that's what I've implemented.

Writing python libraries as shell script wrappers is beyond my pain
threshold. I am writing python-based utilities/web services, not a cron job
or ancient (and insecure) CGI script.

>> +TEST_FIXTURE = 'tests/gdb/fixtures/test_gdb_sanity/gdb-sanity.tar.xz'
>> +PATH_STRING = '/home/fedora33/work/gdb/fsf/virgin/linux/gdb/testsuite'
> If the data is synthetic (and even if it isn't), we could do a search and replace
> to change it to something more generic and less revealing of your work setup :)

This reveals nothing other than how the VM I used to generate the data is setup.

I'll continue hacking at this a bit until I'm satisfied that things are genuinely
useful. I've discovered a few minor issues with the gdb parser that I'd like to
correct, and that will affect this test.

Keith