Re: [RFC] GDB sanity test

["Serhei Makarov" <me@serhei.io>]

On Tue, Apr 19, 2022, at 3:01 PM, Keith Seitz via Bunsen wrote:
> This RFC is meant to start a discussion about testing. As bunsen
> starts to mature and more people rely on it, it is imperative that
> contributions don't randomly break other projects.
>
> In that vein, this is a basic sanity test that I've written for GDB
> which imports a single gdb.{log,sum} from a test fixture (.tar.xz).
> It double-checks all results and test counts.
Correct, & I have other ideas for operations to test
(e.g. duplicate-add, list testruns, update, delete) based on your
example.

Any other operations that come to mind as a priority to test?

> To do this, I've written a custom gdb test script that outputs every
> valid test outcome and run this on a fedora 33 VM.
>
> I'm only including the actual test file, not the glue code
> (tests/core.py) necessary to make it work.
It looks good to me. I'm fine if you commit it and I add some SystemTap variants
subject to the following caveats/questions:

- Perhaps the test data is better to commit as plaintext rather than binary .tar.xz?
  When needed, a tar or tar.xz can be prepared with tar cJf /path/to/sample/data/*

- We'll want to run the same sequence of test operations on different data sets,
  so I would need to rework your fixture to allow that (e.g. perhaps
  expected result counts / etc. could be specified in the fixture and the 
  checking procedure generalized). Similar fixtures could be created
  for more realistic SystemTap and GDB data (to test the parser's edge cases),
  and perhaps for other completely-synthetic data (where I can encode
  some regressions that we would want the downstream analysis to capture).

- Since there can be Bunsen setups operating on private data, ideally
  I'd like to make sure they can set up their own fixtures to run this
  testsuite against and check for breakage, the same way that Bunsen
  itself can invoke local scripts stored in .bunsen/scripts-whatever/.
  Pretty sure a way to do that can be concocted with these standard Python
  testing tools.

Couple of comments on the code:

> diff --git a/tests/gdb/test_gdb_sanity.py b/tests/gdb/test_gdb_sanity.py
> new file mode 100644
> index 0000000..1ce65be
> --- /dev/null
> +++ b/tests/gdb/test_gdb_sanity.py
> @@ -0,0 +1,82 @@
> +# A quick sanity test that the ensures that bunsen can import a really
> +# basic log/sum test run which contains all test outcomes.
> +
> +import os.path
> +from tests.core import setup_bunsen_repo, import_results
Out of curiousity, is that glue code calling out to shell operations,
or is it invoking methods in class Bunsen directly?

> +TEST_FIXTURE = 'tests/gdb/fixtures/test_gdb_sanity/gdb-sanity.tar.xz'
> +PATH_STRING = '/home/fedora33/work/gdb/fsf/virgin/linux/gdb/testsuite'
If the data is synthetic (and even if it isn't), we could do a search and replace
to change it to something more generic and less revealing of your work setup :)