Re: Testing Setup - More Tests and Automation?

[Ben Woodard <woodard@redhat.com>]

> On Apr 29, 2022, at 2:13 PM, Mark Wielaard <mark@klomp.org> wrote:
> 
> That said, if there are other tests, environment variables or
> configure flags that could be used to catch more issues we could
> enable them (on one or more builders). As long as it doesn't take more
> than 10 minutes to run the extra tests on a particular builder.

 I don’t know of any specific configure flags or environmental variables which could be added. I believe that some additional regression tests are needed. 

I have a WIP patch that I haven’t gotten working yet. It amounts to:

diff --git a/tests/runtestfedabipkgdiff.py.in b/tests/runtestfedabipkgdiff.py.in
index dd4e041d..3ce6878d 100755
--- a/tests/runtestfedabipkgdiff.py.in
+++ b/tests/runtestfedabipkgdiff.py.in
@@ -80,6 +80,35 @@ FEDABIPKGDIFF_TEST_SPECS = [
     (['--self-compare', '-a', '--from', 'fc23', 'dbus-glib'],
      'data/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-report-0.txt',
      'output/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'aspell'],
+     'data/test-fedabipkgdiff/test8-self-compare-from-fc36-aspell-report-0.txt',
+     'output/test-fedabipkgdiff/test8-self-compare-from-fc36-aspell-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'dyninst'],
+     'data/test-fedabipkgdiff/test9-self-compare-from-fc36-dyninst-report-0.txt',
+     'output/test-fedabipkgdiff/test9-self-compare-from-fc36-dyninst-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'libabigail'],
+     'data/test-fedabipkgdiff/test10-self-compare-from-fc36-libabigail-report-0.txt',
+     'output/test-fedabipkgdiff/test10-self-compare-from-fc36-libabigail-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'hdf5'],
+     'data/test-fedabipkgdiff/test11-self-compare-from-fc36-hdf5-report-0.txt',
+     'output/test-fedabipkgdiff/test11-self-compare-from-fc36-hdf5-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'gcc'],
+     'data/test-fedabipkgdiff/test12-self-compare-from-fc36-gcc-report-0.txt',
+     'output/test-fedabipkgdiff/test12-self-compare-from-fc36-gcc-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'openmpi'],
+     'data/test-fedabipkgdiff/test13-self-compare-from-fc36-openmpi-report-0.txt',
+     'output/test-fedabipkgdiff/test13-self-compare-from-fc36-openmpi-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'protobuf'],
+     'data/test-fedabipkgdiff/test14-self-compare-from-fc36-protobuf-report-0.txt',
+     'output/test-fedabipkgdiff/test14-self-compare-from-fc36-protobuf-report-0.txt'),
+
 ]

Along with the expected results in the data directory. These are the ones which most frequently seem to break. Other than getting the patch to actually work, I believe that for it to be acceptable to Dodji it would need to be lumped under the slow tests and I haven’t figured out how to do that yet. Python is not my best language.

So if we could have one of the builders run something like:

for i in aspell dyninst libabigail hdf5 gcc openmpi proj protobuf; do
   fedabipkgdiff --self-compare -a --from fc36 $i >/dev/null
  if [ $? != 0 ];then
    exit 1
  fi
done

Then I think that we’d catch many of the problems that I have been seeing.

Notes:
- Having libabigail in the list may be redundant 
- GCC is in there because libstdc++ tends to break things. I haven’t seen a problem with gcc itself.
- When sharing the diff, I realized that I forgot to add ‘proj’
- Note that these are all C++. libabigail rarely fails on C these days. However, the HPC apps that I care most about are all C++ and Fortran.

-ben