'tests/runtestslowselfcompare.sh' (was: [PATCH] Bug 26769 - Fix missing types in abixml output)

[Thomas Schwinge <thomas@codesourcery.com>]

Hi!

On 2020-11-24T11:30:02+0100, Dodji Seketeli via Libabigail <libabigail@sourceware.org> wrote:
>       * tests/runtestslowselfcompare.sh.in: New test that compares
>       libabigail.so against its own ABIXML representation.

I consider this to be a pretty important test case -- "eat our own dog
food".

Thus, I find this a bit unfortunate:

>       * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to
>       source distribution.  This test is too slow to be run during the
>       course of 'make check'.  It takes more than 5 minutes on my slow
>       box here.  Rather, it can be run using 'make check-self-compare'.
>       I plan to run this before releases now.

..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during
standard 'make check'.

Or may we can change that now?  Are you still seeing that "It takes more
than 5 minutes on my slow box here"?  Because, on my eight years old Dell
Precision M4700, I see:

    $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes

    20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k
    20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k
    20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k

So, ~21 s.

All the other test cases, running in parallel (just '-j5'):

    $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no'

    364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k
    359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k
    359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k

So, ~61 s.  Additionally running 'tests/runtestslowselfcompare.sh':

    $ \time make check -j5 ENABLE_SLOW_TEST=yes

    389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k
    387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k
    388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k

So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the
'make check -j5' take just ~5 s longer -- acceptable, in my opinion.


Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030
"CONTRIBUTING: Update instructions about regression tests", it was noted
that...

| This is an important regression test.  The
| problem is that it can takes twice as much time as make distcheck.  So
| we've put it into its own separate target.

Given the "5 minutes" number from above, this comment means that a
'make distcheck' (or rather 'make distcheck-fast', I suppose?) would run
~2.5 min?  I've got the following numbers:

    $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no'

    935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k
    946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k
    935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k

So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone
takes just 1/15 the time of that, not "twice as much".

Additionally enabling 'tests/runtestslowselfcompare.sh' here:

    $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes

    965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k
    981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k
    972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k

..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s
longer -- acceptable, in my opinion.


Therefore I propose to do away with 'ENABLE_SLOW_TEST', and merge
'make check-self-compare' into standard 'make check'.  I'll be happy to
prepare a patch, if we arrive at consensus, but first: happy to hear any
other opinions/data.


Grüße
 Thomas


> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -88,3 +88,9 @@ tag-and-all: distcheck
>  distcheck-fast:
>       $(MAKE) distcheck GZIP_ENV="--fast"
>
> +# This makes us compare libabigail.so against its own ABIXML
> +# representation.  It's super slow (more than 5 minutes on one of my
> +# old boxes) so I do this only before releases.
> +check-self-compare:
> +     $(MAKE) -C tests check-self-compare

> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> [...]
> +check-self-compare:
> +     $(MAKE) check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes

> --- /dev/null
> +++ b/tests/runtestslowselfcompare.sh.in
> @@ -0,0 +1,13 @@
> +#!/bin/sh -x
> +
> +# This program launches "abidw --abidiff src/.libs/libabigail.so"
> +# This compares libabigail.so against its own ABIXML output.
> +abidw=@top_builddir@/tools/abidw
> +objdir=@top_builddir@/src/.libs
> +
> +echo "ENABLE_SLOW_TEST="$ENABLE_SLOW_TEST
> +
> +if test x$ENABLE_SLOW_TEST != x; then
> +    `$abidw --abidiff $objdir/libabigail.so`
> +fi
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955