Re: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh'

[Dodji Seketeli <dodji@redhat.com>]

Hello Thomas,

Thomas Schwinge <thomas@codesourcery.com> a écrit:

> Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97
> "Bug 26769 - Fix missing types in abixml output":
>
> | 	* tests/runtestslowselfcompare.sh.in: New test that compares
> | 	libabigail.so against its own ABIXML representation.
>
> I consider this to be a pretty important test case -- "eat our own dog
> food".
>
> Thus, I find this a bit unfortunate:
>
> | 	* tests/Makefile.am: Add the new test runtestslowselfcompare.sh to
> | 	source distribution.  This test is too slow to be run during the
> | 	course of 'make check'.  It takes more than 5 minutes on my slow
> | 	box here.  Rather, it can be run using 'make check-self-compare'.
> | 	I plan to run this before releases now.
>
> ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during
> standard 'make check'.
>
> On my eight years old Dell Precision M4700, I see:
>
>     $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes
>
>     20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k
>     20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k
>     20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k
>
> So, ~21 s.

On my machine (AMD FX8350 box) it's really still much longer than that, as
you pointed above.  And I know users who are in the same case, still.
Yeah, surprising, I know.  Your box is more than 10 times faster than
mine, it seems.

> All the other test cases, running in parallel (just '-j5'):
>
>     $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no'
>
>     364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k
>     359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k
>     359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k
>
> So, ~61 s.  Additionally running 'tests/runtestslowselfcompare.sh':
>
>     $ \time make check -j5 ENABLE_SLOW_TEST=yes
>
>     389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k
>     387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k
>     388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k
>
> So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the
> 'make check -j5' take just ~5 s longer -- acceptable, in my opinion.
>
> Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030
> "CONTRIBUTING: Update instructions about regression tests", it was noted
> that...
>
> | This is an important regression test.  The
> | problem is that it can takes twice as much time as make distcheck.  So
> | we've put it into its own separate target.
>
> Given the "5 minutes" number from above, this comment means that a
> 'make distcheck' (or rather 'make distcheck-fast', I suppose?)

Well, you are maybe reading too much into that sentence :-)  Since that time,
make distcheck grew slower.  So I guess the test now takes roughly the
same time as make distcheck or make distcheck-fast.

> would run
> ~2.5 min?  I've got the following numbers:
>
>     $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no'
>
>     935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k
>     946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k
>     935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k
>
> So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone
> takes just 1/15 the time of that, not "twice as much".
>
> Additionally enabling 'tests/runtestslowselfcompare.sh' here:
>
>     $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes
>
>     965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k
>     981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k
>     972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k
>
> ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s
> longer -- acceptable, in my opinion.

On my machine, your patch "make check" takes 5 minutes longer than
without.  So I am reluctant to apply it.

In practise, it's not a big deal, I think, as I run 'make
check-self-compare' very regularly.

As bizarre as it seems, the tendency I'd like us to move towards is less
tests "by default".  That is, split out the binaries that we have in the
tarball today and keep a very minimal set of tests.  The tarball is too
big as it is, and yet, there are tons of tests that I'd like to run that
are not present there.

That is why I'd like us to progress towards having much more tests that
are in a separated "test project", somehow.  In that project, we'd
either have binaries locally present or references to binaries (like
distros packages over the interweb) to grab and run comparisons on.

So I would not spent to much time on these tests that are locally
present in the tarball.  Rather, if you are interested in this super
important testing strategy project, we could discuss it a bit more in
depth.


I hope this makes sense.

Thank you for showing interesting in this area.

[...]

Cheers,

-- 
		Dodji