Re: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh'

[Thomas Schwinge <thomas@codesourcery.com>]

Hi Dodji!

Happy New Year to y'all, too!


On 2022-01-04T15:53:57+0100, Dodji Seketeli <dodji@redhat.com> wrote:
> Thomas Schwinge <thomas@codesourcery.com> a écrit:
>
>> Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97
>> "Bug 26769 - Fix missing types in abixml output":
>>
>> |    * tests/runtestslowselfcompare.sh.in: New test that compares
>> |    libabigail.so against its own ABIXML representation.
>>
>> I consider this to be a pretty important test case -- "eat our own dog
>> food".
>>
>> Thus, I find this a bit unfortunate:
>>
>> |    * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to
>> |    source distribution.  This test is too slow to be run during the
>> |    course of 'make check'.  It takes more than 5 minutes on my slow
>> |    box here.  Rather, it can be run using 'make check-self-compare'.
>> |    I plan to run this before releases now.
>>
>> ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during
>> standard 'make check'.
>>
>> On my eight years old Dell Precision M4700, I see:
>>
>>     $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes
>>
>>     20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k
>>     20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k
>>     20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k
>>
>> So, ~21 s.
>
> On my machine (AMD FX8350 box) it's really still much longer than that, as
> you pointed above.  And I know users who are in the same case, still.
> Yeah, surprising, I know.  Your box is more than 10 times faster than
> mine, it seems.

"Interesting" ;-) -- thanks for confirming your numbers.

Per a quick web search, your AMD FX8350 would be just a little older than
my Dell Precision M4700 with "Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz",
so I wonder where the rather big difference is coming from.  Mine has
24 GiB of RAM, and I'm caching the "WDC WD7500BPKT-7" HDD with a
"KINGSTON SA400S3" SSD; maybe that's it.

>> All the other test cases, running in parallel (just '-j5'):
>>
>>     $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no'
>>
>>     364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k
>>     359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k
>>     359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k
>>
>> So, ~61 s.  Additionally running 'tests/runtestslowselfcompare.sh':
>>
>>     $ \time make check -j5 ENABLE_SLOW_TEST=yes
>>
>>     389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k
>>     387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k
>>     388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k
>>
>> So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the
>> 'make check -j5' take just ~5 s longer -- acceptable, in my opinion.
>>
>> Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030
>> "CONTRIBUTING: Update instructions about regression tests", it was noted
>> that...
>>
>> | This is an important regression test.  The
>> | problem is that it can takes twice as much time as make distcheck.  So
>> | we've put it into its own separate target.
>>
>> Given the "5 minutes" number from above, this comment means that a
>> 'make distcheck' (or rather 'make distcheck-fast', I suppose?)
>
> Well, you are maybe reading too much into that sentence :-)  Since that time,
> make distcheck grew slower.  So I guess the test now takes roughly the
> same time as make distcheck or make distcheck-fast.
>
>> would run
>> ~2.5 min?  I've got the following numbers:
>>
>>     $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no'
>>
>>     935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k
>>     946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k
>>     935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k
>>
>> So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone
>> takes just 1/15 the time of that, not "twice as much".
>>
>> Additionally enabling 'tests/runtestslowselfcompare.sh' here:
>>
>>     $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes
>>
>>     965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k
>>     981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k
>>     972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k
>>
>> ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s
>> longer -- acceptable, in my opinion.
>
> On my machine, your patch "make check" takes 5 minutes longer than
> without.  So I am reluctant to apply it.

Sure, understood, and no worries.

> In practise, it's not a big deal, I think, as I run 'make
> check-self-compare' very regularly.
>
> As bizarre as it seems, the tendency I'd like us to move towards is less
> tests "by default".  That is, split out the binaries that we have in the
> tarball today and keep a very minimal set of tests.

My goal is just to establish a testing baseline, so that I can be
reasonably sure that changes I may be doing don't regress anything.
For the time being, I might just locally set 'ENABLE_SLOW_TEST=yes'; and
at least we've now got some numbers (confirmed/new) in the archives.

> The tarball is too
> big as it is, and yet, there are tons of tests that I'd like to run that
> are not present there.
>
> That is why I'd like us to progress towards having much more tests that
> are in a separated "test project", somehow.  In that project, we'd
> either have binaries locally present or references to binaries (like
> distros packages over the interweb) to grab and run comparisons on.

Yes, that makes sense.  (I did wonder about the Git repository/checkout
size as well as the fact that huge binary blobs are stored there without
sources reference.)

> So I would not spent to much time on these tests that are locally
> present in the tarball.  Rather, if you are interested in this super
> important testing strategy project, we could discuss it a bit more in
> depth.

Heh, not at this time, sorry.  ;-)


Grüße
 Thomas
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955