From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa4.mentor.iphmx.com (esa4.mentor.iphmx.com [68.232.137.252]) by sourceware.org (Postfix) with ESMTPS id 82B063858D37 for ; Wed, 5 Jan 2022 16:10:32 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 82B063858D37 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com IronPort-SDR: kUBrW7tAN46T2iYfTOrS82dLdaYT9GXiiyg5dxg4Rr5NuVDMeKycqLIkWEbnSpKuja66l+Vkmz pV1EXVOwsaRBS7Z/QmR5mRN2tuxBkmUzwjKSde2DvLAEvW0DAMAjssbq1tdT1veZdijZLbCfkQ Y5Zs3ideAj+QHI7a0FOycFM8A657rnONAkrPjVyysonGcIXaDizyg75/hK6wNKdSSOdSY/j/Gx 98YOwi/ZOaO7pWF2gYE6gqW0lfPr9s9Xgp6Otk7TdN2CL3cQlqaDyUGhVoJaKm3HNy6cdswvf5 /zpX/vGaDaqZivasFeKHW95z X-IronPort-AV: E=Sophos;i="5.88,264,1635235200"; d="scan'208";a="70418839" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa4.mentor.iphmx.com with ESMTP; 05 Jan 2022 08:10:31 -0800 IronPort-SDR: MV00nQzjsApYCa7Hzr/rQpbWSURSLW6q/pidqajYPDgmYRfSybalH5e8WA+DPFI39Gk3lDgqr4 t3EIJ/Q4Bxj+VYTZwBbHNTft5fdYWw66BcPEz/8jj/CU+ZGNe+l1sWJo3LgHxLsfhu7JyeEUEr YngirgraSHBmkr0oWDzOm2agaiwjnpSNKy2b4vzsQ79UuBJE7vaNnHNxX5hhNNdMBzN1MPwln+ 3rziSAPHmJKenJQxGoW3nxfhm0X+cnGghOtp+QcgzlWqWvQe2N3yoVBzSsec62zZn1XNObJ5nE 2qo= From: Thomas Schwinge To: Dodji Seketeli CC: Subject: Re: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' In-Reply-To: <87a6gbeem2.fsf@seketeli.org> References: <87tuf1ynf3.fsf@euler.schwinge.homeip.net> <20211222104936.3998331-1-thomas@codesourcery.com> <87a6gbeem2.fsf@seketeli.org> User-Agent: Notmuch/0.29.3+94~g74c3f1b (https://notmuchmail.org) Emacs/27.1 (x86_64-pc-linux-gnu) Date: Wed, 5 Jan 2022 17:10:23 +0100 Message-ID: <87wnjew4cw.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-09.mgc.mentorg.com (139.181.222.9) To svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) X-Spam-Status: No, score=-6.1 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2022 16:10:34 -0000 Hi Dodji! Happy New Year to y'all, too! On 2022-01-04T15:53:57+0100, Dodji Seketeli wrote: > Thomas Schwinge a =C3=A9crit: > >> Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97 >> "Bug 26769 - Fix missing types in abixml output": >> >> | * tests/runtestslowselfcompare.sh.in: New test that compares >> | libabigail.so against its own ABIXML representation. >> >> I consider this to be a pretty important test case -- "eat our own dog >> food". >> >> Thus, I find this a bit unfortunate: >> >> | * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to >> | source distribution. This test is too slow to be run during the >> | course of 'make check'. It takes more than 5 minutes on my slow >> | box here. Rather, it can be run using 'make check-self-compare'. >> | I plan to run this before releases now. >> >> ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during >> standard 'make check'. >> >> On my eight years old Dell Precision M4700, I see: >> >> $ \time make check TESTS=3Druntestslowselfcompare.sh ENABLE_SLOW_TES= T=3Dyes >> >> 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 97046= 8maxresident)k >> 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984= maxresident)k >> 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 97001= 6maxresident)k >> >> So, ~21 s. > > On my machine (AMD FX8350 box) it's really still much longer than that, a= s > you pointed above. And I know users who are in the same case, still. > Yeah, surprising, I know. Your box is more than 10 times faster than > mine, it seems. "Interesting" ;-) -- thanks for confirming your numbers. Per a quick web search, your AMD FX8350 would be just a little older than my Dell Precision M4700 with "Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz", so I wonder where the rather big difference is coming from. Mine has 24 GiB of RAM, and I'm caching the "WDC WD7500BPKT-7" HDD with a "KINGSTON SA400S3" SSD; maybe that's it. >> All the other test cases, running in parallel (just '-j5'): >> >> $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=3Dno' >> >> 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605= 568maxresident)k >> 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605= 720maxresident)k >> 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605= 292maxresident)k >> >> So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': >> >> $ \time make check -j5 ENABLE_SLOW_TEST=3Dyes >> >> 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971= 036maxresident)k >> 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971= 000maxresident)k >> 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970= 356maxresident)k >> >> So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the >> 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. >> >> Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 >> "CONTRIBUTING: Update instructions about regression tests", it was noted >> that... >> >> | This is an important regression test. The >> | problem is that it can takes twice as much time as make distcheck. So >> | we've put it into its own separate target. >> >> Given the "5 minutes" number from above, this comment means that a >> 'make distcheck' (or rather 'make distcheck-fast', I suppose?) > > Well, you are maybe reading too much into that sentence :-) Since that t= ime, > make distcheck grew slower. So I guess the test now takes roughly the > same time as make distcheck or make distcheck-fast. > >> would run >> ~2.5 min? I've got the following numbers: >> >> $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=3Dn= o' >> >> 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986= 144maxresident)k >> 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984= 372maxresident)k >> 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985= 388maxresident)k >> >> So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone >> takes just 1/15 the time of that, not "twice as much". >> >> Additionally enabling 'tests/runtestslowselfcompare.sh' here: >> >> $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=3Dyes >> >> 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984= 760maxresident)k >> 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985= 836maxresident)k >> 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985= 224maxresident)k >> >> ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s >> longer -- acceptable, in my opinion. > > On my machine, your patch "make check" takes 5 minutes longer than > without. So I am reluctant to apply it. Sure, understood, and no worries. > In practise, it's not a big deal, I think, as I run 'make > check-self-compare' very regularly. > > As bizarre as it seems, the tendency I'd like us to move towards is less > tests "by default". That is, split out the binaries that we have in the > tarball today and keep a very minimal set of tests. My goal is just to establish a testing baseline, so that I can be reasonably sure that changes I may be doing don't regress anything. For the time being, I might just locally set 'ENABLE_SLOW_TEST=3Dyes'; and at least we've now got some numbers (confirmed/new) in the archives. > The tarball is too > big as it is, and yet, there are tons of tests that I'd like to run that > are not present there. > > That is why I'd like us to progress towards having much more tests that > are in a separated "test project", somehow. In that project, we'd > either have binaries locally present or references to binaries (like > distros packages over the interweb) to grab and run comparisons on. Yes, that makes sense. (I did wonder about the Git repository/checkout size as well as the fact that huge binary blobs are stored there without sources reference.) > So I would not spent to much time on these tests that are locally > present in the tarball. Rather, if you are interested in this super > important testing strategy project, we could discuss it a bit more in > depth. Heh, not at this time, sorry. ;-) Gr=C3=BC=C3=9Fe Thomas ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstra=C3=9Fe 201= , 80634 M=C3=BCnchen; Gesellschaft mit beschr=C3=A4nkter Haftung; Gesch=C3= =A4ftsf=C3=BChrer: Thomas Heurung, Frank Th=C3=BCrauf; Sitz der Gesellschaf= t: M=C3=BCnchen; Registergericht M=C3=BCnchen, HRB 106955