From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa2.mentor.iphmx.com (esa2.mentor.iphmx.com [68.232.141.98]) by sourceware.org (Postfix) with ESMTPS id 0B7613858C2C for ; Wed, 22 Dec 2021 09:55:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 0B7613858C2C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com IronPort-SDR: 7Zt1ujSkiLEQaIrVVVz8t3abVAUQw7cPFatJNNGul8XGVS1pxmg3/24ck24fVPHLwXO3iVn86z eIUhA43/tWluJZZLNnV6A7eGT+lBxpjTQDKjJM3ssk5b6lR2Zpq9kOA7vJNzeL1reavmhctSV/ UCLxDPVl/LKXBjDA+sjgv06p2utSPI4XFOJ2w7IHcbB/r09K8GFA0AVPZxwgEF7RLN5sBzzIW3 GxHN+oLKkhUkxyrqfDKflT+lUknCPqj4kCNyiFqePLnkQVujkNwvpCY8Kt/uE+2kAIOXKR87vu U81CIXCiMNsYcGHYSKyWQ+Pl X-IronPort-AV: E=Sophos;i="5.88,226,1635235200"; d="scan'208";a="69996289" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa2.mentor.iphmx.com with ESMTP; 22 Dec 2021 01:55:06 -0800 IronPort-SDR: fug+TY8RwDWp8GKX0Tdq6nHtfljbCjpA1/TUxoq6zEzik5Jy8F+Tu1RvWIowCUENVPs5vGB+j+ nuEIto13NOt9qV2eAMFZUqc3Ty1e7kOhm2iazyFk9zeMEOyAz+TMG5cyS7/fXYwkc6Ec9Qqotr 3nBxMC+ewEhTaI/ICFD0NrIkpH0wSyDB+BiFlfoHqXmdJPt0/dv5lpm/h9X41ZTPciNSKmgjUl sbwh+isyJIr0t5KwOGzMJpt+YQw6wvrGyBE+MpqzrzXNo/IAOZL2BO6yzIFZoJfKqJgktPeofa oas= From: Thomas Schwinge To: Dodji Seketeli , Subject: 'tests/runtestslowselfcompare.sh' (was: [PATCH] Bug 26769 - Fix missing types in abixml output) In-Reply-To: <86zh37qc7p.fsf@redhat.com> References: <86zh37qc7p.fsf@redhat.com> User-Agent: Notmuch/0.29.3+94~g74c3f1b (https://notmuchmail.org) Emacs/27.1 (x86_64-pc-linux-gnu) Date: Wed, 22 Dec 2021 10:54:56 +0100 Message-ID: <87tuf1ynf3.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-11.mgc.mentorg.com (139.181.222.11) To svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, LIKELY_SPAM_BODY, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Dec 2021 09:55:10 -0000 Hi! On 2020-11-24T11:30:02+0100, Dodji Seketeli via Libabigail wrote: > * tests/runtestslowselfcompare.sh.in: New test that compares > libabigail.so against its own ABIXML representation. I consider this to be a pretty important test case -- "eat our own dog food". Thus, I find this a bit unfortunate: > * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to > source distribution. This test is too slow to be run during the > course of 'make check'. It takes more than 5 minutes on my slow > box here. Rather, it can be run using 'make check-self-compare'. > I plan to run this before releases now. ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during standard 'make check'. Or may we can change that now? Are you still seeing that "It takes more than 5 minutes on my slow box here"? Because, on my eight years old Dell Precision M4700, I see: $ \time make check TESTS=3Druntestslowselfcompare.sh ENABLE_SLOW_TEST= =3Dyes 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468ma= xresident)k 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984max= resident)k 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016ma= xresident)k So, ~21 s. All the other test cases, running in parallel (just '-j5'): $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=3Dno' 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568= maxresident)k 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720= maxresident)k 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292= maxresident)k So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': $ \time make check -j5 ENABLE_SLOW_TEST=3Dyes 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036= maxresident)k 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000= maxresident)k 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356= maxresident)k So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 "CONTRIBUTING: Update instructions about regression tests", it was noted that... | This is an important regression test. The | problem is that it can takes twice as much time as make distcheck. So | we've put it into its own separate target. Given the "5 minutes" number from above, this comment means that a 'make distcheck' (or rather 'make distcheck-fast', I suppose?) would run ~2.5 min? I've got the following numbers: $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=3Dno' 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144= maxresident)k 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372= maxresident)k 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388= maxresident)k So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone takes just 1/15 the time of that, not "twice as much". Additionally enabling 'tests/runtestslowselfcompare.sh' here: $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=3Dyes 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760= maxresident)k 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836= maxresident)k 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224= maxresident)k ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s longer -- acceptable, in my opinion. Therefore I propose to do away with 'ENABLE_SLOW_TEST', and merge 'make check-self-compare' into standard 'make check'. I'll be happy to prepare a patch, if we arrive at consensus, but first: happy to hear any other opinions/data. Gr=C3=BC=C3=9Fe Thomas > --- a/Makefile.am > +++ b/Makefile.am > @@ -88,3 +88,9 @@ tag-and-all: distcheck > distcheck-fast: > $(MAKE) distcheck GZIP_ENV=3D"--fast" > > +# This makes us compare libabigail.so against its own ABIXML > +# representation. It's super slow (more than 5 minutes on one of my > +# old boxes) so I do this only before releases. > +check-self-compare: > + $(MAKE) -C tests check-self-compare > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > [...] > +check-self-compare: > + $(MAKE) check TESTS=3Druntestslowselfcompare.sh ENABLE_SLOW_TEST=3D= yes > --- /dev/null > +++ b/tests/runtestslowselfcompare.sh.in > @@ -0,0 +1,13 @@ > +#!/bin/sh -x > + > +# This program launches "abidw --abidiff src/.libs/libabigail.so" > +# This compares libabigail.so against its own ABIXML output. > +abidw=3D@top_builddir@/tools/abidw > +objdir=3D@top_builddir@/src/.libs > + > +echo "ENABLE_SLOW_TEST=3D"$ENABLE_SLOW_TEST > + > +if test x$ENABLE_SLOW_TEST !=3D x; then > + `$abidw --abidiff $objdir/libabigail.so` > +fi ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstra=C3=9Fe 201= , 80634 M=C3=BCnchen; Gesellschaft mit beschr=C3=A4nkter Haftung; Gesch=C3= =A4ftsf=C3=BChrer: Thomas Heurung, Frank Th=C3=BCrauf; Sitz der Gesellschaf= t: M=C3=BCnchen; Registergericht M=C3=BCnchen, HRB 106955