From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id DBC773858410 for ; Tue, 4 Jan 2022 18:28:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org DBC773858410 Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-617-msNhxT7oM-eqpMRswpnNxA-1; Tue, 04 Jan 2022 13:28:21 -0500 X-MC-Unique: msNhxT7oM-eqpMRswpnNxA-1 Received: by mail-qk1-f198.google.com with SMTP id y24-20020a37e318000000b0047687b4d282so14197505qki.3 for ; Tue, 04 Jan 2022 10:28:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:organization:references:date :in-reply-to:message-id:user-agent:mime-version :content-transfer-encoding; bh=JcDJKLTczpcQEisO5kKghfiqbm/OStXuQKDkq42Vw4U=; b=0jLlVCN0K/0hyGoWt7P+1kvA3x5W9ndL1JTTS85SmRjlMU+aPoerJyZYu7itIezhOh G50NOjriAGz5xIB7XCZASFis0TJpStqZgBcFepxL6r+6cYZkpP3EEGrJU3ottkvc3NvT yqgM5TIAKjB0QmwO5U4M6Wkaw1I2gDR9VdJPVbCr4d1MOhtVl5mKbFxCBxGnINQ9U1Li HgEAMuQt57kLkWOZPQA917a24yy0RAz5d7uSX11gqtffLygPIVXc6GrqMcLwmqmpvBhE FFXZJ8/oheCRwFzmDdJ7XJrHhGED1gG/eYiNuYNvvbDsgum70kPepBTkcyw4WBiIQbjx kGKA== X-Gm-Message-State: AOAM532sLpfDYf6Q6XyxiUpjT7vMps9hPDCZFBrayEGCJKhwx2314mD5 htgGeSCtVoGfHaDDOjcQLPKnSB2ZjIaz1h1XVSNU41xHdoA/ypTcKfp6rXPOgJtoNx7DLtTGhOE nvmCRT2oT9rYcMn+tKw15 X-Received: by 2002:a05:622a:93:: with SMTP id o19mr44956186qtw.90.1641320900722; Tue, 04 Jan 2022 10:28:20 -0800 (PST) X-Google-Smtp-Source: ABdhPJxjb30uH9f6SdHiTkkmNLTLjq1CRX0U/fa49mbhy3vxY4WxDyHfRJ6nL3m+eAyK5AKB5ErBSQ== X-Received: by 2002:a05:622a:93:: with SMTP id o19mr44956169qtw.90.1641320900414; Tue, 04 Jan 2022 10:28:20 -0800 (PST) Received: from localhost ([88.120.130.27]) by smtp.gmail.com with ESMTPSA id q15sm29865745qkj.108.2022.01.04.10.28.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jan 2022 10:28:20 -0800 (PST) Received: by localhost (Postfix, from userid 1000) id 3E78A581C4F; Tue, 4 Jan 2022 15:53:57 +0100 (CET) From: Dodji Seketeli To: Thomas Schwinge Cc: libabigail@sourceware.org, Dodji Seketeli Subject: Re: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' Organization: Me, myself and I References: <87tuf1ynf3.fsf@euler.schwinge.homeip.net> <20211222104936.3998331-1-thomas@codesourcery.com> X-Operating-System: Fedora 36 X-URL: http://www.seketeli.net/~dodji Date: Tue, 04 Jan 2022 15:53:57 +0100 In-Reply-To: <20211222104936.3998331-1-thomas@codesourcery.com> (Thomas Schwinge's message of "Wed, 22 Dec 2021 11:49:36 +0100") Message-ID: <87a6gbeem2.fsf@seketeli.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-5.2 required=5.0 tests=BAYES_00, DATE_IN_PAST_03_06, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2022 18:28:25 -0000 Hello Thomas, Thomas Schwinge a =C3=A9crit: > Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97 > "Bug 26769 - Fix missing types in abixml output": > > | =09* tests/runtestslowselfcompare.sh.in: New test that compares > | =09libabigail.so against its own ABIXML representation. > > I consider this to be a pretty important test case -- "eat our own dog > food". > > Thus, I find this a bit unfortunate: > > | =09* tests/Makefile.am: Add the new test runtestslowselfcompare.sh to > | =09source distribution. This test is too slow to be run during the > | =09course of 'make check'. It takes more than 5 minutes on my slow > | =09box here. Rather, it can be run using 'make check-self-compare'. > | =09I plan to run this before releases now. > > ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during > standard 'make check'. > > On my eight years old Dell Precision M4700, I see: > > $ \time make check TESTS=3Druntestslowselfcompare.sh ENABLE_SLOW_TEST= =3Dyes > > 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468= maxresident)k > 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984m= axresident)k > 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016= maxresident)k > > So, ~21 s. On my machine (AMD FX8350 box) it's really still much longer than that, as you pointed above. And I know users who are in the same case, still. Yeah, surprising, I know. Your box is more than 10 times faster than mine, it seems. > All the other test cases, running in parallel (just '-j5'): > > $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=3Dno' > > 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 6055= 68maxresident)k > 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 6057= 20maxresident)k > 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 6052= 92maxresident)k > > So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': > > $ \time make check -j5 ENABLE_SLOW_TEST=3Dyes > > 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 9710= 36maxresident)k > 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 9710= 00maxresident)k > 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 9703= 56maxresident)k > > So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the > 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. > > Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 > "CONTRIBUTING: Update instructions about regression tests", it was noted > that... > > | This is an important regression test. The > | problem is that it can takes twice as much time as make distcheck. So > | we've put it into its own separate target. > > Given the "5 minutes" number from above, this comment means that a > 'make distcheck' (or rather 'make distcheck-fast', I suppose?) Well, you are maybe reading too much into that sentence :-) Since that tim= e, make distcheck grew slower. So I guess the test now takes roughly the same time as make distcheck or make distcheck-fast. > would run > ~2.5 min? I've got the following numbers: > > $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=3Dno= ' > > 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 9861= 44maxresident)k > 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 9843= 72maxresident)k > 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 9853= 88maxresident)k > > So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone > takes just 1/15 the time of that, not "twice as much". > > Additionally enabling 'tests/runtestslowselfcompare.sh' here: > > $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=3Dyes > > 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 9847= 60maxresident)k > 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 9858= 36maxresident)k > 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 9852= 24maxresident)k > > ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s > longer -- acceptable, in my opinion. On my machine, your patch "make check" takes 5 minutes longer than without. So I am reluctant to apply it. In practise, it's not a big deal, I think, as I run 'make check-self-compare' very regularly. As bizarre as it seems, the tendency I'd like us to move towards is less tests "by default". That is, split out the binaries that we have in the tarball today and keep a very minimal set of tests. The tarball is too big as it is, and yet, there are tons of tests that I'd like to run that are not present there. That is why I'd like us to progress towards having much more tests that are in a separated "test project", somehow. In that project, we'd either have binaries locally present or references to binaries (like distros packages over the interweb) to grab and run comparisons on. So I would not spent to much time on these tests that are locally present in the tarball. Rather, if you are interested in this super important testing strategy project, we could discuss it a bit more in depth. I hope this makes sense. Thank you for showing interesting in this area. [...] Cheers, --=20 =09=09Dodji