From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout-p-202.mailbox.org (mout-p-202.mailbox.org [IPv6:2001:67c:2050:0:465::202]) by sourceware.org (Postfix) with ESMTPS id B108C3858D32 for ; Fri, 15 Jul 2022 22:07:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B108C3858D32 Received: from smtp102.mailbox.org (smtp102.mailbox.org [10.196.197.102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4Ll57q6jd1z9sWC; Sat, 16 Jul 2022 00:07:35 +0200 (CEST) Date: Sat, 16 Jul 2022 00:07:31 +0200 From: Iain Buclaw Subject: Re: [PATCH] libphobos: Fix instability in the parallelized testsuite To: gcc-patches@gcc.gnu.org, Lewis Hyatt References: <20220714215325.GA18923@ldh-imac.local> In-Reply-To: <20220714215325.GA18923@ldh-imac.local> MIME-Version: 1.0 Message-Id: <1657922526.7xsg40t05n.astroid@pulse.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jul 2022 22:07:42 -0000 Excerpts from Lewis Hyatt via Gcc-patches's message of Juli 14, 2022 11:53 = pm: > Hello- >=20 > I get a different number of test results from libphobos.unittest/unittest= .exp, > depending on server load. I believe it's because this testsuite doesn't c= heck > runtest_file_p: >=20 > $ make -j 1 RUNTESTFLAGS=3D'unittest.exp' check-target-libphobos | grep '= ^#' > # of expected passes 10 >=20 > $ make -j 2 RUNTESTFLAGS=3D'unittest.exp' check-target-libphobos | grep '= ^#' > # of expected passes 10 > # of expected passes 10 >=20 > $ make -j 4 RUNTESTFLAGS=3D'unittest.exp' check-target-libphobos | grep '= ^#' > # of expected passes 10 > # of expected passes 10 > # of expected passes 10 > # of expected passes 10 >=20 > When running in parallel along with other tests, even at a fixed argument > for -j, the number of tests that actually execute will depend on how many= of the > parallel sub-makes happened to start prior to the first one finishing, he= nce > it changes from run to run. >=20 > The attached patch fixes it for me, if it looks OK? Thanks, this would re= move > some noise from before/after test comparisons. >=20 > -Lewis > libphobos: Fix instability in the parallelized testsuite >=20 > libphobos.unittest/unittest.exp calls bare dg-test rather than dg-runtest= , and > so it should call runtest_file_p to determine whether to run each test or > not. Without that call, the tests run too many times in parallel mode (th= ey will > run as many times, as the argument to make -j). Hi Lewis, Thanks! Good spot. I think it should be calling dg-runtest however, same as what libphobos.cycles/cycles.exp is doing. Could also fix the test name so each one is unique, just to hit two birds in one - something like the following would suffice (haven't had time to check). Kind Regards, Iain. --- --- a/libphobos/testsuite/libphobos.unittest/unittest.exp +++ b/libphobos/testsuite/libphobos.unittest/unittest.exp @@ -42,8 +42,10 @@ foreach unit_test $unit_test_list { set expected_fail [lindex $unit_test 1] =20 foreach test $tests { - set shouldfail $expected_fail - dg-test $test "" $test_flags + set libphobos_test_name "[dg-trim-dirname $srcdir $test] $test_flags" + set shouldfail $expected_fail + dg-runtest $test "" $test_flags + set libphobos_test_name "" } =20 set shouldfail 0