public inbox for gcc-patches@gcc.gnu.org
 help / color / mirror / Atom feed
* [PATCH] libphobos: Fix instability in the parallelized testsuite
@ 2022-07-14 21:53 Lewis Hyatt
  2022-07-15 22:07 ` Iain Buclaw
  0 siblings, 1 reply; 3+ messages in thread
From: Lewis Hyatt @ 2022-07-14 21:53 UTC (permalink / raw)
  To: gcc-patches

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

Hello-

I get a different number of test results from libphobos.unittest/unittest.exp,
depending on server load. I believe it's because this testsuite doesn't check
runtest_file_p:

$ make -j 1 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#'
 # of expected passes           10

$ make -j 2 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#'
 # of expected passes           10
 # of expected passes           10

$ make -j 4 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#'
 # of expected passes           10
 # of expected passes           10
 # of expected passes           10
 # of expected passes           10

When running in parallel along with other tests, even at a fixed argument
for -j, the number of tests that actually execute will depend on how many of the
parallel sub-makes happened to start prior to the first one finishing, hence
it changes from run to run.

The attached patch fixes it for me, if it looks OK? Thanks, this would remove
some noise from before/after test comparisons.

-Lewis

[-- Attachment #2: libphobos_runtest.txt --]
[-- Type: text/plain, Size: 1020 bytes --]

libphobos: Fix instability in the parallelized testsuite

libphobos.unittest/unittest.exp calls bare dg-test rather than dg-runtest, and
so it should call runtest_file_p to determine whether to run each test or
not. Without that call, the tests run too many times in parallel mode (they will
run as many times, as the argument to make -j).

libphobos/ChangeLog:

	* testsuite/libphobos.unittest/unittest.exp: Call runtest_file_p
	prior to running each test.

diff --git a/libphobos/testsuite/libphobos.unittest/unittest.exp b/libphobos/testsuite/libphobos.unittest/unittest.exp
index 2a019caca8c..175decdc333 100644
--- a/libphobos/testsuite/libphobos.unittest/unittest.exp
+++ b/libphobos/testsuite/libphobos.unittest/unittest.exp
@@ -42,6 +42,9 @@ foreach unit_test $unit_test_list {
     set expected_fail [lindex $unit_test 1]
 
     foreach test $tests {
+	if {![runtest_file_p $runtests $test]} {
+            continue
+        }
         set shouldfail $expected_fail
         dg-test $test "" $test_flags
     }

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-07-17 15:51 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-14 21:53 [PATCH] libphobos: Fix instability in the parallelized testsuite Lewis Hyatt
2022-07-15 22:07 ` Iain Buclaw
2022-07-17 15:51   ` Lewis Hyatt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).