Re: [PATCH] libphobos: Fix instability in the parallelized testsuite

[Iain Buclaw <ibuclaw@gdcproject.org>]

Excerpts from Lewis Hyatt via Gcc-patches's message of Juli 14, 2022 11:53 pm:
> Hello-
> 
> I get a different number of test results from libphobos.unittest/unittest.exp,
> depending on server load. I believe it's because this testsuite doesn't check
> runtest_file_p:
> 
> $ make -j 1 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#'
>  # of expected passes           10
> 
> $ make -j 2 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#'
>  # of expected passes           10
>  # of expected passes           10
> 
> $ make -j 4 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#'
>  # of expected passes           10
>  # of expected passes           10
>  # of expected passes           10
>  # of expected passes           10
> 
> When running in parallel along with other tests, even at a fixed argument
> for -j, the number of tests that actually execute will depend on how many of the
> parallel sub-makes happened to start prior to the first one finishing, hence
> it changes from run to run.
> 
> The attached patch fixes it for me, if it looks OK? Thanks, this would remove
> some noise from before/after test comparisons.
> 
> -Lewis
> libphobos: Fix instability in the parallelized testsuite
> 
> libphobos.unittest/unittest.exp calls bare dg-test rather than dg-runtest, and
> so it should call runtest_file_p to determine whether to run each test or
> not. Without that call, the tests run too many times in parallel mode (they will
> run as many times, as the argument to make -j).


Hi Lewis,

Thanks! Good spot. I think it should be calling dg-runtest however,
same as what libphobos.cycles/cycles.exp is doing. Could also fix the
test name so each one is unique, just to hit two birds in one -
something like the following would suffice (haven't had time to check).

Kind Regards,
Iain.

---

--- a/libphobos/testsuite/libphobos.unittest/unittest.exp
+++ b/libphobos/testsuite/libphobos.unittest/unittest.exp
@@ -42,8 +42,10 @@ foreach unit_test $unit_test_list {
     set expected_fail [lindex $unit_test 1]
 
     foreach test $tests {
-        set shouldfail $expected_fail
-        dg-test $test "" $test_flags
+	set libphobos_test_name "[dg-trim-dirname $srcdir $test] $test_flags"
+	set shouldfail $expected_fail
+	dg-runtest $test "" $test_flags
+	set libphobos_test_name ""
     }
 
     set shouldfail 0