Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes

[Jonathan Wakely <jwakely@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote:
>On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches
><gcc-patches@gcc.gnu.org> wrote:
>>
>> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the
>> same as the DejaGnu default.
>>
>> Libstdc++ overrides this to 600, i.e. 10 minutes.
>>
>> This seems ridiculously long. If any test takes that long on modern
>> hardware, something is wrong. We've seen this a lot recently with
>> buggy tests, and waiting for them to FAIL is tedious.
>>
>> I've already made libstdc++.exp respect the user's setting in
>> ~/.dejagnurc or the global site.exp file. This means anybody testing
>> on slow simulators or old hardware can choose their own timeout.
>>
>> I've added dg-timeout-factor to the slowest std::regex tests and have
>> a patch to do it for the PSTL tests, which both take far too long to
>> compile. That means you can choose a sensible timeout appropriate for
>> most tests (e.g. 60 seconds) and not get spurious failures from the
>> few dozen tests which are just very slow.
>>
>> I'd like to change the default to 6 minutes. If that goes well, I'd
>> like to lower it even further.
>>
>> The main benefit of this will be that buggy tests which hang will get
>> killed sooner, so we waste less time waiting for the inevitable
>> timeout.
>>
>
>I think that's a good idea, I did have problems sometimes when
>many tests timed out, causing the whole 'make check' to be
>killed before completion by our compute farm management system.

Thanks for the feedback. I've pushed this patch now.

It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux,
sparc-solaris and powerpc-aix. They were all fine with much lower
defaults (e.g. 120 seconds). Let's see how this goes for people
testing on older or less powerful hardware.



[-- Attachment #2: patch.txt --]
[-- Type: text/x-patch, Size: 1333 bytes --]

commit 637800c7bbb222a629076a5d20afda03c2dd1043
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Mon Nov 30 14:39:54 2020

    libstdc++: Reduce default test timeout to 360 seconds
    
    The current default of 10 minutes is much longer than most tests need on
    common hardware. The slow tests all now have a dg-timeout-factor
    directive that gives them more time to run relative to the default. The
    default can also be overridden in ~/.dejagnurc or DEJAGNU=site.exp, so
    it seems unnecessary to have such a large default.
    
    This reduces the default from 10 minutes to 6 minutes, which still seems
    more than enough.
    
    libstdc++-v3/ChangeLog:
    
            * testsuite/lib/libstdc++.exp (libstdc++_init): Reduce
            default tool_timeout to 360.

diff --git a/libstdc++-v3/testsuite/lib/libstdc++.exp b/libstdc++-v3/testsuite/lib/libstdc++.exp
index e000dba968f7..35817a8870a4 100644
--- a/libstdc++-v3/testsuite/lib/libstdc++.exp
+++ b/libstdc++-v3/testsuite/lib/libstdc++.exp
@@ -248,7 +248,7 @@ proc libstdc++_init { testfile } {
     # Set the default timeout for v3 tests.
     # You can override this in ~/.dejagnurc or a .exp file named by $DEJAGNU.
     if {![info exists tool_timeout]} {
-      set tool_timeout 600
+      set tool_timeout 360
     }
 
     # Default settings.