Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes

[Jonathan Wakely <jwakely@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2824 bytes --]

On 01/12/20 10:52 +0000, Jonathan Wakely wrote:
>On 01/12/20 10:45 +0100, Christophe Lyon wrote:
>>On Mon, 30 Nov 2020 at 15:58, Jonathan Wakely <jwakely@redhat.com> wrote:
>>>
>>>On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote:
>>>>On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches
>>>><gcc-patches@gcc.gnu.org> wrote:
>>>>>
>>>>> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the
>>>>> same as the DejaGnu default.
>>>>>
>>>>> Libstdc++ overrides this to 600, i.e. 10 minutes.
>>>>>
>>>>> This seems ridiculously long. If any test takes that long on modern
>>>>> hardware, something is wrong. We've seen this a lot recently with
>>>>> buggy tests, and waiting for them to FAIL is tedious.
>>>>>
>>>>> I've already made libstdc++.exp respect the user's setting in
>>>>> ~/.dejagnurc or the global site.exp file. This means anybody testing
>>>>> on slow simulators or old hardware can choose their own timeout.
>>>>>
>>>>> I've added dg-timeout-factor to the slowest std::regex tests and have
>>>>> a patch to do it for the PSTL tests, which both take far too long to
>>>>> compile. That means you can choose a sensible timeout appropriate for
>>>>> most tests (e.g. 60 seconds) and not get spurious failures from the
>>>>> few dozen tests which are just very slow.
>>>>>
>>>>> I'd like to change the default to 6 minutes. If that goes well, I'd
>>>>> like to lower it even further.
>>>>>
>>>>> The main benefit of this will be that buggy tests which hang will get
>>>>> killed sooner, so we waste less time waiting for the inevitable
>>>>> timeout.
>>>>>
>>>>
>>>>I think that's a good idea, I did have problems sometimes when
>>>>many tests timed out, causing the whole 'make check' to be
>>>>killed before completion by our compute farm management system.
>>>
>>>Thanks for the feedback. I've pushed this patch now.
>>>
>>>It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux,
>>>sparc-solaris and powerpc-aix. They were all fine with much lower
>>>defaults (e.g. 120 seconds). Let's see how this goes for people
>>>testing on older or less powerful hardware.
>>>
>>
>>FTR, I've seen two occurrences of a random timeout:
>>WARNING: program timed out.
>>27_io/basic_istream/get/wchar_t/lwg3464.cc execution test (reason: NONE)
>>FAIL: 27_io/basic_istream/get/wchar_t/lwg3464.cc execution test
>>
>>when testing for aarch64-none-elf with -mabi=ilp32 using Arm's
>>Foundation Model as simulator (an old release).
>
>Yes, that test only runs for target { ! lp64 } and does quite a lot of
>work. It should compile quite quickly, but takes a long time to run
>compared to most tests.
>
>I've add dg-timeout-factor to it and its narrow char counterpart.

Similarly for a couple more tests which do extra work on 32-bit
targets.

Tested x86_64-linux, pushed to trunk.


[-- Attachment #2: patch.txt --]
[-- Type: text/x-patch, Size: 1690 bytes --]

commit 8b2c3b5af3da8d06a59276721263c7f326ed64fd
Author: Jonathan Wakely <jwakely@redhat.com>
Date:   Wed Dec 2 12:34:20 2020

    libstdc++: Use longer timeout for istream::gcount() overflow tests
    
    On targets with 32-bit poitners these tests do extra work, so give them
    longer to run.
    
    libstdc++-v3/ChangeLog:
    
            * testsuite/27_io/basic_istream/ignore/char/94749.cc: Add
            dg-timeout-factor for ilp32 targets.
            * testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc:
            Likewise.

diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc
index 21097c2bff1..63b652dc77b 100644
--- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc
+++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc
@@ -17,6 +17,7 @@
 
 // { dg-do run }
 // { dg-options "-DSIMULATOR_TEST" { target simulator } }
+// { dg-timeout-factor 2 { target ilp32 } }
 
 // PR libstdc++/94749
 // basic_istream::ignore(n, c) discards n+1 if next character is equal to c.
diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc
index 2473588d307..dcf0fee4906 100644
--- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc
+++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc
@@ -17,6 +17,7 @@
 
 // { dg-do run }
 // { dg-options "-DSIMULATOR_TEST" { target simulator } }
+// { dg-timeout-factor 2 { target ilp32 } }
 
 // PR libstdc++/94749
 // basic_istream::ignore(n, c) discards n+1 if next character is equal to c.