* [RFC] Decrease default timeout for libstdc++ tests to 6 minutes
@ 2020-11-27 16:13 Jonathan Wakely
2020-11-27 20:17 ` Christophe Lyon
0 siblings, 1 reply; 6+ messages in thread
From: Jonathan Wakely @ 2020-11-27 16:13 UTC (permalink / raw)
To: libstdc++, gcc-patches
The default for the GCC testsuite is 300, i.e. 5 minutes, which is the
same as the DejaGnu default.
Libstdc++ overrides this to 600, i.e. 10 minutes.
This seems ridiculously long. If any test takes that long on modern
hardware, something is wrong. We've seen this a lot recently with
buggy tests, and waiting for them to FAIL is tedious.
I've already made libstdc++.exp respect the user's setting in
~/.dejagnurc or the global site.exp file. This means anybody testing
on slow simulators or old hardware can choose their own timeout.
I've added dg-timeout-factor to the slowest std::regex tests and have
a patch to do it for the PSTL tests, which both take far too long to
compile. That means you can choose a sensible timeout appropriate for
most tests (e.g. 60 seconds) and not get spurious failures from the
few dozen tests which are just very slow.
I'd like to change the default to 6 minutes. If that goes well, I'd
like to lower it even further.
The main benefit of this will be that buggy tests which hang will get
killed sooner, so we waste less time waiting for the inevitable
timeout.
Here's the proposed patch for completeness:
--- a/libstdc++-v3/testsuite/lib/libstdc++.exp
+++ b/libstdc++-v3/testsuite/lib/libstdc++.exp
@@ -248,7 +248,7 @@ proc libstdc++_init { testfile } {
# Set the default timeout for v3 tests.
# You can override this in ~/.dejagnurc or a .exp file named by $DEJAGNU.
if {![info exists tool_timeout]} {
- set tool_timeout 600
+ set tool_timeout 360
}
# Default settings.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes 2020-11-27 16:13 [RFC] Decrease default timeout for libstdc++ tests to 6 minutes Jonathan Wakely @ 2020-11-27 20:17 ` Christophe Lyon 2020-11-30 14:58 ` Jonathan Wakely 0 siblings, 1 reply; 6+ messages in thread From: Christophe Lyon @ 2020-11-27 20:17 UTC (permalink / raw) To: Jonathan Wakely; +Cc: libstdc++, gcc Patches On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches <gcc-patches@gcc.gnu.org> wrote: > > The default for the GCC testsuite is 300, i.e. 5 minutes, which is the > same as the DejaGnu default. > > Libstdc++ overrides this to 600, i.e. 10 minutes. > > This seems ridiculously long. If any test takes that long on modern > hardware, something is wrong. We've seen this a lot recently with > buggy tests, and waiting for them to FAIL is tedious. > > I've already made libstdc++.exp respect the user's setting in > ~/.dejagnurc or the global site.exp file. This means anybody testing > on slow simulators or old hardware can choose their own timeout. > > I've added dg-timeout-factor to the slowest std::regex tests and have > a patch to do it for the PSTL tests, which both take far too long to > compile. That means you can choose a sensible timeout appropriate for > most tests (e.g. 60 seconds) and not get spurious failures from the > few dozen tests which are just very slow. > > I'd like to change the default to 6 minutes. If that goes well, I'd > like to lower it even further. > > The main benefit of this will be that buggy tests which hang will get > killed sooner, so we waste less time waiting for the inevitable > timeout. > I think that's a good idea, I did have problems sometimes when many tests timed out, causing the whole 'make check' to be killed before completion by our compute farm management system. Thanks Christophe > > Here's the proposed patch for completeness: > > --- a/libstdc++-v3/testsuite/lib/libstdc++.exp > +++ b/libstdc++-v3/testsuite/lib/libstdc++.exp > @@ -248,7 +248,7 @@ proc libstdc++_init { testfile } { > # Set the default timeout for v3 tests. > # You can override this in ~/.dejagnurc or a .exp file named by $DEJAGNU. > if {![info exists tool_timeout]} { > - set tool_timeout 600 > + set tool_timeout 360 > } > > # Default settings. > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes 2020-11-27 20:17 ` Christophe Lyon @ 2020-11-30 14:58 ` Jonathan Wakely 2020-12-01 9:45 ` Christophe Lyon 0 siblings, 1 reply; 6+ messages in thread From: Jonathan Wakely @ 2020-11-30 14:58 UTC (permalink / raw) To: Christophe Lyon; +Cc: libstdc++, gcc Patches [-- Attachment #1: Type: text/plain, Size: 1825 bytes --] On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote: >On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches ><gcc-patches@gcc.gnu.org> wrote: >> >> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the >> same as the DejaGnu default. >> >> Libstdc++ overrides this to 600, i.e. 10 minutes. >> >> This seems ridiculously long. If any test takes that long on modern >> hardware, something is wrong. We've seen this a lot recently with >> buggy tests, and waiting for them to FAIL is tedious. >> >> I've already made libstdc++.exp respect the user's setting in >> ~/.dejagnurc or the global site.exp file. This means anybody testing >> on slow simulators or old hardware can choose their own timeout. >> >> I've added dg-timeout-factor to the slowest std::regex tests and have >> a patch to do it for the PSTL tests, which both take far too long to >> compile. That means you can choose a sensible timeout appropriate for >> most tests (e.g. 60 seconds) and not get spurious failures from the >> few dozen tests which are just very slow. >> >> I'd like to change the default to 6 minutes. If that goes well, I'd >> like to lower it even further. >> >> The main benefit of this will be that buggy tests which hang will get >> killed sooner, so we waste less time waiting for the inevitable >> timeout. >> > >I think that's a good idea, I did have problems sometimes when >many tests timed out, causing the whole 'make check' to be >killed before completion by our compute farm management system. Thanks for the feedback. I've pushed this patch now. It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux, sparc-solaris and powerpc-aix. They were all fine with much lower defaults (e.g. 120 seconds). Let's see how this goes for people testing on older or less powerful hardware. [-- Attachment #2: patch.txt --] [-- Type: text/x-patch, Size: 1333 bytes --] commit 637800c7bbb222a629076a5d20afda03c2dd1043 Author: Jonathan Wakely <jwakely@redhat.com> Date: Mon Nov 30 14:39:54 2020 libstdc++: Reduce default test timeout to 360 seconds The current default of 10 minutes is much longer than most tests need on common hardware. The slow tests all now have a dg-timeout-factor directive that gives them more time to run relative to the default. The default can also be overridden in ~/.dejagnurc or DEJAGNU=site.exp, so it seems unnecessary to have such a large default. This reduces the default from 10 minutes to 6 minutes, which still seems more than enough. libstdc++-v3/ChangeLog: * testsuite/lib/libstdc++.exp (libstdc++_init): Reduce default tool_timeout to 360. diff --git a/libstdc++-v3/testsuite/lib/libstdc++.exp b/libstdc++-v3/testsuite/lib/libstdc++.exp index e000dba968f7..35817a8870a4 100644 --- a/libstdc++-v3/testsuite/lib/libstdc++.exp +++ b/libstdc++-v3/testsuite/lib/libstdc++.exp @@ -248,7 +248,7 @@ proc libstdc++_init { testfile } { # Set the default timeout for v3 tests. # You can override this in ~/.dejagnurc or a .exp file named by $DEJAGNU. if {![info exists tool_timeout]} { - set tool_timeout 600 + set tool_timeout 360 } # Default settings. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes 2020-11-30 14:58 ` Jonathan Wakely @ 2020-12-01 9:45 ` Christophe Lyon 2020-12-01 10:52 ` Jonathan Wakely 0 siblings, 1 reply; 6+ messages in thread From: Christophe Lyon @ 2020-12-01 9:45 UTC (permalink / raw) To: Jonathan Wakely; +Cc: libstdc++, gcc Patches On Mon, 30 Nov 2020 at 15:58, Jonathan Wakely <jwakely@redhat.com> wrote: > > On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote: > >On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches > ><gcc-patches@gcc.gnu.org> wrote: > >> > >> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the > >> same as the DejaGnu default. > >> > >> Libstdc++ overrides this to 600, i.e. 10 minutes. > >> > >> This seems ridiculously long. If any test takes that long on modern > >> hardware, something is wrong. We've seen this a lot recently with > >> buggy tests, and waiting for them to FAIL is tedious. > >> > >> I've already made libstdc++.exp respect the user's setting in > >> ~/.dejagnurc or the global site.exp file. This means anybody testing > >> on slow simulators or old hardware can choose their own timeout. > >> > >> I've added dg-timeout-factor to the slowest std::regex tests and have > >> a patch to do it for the PSTL tests, which both take far too long to > >> compile. That means you can choose a sensible timeout appropriate for > >> most tests (e.g. 60 seconds) and not get spurious failures from the > >> few dozen tests which are just very slow. > >> > >> I'd like to change the default to 6 minutes. If that goes well, I'd > >> like to lower it even further. > >> > >> The main benefit of this will be that buggy tests which hang will get > >> killed sooner, so we waste less time waiting for the inevitable > >> timeout. > >> > > > >I think that's a good idea, I did have problems sometimes when > >many tests timed out, causing the whole 'make check' to be > >killed before completion by our compute farm management system. > > Thanks for the feedback. I've pushed this patch now. > > It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux, > sparc-solaris and powerpc-aix. They were all fine with much lower > defaults (e.g. 120 seconds). Let's see how this goes for people > testing on older or less powerful hardware. > FTR, I've seen two occurrences of a random timeout: WARNING: program timed out. 27_io/basic_istream/get/wchar_t/lwg3464.cc execution test (reason: NONE) FAIL: 27_io/basic_istream/get/wchar_t/lwg3464.cc execution test when testing for aarch64-none-elf with -mabi=ilp32 using Arm's Foundation Model as simulator (an old release). Christophe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes 2020-12-01 9:45 ` Christophe Lyon @ 2020-12-01 10:52 ` Jonathan Wakely 2020-12-02 12:35 ` Jonathan Wakely 0 siblings, 1 reply; 6+ messages in thread From: Jonathan Wakely @ 2020-12-01 10:52 UTC (permalink / raw) To: Christophe Lyon; +Cc: libstdc++, gcc Patches [-- Attachment #1: Type: text/plain, Size: 2684 bytes --] On 01/12/20 10:45 +0100, Christophe Lyon wrote: >On Mon, 30 Nov 2020 at 15:58, Jonathan Wakely <jwakely@redhat.com> wrote: >> >> On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote: >> >On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches >> ><gcc-patches@gcc.gnu.org> wrote: >> >> >> >> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the >> >> same as the DejaGnu default. >> >> >> >> Libstdc++ overrides this to 600, i.e. 10 minutes. >> >> >> >> This seems ridiculously long. If any test takes that long on modern >> >> hardware, something is wrong. We've seen this a lot recently with >> >> buggy tests, and waiting for them to FAIL is tedious. >> >> >> >> I've already made libstdc++.exp respect the user's setting in >> >> ~/.dejagnurc or the global site.exp file. This means anybody testing >> >> on slow simulators or old hardware can choose their own timeout. >> >> >> >> I've added dg-timeout-factor to the slowest std::regex tests and have >> >> a patch to do it for the PSTL tests, which both take far too long to >> >> compile. That means you can choose a sensible timeout appropriate for >> >> most tests (e.g. 60 seconds) and not get spurious failures from the >> >> few dozen tests which are just very slow. >> >> >> >> I'd like to change the default to 6 minutes. If that goes well, I'd >> >> like to lower it even further. >> >> >> >> The main benefit of this will be that buggy tests which hang will get >> >> killed sooner, so we waste less time waiting for the inevitable >> >> timeout. >> >> >> > >> >I think that's a good idea, I did have problems sometimes when >> >many tests timed out, causing the whole 'make check' to be >> >killed before completion by our compute farm management system. >> >> Thanks for the feedback. I've pushed this patch now. >> >> It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux, >> sparc-solaris and powerpc-aix. They were all fine with much lower >> defaults (e.g. 120 seconds). Let's see how this goes for people >> testing on older or less powerful hardware. >> > >FTR, I've seen two occurrences of a random timeout: >WARNING: program timed out. >27_io/basic_istream/get/wchar_t/lwg3464.cc execution test (reason: NONE) >FAIL: 27_io/basic_istream/get/wchar_t/lwg3464.cc execution test > >when testing for aarch64-none-elf with -mabi=ilp32 using Arm's >Foundation Model as simulator (an old release). Yes, that test only runs for target { ! lp64 } and does quite a lot of work. It should compile quite quickly, but takes a long time to run compared to most tests. I've add dg-timeout-factor to it and its narrow char counterpart. Tested x86_64-linux, pushed to trunk. [-- Attachment #2: patch.txt --] [-- Type: text/x-patch, Size: 1404 bytes --] commit 0fb378761f2c5eb906bf0a9698112dd65d4dc73b Author: Jonathan Wakely <jwakely@redhat.com> Date: Tue Dec 1 10:51:25 2020 libstdc++: Use longer timeout for slow running tests libstdc++-v3/ChangeLog: * testsuite/27_io/basic_istream/get/char/lwg3464.cc: Add dg-timeout-factor directive. * testsuite/27_io/basic_istream/get/wchar_t/lwg3464.cc: Likewise. diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/get/char/lwg3464.cc b/libstdc++-v3/testsuite/27_io/basic_istream/get/char/lwg3464.cc index 6123ca5b713..85d7fc0ebd0 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/get/char/lwg3464.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/get/char/lwg3464.cc @@ -16,6 +16,7 @@ // <http://www.gnu.org/licenses/>. // { dg-do run { target { ! lp64 } } } +// { dg-timeout-factor 2 } #include <istream> #include <streambuf> diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/get/wchar_t/lwg3464.cc b/libstdc++-v3/testsuite/27_io/basic_istream/get/wchar_t/lwg3464.cc index 6df244cc32d..1d8411c267a 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/get/wchar_t/lwg3464.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/get/wchar_t/lwg3464.cc @@ -16,6 +16,7 @@ // <http://www.gnu.org/licenses/>. // { dg-do run { target { ! lp64 } } } +// { dg-timeout-factor 2 } #include <istream> #include <streambuf> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes 2020-12-01 10:52 ` Jonathan Wakely @ 2020-12-02 12:35 ` Jonathan Wakely 0 siblings, 0 replies; 6+ messages in thread From: Jonathan Wakely @ 2020-12-02 12:35 UTC (permalink / raw) To: Christophe Lyon; +Cc: libstdc++, gcc Patches [-- Attachment #1: Type: text/plain, Size: 2824 bytes --] On 01/12/20 10:52 +0000, Jonathan Wakely wrote: >On 01/12/20 10:45 +0100, Christophe Lyon wrote: >>On Mon, 30 Nov 2020 at 15:58, Jonathan Wakely <jwakely@redhat.com> wrote: >>> >>>On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote: >>>>On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches >>>><gcc-patches@gcc.gnu.org> wrote: >>>>> >>>>> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the >>>>> same as the DejaGnu default. >>>>> >>>>> Libstdc++ overrides this to 600, i.e. 10 minutes. >>>>> >>>>> This seems ridiculously long. If any test takes that long on modern >>>>> hardware, something is wrong. We've seen this a lot recently with >>>>> buggy tests, and waiting for them to FAIL is tedious. >>>>> >>>>> I've already made libstdc++.exp respect the user's setting in >>>>> ~/.dejagnurc or the global site.exp file. This means anybody testing >>>>> on slow simulators or old hardware can choose their own timeout. >>>>> >>>>> I've added dg-timeout-factor to the slowest std::regex tests and have >>>>> a patch to do it for the PSTL tests, which both take far too long to >>>>> compile. That means you can choose a sensible timeout appropriate for >>>>> most tests (e.g. 60 seconds) and not get spurious failures from the >>>>> few dozen tests which are just very slow. >>>>> >>>>> I'd like to change the default to 6 minutes. If that goes well, I'd >>>>> like to lower it even further. >>>>> >>>>> The main benefit of this will be that buggy tests which hang will get >>>>> killed sooner, so we waste less time waiting for the inevitable >>>>> timeout. >>>>> >>>> >>>>I think that's a good idea, I did have problems sometimes when >>>>many tests timed out, causing the whole 'make check' to be >>>>killed before completion by our compute farm management system. >>> >>>Thanks for the feedback. I've pushed this patch now. >>> >>>It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux, >>>sparc-solaris and powerpc-aix. They were all fine with much lower >>>defaults (e.g. 120 seconds). Let's see how this goes for people >>>testing on older or less powerful hardware. >>> >> >>FTR, I've seen two occurrences of a random timeout: >>WARNING: program timed out. >>27_io/basic_istream/get/wchar_t/lwg3464.cc execution test (reason: NONE) >>FAIL: 27_io/basic_istream/get/wchar_t/lwg3464.cc execution test >> >>when testing for aarch64-none-elf with -mabi=ilp32 using Arm's >>Foundation Model as simulator (an old release). > >Yes, that test only runs for target { ! lp64 } and does quite a lot of >work. It should compile quite quickly, but takes a long time to run >compared to most tests. > >I've add dg-timeout-factor to it and its narrow char counterpart. Similarly for a couple more tests which do extra work on 32-bit targets. Tested x86_64-linux, pushed to trunk. [-- Attachment #2: patch.txt --] [-- Type: text/x-patch, Size: 1690 bytes --] commit 8b2c3b5af3da8d06a59276721263c7f326ed64fd Author: Jonathan Wakely <jwakely@redhat.com> Date: Wed Dec 2 12:34:20 2020 libstdc++: Use longer timeout for istream::gcount() overflow tests On targets with 32-bit poitners these tests do extra work, so give them longer to run. libstdc++-v3/ChangeLog: * testsuite/27_io/basic_istream/ignore/char/94749.cc: Add dg-timeout-factor for ilp32 targets. * testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc: Likewise. diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc index 21097c2bff1..63b652dc77b 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc @@ -17,6 +17,7 @@ // { dg-do run } // { dg-options "-DSIMULATOR_TEST" { target simulator } } +// { dg-timeout-factor 2 { target ilp32 } } // PR libstdc++/94749 // basic_istream::ignore(n, c) discards n+1 if next character is equal to c. diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc index 2473588d307..dcf0fee4906 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc @@ -17,6 +17,7 @@ // { dg-do run } // { dg-options "-DSIMULATOR_TEST" { target simulator } } +// { dg-timeout-factor 2 { target ilp32 } } // PR libstdc++/94749 // basic_istream::ignore(n, c) discards n+1 if next character is equal to c. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-12-02 12:35 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-11-27 16:13 [RFC] Decrease default timeout for libstdc++ tests to 6 minutes Jonathan Wakely 2020-11-27 20:17 ` Christophe Lyon 2020-11-30 14:58 ` Jonathan Wakely 2020-12-01 9:45 ` Christophe Lyon 2020-12-01 10:52 ` Jonathan Wakely 2020-12-02 12:35 ` Jonathan Wakely
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).