Re: [RFC] Decrease default timeout for libstdc++ tests to 6 minutes

[Christophe Lyon <christophe.lyon@linaro.org>]

On Mon, 30 Nov 2020 at 15:58, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On 27/11/20 21:17 +0100, Christophe Lyon via Libstdc++ wrote:
> >On Fri, 27 Nov 2020 at 17:13, Jonathan Wakely via Gcc-patches
> ><gcc-patches@gcc.gnu.org> wrote:
> >>
> >> The default for the GCC testsuite is 300, i.e. 5 minutes, which is the
> >> same as the DejaGnu default.
> >>
> >> Libstdc++ overrides this to 600, i.e. 10 minutes.
> >>
> >> This seems ridiculously long. If any test takes that long on modern
> >> hardware, something is wrong. We've seen this a lot recently with
> >> buggy tests, and waiting for them to FAIL is tedious.
> >>
> >> I've already made libstdc++.exp respect the user's setting in
> >> ~/.dejagnurc or the global site.exp file. This means anybody testing
> >> on slow simulators or old hardware can choose their own timeout.
> >>
> >> I've added dg-timeout-factor to the slowest std::regex tests and have
> >> a patch to do it for the PSTL tests, which both take far too long to
> >> compile. That means you can choose a sensible timeout appropriate for
> >> most tests (e.g. 60 seconds) and not get spurious failures from the
> >> few dozen tests which are just very slow.
> >>
> >> I'd like to change the default to 6 minutes. If that goes well, I'd
> >> like to lower it even further.
> >>
> >> The main benefit of this will be that buggy tests which hang will get
> >> killed sooner, so we waste less time waiting for the inevitable
> >> timeout.
> >>
> >
> >I think that's a good idea, I did have problems sometimes when
> >many tests timed out, causing the whole 'make check' to be
> >killed before completion by our compute farm management system.
>
> Thanks for the feedback. I've pushed this patch now.
>
> It's been tested on powercp64le-linux, x86_64-linux, aarch64-linux,
> sparc-solaris and powerpc-aix. They were all fine with much lower
> defaults (e.g. 120 seconds). Let's see how this goes for people
> testing on older or less powerful hardware.
>

FTR, I've seen two occurrences of a random timeout:
WARNING: program timed out.
27_io/basic_istream/get/wchar_t/lwg3464.cc execution test (reason: NONE)
FAIL: 27_io/basic_istream/get/wchar_t/lwg3464.cc execution test

when testing for aarch64-none-elf with -mabi=ilp32 using Arm's
Foundation Model as simulator (an old release).

Christophe