[Bug modula2/114886] New: gm2 testsuite arbitrarily reduces timeouts

[Bug modula2/114886] New: gm2 testsuite arbitrarily reduces timeouts

[ro at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114886

            Bug ID: 114886
           Summary: gm2 testsuite arbitrarily reduces timeouts
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: modula2
          Assignee: gaius at gcc dot gnu.org
          Reporter: ro at gcc dot gnu.org
  Target Milestone: ---

When bootstrapping the gcc-14 branch on sparc-sun-solaris2.11, quite a number
of gm2 tests FAILed due to timeouts.

I had seen this before and could trace this to the gm2 testsuite reducing way
beyond the default of 300 seconds (sometimes as low as 10 seconds).  I think
this is badly misguided for a couple of reasons:

* If a test completes (PASS or FAIL) within the reduced reduced timeout, it
  will do so with the default timeout, too.

* If a test takes either longer than the default timeout (either because
  compilation of execution is particularly slow or because some step loops
  indefinitely), reducing the timeout would give a result earlier.  However,
  I've rarely if ever seen such a case in all my testing.

* If a test takes longer than the reduced timeout, it will fail while it would
  pass just fine with the default timeout.

* Using hardcoded timeouts is always wrong: while the values currenly used
  may work one some particular target, it will cause failures on systems that
  are either slower or under higher load.  Besides, it is possible to globally
  change the derfault in some expect snippet (like

  global board_info
  set board_info(unix,gcc,timeout) 600

  (even differently per multilib if desired).

IMO this whole approach is completely misguided, has very little benefit even
in the best case, and demonstrably introduces failures that wouldn't exist
without
it.

I'm attaching my current hack for reference, but it's completely inappropriate 
as is.  It only identifies some (all?) places that need changing.

[Bug modula2/114886] gm2 testsuite arbitrarily reduces timeouts

[ro at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114886

--- Comment #1 from Rainer Orth <ro at gcc dot gnu.org> ---
Created attachment 58068
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=58068&action=edit
Hack to identify places that reduce timeouts.

[Bug modula2/114886] gm2 testsuite arbitrarily reduces timeouts

[ro at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114886

Rainer Orth <ro at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |https://gcc.gnu.org/piperma
                   |                            |il/gcc-patches/2024-April/6
                   |                            |50222.html
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2024-04-30
   Target Milestone|---                         |14.2
     Ever confirmed|0                           |1

--- Comment #2 from Rainer Orth <ro at gcc dot gnu.org> ---
Proper patch posted.

[Bug modula2/114886] gm2 testsuite arbitrarily reduces timeouts

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114886

--- Comment #3 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Rainer Orth <ro@gcc.gnu.org>:

https://gcc.gnu.org/g:aff63ac11099d100b6891f3bcc3dc6cbc4fad654

commit r15-69-gaff63ac11099d100b6891f3bcc3dc6cbc4fad654
Author: Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE>
Date:   Tue Apr 30 13:49:28 2024 +0200

    testsuite: gm2: Remove timeout overrides [PR114886]

    A large number of gm2 tests are timing out even on current Solaris/SPARC
    systems.  As detailed in the PR, the problem is that the gm2 testsuite
    artificially lowers many timeouts way below the DejaGnu default of 300
    seconds, often as short as 10 seconds.  The problem lies both in the
    values (they may be appropriate for some targets, but too low for
    others, especially under high load) and the fact that it uses absolute
    values, overriding e.g. settings from a build-wide site.exp.

    Therefore this patch removes all those overrides, restoring the
    defaults.

    Tested on sparc-sun-solaris2.11 (where all the previous timeouts are
    gone) and i386-pc-solaris2.11.

    2024-04-29  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>

            gcc/testsuite:
            PR modula2/114886
            * lib/gm2.exp: Don't load timeout-dg.exp.
            Don't set gm2_previous_timeout.
            Don't call dg-timeout.
            (gm2_push_timeout, gm2_pop_timeout): Remove.
            (gm2_init): Don't call dg-timeout.
            * lib/gm2-torture.exp: Don't load timeout-dg.exp.
            Don't set gm2_previous_timeout.
            Don't call dg-timeout.
            (gm2_push_timeout, gm2_pop_timeout): Remove.

            * gm2/coroutines/pim/run/pass/coroutines-pim-run-pass.exp: Don't
            load timeout-dg.exp.
            Don't call gm2_push_timeout, gm2_pop_timeout.
            * gm2/examples/map/pass/examples-map-pass.exp: Don't call
            gm2_push_timeout, gm2_pop_timeout.
            * gm2/iso/run/pass/iso-run-pass.exp: Don't load timeout-dg.exp.
            Don't call gm2_push_timeout, gm2_pop_timeout.
            * gm2/pimlib/base/run/pass/pimlib-base-run-pass.exp: Don't load
            timeout-dg.exp.
            Don't call gm2_push_timeout, gm2_pop_timeout.
            * gm2/projects/iso/run/pass/halma/projects-iso-run-pass-halma.exp:
            Don't call gm2_push_timeout, gm2_pop_timeout.
            *
gm2/switches/whole-program/pass/run/switches-whole-program-pass-run.exp:
            Don't load timeout-dg.exp.
            Don't call gm2_push_timeout, gm2_pop_timeout.

[Bug modula2/114886] gm2 testsuite arbitrarily reduces timeouts

[ro at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114886

Rainer Orth <ro at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|gaius at gcc dot gnu.org           |ro at gcc dot gnu.org

--- Comment #4 from Rainer Orth <ro at gcc dot gnu.org> ---
Fixed for GCC 15.