[Bug libstdc++/115433] New: unexpected increase of testsuite execution time

[Bug libstdc++/115433] New: unexpected increase of testsuite execution time

[jbeulich at suse dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115433

            Bug ID: 115433
           Summary: unexpected increase of testsuite execution time
           Product: gcc
           Version: 14.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libstdc++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: jbeulich at suse dot com
  Target Milestone: ---

With the usual, moderate increase of the number of tests it is somewhat
unexpected that a testsuite run now takes 50% more time for x86-64 targets, and
about 100% more for i686 targets (and there already with a 64-bit kernel
underneath). For comparison, taking a representative 64-bit system, testsuite
runs took
- gcc: a little over an hour
- g++: less than 40 min
- libstdc++: 3 hours
IOW what already took longest before has now further increased by a significant
percentage. (Absolute numbers are bigger on two or three generations older
hardware, but the relative values remain largely the same.)

All of this said - I certainly understand that moderate increases of overall
execution time is always to be expected for a new major release. It's just that
this time it's not quite "moderate" anymore.

[Bug libstdc++/115433] unexpected increase of testsuite execution time

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115433

--- Comment #1 from Jonathan Wakely <redi at gcc dot gnu.org> ---
What's the baseline for comparisons, the 13.x releases?

We could make this change for release branches, so that C++20 and C++23 tests
are not also run for C++26:

--- a/libstdc++-v3/testsuite/lib/libstdc++.exp
+++ b/libstdc++-v3/testsuite/lib/libstdc++.exp
@@ -536,10 +536,6 @@ proc v3-dg-runtest { testcases flags default-extra-flags }
{
                set min_std [v3-minimum-std $test $default_std $max_std]
                if { $min_std > $default_std } {
                    set std_list $min_std
-                   if { $min_std != $max_std } {
-                       # Also test the latest version.
-                       lappend std_list "$max_std"
-                   }
                } else {
                    # Only run each test once with the default -std option.
                    # This avoids increasing the run time for most testers.


That could make a difference, because many of the C++20 tests are very slow to
compile (std::ranges, std::format, std::chrono::parse are all slow to compile).

Another possible culprit is that you now have tbb-devel installed on the test
machine, which causes the PSTL tests to be run instead of skipped. Some of
those are slow too.

[Bug libstdc++/115433] unexpected increase of testsuite execution time

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115433

--- Comment #2 from Jonathan Wakely <redi at gcc dot gnu.org> ---
I haven't noticed this myself because my ~/.dejagnurc contains:

if { [info exists v3-use-std-list] } {
  # v3-dg-runtest supports running tests with multiple -std options.
  set v3_std_list { 98 11 14 17 20 23 26 }
  set target_list {
"unix{-D_GLIBCXX_ASSERTIONS,-D_GLIBCXX_USE_CXX11_ABI=0,-m32}" }
} else {
  set target_list {
"unix{,-D_GLIBCXX_USE_CXX11_ABI=0,-std=gnu++2b,-std=gnu++11}" }
}

So I run all tests 21 times (seven -std options, each repeated for three target
boards).

[Bug libstdc++/115433] unexpected increase of testsuite execution time

[jbeulich at suse dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115433

--- Comment #3 from jbeulich at suse dot com ---
(In reply to Jonathan Wakely from comment #1)
> What's the baseline for comparisons, the 13.x releases?

Oh, sorry, I should of course have mentioned that: Yes, 13.3.0.

> Another possible culprit is that you now have tbb-devel installed on the
> test machine, which causes the PSTL tests to be run instead of skipped. Some
> of those are slow too.

I've never heard of tbb-devel, to be honest. And I can't spot any package with
"tbb" in its name on any of the hosts I've been using. (These are all somewhat
older distros, fwiw.)