Re: [PATCH 00/13] libstdc++: Add support for running tests with multiple -std options

[Jonathan Wakely <jwakely@redhat.com>]

On Mon, 11 Sept 2023 at 17:37, Jonathan Wakely via Libstdc++
<libstdc++@gcc.gnu.org> wrote:
>
> This patch series replicates the behaviour of the g++ testsuite, so that
> libstdc++ tests can easily be run for multiple different -std options in
> a single testsuite run.  As described in the updated docs, the -std
> options to use for every test can be overridden by setting v3_std_list
> in ~/.dejagnurc or $DEJAGNU, or setting $GLIBCXX_TESTSUITE_STDS in the
> environment.  If not overridden, the default is just to run with
> -std=gnu++17 (so that we don't increase the time taken for a full
> testsuite run).
>
> Tests that require a newer standard than C++17 will default to that
> newer standard and C++26, so e.g. std::format tests will be run with
> both -std=gnu++20 and -std=gnu++26.  This does increase the number of
> tests, but only for the subset of tests for C++20/23/26 features.  If
> this is too costly for testers, we can change that (this might be
> needed, because the C++20 tests for std::ranges and std::format are
> particularly slow to compile).
>
> Because a correct default will be chosen for tests that require
> something newer than C++17, we no longer need dg-options "-std=gnu++20"
> or similar in any tests.  Removing the explicit -std option allows the
> test to be run for later standards via the v3_std_list settings, so that
> we can verify that C++20 features still work in C++23 and C++26, for
> example.  This change already found some tests which failed when run
> with a later standard (see r14-3771-gf12e26f3496275).
>
> Patches 2-13 in the series remove those unnecessary dg-options from
> about half the relevant tests, but there are more than 500 others that
> still need adjusting.
>
> We can remove files like testsuite/std/format/functions/format_c++23.cc
> which only exist to duplicate existing tests with a different -std
> option.  We can remove that file now, and rely on format.cc being run
> with multiple -std options by libstdc++ maintainers.
>
> It might also be useful to add a 'make check-quick' target which runs a
> small subset of smoke tests with every standard version in v3_std_list.
> This would be a suitable target for CI bots and for packagers who want
> to verify that a build of GCC is functional, without running the entire
> libstdc++ testsuite.

There's a problem with this change. Some of our tests fail if they're
run more than once.

We have some static data files which are copied into the test
directory by libstdc++_init at the start of the run. But some tests
modify those files, so if the same test gets run multiple times, the
file is no longer in the expected state after the first test.

This only shows up when overriding the list of -std modes to include
more than one option. The tests pass on the first run, and fail for
subsequent ones:

Running /home/test/src/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp
...
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++98 (test for
excess errors)
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++98 execution test
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++11 (test for
excess errors)
FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++11 execution test
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++14 (test for
excess errors)
FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++14 execution test
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++17 (test for
excess errors)
FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++17 execution test
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++20 (test for
excess errors)
FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++20 execution test
PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++23 (test for
excess errors)
FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc  -std=gnu++23 execution test

We either need to copy the data files again after each test, or
rewrite the tests to be idempotent.