Re: [RFC] c++, libstdc++: Default make check vs. tests for newest C++ standard

[Jonathan Wakely <jwakely@redhat.com>]

On Wed, 19 Oct 2022 at 09:40, Jakub Jelinek <jakub@redhat.com> wrote:
>
> Hi!
>
> The screw-up on my side with libstdc++ testing (tested normally rather
> than in C++23 mode) makes me wonder if we couldn't tweak the default
> testing.
> Dunno what libstdc++ testing normally does (just C++17?),

That's the default unless a test has something else in -std=gnu++17
but I do my local testing with:

set target_list {
"unix{,-D_GLIBCXX_USE_CXX11_ABI=0,-std=gnu++2b,-std=gnu++11}" }

and then push to the compile farm and test with:

set target_list {
"unix{,-std=c++98,-std=gnu++11,-std=gnu++20,-D_GLIBCXX_USE_CXX11_ABI=0/-D_GLIBCXX_DEBUG,-D_GLIBCXX_DEBUG,-std=gnu++23}"
}

That's far too slow to force on everybody though.


> make check-g++
> tests by default { 98, 14, 17, 20 } (and I regularly use
> GXX_TESTSUITE_STDS=98,11,14,17,20,2b in environment but that doesn't
> cover libstdc++ I guess).

It doesn't, correct. It's been on my TODO list for a couple of years.

> When adding tests for upcoming C++ version, one always has a dilemma
> whether to use explicit // { dg-options "-std=c++2b" }
> or -std=gnu++2b and similar, then the test works in all modes, but it might
> be forgotten later on to be converted into // { dg-do whatever { target c++23 } }
> test so that when 23 is tested by default and say 26 or 29 appears too,
> we test it also in those modes, or just go with
> // { dg-do whatever { target c++23 } }
> which has the disadvantage that it is skipped when testing by default and
> one only tests it if he asks for the newer version.

The convention is:
// { dg-options "-std=gnu++23" }
// { dg-do whatever { target c++23 } }

When that becomes the default, we'll remove the first line, so that it
runs for all later versions.
See r12-678 to r12-686 which removed the dg-options "-std=gnu++17"
after that became the default for g++.

I should have noticed you were missing that from some of the new
tests, sorry. I saw it in a few and didn't check them all.


> I wonder if we couldn't for the default testing (when one doesn't
> specify GXX_TESTSUITE_STDS or uses make check-c++-all and similar)
> improve things a little bit by automatically treat those
> // { dg-do whatever { target c++23 } }
> tests as // { dg-options "-std=c++2b" }.
>
> g++-dg.exp has:
>         # If the testcase specifies a standard, use that one.
>         # If not, run it under several standards, allowing GNU extensions
>         # if there's a dg-options line.
>         if ![search_for $test "-std=*++"] {
>             if [search_for $test "dg-options"] {
>                 set std_prefix "-std=gnu++"
>             } else {
>                 set std_prefix "-std=c++"
>             }
>
>             # See g++.exp for the initial value of this list.
>             global gpp_std_list
>             if { [llength $gpp_std_list] > 0 } {
>                 set std_list $gpp_std_list
>             } else {
>                 set std_list { 98 14 17 20 }
>             }
>             set option_list { }
>             foreach x $std_list {
>                 # Handle "concepts" as C++17 plus Concepts TS.
>                 if { $x eq "concepts" } then { set x "17 -fconcepts"
>                 } elseif { $x eq "impcx" } then { set x "23 -fimplicit-constexpr" }
>                 lappend option_list "${std_prefix}$x"
>             }
>         } else {
>             set option_list { "" }
>         }
>
>         set nshort [file tail [file dirname $test]]/[file tail $test]
>
>         foreach flags_t $option_list {
>             verbose "Testing $nshort, $flags $flags_t" 1
>             dg-test $test "$flags $flags_t" ${default-extra-flags}
>         }
> so I wonder if in the set std_list { 98 14 17 20 } spot we couldn't do
> something like special search_for for "{ dg-do * { target c++23 } }"
> and if so, set std_list { 2b } instead of set std_list { 98 14 17 20 }?
> It wouldn't handle more complex cases like
> // { dg-do compile { target { c++23 && { aarch64*-*-* powerpc64le*-*-linux* riscv*-*-* s390*-*-* sparc*-*-linux* } } } }
> but at least for the majority of tests for the new language version
> it would run them even in default testing where they'd be otherwise
> skipped (we'd cycle over 98 14 17 20 only to see it doesn't satisfy any of
> them).
> If we wanted to go even further, we could handle similarly say c++11_only
> tests.
>
> What do you think?

But libstdc++ doesn't use g++.exp so we need to start using that (or
something like it) in libstdc++ before any such changes would help.