From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 585263858C50 for ; Tue, 12 Sep 2023 13:06:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 585263858C50 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694523972; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=z3zsFhPlqNn7KkeGoAT2T5BiXtel+0MFt202DtomyA4=; b=SZhBdxk9eh8iY+CtalPOJctkNtBvm/kFrjCpZF4scnMHvnaUwfEIaGQmCIpa7W55XQsKf7 lPRf9UxqKOVq5IMlco1pO03FziMvRX86HZLe6n7iTBX4eCQ8rXRViv3W9OBnKK41mrDGru 1iQ2rrIhGTUn8fB5noj0cGpVUFLS4Gw= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-624-3aGVEhP0NieWBG4DQ04PFA-1; Tue, 12 Sep 2023 09:06:08 -0400 X-MC-Unique: 3aGVEhP0NieWBG4DQ04PFA-1 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-2bcc4347d48so62869341fa.0 for ; Tue, 12 Sep 2023 06:06:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694523967; x=1695128767; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=z3zsFhPlqNn7KkeGoAT2T5BiXtel+0MFt202DtomyA4=; b=WSGO1vasgDO5futZ/ZiFxFmJZNVFONeJE6BAPS4NQ6MzA6hKCjs8v2rRjYgO0IVSqf LYxwYh2v29vUBBtpdD5pK12C/plf8lwm4/vHnnj+i4FCICsbKDui9SBsL89YXY9pXp1M hosCp9fvnhw7x7Ekt40qw7R5OByB5dt0/tIbRTfIxwnbZRTycUIkrnp9ZfHRbV15wmMP tiTPam32jov9LHOqWJa2UTj24TWZjqZ8zRwc38xeyg/Bfh88mfWFiL65Ttph7LcAypYa LgKgOvDBZ5QXhs0F846fPWXpRSF0bQEE9T02Unr4O6cfBN+HseGKEZrtEpg7ydOBBOET QIYw== X-Gm-Message-State: AOJu0YwwYGyeiq2hUI/PS4E2bg+FJN5BR/5ns10psswewlmp+NnOdrkc i0nbzRxuU+u3xzRbEEkrmJ/sttiwblLsU0zuck7fAI0sj+q6QTKND4Y/fUsxNU8SP3hAaMZocWD MZ2WkCl26fKRBAa9YpjRdo9yx5RY5YWg= X-Received: by 2002:a2e:918f:0:b0:2b7:7c:d5a1 with SMTP id f15-20020a2e918f000000b002b7007cd5a1mr9988790ljg.23.1694523967103; Tue, 12 Sep 2023 06:06:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDk5u3JQVFhbSw4cP75BUPdBjTPKj06hhiZc6p1PqlJ3Q57Al3IWb3vb5MaZGjcvwGybl7JbfEV1wtLyFyN8U= X-Received: by 2002:a2e:918f:0:b0:2b7:7c:d5a1 with SMTP id f15-20020a2e918f000000b002b7007cd5a1mr9988775ljg.23.1694523966761; Tue, 12 Sep 2023 06:06:06 -0700 (PDT) MIME-Version: 1.0 References: <20230911163534.1913512-1-jwakely@redhat.com> In-Reply-To: <20230911163534.1913512-1-jwakely@redhat.com> From: Jonathan Wakely Date: Tue, 12 Sep 2023 14:05:55 +0100 Message-ID: Subject: Re: [PATCH 00/13] libstdc++: Add support for running tests with multiple -std options To: Jonathan Wakely Cc: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-6.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Mon, 11 Sept 2023 at 17:37, Jonathan Wakely via Libstdc++ wrote: > > This patch series replicates the behaviour of the g++ testsuite, so that > libstdc++ tests can easily be run for multiple different -std options in > a single testsuite run. As described in the updated docs, the -std > options to use for every test can be overridden by setting v3_std_list > in ~/.dejagnurc or $DEJAGNU, or setting $GLIBCXX_TESTSUITE_STDS in the > environment. If not overridden, the default is just to run with > -std=gnu++17 (so that we don't increase the time taken for a full > testsuite run). > > Tests that require a newer standard than C++17 will default to that > newer standard and C++26, so e.g. std::format tests will be run with > both -std=gnu++20 and -std=gnu++26. This does increase the number of > tests, but only for the subset of tests for C++20/23/26 features. If > this is too costly for testers, we can change that (this might be > needed, because the C++20 tests for std::ranges and std::format are > particularly slow to compile). > > Because a correct default will be chosen for tests that require > something newer than C++17, we no longer need dg-options "-std=gnu++20" > or similar in any tests. Removing the explicit -std option allows the > test to be run for later standards via the v3_std_list settings, so that > we can verify that C++20 features still work in C++23 and C++26, for > example. This change already found some tests which failed when run > with a later standard (see r14-3771-gf12e26f3496275). > > Patches 2-13 in the series remove those unnecessary dg-options from > about half the relevant tests, but there are more than 500 others that > still need adjusting. > > We can remove files like testsuite/std/format/functions/format_c++23.cc > which only exist to duplicate existing tests with a different -std > option. We can remove that file now, and rely on format.cc being run > with multiple -std options by libstdc++ maintainers. > > It might also be useful to add a 'make check-quick' target which runs a > small subset of smoke tests with every standard version in v3_std_list. > This would be a suitable target for CI bots and for packagers who want > to verify that a build of GCC is functional, without running the entire > libstdc++ testsuite. There's a problem with this change. Some of our tests fail if they're run more than once. We have some static data files which are copied into the test directory by libstdc++_init at the start of the run. But some tests modify those files, so if the same test gets run multiple times, the file is no longer in the expected state after the first test. This only shows up when overriding the list of -std modes to include more than one option. The tests pass on the first run, and fail for subsequent ones: Running /home/test/src/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp ... PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++98 (test for excess errors) PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++98 execution test PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++11 (test for excess errors) FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++11 execution test PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++14 (test for excess errors) FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++14 execution test PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++17 (test for excess errors) FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++17 execution test PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++20 (test for excess errors) FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++20 execution test PASS: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++23 (test for excess errors) FAIL: 27_io/basic_filebuf/seekoff/char/1-io.cc -std=gnu++23 execution test We either need to copy the data files again after each test, or rewrite the tests to be idempotent.