From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by sourceware.org (Postfix) with ESMTPS id 7EBA63858D37 for ; Thu, 14 Jul 2022 21:53:29 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7EBA63858D37 Received: by mail-qk1-x72e.google.com with SMTP id b25so2305558qkl.1 for ; Thu, 14 Jul 2022 14:53:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition; bh=E/iN3BZNT1aNyuQsuL9+ussYtFnVTQAkqG4PGqmuyNo=; b=7+YqUeh4Y/DAfFrKl50lxp4VWtwrZEGIw73RZLMw1eIO6S8+bTNoAvDGZql0N5FYXt v+HcQV+Kntl2PsmhKXq1uiFpeKSGvtun9dIP3+j99Ru2KZoRHHobZpbGrzHvj1vW9AAx +UdEM6hyO1CloWI/1duEN4Eb7nqQEFVtRjhrEw2wU0spLhQUV2GOKWFXbJneNNxJCj86 GTEyoIpGFY+4VrJHpleDaiDHgAjaGxgRpgzC+Z06HLcFXb/URq4wMIF7Gf9UTnmiUQAH Bvgs1ZgCoU1sThsfAn+80COnT3FoKfiR/E23SSUicicZkFrCjfdGFY6TTLYO9x9lVfHR TVjg== X-Gm-Message-State: AJIora/RvEZYL6qNV60vHZgage/W3/SLVFMqXE0gS8UD7/GBpdYFNVwG ebNqDxAW0Hw84ZY44bu7I1fsSRz2Pqc= X-Google-Smtp-Source: AGRyM1tXWeZl9apXxZ5MddgxrENvrpGZH9fKmUFBOY48CPwn2koYtTt8lz7uoUU+9rw5ZB8mcWnxCA== X-Received: by 2002:a05:620a:44c9:b0:6b2:1f49:f0f5 with SMTP id y9-20020a05620a44c900b006b21f49f0f5mr7486961qkp.327.1657835608492; Thu, 14 Jul 2022 14:53:28 -0700 (PDT) Received: from ldh-imac.local (96-67-140-173-static.hfc.comcastbusiness.net. [96.67.140.173]) by smtp.gmail.com with ESMTPSA id k10-20020ac8474a000000b00304bc2acc25sm2384383qtp.6.2022.07.14.14.53.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Jul 2022 14:53:27 -0700 (PDT) Date: Thu, 14 Jul 2022 17:53:25 -0400 From: Lewis Hyatt To: gcc-patches@gcc.gnu.org Subject: [PATCH] libphobos: Fix instability in the parallelized testsuite Message-ID: <20220714215325.GA18923@ldh-imac.local> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="jRHKVT23PllUwdXP" Content-Disposition: inline X-Spam-Status: No, score=-3039.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jul 2022 21:53:31 -0000 --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello- I get a different number of test results from libphobos.unittest/unittest.exp, depending on server load. I believe it's because this testsuite doesn't check runtest_file_p: $ make -j 1 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' # of expected passes 10 $ make -j 2 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' # of expected passes 10 # of expected passes 10 $ make -j 4 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' # of expected passes 10 # of expected passes 10 # of expected passes 10 # of expected passes 10 When running in parallel along with other tests, even at a fixed argument for -j, the number of tests that actually execute will depend on how many of the parallel sub-makes happened to start prior to the first one finishing, hence it changes from run to run. The attached patch fixes it for me, if it looks OK? Thanks, this would remove some noise from before/after test comparisons. -Lewis --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline; filename="libphobos_runtest.txt" libphobos: Fix instability in the parallelized testsuite libphobos.unittest/unittest.exp calls bare dg-test rather than dg-runtest, and so it should call runtest_file_p to determine whether to run each test or not. Without that call, the tests run too many times in parallel mode (they will run as many times, as the argument to make -j). libphobos/ChangeLog: * testsuite/libphobos.unittest/unittest.exp: Call runtest_file_p prior to running each test. diff --git a/libphobos/testsuite/libphobos.unittest/unittest.exp b/libphobos/testsuite/libphobos.unittest/unittest.exp index 2a019caca8c..175decdc333 100644 --- a/libphobos/testsuite/libphobos.unittest/unittest.exp +++ b/libphobos/testsuite/libphobos.unittest/unittest.exp @@ -42,6 +42,9 @@ foreach unit_test $unit_test_list { set expected_fail [lindex $unit_test 1] foreach test $tests { + if {![runtest_file_p $runtests $test]} { + continue + } set shouldfail $expected_fail dg-test $test "" $test_flags } --jRHKVT23PllUwdXP--