From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by sourceware.org (Postfix) with ESMTPS id 56B463858C53 for ; Sun, 17 Jul 2022 15:51:47 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 56B463858C53 Received: by mail-pl1-x62d.google.com with SMTP id x21so7077500plb.3 for ; Sun, 17 Jul 2022 08:51:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=b44JJKtbMMNhVfe8LVzMNLI56WWYJC2RP/tR7uW8Pmw=; b=si4UB7kNXaQXZfLcV9gl/B0WTkKRfZkL9dnJG9wHGgwkWuo9+M0ypmExvQE1uXuMXg l+CwdqcWvX98TBePlA/8rfqqEL+jrn/OQStRx0FgdSAT9pdBD9Rh2lTAk0siZz2PDARh 9HjpmF+SwfBNI+kfPRGW6kU1qypBuwjr+HK9O62i7KsgpzYaFwf5GlKjTz2CdUAv+zj1 UUgaHZe0bisU144LSPl9nHh2Q2yLoRtxtDPbS45PMQnvQaQc77D/eOOEtZClqWVegHMH ayeJTdzZTJ3+nPxguhE15wW5A5f8pWJlloI4WQD+CALN2obpYYy0Z1NJWRbKvpWOBSNv WI3Q== X-Gm-Message-State: AJIora/fyOjQYXtCrf/lpcUqNVZ04ZWfsTfY2HVrNc3NcsPdZOta+BHp pZZSwVnFJ5nJhd5h2SmPogVbIvxZlkdSdw== X-Google-Smtp-Source: AGRyM1sjG63USPyT1Z3d1436A2oMPv/mMZWYV2Cf/OK+I8p1TvT/oeRiRmVXYb5KEMb/z1cOG7HHLg== X-Received: by 2002:a17:90a:df89:b0:1f0:916c:284f with SMTP id p9-20020a17090adf8900b001f0916c284fmr26046000pjv.78.1658073106117; Sun, 17 Jul 2022 08:51:46 -0700 (PDT) Received: from ldh-macbook.local ([2601:648:8281:4310:958b:b934:e1ee:7a2a]) by smtp.gmail.com with ESMTPSA id c11-20020a056a00008b00b005289fad1bbesm7648984pfj.94.2022.07.17.08.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jul 2022 08:51:45 -0700 (PDT) Date: Sun, 17 Jul 2022 11:51:43 -0400 From: Lewis Hyatt To: Iain Buclaw Cc: gcc-patches@gcc.gnu.org Subject: Re: [PATCH] libphobos: Fix instability in the parallelized testsuite Message-ID: References: <20220714215325.GA18923@ldh-imac.local> <1657922526.7xsg40t05n.astroid@pulse.none> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1657922526.7xsg40t05n.astroid@pulse.none> X-Spam-Status: No, score=-3032.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jul 2022 15:51:49 -0000 > Hi Lewis, > > Thanks! Good spot. I think it should be calling dg-runtest however, > same as what libphobos.cycles/cycles.exp is doing. Could also fix the > test name so each one is unique, just to hit two birds in one - > something like the following would suffice (haven't had time to check). > > Kind Regards, > Iain. > > --- > > --- a/libphobos/testsuite/libphobos.unittest/unittest.exp > +++ b/libphobos/testsuite/libphobos.unittest/unittest.exp > @@ -42,8 +42,10 @@ foreach unit_test $unit_test_list { > set expected_fail [lindex $unit_test 1] > > foreach test $tests { > - set shouldfail $expected_fail > - dg-test $test "" $test_flags > + set libphobos_test_name "[dg-trim-dirname $srcdir $test] $test_flags" > + set shouldfail $expected_fail > + dg-runtest $test "" $test_flags > + set libphobos_test_name "" > } > > set shouldfail 0 > Thanks for the followup. I tested and can confirm your version works fine: PASS: libphobos.unittest/customhandler.d -fversion=FailNoPrintout (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=FailNoPrintout execution test PASS: libphobos.unittest/customhandler.d -fversion=FailedTests (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=FailedTests execution test PASS: libphobos.unittest/customhandler.d -fversion=GoodTests (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=GoodTests execution test PASS: libphobos.unittest/customhandler.d -fversion=NoTests (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=NoTests execution test PASS: libphobos.unittest/customhandler.d -fversion=PassNoPrintout (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=PassNoPrintout execution test Let me know if you want me to do anything from there please? By the way, there are a few other tests that cause some minor glitches with comparing results: libphobos.sum:PASS: libphobos.shared/link.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared lib.so -shared-libphobos (test for excess errors) libphobos.sum:PASS: libphobos.shared/link.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared lib.so -shared-libphobos execution test libphobos.sum:PASS: libphobos.shared/link_linkdep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared liblinkdep.so lib.so -shared-libphobos (test for excess errors) libphobos.sum:PASS: libphobos.shared/link_linkdep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared liblinkdep.so lib.so -shared-libphobos execution test libphobos.sum:PASS: libphobos.shared/link_loaddep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared libloaddep.so -shared-libphobos (test for excess errors) libphobos.sum:PASS: libphobos.shared/link_loaddep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared libloaddep.so -shared-libphobos execution test The problem here is that the absolute path to the test dir ends up in the results summary, since it appears in the options string that is part of the test name. It's not so hard to work around when doing the comparisons, but it seems to be the only case where this happens in the whole testsuite, other than one other similar case from libgo. Is there a standard way to handle it I take it? Thanks... -Lewis