From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 24637 invoked by alias); 25 Sep 2014 12:22:41 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Received: (qmail 24621 invoked by uid 89); 25 Sep 2014 12:22:40 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-3.3 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Thu, 25 Sep 2014 12:22:39 +0000 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s8PCMVIj017619 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 25 Sep 2014 08:22:31 -0400 Received: from [10.10.51.195] (vpn-51-195.rdu2.redhat.com [10.10.51.195]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s8PCMTsZ022376; Thu, 25 Sep 2014 08:22:29 -0400 Message-ID: <54240905.70600@redhat.com> Date: Thu, 25 Sep 2014 12:22:00 -0000 From: Andrew MacLeod User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Segher Boessenkool CC: Jakub Jelinek , Bernd Schmidt , gcc-patches , richard.sandiford@arm.com Subject: Re: parallel check output changes? References: <541AD692.4030907@redhat.com> <20140918130109.GH17454@tucnak.redhat.com> <541AD880.7080703@redhat.com> <541AF451.3070406@redhat.com> <541B1710.8060809@codesourcery.com> <20140918173609.GM17454@tucnak.redhat.com> <20140918184455.GB28595@gate.crashing.org> <20140919093723.GA26414@gate.crashing.org> <87iokel5c0.fsf@e105548-lin.cambridge.arm.com> <5422DB41.1090800@redhat.com> <20140924161039.GA27079@gate.crashing.org> <5422F174.2020001@redhat.com> <5423065F.7020308@redhat.com> In-Reply-To: <5423065F.7020308@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2014-09/txt/msg02252.txt.bz2 On 09/24/2014 01:58 PM, Andrew MacLeod wrote: > On 09/24/2014 12:29 PM, Andrew MacLeod wrote: >> > > AH. interesting. > > The third run has a gcc.sum that is exactly the same as the first run. > so only the second run differs, and it seems to be from an > alphabetical sort. So run 3 and 1 match. > the gfortran.sum from the third run is identical to the *second* run, > but it is different from the *first* run. so run 2 and 3 match. > > the two runs that match (2nd and 3rd run) look like: > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single -O2 (test > for excess errors) > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single -O2 > execution test > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib -O2 > -lcaf_single (test for excess errors) > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib -O2 > -lcaf_single execution test > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single -O2 (test > for excess errors) > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single -O2 > execution test > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib -O2 > -lcaf_single (test for excess errors) > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib -O2 > -lcaf_single execution test > > and the odd one out (firstrun:) > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib -O2 > -lcaf_single (test for excess errors) > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib -O2 > -lcaf_single execution test > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single -O2 (test > for excess errors) > PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single -O2 > execution test > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib -O2 > -lcaf_single (test for excess errors) > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib -O2 > -lcaf_single execution test > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single -O2 (test > for excess errors) > PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single -O2 > execution test > > looks like the first run was sorted, and the other 2 weren't. > > There must be some condition under which we don't sort the results? or > another place which needs to be tweaked to do the sort as well...? > > Andrew > So to be fair, I could use test_summary, but I think the concern is warranted because if this inconsistent ordering can happen to PASS, I would expect the same non-deterministic behaviour if those tests happen to FAIL. we just have far less FAILS so we aren't seeing it with test_summary at the moment... Aggregating all my .sum files, I see a sampling of about 257,000 PASSs, whereas I see a total of 141 FAILs. FAILs only account for < 0.06% of the output. ( I'm getting an average of about 510 mis-ordered PASSs, so it only affects a small portion of them as well.) I would think the output of .sum needs to be consistent from one run to the next in order for test_summary to consistently report its results as well. Andrew