From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 21681 invoked by alias); 21 Dec 2012 13:01:47 -0000 Received: (qmail 21673 invoked by uid 22791); 21 Dec 2012 13:01:46 -0000 X-SWARE-Spam-Status: No, hits=-2.7 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,KHOP_THREADED,RCVD_IN_DNSWL_NONE,RCVD_IN_HOSTKARMA_YE,T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from imr-ma01.mx.aol.com (HELO imr-ma01.mx.aol.com) (64.12.206.39) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Fri, 21 Dec 2012 13:01:37 +0000 Received: from mtaout-da04.r1000.mx.aol.com (mtaout-da04.r1000.mx.aol.com [172.29.51.132]) by imr-ma01.mx.aol.com (Outbound Mail Relay) with ESMTP id 51383380000AA for ; Fri, 21 Dec 2012 08:01:36 -0500 (EST) Received: from [134.134.139.76] (unknown [134.134.139.76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mtaout-da04.r1000.mx.aol.com (MUA/Third Party Client Interface) with ESMTPSA id B5D9DE000138 for ; Fri, 21 Dec 2012 08:01:35 -0500 (EST) Message-ID: <50D45DAA.40308@aol.com> Date: Fri, 21 Dec 2012 13:01:00 -0000 From: Tim Prince Reply-To: tprince@computer.org User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: gcc-help@gcc.gnu.org Subject: Re: parallel testsuite results differ from a single task approach References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit x-aol-global-disposition: G X-AOL-SCOLL-SCORE: 0:2:433093024:93952408 X-AOL-SCOLL-URL_COUNT: 0 x-aol-sid: 3039ac1d338450d45dad0f24 X-AOL-IP: 134.134.139.76 Mailing-List: contact gcc-help-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-help-owner@gcc.gnu.org X-SW-Source: 2012-12/txt/msg00117.txt.bz2 On 12/20/2012 11:06 PM, Dennis Clarke wrote: > I gad received some good advice to try using gmake -j X -k check for some X > 1 when running a testsuite in order to speed the process up. So I decided to test that theory with an eye on the results. I was surprised to see that the results when running "gmake -j 8 -k check" differ from a simple "gmake -k check" in the area of go and libmudflap : > > 1 ) parallel testsuite : > > http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01903.html > > 2 ) single process testsuite : > > http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01935.html > > > In parallel we see this for "go" : > > === go tests === > > > Running target unix > FAIL: go.test/test/stack.go execution, -O2 -g > > === go Summary === > > # of expected passes 3293 > # of unexpected failures 1 > # of expected failures 4 > # of untested testcases 4 > /home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00) > > > > > While the single process apporach reports : > > > === go Summary === > > # of expected passes 3294 > # of expected failures 4 > # of untested testcases 4 > /home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00) > > > Perhaps the go test "go.test/test/stack.go" is somehow sensitive to running on a busy machine? I am only guessing. > > The "libmudflap tests" results are wildly different. > > So at first glance it appears that the testsuite will in fact product different results if one chooses to use "gmake -j X" for X>1 but I am not sure why. Felt is was worth pointing out as maybe no one else has done the comparison. > > Dennis > > Did you read any reports from recent years about problems with this? If a test dupiicates itself while accessing the same file system, it may create a kind of race condition. If the additional failures are time-outs, it's to be expected on a "busy machine." In particular, you may have multiple tests contending for a single rotating disk. Number of cores, OS, and other factors will make a large difference. How are we to know what you aren't telling? -- Tim Prince