From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 5616 invoked by alias); 1 Jun 2006 19:35:33 -0000 Received: (qmail 5602 invoked by uid 22791); 1 Jun 2006 19:35:31 -0000 X-Spam-Check-By: sourceware.org Received: from thelemkes.ca (HELO shoal.thelemkes.ca) (66.96.18.66) by sourceware.org (qpsmtpd/0.31) with ESMTP; Thu, 01 Jun 2006 19:35:26 +0000 Received: from winch.thelemkes.ca (unknown [192.168.62.33]) by shoal.thelemkes.ca (Postfix) with ESMTP id 7745B40B3E6; Thu, 1 Jun 2006 15:35:23 -0400 (EDT) Subject: Re: comparing DejaGNU results From: James Lemke To: Mike Stump Cc: gcc@gcc.gnu.org In-Reply-To: References: <1149099224.7265.396.camel@winch.thelemkes.ca> Content-Type: text/plain Date: Thu, 01 Jun 2006 19:35:00 -0000 Message-Id: <1149190523.4755.59.camel@winch.thelemkes.ca> Mime-Version: 1.0 X-Mailer: Evolution 2.2.1 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org X-SW-Source: 2006-06/txt/msg00032.txt.bz2 > Please do. I'd welcome it (and scripts to generate html, to track > known issues, to trim log files, to drive things and do on)... I > think having a few different styles would be good, then people can > try them all out and see which ones they like and why. Anyway, for > me, it isn't yet better: : > I tried all the results and couldn't find any that your script could > analyze, except libgomp, but only because libgomp doesn't run on my > system, so there are no results in it? Is it just me? I hand edited > my results to be the first 30 or so from the Objective C++ testsuite, > and then I got it to analyze and not dump out. > > When I tried gcc, I had a chance to notice the timings, your version: > > real 8m44.413s > user 2m0.714s > sys 7m54.847s > > mine: > > real 0m1.994s > user 0m1.756s > sys 0m0.556s As I said I had only used it on Linux, I assume you're using Darwin. I tried it there and had to fix the following. You got the first two but not the last so my script was way non-functional. sed uses -E instead of -r (as you noted) sort uses -m but not --merge (as you noted) sed doesn't honour \t or \s > :-) Maybe you only run the script on toy results? Or, do you just > drink lots of coffee? Now, I know mine is well more than 10-100x > slower than it could be, but waiting 2 seconds isn't a hardship for > me. Waiting 8 minutes strikes me as just a little slow. I said it was slow :-)> On NetBSD your approach is at least 50x faster. Strangely, on FC3 the two have similar elapsed times: time ./compare_tests gcc-sim-20050503.sum gcc-sim-20060528.sum >compare2.lst real 4m0.729s user 4m5.880s sys 0m0.231s time ./dg-cmp-results.sh -v "" gcc-sim-20050503.sum gcc-sim-20060528.sum >dg-cmp2.lst real 4m25.941s user 1m29.850s sys 3m8.021s > The output of yours doesn't seem to be targeted for human eyes, the > verbosity (at the least verbose setting) is about 121x more than mine > for two random sets I had lying around, and that is with it cutting > the output off really early due to the above problem. I predict that > in normal day to day use, it is well better than 120,000x larger. > What use do you get out of it? It obviously wasn't working properly for you due to linux - netbsd differences. wc -l *2.lst 66 compare2.lst 63 dg-cmp2.lst Results from mine should have looked like: dg-cmp-results.sh: Verbosity is 1, Variant is "" Older log file: gcc-sim-20050503.sum Test Run By jim on Mon May 2 12:29:08 2005 Target is xscale-unknown-elf Host is i686-pc-linux-gnu Newer log file: gcc-sim-20060528.sum Test Run By jim on Mon May 29 12:55:01 2006 Target is xscale-unknown-elf Host is i686-pc-linux-gnu PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O1 PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O2 PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O3 -fomit-frame-pointer PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O3 -g PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -Os PASS->UNRESOLVED: gcc.c-torture/execute/920428-2.c execution, -O1 PASS->UNRESOLVED: gcc.c-torture/execute/920428-2.c execution, -O2 : PASS->UNRESOLVED: gcc.c-torture/execute/nestfunc-6.c execution, -Os FAIL->PASS: gcc.dg/20030324-1.c execution test FAIL->PASS: gcc.dg/debug/dwarf2/dwarf-die7.c scan-assembler 1.*DW_AT_inline FAIL->PASS: gcc.dg/range-test-1.c execution test > Mine was designed to do two things, figure out if the results are > interesting and not send email, if they are not, and to show > engineers the `interesting' detailed results in priority order. It's > meant to be run daily, and on good days, it produces no output. On > normal days, output is 20-30 lines at most. It tries to balance the > complaints with the occasional atta boy, to help with moral. Mine is used at the end of full build & test runs so a few minutes didn't bother me. A look at output like that above tells me what I need. Your approach is faster, esp. on Darwin / NetBSD. The only advantages I see to mine is handling variants (Richard's patch fixes that), verbosity control, and detail -- compare_tests only looks at X?(PASS|X?FAIL). Jim. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario