* comparing DejaGNU results @ 2006-05-31 18:13 James Lemke 2006-05-31 18:25 ` Joe Buck ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: James Lemke @ 2006-05-31 18:13 UTC (permalink / raw) To: gcc [-- Attachment #1: Type: text/plain, Size: 1006 bytes --] I wanted some mechanical way to compare the output of dejagnu runs between releases, etc. I asked a few people at the GCC Summit last year what they used or knew about. Not much came to light, so I ended up writing something of my own. It's a shell script I've been using on Linux with only trivial changes since December. The one tricky part was dealing with lines in the results file that are identical. I made the rash assumption that the relative order wouldn't change between runs and that seems to hold. The script works well although it isn't blazingly fast. My current version is attached. If others find it useful I can contribute it. Comments and suggestions are welcome. Typical invocation is something like: dg-cmp-results.sh -v "" gcc-20060328.sum gcc-20060510.sum |less for the first variation in the files, or dg-cmp-results.sh -v arm-sim gcc-20060328.sum gcc-20060510.sum |less for a specific variation. Cheers, Jim. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario [-- Attachment #2: Type: application/x-shellscript, Size: 6646 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:13 comparing DejaGNU results James Lemke @ 2006-05-31 18:25 ` Joe Buck 2006-05-31 18:28 ` Jeffrey Law 2006-06-01 1:14 ` Jim Wilson 2006-06-01 2:43 ` Mike Stump 2 siblings, 1 reply; 20+ messages in thread From: Joe Buck @ 2006-05-31 18:25 UTC (permalink / raw) To: James Lemke; +Cc: gcc On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote: > I wanted some mechanical way to compare the output of dejagnu runs > between releases, etc. I asked a few people at the GCC Summit last year > what they used or knew about. Not much came to light, so I ended up > writing something of my own. I thought that Jeff Law had something that compared .sum files back when he was RM. The description of what you wrote sounds similar. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:25 ` Joe Buck @ 2006-05-31 18:28 ` Jeffrey Law 2006-05-31 18:33 ` James Lemke 2006-06-01 0:56 ` Ben Elliston 0 siblings, 2 replies; 20+ messages in thread From: Jeffrey Law @ 2006-05-31 18:28 UTC (permalink / raw) To: Joe Buck; +Cc: James Lemke, gcc On Wed, 2006-05-31 at 11:25 -0700, Joe Buck wrote: > On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote: > > I wanted some mechanical way to compare the output of dejagnu runs > > between releases, etc. I asked a few people at the GCC Summit last year > > what they used or knew about. Not much came to light, so I ended up > > writing something of my own. > > I thought that Jeff Law had something that compared .sum files back when > he was RM. The description of what you wrote sounds similar. Don't we have a comparison script in the "contrib" subdirectory? Jeff ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:28 ` Jeffrey Law @ 2006-05-31 18:33 ` James Lemke 2006-05-31 18:45 ` Joe Buck 2006-06-01 0:56 ` Ben Elliston 1 sibling, 1 reply; 20+ messages in thread From: James Lemke @ 2006-05-31 18:33 UTC (permalink / raw) To: law; +Cc: Joe Buck, gcc On Wed, 2006-05-31 at 12:27 -0600, Jeffrey Law wrote: > On Wed, 2006-05-31 at 11:25 -0700, Joe Buck wrote: > > On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote: > > > I wanted some mechanical way to compare the output of dejagnu runs > > > between releases, etc. I asked a few people at the GCC Summit last year > > > what they used or knew about. Not much came to light, so I ended up > > > writing something of my own. > > > > I thought that Jeff Law had something that compared .sum files back when > > he was RM. The description of what you wrote sounds similar. > Don't we have a comparison script in the "contrib" subdirectory? There is contrib/compare_tests although dg-cmp-results gives more control over what is considered "interesting". -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:33 ` James Lemke @ 2006-05-31 18:45 ` Joe Buck 0 siblings, 0 replies; 20+ messages in thread From: Joe Buck @ 2006-05-31 18:45 UTC (permalink / raw) To: James Lemke; +Cc: law, gcc On Wed, May 31, 2006 at 02:33:29PM -0400, James Lemke wrote: > On Wed, 2006-05-31 at 12:27 -0600, Jeffrey Law wrote: > > On Wed, 2006-05-31 at 11:25 -0700, Joe Buck wrote: > > > On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote: > > > > I wanted some mechanical way to compare the output of dejagnu runs > > > > between releases, etc. I asked a few people at the GCC Summit last year > > > > what they used or knew about. Not much came to light, so I ended up > > > > writing something of my own. > > > > > > I thought that Jeff Law had something that compared .sum files back when > > > he was RM. The description of what you wrote sounds similar. > > Don't we have a comparison script in the "contrib" subdirectory? > There is contrib/compare_tests although dg-cmp-results gives more > control over what is considered "interesting". Well, I suppose people would welcome useful improvements to the contrib script. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:28 ` Jeffrey Law 2006-05-31 18:33 ` James Lemke @ 2006-06-01 0:56 ` Ben Elliston 2006-06-01 0:59 ` Andrew Pinski 2006-06-01 2:17 ` Mike Stump 1 sibling, 2 replies; 20+ messages in thread From: Ben Elliston @ 2006-06-01 0:56 UTC (permalink / raw) To: Jeffrey Law; +Cc: Joe Buck, James Lemke, gcc > > I thought that Jeff Law had something that compared .sum files back when > > he was RM. The description of what you wrote sounds similar. > Don't we have a comparison script in the "contrib" subdirectory? If that script does indeed work, then I'd like to distribute it with DejaGnu. There used to be a number of similar scripts in DejaGnu's contrib directory, but none of them worked, so I removed them all. Ben ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 0:56 ` Ben Elliston @ 2006-06-01 0:59 ` Andrew Pinski 2006-06-01 2:17 ` Mike Stump 1 sibling, 0 replies; 20+ messages in thread From: Andrew Pinski @ 2006-06-01 0:59 UTC (permalink / raw) To: Ben Elliston; +Cc: Jeffrey Law, Joe Buck, James Lemke, gcc > > > > I thought that Jeff Law had something that compared .sum files back when > > > he was RM. The description of what you wrote sounds similar. > > > Don't we have a comparison script in the "contrib" subdirectory? > > If that script does indeed work, then I'd like to distribute it with > DejaGnu. There used to be a number of similar scripts in DejaGnu's > contrib directory, but none of them worked, so I removed them all. It does work the last time I used it which was last year. It might also need to be expanded for KFAIL/KPASS for gdb. Thanks, Andrew Pinski ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 0:56 ` Ben Elliston 2006-06-01 0:59 ` Andrew Pinski @ 2006-06-01 2:17 ` Mike Stump 1 sibling, 0 replies; 20+ messages in thread From: Mike Stump @ 2006-06-01 2:17 UTC (permalink / raw) To: Ben Elliston; +Cc: Jeffrey Law, Joe Buck, James Lemke, gcc On May 31, 2006, at 5:57 PM, Ben Elliston wrote: >> Don't we have a comparison script in the "contrib" subdirectory? > > If that script does indeed work, then I'd like to distribute it with > DejaGnu. Please. Yes, it does work and has been working just fine for the past decade to decade and a half. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:13 comparing DejaGNU results James Lemke 2006-05-31 18:25 ` Joe Buck @ 2006-06-01 1:14 ` Jim Wilson 2006-06-01 2:43 ` Mike Stump 2 siblings, 0 replies; 20+ messages in thread From: Jim Wilson @ 2006-06-01 1:14 UTC (permalink / raw) To: James Lemke; +Cc: gcc James Lemke wrote: > I wanted some mechanical way to compare the output of dejagnu runs > between releases, etc. Did you look at contrib/compare_tests? It does something very similar to what your script is doing. -- Jim Wilson, GNU Tools Support, http://www.specifix.com ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-05-31 18:13 comparing DejaGNU results James Lemke 2006-05-31 18:25 ` Joe Buck 2006-06-01 1:14 ` Jim Wilson @ 2006-06-01 2:43 ` Mike Stump 2006-06-01 8:46 ` Richard Earnshaw ` (2 more replies) 2 siblings, 3 replies; 20+ messages in thread From: Mike Stump @ 2006-06-01 2:43 UTC (permalink / raw) To: James Lemke; +Cc: gcc [-- Attachment #1: Type: text/plain, Size: 1338 bytes --] On May 31, 2006, at 11:13 AM, James Lemke wrote: > My current version is attached. If others find it useful I can > contribute it. Comments and suggestions are welcome. Please do. I'd welcome it (and scripts to generate html, to track known issues, to trim log files, to drive things and do on)... I think having a few different styles would be good, then people can try them all out and see which ones they like and why. Anyway, for me, it isn't yet better: dg-cmp-results.sh: Verbosity is 0, Variant is "unix" Older log file: 06031600/gcc.sum sed: illegal option -- r usage: sed script [-Ean] [-i extension] [file ...] sed [-an] [-i extension] [-e script] ... [-f script_file] ... [file ...] Newer log file: 06052905/gcc.sum sed: illegal option -- r usage: sed script [-Ean] [-i extension] [file ...] sed [-an] [-i extension] [-e script] ... [-f script_file] ... [file ...] sed: illegal option -- r usage: sed script [-Ean] [-i extension] [file ...] sed [-an] [-i extension] [-e script] ... [-f script_file] ... [file ...] sed: illegal option -- r usage: sed script [-Ean] [-i extension] [file ...] sed [-an] [-i extension] [-e script] ... [-f script_file] ... [file ...] sort: unrecognized option `--' Try `sort --help' for more information. :-( When I fixed it up like this: [-- Attachment #2: t.txt --] [-- Type: text/plain, Size: 2392 bytes --] Doing diffs in /Network/Servers/madrid/Volumes/huis/mstump/bin/dg-cmp-results.~1~: --- /Network/Servers/madrid/Volumes/huis/mstump/bin/dg-cmp-results.~1~ 2006-05-31 15:57:42.000000000 -0700 +++ /Network/Servers/madrid/Volumes/huis/mstump/bin/dg-cmp-results 2006-05-31 17:24:21.000000000 -0700 @@ -42,6 +42,8 @@ OFILE="$2" OBASE=`basename "$2"` NFILE="$3" NBASE=`basename "$3"` +#export R=-r +export R=-E echo "dg-cmp-results.sh: Verbosity is ${verbose}, Variant is \"${VARIANT}\"" echo @@ -62,14 +64,14 @@ unset temp # Copy out the old file's section 0. echo "Older log file: $OFILE" -sed -r -e '/^\t+===/,$d' $OFILE +sed $R -e '/^\t+===/,$d' $OFILE # Copy out the new file's section 0. echo "Newer log file: $NFILE" -sed -r -e '/^\t+===/,$d' $NFILE +sed $R -e '/^\t+===/,$d' $NFILE # Create a temporary file from the old file's interesting section. -sed -r -e "1,/$header/d" \ +sed $R -e "1,/$header/d" \ -e '/^\t+===/,$d' \ -e '/^[A-Z]+:/!d' \ -e 's/\r$//' \ @@ -79,7 +81,7 @@ sed -r -e "1,/$header/d" \ >/tmp/o$$-$OBASE # Create a temporary file from the new file's interesting section. -sed -r -e "1,/$header/d" \ +sed $R -e "1,/$header/d" \ -e '/^\t+===/,$d' \ -e '/^[A-Z]+:/!d' \ -e 's/\r$//' \ @@ -94,9 +96,9 @@ sed -r -e "1,/$header/d" \ # If so, we assume that the order is the same in both files. IFS=: firstread=Y -sort --merge -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE | +sort -m -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE | while read -r lineon linestatus linename; do - linename=`echo "$linename" |sed -r -e 's/^ *(.*) *$/\1/'` + linename=`echo "$linename" |sed $R -e 's/^ *(.*) *$/\1/'` if test $verbose -ge 4; then case "$linestatus" in @@ -231,9 +233,9 @@ while read -r lineon linestatus linename else ##echo "$lines" |read -r prevon prevstatus prevname line=`echo "$lines" |head -n 1` - prevon=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\1/'` - prevstatus=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\2/'` - prevname=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\3/'` + prevon=`echo "$line" |sed $R -e 's/\s*(.):([A-Z]+):(.*)$/\1/'` + prevstatus=`echo "$line" |sed $R -e 's/\s*(.):([A-Z]+):(.*)$/\2/'` + prevname=`echo "$line" |sed $R -e 's/\s*(.):([A-Z]+):(.*)$/\3/'` if test $verbose -ge 4; then echo " debug: pulled a line from the stack" -------------- [-- Attachment #3: Type: text/plain, Size: 2760 bytes --] I got: Script error: trying to process: O: PASS: objc.dg/bad-receiver-type.m (test for warnings, line 13) N:PASS:objc.dg/bad-receiver-type.m (test for warnings, line 13) :-( mrs $ grep 'objc.dg/bad-receiver-type.m' 06031600/objc.sum 06052905/ objc.sum 06031600/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for warnings, line 13) 06031600/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for warnings, line 13) 06031600/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for excess errors) 06052905/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for warnings, line 13) 06052905/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for warnings, line 13) 06052905/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for excess errors) Mine btw gives: mrs $ cmp_logs 06031600/objc.sum 06052905/objc.sum New tests that PASS: objc.dg/dwarf-2.m (test for excess errors) objc.dg/dwarf-2.m scan-assembler 0x10[^0-9a-f].*DW_AT_language objc.dg/member-1.m (test for errors, line 3) objc.dg/member-1.m (test for excess errors) Old tests that passed, that have disappeared: (Eeek!) objc.dg/stubify-2.m scan-rtl-dump-not symbol_ref.*"objc_msgSend" I tried all the results and couldn't find any that your script could analyze, except libgomp, but only because libgomp doesn't run on my system, so there are no results in it? Is it just me? I hand edited my results to be the first 30 or so from the Objective C++ testsuite, and then I got it to analyze and not dump out. When I tried gcc, I had a chance to notice the timings, your version: real 8m44.413s user 2m0.714s sys 7m54.847s mine: real 0m1.994s user 0m1.756s sys 0m0.556s :-) Maybe you only run the script on toy results? Or, do you just drink lots of coffee? Now, I know mine is well more than 10-100x slower than it could be, but waiting 2 seconds isn't a hardship for me. Waiting 8 minutes strikes me as just a little slow. The output of yours doesn't seem to be targeted for human eyes, the verbosity (at the least verbose setting) is about 121x more than mine for two random sets I had lying around, and that is with it cutting the output off really early due to the above problem. I predict that in normal day to day use, it is well better than 120,000x larger. What use do you get out of it? Mine was designed to do two things, figure out if the results are interesting and not send email, if they are not, and to show engineers the `interesting' detailed results in priority order. It's meant to be run daily, and on good days, it produces no output. On normal days, output is 20-30 lines at most. It tries to balance the complaints with the occasional atta boy, to help with moral. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 2:43 ` Mike Stump @ 2006-06-01 8:46 ` Richard Earnshaw 2006-06-01 17:27 ` Mike Stump 2006-06-01 19:35 ` James Lemke 2006-06-01 19:44 ` James Lemke 2 siblings, 1 reply; 20+ messages in thread From: Richard Earnshaw @ 2006-06-01 8:46 UTC (permalink / raw) To: Mike Stump; +Cc: James Lemke, gcc [-- Attachment #1: Type: text/plain, Size: 1452 bytes --] On Thu, 2006-06-01 at 03:43, Mike Stump wrote: > Mine was designed to do two things, figure out if the results are > interesting and not send email, if they are not, and to show > engineers the `interesting' detailed results in priority order. It's > meant to be run daily, and on good days, it produces no output. On > normal days, output is 20-30 lines at most. It tries to balance the > complaints with the occasional atta boy, to help with moral. The only problem I have with Mike's script is that it doesn't handle runs with multiple multi-lib variants nicely. Attached is a local hack that I have to address this, but you'll need to re-seed your reference files if you apply it. With the change the output generated is not something like Tests that now work, but didn't before: arm-sim/-mthumb: g++.old-deja/g++.oliva/template9.C (test for errors, line 10) arm-sim: g++.old-deja/g++.oliva/template9.C (test for errors, line 10) New tests that FAIL: arm-sim/-mthumb: g++.dg/init/const3.C (test for excess errors) arm-sim/-mthumb: g++.dg/opt/temp2.C (test for excess errors) arm-sim/-mthumb: g++.dg/other/unused1.C scan-assembler stringz?\t"class2" arm-sim/-mthumb: g++.dg/other/unused1.C scan-assembler stringz?\t"printer" arm-sim/-mthumb: g++.dg/template/fntry1.C (test for excess errors) arm-sim: g++.dg/other/unused1.C scan-assembler stringz?\t"class2" arm-sim: g++.dg/other/unused1.C scan-assembler stringz?\t"printer" [-- Attachment #2: compare.patch --] [-- Type: text/plain, Size: 580 bytes --] --- /home/rearnsha/gnusrc/gcc/trunk/contrib/compare_tests 2005-11-12 16:36:57.000000000 +0000 +++ compare_gcc_tests 2006-01-24 18:49:05.000000000 +0000 @@ -22,8 +22,8 @@ exit 2 fi -sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" >$tmp1 -sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" >$tmp2 +sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp1 +sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp2 before=$tmp1 now=$tmp2 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 8:46 ` Richard Earnshaw @ 2006-06-01 17:27 ` Mike Stump 0 siblings, 0 replies; 20+ messages in thread From: Mike Stump @ 2006-06-01 17:27 UTC (permalink / raw) To: Richard Earnshaw; +Cc: James Lemke, gcc On Jun 1, 2006, at 1:45 AM, Richard Earnshaw wrote: > The only problem I have with Mike's script is that it doesn't handle > runs with multiple multi-lib variants nicely. Yeah, in the past, we drove the multilib pass from above as in general we had to select different hardware for testing. I like the idea. > Attached is a local hack that I have to address this Let's see if anyone can spot any non-portabilities in it. I think it is safe. If safe, I'd say let's add this, as it helps. I do have a slight additional change: --- cmp_logs.~2~ 2006-06-01 09:41:12.000000000 -0700 +++ cmp_logs 2006-06-01 10:05:56.000000000 -0700 @@ -25,2 +25,2 @@ -sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp1 -sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp2 +sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running target / {target = $3} { if (target != "unix") { sub(/: /, "&"target": " ); }; print $0; }' >$tmp1 +sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running target / {target = $3} { if (target != "unix") { sub(/: /, "&"target": " ); }; print $0; }' >$tmp2 This preserves the simplicity in the non-multilib case and keeps future results comparable with older results. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 2:43 ` Mike Stump 2006-06-01 8:46 ` Richard Earnshaw @ 2006-06-01 19:35 ` James Lemke 2006-06-01 20:26 ` James Lemke 2006-06-01 19:44 ` James Lemke 2 siblings, 1 reply; 20+ messages in thread From: James Lemke @ 2006-06-01 19:35 UTC (permalink / raw) To: Mike Stump; +Cc: gcc > Please do. I'd welcome it (and scripts to generate html, to track > known issues, to trim log files, to drive things and do on)... I > think having a few different styles would be good, then people can > try them all out and see which ones they like and why. Anyway, for > me, it isn't yet better: : > I tried all the results and couldn't find any that your script could > analyze, except libgomp, but only because libgomp doesn't run on my > system, so there are no results in it? Is it just me? I hand edited > my results to be the first 30 or so from the Objective C++ testsuite, > and then I got it to analyze and not dump out. > > When I tried gcc, I had a chance to notice the timings, your version: > > real 8m44.413s > user 2m0.714s > sys 7m54.847s > > mine: > > real 0m1.994s > user 0m1.756s > sys 0m0.556s As I said I had only used it on Linux, I assume you're using Darwin. I tried it there and had to fix the following. You got the first two but not the last so my script was way non-functional. sed uses -E instead of -r (as you noted) sort uses -m but not --merge (as you noted) sed doesn't honour \t or \s > :-) Maybe you only run the script on toy results? Or, do you just > drink lots of coffee? Now, I know mine is well more than 10-100x > slower than it could be, but waiting 2 seconds isn't a hardship for > me. Waiting 8 minutes strikes me as just a little slow. I said it was slow :-)> On NetBSD your approach is at least 50x faster. Strangely, on FC3 the two have similar elapsed times: time ./compare_tests gcc-sim-20050503.sum gcc-sim-20060528.sum >compare2.lst real 4m0.729s user 4m5.880s sys 0m0.231s time ./dg-cmp-results.sh -v "" gcc-sim-20050503.sum gcc-sim-20060528.sum >dg-cmp2.lst real 4m25.941s user 1m29.850s sys 3m8.021s > The output of yours doesn't seem to be targeted for human eyes, the > verbosity (at the least verbose setting) is about 121x more than mine > for two random sets I had lying around, and that is with it cutting > the output off really early due to the above problem. I predict that > in normal day to day use, it is well better than 120,000x larger. > What use do you get out of it? It obviously wasn't working properly for you due to linux - netbsd differences. wc -l *2.lst 66 compare2.lst 63 dg-cmp2.lst Results from mine should have looked like: dg-cmp-results.sh: Verbosity is 1, Variant is "" Older log file: gcc-sim-20050503.sum Test Run By jim on Mon May 2 12:29:08 2005 Target is xscale-unknown-elf Host is i686-pc-linux-gnu Newer log file: gcc-sim-20060528.sum Test Run By jim on Mon May 29 12:55:01 2006 Target is xscale-unknown-elf Host is i686-pc-linux-gnu PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O1 PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O2 PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O3 -fomit-frame-pointer PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -O3 -g PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation, -Os PASS->UNRESOLVED: gcc.c-torture/execute/920428-2.c execution, -O1 PASS->UNRESOLVED: gcc.c-torture/execute/920428-2.c execution, -O2 : PASS->UNRESOLVED: gcc.c-torture/execute/nestfunc-6.c execution, -Os FAIL->PASS: gcc.dg/20030324-1.c execution test FAIL->PASS: gcc.dg/debug/dwarf2/dwarf-die7.c scan-assembler 1.*DW_AT_inline FAIL->PASS: gcc.dg/range-test-1.c execution test > Mine was designed to do two things, figure out if the results are > interesting and not send email, if they are not, and to show > engineers the `interesting' detailed results in priority order. It's > meant to be run daily, and on good days, it produces no output. On > normal days, output is 20-30 lines at most. It tries to balance the > complaints with the occasional atta boy, to help with moral. Mine is used at the end of full build & test runs so a few minutes didn't bother me. A look at output like that above tells me what I need. Your approach is faster, esp. on Darwin / NetBSD. The only advantages I see to mine is handling variants (Richard's patch fixes that), verbosity control, and detail -- compare_tests only looks at X?(PASS|X?FAIL). Jim. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 19:35 ` James Lemke @ 2006-06-01 20:26 ` James Lemke 2006-06-01 20:36 ` Joseph S. Myers 2006-06-02 18:08 ` James Lemke 0 siblings, 2 replies; 20+ messages in thread From: James Lemke @ 2006-06-01 20:26 UTC (permalink / raw) To: Mike Stump; +Cc: gcc > Your approach is faster, esp. on Darwin / NetBSD. > The only advantages I see to mine is handling variants (Richard's patch > fixes that), verbosity control, and detail -- compare_tests only looks > at X?(PASS|FAIL). Hmm.. another small point, FWIW. Both the results files I used contained the following ssequence of results: PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) compare_tests reported the following: Tests that now fail, but worked before: gcc.c-torture/compile/930210-1.c (test for excess errors) gcc.c-torture/compile/930210-1.c (test for excess errors) gcc.c-torture/compile/930210-1.c (test for excess errors) : Tests that now work, but didn't before: gcc.c-torture/compile/930210-1.c (test for excess errors) gcc.c-torture/compile/930210-1.c (test for excess errors) gcc.c-torture/compile/930210-1.c (test for excess errors) dg-cmp-results didn't report anything (at that verbosity) because nothing had changed. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 20:26 ` James Lemke @ 2006-06-01 20:36 ` Joseph S. Myers 2006-06-01 20:42 ` James Lemke 2006-06-02 18:08 ` James Lemke 1 sibling, 1 reply; 20+ messages in thread From: Joseph S. Myers @ 2006-06-01 20:36 UTC (permalink / raw) To: James Lemke; +Cc: Mike Stump, gcc On Thu, 1 Jun 2006, James Lemke wrote: > Both the results files I used contained the following ssequence of > results: > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) This probably means you are using too old a version of DejaGnu; make sure you are using at least 1.4.4, the minimum supported version. There are still problems with duplicate test assertion names (e.g. bug 20771), but there shouldn't be in c-torture if you use current DejaGnu. -- Joseph S. Myers http://www.srcf.ucam.org/~jsm28/gcc/ jsm@polyomino.org.uk (personal mail) joseph@codesourcery.com (CodeSourcery mail) jsm28@gcc.gnu.org (Bugzilla assignments and CCs) ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 20:36 ` Joseph S. Myers @ 2006-06-01 20:42 ` James Lemke 0 siblings, 0 replies; 20+ messages in thread From: James Lemke @ 2006-06-01 20:42 UTC (permalink / raw) To: Joseph S. Myers; +Cc: Mike Stump, gcc > > Both the results files I used contained the following ssequence of > > results: > > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) > > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) > > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors) > > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors) > > This probably means you are using too old a version of DejaGnu; make sure > you are using at least 1.4.4, the minimum supported version. There are > still problems with duplicate test assertion names (e.g. bug 20771), but > there shouldn't be in c-torture if you use current DejaGnu. Thanks. Regardless, dg-cmp-results handles that situation, albeit slowly ; - )> -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 20:26 ` James Lemke 2006-06-01 20:36 ` Joseph S. Myers @ 2006-06-02 18:08 ` James Lemke 2006-06-02 23:27 ` Mike Stump 1 sibling, 1 reply; 20+ messages in thread From: James Lemke @ 2006-06-02 18:08 UTC (permalink / raw) To: gcc [-- Attachment #1: Type: text/plain, Size: 368 bytes --] I took a quick pass at implementing the comparisons in a more suitable lanugage. Run time is now a few seconds on both platforms. About the same as compare_tests on my old ibook/OSX and much faster on FC3. Trials show the same results as before. For anyone interested, the new version is attached. Jim. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario [-- Attachment #2: Type: application/x-shellscript, Size: 5100 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-02 18:08 ` James Lemke @ 2006-06-02 23:27 ` Mike Stump 2006-06-03 12:35 ` James Lemke 0 siblings, 1 reply; 20+ messages in thread From: Mike Stump @ 2006-06-02 23:27 UTC (permalink / raw) To: James Lemke; +Cc: gcc On Jun 2, 2006, at 11:08 AM, James Lemke wrote: > I took a quick pass at implementing the comparisons in a more suitable > lanugage. Run time is now a few seconds on both platforms. About the > same as compare_tests on my old ibook/OSX and much faster on FC3. Since Ben and I seem interested in this, I think we should check in this version. It seems portable enough and useful enough. Any objections from the crowd? ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-02 23:27 ` Mike Stump @ 2006-06-03 12:35 ` James Lemke 0 siblings, 0 replies; 20+ messages in thread From: James Lemke @ 2006-06-03 12:35 UTC (permalink / raw) To: Mike Stump; +Cc: gcc On Fri, 2006-06-02 at 16:26 -0700, Mike Stump wrote: > On Jun 2, 2006, at 11:08 AM, James Lemke wrote: > > I took a quick pass at implementing the comparisons in a more suitable > > lanugage. Run time is now a few seconds on both platforms. About the > > same as compare_tests on my old ibook/OSX and much faster on FC3. > > Since Ben and I seem interested in this, I think we should check in > this version. It seems portable enough and useful enough. Any > objections from the crowd? If not, I'll commit it to contrib/ on Monday. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: comparing DejaGNU results 2006-06-01 2:43 ` Mike Stump 2006-06-01 8:46 ` Richard Earnshaw 2006-06-01 19:35 ` James Lemke @ 2006-06-01 19:44 ` James Lemke 2 siblings, 0 replies; 20+ messages in thread From: James Lemke @ 2006-06-01 19:44 UTC (permalink / raw) To: Mike Stump; +Cc: gcc [-- Attachment #1: Type: text/plain, Size: 131 bytes --] Whoops... I forgot to attach my fixes, for anyone that's interested. -- Jim Lemke jwlemke@wasabisystems.com Orillia, Ontario [-- Attachment #2: Type: text/x-patch, Size: 2652 bytes --] --- dg-cmp-results.sh 2006/05/31 19:22:14 1.18 +++ dg-cmp-results.sh 2006/06/01 17:53:21 @@ -31,6 +31,16 @@ if test $# -ne 3 -o ! -f "$2" -o ! -f "$ exit 1 fi +# Command differences for various platforms. +case `uname -s` in +Darwin|NetBSD) + E=-E # sed + ;; +*) + E=-r # sed + ;; +esac + # sections are identified by separator lines beginning with '\t\t==='. # section 0 identifies run date, target, and host. # section 1 and subsequent contain test data for a target variant. @@ -62,15 +72,15 @@ unset temp # Copy out the old file's section 0. echo "Older log file: $OFILE" -sed -r -e '/^\t+===/,$d' $OFILE +sed $E -e '/^[[:space:]]+===/,$d' $OFILE # Copy out the new file's section 0. echo "Newer log file: $NFILE" -sed -r -e '/^\t+===/,$d' $NFILE +sed $E -e '/^[[:space:]]+===/,$d' $NFILE # Create a temporary file from the old file's interesting section. -sed -r -e "1,/$header/d" \ - -e '/^\t+===/,$d' \ +sed $E -e "1,/$header/d" \ + -e '/^[[:space:]]+===/,$d' \ -e '/^[A-Z]+:/!d' \ -e 's/\r$//' \ -e 's/^/O:/' \ @@ -79,8 +89,8 @@ sed -r -e "1,/$header/d" \ >/tmp/o$$-$OBASE # Create a temporary file from the new file's interesting section. -sed -r -e "1,/$header/d" \ - -e '/^\t+===/,$d' \ +sed $E -e "1,/$header/d" \ + -e '/^[[:space:]]+===/,$d' \ -e '/^[A-Z]+:/!d' \ -e 's/\r$//' \ -e 's/^/N:/' \ @@ -94,9 +104,9 @@ sed -r -e "1,/$header/d" \ # If so, we assume that the order is the same in both files. IFS=: firstread=Y -sort --merge -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE | +sort -m -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE | while read -r lineon linestatus linename; do - linename=`echo "$linename" |sed -r -e 's/^ *(.*) *$/\1/'` + linename=`echo "$linename" |sed $E -e 's/^ *(.*) *$/\1/'` if test $verbose -ge 4; then case "$linestatus" in @@ -231,15 +241,15 @@ while read -r lineon linestatus linename else ##echo "$lines" |read -r prevon prevstatus prevname line=`echo "$lines" |head -n 1` - prevon=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\1/'` - prevstatus=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\2/'` - prevname=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\3/'` + prevon=`echo "$line" |sed $E -e 's/ *(.):([A-Z]+):(.*)$/\1/'` + prevstatus=`echo "$line" |sed $E -e 's/ *(.):([A-Z]+):(.*)$/\2/'` + prevname=`echo "$line" |sed $E -e 's/ *(.):([A-Z]+):(.*)$/\3/'` if test $verbose -ge 4; then echo " debug: pulled a line from the stack" fi #lines=`echo "$lines" |tail -n $numlines` - lines=`echo "$lines" |sed -e 1d` + lines=`echo "$lines" |sed $E -e 1d` fi ;; *) ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2006-06-03 12:35 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2006-05-31 18:13 comparing DejaGNU results James Lemke 2006-05-31 18:25 ` Joe Buck 2006-05-31 18:28 ` Jeffrey Law 2006-05-31 18:33 ` James Lemke 2006-05-31 18:45 ` Joe Buck 2006-06-01 0:56 ` Ben Elliston 2006-06-01 0:59 ` Andrew Pinski 2006-06-01 2:17 ` Mike Stump 2006-06-01 1:14 ` Jim Wilson 2006-06-01 2:43 ` Mike Stump 2006-06-01 8:46 ` Richard Earnshaw 2006-06-01 17:27 ` Mike Stump 2006-06-01 19:35 ` James Lemke 2006-06-01 20:26 ` James Lemke 2006-06-01 20:36 ` Joseph S. Myers 2006-06-01 20:42 ` James Lemke 2006-06-02 18:08 ` James Lemke 2006-06-02 23:27 ` Mike Stump 2006-06-03 12:35 ` James Lemke 2006-06-01 19:44 ` James Lemke
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).