comparing DejaGNU results

comparing DejaGNU results

[James Lemke]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

I wanted some mechanical way to compare the output of dejagnu runs
between releases, etc.  I asked a few people at the GCC Summit last year
what they used or knew about.  Not much came to light, so I ended up
writing something of my own.

It's a shell script I've been using on Linux with only trivial changes
since December.  The one tricky part was dealing with lines in the
results file that are identical.  I made the rash assumption that the
relative order wouldn't change between runs and that seems to hold.  The
script works well although it isn't blazingly fast.

My current version is attached.  If others find it useful I can
contribute it.  Comments and suggestions are welcome.

Typical invocation is something like:
dg-cmp-results.sh -v "" gcc-20060328.sum gcc-20060510.sum |less
for the first variation in the files, or
dg-cmp-results.sh -v arm-sim gcc-20060328.sum gcc-20060510.sum |less
for a specific variation.

Cheers,
Jim.

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

[-- Attachment #2: Type: application/x-shellscript, Size: 6646 bytes --]

Re: comparing DejaGNU results

[Joe Buck]

On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote:
> I wanted some mechanical way to compare the output of dejagnu runs
> between releases, etc.  I asked a few people at the GCC Summit last year
> what they used or knew about.  Not much came to light, so I ended up
> writing something of my own.

I thought that Jeff Law had something that compared .sum files back when
he was RM.  The description of what you wrote sounds similar.

Re: comparing DejaGNU results

[Jeffrey Law]

On Wed, 2006-05-31 at 11:25 -0700, Joe Buck wrote:
> On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote:
> > I wanted some mechanical way to compare the output of dejagnu runs
> > between releases, etc.  I asked a few people at the GCC Summit last year
> > what they used or knew about.  Not much came to light, so I ended up
> > writing something of my own.
> 
> I thought that Jeff Law had something that compared .sum files back when
> he was RM.  The description of what you wrote sounds similar.
Don't we have a comparison script in the "contrib" subdirectory?  

Jeff

Re: comparing DejaGNU results

[James Lemke]

On Wed, 2006-05-31 at 12:27 -0600, Jeffrey Law wrote:
> On Wed, 2006-05-31 at 11:25 -0700, Joe Buck wrote:
> > On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote:
> > > I wanted some mechanical way to compare the output of dejagnu runs
> > > between releases, etc.  I asked a few people at the GCC Summit last year
> > > what they used or knew about.  Not much came to light, so I ended up
> > > writing something of my own.
> > 
> > I thought that Jeff Law had something that compared .sum files back when
> > he was RM.  The description of what you wrote sounds similar.
> Don't we have a comparison script in the "contrib" subdirectory?  
There is contrib/compare_tests although dg-cmp-results gives more
control over what is considered "interesting".

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

Re: comparing DejaGNU results

[Joe Buck]

On Wed, May 31, 2006 at 02:33:29PM -0400, James Lemke wrote:
> On Wed, 2006-05-31 at 12:27 -0600, Jeffrey Law wrote:
> > On Wed, 2006-05-31 at 11:25 -0700, Joe Buck wrote:
> > > On Wed, May 31, 2006 at 02:13:44PM -0400, James Lemke wrote:
> > > > I wanted some mechanical way to compare the output of dejagnu runs
> > > > between releases, etc.  I asked a few people at the GCC Summit last year
> > > > what they used or knew about.  Not much came to light, so I ended up
> > > > writing something of my own.
> > > 
> > > I thought that Jeff Law had something that compared .sum files back when
> > > he was RM.  The description of what you wrote sounds similar.
> > Don't we have a comparison script in the "contrib" subdirectory?  
> There is contrib/compare_tests although dg-cmp-results gives more
> control over what is considered "interesting".

Well, I suppose people would welcome useful improvements to the contrib
script.

Re: comparing DejaGNU results

[Ben Elliston]

> > I thought that Jeff Law had something that compared .sum files back when
> > he was RM.  The description of what you wrote sounds similar.

> Don't we have a comparison script in the "contrib" subdirectory?

If that script does indeed work, then I'd like to distribute it with
DejaGnu.  There used to be a number of similar scripts in DejaGnu's
contrib directory, but none of them worked, so I removed them all.

Ben

Re: comparing DejaGNU results

[Andrew Pinski]

> 
> > > I thought that Jeff Law had something that compared .sum files back when
> > > he was RM.  The description of what you wrote sounds similar.
> 
> > Don't we have a comparison script in the "contrib" subdirectory?
> 
> If that script does indeed work, then I'd like to distribute it with
> DejaGnu.  There used to be a number of similar scripts in DejaGnu's
> contrib directory, but none of them worked, so I removed them all.

It does work the last time I used it which was last year.
It might also need to be expanded for KFAIL/KPASS for gdb.

Thanks,
Andrew Pinski

Re: comparing DejaGNU results

[Jim Wilson]

James Lemke wrote:
> I wanted some mechanical way to compare the output of dejagnu runs
> between releases, etc.

Did you look at contrib/compare_tests?  It does something very similar 
to what your script is doing.
-- 
Jim Wilson, GNU Tools Support, http://www.specifix.com

Re: comparing DejaGNU results

[Mike Stump]

On May 31, 2006, at 5:57 PM, Ben Elliston wrote:
>> Don't we have a comparison script in the "contrib" subdirectory?
>
> If that script does indeed work, then I'd like to distribute it with
> DejaGnu.

Please.  Yes, it does work and has been working just fine for the  
past decade to decade and a half.

Re: comparing DejaGNU results

[Mike Stump]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

On May 31, 2006, at 11:13 AM, James Lemke wrote:
> My current version is attached.  If others find it useful I can
> contribute it.  Comments and suggestions are welcome.

Please do.  I'd welcome it (and scripts to generate html, to track  
known issues, to trim log files, to drive things and do on)...  I  
think having a few different styles would be good, then people can  
try them all out and see which ones they like and why.  Anyway, for  
me, it isn't yet better:

dg-cmp-results.sh: Verbosity is 0, Variant is "unix"

Older log file: 06031600/gcc.sum
sed: illegal option -- r
usage: sed script [-Ean] [-i extension] [file ...]
        sed [-an] [-i extension] [-e script] ... [-f script_file] ...  
[file ...]
Newer log file: 06052905/gcc.sum
sed: illegal option -- r
usage: sed script [-Ean] [-i extension] [file ...]
        sed [-an] [-i extension] [-e script] ... [-f script_file] ...  
[file ...]
sed: illegal option -- r
usage: sed script [-Ean] [-i extension] [file ...]
        sed [-an] [-i extension] [-e script] ... [-f script_file] ...  
[file ...]
sed: illegal option -- r
usage: sed script [-Ean] [-i extension] [file ...]
        sed [-an] [-i extension] [-e script] ... [-f script_file] ...  
[file ...]
sort: unrecognized option `--'
Try `sort --help' for more information.

:-(

When I fixed it up like this:


[-- Attachment #2: t.txt --]
[-- Type: text/plain, Size: 2392 bytes --]

Doing diffs in /Network/Servers/madrid/Volumes/huis/mstump/bin/dg-cmp-results.~1~:
--- /Network/Servers/madrid/Volumes/huis/mstump/bin/dg-cmp-results.~1~	2006-05-31 15:57:42.000000000 -0700
+++ /Network/Servers/madrid/Volumes/huis/mstump/bin/dg-cmp-results	2006-05-31 17:24:21.000000000 -0700
@@ -42,6 +42,8 @@ OFILE="$2"
 OBASE=`basename "$2"`
 NFILE="$3"
 NBASE=`basename "$3"`
+#export R=-r
+export R=-E
 
 echo "dg-cmp-results.sh: Verbosity is ${verbose}, Variant is \"${VARIANT}\""
 echo
@@ -62,14 +64,14 @@ unset temp
 
 # Copy out the old file's section 0.
 echo "Older log file: $OFILE"
-sed -r -e '/^\t+===/,$d' $OFILE
+sed $R -e '/^\t+===/,$d' $OFILE
 
 # Copy out the new file's section 0.
 echo "Newer log file: $NFILE"
-sed -r -e '/^\t+===/,$d' $NFILE
+sed $R -e '/^\t+===/,$d' $NFILE
 
 # Create a temporary file from the old file's interesting section.
-sed -r -e "1,/$header/d" \
+sed $R -e "1,/$header/d" \
   -e '/^\t+===/,$d' \
   -e '/^[A-Z]+:/!d' \
   -e 's/\r$//' \
@@ -79,7 +81,7 @@ sed -r -e "1,/$header/d" \
   >/tmp/o$$-$OBASE
 
 # Create a temporary file from the new file's interesting section.
-sed -r -e "1,/$header/d" \
+sed $R -e "1,/$header/d" \
   -e '/^\t+===/,$d' \
   -e '/^[A-Z]+:/!d' \
   -e 's/\r$//' \
@@ -94,9 +96,9 @@ sed -r -e "1,/$header/d" \
 # If so, we assume that the order is the same in both files.
 IFS=:
 firstread=Y
-sort --merge -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE |
+sort -m -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE |
 while read -r lineon linestatus linename; do
-    linename=`echo "$linename" |sed -r -e 's/^ *(.*) *$/\1/'`
+    linename=`echo "$linename" |sed $R -e 's/^ *(.*) *$/\1/'`
 
     if test $verbose -ge 4; then
 	case "$linestatus" in
@@ -231,9 +233,9 @@ while read -r lineon linestatus linename
 	    else
 		##echo "$lines" |read -r prevon prevstatus prevname
 		line=`echo "$lines" |head -n 1`
-		prevon=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\1/'`
-		prevstatus=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\2/'`
-		prevname=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\3/'`
+		prevon=`echo "$line" |sed $R -e 's/\s*(.):([A-Z]+):(.*)$/\1/'`
+		prevstatus=`echo "$line" |sed $R -e 's/\s*(.):([A-Z]+):(.*)$/\2/'`
+		prevname=`echo "$line" |sed $R -e 's/\s*(.):([A-Z]+):(.*)$/\3/'`
 
 		if test $verbose -ge 4; then
 		    echo "  debug: pulled a line from the stack"
--------------

[-- Attachment #3: Type: text/plain, Size: 2760 bytes --]


I got:

Script error: trying to process:
          O: PASS: objc.dg/bad-receiver-type.m  (test for warnings,  
line 13)
         N:PASS:objc.dg/bad-receiver-type.m  (test for warnings, line  
13)

:-(

mrs $ grep 'objc.dg/bad-receiver-type.m'  06031600/objc.sum 06052905/ 
objc.sum
06031600/objc.sum:PASS: objc.dg/bad-receiver-type.m  (test for  
warnings, line 13)
06031600/objc.sum:PASS: objc.dg/bad-receiver-type.m  (test for  
warnings, line 13)
06031600/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for excess  
errors)
06052905/objc.sum:PASS: objc.dg/bad-receiver-type.m  (test for  
warnings, line 13)
06052905/objc.sum:PASS: objc.dg/bad-receiver-type.m  (test for  
warnings, line 13)
06052905/objc.sum:PASS: objc.dg/bad-receiver-type.m (test for excess  
errors)

Mine btw gives:

mrs $ cmp_logs  06031600/objc.sum 06052905/objc.sum
New tests that PASS:

objc.dg/dwarf-2.m (test for excess errors)
objc.dg/dwarf-2.m scan-assembler 0x10[^0-9a-f].*DW_AT_language
objc.dg/member-1.m  (test for errors, line 3)
objc.dg/member-1.m (test for excess errors)

Old tests that passed, that have disappeared: (Eeek!)

objc.dg/stubify-2.m scan-rtl-dump-not symbol_ref.*"objc_msgSend"


I tried all the results and couldn't find any that your script could  
analyze, except libgomp, but only because libgomp doesn't run on my  
system, so there are no results in it?  Is it just me?  I hand edited  
my results to be the first 30 or so from the Objective C++ testsuite,  
and then I got it to analyze and not dump out.

When I tried gcc, I had a chance to notice the timings, your version:

real    8m44.413s
user    2m0.714s
sys     7m54.847s

mine:

real    0m1.994s
user    0m1.756s
sys     0m0.556s

:-)  Maybe you only run the script on toy results?  Or, do you just  
drink lots of coffee?  Now, I know mine is well more than 10-100x  
slower than it could be, but waiting 2 seconds isn't a hardship for  
me.  Waiting 8 minutes strikes me as just a little slow.

The output of yours doesn't seem to be targeted for human eyes, the  
verbosity (at the least verbose setting) is about 121x more than mine  
for two random sets I had lying around, and that is with it cutting  
the output off really early due to the above problem.  I predict that  
in normal day to day use, it is well better than 120,000x larger.   
What use do you get out of it?

Mine was designed to do two things, figure out if the results are  
interesting and not send email, if they are not, and to show  
engineers the `interesting' detailed results in priority order.  It's  
meant to be run daily, and on good days, it produces no output.  On  
normal days, output is 20-30 lines at most.  It tries to balance the  
complaints with the occasional atta boy, to help with moral.

Re: comparing DejaGNU results

[Richard Earnshaw]

[-- Attachment #1: Type: text/plain, Size: 1452 bytes --]

On Thu, 2006-06-01 at 03:43, Mike Stump wrote:

> Mine was designed to do two things, figure out if the results are  
> interesting and not send email, if they are not, and to show  
> engineers the `interesting' detailed results in priority order.  It's  
> meant to be run daily, and on good days, it produces no output.  On  
> normal days, output is 20-30 lines at most.  It tries to balance the  
> complaints with the occasional atta boy, to help with moral.

The only problem I have with Mike's script is that it doesn't handle
runs with multiple multi-lib variants nicely.  Attached is a local hack
that I have to address this, but you'll need to re-seed your reference
files if you apply it.

With the change the output generated is not something like

Tests that now work, but didn't before:

arm-sim/-mthumb: g++.old-deja/g++.oliva/template9.C  (test for errors, line 10)
arm-sim: g++.old-deja/g++.oliva/template9.C  (test for errors, line 10)

New tests that FAIL:

arm-sim/-mthumb: g++.dg/init/const3.C (test for excess errors)
arm-sim/-mthumb: g++.dg/opt/temp2.C (test for excess errors)
arm-sim/-mthumb: g++.dg/other/unused1.C scan-assembler stringz?\t"class2"
arm-sim/-mthumb: g++.dg/other/unused1.C scan-assembler stringz?\t"printer"
arm-sim/-mthumb: g++.dg/template/fntry1.C (test for excess errors)
arm-sim: g++.dg/other/unused1.C scan-assembler stringz?\t"class2"
arm-sim: g++.dg/other/unused1.C scan-assembler stringz?\t"printer"


[-- Attachment #2: compare.patch --]
[-- Type: text/plain, Size: 580 bytes --]

--- /home/rearnsha/gnusrc/gcc/trunk/contrib/compare_tests	2005-11-12 16:36:57.000000000 +0000
+++ compare_gcc_tests	2006-01-24 18:49:05.000000000 +0000
@@ -22,8 +22,8 @@
 	exit 2
 fi
 
-sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" >$tmp1
-sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" >$tmp2
+sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp1
+sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp2
 
 before=$tmp1
 now=$tmp2

Re: comparing DejaGNU results

[Mike Stump]

On Jun 1, 2006, at 1:45 AM, Richard Earnshaw wrote:
> The only problem I have with Mike's script is that it doesn't handle
> runs with multiple multi-lib variants nicely.

Yeah, in the past, we drove the multilib pass from above as in  
general we had to select different hardware for testing.  I like the  
idea.

> Attached is a local hack that I have to address this

Let's see if anyone can spot any non-portabilities in it.  I think it  
is safe.  If safe, I'd say let's add this, as it helps.

I do have a slight additional change:

--- cmp_logs.~2~        2006-06-01 09:41:12.000000000 -0700
+++ cmp_logs    2006-06-01 10:05:56.000000000 -0700
@@ -25,2 +25,2 @@
-sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running  
target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp1
-sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running  
target / {target = $3} {sub(/: /, "&"target": " ); print $0;}' >$tmp2
+sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running  
target / {target = $3} { if (target != "unix") { sub(/: /,  
"&"target": " ); }; print $0; }' >$tmp1
+sed 's/^XFAIL/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running  
target / {target = $3} { if (target != "unix") { sub(/: /,  
"&"target": " ); }; print $0; }' >$tmp2

This preserves the simplicity in the non-multilib case and keeps  
future results comparable with older results.

Re: comparing DejaGNU results

[James Lemke]

> Please do.  I'd welcome it (and scripts to generate html, to track  
> known issues, to trim log files, to drive things and do on)...  I  
> think having a few different styles would be good, then people can  
> try them all out and see which ones they like and why.  Anyway, for  
> me, it isn't yet better:
:
> I tried all the results and couldn't find any that your script could  
> analyze, except libgomp, but only because libgomp doesn't run on my  
> system, so there are no results in it?  Is it just me?  I hand edited  
> my results to be the first 30 or so from the Objective C++ testsuite,  
> and then I got it to analyze and not dump out.
> 
> When I tried gcc, I had a chance to notice the timings, your version:
> 
> real    8m44.413s
> user    2m0.714s
> sys     7m54.847s
> 
> mine:
> 
> real    0m1.994s
> user    0m1.756s
> sys     0m0.556s

As I said I had only used it on Linux, I assume you're using Darwin.
I tried it there and had to fix the following.  You got the first two
but not the last so my script was way non-functional.
sed uses -E instead of -r (as you noted)
sort uses -m but not --merge (as you noted)
sed doesn't honour \t or \s

> :-)  Maybe you only run the script on toy results?  Or, do you just  
> drink lots of coffee?  Now, I know mine is well more than 10-100x  
> slower than it could be, but waiting 2 seconds isn't a hardship for  
> me.  Waiting 8 minutes strikes me as just a little slow.
I said it was slow :-)>
On NetBSD your approach is at least 50x faster.  Strangely, on FC3 the
two have similar elapsed times:

time ./compare_tests gcc-sim-20050503.sum gcc-sim-20060528.sum >compare2.lst
real    4m0.729s
user    4m5.880s
sys     0m0.231s

time ./dg-cmp-results.sh -v "" gcc-sim-20050503.sum gcc-sim-20060528.sum >dg-cmp2.lst
real    4m25.941s
user    1m29.850s
sys     3m8.021s

> The output of yours doesn't seem to be targeted for human eyes, the  
> verbosity (at the least verbose setting) is about 121x more than mine  
> for two random sets I had lying around, and that is with it cutting  
> the output off really early due to the above problem.  I predict that  
> in normal day to day use, it is well better than 120,000x larger.   
> What use do you get out of it?
It obviously wasn't working properly for you due to linux - netbsd
differences.
wc -l *2.lst
  66 compare2.lst
  63 dg-cmp2.lst

Results from mine should have looked like:
dg-cmp-results.sh: Verbosity is 1, Variant is ""

Older log file: gcc-sim-20050503.sum
Test Run By jim on Mon May  2 12:29:08 2005
Target is xscale-unknown-elf
Host   is i686-pc-linux-gnu

Newer log file: gcc-sim-20060528.sum
Test Run By jim on Mon May 29 12:55:01 2006
Target is xscale-unknown-elf
Host   is i686-pc-linux-gnu

PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation,  -O1
PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation,  -O2
PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation,  -O3 -fomit-frame-pointer
PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation,  -O3 -g
PASS->FAIL: gcc.c-torture/execute/920428-2.c compilation,  -Os
PASS->UNRESOLVED: gcc.c-torture/execute/920428-2.c execution,  -O1
PASS->UNRESOLVED: gcc.c-torture/execute/920428-2.c execution,  -O2
:
PASS->UNRESOLVED: gcc.c-torture/execute/nestfunc-6.c execution,  -Os
FAIL->PASS: gcc.dg/20030324-1.c execution test
FAIL->PASS: gcc.dg/debug/dwarf2/dwarf-die7.c scan-assembler 1.*DW_AT_inline
FAIL->PASS: gcc.dg/range-test-1.c execution test

> Mine was designed to do two things, figure out if the results are  
> interesting and not send email, if they are not, and to show  
> engineers the `interesting' detailed results in priority order.  It's  
> meant to be run daily, and on good days, it produces no output.  On  
> normal days, output is 20-30 lines at most.  It tries to balance the  
> complaints with the occasional atta boy, to help with moral.
Mine is used at the end of full build & test runs so a few minutes
didn't bother me.  A look at output like that above tells me what I
need.

Your approach is faster, esp. on Darwin / NetBSD.
The only advantages I see to mine is handling variants (Richard's patch
fixes that), verbosity control, and detail -- compare_tests only looks
at X?(PASS|X?FAIL).

Jim.

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

Re: comparing DejaGNU results

[James Lemke]

[-- Attachment #1: Type: text/plain, Size: 131 bytes --]

Whoops... I forgot to attach my fixes, for anyone that's interested.

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

[-- Attachment #2: Type: text/x-patch, Size: 2652 bytes --]

--- dg-cmp-results.sh	2006/05/31 19:22:14	1.18
+++ dg-cmp-results.sh	2006/06/01 17:53:21
@@ -31,6 +31,16 @@ if test $# -ne 3 -o ! -f "$2" -o ! -f "$
     exit 1
 fi
 
+# Command differences for various platforms.
+case `uname -s` in
+Darwin|NetBSD)
+    E=-E	# sed
+    ;;
+*)
+    E=-r	# sed
+    ;;
+esac
+
 # sections are identified by separator lines beginning with '\t\t==='.
 # section 0 identifies run date, target, and host.
 # section 1 and subsequent contain test data for a target variant.
@@ -62,15 +72,15 @@ unset temp
 
 # Copy out the old file's section 0.
 echo "Older log file: $OFILE"
-sed -r -e '/^\t+===/,$d' $OFILE
+sed $E -e '/^[[:space:]]+===/,$d' $OFILE
 
 # Copy out the new file's section 0.
 echo "Newer log file: $NFILE"
-sed -r -e '/^\t+===/,$d' $NFILE
+sed $E -e '/^[[:space:]]+===/,$d' $NFILE
 
 # Create a temporary file from the old file's interesting section.
-sed -r -e "1,/$header/d" \
-  -e '/^\t+===/,$d' \
+sed $E -e "1,/$header/d" \
+  -e '/^[[:space:]]+===/,$d' \
   -e '/^[A-Z]+:/!d' \
   -e 's/\r$//' \
   -e 's/^/O:/' \
@@ -79,8 +89,8 @@ sed -r -e "1,/$header/d" \
   >/tmp/o$$-$OBASE
 
 # Create a temporary file from the new file's interesting section.
-sed -r -e "1,/$header/d" \
-  -e '/^\t+===/,$d' \
+sed $E -e "1,/$header/d" \
+  -e '/^[[:space:]]+===/,$d' \
   -e '/^[A-Z]+:/!d' \
   -e 's/\r$//' \
   -e 's/^/N:/' \
@@ -94,9 +104,9 @@ sed -r -e "1,/$header/d" \
 # If so, we assume that the order is the same in both files.
 IFS=:
 firstread=Y
-sort --merge -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE |
+sort -m -s -t : -k 3b /tmp/o$$-$OBASE /tmp/n$$-$NBASE |
 while read -r lineon linestatus linename; do
-    linename=`echo "$linename" |sed -r -e 's/^ *(.*) *$/\1/'`
+    linename=`echo "$linename" |sed $E -e 's/^ *(.*) *$/\1/'`
 
     if test $verbose -ge 4; then
 	case "$linestatus" in
@@ -231,15 +241,15 @@ while read -r lineon linestatus linename
 	    else
 		##echo "$lines" |read -r prevon prevstatus prevname
 		line=`echo "$lines" |head -n 1`
-		prevon=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\1/'`
-		prevstatus=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\2/'`
-		prevname=`echo "$line" |sed -r -e 's/\s*(.):([A-Z]+):(.*)$/\3/'`
+		prevon=`echo "$line" |sed $E -e 's/ *(.):([A-Z]+):(.*)$/\1/'`
+		prevstatus=`echo "$line" |sed $E -e 's/ *(.):([A-Z]+):(.*)$/\2/'`
+		prevname=`echo "$line" |sed $E -e 's/ *(.):([A-Z]+):(.*)$/\3/'`
 
 		if test $verbose -ge 4; then
 		    echo "  debug: pulled a line from the stack"
 		fi
 		#lines=`echo "$lines" |tail -n $numlines`
-		lines=`echo "$lines" |sed -e 1d`
+		lines=`echo "$lines" |sed $E -e 1d`
 	    fi
 	    ;;
 	*)

Re: comparing DejaGNU results

[James Lemke]

> Your approach is faster, esp. on Darwin / NetBSD.
> The only advantages I see to mine is handling variants (Richard's patch
> fixes that), verbosity control, and detail -- compare_tests only looks
> at X?(PASS|FAIL).

Hmm.. another small point, FWIW.

Both the results files I used contained the following ssequence of
results:
PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)

compare_tests reported the following:
Tests that now fail, but worked before:
gcc.c-torture/compile/930210-1.c (test for excess errors)
gcc.c-torture/compile/930210-1.c (test for excess errors)
gcc.c-torture/compile/930210-1.c (test for excess errors)
:
Tests that now work, but didn't before:
gcc.c-torture/compile/930210-1.c (test for excess errors)
gcc.c-torture/compile/930210-1.c (test for excess errors)
gcc.c-torture/compile/930210-1.c (test for excess errors)

dg-cmp-results didn't report anything (at that verbosity) because
nothing had changed.

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

Re: comparing DejaGNU results

[Joseph S. Myers]

On Thu, 1 Jun 2006, James Lemke wrote:

> Both the results files I used contained the following ssequence of
> results:
> PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
> PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
> FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
> FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)

This probably means you are using too old a version of DejaGnu; make sure 
you are using at least 1.4.4, the minimum supported version.  There are 
still problems with duplicate test assertion names (e.g. bug 20771), but 
there shouldn't be in c-torture if you use current DejaGnu.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: comparing DejaGNU results

[James Lemke]

> > Both the results files I used contained the following ssequence of
> > results:
> > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > PASS: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> > FAIL: gcc.c-torture/compile/930210-1.c (test for excess errors)
> 
> This probably means you are using too old a version of DejaGnu; make sure 
> you are using at least 1.4.4, the minimum supported version.  There are 
> still problems with duplicate test assertion names (e.g. bug 20771), but 
> there shouldn't be in c-torture if you use current DejaGnu.

Thanks.
Regardless, dg-cmp-results handles that situation, albeit slowly  ; - )>

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

Re: comparing DejaGNU results

[James Lemke]

[-- Attachment #1: Type: text/plain, Size: 368 bytes --]

I took a quick pass at implementing the comparisons in a more suitable
lanugage.  Run time is now a few seconds on both platforms.  About the
same as compare_tests on my old ibook/OSX and much faster on FC3.
Trials show the same results as before.

For anyone interested, the new version is attached.
Jim.

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario

[-- Attachment #2: Type: application/x-shellscript, Size: 5100 bytes --]

Re: comparing DejaGNU results

[Mike Stump]

On Jun 2, 2006, at 11:08 AM, James Lemke wrote:
> I took a quick pass at implementing the comparisons in a more suitable
> lanugage.  Run time is now a few seconds on both platforms.  About the
> same as compare_tests on my old ibook/OSX and much faster on FC3.

Since Ben and I seem interested in this, I think we should check in  
this version.  It seems portable enough and useful enough.  Any  
objections from the crowd?

Re: comparing DejaGNU results

[James Lemke]

On Fri, 2006-06-02 at 16:26 -0700, Mike Stump wrote:
> On Jun 2, 2006, at 11:08 AM, James Lemke wrote:
> > I took a quick pass at implementing the comparisons in a more suitable
> > lanugage.  Run time is now a few seconds on both platforms.  About the
> > same as compare_tests on my old ibook/OSX and much faster on FC3.
> 
> Since Ben and I seem interested in this, I think we should check in  
> this version.  It seems portable enough and useful enough.  Any  
> objections from the crowd?

If not, I'll commit it to contrib/ on Monday.

-- 
Jim Lemke   jwlemke@wasabisystems.com   Orillia, Ontario