How to adequately test 'sim' changes?

How to adequately test 'sim' changes?

[Chris G. Demetriou]

Hi,

I've got a bunch of sim changes that I'd like to submit in the next
week or so, and I want to make sure they're tested according to
community standards before I submit them.

Therefore, i'm wondering:

(1) What's the recommended method for testing 'sim' changes?  the
sim subtree seems to have a minimal testsuite, but I was hoping for
something more substantial?

(2) which MIPS targets should be used for testing of
architecture-dependent changes?

(3) which non-MIPS targets should be used for (additional) testing of
architecture-independent changes?  (I'm not thinking anything big
here, just the need for compile/smoke-test of minor additions.)


(Obviously, everything we're going to submit works great for us in our
source tree... but issues always may arise in extracting diffs from
our tree, integrating them into the public tree, etc.  Unfortunately,
most of the 'interesting' tests we run on our local tree won't apply
to piecewise integration of our improvements into the public source
base...)


thanks!


chris

Re: How to adequately test 'sim' changes?

[Frank Ch. Eigler]

cgd@sibyte.com (Chris G. Demetriou) writes:

: [...]
: I've got a bunch of sim changes that I'd like to submit in the next
: week or so [...]

Great.

: (1) What's the recommended method for testing 'sim' changes?  the
: sim subtree seems to have a minimal testsuite, but I was hoping for
: something more substantial?

For simulators without their own tests, we normally rely on the
cross-gcc and cross-gdb tests.  For this, you'd need to build the
remainder of the cross toolchain (compiler/binutils/debugger), and
run the respective tests against the sim.

: (2) which MIPS targets should be used for testing of
: architecture-dependent changes?

AFAIK, mipstx39-elf and mips64vr5000-elf are a reasonable pair.

: (3) which non-MIPS targets should be used for (additional) testing of
: architecture-independent changes?  (I'm not thinking anything big
: here, just the need for compile/smoke-test of minor additions.) [...]

The fr30, arm, mn10300, erc32 (sparc) would give reasonably wide
coverage for build-breakage testing.

: thanks!

No, thank you!  (I presume you know about the copyright
assignment/disclaimer procedure too.)


- FChE

Re: How to adequately test 'sim' changes?

[Chris G. Demetriou]

fche@redhat.com (Frank Ch. Eigler) writes:
> : (1) What's the recommended method for testing 'sim' changes?  the
> : sim subtree seems to have a minimal testsuite, but I was hoping for
> : something more substantial?
> 
> For simulators without their own tests, we normally rely on the
> cross-gcc and cross-gdb tests.  For this, you'd need to build the
> remainder of the cross toolchain (compiler/binutils/debugger), and
> run the respective tests against the sim.

Are there any details on how to configure and run those tests against
the simulator?  I've not had much luck in that regard, to be honest,
so most of my runs of 'make check' don't actually do much...

Also, should i be building everything from a single tree, from a
single tree for binutils + debugger & separate tree for gcc, or is all
independent trees OK?


> [ targets ]

great.


> : thanks!
> 
> No, thank you!  (I presume you know about the copyright
> assignment/disclaimer procedure too.)

Yes; I (and any of other of my coworkers whose code will be submitted)
already have the proper assignment forms on file with the FSF, and
should be good to go to submit code and have it checked in once
accepted.

Getting that hairball taken care of was one of the first things I made
sure to do, when I embarked on this process.  8-)


chris

Re: How to adequately test 'sim' changes?

[Frank Ch. Eigler]

Hi -

On Fri, Dec 01, 2000 at 12:59:58PM -0800, Chris G. Demetriou wrote:
: [...]
: Are there any details on how to configure and run those tests against
: the simulator?  I've not had much luck in that regard, to be honest,
: so most of my runs of 'make check' don't actually do much...

The key is to get dejagnu to identify the correct target for the tests.
Look for the most appropriate "*-sim.exp" in $srcdir/dejagnu/baseboards,
then run tests thusly:

	cd $build/gdb
	make check RUNTESTFLAGS="--target_board=foo-sim"

: Also, should i be building everything from a single tree, from a
: single tree for binutils + debugger & separate tree for gcc, or is all
: independent trees OK?  [...]

Separately installed tools are probably fine.  In a standard Cygnus toolchain
build tree, all the various programs (gcc, binutils, etc.) are built together
as siblings.  Perhaps such a combined source tree will exist in the public
GNU repositories too before long.


- FChE
-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE6KBNQVZbdDOm/ZT0RArZbAJsHlUJLeW2clXX2dWTNYtsErLC3HgCdEq62
leQRzkqnulRhJpRYpwxcncs=
=6KRo
-----END PGP SIGNATURE-----

Re: How to adequately test 'sim' changes?

[Chris G. Demetriou]

"Frank Ch. Eigler" <fche@redhat.com> writes:
> The key is to get dejagnu to identify the correct target for the tests.
> Look for the most appropriate "*-sim.exp" in $srcdir/dejagnu/baseboards,
> then run tests thusly:
> 
> 	cd $build/gdb
> 	make check RUNTESTFLAGS="--target_board=foo-sim"

ahh!

(I don't think i'll end up with dejagnu in my source tree, since i've
already got a recent version installed, but I get the point.)



> : Also, should i be building everything from a single tree, from a
> : single tree for binutils + debugger & separate tree for gcc, or is all
> : independent trees OK?  [...]
> 
> Separately installed tools are probably fine.  In a standard Cygnus toolchain
> build tree, all the various programs (gcc, binutils, etc.) are built together
> as siblings.  Perhaps such a combined source tree will exist in the public
> GNU repositories too before long.

Yeah.

I'm thinking that i'll check out a tree from the src anoncvs
repos. with gdb, binutils, newlib, and libgloss, and then a separate
tree from the gcc anoncvs repos. with just gcc.

If that doesn't work right, i'll combine them.


thanks for your help.


chris

Re: How to adequately test 'sim' changes?

[Chris G. Demetriou]

fche@redhat.com (Frank Ch. Eigler) writes:
> : (2) which MIPS targets should be used for testing of
> : architecture-dependent changes?
> 
> AFAIK, mipstx39-elf and mips64vr5000-elf are a reasonable pair.
> 
> : (3) which non-MIPS targets should be used for (additional) testing of
> : architecture-independent changes?  (I'm not thinking anything big
> : here, just the need for compile/smoke-test of minor additions.) [...]
> 
> The fr30, arm, mn10300, erc32 (sparc) would give reasonably wide
> coverage for build-breakage testing.

FYI, what i settled on was the script below.

I've been running it on a solaris-sparc host, with a single source
tree containing:

	* binutils
	* gdb + sim
	* libgloss + newlib
	* gcc (_only_ gcc, not the additional subtrees)

("it only chews about 1.3GB per test tree!"  8-)


Anyway, the setup seems to work well for some of the boards, and of
course there are classes of optimizations, etc., that cause some of
the targets GCC tests to fall right over.


I've got a auestion about this type of testing procedure.  I've been
going under the assumption that, if a reasonable number of the GCC
'execute' tests pass, this configuration is reasonable for testing the
simulator for that target.  As it turns out, some of the
configurations _don't_ actually pass any of the execute tests
(fr30-elf and sparclite-elf lose building them, and so don't even
attempt to execute), and e.g. fr30 runs into some issues in the sim
tests.  I'm using dejagnu 1.3.1-19990614 from
ftp://gcc.gnu.org/pub/egcs/infrastructure .  Any advice on this, either
targets/boards to use, or to upgrade & use the dejagnu from CVS?




cgd
========
#!/bin/ksh

do_date() {
	echo "*** `date`: $*"
}

do_target() {

	tgt=$1
	case $target in
	mipstx39-elf)
		board=tx39-sim
		;;
	*)
		board=`echo $tgt | sed -e 's,-elf,-sim,'`
		;;
	esac

	do_date start
	mkdir BUILD.$tgt
	cd BUILD.$tgt
	../src/configure --disable-shared --disable-nls --enable-languages=c \
	    --target=$tgt --prefix=`pwd`/../INSTALL.$tgt --with-gnu-as \
	    --with-gnu-ld
	do_date configured
	gmake
	do_date made
	gmake -k check RUNTESTFLAGS="--target_board=$board"
	do_date checked
}

do_date start

for target in \
    mipstx39-elf mips64-elf fr30-elf arm-elf     mn10300-elf sparclite-elf
do
	( do_target $target ) > LOG.$target 2>&1 &
done

wait

do_date all done

Re: How to adequately test 'sim' changes?

[Andrew Cagney]

"Chris G. Demetriou" wrote:

> > : (3) which non-MIPS targets should be used for (additional) testing of
> > : architecture-independent changes?  (I'm not thinking anything big
> > : here, just the need for compile/smoke-test of minor additions.) [...]
> >
> > The fr30, arm, mn10300, erc32 (sparc) would give reasonably wide
> > coverage for build-breakage testing.

FYI,

I'm trying to prepare a list of targets that are known to cross
compile.  See:

http://sources.redhat.com/ml/gdb-patches/2000-12/msg00024.html

I've an SH script that goes with the changes :-)

	Andrew