On 16 Feb 2022 07:07, Hans-Peter Nilsson wrote:
> Date: Wed, 16 Feb 2022 00:25:07 -0500 Mike Frysinger <vapier@gentoo.org>
> > On 15 Feb 2022 00:02, Hans-Peter Nilsson via Gdb-patches wrote:
> > > Commit a39487c6685f "sim: cris: use -sim with C tests for cris-elf
> > > targets" caused " -sim" to be appended to CFLAGS_FOR_TARGET for
> > > cris*-*-elf, where testing had until then relied on
> > > "RUNTESTFLAGS=--target_board=cris-sim" being passed when running "make
> > > check-sim", adding the right options.  While "-sim" happens to work,
> > > the baseboard-file cris-sim.exp uses "-sim3" so for consistency use
> > > that instead.
> > > 
> > > Then commit b42f20d2ac72 "sim: testsuite: drop most specific istarget
> > > checks" caused " -sim" to be appended for *all* targets, which just
> > > doesn't work.  For example, for crisv32-linux-gnu, that's not a
> > > recognized option and will cause a dejagnu error and further testing
> > > in c.exp will be aborted.
> > > 
> > > While cris-sim.exp appends "-static" for *-linux-gnu, further changes
> > > in the test-suite have caused "linux"-specific tests to break, so that
> > > part will be tended to separately.
> > > 
> > > But, save and restore CFLAGS_FOR_TARGET around the modification and
> > > use where needed, to not have the CRIS-specific modification affect a
> > > continuing test-run (possibly for other targets).
> > 
> > i'm trying to get away from needing dejagnu boards at all.  it brings nothing
> > to the table when it comes to testing the sim itself.
> 
> I know you're of that opinion, but I'll say this once again (it
> was decades ago last time): you're wrong.  This leads you to
> discarding half of dejagnu and working *against* it rather than
> *with it*.

i think you're overstating what it has to offer to the sim.  it can be useful
for people to define targets to run programs against (e.g. defining different
sim or hardware or remote systems), but none of that matters here.  we have
full control over the sim runtime and do not care about any of the other
aspects like running against real hardware.

> >  ideally we should have
> > a single sim binary that supports all targets simultaneously, and it's only
> > runtime options (or dynamic probing) that selects between them.  that's why
> > #progos was introduced -- so tests could declare which env they're targeting
> > and the test framework can run the simulator with the right settings.
> > 
> > one can now do:
> > $ ./configure --enable-targets=all
> > $ make check-sim
> > 
> > and every architecture will be built & tested.  no need for multiple build
> > dirs for diff targets or sep runs with diff runtestflags.
> 
> Right, I figured that's your preferred setup.  You can certainly
> make dejagnu baseboards work in this scenario: just call *each*
> of the one one that fits when testing *each* simulator, and
> clear the slate in-between.

this requires people to get their baseboards into dejagnu, and for a dejagnu
release to be made, and for these settings to be kept in sync.  releases are
way too slow on the dejagnu front.

people also tend to write their baseboards for their own setup they're using
which doesn't generalize.

`make check` should work out of the box for all targets & sim functionality
for everyone.  if they have to fumble about with runtestflags or any other
dejagnu settings, then the testsuite is doing it wrong.

if gas & ld ever get multitarget support, we'd have an even easier time with
all the xxx_FOR_TARGET_xxx settings.

again, i'm speaking only about the sim here.  something like gcc going through
dejagnu baseboards for the sim or real hardware makes sense.

> > at some point i also want to delete all the custom compile+run logic in the
> > testsuite/cris/ tree.  that's why i spent so much time pulling code out and
> > into the common one.
> 
> The baby went with the bath-water here.  (Or IOW, if you pull
> out *all* of the test-suites the testsuite will be really
> simple!)

the "baby" is the tests.  i'm not talking about deleting the tests.  there is
no reason cris should be parsing or compiling or linking or running any tests
on its own.  the functionality it wants is by no means unique to cris.

> > i even have a poc locally that deletes the dejagnu framework entirely and
> > switches to Automake test harness, but i haven't quite figured out how to
> > cleanly handle the all_machs multiplex logic in Automake.  i eventually
> > pulled out individual cleanups and merged them so at least `make check`
> > works in a multi-target build, and isn't nearly as slow as it was.
> 
> Ouch.  I haven't found the "automake test harness" to be of use
> for serious testing.

i'm not a fan of a framework that can't fathom a system with more than 1 cpu.
-mike