merged dejagnu testsuites

merged dejagnu testsuites

[Frank Ch. Eigler]

Hi -

The src/testsuite/ tree now contains a unified dejagnu suite with all
pass 1 ... 5 tests in there.  I hope that with dejagnu on the case, it
will be possible to make more sophisticated judgements about whether
tests should pass/kfail, so that we can have a clean run again.

"make check" runs the old pass 1-4 tests against the build tree.
"make installcheck" runs pass 1-5 tests against the install tree,
and requires root privileges.

Please give it a try and fix what you can.  Some work is incomplete,
such as the systemtap.syscalls test bucket, which made more than the
usual number of assumptions about its invocation environment.

- FChE

RE: merged dejagnu testsuites

[Stone, Joshua I]

On Tuesday, August 29, 2006 9:05 AM, Frank Ch. Eigler wrote:
> You probably mean the new pass1-4/*.exp files.  Only a few weeks went
> by, but already I don't recall why the ">>& systemtap.log" is there
> :-). It wasn't intentional to lose the *ko error messages.

Yes I mean the pass1-4, but I'm actually worried about losing the
messages from *ok failures.

Before I wasn't waiting until the end of the testsuite run -- I waited
until I saw a failure and then hit crtl-c and looked at the logfile.

Now I see that when you let the testsuite run through completely, some
output ends up at the end of the file.  It's not all that useful though
-- I see a deluge of information from what appears to be both passing
and failing tests, with no clear way to tell what output came from what
test.  A 7MB log file is far too verbose for me to figure out what
happened.  It seems to be giving be something like this:

	Running [...]/buildok/foo.stp
	PASS: buildok/foo.stp
	Running [...]/buildok/bar.stp
	FAIL: buildok/bar.stp
	Running [...]/buildok/baz.stp
	PASS: buildok/baz.stp
	[foo's output]
	[bar's output]
	[baz's output]

I'm not even sure if all that output at the end really has the error
information I need.  For example, buildok/syscall.stp is failing (known
problem -- missing some syscalls), but I can't find anything in the log
file that tells me the failure message.

The deluge wouldn't be so bad if it was kept inline with the test
status, i.e.,

	Running [...]/buildok/foo.stp
	[foo's output]
	PASS: buildok/foo.stp
	Running [...]/buildok/bar.stp
	[bar's output]
	FAIL: buildok/bar.stp
	Running [...]/buildok/baz.stp
	[baz's output]
	PASS: buildok/baz.stp


I tried taking out your mysterious ">>& systemtap.log", and that just
made it so I don't see the output in the log at all.  So to refresh your
memory, Frank, you probably added that so the output would not be
completely lost.  ;)


Josh

Re: merged dejagnu testsuites

[Frank Ch. Eigler]

Hi -

> With the new testsuite, is there a log that captures the error output?

In general dejagnu is supposed to capture stdout/stderr into
systemtap.log.

> We used to get a testresults directory, but that's gone.  The test
> drivers look like they're trying to capture the output to
> systemtap.log, but that doesn't appear to be working.  [...]

You probably mean the new pass1-4/*.exp files.  Only a few weeks went
by, but already I don't recall why the ">>& systemtap.log" is there :-).
It wasn't intentional to lose the *ko error messages.

- FChE

RE: merged dejagnu testsuites

[Stone, Joshua I]

With the new testsuite, is there a log that captures the error output?
We used to get a testresults directory, but that's gone.  The test
drivers look like they're trying to capture the output to systemtap.log,
but that doesn't appear to be working.  I would guess that since dejagnu
is writing to that file, all the stap redirects to that file get lost.

Josh


On Friday, August 11, 2006 10:22 PM, Frank Ch. Eigler wrote:
> Hi -
> 
> The src/testsuite/ tree now contains a unified dejagnu suite with all
> pass 1 ... 5 tests in there.  I hope that with dejagnu on the case, it
> will be possible to make more sophisticated judgements about whether
> tests should pass/kfail, so that we can have a clean run again.
> 
> "make check" runs the old pass 1-4 tests against the build tree.
> "make installcheck" runs pass 1-5 tests against the install tree,
> and requires root privileges.
> 
> Please give it a try and fix what you can.  Some work is incomplete,
> such as the systemtap.syscalls test bucket, which made more than the
> usual number of assumptions about its invocation environment.
> 
> - FChE