Re: Automated build-and-test summary report (2007/06/18)

Re: Automated build-and-test summary report (2007/06/18)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

Hi,
(From the test-results list)

On Mon, 2007-06-18 at 05:35 -0700, Kris Van Hees wrote:
> For more detailed results, please check out the reports at:
> 
>     http://build.alchar.org/~aedil/rep/
>  [...]
>   Old   -> New   Test
>   -----    ----- ----------------------------------
>   PASS  -> FAIL  frysk-core - testStepOver(frysk.rt.TestSteppingEngine)
>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferMapOverrun(frysk.proc.corefile.TestCorefileByteBuffer)
>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPoke(frysk.proc.corefile.TestCorefileByteBuffer)
>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPeek(frysk.proc.corefile.TestCorefileByteBuffer)
>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPeekArray(frysk.proc.corefile.TestCorefileByteBuffer)
>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer)
>   FAIL  -> PASS  frysk-core - testSliceAddressSpace(frysk.proc.ptrace.TestByteBuffer)
>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferSequentialGet(frysk.proc.corefile.TestCorefileByteBuffer)

The core file tests did once fail for me with a pending SIGUSR1 during
teardown. But I cannot replicate it now locally. It seems to be some
kind of race during TestLib cleanup. Kris, are the full test results
logs also available? I couldn't find them easily, and having the
backtrace available would help with tracking these down.

Thanks,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Automated build-and-test summary report (2007/06/18)

[Kris Van Hees]

On Mon, Jun 18, 2007 at 02:44:37PM +0200, Mark Wielaard wrote:
> Hi,
> (From the test-results list)
> 
> On Mon, 2007-06-18 at 05:35 -0700, Kris Van Hees wrote:
> > For more detailed results, please check out the reports at:
> > 
> >     http://build.alchar.org/~aedil/rep/
> >  [...]
> >   Old   -> New   Test
> >   -----    ----- ----------------------------------
> >   PASS  -> FAIL  frysk-core - testStepOver(frysk.rt.TestSteppingEngine)
> >   PASS  -> FAIL  frysk-core - testCoreFileByteBufferMapOverrun(frysk.proc.corefile.TestCorefileByteBuffer)
> >   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPoke(frysk.proc.corefile.TestCorefileByteBuffer)
> >   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPeek(frysk.proc.corefile.TestCorefileByteBuffer)
> >   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPeekArray(frysk.proc.corefile.TestCorefileByteBuffer)
> >   PASS  -> FAIL  frysk-core - testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer)
> >   FAIL  -> PASS  frysk-core - testSliceAddressSpace(frysk.proc.ptrace.TestByteBuffer)
> >   PASS  -> FAIL  frysk-core - testCoreFileByteBufferSequentialGet(frysk.proc.corefile.TestCorefileByteBuffer)
> 
> The core file tests did once fail for me with a pending SIGUSR1 during
> teardown. But I cannot replicate it now locally. It seems to be some
> kind of race during TestLib cleanup. Kris, are the full test results
> logs also available? I couldn't find them easily, and having the
> backtrace available would help with tracking these down.

Hm, I can see how easy it is to include links for the individual latest
builds to help with this.  In this case, if you click on the score (X /
Y number in the grid) for the specific host-package combination you get
the specific build result page.  It has a download link for the build
log in the upper right section of the page.

	Kris

Re: Automated build-and-test summary report (2007/06/18)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On Mon, 2007-06-18 at 06:22 -0700, Kris Van Hees wrote:
> Hm, I can see how easy it is to include links for the individual latest
> builds to help with this.  In this case, if you click on the score (X /
> Y number in the grid) for the specific host-package combination you get
> the specific build result page.  It has a download link for the build
> log in the upper right section of the page.

It seems the build log only contains the harness output of setting up
the rpms, build infrastructure, running the testharness, but not the
actual make or make check output. Or am I looking for the wrong thing,
at the wrong place? Following your instructions I end up with:
http://build.alchar.org/~aedil/rep/coldstone/dl/frysk_dist.20070618-041627.log.bz2

In particular I would like to have the make check logs for the above new
FAILs in this message:
http://sourceware.org/ml/frysk-testresults/2007-q2/msg00017.html
What would be the URL for those?

Thanks,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Automated build-and-test summary report (2007/06/18)

[Phil Muldoon]

Mark Wielaard wrote:
> Hi,
> (From the test-results list)
>
> On Mon, 2007-06-18 at 05:35 -0700, Kris Van Hees wrote:
>   
>> For more detailed results, please check out the reports at:
>>
>>     http://build.alchar.org/~aedil/rep/
>>  [...]
>>   Old   -> New   Test
>>   -----    ----- ----------------------------------
>>   PASS  -> FAIL  frysk-core - testStepOver(frysk.rt.TestSteppingEngine)
>>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferMapOverrun(frysk.proc.corefile.TestCorefileByteBuffer)
>>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPoke(frysk.proc.corefile.TestCorefileByteBuffer)
>>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPeek(frysk.proc.corefile.TestCorefileByteBuffer)
>>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferPeekArray(frysk.proc.corefile.TestCorefileByteBuffer)
>>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer)
>>   FAIL  -> PASS  frysk-core - testSliceAddressSpace(frysk.proc.ptrace.TestByteBuffer)
>>   PASS  -> FAIL  frysk-core - testCoreFileByteBufferSequentialGet(frysk.proc.corefile.TestCorefileByteBuffer)
>>     
>
> The core file tests did once fail for me with a pending SIGUSR1 during
> teardown.
The test pass for me on Fedora 7, with:

./TestRunner -r 1000 frysk.proc.corefile.TestCorefileByteBuffer

Running 
testCorefileByteBufferSlice(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS
Running 
testCoreFileByteBufferPeek(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS
Running 
testCoreFileByteBufferMapOverrun(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS
Running 
testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS
Running 
testCoreFileByteBufferSequentialGet(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS
Running 
testCoreFileByteBufferPeekArray(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS
Running 
testCoreFileByteBufferPoke(frysk.proc.corefile.TestCorefileByteBuffer) 
...PASS

Time: 0.056

OK (7 tests)


And it does that for * 1000

in make check I do see the intermittent  test fails due to

1) 
testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer)junit.framework.AssertionFailedError: 
pending signal Sig_USR1


But all these tests are not emitting or listening for signals, I've no 
idea where that signal is coming from, or how it found it's way into the 
test.

However when the corefile tests pass in make check, I see some other 
test fails such as the one below, come and go:

There were 6 failures:
1) 
testSliceAddressSpace(frysk.proc.ptrace.TestByteBuffer)junit.framework.AssertionFailedError: 
unexpected signal Sig_IO
   at frysk.testbed.AttachedSelf$2.signal(TestRunner)
   at frysk.sys.Wait.wait(TestRunner)
   at frysk.sys.Wait.wait(TestRunner)
   at frysk.testbed.AttachedSelf.<init>(TestRunner)
   at frysk.proc.ptrace.TestByteBuffer.setUp(TestRunner)
   at frysk.junit.Runner.runCases(TestRunner)
   at frysk.junit.Runner.runArchCases(TestRunner)
   at frysk.junit.Runner.runTestCases(TestRunner)
   at TestRunner.main(TestRunner)

Re: Automated build-and-test summary report (2007/06/18)

[Kris Van Hees]

If you look at the frysk_dist logs, indeed there will be no test harness
output because building from a 'make dist' tree simply doesn't work, and
hasn't been working for several months now.

Make sure you look for the build of the correct 'package', e.g.
frysk_fresh or frysk_incr.  The summary email mentions the package that
the reported test result diffs belong to in the header above the listed
diff.

So, for the FAILs you are referring to, the original email to
frysk-testresults states that this was for package frysk_incr on
coldstone.us.oracle.com, so when you look in the frysk_incr grid, take
the row for coldstone.us.oracle.com, and the last build in that row (for
Jun 18th in fact) is the one you want.

For reference, here is the link to that particular build result page:
http://build.alchar.org/~aedil/rep/coldstone/frysk_incr.20070618-033404.html

And here is the log:
http://build.alchar.org/~aedil/rep/coldstone/dl/frysk_incr.20070618-033404.log.bz2

	Cheers,
	Kris

On Mon, Jun 18, 2007 at 04:56:18PM +0200, Mark Wielaard wrote:
> On Mon, 2007-06-18 at 06:22 -0700, Kris Van Hees wrote:
> > Hm, I can see how easy it is to include links for the individual latest
> > builds to help with this.  In this case, if you click on the score (X /
> > Y number in the grid) for the specific host-package combination you get
> > the specific build result page.  It has a download link for the build
> > log in the upper right section of the page.
> 
> It seems the build log only contains the harness output of setting up
> the rpms, build infrastructure, running the testharness, but not the
> actual make or make check output. Or am I looking for the wrong thing,
> at the wrong place? Following your instructions I end up with:
> http://build.alchar.org/~aedil/rep/coldstone/dl/frysk_dist.20070618-041627.log.bz2
> 
> In particular I would like to have the make check logs for the above new
> FAILs in this message:
> http://sourceware.org/ml/frysk-testresults/2007-q2/msg00017.html
> What would be the URL for those?
> 
> Thanks,
> 
> Mark

Re: Automated build-and-test summary report (2007/06/18)

[Kris Van Hees]

As reported before on IRC, I am fairly certain that the signals
triggering the failure are actually coming from an earlier test and not
being cleaned up correctly between tests, i.e. a problem with either the
test harness code or some earlier test.  That's also why running just this
particular test tons of time won't bring the problem to light because
nothing is being executed that even posts a SIGUSR1.

	Cheers,
	Kris

On Mon, Jun 18, 2007 at 09:59:12AM -0500, Phil Muldoon wrote:
> Mark Wielaard wrote:
> >Hi,
> >(From the test-results list)
> >
> >On Mon, 2007-06-18 at 05:35 -0700, Kris Van Hees wrote:
> >  
> >>For more detailed results, please check out the reports at:
> >>
> >>    http://build.alchar.org/~aedil/rep/
> >> [...]
> >>  Old   -> New   Test
> >>  -----    ----- ----------------------------------
> >>  PASS  -> FAIL  frysk-core - testStepOver(frysk.rt.TestSteppingEngine)
> >>  PASS  -> FAIL  frysk-core - 
> >>  testCoreFileByteBufferMapOverrun(frysk.proc.corefile.TestCorefileByteBuffer)
> >>  PASS  -> FAIL  frysk-core - 
> >>  testCoreFileByteBufferPoke(frysk.proc.corefile.TestCorefileByteBuffer)
> >>  PASS  -> FAIL  frysk-core - 
> >>  testCoreFileByteBufferPeek(frysk.proc.corefile.TestCorefileByteBuffer)
> >>  PASS  -> FAIL  frysk-core - 
> >>  testCoreFileByteBufferPeekArray(frysk.proc.corefile.TestCorefileByteBuffer)
> >>  PASS  -> FAIL  frysk-core - 
> >>  testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer)
> >>  FAIL  -> PASS  frysk-core - 
> >>  testSliceAddressSpace(frysk.proc.ptrace.TestByteBuffer)
> >>  PASS  -> FAIL  frysk-core - 
> >>  testCoreFileByteBufferSequentialGet(frysk.proc.corefile.TestCorefileByteBuffer)
> >>    
> >
> >The core file tests did once fail for me with a pending SIGUSR1 during
> >teardown.
> The test pass for me on Fedora 7, with:
> 
> ./TestRunner -r 1000 frysk.proc.corefile.TestCorefileByteBuffer
> 
> Running 
> testCorefileByteBufferSlice(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> Running 
> testCoreFileByteBufferPeek(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> Running 
> testCoreFileByteBufferMapOverrun(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> Running 
> testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> Running 
> testCoreFileByteBufferSequentialGet(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> Running 
> testCoreFileByteBufferPeekArray(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> Running 
> testCoreFileByteBufferPoke(frysk.proc.corefile.TestCorefileByteBuffer) 
> ...PASS
> 
> Time: 0.056
> 
> OK (7 tests)
> 
> 
> And it does that for * 1000
> 
> in make check I do see the intermittent  test fails due to
> 
> 1) 
> testCoreFileByteBufferMapUnderrun(frysk.proc.corefile.TestCorefileByteBuffer)junit.framework.AssertionFailedError: 
> pending signal Sig_USR1
> 
> 
> But all these tests are not emitting or listening for signals, I've no 
> idea where that signal is coming from, or how it found it's way into the 
> test.
> 
> However when the corefile tests pass in make check, I see some other 
> test fails such as the one below, come and go:
> 
> There were 6 failures:
> 1) 
> testSliceAddressSpace(frysk.proc.ptrace.TestByteBuffer)junit.framework.AssertionFailedError: 
> unexpected signal Sig_IO
>   at frysk.testbed.AttachedSelf$2.signal(TestRunner)
>   at frysk.sys.Wait.wait(TestRunner)
>   at frysk.sys.Wait.wait(TestRunner)
>   at frysk.testbed.AttachedSelf.<init>(TestRunner)
>   at frysk.proc.ptrace.TestByteBuffer.setUp(TestRunner)
>   at frysk.junit.Runner.runCases(TestRunner)
>   at frysk.junit.Runner.runArchCases(TestRunner)
>   at frysk.junit.Runner.runTestCases(TestRunner)
>   at TestRunner.main(TestRunner)
>