Automated build-and-test summary report (2007/05/23)

Automated build-and-test summary report (2007/05/23)

[Kris Van Hees]

------------------------------------------------------------------------------
Package:        frysk_fresh
Previous build: 2007/05/22 03:34:03 - 2007/05/22 03:56:56
Current build:  2007/05/23 03:34:02 - 2007/05/23 03:56:48
Host:           coldstone.us.oracle.com
Platform:       Linux 2.6.20-1.2948.fc6 (FC6)
First failure:  test
Test result:    602: 556 +, 12 -, 8 S, 26 E (was 599: 553 +, 12 -, 8 S, 26 E)

  Old   -> New   Test
  -----    ----- ----------------------------------
  PASS  -> ERROR ADDED (NEW) TESTS:
  PASS  -> ERROR frysk-core - testAsyncAddressSpace(frysk.proc.ptrace.TestByteBuffer)
  PASS  -> ERROR frysk-core - testAbandonThenRemoveAttached(frysk.proc.TestAbandon)
  PASS  -> ERROR frysk-core - testSliceRegisterSet(frysk.proc.ptrace.TestByteBuffer)

------------------------------------------------------------------------------
Package:        frysk_incr
Previous build: 2007/05/22 03:57:09 - 2007/05/22 04:12:00
Current build:  2007/05/23 03:57:00 - 2007/05/23 04:12:03
Host:           coldstone.us.oracle.com
Platform:       Linux 2.6.20-1.2948.fc6 (FC6)
First failure:  test
Test result:    602: 556 +, 12 -, 8 S, 26 E (was 599: 551 +, 14 -, 8 S, 26 E)

  Old   -> New   Test
  -----    ----- ----------------------------------
  FAIL  -> PASS  frysk-imports - testShortTimeout(frysk.sys.TestWait)
  PASS  -> ERROR ADDED (NEW) TESTS:
  PASS  -> ERROR frysk-core - testAsyncAddressSpace(frysk.proc.ptrace.TestByteBuffer)
  FAIL  -> PASS  frysk-imports - TestRunner
  PASS  -> ERROR frysk-core - testAbandonThenRemoveAttached(frysk.proc.TestAbandon)
  PASS  -> ERROR frysk-core - testSliceRegisterSet(frysk.proc.ptrace.TestByteBuffer)

------------------------------------------------------------------------------
Package:        frysk_dist
Previous build: 2007/05/17 14:54:42 - 2007/05/17 14:54:55
Current build:  2007/05/23 04:12:14 - 2007/05/23 04:23:52
Host:           coldstone.us.oracle.com
Platform:       Linux 2.6.20-1.2948.fc6 (FC6)
First failure:  build (was pre_fetch)

------------------------------------------------------------------------------
Package:        frysk_fresh
Previous build: 2007/05/22 03:45:04 - 2007/05/22 04:00:23
Current build:  2007/05/23 03:45:05 - 2007/05/23 04:01:49
Host:           ca-tools2
Platform:       Linux 2.6.20-1.2948.fc6 (FC6)
First failure:  test
Test result:    602: 560 +, 12 -, 8 S, 22 E (was 599: 554 +, 13 -, 8 S, 24 E)

  Old   -> New   Test
  -----    ----- ----------------------------------
  PASS  -> ERROR ADDED (NEW) TESTS:
  PASS  -> ERROR frysk-core - testAsyncAddressSpace(frysk.proc.ptrace.TestByteBuffer)
  ERROR -> PASS  frysk-imports - testPreadValidity(frysk.sys.TestStatelessFile)
  FAIL  -> PASS  frysk-imports - TestRunner
  ERROR -> PASS  frysk-imports - testPwriteValidity(frysk.sys.TestStatelessFile)
  PASS  -> ERROR frysk-core - testAbandonThenRemoveAttached(frysk.proc.TestAbandon)
  PASS  -> ERROR frysk-core - testSliceRegisterSet(frysk.proc.ptrace.TestByteBuffer)

------------------------------------------------------------------------------
Package:        frysk_incr
Previous build: 2007/05/22 04:00:24 - 2007/05/22 04:15:11
Current build:  2007/05/23 04:01:50 - 2007/05/23 04:16:37
Host:           ca-tools2
Platform:       Linux 2.6.20-1.2948.fc6 (FC6)
First failure:  test
Test result:    602: 558 +, 14 -, 8 S, 22 E (was 599: 557 +, 12 -, 8 S, 22 E)

  Old   -> New   Test
  -----    ----- ----------------------------------
  PASS  -> FAIL  frysk-imports - testShortTimeout(frysk.sys.TestWait)
  PASS  -> ERROR ADDED (NEW) TESTS:
  PASS  -> ERROR frysk-core - testAsyncAddressSpace(frysk.proc.ptrace.TestByteBuffer)
  PASS  -> FAIL  frysk-imports - TestRunner
  PASS  -> ERROR frysk-core - testAbandonThenRemoveAttached(frysk.proc.TestAbandon)
  PASS  -> ERROR frysk-core - testSliceRegisterSet(frysk.proc.ptrace.TestByteBuffer)

------------------------------------------------------------------------------
Package:        frysk_dist
Previous build: 2007/04/20 22:57:40 - 2007/04/20 23:10:03
Current build:  2007/05/23 04:16:38 - 2007/05/23 04:28:05
Host:           ca-tools2
Platform:       Linux 2.6.20-1.2948.fc6 (FC6) (was Linux 2.6.20-1.2944.fc6 (FC6))
First failure:  build

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Kris,

Does this really need to be semi-daily?  I thought it was happily 
posting weekly summaries.  As developers we can more directly detect and 
report immediate problems.

Andrew

Re: Automated build-and-test summary report (2007/05/23)

[Kris Van Hees]

I may be missing something, but I honestly do not understand the issue
here.  Obviously, an automated build-and-test system that performs
nightly builds is hardly any use if the results are only emailed out
once a week.  The continuing existence of tests that show intermittent
failures, and the continued failure of the dist-builds (that you in fact
suggested be added to the list of configuration) indicates that we (as
developers) are not detecting and reporting all problems.  The automated
system is performing a full-scale test, and extracts the relevant output
from the tests, consolidating those in clear reports.

However, if receiving a daily report in email places an undue burden on
you, I can certainly disable the scheduling of nightly builds for Frysk.
There is no point in running tests when you don't want the results
distributed to the developers.

	Cheers,
	Kris

On Wed, May 23, 2007 at 11:24:35AM -0400, Andrew Cagney wrote:
> Kris,
> 
> Does this really need to be semi-daily?  I thought it was happily 
> posting weekly summaries.  As developers we can more directly detect and 
> report immediate problems.
> 
> Andrew
> 

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Hi Kris,

Is it possible to separate out the building and reporting functions so 
that there's a weekly summary showing trends?  In general, crasher bugs 
are found quickly and directly by running the test-suite locally.  Its 
the more obscure failures, that are missed by normal testing, and 
require drilling down through change/build history that are really 
helped by this system.

Andrew

Kris Van Hees wrote:
> I may be missing something, but I honestly do not understand the issue
> here.  Obviously, an automated build-and-test system that performs
> nightly builds is hardly any use if the results are only emailed out
> once a week.  The continuing existence of tests that show intermittent
> failures, and the continued failure of the dist-builds (that you in fact
> suggested be added to the list of configuration) indicates that we (as
> developers) are not detecting and reporting all problems.  The automated
> system is performing a full-scale test, and extracts the relevant output
> from the tests, consolidating those in clear reports.
>
> However, if receiving a daily report in email places an undue burden on
> you, I can certainly disable the scheduling of nightly builds for Frysk.
> There is no point in running tests when you don't want the results
> distributed to the developers.
>
> 	Cheers,
> 	Kris
>
> On Wed, May 23, 2007 at 11:24:35AM -0400, Andrew Cagney wrote:
>   
>> Kris,
>>
>> Does this really need to be semi-daily?  I thought it was happily 
>> posting weekly summaries.  As developers we can more directly detect and 
>> report immediate problems.
>>
>> Andrew
>>
>>     

Re: Automated build-and-test summary report (2007/05/23)

[Kris Van Hees]

I somehow fail to see the value of crippling the system by reducing the
amount of useful reporting, when it should be quite easy for you to
obtain the information you want based on the more detailed, more
frequent reports.  The point of the automated build-and-test system is
to perform the build-and-test runs, and collect the output for reporting
purposes.  The point of the system is not to do project-specific analysis
based on that data.  All the data (even the raw data) is available to
you from the the web pages generated by the automated build-and-test
system.

Also note that I am a bit reluctant to make significant changes to a
generic system (i.e. that is *not* frysk specific) that has yet to
receive any outside contribution in terms of build-and-test resources.
Note that the system is in fact generic enough to allow you to do
exactly what you want on your own system, i.e. you could use it to have
nightly (or even more frequent) build-and-test runs executed, capture
the logs and parse them at your heart's content.  Or let the central
reporting system process them, and you pull them back off the system for
your own reporting.

The opportunities are virtually endless...  let's not limit the
flexibility it offers for a single very specific request that could
easily be implemented on top of the existing system.

	Cheers,
	Kris

On Thu, May 24, 2007 at 01:35:50PM -0400, Andrew Cagney wrote:
> Hi Kris,
> 
> Is it possible to separate out the building and reporting functions so 
> that there's a weekly summary showing trends?  In general, crasher bugs 
> are found quickly and directly by running the test-suite locally.  Its 
> the more obscure failures, that are missed by normal testing, and 
> require drilling down through change/build history that are really 
> helped by this system.
> 
> Andrew
> 
> Kris Van Hees wrote:
> >I may be missing something, but I honestly do not understand the issue
> >here.  Obviously, an automated build-and-test system that performs
> >nightly builds is hardly any use if the results are only emailed out
> >once a week.  The continuing existence of tests that show intermittent
> >failures, and the continued failure of the dist-builds (that you in fact
> >suggested be added to the list of configuration) indicates that we (as
> >developers) are not detecting and reporting all problems.  The automated
> >system is performing a full-scale test, and extracts the relevant output
> >from the tests, consolidating those in clear reports.
> >
> >However, if receiving a daily report in email places an undue burden on
> >you, I can certainly disable the scheduling of nightly builds for Frysk.
> >There is no point in running tests when you don't want the results
> >distributed to the developers.
> >
> >	Cheers,
> >	Kris
> >
> >On Wed, May 23, 2007 at 11:24:35AM -0400, Andrew Cagney wrote:
> >  
> >>Kris,
> >>
> >>Does this really need to be semi-daily?  I thought it was happily 
> >>posting weekly summaries.  As developers we can more directly detect and 
> >>report immediate problems.
> >>
> >>Andrew
> >>
> >>    
> 

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Kris,

Again, I would really appreciate it if we didn't fill the list with your 
test-results.  If the system must always post out its results then it 
unfortunatly sounds like the best solution is to limit them to once a 
week.  Sigh.

Andrew

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Kris, ping.

Andrew Cagney wrote:
> Kris,
>
> Again, I would really appreciate it if we didn't fill the list with 
> your test-results.  If the system must always post out its results 
> then it unfortunatly sounds like the best solution is to limit them to 
> once a week.  Sigh.
>
> Andrew
>

Re: Automated build-and-test summary report (2007/05/23)

[Kris Van Hees]

Pong.  As you are probably aware, yesterday was a US holiday.

I must not be getting your point...  Why would you support the notion
that automated builds and testing is a benefit (at least that is my
impression based on conversations on #frysk and conf calls), while at
the same time trying to pressure me into not posting the results.  I
have yet to find any testing methodology that promotes executing tests
without presenting the results for investigation.  Nor have I found any
(yet) that promote executing tests and verifying the results at a much
later time (like running tests all week, but only getting the results at
the end of that week).  And note that these are all *project*
test-results.

At the onset of getting preliminary results from the build-and-test
system, various people stepped in to deal with some issues that were
obvious (to the people working on it) once the summarized report was
posted.  That turned out to be rather low hanging fruit that simply had
not been dealt with.

The build system has also been exercising (on your suggestion, by the
way) the state of building from a 'make dist' tree.  We started doing
that on Mar 29th, and to date there has not been a single successful
build using that configuration, nor does it seem like anyone has
actually even bothered to look into that problem.  While I fully agree
that testing that configuration is important, I am a bit concerned about
you being the one to strongly suggest we should add that and at the same
time not having said a word about it even though it has been failing for
2 months straight so far with no indication that it is about to improve.

Throughout the months the system has been in development, and then
became fully functioning, we suffered through quite a few iterations of
finding kernel problems relating to utrace.  More often than not, these
were problems that others had not reported (either due to not testing on
those configurations or otherwise).  We got quite a bit of traction on
that and largely due to Roland's work, the situation improved a whole
lot.

It is also very noticeable when you look at the today-vs-yesterday
comparison reports that I have recently been posting on a daily basis
(and that you clearly have a personal objection to), that there are
quite a few tests that show intermittent failures (some are almost on an
every-other-day cycle) in specific configurations.  You have previously
stated that no test should ever PASS part of the time, and FAIL (or be
an ERROR) the rest of the time - that would constitute a "bad test".  I
fully agree.  And that is the kind of information that these reports
are bringing to the surface.  These situations are one of the reasons to
not just depend on developers running tests and fixing things that are
broken.  Note the current issue with pending USR1 signals?  For you (as
a developer) the problem occurred, you did a clean rebuild, and it went
away.  The build system exercises clean builds all the time, and clearly
shows that testing as a developer is simply not sufficient to make a
correct determination on whether a problem exists or not.

So, please enlighten me: what is the *real* problem here?

	Cheers,
	Kris

On Tue, May 29, 2007 at 01:23:35PM -0400, Andrew Cagney wrote:
> Kris, ping.
> 
> Andrew Cagney wrote:
> >Kris,
> >
> >Again, I would really appreciate it if we didn't fill the list with 
> >your test-results.  If the system must always post out its results 
> >then it unfortunatly sounds like the best solution is to limit them to 
> >once a week.  Sigh.
> >
> >Andrew
> >
> 

Re: Automated build-and-test summary report (2007/05/23)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

On Tue, 2007-05-29 at 18:12 -0700, Kris Van Hees wrote:
> So, please enlighten me: what is the *real* problem here?

I like the test results and do always quickly look at them to see if
they correspond to what I am seeing locally. But I can see how it might
be annoying for them to appear on the main list. Lets create a
frysk-testresults@sourceware.org and encourage everybody to publish
their frysk test run results there. That way we have an easy way for
people to filter out the test runs from the main discussion list and
still keep an archive to consult for any historical issues which is
really valuable when some failure doesn't occur for someone locally, but
is visible on one of the test machines.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Automated build-and-test summary report (2007/05/23)

[Kris Van Hees]

Mark,

That is indeed a good suggestion.  If someone can create that list, I'll
point the posting process to that location for the test results.

	Cheers,
	Kris

On Wed, May 30, 2007 at 02:07:25PM +0200, Mark Wielaard wrote:
> On Tue, 2007-05-29 at 18:12 -0700, Kris Van Hees wrote:
> > So, please enlighten me: what is the *real* problem here?
> 
> I like the test results and do always quickly look at them to see if
> they correspond to what I am seeing locally. But I can see how it might
> be annoying for them to appear on the main list. Lets create a
> frysk-testresults@sourceware.org and encourage everybody to publish
> their frysk test run results there. That way we have an easy way for
> people to filter out the test runs from the main discussion list and
> still keep an archive to consult for any historical issues which is
> really valuable when some failure doesn't occur for someone locally, but
> is visible on one of the test machines.
> 
> Cheers,
> 
> Mark

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Kris Van Hees wrote:
> Mark,
>
> That is indeed a good suggestion.  If someone can create that list, I'll
> point the posting process to that location for the test results.
>   

I see you're continuing to post thes results to frysk@ :-/  Can that be 
disabled while I get a everything to do with the new list established?

Andrew

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Kris,

This isn't about the need for a test-farm, or about methodology, and 
certainly isn't some sort of "personal" objection.  It is just about 
considering the most effective way to communicate results to its 
intended audience.  frysk@ is used both as a discussion and status 
list.   Having weekly testing results identifying unstable tests (based 
on some history) would certainly be part of that.  However, is drowning 
the list with daily results really that reasonable?

Mark has suggested, and what many other projects have adopted, is a 
dedicated frysk-testresults list (and like frysk-bugzilla and frysk-cvs, 
it will have Reply-To: pointing back to frysk@).  Both your and any 
other daily test reports can go there.

Does this sound reasonable?

Andrew

Re: Automated build-and-test summary report (2007/05/23)

[Andrew Cagney]

Kris Van Hees wrote:
> Throughout the months the system has been in development, and then
> became fully functioning, we suffered through quite a few iterations of
> finding kernel problems relating to utrace.  More often than not, these
> were problems that others had not reported (either due to not testing on
> those configurations or otherwise).  We got quite a bit of traction on
> that and largely due to Roland's work, the situation improved a whole
> lot.
>   

To clarify.

The improvements to Frysk on Fedora Core 6 largely came about as a 
consequence of bugs identified by Roland, Moller, and myself when 
testing on RHEL 5 (the kernels were relatively close).  For instance, 
the need to re-implement the event-loop, that Red Hat undertook, was 
motivated by these bugs.

Andrew

Re: Automated build-and-test summary report (2007/05/23)

[Kris Van Hees]

No one is denying any contribution you and others at Red Hat have made
with regard to running Frysk on FC6.  Nevertheless you used part of my
message as a trampoline to provide a statement of credit, lacking any
form of recognition of contributions by anyone other than Red Hat.
Must we therefore conclude that bugs reports like #227952 and
#232800 (in Red Hat's bugzilla, against the FC6 kernel) and numerous
discussions on #frysk are a figment of my imagination?

You and everyone on the team (past and present) deserve credit for the
work you do and have done.  I never have and never will deny that.  But
as a professional courtesy, I'd expect the same in return.

	Cheers,
	Kris

On Wed, May 30, 2007 at 09:31:46AM -0400, Andrew Cagney wrote:
> Kris Van Hees wrote:
> >Throughout the months the system has been in development, and then
> >became fully functioning, we suffered through quite a few iterations of
> >finding kernel problems relating to utrace.  More often than not, these
> >were problems that others had not reported (either due to not testing on
> >those configurations or otherwise).  We got quite a bit of traction on
> >that and largely due to Roland's work, the situation improved a whole
> >lot.
> >  
> 
> To clarify.
> 
> The improvements to Frysk on Fedora Core 6 largely came about as a 
> consequence of bugs identified by Roland, Moller, and myself when 
> testing on RHEL 5 (the kernels were relatively close).  For instance, 
> the need to re-implement the event-loop, that Red Hat undertook, was 
> motivated by these bugs.
> 
> Andrew
> 

Re: Automated build-and-test summary report (2007/05/23)

[Mark Wielaard]

On Wed, 2007-05-30 at 14:09 -0700, Kris Van Hees wrote:
> You and everyone on the team (past and present) deserve credit for the
> work you do and have done.  I never have and never will deny that.  But
> as a professional courtesy, I'd expect the same in return.

To clarify.

The fact that Kris could often test and confirm or deny failures and
crashes happening on specific setups was a great way for me to know
whether or not I was looking at a kernel issue or whether we had
identified a bug in the frysk-core/state machine. Thanks Kris.

Mark