Regular git master HEAD testing

Regular git master HEAD testing

[Petr Muller]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

Hi,

I'm a member of Red Hat tools QA team. I've set up a regular testing
script which, once a day, will fetch a HEAD of master git branch, build
it, and run the testsuite, submitting logs to dejazilla. In the case of
build failure, it will send an email to this list. Here are some
details:

1) The distro where all this happen is currently RHEL5.2. I will try to
add Fedora too in the future.
2) Architectures being tested are i686,ia64,s390x,powerpc64,x86_64
3) There are no special configure options, except for bundled elfutils,
for which the source will be always the most recent fetched from
elfutils upstream. Currently it is 0.137.
4) The build failure mails are currently directed at me, not this list,
because I'm gonna test this a bit more before I let some script spam
some mailing list.

I hope this can be useful for you, please let me know, if you find it
needless, or things you would like or do not like.

Some first runs are already at dejazilla, and I've noticed quite a high
number of 'UNTESTED' in my results (cca 230), in comparison to other
results already there (cca 20). What can be the reason, and what should
I do to make these testcases working?

Thanks,

Petr

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Regular git master HEAD testing

[Masami Hiramatsu]

Hi,

Petr Muller wrote:
> Hi,
> 
> I'm a member of Red Hat tools QA team. I've set up a regular testing
> script which, once a day, will fetch a HEAD of master git branch, build
> it, and run the testsuite, submitting logs to dejazilla. In the case of
> build failure, it will send an email to this list. Here are some
> details:

Great! thank you for making it automated.

> 1) The distro where all this happen is currently RHEL5.2. I will try to
> add Fedora too in the future.
> 2) Architectures being tested are i686,ia64,s390x,powerpc64,x86_64
> 3) There are no special configure options, except for bundled elfutils,
> for which the source will be always the most recent fetched from
> elfutils upstream. Currently it is 0.137.
> 4) The build failure mails are currently directed at me, not this list,
> because I'm gonna test this a bit more before I let some script spam
> some mailing list.
> 
> I hope this can be useful for you, please let me know, if you find it
> needless, or things you would like or do not like.

In the case of that you test once a day, I think it would be better to
send the result only if there are some differences from previous.

> Some first runs are already at dejazilla, and I've noticed quite a high
> number of 'UNTESTED' in my results (cca 230), in comparison to other
> results already there (cca 20). What can be the reason, and what should
> I do to make these testcases working?

Would you run 'make installcheck'? 'make check' will skip some tests
which install some test modules.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Regular git master HEAD testing

[Petr Muller]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

On Wed, 2008-09-10 at 10:50 -0400, Masami Hiramatsu wrote:
> Petr Muller wrote:
> > I'm a member of Red Hat tools QA team. I've set up a regular testing
> > script which, once a day, will fetch a HEAD of master git branch, build
> > it, and run the testsuite, submitting logs to dejazilla. 
> > 
> > I hope this can be useful for you, please let me know, if you find it
> > needless, or things you would like or do not like.
> 
> In the case of that you test once a day, I think it would be better to
> send the result only if there are some differences from previous.

Makes sense. I will change the scripts so they won't submit the results
if they are same as the previous day. I wonder - how much is dejazilla
monitored for regressions? I mean - I can easily spot regressions "from
yesterday" too as a part of that comparison, how can I give an alert? Is
uploading results to dejazilla sufficient to trigger someone's alarm?

> > Some first runs are already at dejazilla, and I've noticed quite a high
> > number of 'UNTESTED' in my results (cca 230), in comparison to other
> > results already there (cca 20). What can be the reason, and what should
> > I do to make these testcases working?
> 
> Would you run 'make installcheck'? 'make check' will skip some tests
> which install some test modules.

Yes, this was the reason. I modified the scripts to do installcheck
instead now.

Petr

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Regular git master HEAD testing

[Frank Ch. Eigler]

Petr Muller <pmuller@redhat.com> writes:

> [...]
>> In the case of that you test once a day, I think it would be better to
>> send the result only if there are some differences from previous.
>
> Makes sense. I will change the scripts so they won't submit the results
> if they are same as the previous day. I wonder - how much is dejazilla
> monitored for regressions? [...]

Only casually at present.  I'll extend dejazilla at some point soon to
focus on such data, probably via RSS.

- FChE

Re: Regular git master HEAD testing

[Mark Wielaard]

Hi Petr,

On Wed, 2008-09-10 at 14:33 +0200, Petr Muller wrote:
> 1) The distro where all this happen is currently RHEL5.2. I will try to
> add Fedora too in the future.
> 2) Architectures being tested are i686,ia64,s390x,powerpc64,x86_64
> 3) There are no special configure options, except for bundled elfutils,
> for which the source will be always the most recent fetched from
> elfutils upstream. Currently it is 0.137.
> 4) The build failure mails are currently directed at me, not this list,
> because I'm gonna test this a bit more before I let some script spam
> some mailing list.
> 
> I hope this can be useful for you, please let me know, if you find it
> needless, or things you would like or do not like.

Was this how you found bug #6904 buildok/syscall.stp and
buildok/aux_syscalls.stp started failing on ia64 and s390x?
Then, sorry for breaking that on those architectures. But cool your
autobuilder found it. Very useful.

Could you show how the failure email looked?

Thanks,

Mark

Re: Regular git master HEAD testing

[Petr Muller]

[-- Attachment #1: Type: text/plain, Size: 2472 bytes --]

On Thu, 2008-09-18 at 23:48 +0200, Mark Wielaard wrote:
> Hi Petr,
> 
> On Wed, 2008-09-10 at 14:33 +0200, Petr Muller wrote:
> > 1) The distro where all this happen is currently RHEL5.2. I will try to
> > add Fedora too in the future.
> > 2) Architectures being tested are i686,ia64,s390x,powerpc64,x86_64
> > 3) There are no special configure options, except for bundled elfutils,
> > for which the source will be always the most recent fetched from
> > elfutils upstream. Currently it is 0.137.
> > 4) The build failure mails are currently directed at me, not this list,
> > because I'm gonna test this a bit more before I let some script spam
> > some mailing list.
> > 
> > I hope this can be useful for you, please let me know, if you find it
> > needless, or things you would like or do not like.
> 
> Was this how you found bug #6904 buildok/syscall.stp and
> buildok/aux_syscalls.stp started failing on ia64 and s390x?

Yeah, it was. Although I did the "which commit did that"  part manually
with git bisect. It would save me time if I looked at the commit log in
the first place :) Actually I'll probably add new commit logs to my
alarm messages. 

> Then, sorry for breaking that on those architectures. But cool your
> autobuilder found it. Very useful.
> 
> Could you show how the failure email looked?

Currently, it looks like this (it is just an output of deja-summarize
script, which is a blackbox for me, because it's in perl :)

REGRESSION INFO:
pass: buildok/aux_syscalls-embedded.stp: 
fail: buildok/aux_syscalls-embedded.stp: 
pass: buildok/syscall.stp: 
fail: buildok/syscall.stp: 

Filename: /mnt/qa/scratch/pmuller/stap-upstream-tester/Red Hat
Enterprise Linux Server release 5.2 (Tikanga)/ia64/systemtap.sum
PASS       :     517
FAIL       :      15
REGRESSION :       0
KFAIL      :       6
default    :     246
XFAIL      :       2
SUITES     :     157
UNSUPPORTED:       1
UNRESOLVED :       0
XPASS      :       9
WARN       :       0
UNTESTED   :      24
ERROR      :       2
KPASS      :       1
SCORE      :     341

Filename: systemtap.sum
PASS       :     490
FAIL       :      42
REGRESSION :       2
KFAIL      :       6
default    :     246
XFAIL      :       2
SUITES     :     157
UNSUPPORTED:       1
UNRESOLVED :       0
XPASS      :       9
WARN       :       0
UNTESTED   :      24
ERROR      :       2
KPASS      :       1
SCORE      :       6


Petr

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]