RFC: libc-testresults mailing list

RFC: libc-testresults mailing list

[Joseph Myers]

Andrew Pinski asked in 
<https://sourceware.org/ml/libc-alpha/2016-07/msg00460.html> about a 
mailing list for sending test results to.

I propose that we create a libc-testresults mailing list for test results 
to be sent to by any kind of automation (not for discussions, if people 
wish to discuss things about the results they should take replies to 
libc-alpha with a meaningful Subject line rather than the subject line of 
the results posting).

This would explicitly include results reporting on e.g. whether glibc 
builds in various configurations, not just results of the glibc testsuite 
itself.  It would also include any systems automatically detecting 
regressions and reporting on those regressions.  It might be useful to 
have something like GCC's contrib/test_summary to collect results and 
information about the configuration and versions of various components 
(and indeed for test runs to automatically record some information such as 
the kernel version on the system, possibly remote, that test programs are 
run on), but this is not required for such a list to be useful.

Comments?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RFC: libc-testresults mailing list

[Carlos O'Donell]

On 11/02/2016 09:05 AM, Joseph Myers wrote:
> Andrew Pinski asked in 
> <https://sourceware.org/ml/libc-alpha/2016-07/msg00460.html> about a 
> mailing list for sending test results to.
> 
> I propose that we create a libc-testresults mailing list for test results 
> to be sent to by any kind of automation (not for discussions, if people 
> wish to discuss things about the results they should take replies to 
> libc-alpha with a meaningful Subject line rather than the subject line of 
> the results posting).
> 
> This would explicitly include results reporting on e.g. whether glibc 
> builds in various configurations, not just results of the glibc testsuite 
> itself.  It would also include any systems automatically detecting 
> regressions and reporting on those regressions.  It might be useful to 
> have something like GCC's contrib/test_summary to collect results and 
> information about the configuration and versions of various components 
> (and indeed for test runs to automatically record some information such as 
> the kernel version on the system, possibly remote, that test programs are 
> run on), but this is not required for such a list to be useful.
> 
> Comments?

In general this seems like a good idea.

We could then link from the per-release wiki page to a mailing list URL
to show full test result details as we approach the final release.

Do we expect that libc-testresults is a superset of those builds we have
connected to our buildbot CI infrastructure?

Do we expect buildbot test results to go to libc-testresults for archiving?

Cheers,
Carlos.

Re: RFC: libc-testresults mailing list

[Joseph Myers]

On Wed, 2 Nov 2016, Carlos O'Donell wrote:

> Do we expect that libc-testresults is a superset of those builds we have
> connected to our buildbot CI infrastructure?
> 
> Do we expect buildbot test results to go to libc-testresults for archiving?

If we add a script to mail out results then I think it would be 
appropriate to make our buildbot scripts use it (having made sure that 
existing buildbot instances have working outgoing mail and documented that 
as a requirement for buildbot instances).  In that case, yes, it would be 
a superset of the buildbot builds (note Andrew was asking about where to 
send results from a system not hooked up to our buildbot).

We can consider questions such as how much goes in test results mails from 
such a script, e.g. do we include the full tests.sum (possibly gzipped and 
attached) or just results that aren't PASS, what standard information 
about the test environment goes in mails, whether to attach (gzipped) .out 
files of failing tests if they aren't too many / too big, how mails should 
declare that they are the standard output of the script to allow for 
automatic parsing.

It would of course be expected that if someone's build system starts 
flooding the list with multiple mails from identical builds (keeping 
rebuilding the same versions continuously, or mail system problems 
resulting in duplicate mails) then those mails would be blocked until the 
problem is sorted out.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RFC: libc-testresults mailing list

[Andrew Pinski]

On Wed, Nov 2, 2016 at 9:48 AM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Wed, 2 Nov 2016, Carlos O'Donell wrote:
>
>> Do we expect that libc-testresults is a superset of those builds we have
>> connected to our buildbot CI infrastructure?
>>
>> Do we expect buildbot test results to go to libc-testresults for archiving?
>
> If we add a script to mail out results then I think it would be
> appropriate to make our buildbot scripts use it (having made sure that
> existing buildbot instances have working outgoing mail and documented that
> as a requirement for buildbot instances).  In that case, yes, it would be
> a superset of the buildbot builds (note Andrew was asking about where to
> send results from a system not hooked up to our buildbot).

Yes; the main reason why I don't have it hooked up to the buildbot is
that I need/want the ability to run other things on the same machine
like even GCC build and test since build/test of glibc only takes an
hour and there is not enough changes going into glibc to dedicate one
machine just for glibc testing and one for gdb testing and one of gcc
testing.

> We can consider questions such as how much goes in test results mails from
> such a script, e.g. do we include the full tests.sum (possibly gzipped and
> attached) or just results that aren't PASS, what standard information
> about the test environment goes in mails, whether to attach (gzipped) .out
> files of failing tests if they aren't too many / too big, how mails should
> declare that they are the standard output of the script to allow for
> automatic parsing.

GCC only sends summary of what fails and what I know, that is usually
enough to track down failures on some target and when it started to
fail and then go back find which exact change broke it.  There are
many machines available on the "GCC" Compile farm to be able to run
tests like this.

>
> It would of course be expected that if someone's build system starts
> flooding the list with multiple mails from identical builds (keeping
> rebuilding the same versions continuously, or mail system problems
> resulting in duplicate mails) then those mails would be blocked until the
> problem is sorted out.

We have had that issue once or twice but nothing in the last 2-3 years
as far as I know.

Thanks,
Andrew

>
> --
> Joseph S. Myers
> joseph@codesourcery.com

Re: RFC: libc-testresults mailing list

[Joseph Myers]

On Wed, 2 Nov 2016, Andrew Pinski wrote:

> > It would of course be expected that if someone's build system starts
> > flooding the list with multiple mails from identical builds (keeping
> > rebuilding the same versions continuously, or mail system problems
> > resulting in duplicate mails) then those mails would be blocked until the
> > problem is sorted out.
> 
> We have had that issue once or twice but nothing in the last 2-3 years
> as far as I know.

gcc-regression, April this year, an interaction between IBM's mail system 
and sourceware resulted in 750 MB of mail to the list in a few days.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RFC: libc-testresults mailing list

[Joseph Myers]

This list is now operational.

https://sourceware.org/ml/libc-testresults/

-- 
Joseph S. Myers
joseph@codesourcery.com