* RFC: libc-testresults mailing list
@ 2016-11-02 13:06 Joseph Myers
2016-11-02 14:30 ` Carlos O'Donell
2016-11-21 19:08 ` Joseph Myers
0 siblings, 2 replies; 6+ messages in thread
From: Joseph Myers @ 2016-11-02 13:06 UTC (permalink / raw)
To: libc-alpha
Andrew Pinski asked in
<https://sourceware.org/ml/libc-alpha/2016-07/msg00460.html> about a
mailing list for sending test results to.
I propose that we create a libc-testresults mailing list for test results
to be sent to by any kind of automation (not for discussions, if people
wish to discuss things about the results they should take replies to
libc-alpha with a meaningful Subject line rather than the subject line of
the results posting).
This would explicitly include results reporting on e.g. whether glibc
builds in various configurations, not just results of the glibc testsuite
itself. It would also include any systems automatically detecting
regressions and reporting on those regressions. It might be useful to
have something like GCC's contrib/test_summary to collect results and
information about the configuration and versions of various components
(and indeed for test runs to automatically record some information such as
the kernel version on the system, possibly remote, that test programs are
run on), but this is not required for such a list to be useful.
Comments?
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: libc-testresults mailing list
2016-11-02 13:06 RFC: libc-testresults mailing list Joseph Myers
@ 2016-11-02 14:30 ` Carlos O'Donell
2016-11-02 16:48 ` Joseph Myers
2016-11-21 19:08 ` Joseph Myers
1 sibling, 1 reply; 6+ messages in thread
From: Carlos O'Donell @ 2016-11-02 14:30 UTC (permalink / raw)
To: libc-alpha
On 11/02/2016 09:05 AM, Joseph Myers wrote:
> Andrew Pinski asked in
> <https://sourceware.org/ml/libc-alpha/2016-07/msg00460.html> about a
> mailing list for sending test results to.
>
> I propose that we create a libc-testresults mailing list for test results
> to be sent to by any kind of automation (not for discussions, if people
> wish to discuss things about the results they should take replies to
> libc-alpha with a meaningful Subject line rather than the subject line of
> the results posting).
>
> This would explicitly include results reporting on e.g. whether glibc
> builds in various configurations, not just results of the glibc testsuite
> itself. It would also include any systems automatically detecting
> regressions and reporting on those regressions. It might be useful to
> have something like GCC's contrib/test_summary to collect results and
> information about the configuration and versions of various components
> (and indeed for test runs to automatically record some information such as
> the kernel version on the system, possibly remote, that test programs are
> run on), but this is not required for such a list to be useful.
>
> Comments?
In general this seems like a good idea.
We could then link from the per-release wiki page to a mailing list URL
to show full test result details as we approach the final release.
Do we expect that libc-testresults is a superset of those builds we have
connected to our buildbot CI infrastructure?
Do we expect buildbot test results to go to libc-testresults for archiving?
Cheers,
Carlos.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: libc-testresults mailing list
2016-11-02 14:30 ` Carlos O'Donell
@ 2016-11-02 16:48 ` Joseph Myers
2016-11-03 6:00 ` Andrew Pinski
0 siblings, 1 reply; 6+ messages in thread
From: Joseph Myers @ 2016-11-02 16:48 UTC (permalink / raw)
To: Carlos O'Donell; +Cc: libc-alpha
On Wed, 2 Nov 2016, Carlos O'Donell wrote:
> Do we expect that libc-testresults is a superset of those builds we have
> connected to our buildbot CI infrastructure?
>
> Do we expect buildbot test results to go to libc-testresults for archiving?
If we add a script to mail out results then I think it would be
appropriate to make our buildbot scripts use it (having made sure that
existing buildbot instances have working outgoing mail and documented that
as a requirement for buildbot instances). In that case, yes, it would be
a superset of the buildbot builds (note Andrew was asking about where to
send results from a system not hooked up to our buildbot).
We can consider questions such as how much goes in test results mails from
such a script, e.g. do we include the full tests.sum (possibly gzipped and
attached) or just results that aren't PASS, what standard information
about the test environment goes in mails, whether to attach (gzipped) .out
files of failing tests if they aren't too many / too big, how mails should
declare that they are the standard output of the script to allow for
automatic parsing.
It would of course be expected that if someone's build system starts
flooding the list with multiple mails from identical builds (keeping
rebuilding the same versions continuously, or mail system problems
resulting in duplicate mails) then those mails would be blocked until the
problem is sorted out.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: libc-testresults mailing list
2016-11-02 16:48 ` Joseph Myers
@ 2016-11-03 6:00 ` Andrew Pinski
2016-11-03 13:21 ` Joseph Myers
0 siblings, 1 reply; 6+ messages in thread
From: Andrew Pinski @ 2016-11-03 6:00 UTC (permalink / raw)
To: Joseph Myers; +Cc: Carlos O'Donell, GNU C Library
On Wed, Nov 2, 2016 at 9:48 AM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Wed, 2 Nov 2016, Carlos O'Donell wrote:
>
>> Do we expect that libc-testresults is a superset of those builds we have
>> connected to our buildbot CI infrastructure?
>>
>> Do we expect buildbot test results to go to libc-testresults for archiving?
>
> If we add a script to mail out results then I think it would be
> appropriate to make our buildbot scripts use it (having made sure that
> existing buildbot instances have working outgoing mail and documented that
> as a requirement for buildbot instances). In that case, yes, it would be
> a superset of the buildbot builds (note Andrew was asking about where to
> send results from a system not hooked up to our buildbot).
Yes; the main reason why I don't have it hooked up to the buildbot is
that I need/want the ability to run other things on the same machine
like even GCC build and test since build/test of glibc only takes an
hour and there is not enough changes going into glibc to dedicate one
machine just for glibc testing and one for gdb testing and one of gcc
testing.
> We can consider questions such as how much goes in test results mails from
> such a script, e.g. do we include the full tests.sum (possibly gzipped and
> attached) or just results that aren't PASS, what standard information
> about the test environment goes in mails, whether to attach (gzipped) .out
> files of failing tests if they aren't too many / too big, how mails should
> declare that they are the standard output of the script to allow for
> automatic parsing.
GCC only sends summary of what fails and what I know, that is usually
enough to track down failures on some target and when it started to
fail and then go back find which exact change broke it. There are
many machines available on the "GCC" Compile farm to be able to run
tests like this.
>
> It would of course be expected that if someone's build system starts
> flooding the list with multiple mails from identical builds (keeping
> rebuilding the same versions continuously, or mail system problems
> resulting in duplicate mails) then those mails would be blocked until the
> problem is sorted out.
We have had that issue once or twice but nothing in the last 2-3 years
as far as I know.
Thanks,
Andrew
>
> --
> Joseph S. Myers
> joseph@codesourcery.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: libc-testresults mailing list
2016-11-03 6:00 ` Andrew Pinski
@ 2016-11-03 13:21 ` Joseph Myers
0 siblings, 0 replies; 6+ messages in thread
From: Joseph Myers @ 2016-11-03 13:21 UTC (permalink / raw)
To: Andrew Pinski; +Cc: Carlos O'Donell, GNU C Library
On Wed, 2 Nov 2016, Andrew Pinski wrote:
> > It would of course be expected that if someone's build system starts
> > flooding the list with multiple mails from identical builds (keeping
> > rebuilding the same versions continuously, or mail system problems
> > resulting in duplicate mails) then those mails would be blocked until the
> > problem is sorted out.
>
> We have had that issue once or twice but nothing in the last 2-3 years
> as far as I know.
gcc-regression, April this year, an interaction between IBM's mail system
and sourceware resulted in 750 MB of mail to the list in a few days.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RFC: libc-testresults mailing list
2016-11-02 13:06 RFC: libc-testresults mailing list Joseph Myers
2016-11-02 14:30 ` Carlos O'Donell
@ 2016-11-21 19:08 ` Joseph Myers
1 sibling, 0 replies; 6+ messages in thread
From: Joseph Myers @ 2016-11-21 19:08 UTC (permalink / raw)
To: libc-alpha
This list is now operational.
https://sourceware.org/ml/libc-testresults/
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-11-21 19:08 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-02 13:06 RFC: libc-testresults mailing list Joseph Myers
2016-11-02 14:30 ` Carlos O'Donell
2016-11-02 16:48 ` Joseph Myers
2016-11-03 6:00 ` Andrew Pinski
2016-11-03 13:21 ` Joseph Myers
2016-11-21 19:08 ` Joseph Myers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).