[RFC] Proposal for hosting GDB CI builds

[RFC] Proposal for hosting GDB CI builds

[Luis Machado]

Hi,

This proposal comes as an attempt to revive the GDB CI builds, given the 
previous one (buildbot) is no longer being maintained by Sergio (thanks 
a lot for keeping it working for all these years by the way).

CI GDB builds are a great help for spotting regressions without having 
to do the tedious and time-consuming work of running the GDB testsuite 
for each git revision, for each architecture and reading through 
hundreds of lines of summaries. If a regression is spotted, then one 
needs to bisect to find the culprit. This isn't great, specially for 
architectures without great availability of desktop hardware.

Ideally, for each commit, we'd run full builds to validate the state of 
the tree, but we're not quite there yet. So meanwhile, having some level 
of automation to get the builds done without manual intervention sounds 
like a reasonable step forward.

 From previous IRC conversations, it seems to be a consensus that 
availability of processing power is not a problem. It is reasonably easy 
to find hardware to do some builds. The most difficult resource to find 
is manpower to setup the CI infrastructure and keep it running.

With the above said, I've discussed this internally at Linaro and we can 
spare some manpower to setup and maintain an isolated Linaro-hosted 
Jenkins instance for GDB CI builds.

Linaro can take care of providing builders and build jobs for ARM. Other 
architectures would be handled by their respective contributors. Those 
contributors can write jobs and plug builders as needed.

Setting up new jobs doesn't require the use of the web interface. It can 
be done with yaml files in a git repo. It is reasonably simple.

You can see an example of the Linaro CI here: https://ci.linaro.org/

Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/

And also the summaries for GDB testsuite runs: 
https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html

Of course, this effort only makes sense if the community is OK with 
using Jenkins as the CI mechanism and if it actually sees value in 
having a system like this in place.

Thoughts? Questions?

Re: [RFC] Proposal for hosting GDB CI builds

[Rainer Orth]

Hi Luis,

> Linaro can take care of providing builders and build jobs for ARM. Other
> architectures would be handled by their respective contributors. Those 
> contributors can write jobs and plug builders as needed.

thanks for coming forward with this: this is very welcome, given how
easy it is to miss build failures and other issues especially on
not-so-common targets.

However, is there any documentation on setting up new builders?  I've
never dealt with Jenkins before, and from glimpsing over the docs some
time ago when Jeff Law talked about extending this GCC builders to a
wider range of architectures left me completely at a loss: the whole
thing felt like a total moloch with an incredible range of abilities,
but little to no guidance on how to start.  If the GDB CI wants to
extend beyond a Linux-only range of targets, I believe considerable
documentation is necessary to make this happen.

Besides, I seem to have glimpsed from the Linaro instance that the
builders use Docker.  Is this a requirement or just a convenience?  I'm
asking because there's no current Docker port to Solaris (there used to
be one based on zones, but it's no longer maintained) and the
buildbot-based builders I'm running (for both LLVM and GDB) do fine
without.

Thanks.
        Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: [RFC] Proposal for hosting GDB CI builds

[Christophe LYON]

Hi Luis,

On 30/06/2021 18:46, Luis Machado via Gdb wrote:
> Hi,
>
> This proposal comes as an attempt to revive the GDB CI builds, given 
> the previous one (buildbot) is no longer being maintained by Sergio 
> (thanks a lot for keeping it working for all these years by the way).
>
> CI GDB builds are a great help for spotting regressions without having 
> to do the tedious and time-consuming work of running the GDB testsuite 
> for each git revision, for each architecture and reading through 
> hundreds of lines of summaries. If a regression is spotted, then one 
> needs to bisect to find the culprit. This isn't great, specially for 
> architectures without great availability of desktop hardware.
>
> Ideally, for each commit, we'd run full builds to validate the state 
> of the tree, but we're not quite there yet. So meanwhile, having some 
> level of automation to get the builds done without manual intervention 
> sounds like a reasonable step forward.
>
> From previous IRC conversations, it seems to be a consensus that 
> availability of processing power is not a problem. It is reasonably 
> easy to find hardware to do some builds. The most difficult resource 
> to find is manpower to setup the CI infrastructure and keep it running.
>
> With the above said, I've discussed this internally at Linaro and we 
> can spare some manpower to setup and maintain an isolated 
> Linaro-hosted Jenkins instance for GDB CI builds.


As you could guess, you have a big +1 from me ;-)


>
> Linaro can take care of providing builders and build jobs for ARM. 
> Other architectures would be handled by their respective contributors. 
> Those contributors can write jobs and plug builders as needed.
>
> Setting up new jobs doesn't require the use of the web interface. It 
> can be done with yaml files in a git repo. It is reasonably simple.
>
> You can see an example of the Linaro CI here: https://ci.linaro.org/
>
> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/
>
> And also the summaries for GDB testsuite runs: 
> https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html
>
> Of course, this effort only makes sense if the community is OK with 
> using Jenkins as the CI mechanism and if it actually sees value in 
> having a system like this in place.
>
> Thoughts? Questions?

 From the reports you shared above, it seems there are lots of tests 
with "random" results (or rather lots of differences between two runs). 
My fear is that reports will soon get ignored if there are too many 
false regressions. But I hope that making such variations public will 
help fix the root cause.

Thanks,


Christophe

Re: [RFC] Proposal for hosting GDB CI builds

[Luis Machado]

Hi Rainer,

On 7/1/21 6:40 AM, Rainer Orth wrote:
> Hi Luis,
> 
>> Linaro can take care of providing builders and build jobs for ARM. Other
>> architectures would be handled by their respective contributors. Those
>> contributors can write jobs and plug builders as needed.
> 
> thanks for coming forward with this: this is very welcome, given how
> easy it is to miss build failures and other issues especially on
> not-so-common targets.
> 
> However, is there any documentation on setting up new builders?  I've
> never dealt with Jenkins before, and from glimpsing over the docs some
> time ago when Jeff Law talked about extending this GCC builders to a
> wider range of architectures left me completely at a loss: the whole
> thing felt like a total moloch with an incredible range of abilities,
> but little to no guidance on how to start.  If the GDB CI wants to
> extend beyond a Linux-only range of targets, I believe considerable
> documentation is necessary to make this happen.

I know the feeling. I've been there myself as Jenkins is indeed a very 
flexible tool. And I don't, by any means, claim to be an expert on 
dealing with it.

Usually what I do is to peak at the documentation and at what others 
have implemented. Then I work on extending/modifying things to my needs.

The GDB build job (currently running for aarch64/armhf/x86_64) I put 
together is here:

https://git.linaro.org/ci/job/configs.git/tree/tcwg-gdb.yaml

But I expect we will have to adjust this to make it a bit more generic 
so other architectures can use it.

> 
> Besides, I seem to have glimpsed from the Linaro instance that the
> builders use Docker.  Is this a requirement or just a convenience?  I'm
> asking because there's no current Docker port to Solaris (there used to
> be one based on zones, but it's no longer maintained) and the
> buildbot-based builders I'm running (for both LLVM and GDB) do fine
> without.

That is a convenience so we can share hardware resources. It is possible 
to use real hardware to run the jobs. One may need to adjust the 
configurations a bit (distro, packages etc), but a job can automate some 
of that. Details about distros to use and packages to install still need 
to be investigated/discussed.

> 
> Thanks.
>          Rainer
> 

Re: [RFC] Proposal for hosting GDB CI builds

[Luis Machado]

Hi Christophe,

On 7/1/21 8:37 AM, Christophe LYON wrote:
> Hi Luis,
> 
> On 30/06/2021 18:46, Luis Machado via Gdb wrote:
>> Hi,
>>
>> This proposal comes as an attempt to revive the GDB CI builds, given 
>> the previous one (buildbot) is no longer being maintained by Sergio 
>> (thanks a lot for keeping it working for all these years by the way).
>>
>> CI GDB builds are a great help for spotting regressions without having 
>> to do the tedious and time-consuming work of running the GDB testsuite 
>> for each git revision, for each architecture and reading through 
>> hundreds of lines of summaries. If a regression is spotted, then one 
>> needs to bisect to find the culprit. This isn't great, specially for 
>> architectures without great availability of desktop hardware.
>>
>> Ideally, for each commit, we'd run full builds to validate the state 
>> of the tree, but we're not quite there yet. So meanwhile, having some 
>> level of automation to get the builds done without manual intervention 
>> sounds like a reasonable step forward.
>>
>> From previous IRC conversations, it seems to be a consensus that 
>> availability of processing power is not a problem. It is reasonably 
>> easy to find hardware to do some builds. The most difficult resource 
>> to find is manpower to setup the CI infrastructure and keep it running.
>>
>> With the above said, I've discussed this internally at Linaro and we 
>> can spare some manpower to setup and maintain an isolated 
>> Linaro-hosted Jenkins instance for GDB CI builds.
> 
> 
> As you could guess, you have a big +1 from me ;-)
> 
> 
>>
>> Linaro can take care of providing builders and build jobs for ARM. 
>> Other architectures would be handled by their respective contributors. 
>> Those contributors can write jobs and plug builders as needed.
>>
>> Setting up new jobs doesn't require the use of the web interface. It 
>> can be done with yaml files in a git repo. It is reasonably simple.
>>
>> You can see an example of the Linaro CI here: https://ci.linaro.org/
>>
>> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/
>>
>> And also the summaries for GDB testsuite runs: 
>> https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html 
>>
>>
>> Of course, this effort only makes sense if the community is OK with 
>> using Jenkins as the CI mechanism and if it actually sees value in 
>> having a system like this in place.
>>
>> Thoughts? Questions?
> 
>  From the reports you shared above, it seems there are lots of tests 
> with "random" results (or rather lots of differences between two runs). 
> My fear is that reports will soon get ignored if there are too many 
> false regressions. But I hope that making such variations public will 
> help fix the root cause.

Unfortunately that's always been the nature of GDB's testsuite. It has 
improved over the year, but there are still problematic tests. We will 
certainly have to put those into an exception list when reporting the 
summaries, at least at an initial stage.

> 
> Thanks,
> 
> 
> Christophe
> 
> 

Re: [RFC] Proposal for hosting GDB CI builds

[Rainer Orth]

Hi Luis,

>>> Linaro can take care of providing builders and build jobs for ARM. Other
>>> architectures would be handled by their respective contributors. Those
>>> contributors can write jobs and plug builders as needed.
>> thanks for coming forward with this: this is very welcome, given how
>> easy it is to miss build failures and other issues especially on
>> not-so-common targets.
>> However, is there any documentation on setting up new builders?  I've
>> never dealt with Jenkins before, and from glimpsing over the docs some
>> time ago when Jeff Law talked about extending this GCC builders to a
>> wider range of architectures left me completely at a loss: the whole
>> thing felt like a total moloch with an incredible range of abilities,
>> but little to no guidance on how to start.  If the GDB CI wants to
>> extend beyond a Linux-only range of targets, I believe considerable
>> documentation is necessary to make this happen.
>
> I know the feeling. I've been there myself as Jenkins is indeed a very
> flexible tool. And I don't, by any means, claim to be an expert on 
> dealing with it.
>
> Usually what I do is to peak at the documentation and at what others have
> implemented. Then I work on extending/modifying things to my needs.

same here.  In the case of the GDB and LLVM buildbots there was some
documentation to start from, although I'd still to figure several things
out by myself.  Fortunately, the buildbot docs aren't that forbidding ;-)

> The GDB build job (currently running for aarch64/armhf/x86_64) I put
> together is here:
>
> https://git.linaro.org/ci/job/configs.git/tree/tcwg-gdb.yaml

Fine, thanks.

> But I expect we will have to adjust this to make it a bit more generic so
> other architectures can use it.

I'll certainly have look.  If nothing else, it's a start and way way
easier than having to start from scratch.

>> Besides, I seem to have glimpsed from the Linaro instance that the
>> builders use Docker.  Is this a requirement or just a convenience?  I'm
>> asking because there's no current Docker port to Solaris (there used to
>> be one based on zones, but it's no longer maintained) and the
>> buildbot-based builders I'm running (for both LLVM and GDB) do fine
>> without.
>
> That is a convenience so we can share hardware resources. It is possible to
> use real hardware to run the jobs. One may need to adjust the

For my existing buildbots, I let them run inside Solaris zones (the
equivalent of Linux containers) and could use ressource control features
to provide additional containment (e.g. cpu, memory use) if need be.

> configurations a bit (distro, packages etc), but a job can automate some 
> of that. Details about distros to use and packages to install still need to
> be investigated/discussed.

In my case, I start from a configuration matching what I use for manual
GDB builds, afterwards keeping the system up to date about once a
months.  This makes the host somewhat a moving target, but the rate of
chance hasn't ever caused problems.

Documenting the set of necessary packages is similar to what one needs
for manual GDB builds, just a bit more formalized.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: [RFC] Proposal for hosting GDB CI builds

[Rainer Orth]

Hi Luis,

>>  From the reports you shared above, it seems there are lots of tests 
>> with "random" results (or rather lots of differences between two
>> runs). My fear is that reports will soon get ignored if there are too
>> many false regressions. But I hope that making such variations public
>> will help fix the root cause.
>
> Unfortunately that's always been the nature of GDB's testsuite. It has
> improved over the year, but there are still problematic tests. We will 
> certainly have to put those into an exception list when reporting the
> summaries, at least at an initial stage.

on the Solaris side, things are considerably worse: there are currently
2000+ FAILs for a GDB testsuite run, with considerable variation between
reruns.  I've never got a grip on what's going on, unfortunately, and in
the end turned the Solaris GDB buildbots to be build-only so I'd at
least catch build failures early.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: [RFC] Proposal for hosting GDB CI builds

[Luis Machado]

On 7/1/21 9:06 AM, Rainer Orth wrote:
> Hi Luis,
> 
>>>> Linaro can take care of providing builders and build jobs for ARM. Other
>>>> architectures would be handled by their respective contributors. Those
>>>> contributors can write jobs and plug builders as needed.
>>> thanks for coming forward with this: this is very welcome, given how
>>> easy it is to miss build failures and other issues especially on
>>> not-so-common targets.
>>> However, is there any documentation on setting up new builders?  I've
>>> never dealt with Jenkins before, and from glimpsing over the docs some
>>> time ago when Jeff Law talked about extending this GCC builders to a
>>> wider range of architectures left me completely at a loss: the whole
>>> thing felt like a total moloch with an incredible range of abilities,
>>> but little to no guidance on how to start.  If the GDB CI wants to
>>> extend beyond a Linux-only range of targets, I believe considerable
>>> documentation is necessary to make this happen.
>>
>> I know the feeling. I've been there myself as Jenkins is indeed a very
>> flexible tool. And I don't, by any means, claim to be an expert on
>> dealing with it.
>>
>> Usually what I do is to peak at the documentation and at what others have
>> implemented. Then I work on extending/modifying things to my needs.
> 
> same here.  In the case of the GDB and LLVM buildbots there was some
> documentation to start from, although I'd still to figure several things
> out by myself.  Fortunately, the buildbot docs aren't that forbidding ;-)
> 
>> The GDB build job (currently running for aarch64/armhf/x86_64) I put
>> together is here:
>>
>> https://git.linaro.org/ci/job/configs.git/tree/tcwg-gdb.yaml
> 
> Fine, thanks.
> 
>> But I expect we will have to adjust this to make it a bit more generic so
>> other architectures can use it.
> 
> I'll certainly have look.  If nothing else, it's a start and way way
> easier than having to start from scratch.
> 
>>> Besides, I seem to have glimpsed from the Linaro instance that the
>>> builders use Docker.  Is this a requirement or just a convenience?  I'm
>>> asking because there's no current Docker port to Solaris (there used to
>>> be one based on zones, but it's no longer maintained) and the
>>> buildbot-based builders I'm running (for both LLVM and GDB) do fine
>>> without.
>>
>> That is a convenience so we can share hardware resources. It is possible to
>> use real hardware to run the jobs. One may need to adjust the
> 
> For my existing buildbots, I let them run inside Solaris zones (the
> equivalent of Linux containers) and could use ressource control features
> to provide additional containment (e.g. cpu, memory use) if need be.
> 

That's good. One other benefit of using docker images is the consistency 
between runs. You have control over the exact distro + set of packages 
that are installed in the image, so you have a better chance of being 
able to repeat a run.

>> configurations a bit (distro, packages etc), but a job can automate some
>> of that. Details about distros to use and packages to install still need to
>> be investigated/discussed.
> 
> In my case, I start from a configuration matching what I use for manual
> GDB builds, afterwards keeping the system up to date about once a
> months.  This makes the host somewhat a moving target, but the rate of
> chance hasn't ever caused problems.

I think that's reasonable. Regular updates shouldn't cause breakage to 
GDB. If they do, that's a sign that something was/got broken anyway.

> 
> Documenting the set of necessary packages is similar to what one needs
> for manual GDB builds, just a bit more formalized.

We maintain a set of required packages in a separate file. That file 
gets loaded and processed so we are sure all the dependencies are met. 
So coming up with a similar non-docker-based mechanism shouldn't be hard.

Re: [RFC] Proposal for hosting GDB CI builds

[Rainer Orth]

Hi Luis,

>>>> Besides, I seem to have glimpsed from the Linaro instance that the
>>>> builders use Docker.  Is this a requirement or just a convenience?  I'm
>>>> asking because there's no current Docker port to Solaris (there used to
>>>> be one based on zones, but it's no longer maintained) and the
>>>> buildbot-based builders I'm running (for both LLVM and GDB) do fine
>>>> without.
>>>
>>> That is a convenience so we can share hardware resources. It is possible to
>>> use real hardware to run the jobs. One may need to adjust the
>> For my existing buildbots, I let them run inside Solaris zones (the
>> equivalent of Linux containers) and could use ressource control features
>> to provide additional containment (e.g. cpu, memory use) if need be.
>> 
>
> That's good. One other benefit of using docker images is the consistency
> between runs. You have control over the exact distro + set of packages 
> that are installed in the image, so you have a better chance of being able
> to repeat a run.

understood.  You can achieve somethine similar when installing Solaris
zones using particular profiles.  IIUC that's what had been used for the
Solaris port of Docker.

>>> configurations a bit (distro, packages etc), but a job can automate some
>>> of that. Details about distros to use and packages to install still need to
>>> be investigated/discussed.
>> In my case, I start from a configuration matching what I use for manual
>> GDB builds, afterwards keeping the system up to date about once a
>> months.  This makes the host somewhat a moving target, but the rate of
>> chance hasn't ever caused problems.
>
> I think that's reasonable. Regular updates shouldn't cause breakage to
> GDB. If they do, that's a sign that something was/got broken anyway.

Exactly.  Besides, I've got good contacts to Solaris engineering in case
I cannot figure out what's wrong by myself.

>> Documenting the set of necessary packages is similar to what one needs
>> for manual GDB builds, just a bit more formalized.
>
> We maintain a set of required packages in a separate file. That file gets
> loaded and processed so we are sure all the dependencies are met. So coming
> up with a similar non-docker-based mechanism shouldn't be hard.

Agreed.  I'll certainly try that once we get there.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: [RFC] Proposal for hosting GDB CI builds

[Luis Machado]

Any other feedback on this?

On 6/30/21 1:46 PM, Luis Machado wrote:
> Hi,
> 
> This proposal comes as an attempt to revive the GDB CI builds, given the 
> previous one (buildbot) is no longer being maintained by Sergio (thanks 
> a lot for keeping it working for all these years by the way).
> 
> CI GDB builds are a great help for spotting regressions without having 
> to do the tedious and time-consuming work of running the GDB testsuite 
> for each git revision, for each architecture and reading through 
> hundreds of lines of summaries. If a regression is spotted, then one 
> needs to bisect to find the culprit. This isn't great, specially for 
> architectures without great availability of desktop hardware.
> 
> Ideally, for each commit, we'd run full builds to validate the state of 
> the tree, but we're not quite there yet. So meanwhile, having some level 
> of automation to get the builds done without manual intervention sounds 
> like a reasonable step forward.
> 
>  From previous IRC conversations, it seems to be a consensus that 
> availability of processing power is not a problem. It is reasonably easy 
> to find hardware to do some builds. The most difficult resource to find 
> is manpower to setup the CI infrastructure and keep it running.
> 
> With the above said, I've discussed this internally at Linaro and we can 
> spare some manpower to setup and maintain an isolated Linaro-hosted 
> Jenkins instance for GDB CI builds.
> 
> Linaro can take care of providing builders and build jobs for ARM. Other 
> architectures would be handled by their respective contributors. Those 
> contributors can write jobs and plug builders as needed.
> 
> Setting up new jobs doesn't require the use of the web interface. It can 
> be done with yaml files in a git repo. It is reasonably simple.
> 
> You can see an example of the Linaro CI here: https://ci.linaro.org/
> 
> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/
> 
> And also the summaries for GDB testsuite runs: 
> https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html 
> 
> 
> Of course, this effort only makes sense if the community is OK with 
> using Jenkins as the CI mechanism and if it actually sees value in 
> having a system like this in place.
> 
> Thoughts? Questions?

Re: [RFC] Proposal for hosting GDB CI builds

[Simon Marchi]

On 2021-07-20 11:21 a.m., Luis Machado wrote:
> Any other feedback on this?

Sorry, I procrastinated replying to this thread for a while - too many
things to do.

> On 6/30/21 1:46 PM, Luis Machado wrote:
>> Hi,
>>
>> This proposal comes as an attempt to revive the GDB CI builds, given the previous one (buildbot) is no longer being maintained by Sergio (thanks a lot for keeping it working for all these years by the way).
>>
>> CI GDB builds are a great help for spotting regressions without having to do the tedious and time-consuming work of running the GDB testsuite for each git revision, for each architecture and reading through hundreds of lines of summaries. If a regression is spotted, then one needs to bisect to find the culprit. This isn't great, specially for architectures without great availability of desktop hardware.
>>
>> Ideally, for each commit, we'd run full builds to validate the state of the tree, but we're not quite there yet. So meanwhile, having some level of automation to get the builds done without manual intervention sounds like a reasonable step forward.
>>
>>  From previous IRC conversations, it seems to be a consensus that availability of processing power is not a problem. It is reasonably easy to find hardware to do some builds. The most difficult resource to find is manpower to setup the CI infrastructure and keep it running.

I agree, it's almost a full time job to babysit a CI.

>> With the above said, I've discussed this internally at Linaro and we can spare some manpower to setup and maintain an isolated Linaro-hosted Jenkins instance for GDB CI builds.
>>
>> Linaro can take care of providing builders and build jobs for ARM. Other architectures would be handled by their respective contributors. Those contributors can write jobs and plug builders as needed.
>>
>> Setting up new jobs doesn't require the use of the web interface. It can be done with yaml files in a git repo. It is reasonably simple.
>>
>> You can see an example of the Linaro CI here: https://ci.linaro.org/
>>
>> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/
>>
>> And also the summaries for GDB testsuite runs: https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html

The reports look really good.

When I looked into setting up a CI, the most difficult / annoying part
was analyzing the results.  Not only the racy tests that intermittently
fail, but also some tests that alternate between PASS and KFAIL (I don't
remember which exactly, I think the "many short lived threads" one).

The previous buildbot had a racy test analysis to detemine which tests
were flaky, and (I suppose) ignore them in the results.  I don't know
how well that worked.  But if that makes it simpler, I'd be ok with an
hand-maintained list of known flaky/racy tests that should be ignored
when comparing the results.  Ignoring some test results is not great,
but if it can make the CI's results more trustworthy, I say it's better
than letting the racy/flaky tests "contaminate" the results of all the
tests that aren't.

In the end, we want a CI with results we can trust.  With the racy/flaky
tests, it often ends up that builds fail but "oh it's just that test
failing, it's fine".  And this is how regressions can slip in.

>> Of course, this effort only makes sense if the community is OK with using Jenkins as the CI mechanism and if it actually sees value in having a system like this in place.

I am of course in favor of it.  And in my opinion, whoever makes the
effort has the final word on how it's done.  So if you are used to
Jenkins, let's go with Jenkins.

Simon