Re: Regressions getting more common

[Simon Marchi <simark@simark.ca>]

On 2020-10-13 1:05 p.m., Luis Machado via Gdb wrote:
> Hi,
>
> I don't know about other non-x86 architectures, but over the past year I've been noticing more and more regressions being introduced, unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if you have to keep tracking things manually, like we do now.
>
> The buildbots worked great for this very purpose, but Sergio has moved on to other duties (thanks for all the work!) and can't maintain it anymore. The builders are still there though, sitting mostly idle.
> We have a beefy ARM/AArch64 builder, which I can maintain for others to use.
>
> We can do better than to declare things OK after a single round of tests under x86, which has been the trend unfortunately.
>
> The subject of better CI has come up multiple times on IRC, with sad memories of the gerrit experiment's demise. Now we're left with review by e-mail and no broad testing.
>
> I think we need to discuss better validation pre-commit and possible CI solutions for GDB. It is pretty easy to exercise x86, but it doesn't sound fair to other architectures to have to keep cleaning up after things that have only been validated on that architecture.
>
> It would be great to establish a roadmap so we can get GDB's testing to today's standards, and maybe revisit the use of more modern patch review tools while at it.
>
> What do you think?

I agree with all of this.

It all comes down to:

- human resources: a system like that is not fire and forget, there's
  always something to look into to make sure it runs smoothly
- hardware resources: it takes a lot of CPU time, that takes some
  dedicated machines

I think it would be good to reboot the buildbot, but start by focusing
on what delivers the best "bang for the buck".  I remember workers
lagging a lot behind, meaning we would get notifications of breakage
quite a lot after a commit was pushed.  I think it would be good to not
make the workers build all commits on master.  Either do one build a
day, or have them constantly build the current master (as a background /
low priority task).  If there's a regression, then there's a window of
commits that might be responsible for it.  But in any case, I would see
having a stable post-commit CI as the first step.

Then, we can look at having try jobs work again.  I never really liked
submitting try jobs through the buildbot command line tool, I always
found there was to little feedback (did my upload work? what jobs did I
initiate?).  I would receive test results by email and would have a ahrd
time figuring out which result was for which patch.

One idea would be to re-use the now abandonned Gerrit instance for this.
Those who have use Gerrit will probably agree that it integrates quite
well with CI.  After pushing your patch for review, you can ask the CI
to test a given patch (often by setting a label on it).  The CI posts a
comment on the patch with a link to the build, so you can follow it if
you want.  Once it has ran, the CI posts comments on the patch to say
that version N of the patch has this result, again with a link to the
build.

I would be happy to help take over maintenance of the buildbot master
from Sergio, but I wouldn't want to be alone in this.

Simon