Re: Regressions getting more common

[Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE>]

Hi Simon,

> I think it would be good to reboot the buildbot, but start by focusing
> on what delivers the best "bang for the buck".  I remember workers
> lagging a lot behind, meaning we would get notifications of breakage
> quite a lot after a commit was pushed.  I think it would be good to not
> make the workers build all commits on master.  Either do one build a
> day, or have them constantly build the current master (as a background /
> low priority task).  If there's a regression, then there's a window of
> commits that might be responsible for it.  But in any case, I would see
> having a stable post-commit CI as the first step.

maybe we can learn a thing or two from the LLVM buildbots

	https://llvm.org/docs/HowToAddABuilder.html

with the config at

	https://github.com/llvm/llvm-zorg

For one, they do group several commits into one build if the bots lag
too far behind (which is a strict necessity given the enormous size of
LLVM, even if many bots have a smaller scope like only testing clang,
compiler-rt/the sanitizers, ...).  Another possibility is not to run
full builds every time but only under special circumstances (haven't
fully understood which those are, but I'm operating the Solaris
buildbots in that mode to avoid overwhelming them), defaulting to
incremental builds.  Given that GDB build times are reasonably short (a
matter of minutes usually), I'm unsure if this is necessary, though.

> Then, we can look at having try jobs work again.  I never really liked
> submitting try jobs through the buildbot command line tool, I always
> found there was to little feedback (did my upload work? what jobs did I
> initiate?).  I would receive test results by email and would have a ahrd
> time figuring out which result was for which patch.

I have to admit I haven't ever used the try jobs myself.  However, AFAIK
they still do work with the current buildmaster.  Only the jobs
triggered by commits seem to have stopped.

That said, they are certainly useful, especially to allow people to test
their patches on uncommon targets.  I know for a fact that several GDB
developers used my Solaris bots that way.  While the GCC compile farm
could be used as well, this means much more effort for the developers.

However, as a maintainer of two of those bots, I'd be little pleased to
have to maintain and watch over two different packages for this purpose:
the charm of the current buildbot setup is that is serves both (as does,
e.g. the trybot component of the golang farm).

> One idea would be to re-use the now abandonned Gerrit instance for this.
> Those who have use Gerrit will probably agree that it integrates quite
> well with CI.  After pushing your patch for review, you can ask the CI
> to test a given patch (often by setting a label on it).  The CI posts a
> comment on the patch with a link to the build, so you can follow it if
> you want.  Once it has ran, the CI posts comments on the patch to say
> that version N of the patch has this result, again with a link to the
> build.
>
> I would be happy to help take over maintenance of the buildbot master
> from Sergio, but I wouldn't want to be alone in this.

It would be really great if the GDB bots could get started again.
However, I'm unable to help: way too much on my plate already.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University