Re: Moving buildbot master to sourceware

[Philip Herron <philip.herron@embecosm.com>]

Hi all,

Just want to say how useful these built bots have been for our
project. We are on GitHub which has been good in some respects in
terms of engagement but I personally don't like it, it does offer
automation but these build bots have time and time proven to be more
reliable in finding issues/warnings and failures on different
platforms. I wanted to reply here because we have started a side
project to investigate test automation for the compiler, so that means
we have a git repo that has some rust code to invoke our compiler on
the rustc test suite, we have been toying with the idea of building a
basic dashboard out of this but I bet there is overlap with this and
what built bots do.

Overall just want to say a big thank you to all that's involved in
keeping this going for us.

--Phil

On Wed, 6 Apr 2022 at 17:27, Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 4/6/22 04:39, Mark Wielaard wrote:
> > Hi Carlos,
> >
> > On Mon, Apr 04, 2022 at 06:15:01PM -0400, Carlos O'Donell via Overseers wrote:
> >>> Maintenance of the master is fairly minimal and it doesn't require
> >>> that many resources. The current state database, git poller and build
> >>> logs total less than 3.5G for hundreds to thousands of builds for all
> >>> of the above projects.
> >>>
> >>> The bike shed items for installing this on sourceware are:
> >>>
> >>> - Whether to install buildbot from pip or epel. My preference is
> >>>   throug pip under a user account since that is how I run it
> >>>   locally. But using the epel package is fine too.
> >>> - Running it under sourceware.org/buildbot or adding a vhost
> >>>   builder.sourceware.org. My preference is for a vhost.
> >>> - A git repo for the master.cfg file builder.git (*)
> >>>   Plus a git trigger to reload to config by the buildbot master.
> >>> - A user, group and homedir to store the state and password/key files
> >>>   /sourceware/home/builder/
> >>> - The initial members of the builder group which can update the git
> >>>   repo and key files. The people on CC?
> >>> - A mailinglist to coordinate builder@sourceware.org or we could
> >>>   simply use the overseers mailinglist.
> >>
> >> Can we deploy this as a buildbot master container with some customization?
> >>
> >> This would make the service easier to keep going during sourcware updates because
> >> dependencies would be decoupled.
> >>
> >> It would allow developers to test pre-production changes locally by using the same
> >> container.
> >
> > The most natural "container" for a pure python app that we'll be
> > maintaining through pip is a python virtualenv. I have scripts to
> > setup and update the virtualenv so it is easy to replicate and play
> > with independently. You can wrap those in a whole system image of
> > course. I'll make sure to document that process.
>
> Agreed, virtualenv is a good alterantive.
>
> >> This is how upstream patchwork changes are being developed, which DJ and I are
> >> looking at because we will need to update patchwork to v3.1 upstream when it's
> >> released with DJ's new changes.
> >
> > Is there documentation on how sourceware patchwork is currently
> > deployed? Are you actually running a patches upstream version? I am
> > happy to help upgrading.
>
> Siddhesh has notes.
>
> We are running Patchwork 2.2 stable with just a few fixes related to our environment.
>
> Thanks for the offer. We're waiting for upstream v3.1 to be released with DJ's patches.
>
> > I haven't had to make any changes to upstream/packaged/pip buildbot
> > because all configuration is done through a python file (so if you do
> > really have to monkey patch the upstream version you can add those to
> > the config file).
>
> Right.
>
> >> My plan was to work with downstream distributions, ISVs, and IHVs who are interested
> >> in a particular project to donate workers. In order to do that though we have to make
> >> the act of installing and setting up a worker very low friction and very safe.
> >>
> >> The workers should arguably have a net positive impact for the downstream, or ISV, or
> >> IHV, so they need a way to measure that they prevented a patch that breaks their
> >> configuration of interest. So I also need to make sure they can gather metrics for
> >> reporting on the resources they are spending e.g. how soon after going red on their
> >> build was it corrected?
> >>
> >> For example, could Red Hat or IBM donate workers?
> >>
> >> Can we grow best practice CI adoption in our mature projects by making it easy?
> >
> > All good ideas and, if you have the matching project developers, then
> > adding more workers is in general a good thing.
> >
> > There are a few lessons about making CI most effective:
> >
> > - Start small with a minimal environment and a minimal subset of tests
> >   that just have to work.
> > - Always keep the builder green.
> > - If it is a flaky test leave it out.
> > - If there is a test that is failing consistently, but nobody fixes
> >   it, file a bug and take it out.
> > - Only expand if you have developers that care to keep that environment green.
> > - Only add workers if you have devxelopers on the project that can take
> >   care of the new build environments created for it.
> > - If you have a worker that is unique in some way (say for a
> >   non-common architecture) arrange for project maintainers to get
> >   access to it outside the buildbot so they can actually debug
> >   problems.
>
> Agreed.
>
> > For tracking actual success/failures/added/removed tests we could look
> > at feeding the test logs from buildbot to bunsen.
>
> Agreed. Though I sill want to point out that each individual may have different
> metrics for what means success and we need to ensure they can easily gather
> those metrics.
>
> --
> Cheers,
> Carlos.
>