Re: Moving buildbot master to sourceware

[Carlos O'Donell <carlos@redhat.com>]

On 4/6/22 04:39, Mark Wielaard wrote:
> Hi Carlos,
> 
> On Mon, Apr 04, 2022 at 06:15:01PM -0400, Carlos O'Donell via Overseers wrote:
>>> Maintenance of the master is fairly minimal and it doesn't require
>>> that many resources. The current state database, git poller and build
>>> logs total less than 3.5G for hundreds to thousands of builds for all
>>> of the above projects.
>>>
>>> The bike shed items for installing this on sourceware are:
>>>
>>> - Whether to install buildbot from pip or epel. My preference is
>>>   throug pip under a user account since that is how I run it
>>>   locally. But using the epel package is fine too.
>>> - Running it under sourceware.org/buildbot or adding a vhost
>>>   builder.sourceware.org. My preference is for a vhost.
>>> - A git repo for the master.cfg file builder.git (*)
>>>   Plus a git trigger to reload to config by the buildbot master.
>>> - A user, group and homedir to store the state and password/key files
>>>   /sourceware/home/builder/
>>> - The initial members of the builder group which can update the git
>>>   repo and key files. The people on CC?
>>> - A mailinglist to coordinate builder@sourceware.org or we could
>>>   simply use the overseers mailinglist.
>>
>> Can we deploy this as a buildbot master container with some customization?
>>
>> This would make the service easier to keep going during sourcware updates because
>> dependencies would be decoupled.
>>
>> It would allow developers to test pre-production changes locally by using the same
>> container.
> 
> The most natural "container" for a pure python app that we'll be
> maintaining through pip is a python virtualenv. I have scripts to
> setup and update the virtualenv so it is easy to replicate and play
> with independently. You can wrap those in a whole system image of
> course. I'll make sure to document that process.

Agreed, virtualenv is a good alterantive.
 
>> This is how upstream patchwork changes are being developed, which DJ and I are
>> looking at because we will need to update patchwork to v3.1 upstream when it's
>> released with DJ's new changes.
> 
> Is there documentation on how sourceware patchwork is currently
> deployed? Are you actually running a patches upstream version? I am
> happy to help upgrading.

Siddhesh has notes.

We are running Patchwork 2.2 stable with just a few fixes related to our environment.

Thanks for the offer. We're waiting for upstream v3.1 to be released with DJ's patches.

> I haven't had to make any changes to upstream/packaged/pip buildbot
> because all configuration is done through a python file (so if you do
> really have to monkey patch the upstream version you can add those to
> the config file).

Right.
 
>> My plan was to work with downstream distributions, ISVs, and IHVs who are interested
>> in a particular project to donate workers. In order to do that though we have to make
>> the act of installing and setting up a worker very low friction and very safe.
>>
>> The workers should arguably have a net positive impact for the downstream, or ISV, or
>> IHV, so they need a way to measure that they prevented a patch that breaks their
>> configuration of interest. So I also need to make sure they can gather metrics for
>> reporting on the resources they are spending e.g. how soon after going red on their
>> build was it corrected?
>>
>> For example, could Red Hat or IBM donate workers?
>>
>> Can we grow best practice CI adoption in our mature projects by making it easy?
> 
> All good ideas and, if you have the matching project developers, then
> adding more workers is in general a good thing.
> 
> There are a few lessons about making CI most effective:
> 
> - Start small with a minimal environment and a minimal subset of tests
>   that just have to work.
> - Always keep the builder green.
> - If it is a flaky test leave it out.
> - If there is a test that is failing consistently, but nobody fixes
>   it, file a bug and take it out.
> - Only expand if you have developers that care to keep that environment green.
> - Only add workers if you have devxelopers on the project that can take
>   care of the new build environments created for it.
> - If you have a worker that is unique in some way (say for a
>   non-common architecture) arrange for project maintainers to get
>   access to it outside the buildbot so they can actually debug
>   problems.

Agreed.
 
> For tracking actual success/failures/added/removed tests we could look
> at feeding the test logs from buildbot to bunsen.

Agreed. Though I sill want to point out that each individual may have different
metrics for what means success and we need to ensure they can easily gather
those metrics.

-- 
Cheers,
Carlos.