Moving buildbot master to sourceware

Moving buildbot master to sourceware

[Mark Wielaard]

Hi,

For several years now I have been running the buildbot master for
various sourceware projects (bzip2, debugedit, dwz, elfutils,
libabigail, valgrind and gccrs - that last one not actually on
sourceware yet, but we'll get there eventually) on
https://builder.wildebeest.org/buildbot/

I would like to move this officially to sourceware now to make working
on this together easier and to see if we can extend the service to
other sourceware projects and see if we can get more workers. e.g. in
the past Sergio ran a separate buildbot for gdb which we might merge
together. One future integration idea would be to add buildbot
triggers for patchwork to test out proposed patches before they are
integrated.

Maintenance of the master is fairly minimal and it doesn't require
that many resources. The current state database, git poller and build
logs total less than 3.5G for hundreds to thousands of builds for all
of the above projects.

The bike shed items for installing this on sourceware are:

- Whether to install buildbot from pip or epel. My preference is
  throug pip under a user account since that is how I run it
  locally. But using the epel package is fine too.
- Running it under sourceware.org/buildbot or adding a vhost
  builder.sourceware.org. My preference is for a vhost.
- A git repo for the master.cfg file builder.git (*)
  Plus a git trigger to reload to config by the buildbot master.
- A user, group and homedir to store the state and password/key files
  /sourceware/home/builder/
- The initial members of the builder group which can update the git
  repo and key files. The people on CC?
- A mailinglist to coordinate builder@sourceware.org or we could
  simply use the overseers mailinglist.

(*) The master.cfg file is actually a python script, and I am slightly
    embarrased by it. It would be great if someone with a bit more python
    experience could refactor it so that the Builder setups aren't just
    done by copy/paste tens of identical lines.

Once setup the real work is on the projects using the buildbot to
define and coordinating with the volunteers running the buildbot
workers in case there is some issue that cannot be explained by the
buildbot test run. This can in general be done directly between the
project maintainers and the buildbot worker volunteers, who can also
provide direct access to their setups. But it would be good to have a
central list to coordinate such issues.

There are currently the following 10 workers:
- centos-x86_64, debian-amd64, debian-i386, fedora-x86_64
  Ran by me as kvm libvirt instances.
- debian-ppc64 ran by Thomas Fitzsimmons
- fedora-ppc64 and fedora-ppc64le ran by Dan Horák
  thanks to the Brno University
- fedora-s390x Ran by Dan Horák
  thanks to the Marist University
- debian-arm64 and debian-armhf small boards ran by me.

I am also in the process of setting up a Ryzen9 machine with 3 extra
VMs using Fedora CoreOS that can run container images using Podman. It
does need a workaround for a small docker/podman incompatibility
https://github.com/containers/podman/issues/11668 But once it works it
could be a model of extending the workers in a simpler way, where the
container images are defined together with the master to give project
maintainers more flexibility in their testing environment.

Let me know what you think and whether you have ideas for getting more
worker machines.

Cheers,

Mark

Re: Moving buildbot master to sourceware

[Frank Ch. Eigler]

Hi -

On Mon, Apr 04, 2022 at 12:02:06PM +0200, Mark Wielaard via Overseers wrote:
> For several years now I have been running the buildbot master for
> various sourceware projects (bzip2, debugedit, dwz, elfutils,
> libabigail, valgrind and gccrs - that last one not actually on
> sourceware yet, but we'll get there eventually) on
> https://builder.wildebeest.org/buildbot/
> 
> I would like to move this officially to sourceware now [...]

Why not.

> The bike shed items for installing this on sourceware are:
> [...]

All those look fine to me.

> Once setup the real work is on the projects using the buildbot to
> define and coordinating with the volunteers running the buildbot
> workers [...]

If people are happy with a pooled trust model where all project
worker/buildbot operators trust all buildbot-master admins, sure.

Perhaps one can encourage further donation of buildbots by publishing
instructions about how to set them up in a secure way, meaning for
example that the bots can't access/hurt the donor's network.

- FChE

Re: Moving buildbot master to sourceware

[Mark Wielaard]

Hi Frank,

On Mon, Apr 04, 2022 at 11:24:39AM -0400, Frank Ch. Eigler wrote:
> On Mon, Apr 04, 2022 at 12:02:06PM +0200, Mark Wielaard via Overseers wrote:
> > For several years now I have been running the buildbot master for
> > various sourceware projects (bzip2, debugedit, dwz, elfutils,
> > libabigail, valgrind and gccrs - that last one not actually on
> > sourceware yet, but we'll get there eventually) on
> > https://builder.wildebeest.org/buildbot/
> > 
> > I would like to move this officially to sourceware now [...]
> 
> Why not.
> 
> > The bike shed items for installing this on sourceware are:
> > [...]
> 
> All those look fine to me.

Great thanks. This is mainly the setup as it is on my centos7 server,
so it should translate fairly smoothly to the sourceware rhel8
setup. I'll make sure to document and automate it all so it is easy to
replicate and update.

> > Once setup the real work is on the projects using the buildbot to
> > define and coordinating with the volunteers running the buildbot
> > workers [...]
> 
> If people are happy with a pooled trust model where all project
> worker/buildbot operators trust all buildbot-master admins, sure.

The buildbot worker operators really have to even trust all project
committers, in theory any of them could commit any code that is then
executed on the workers. So if we trust project committers I think we
can also trust the buildbot-master admins.

If projects have their own dedicated workers then we could setup a
multi-master model so only builds of certain projects are executed on
specific workers. But I think as long as we share the same workers
between the projects it makes sense to have a group of shared
buildbot-master admins. So they can coordinate to not overload the
workers.

> Perhaps one can encourage further donation of buildbots by publishing
> instructions about how to set them up in a secure way, meaning for
> example that the bots can't access/hurt the donor's network.

Good idea. I don't have any special recommendations for network
isolation but can certainly document the VM/user/container setups to
run the buildbot-worker in isolated environments.

Cheers,

Mark

Re: Moving buildbot master to sourceware

[Carlos O'Donell]

On 4/4/22 06:02, Mark Wielaard via Overseers wrote:
> I would like to move this officially to sourceware now to make working
> on this together easier and to see if we can extend the service to
> other sourceware projects and see if we can get more workers. e.g. in
> the past Sergio ran a separate buildbot for gdb which we might merge
> together. One future integration idea would be to add buildbot
> triggers for patchwork to test out proposed patches before they are
> integrated.

Thank you for volunteering to do this!

A central buildbot would be nice.

> Maintenance of the master is fairly minimal and it doesn't require
> that many resources. The current state database, git poller and build
> logs total less than 3.5G for hundreds to thousands of builds for all
> of the above projects.
> 
> The bike shed items for installing this on sourceware are:
> 
> - Whether to install buildbot from pip or epel. My preference is
>   throug pip under a user account since that is how I run it
>   locally. But using the epel package is fine too.
> - Running it under sourceware.org/buildbot or adding a vhost
>   builder.sourceware.org. My preference is for a vhost.
> - A git repo for the master.cfg file builder.git (*)
>   Plus a git trigger to reload to config by the buildbot master.
> - A user, group and homedir to store the state and password/key files
>   /sourceware/home/builder/
> - The initial members of the builder group which can update the git
>   repo and key files. The people on CC?
> - A mailinglist to coordinate builder@sourceware.org or we could
>   simply use the overseers mailinglist.

Can we deploy this as a buildbot master container with some customization?

This would make the service easier to keep going during sourcware updates because
dependencies would be decoupled.

It would allow developers to test pre-production changes locally by using the same
container.

This is how upstream patchwork changes are being developed, which DJ and I are
looking at because we will need to update patchwork to v3.1 upstream when it's
released with DJ's new changes. Upstream patchwork can be deployed in a container.
We should switch to container deployments for the current instance.

Reference:
http://docs.buildbot.net/current/tutorial/docker.html?highlight=docker
https://github.com/getpatchwork/patchwork/blob/master/docker-compose-pg.yml

> (*) The master.cfg file is actually a python script, and I am slightly
>     embarrased by it. It would be great if someone with a bit more python
>     experience could refactor it so that the Builder setups aren't just
>     done by copy/paste tens of identical lines.
> 
> Once setup the real work is on the projects using the buildbot to
> define and coordinating with the volunteers running the buildbot
> workers in case there is some issue that cannot be explained by the
> buildbot test run. This can in general be done directly between the
> project maintainers and the buildbot worker volunteers, who can also
> provide direct access to their setups. But it would be good to have a
> central list to coordinate such issues.
> 
> There are currently the following 10 workers:
> - centos-x86_64, debian-amd64, debian-i386, fedora-x86_64
>   Ran by me as kvm libvirt instances.
> - debian-ppc64 ran by Thomas Fitzsimmons
> - fedora-ppc64 and fedora-ppc64le ran by Dan Horák
>   thanks to the Brno University
> - fedora-s390x Ran by Dan Horák
>   thanks to the Marist University
> - debian-arm64 and debian-armhf small boards ran by me.
> 
> I am also in the process of setting up a Ryzen9 machine with 3 extra
> VMs using Fedora CoreOS that can run container images using Podman. It
> does need a workaround for a small docker/podman incompatibility
> https://github.com/containers/podman/issues/11668 But once it works it
> could be a model of extending the workers in a simpler way, where the
> container images are defined together with the master to give project
> maintainers more flexibility in their testing environment.

This is similar to how DJ's CI/CD runners work. We have a VM, that runs the trybot
runner which runs tests in a disconnected container. The goal is to eventually ship an
Ansible playbook to setup the VM with some templated defaults for "runner" volunteers
to use for their testing. 
> Let me know what you think and whether you have ideas for getting more
> worker machines.

My plan was to work with downstream distributions, ISVs, and IHVs who are interested
in a particular project to donate workers. In order to do that though we have to make
the act of installing and setting up a worker very low friction and very safe.

The workers should arguably have a net positive impact for the downstream, or ISV, or
IHV, so they need a way to measure that they prevented a patch that breaks their
configuration of interest. So I also need to make sure they can gather metrics for
reporting on the resources they are spending e.g. how soon after going red on their
build was it corrected?

For example, could Red Hat or IBM donate workers?

Can we grow best practice CI adoption in our mature projects by making it easy?

A example of this is the Kernel CI project at the Linux Foundation:
https://linux.kernelci.org/job/
https://foundation.kernelci.org/

-- 
Cheers,
Carlos.

Re: Moving buildbot master to sourceware

[Mark Wielaard]

Hi Carlos,

On Mon, Apr 04, 2022 at 06:15:01PM -0400, Carlos O'Donell via Overseers wrote:
> > Maintenance of the master is fairly minimal and it doesn't require
> > that many resources. The current state database, git poller and build
> > logs total less than 3.5G for hundreds to thousands of builds for all
> > of the above projects.
> > 
> > The bike shed items for installing this on sourceware are:
> > 
> > - Whether to install buildbot from pip or epel. My preference is
> >   throug pip under a user account since that is how I run it
> >   locally. But using the epel package is fine too.
> > - Running it under sourceware.org/buildbot or adding a vhost
> >   builder.sourceware.org. My preference is for a vhost.
> > - A git repo for the master.cfg file builder.git (*)
> >   Plus a git trigger to reload to config by the buildbot master.
> > - A user, group and homedir to store the state and password/key files
> >   /sourceware/home/builder/
> > - The initial members of the builder group which can update the git
> >   repo and key files. The people on CC?
> > - A mailinglist to coordinate builder@sourceware.org or we could
> >   simply use the overseers mailinglist.
> 
> Can we deploy this as a buildbot master container with some customization?
> 
> This would make the service easier to keep going during sourcware updates because
> dependencies would be decoupled.
> 
> It would allow developers to test pre-production changes locally by using the same
> container.

The most natural "container" for a pure python app that we'll be
maintaining through pip is a python virtualenv. I have scripts to
setup and update the virtualenv so it is easy to replicate and play
with independently. You can wrap those in a whole system image of
course. I'll make sure to document that process.

> This is how upstream patchwork changes are being developed, which DJ and I are
> looking at because we will need to update patchwork to v3.1 upstream when it's
> released with DJ's new changes.

Is there documentation on how sourceware patchwork is currently
deployed? Are you actually running a patches upstream version? I am
happy to help upgrading.

I haven't had to make any changes to upstream/packaged/pip buildbot
because all configuration is done through a python file (so if you do
really have to monkey patch the upstream version you can add those to
the config file).

> My plan was to work with downstream distributions, ISVs, and IHVs who are interested
> in a particular project to donate workers. In order to do that though we have to make
> the act of installing and setting up a worker very low friction and very safe.
> 
> The workers should arguably have a net positive impact for the downstream, or ISV, or
> IHV, so they need a way to measure that they prevented a patch that breaks their
> configuration of interest. So I also need to make sure they can gather metrics for
> reporting on the resources they are spending e.g. how soon after going red on their
> build was it corrected?
> 
> For example, could Red Hat or IBM donate workers?
> 
> Can we grow best practice CI adoption in our mature projects by making it easy?

All good ideas and, if you have the matching project developers, then
adding more workers is in general a good thing.

There are a few lessons about making CI most effective:

- Start small with a minimal environment and a minimal subset of tests
  that just have to work.
- Always keep the builder green.
- If it is a flaky test leave it out.
- If there is a test that is failing consistently, but nobody fixes
  it, file a bug and take it out.
- Only expand if you have developers that care to keep that environment green.
- Only add workers if you have devxelopers on the project that can take
  care of the new build environments created for it.
- If you have a worker that is unique in some way (say for a
  non-common architecture) arrange for project maintainers to get
  access to it outside the buildbot so they can actually debug
  problems.

For tracking actual success/failures/added/removed tests we could look
at feeding the test logs from buildbot to bunsen.

Cheers,

Mark

Re: Moving buildbot master to sourceware

[Carlos O'Donell]

On 4/6/22 04:39, Mark Wielaard wrote:
> Hi Carlos,
> 
> On Mon, Apr 04, 2022 at 06:15:01PM -0400, Carlos O'Donell via Overseers wrote:
>>> Maintenance of the master is fairly minimal and it doesn't require
>>> that many resources. The current state database, git poller and build
>>> logs total less than 3.5G for hundreds to thousands of builds for all
>>> of the above projects.
>>>
>>> The bike shed items for installing this on sourceware are:
>>>
>>> - Whether to install buildbot from pip or epel. My preference is
>>>   throug pip under a user account since that is how I run it
>>>   locally. But using the epel package is fine too.
>>> - Running it under sourceware.org/buildbot or adding a vhost
>>>   builder.sourceware.org. My preference is for a vhost.
>>> - A git repo for the master.cfg file builder.git (*)
>>>   Plus a git trigger to reload to config by the buildbot master.
>>> - A user, group and homedir to store the state and password/key files
>>>   /sourceware/home/builder/
>>> - The initial members of the builder group which can update the git
>>>   repo and key files. The people on CC?
>>> - A mailinglist to coordinate builder@sourceware.org or we could
>>>   simply use the overseers mailinglist.
>>
>> Can we deploy this as a buildbot master container with some customization?
>>
>> This would make the service easier to keep going during sourcware updates because
>> dependencies would be decoupled.
>>
>> It would allow developers to test pre-production changes locally by using the same
>> container.
> 
> The most natural "container" for a pure python app that we'll be
> maintaining through pip is a python virtualenv. I have scripts to
> setup and update the virtualenv so it is easy to replicate and play
> with independently. You can wrap those in a whole system image of
> course. I'll make sure to document that process.

Agreed, virtualenv is a good alterantive.
 
>> This is how upstream patchwork changes are being developed, which DJ and I are
>> looking at because we will need to update patchwork to v3.1 upstream when it's
>> released with DJ's new changes.
> 
> Is there documentation on how sourceware patchwork is currently
> deployed? Are you actually running a patches upstream version? I am
> happy to help upgrading.

Siddhesh has notes.

We are running Patchwork 2.2 stable with just a few fixes related to our environment.

Thanks for the offer. We're waiting for upstream v3.1 to be released with DJ's patches.

> I haven't had to make any changes to upstream/packaged/pip buildbot
> because all configuration is done through a python file (so if you do
> really have to monkey patch the upstream version you can add those to
> the config file).

Right.
 
>> My plan was to work with downstream distributions, ISVs, and IHVs who are interested
>> in a particular project to donate workers. In order to do that though we have to make
>> the act of installing and setting up a worker very low friction and very safe.
>>
>> The workers should arguably have a net positive impact for the downstream, or ISV, or
>> IHV, so they need a way to measure that they prevented a patch that breaks their
>> configuration of interest. So I also need to make sure they can gather metrics for
>> reporting on the resources they are spending e.g. how soon after going red on their
>> build was it corrected?
>>
>> For example, could Red Hat or IBM donate workers?
>>
>> Can we grow best practice CI adoption in our mature projects by making it easy?
> 
> All good ideas and, if you have the matching project developers, then
> adding more workers is in general a good thing.
> 
> There are a few lessons about making CI most effective:
> 
> - Start small with a minimal environment and a minimal subset of tests
>   that just have to work.
> - Always keep the builder green.
> - If it is a flaky test leave it out.
> - If there is a test that is failing consistently, but nobody fixes
>   it, file a bug and take it out.
> - Only expand if you have developers that care to keep that environment green.
> - Only add workers if you have devxelopers on the project that can take
>   care of the new build environments created for it.
> - If you have a worker that is unique in some way (say for a
>   non-common architecture) arrange for project maintainers to get
>   access to it outside the buildbot so they can actually debug
>   problems.

Agreed.
 
> For tracking actual success/failures/added/removed tests we could look
> at feeding the test logs from buildbot to bunsen.

Agreed. Though I sill want to point out that each individual may have different
metrics for what means success and we need to ensure they can easily gather
those metrics.

-- 
Cheers,
Carlos.

Re: Moving buildbot master to sourceware

[Philip Herron]

Hi all,

Just want to say how useful these built bots have been for our
project. We are on GitHub which has been good in some respects in
terms of engagement but I personally don't like it, it does offer
automation but these build bots have time and time proven to be more
reliable in finding issues/warnings and failures on different
platforms. I wanted to reply here because we have started a side
project to investigate test automation for the compiler, so that means
we have a git repo that has some rust code to invoke our compiler on
the rustc test suite, we have been toying with the idea of building a
basic dashboard out of this but I bet there is overlap with this and
what built bots do.

Overall just want to say a big thank you to all that's involved in
keeping this going for us.

--Phil

On Wed, 6 Apr 2022 at 17:27, Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 4/6/22 04:39, Mark Wielaard wrote:
> > Hi Carlos,
> >
> > On Mon, Apr 04, 2022 at 06:15:01PM -0400, Carlos O'Donell via Overseers wrote:
> >>> Maintenance of the master is fairly minimal and it doesn't require
> >>> that many resources. The current state database, git poller and build
> >>> logs total less than 3.5G for hundreds to thousands of builds for all
> >>> of the above projects.
> >>>
> >>> The bike shed items for installing this on sourceware are:
> >>>
> >>> - Whether to install buildbot from pip or epel. My preference is
> >>>   throug pip under a user account since that is how I run it
> >>>   locally. But using the epel package is fine too.
> >>> - Running it under sourceware.org/buildbot or adding a vhost
> >>>   builder.sourceware.org. My preference is for a vhost.
> >>> - A git repo for the master.cfg file builder.git (*)
> >>>   Plus a git trigger to reload to config by the buildbot master.
> >>> - A user, group and homedir to store the state and password/key files
> >>>   /sourceware/home/builder/
> >>> - The initial members of the builder group which can update the git
> >>>   repo and key files. The people on CC?
> >>> - A mailinglist to coordinate builder@sourceware.org or we could
> >>>   simply use the overseers mailinglist.
> >>
> >> Can we deploy this as a buildbot master container with some customization?
> >>
> >> This would make the service easier to keep going during sourcware updates because
> >> dependencies would be decoupled.
> >>
> >> It would allow developers to test pre-production changes locally by using the same
> >> container.
> >
> > The most natural "container" for a pure python app that we'll be
> > maintaining through pip is a python virtualenv. I have scripts to
> > setup and update the virtualenv so it is easy to replicate and play
> > with independently. You can wrap those in a whole system image of
> > course. I'll make sure to document that process.
>
> Agreed, virtualenv is a good alterantive.
>
> >> This is how upstream patchwork changes are being developed, which DJ and I are
> >> looking at because we will need to update patchwork to v3.1 upstream when it's
> >> released with DJ's new changes.
> >
> > Is there documentation on how sourceware patchwork is currently
> > deployed? Are you actually running a patches upstream version? I am
> > happy to help upgrading.
>
> Siddhesh has notes.
>
> We are running Patchwork 2.2 stable with just a few fixes related to our environment.
>
> Thanks for the offer. We're waiting for upstream v3.1 to be released with DJ's patches.
>
> > I haven't had to make any changes to upstream/packaged/pip buildbot
> > because all configuration is done through a python file (so if you do
> > really have to monkey patch the upstream version you can add those to
> > the config file).
>
> Right.
>
> >> My plan was to work with downstream distributions, ISVs, and IHVs who are interested
> >> in a particular project to donate workers. In order to do that though we have to make
> >> the act of installing and setting up a worker very low friction and very safe.
> >>
> >> The workers should arguably have a net positive impact for the downstream, or ISV, or
> >> IHV, so they need a way to measure that they prevented a patch that breaks their
> >> configuration of interest. So I also need to make sure they can gather metrics for
> >> reporting on the resources they are spending e.g. how soon after going red on their
> >> build was it corrected?
> >>
> >> For example, could Red Hat or IBM donate workers?
> >>
> >> Can we grow best practice CI adoption in our mature projects by making it easy?
> >
> > All good ideas and, if you have the matching project developers, then
> > adding more workers is in general a good thing.
> >
> > There are a few lessons about making CI most effective:
> >
> > - Start small with a minimal environment and a minimal subset of tests
> >   that just have to work.
> > - Always keep the builder green.
> > - If it is a flaky test leave it out.
> > - If there is a test that is failing consistently, but nobody fixes
> >   it, file a bug and take it out.
> > - Only expand if you have developers that care to keep that environment green.
> > - Only add workers if you have devxelopers on the project that can take
> >   care of the new build environments created for it.
> > - If you have a worker that is unique in some way (say for a
> >   non-common architecture) arrange for project maintainers to get
> >   access to it outside the buildbot so they can actually debug
> >   problems.
>
> Agreed.
>
> > For tracking actual success/failures/added/removed tests we could look
> > at feeding the test logs from buildbot to bunsen.
>
> Agreed. Though I sill want to point out that each individual may have different
> metrics for what means success and we need to ensure they can easily gather
> those metrics.
>
> --
> Cheers,
> Carlos.
>

Re: Moving buildbot master to sourceware

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

Hi,

On Mon, Apr 04, 2022 at 12:02:06PM +0200, Mark Wielaard via Overseers wrote:
> I would like to move this officially to sourceware now to make working
> on this together easier and to see if we can extend the service to
> other sourceware projects and see if we can get more workers. e.g. in
> the past Sergio ran a separate buildbot for gdb which we might merge
> together. One future integration idea would be to add buildbot
> triggers for patchwork to test out proposed patches before they are
> integrated.

This has finally been done:
https://builder.sourceware.org/buildbot/
With configs in https://sourceware.org/git/builder.git

All current builders and workers have been migrated and should work as
before. The SETUP file should explain how it was set up and how to
create a python virtual env to hack on it offline. But the
documentation (in README) on how to add new builders or workers is not
complete yet. There are a couple of TODO items (also attached) that we
can use to document the process.

I'll work with Tom to add new builders for the debian-ppc64 builder,
if the works out then I think we should deprecate the fedora-ppc64
worker if Dan agrees. Nick has volunteered to look into adding a
builder for binutils. And I'll try to work with the gdb hackers to see
which parts of the old gdb buildbot we can integrate.

Please let me know if there are other TODO items you would like to
see. And of course if you can help with any of them.

Cheers,

Mark

[-- Attachment #2: TODO --]
[-- Type: text/plain, Size: 1929 bytes --]

- Automate buildbot reconfig
Create a worker running on sourceware under the builder user with a builder
that listens to builder.git repo, copies files and reconfigs the master.

- More documentation on adding builders and workers.
Can use the debian-ppc64/fedora-ppc64 and binutils/gdb builders projects
as examples.

- Add more builders (bzip2 debugedit dwz elfutils libabigail valgrind) for
the debian-ppc64 workers. Then we can Deprecate the fedora-ppc64 worker,
somewhat slow, no working ccache and runs ancient/unsupported Fedora 22.

- Add a binutils builder (start small).
- Integrate parts of the old gdb buildbot
Old GDB buildbot documentation and config files
https://sourceware.org/gdb/wiki/BuildBot
git clone https://git.sergiodj.net/gdb-buildbot

- Add DockerLatentWorker
https://docs.buildbot.net/current/manual/configuration/workers-docker.html
A machine already has been setup runnin Debian libvirt VMs using
Fedora CoreOS that can run container images using Podman. It does need
a workaround for a small docker/podman incompatibility
https://github.com/containers/podman/issues/11668

- Use TLS and private cert files for worker pb connection
https://docs.buildbot.net/current/manual/installation/worker.html#worker-tls-config

- Use SecretInAFile instead of properties file
https://docs.buildbot.net/current/manual/secretsmanagement.html

- Prometheus Metrics:
https://pypi.org/project/buildbot-prometheus/

- Badges:
http://docs.buildbot.net/current/manual/configuration/www.html#the-badgeio-template

- Revlinks:
https://docs.buildbot.net/current/manual/configuration/global.html#revision-links

- Codebase generator:
https://docs.buildbot.net/current/manual/configuration/global.html#codebase-generator

- IRC reporter:
Might need one per project, be careful about authorization, don't want random
people to trigger or cancel builds
https://docs.buildbot.net/current/manual/configuration/reporters/irc.html

Re: Moving buildbot master to sourceware

[Mark Wielaard]

Hi,

On Sun, Apr 17, 2022 at 10:55:30PM +0200, Mark Wielaard via Overseers wrote:
> This has finally been done:
> https://builder.sourceware.org/buildbot/
> With configs in https://sourceware.org/git/builder.git
> 
> All current builders and workers have been migrated and should work as
> before. The SETUP file should explain how it was set up and how to
> create a python virtual env to hack on it offline. But the
> documentation (in README) on how to add new builders or workers is not
> complete yet. There are a couple of TODO items (also attached) that we
> can use to document the process.

I automated the updating of the master.cfg so that if someone commits
a change the buildbot picks it up (with its own worker) and reloads
its own config. This should make it possible for anybody in the
builder group to just commit updates to git without needing shell
access. The same didn't work for upgrading the virtual env that
buildbot runs in (I tried, but it deadlocked, the buildbot sees the
shutdown request, but waits for the worker to finish first, which is
waiting for the buildbot to shutdown before upgrading...)

At the moment it sents email to overseers for any such update, but
that might be overkill.

It is a nice example of how to add a new project with its own worker,
change source, scheduler, builder and mail reporters.

> I'll work with Tom to add new builders for the debian-ppc64 builder

This is next on my list. Tom already updated the worker to include all
build dependencies.

> if the works out then I think we should deprecate the fedora-ppc64
> worker if Dan agrees. Nick has volunteered to look into adding a
> builder for binutils. And I'll try to work with the gdb hackers to see
> which parts of the old gdb buildbot we can integrate.
> 
> Please let me know if there are other TODO items you would like to
> see. And of course if you can help with any of them.

Cheers,

Mark

> - Add more builders (bzip2 debugedit dwz elfutils libabigail valgrind) for
> the debian-ppc64 workers. Then we can Deprecate the fedora-ppc64 worker,
> somewhat slow, no working ccache and runs ancient/unsupported Fedora 22.
> 
> - Add a binutils builder (start small).
> - Integrate parts of the old gdb buildbot
> Old GDB buildbot documentation and config files
> https://sourceware.org/gdb/wiki/BuildBot
> git clone https://git.sergiodj.net/gdb-buildbot
> 
> - Add DockerLatentWorker
> https://docs.buildbot.net/current/manual/configuration/workers-docker.html
> A machine already has been setup runnin Debian libvirt VMs using
> Fedora CoreOS that can run container images using Podman. It does need
> a workaround for a small docker/podman incompatibility
> https://github.com/containers/podman/issues/11668
> 
> - Use TLS and private cert files for worker pb connection
> https://docs.buildbot.net/current/manual/installation/worker.html#worker-tls-config
> 
> - Use SecretInAFile instead of properties file
> https://docs.buildbot.net/current/manual/secretsmanagement.html
> 
> - Prometheus Metrics:
> https://pypi.org/project/buildbot-prometheus/
> 
> - Badges:
> http://docs.buildbot.net/current/manual/configuration/www.html#the-badgeio-template
> 
> - Revlinks:
> https://docs.buildbot.net/current/manual/configuration/global.html#revision-links
> 
> - Codebase generator:
> https://docs.buildbot.net/current/manual/configuration/global.html#codebase-generator
> 
> - IRC reporter:
> Might need one per project, be careful about authorization, don't want random
> people to trigger or cancel builds
> https://docs.buildbot.net/current/manual/configuration/reporters/irc.html

Re: Moving buildbot master to sourceware

[Mark Wielaard]

Hi,

Added Dodji to CC since libabigail has also moved. It hasn't seen any
new builds since the move, but I think everything is fine. There is
one extra builder now, libabigail-debian-ppc64. Please ping me if the
libabigail builder starts failing unexpectedly.

TLDR; Badges on https://builder.sourceware.org/

On Mon, Apr 18, 2022 at 11:56:01PM +0200, Mark Wielaard via Overseers wrote:
> > I'll work with Tom to add new builders for the debian-ppc64 builder
> 
> This is next on my list. Tom already updated the worker to include all
> build dependencies.

This has now been done and Tom is a member of the builder group so he
can help maintain the buildbot.

> > - Add a binutils builder (start small).

Nick and I are working on that right now. We are looking into a
unexpected FAIL. Which seems exactly what the buildbot was supposed to
find early. So even though not integrated yet, it is already working
:)

> > - Badges:
> > http://docs.buildbot.net/current/manual/configuration/www.html#the-badgeio-template

I added this, which was really just one line. But I took the
oppertunity to document how to do local hacking in a python virtual
env. See the README (not included yet is a description on how to
actually add new projects, change-sources, workers, builders and
reports - that will be added when we add binutils)
https://sourceware.org/git/?p=builder.git;a=blob_plain;f=README;hb=HEAD

It works pretty nicely IMHO. And except for adding plugins or
upgrading the buildbot packages you don't need shell access as builder
on the sourceware server now. If you just hack on the master.cfg you
can fully test it out locally, commit and push, and the buildbot will
automatically reconfig itself.

So do not do what I did and hack on the server itself. The automatic
reconfig will fail in that case and the buildbot will sent an angry
email to overseers...

You can find example badges on https://builder.sourceware.org/ now.

Cheers,

Mark

Re: Moving buildbot master to sourceware

[Mark Wielaard]

On Mon, Apr 04, 2022 at 12:02:06PM +0200, Mark Wielaard via Overseers wrote:
> I am also in the process of setting up a Ryzen9 machine with 3 extra
> VMs using Fedora CoreOS that can run container images using Podman. It
> does need a workaround for a small docker/podman incompatibility
> https://github.com/containers/podman/issues/11668 But once it works it
> could be a model of extending the workers in a simpler way, where the
> container images are defined together with the master to give project
> maintainers more flexibility in their testing environment.

This has finally been done. In the end it turned out that there were
just to many little issues/differences between podman and docker that
I had to just use docker to make it all work with the standard
DockerLatentWorker which uses docker.py. There are bug reports and
workarounds for all the issues I encountered, some should be fixed in
newer podman versions. So maybe in the future.

There is no documentation yet. Sorry.

But hopefully the commit makes clear how it works:
https://sourceware.org/git/?p=builder.git;a=commitdiff;h=b5aa4e8f3171b9494c4e649704545f21d8067252

Basically you define a container file under containers/ContainerFile-image-name
Then any builder that wants to use that container should add:

    properties={'container-file': readContainerFile('image-name')}

And use the new bb workers:

    workernames=["bb1", "bb2", "bb3"]

And that is it.

The tricky bit is embedding the buildbot-worker startup script in the image.
But there is an example for fedora and debian.

Cheers,

Mark

Re: Moving buildbot master to sourceware

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 245 bytes --]

On Tue, May 24, 2022 at 03:55:08AM +0200, Mark Wielaard via Overseers wrote:
> There is no documentation yet. Sorry.

Documentation on adding workers and container files attached (and
committed). Let me know if I missed something.

Cheers,

Mark

[-- Attachment #2: 0001-Add-documentation-on-adding-a-dedicated-or-container.patch --]
[-- Type: text/x-diff, Size: 5305 bytes --]

From cd13ea399e2b24d607fdcaa26b5405ee89b95468 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Wed, 25 May 2022 01:52:36 +0200
Subject: [PATCH] Add documentation on adding a dedicated or container worker

---
 README | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 SETUP  | 10 ++++++++
 2 files changed, 84 insertions(+), 1 deletion(-)

diff --git a/README b/README
index 72779d1..2574424 100644
--- a/README
+++ b/README
@@ -72,6 +72,19 @@ htdocs: document root of https://builder.sourceware.org/
        factory=project_factory)
   c['builders'].append(project__builder)
 
+  The above uses a dedicated distro-arch worker. If you want to use a
+  container image worker then you need to add a container-file
+  property to the builder and use one or more of the container image
+  workers (currently there are only 3 x86_64 container image workers
+  bb1, bb2 and bb3).
+
+        properties={'container-file':
+                    readContainerFile('fedora-latest')},
+        workernames=["bb1", "bb2", "bb3"],
+
+  Look under builder/containers/ContainerFile-* to the container image
+  names you can use.
+
   There are a couple of common steps that might be useful to
   reuse in you factory. A factory using common steps look something
   like:
@@ -95,7 +108,67 @@ htdocs: document root of https://builder.sourceware.org/
       <img src="/buildbot/badges/project-distro-arch.svg"></a></td>
 
 - How to add a new worker
-  ...
+
+  There are two kinds of workers. Dedicated distr-arch workers
+  (worker.Worker) and container image workers
+  (worker.DockerLatentWorker). The first has a whole distro installed
+  and runs a dedicated buildbot-worker which connects to
+  builder.sourceware.org to know what to build. The second runs a
+  (probably minimal) install with docker (or podman, but there are
+  still some incompatabilities) to which builder.sourceware.org
+  connects when there is a build providing a container file inside
+  which the build should be run (the container file will actually
+  contain a buildbot-worker which connects back to
+  builder.sourceware.org).
+
+  Both kind of workers need a little setup on the server. The
+  dedicated workers need a worker name and password, the container
+  image builders need a ssh setup (see SETUP).
+
+  The dedicated workers need the have all the developer packages
+  installed (look at one of the ContainerFiles to see which
+  ones). Since most builds will be almost the same with just a few
+  file changes, it is a good idea to have ccache installed and make
+  sure the ccache wrapper binaries are in the PATH before running the
+  buildbot-worker.
+
+  The container image builders should have a dedicated user (builder,
+  uid 1001) and a "shared" directory in which the container images can
+  put shared data (worker status, git checkouts, ccache). They should
+  also periodically clean up the images to make sure they get
+  regenerated with new packages. Run docker system prune -a -f once a
+  week.
+
+- How to add a new ContainerFile
+
+  The container files are all under builder/containers/ and should be
+  named ContainerFile-image-name so a builder can set the container
+  file property with readContainerFile('image-name').
+
+  The ContainerFile has four parts. First the base image and package
+  setup. Second the builder user setup (needs to match the builder uid
+  1001 from the host). Third the installation of the buildbot-worker
+  startup wrapper script. Fourth launching the startup wrapper script.
+
+  The base image and packages should install all development packages
+  that are needed to build the projects.
+
+  The builder user setup needs to be the same uid 1001 as the host
+  builder user and have /home/builder as $HOME so the host shared
+  directory can be mapped.
+
+  The startup script installation is slightly tricky. It either needs
+  to create a new buildbot-worker setup or edit an existing worker
+  buildbot.tac file to set the new password (which changes on each
+  run). The current ContainerFiles use some bashisms to create the
+  script, so make sure the image actually uses bash as /bin/sh (see
+  the debian containers for an example how to make sure it really is
+  bash). When using the existing setup script you must change the
+  image_name=... and maybe the export PATH=/usr/lib/ccache:$PATH line
+  (if the ccache wrappers are installed somewhere else).
+
+  Finally the script (that launches buildbot-worker) should be run as
+  the builder user.
 
 = Local hacking
 
diff --git a/SETUP b/SETUP
index 972e725..ed82db3 100644
--- a/SETUP
+++ b/SETUP
@@ -136,3 +136,13 @@ WantedBy=multi-user.target
 
 systemctl enable buildbot-worker
 systemctl start buildbot-worker
+
+For each dedicated distro-arch worker make sure there is a name and
+password in ~/buildbot.config.
+
+For the DockerLatentWorkers make sure that there is a private/public
+key pair under /sourceware/builder/.ssh (ssh-keygen) with the public
+part going into the docker_host builder user .ssh/authorized_keys. So
+the DockerLatentWorker can run docker system dial-stdio on the
+docker_host. Since the buildbot will use the ~/.ssh config make sure
+that the hosts are in .known_hosts (by logging in by hand once).
\ No newline at end of file
-- 
2.30.2