From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id D9332385828E for ; Tue, 5 Jul 2022 10:37:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D9332385828E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from reform (deer0x0e.wildebeest.org [172.31.17.144]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 321C4302AB2C; Tue, 5 Jul 2022 12:37:28 +0200 (CEST) Received: by reform (Postfix, from userid 1000) id B2C872E832D3; Tue, 5 Jul 2022 12:37:28 +0200 (CEST) From: Mark Wielaard To: buildbot@sourceware.org Cc: Mark Wielaard Subject: [COMMITTED] Add README_workers Date: Tue, 5 Jul 2022 12:37:06 +0200 Message-Id: <20220705103705.12117-1-mark@klomp.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-11.2 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_ASCII_DIVIDERS, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: buildbot@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "The https://builder.sourceware.org/ buildbot" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2022 10:37:33 -0000 All the information to add a new buildbot worker. --- README_workers | 182 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 README_workers diff --git a/README_workers b/README_workers new file mode 100644 index 0000000..76ed9c3 --- /dev/null +++ b/README_workers @@ -0,0 +1,182 @@ +Adding a new buildbot-worker +============================ + +A worker is a (vitual) machine that does the actual builds. +builder.sourceware.org supports two kinds of workers. A worker in a +full distro setup with all build dependencies already insalled. And a +container latent worker which can create containers from a buildbot +provided file to run the build inside. + +The container latent worker is a bit more work to install and +requires ssh access to the host running the worker. + +The full distro worker is easier to setup, doesn't need to be +reachable as long as it can make a connection to +builder.sourceware.org, but requires all build dependency packages to +be install up front. + +Adding a (virtual) machine (distro) builder +------------------------------------------- + +Setup a normal distro install and make sure all build dependencies are +installed. A good list can be in the container files for various +distros that builder.sourceware.org uses (examples for debian, fedora +and opensuse): +https://sourceware.org/git/?p=builder.git;a=tree;f=builder/containers;hb=HEAD + +And it is a good idea to install ccache and put it in the worker user +PATH because most builds will compile the exact same sources. (See below) + +Install buildbot-worker either as distro package or through pip: +$ pip install buildbot-worker + +To create the worker you'll need a name and a password. Please +contact buildbot@sourceware.org to get a password. Also say what the +"normal" number of cpus/cores that builds can be used and what the max +number of cpus/cores is that a "heavy" build can use (these values can +be the same). Also say what the maximum number of concurrent builds +can be. + +e.g. +- ncpus: 4 +- maxcpus: 16 +- max_builds: 3 + +We'll add something like the following to the master.cfg file: + +name_worker = worker.Worker("name", + getpw("name"), + max_builds=3, + properties={'ncpus': 4, 'maxcpus': 16}, + notify_on_missing='your@example.org'); +c['workers'].append(name_worker) + +The password will be provided off-list and won't be checked into git. +It will have to be added to the /sourceware/builder/buildbot.conf file +by someone with builder permissions. And an extra entry will be added +to the buildbot.config.sample file. + +Then you can create the worker, do this under a user where the builds +will be running: + +- buildbot-worker create-worker workerdir builder.sourceware.org name passwd + +Create a startup script like: + += start-bb-worker.sh = +#!/bin/bash + +# ccache should be in front, sometimes in /usr/lib64 +export PATH=/usr/lib/ccache:$PATH + +# the worker dir, same as first argument to create-worker +cd ~/workerdir + +# Add info for buildbot users +echo "Worker admin contact " > info/admin + +# Distro name, glibc version, binutils version, gcc version +echo "Distro name" > info/host +iconv --version | head -1 >> info/host +objcopy --version | head -1 >> info/host +gcc --version | head -1 >> info/host + +# This will demonize after startup +buildbot-worker start + +And put it in a cronjob so it starts on system reboot: +@reboot rm -f ~/workerdir/twistd.pid; ~/start-bb-worker.sh + +Adding a container (latent) worker +---------------------------------- + +This is a bit more work, but has as advantage that you don't have to +keep track of updates or any missing packages. It also take a bit more +disk space since it creates (full distro) container images. + +The setup requires you to provide a builder user with uid 1001 and a +/home/builder/shared directory that keeps ccache, git checkouts and +builds with the right permissions. And allows builder to connect to +that user through ssh to setup a docker socket connection. Note that +podman doesn't work here since it has a couple of small, but +significant incompatibilities. e.g. +https://github.com/containers/podman/issues/11668 + +Easiest setup is using something like coreos with a butane/ignotion +file. The advantage of coreos is that it autoupdates, already comes +with docker/moby, uses selinux for isolation and that you can simply +reinstall (the images and shared directory will just be recreated by +the buildbot). This sample butane.yaml file is for a libvirt setup, +you will likely need to change the device:, /etc/hostname contents and +core ssh_authorized_keys. + +variant: fcos +version: 1.4.0 +storage: + disks: + - device: /dev/vda + wipe_table: false + partitions: + # 10GB root (already exists, resize) + - number: 4 + label: root + size_mib: 10240 + resize: true + # 8GB swap + - number: 5 + label: swap + size_mib: 8192 + # Everything else for /var (which includes everything writable) + - size_mib: 0 + label: var + filesystems: + - device: /dev/disk/by-partlabel/swap + format: swap + wipe_filesystem: true + with_mount_unit: true + - path: /var + device: /dev/disk/by-partlabel/var + # We can select the filesystem we'd like. + format: ext4 + wipe_filesystem: true + # Ask Butane to generate a mount unit for us so that this filesystem + # gets mounted in the real root. + with_mount_unit: true + files: + - path: /etc/hostname + mode: 0644 + contents: + inline: sourceware-builder1 +systemd: + units: + # Create and set selinux context for shared builder dir + - name: builder-shared-dir.service + enabled: true + contents: | + [Unit] + Description=Builder shared dir + Wants=network-online.target + After=network-online.target + Before=zincati.service + ConditionPathExists=!/var/lib/%N.stamp + + [Service] + Type=oneshot + RemainAfterExit=yes + ExecStart=/bin/mkdir -p /home/builder/shared + ExecStart=/bin/chown builder.builder /home/builder/shared + ExecStart=/bin/chcon -t container_file_t /home/builder/shared + ExecStart=/bin/touch /var/lib/%N.stamp + + [Install] + WantedBy=multi-user.target +passwd: + users: + - name: core + ssh_authorized_keys: + - YOUR_PUB_KEY + - name: builder + uid: 1001 + groups: [docker] + ssh_authorized_keys: + - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnB//uE66TX+nmLuox3ETfZ7dvjUl2mlwN4Q+107EWCuP05eKxcahyzwenmzODzzC8/hUVeUtuCJJyMW8+CpqCluHd2bEUT9WIzP1C/T22jBUbhY7zXLufSYfKmZe9zHNUizEbqbGAfR01D3jv3wis8JNOq/yopGLKJmAPd/Ye8XN5lsfAUTREIF+xygjxSb6SZPU9UvKlJmnwZoCN2Fd1u4IhSWBcD3dyZyirD80ZLGsSdOUmzX2Z2NaSI5tG6MxjYtQZbzmsG/Z0HY1Y/KiSolX989+7dMKMy2V911SkS6Cx9aVuSY+ig9Rq6GpfjGkBnHrNCCeVFe8PlYJl6DXZ -- 2.30.2