From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id CE3EC3858D37 for ; Mon, 4 Apr 2022 22:15:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org CE3EC3858D37 Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-554-oK6fUjxrMZCcVa3ihuIb0Q-1; Mon, 04 Apr 2022 18:15:04 -0400 X-MC-Unique: oK6fUjxrMZCcVa3ihuIb0Q-1 Received: by mail-qk1-f197.google.com with SMTP id c19-20020a05620a0cf300b005f17891c015so7126266qkj.18 for ; Mon, 04 Apr 2022 15:15:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:organization:in-reply-to :content-transfer-encoding; bh=Vlhsu9DhmIct1FrG+CoQsIhwVHFIpTKp7n//HGpPTbI=; b=BiesQn/GnjFMNOTh8+JiyrPlrApvR7r3SwEfQZNeYykbaLdoATWgxvwQ+yKWRfmA4G Ums6mAYD3TgBxflBUcl7kk43ZPtkHpNtsyUJ1SZ1joPmGGqdKZS9mOM69L9BuKL58pyf NQUvB2e0r64PxtdLGlqcJZctg9eZLV1WEFrUZP9dFwB5R7GlFF/zWoki2Aj496rdS0cv fEMALu+3/fUhkJQOg5yISy3lWkpPCns83WJdKx7GE1gv79ncJcGqZiBsdUZ1N9PBLbBc tcGSSLlDNCIqdIgc4PPiakt86rz9Z7rUUjbLey6pHEuwYe26hdaqjItChzsqenwXSvsX Rt5g== X-Gm-Message-State: AOAM531c8NsRk2bH2mLkzpWlDadm6z0Qp8dps1wP4oZN7VUErCe/UMeF eeoQzRA8/1r404IHErb1S+kyW96SeO1r4A5WanNa5wv4Lff3fergQJQVFic37jRBSGoThzgZiSS pw1U7hx2862YnLJ1iSySrzdAAmSjPNSt4fTOzDrRPB1MLMkd+1IxMQLOrM3gsMd+LGST8 X-Received: by 2002:a05:6214:519b:b0:441:2307:91d5 with SMTP id kl27-20020a056214519b00b00441230791d5mr320418qvb.81.1649110503867; Mon, 04 Apr 2022 15:15:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwbSDkqQlvOeC5z2B9eg5TdjU9GQKAGriPvv4PaPKNXo7Si++ZUeDxD8NSF1YCi69ePbK/YkA== X-Received: by 2002:a05:6214:519b:b0:441:2307:91d5 with SMTP id kl27-20020a056214519b00b00441230791d5mr320393qvb.81.1649110503521; Mon, 04 Apr 2022 15:15:03 -0700 (PDT) Received: from [192.168.0.241] (135-23-175-80.cpe.pppoe.ca. [135.23.175.80]) by smtp.gmail.com with ESMTPSA id f8-20020ac859c8000000b002e232e04cafsm9776037qtf.88.2022.04.04.15.15.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 04 Apr 2022 15:15:03 -0700 (PDT) Message-ID: Date: Mon, 4 Apr 2022 18:15:01 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: Moving buildbot master to sourceware To: Overseers mailing list Cc: Mark Wielaard , Philip Herron , Sergio Durigan Junior , Julian Seward , Thomas Fitzsimmons , =?UTF-8?Q?Dan_Hor=c3=a1k?= References: From: Carlos O'Donell Organization: Red Hat In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-6.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: overseers@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Overseers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Apr 2022 22:15:07 -0000 On 4/4/22 06:02, Mark Wielaard via Overseers wrote: > I would like to move this officially to sourceware now to make working > on this together easier and to see if we can extend the service to > other sourceware projects and see if we can get more workers. e.g. in > the past Sergio ran a separate buildbot for gdb which we might merge > together. One future integration idea would be to add buildbot > triggers for patchwork to test out proposed patches before they are > integrated. Thank you for volunteering to do this! A central buildbot would be nice. > Maintenance of the master is fairly minimal and it doesn't require > that many resources. The current state database, git poller and build > logs total less than 3.5G for hundreds to thousands of builds for all > of the above projects. > > The bike shed items for installing this on sourceware are: > > - Whether to install buildbot from pip or epel. My preference is > throug pip under a user account since that is how I run it > locally. But using the epel package is fine too. > - Running it under sourceware.org/buildbot or adding a vhost > builder.sourceware.org. My preference is for a vhost. > - A git repo for the master.cfg file builder.git (*) > Plus a git trigger to reload to config by the buildbot master. > - A user, group and homedir to store the state and password/key files > /sourceware/home/builder/ > - The initial members of the builder group which can update the git > repo and key files. The people on CC? > - A mailinglist to coordinate builder@sourceware.org or we could > simply use the overseers mailinglist. Can we deploy this as a buildbot master container with some customization? This would make the service easier to keep going during sourcware updates because dependencies would be decoupled. It would allow developers to test pre-production changes locally by using the same container. This is how upstream patchwork changes are being developed, which DJ and I are looking at because we will need to update patchwork to v3.1 upstream when it's released with DJ's new changes. Upstream patchwork can be deployed in a container. We should switch to container deployments for the current instance. Reference: http://docs.buildbot.net/current/tutorial/docker.html?highlight=docker https://github.com/getpatchwork/patchwork/blob/master/docker-compose-pg.yml > (*) The master.cfg file is actually a python script, and I am slightly > embarrased by it. It would be great if someone with a bit more python > experience could refactor it so that the Builder setups aren't just > done by copy/paste tens of identical lines. > > Once setup the real work is on the projects using the buildbot to > define and coordinating with the volunteers running the buildbot > workers in case there is some issue that cannot be explained by the > buildbot test run. This can in general be done directly between the > project maintainers and the buildbot worker volunteers, who can also > provide direct access to their setups. But it would be good to have a > central list to coordinate such issues. > > There are currently the following 10 workers: > - centos-x86_64, debian-amd64, debian-i386, fedora-x86_64 > Ran by me as kvm libvirt instances. > - debian-ppc64 ran by Thomas Fitzsimmons > - fedora-ppc64 and fedora-ppc64le ran by Dan Horák > thanks to the Brno University > - fedora-s390x Ran by Dan Horák > thanks to the Marist University > - debian-arm64 and debian-armhf small boards ran by me. > > I am also in the process of setting up a Ryzen9 machine with 3 extra > VMs using Fedora CoreOS that can run container images using Podman. It > does need a workaround for a small docker/podman incompatibility > https://github.com/containers/podman/issues/11668 But once it works it > could be a model of extending the workers in a simpler way, where the > container images are defined together with the master to give project > maintainers more flexibility in their testing environment. This is similar to how DJ's CI/CD runners work. We have a VM, that runs the trybot runner which runs tests in a disconnected container. The goal is to eventually ship an Ansible playbook to setup the VM with some templated defaults for "runner" volunteers to use for their testing. > Let me know what you think and whether you have ideas for getting more > worker machines. My plan was to work with downstream distributions, ISVs, and IHVs who are interested in a particular project to donate workers. In order to do that though we have to make the act of installing and setting up a worker very low friction and very safe. The workers should arguably have a net positive impact for the downstream, or ISV, or IHV, so they need a way to measure that they prevented a patch that breaks their configuration of interest. So I also need to make sure they can gather metrics for reporting on the resources they are spending e.g. how soon after going red on their build was it corrected? For example, could Red Hat or IBM donate workers? Can we grow best practice CI adoption in our mature projects by making it easy? A example of this is the Kernel CI project at the Linux Foundation: https://linux.kernelci.org/job/ https://foundation.kernelci.org/ -- Cheers, Carlos.