From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id 3B56F3858D33; Sun, 9 Apr 2023 11:30:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3B56F3858D33 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: by gnu.wildebeest.org (Postfix, from userid 1000) id 3FAE9302BB02; Sun, 9 Apr 2023 13:30:50 +0200 (CEST) Date: Sun, 9 Apr 2023 13:30:50 +0200 From: Mark Wielaard To: buildbot@sourceware.org, overseers@sourceware.org Cc: Arsen =?utf-8?Q?Arsenovi=C4=87?= , Gerald Pfeifer , Nick Clifton , Dodji Seketeli , "Jose E. Marchesi" Subject: Using builder for (documentation) snapshots Message-ID: <20230409113050.GA6496@gnu.wildebeest.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-3030.6 required=5.0 tests=BAYES_00,JMQ_SPF_NEUTRAL,KAM_DMARC_STATUS,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi, Thanks to OSUOSL we now have an extra buildbot worker with extra storage so we can do container builds and publish (documentation) snapshots. https://snapshots.sourceware.org/ It acts like a normal buildbot (x86_64) container worker, except that whatever is left in the output directory will be available through the snapshots server. Some examples of where this is useful: - elfutils test coverage reports. Currently they are only produced (by hand) once an official release is made. Ideally we have coverage reports of current git. - snapshots of binutils or gnupoke which are either currently done by hand (and so need a script to be written to automate) or which have a non-trivial boostrap script. - generating the gcc documentation using the lowest and highest supported version of texinfo to make sure it works and looks as intended. - Generating the documentation/website of libabigail which is currently done by hand and needs a specific setup to generate from the source code with sphynx. - valgrind the html and pdf manuals are currently only generated by hand during a release. - dwarfstd.org is automatically generated through a git hook from the main branch, but people have wanted to show how things look from a specific branch. There is an example of how to do the elfutils coverage report in builbot.git: https://sourceware.org/cgit/builder/commit/?id=365a64a19b9a4b3cd677ec377dd1574e4dcdf022 This also contains the setup instructions of the snapshots server plus scripts used. The output can be seen here: https://snapshots.sourceware.org/elfutils/coverage/ I'll help setup the other examples. Retention --------- The server has 600GB free storage, so even if we really create a new snapshot every 15 minutes we have some way to go. We could install a script that scrubs snapshots after a year, or that only keeps one snapshot a day after a month. A snapshot can be recreated from some git checkout and aren't intended to be official releases so we can erase them whenever we want. We just need to write a script to do it. Write back ---------- Although the original idea was that the "latest" snapshot could be written back to the sourceware.org webserver dir and/or htdocs git repo I didn't add such functionality. It is probably better to see how useful these documentation/web snapshots are before replacing existing update machanisms. Cheers, Mark