From: Mark Wielaard <mark@klomp.org>
To: binutils@sourceware.org, overseers@sourceware.org
Subject: Adding binutils to the GNU Toolchain buildbot on sourceware
Date: Mon, 25 Apr 2022 11:04:41 +0200 [thread overview]
Message-ID: <YmZkKRO+yUHeFqV0@wildebeest.org> (raw)
[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]
Hi,
I was talking to Nick what the best way was to add binutils to the
buildbot at https://builder.sourceware.org/. We came up with the
following which tests a minimal setup on a somewhat older but stable
distro:
- Debian 10, x86_64 which has Linux 4.19.235, GLIBC 2.28, gcc 8.3.0
GNU and Binutils 2.31.1
- Only triggers a build when any files under bfd/ binutils/ gas/ ld/
gold/ elfcpp/ include/ libiberty/ opcodes/ or the top-level
configure or Makefile.in changes.
- configure --enable-gold --enable-shared --enable-target=all
- make all-gas all-ld all-binutils all-gold
- make check-ld check-gas check-binutils
- log files ld/ld.{log,sum} gas/testsuite/gas.{log,sum}
binutils/binutils.{log,sum}
- Sents email to this list and any committers when the build changes
from sucess to failure (or back).
All of this can of course be changed/extended.
There are other distros/arches available:
https://builder.sourceware.org/buildbot/#/workers
And we hope to add container based workers to support even more
distros.
We only build a smaller set of tools to make sure you get a
notification within 10 minutes if anything was broken (or fixed). We
do build gold, but don't test it because the testsuite isn't
zero-fail.
If you want Frank also has scripts to add the log files to bunsen.
The intention was that this was a zero-fail testsuite subset
(gas,ld,binutils) but we did find two ld test failures:
FAIL: Run p_align-1b with PIE
FAIL: Run p_align-1d with -Wl,-z,max-page-size=0x1000 with PIE
These tests seem to rely on a specific glibc version/bug fix. Nick is
working on a fix.
Please let me know if any if this should be changed. The intention is
that the buildbot will help with maintaining binutils, keep it
building and zero-fail. If it isn't doing that, but get annoying let
change the configuration.
Cheers,
Mark
[-- Attachment #2: 0001-Add-binutils-as-project-and-documentation-on-adding-.patch --]
[-- Type: text/x-diff, Size: 10863 bytes --]
From 7e67984ba110dad6bafea0b22a56584494f11c56 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 25 Apr 2022 11:02:30 +0200
Subject: [PATCH] Add binutils as project and documentation on adding a new
project
---
README | 90 +++++++++++++++++++++++++++++++++++++++++-----
TODO | 7 ++--
builder/master.cfg | 90 ++++++++++++++++++++++++++++++++++++++++++++++
htdocs/index.html | 5 +++
4 files changed, 180 insertions(+), 12 deletions(-)
diff --git a/README b/README
index 3dda795..3afc8f2 100644
--- a/README
+++ b/README
@@ -8,13 +8,87 @@ builder/master.cfg: buildbot configuration
htdocs: document root of https://builder.sourceware.org/
- How to add a new project
- ... Add a change_source
- ... Add a scheduler
- ... Add a builder, see below
- ... Add a mail notifier
-
-- How to add a new builder
- ...
+ - Define a repourl to be used in the GitPoller and any Build Steps
+ project_repourl = 'git://sourceware.org/git/project.git'
+
+ - Define a change source, e.g. a GitPoller and add it
+ project_gitpoller = changes.GitPoller(repourl=project_repourl,
+ branches=['main'],
+ pollInterval=227,
+ project='project')
+ c['change_source'].append(project_gitpoller)
+
+ You can make it listen to multiple branches, use a pollInterval
+ not used by any other poller. Note that one repourl can only be
+ used once, if projects share a git repo the schedulers for the
+ project can select which files in the repo belong to which project.
+ See for example the gccrs or binutils-gdb schedulers.
+
+ - Add a scheduler
+ project_scheduler = schedulers.SingleBranchScheduler(
+ name="project",
+ change_filter=util.ChangeFilter(project="project",
+ branch="master"),
+ builderNames=["project-distro-arch", ...])
+ c['schedulers'].append(project_scheduler)
+
+ You might want to add a isImportantFile filter if not
+ all file changes must trigger a build (see gccrust or binutils
+ for examples).
+
+ - Add a builder, factory and build steps see below...
+
+ - Add a mail notifier
+ In general it is a good idea to only sent email for all
+ the builders from a scheduler combined based on the scheduler
+ project tag and only for changes (from good to bad or bad to good):
+
+ generator_project = reporters.BuildSetStatusGenerator(
+ mode=('change',), tags=['project'])
+ mn_project = reporters.MailNotifier(
+ fromaddr="builder@sourceware.org",
+ sendToInterestedUsers=True, # The patch committers.
+ extraRecipients=['project-list@sourceware.org'],
+ generators=[generator_project])
+ c['services'].append(mn_project)
+
+- How to add a new builder/fectory/steps
+ Asssuming there is already a project defined as above with a
+ repourl, change-source, scheduler and (mail) notifiers
+
+ The scheduler lists one or more builder names, each builder
+ defines a factory that generates the build steps, one or more
+ workers to execute the steps and defines tags which can be used
+ by reporters.
+
+ project_distro_arch_builder = util.BuilderConfig(
+ name="binutils-debian-amd64",
+ workernames=["distro-arch"],
+ tags=["project", "distro", "arch"],
+ factory=project_factory)
+ c['builders'].append(project__builder)
+
+ There are a couple of common steps that might be useful to
+ reuse in you factory. A factory using common steps look something
+ like:
+
+ project_factory = util.BuildFactory()
+ project_factory.addStep(steps.Git(
+ repourl=project_repourl,
+ mode='full', method='fresh',
+ name="git checkout",
+ haltOnFailure=True))
+ project_factory.addStep(autoreconf_step)
+ project_factory.addStep(configure_step)
+ project_factory.addStep(make_step)
+ project_factory.addStep(make_check_test_suite_step)
+
+ But often you need to tweak the steps a little.
+
+ Finally add the new builder to htdocs/index.html under the right project
+
+ <td><a href="/buildbot/#builders/project-distro-arch">distro-arch<br>
+ <img src="/buildbot/badges/project-distro-arch.svg"></a></td>
- How to add a new worker
...
@@ -118,4 +192,4 @@ Commit the changes. NOTE for actually upgrading the virtual-env on
sourceware you will need a shell account and run
/sourceware/builder/bin/buildbot-upgrade.sh as user builder. This
will shutdown the builder, do the above updates based on the
-buildbot.requirements file, upgrade-master and restart the builder.
\ No newline at end of file
+buildbot.requirements file, upgrade-master and restart the builder.
diff --git a/TODO b/TODO
index 495ac99..fe82827 100644
--- a/TODO
+++ b/TODO
@@ -1,11 +1,10 @@
-- More documentation on adding builders and workers.
- Can use the debian-ppc64/fedora-ppc64 and binutils/gdb builders projects
- as examples.
+- More documentation on adding workers.
+ Can use the debian-ppc64/fedora-ppc64 as examples.
- We can deprecate the fedora-ppc64 worker if the debian-ppc64 works out.
fedora-ppc64 is slow, no working ccache and runs unsupported Fedora 22.
-- Add a binutils builder (start small).
+- Generate the "dashboard" in htdocs/index.html automatically.
- Integrate parts of the old gdb buildbot
Old GDB buildbot documentation and config files
diff --git a/builder/master.cfg b/builder/master.cfg
index 5e8b544..e66e60a 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -201,6 +201,13 @@ gccrust_gitpoller = changes.GitPoller(repourl=gccrust_repourl,
project='gccrust')
c['change_source'].append(gccrust_gitpoller)
+binutils_gdb_repourl='git://sourceware.org/git/binutils-gdb.git'
+binutils_gdb_gitpoller = changes.GitPoller(repourl=binutils_gdb_repourl,
+ branches=['main'],
+ pollInterval=227,
+ project='binutils-gdb')
+c['change_source'].append(binutils_gdb_gitpoller)
+
####### SCHEDULERS
# Configure the Schedulers, which decide how to react to incoming changes.
@@ -338,6 +345,29 @@ gccrust_scheduler = schedulers.SingleBranchScheduler(
"gccrust-debian-ppc64"])
c['schedulers'].append(gccrust_scheduler)
+# Only trigger scheduler for changes to binutils (or deps)
+binutils_files = ["bfd/",
+ "binutils/", "gas/", "ld/",
+ "gold/", "elfcpp/",
+ "include/", "libiberty/", "opcodes/",
+ "configure", "Makefile.in"]
+
+def binutilsImportant(change):
+ for file in change.files:
+ for pattern in binutils_files:
+ match = re.match(pattern, file)
+ if match:
+ return True
+ return False
+
+binutils_scheduler = schedulers.SingleBranchScheduler(
+ name="binutils",
+ change_filter=util.ChangeFilter(project="binutils",
+ branch="master"),
+ fileIsImportant=binutilsImportant,
+ builderNames=["binutils-debian-amd64"])
+c['schedulers'].append(binutils_scheduler)
+
####### BUILDERS
# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
@@ -1021,6 +1051,56 @@ gccrust_debian_ppc64_builder = util.BuilderConfig(
factory=gccrust_factory)
c['builders'].append(gccrust_debian_ppc64_builder)
+# binutils build steps, factory and builders
+
+binutils_factory = util.BuildFactory()
+binutils_factory.addStep(steps.Git(
+ workdir='binutils-gdb',
+ repourl=binutils_gdb_repourl,
+ mode='full', method='fresh',
+ name="git checkout",
+ haltOnFailure=True))
+binutils_factory.addStep(steps.ShellCommand(
+ command=["rm", "-rf",
+ util.Interpolate ("%(prop:builddir)s/binutils-build")],
+ name="rm -rf binutils-build",
+ haltOnFailure=True))
+binutils_factory.addStep(steps.Configure(
+ workdir='binutils-build',
+ command=['../binutils-gdb/configure',
+ '--enable-gold',
+ '--enable-shared',
+ '--enable-target=all'],
+ name='configure',
+ haltOnFailure=True))
+binutils_factory.addStep(steps.Compile(
+ workdir='binutils-build',
+ command=['make',
+ util.Interpolate('-j%(prop:ncpus)s'),
+ 'all-gas', 'all-ld', 'all-binutils', 'all-gold'],
+ name='make',
+ haltOnFailure=True))
+binutils_factory.addStep(steps.Compile(
+ workdir='binutils-build',
+ command=['make',
+ util.Interpolate('-j%(prop:ncpus)s'),
+ 'check-ld', 'check-gas', 'check-binutils'],
+ name='make check',
+ logfiles={ "ld.sum": "ld/ld.sum",
+ "ld.log": "ld/ld.log",
+ "gas.sum": "gas/testsuite/gas.sum",
+ "gas.log": "gas/testsuite/gas.log",
+ "binutils.sum": "binutils/binutils.sum",
+ "binutils.log": "binutils/binutils.log" },
+ haltOnFailure=True))
+
+binutils_debian_amd64_builder = util.BuilderConfig(
+ name="binutils-debian-amd64",
+ workernames=["debian-amd64"],
+ tags=["binutils", "debian", "x86_64"],
+ factory=binutils_factory)
+c['builders'].append(binutils_debian_amd64_builder)
+
# libabigail build steps, factory and builders
libabigail_package_name_step = steps.SetProperty(
@@ -1309,6 +1389,16 @@ mn_gccrust = reporters.MailNotifier(
generators=[generator_gccrust])
c['services'].append(mn_gccrust)
+# Report for the whole binutils tagged builder set
+generator_binutils = reporters.BuildSetStatusGenerator(
+ mode=('change',), tags=['binutils'])
+mn_binutils = reporters.MailNotifier(
+ fromaddr="builder@sourceware.org",
+ sendToInterestedUsers=True,
+ extraRecipients=['binutils@sourceware.org'],
+ generators=[generator_binutils])
+c['services'].append(mn_binutils)
+
# Report for the whole libabigail tagged builder set
generator_libabigail = reporters.BuildSetStatusGenerator(
mode=('change',), tags=['libabigail'])
diff --git a/htdocs/index.html b/htdocs/index.html
index 9a2d64e..9d12fb9 100644
--- a/htdocs/index.html
+++ b/htdocs/index.html
@@ -30,6 +30,11 @@
<h2>Projects</h2>
<p>
<table>
+ <tr>
+ <td><a href="/buildbot/#/builders?tags=binutils">binutils</a></td>
+ <td><a href="/buildbot/#builders/binutils-debian-amd64">debian-amd64<br>
+ <img src="/buildbot/badges/binutils-debian-amd64.svg"></a></td>
+ </tr>
<tr>
<td><a href="/buildbot/#/builders?tags=bzip2">bzip2</a></td>
<td><a href="/buildbot/#builders/bzip2-centos-x86_64">centos-x86_64<br>
--
2.30.2
next reply other threads:[~2022-04-25 9:04 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-25 9:04 Mark Wielaard [this message]
2022-04-25 10:37 ` Luis Machado
2022-04-25 10:43 ` Frank Ch. Eigler
2022-04-25 12:16 ` Luis Machado
2022-04-25 12:30 ` Frank Ch. Eigler
2022-04-25 18:20 ` Mark Wielaard
2022-04-25 18:27 ` Frank Ch. Eigler
2022-04-25 22:11 ` Mark Wielaard
2022-04-26 3:33 ` Alan Modra
2022-04-26 6:22 ` Jan Beulich
2022-04-26 12:27 ` Nick Clifton
2022-04-26 13:49 ` Jan Beulich
2022-04-26 15:47 ` H.J. Lu
2022-04-27 6:15 ` Jan Beulich
2022-04-28 12:10 ` Nick Clifton
2022-04-28 13:07 ` Jan Beulich
2022-04-26 15:54 ` H.J. Lu
2022-04-26 23:33 ` Alan Modra
2022-04-27 18:32 ` [PATCH] x86: Disable 2 tests with large memory requirement H.J. Lu
2022-04-26 7:01 ` Adding binutils to the GNU Toolchain buildbot on sourceware Luis Machado
2022-04-26 9:40 ` Frank Ch. Eigler
2022-04-26 22:59 ` Mark Wielaard
2022-04-26 22:34 ` Mark Wielaard
2022-04-28 12:23 ` Luis Machado
2022-04-28 13:50 ` Frank Ch. Eigler
2022-04-28 13:53 ` Luis Machado
2022-04-28 14:22 ` Frank Ch. Eigler
2022-04-28 17:04 ` Mark Wielaard
2022-04-28 14:48 ` Mark Wielaard
2022-04-28 14:19 ` Mark Wielaard
2022-04-28 14:47 ` Thomas Fitzsimmons
2022-04-28 16:28 ` Mark Wielaard
2022-05-01 16:37 ` binutils builder status (Was: Adding binutils to the GNU Toolchain buildbot on sourceware) Mark Wielaard
2022-05-05 7:53 ` Jan Beulich
2022-05-07 20:25 ` Mark Wielaard
2022-05-16 9:40 ` Mark Wielaard
2022-04-28 17:50 ` Adding binutils to the GNU Toolchain buildbot on sourceware Nick Alcock
2022-04-29 17:54 ` Mark Wielaard
2022-04-30 0:12 ` Nick Alcock
2022-04-30 22:27 ` Mark Wielaard
2022-05-03 12:48 ` Nick Alcock
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YmZkKRO+yUHeFqV0@wildebeest.org \
--to=mark@klomp.org \
--cc=binutils@sourceware.org \
--cc=overseers@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).