Re: binutils builder status (Was: Adding binutils to the GNU Toolchain buildbot on sourceware)

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 4729 bytes --]

Hi Jan,

On Thu, May 05, 2022 at 09:53:25AM +0200, Jan Beulich wrote:
> On 01.05.2022 18:37, Mark Wielaard wrote:
> > The gas and binutils testsuites seem clean on all builders. But the ld
> > testsuite does see some unexpected failures or passes on some
> > builders. It would be great if we could fix these. If not it might
> > make sense to run the ld testsuite separately.
> > 
> > Note that you can see the used linux kernel, gcc, binutils versions on
> > the workers page:
> > https://builder.sourceware.org/buildbot/#/workers
> > 
> > binutils-fedora-s390x
> > https://builder.sourceware.org/buildbot/#/builders/binutils-fedora-s390x
> > has 2 unexpected ld failures:
> > FAIL: Run pr19719 fun undefined
> > FAIL: pr26580-3
> > 
> > binutils-debian-amd64
> > https://builder.sourceware.org/buildbot/#/builders/binutils-debian-amd64
> > has 2 unexpected ld failures:
> > FAIL: Run p_align-1b with PIE
> > FAIL: Run p_align-1d with -Wl,-z,max-page-size=0x1000 with PIE
> > 
> > binutils-debian-arm64
> > https://builder.sourceware.org/buildbot/#/builders/binutils-debian-arm64
> > has 1 unexpected success:
> > XPASS: Run pr19719 fun undefined
> > 
> > binutils-debian-armhf
> > https://builder.sourceware.org/buildbot/#/builders/binutils-debian-armhf
> > has 8 unexpected ld failures and 7 unexpected successes
> > XPASS: Run pr19719 fun undefined
> > FAIL: Common symbol override ifunc test 1a
> > FAIL: Common symbol override ifunc test 1b
> > FAIL: Run pr18841 with libpr18841b.so
> > FAIL: Run pr18841 with libpr18841c.so
> > FAIL: Run pr18841 with libpr18841bn.so (-z now)
> > FAIL: Run pr18841 with libpr18841cn.so (-z now)
> > FAIL: Run pr23169a
> > FAIL: Run pr23169d
> > XPASS: visibility (hidden_undef) (non PIC)
> > XPASS: visibility (hidden_undef) (non PIC, load offset)
> > XPASS: visibility (hidden_undef) (PIC main, non PIC so)
> > XPASS: visibility (protected_undef) (non PIC)
> > XPASS: visibility (protected_undef) (non PIC, load offset)
> > XPASS: visibility (protected_undef) (PIC main, non PIC so)
> > 
> > The binutils-debian-armhf builder is also the slowest (takes 15
> > minutes). The rest take a few minutes. They should sent email once a
> > new failure occurs (or if one if the currently failing builders starts
> > passing).
> > 
> > There is also one build and check everything builder
> > https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-fedrawhide-x86_64
> > I haven't looked at the test results yet, but they are all stored in
> > the bunsendb.git for later analysis. This builder doesn't sent emails
> > on bad builds. It also takes a very long time to run (from 1 to 7 hours).
> 
> I've received the first email from that build, as it looks. May I please
> ask that this be Cc-ed to individuals (rather than putting them on To: )?
> I view To: as reasonable only if there was breakage which was possible to
> associate with a particular commit at quite high a probability (which
> would require either bisection of there being merely a single commit on
> top of what was tested in the prior run).
> 
> To: otoh quite likely wants to include binutils@ ?

I think the email you received was for the "recovered"
gdb-fedora-s390x builder. This builder failed from the start since the
fix on Friday:
https://builder.sourceware.org/buildbot/#builders/gdb-fedora-s390x

The buildbot assumed everybody who had done a commit since the first
fail should hear the happy news that the build now passes. Although it
is good news, this was probably too many people.

I changed the reporters so only people who committed a change that
caused a build to go from success to failure will get a problem report
email. Then I added a new reporter that just reports to the
mailinglist for any "changed" build (from success to failure or from
failure to success).

That way you should only be on the To: when you most likely caused the
problem. There is one small issue with binutils and gdb because they
share a source repository. You might be included in a report about gdb
or binutils even if you meant to only change one or the other
project. The buildbot knows which directories are part of which
project, but some overlap and it doesn't know that a change it didn't
explicitly tests when the build for one project fails couldn't have
caused that failure.

You can write more complicated reporters in python if you want:
https://docs.buildbot.net/current/manual/configuration/report_generators/
https://docs.buildbot.net/current/manual/configuration/reporters/mail_notifier.html

And as you can see from the attached patch I am probably not a great
python programmer. If someones wants to try to clean the master.cfg
file up a little to remove all the replications, please do!

Cheers,

Mark

[-- Attachment #2: 0001-Change-mail-reports-to-only-sent-problems-to-interes.patch --]
[-- Type: text/x-diff, Size: 10265 bytes --]

From c3de0dd3c59818819b9d09c7544ca2feb370770d Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 7 May 2022 22:02:49 +0200
Subject: [PATCH] Change mail reports to only sent problems to interested
 users.

Report for changes (failure to success and success to failure) go to
the mailinglist only. Do this by having two reporters. One sents email
only to interested users, the other only the the extra recipients (the
mailing list).
---
 builder/master.cfg | 145 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 118 insertions(+), 27 deletions(-)

diff --git a/builder/master.cfg b/builder/master.cfg
index c131974..988e584 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1584,6 +1584,7 @@ c['services'].append(mn)
 
 # Sent reports about sourceware buildbot updates to overseers
 # 2022-05-02 ... don't bother overseers@ anymore about this
+# Just sent a report to the interested user (committer).
 generator_sourceware = reporters.BuildStatusGenerator(tags=["sourceware"])
 mn_sourceware = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
@@ -1591,97 +1592,187 @@ mn_sourceware = reporters.MailNotifier(
         generators=[generator_sourceware])
 c['services'].append(mn_sourceware)
 
+# Only sent problem reports to overseers
+generator_sourceware_problem = reporters.BuildStatusGenerator(
+        mode=('problem',), tags=["sourceware"])
+mn_sourceware_problem = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['overseers@sourceware.org'],
+        generators=[generator_sourceware_problem])
+c['services'].append(mn_sourceware_problem)
 
-# Report for the whole bzip2 tagged builder set
+# Problem report for the whole bzip2 tagged builder set
 generator_bzip2 = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['bzip2'])
+        mode=('problem',), tags=['bzip2'])
 mn_bzip2 = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['bzip2-devel@sourceware.org'],
         generators=[generator_bzip2])
 c['services'].append(mn_bzip2)
 
-# Report for the whole debugedit tagged builder set
+# Change report for the whole bzip2 tagged builder set
+generator_bzip2_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['bzip2'])
+mn_bzip2_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['bzip2-devel@sourceware.org'],
+        generators=[generator_bzip2_change])
+c['services'].append(mn_bzip2_change)
+
+# Problem report for the whole debugedit tagged builder set
 generator_debugedit = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['debugedit'])
+        mode=('problem',), tags=['debugedit'])
 mn_debugedit = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['debugedit@sourceware.org'],
         generators=[generator_debugedit])
 c['services'].append(mn_debugedit)
 
-# Report for the whole dwz tagged builder set
+# Change report for the whole debugedit tagged builder set
+generator_debugedit_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['debugedit'])
+mn_debugedit_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['debugedit@sourceware.org'],
+        generators=[generator_debugedit_change])
+c['services'].append(mn_debugedit_change)
+
+# Problem report for the whole dwz tagged builder set
 generator_dwz = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['dwz'])
+        mode=('problem',), tags=['dwz'])
 mn_dwz = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['dwz@sourceware.org'],
         generators=[generator_dwz])
 c['services'].append(mn_dwz)
 
-# Report for the whole elfutils tagged builder set
+# Change report for the whole dwz tagged builder set
+generator_dwz_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['dwz'])
+mn_dwz_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['dwz@sourceware.org'],
+        generators=[generator_dwz_change])
+c['services'].append(mn_dwz_change)
+
+# Problem report for the whole elfutils tagged builder set
 generator_elfutils = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['elfutils'])
+        mode=('problem',), tags=['elfutils'])
 mn_elfutils = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['elfutils-devel@sourceware.org'],
         generators=[generator_elfutils])
 c['services'].append(mn_elfutils)
 
-# Report for the whole gccrust tagged builder set
+# Change report for the whole elfutils tagged builder set
+generator_elfutils_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['elfutils'])
+mn_elfutils_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['elfutils-devel@sourceware.org'],
+        generators=[generator_elfutils_change])
+c['services'].append(mn_elfutils_change)
+
+# Problem report for the whole gccrust tagged builder set
 generator_gccrust = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['gccrust'])
+        mode=('problem',), tags=['gccrust'])
 mn_gccrust = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['gcc-rust@gcc.gnu.org'],
         generators=[generator_gccrust])
 c['services'].append(mn_gccrust)
 
-# Report for the whole binutils tagged builder set
+# Change report for the whole gccrust tagged builder set
+generator_gccrust_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['gccrust'])
+mn_gccrust_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['gcc-rust@gcc.gnu.org'],
+        generators=[generator_gccrust_change])
+c['services'].append(mn_gccrust_change)
+
+# Problem report for the whole binutils tagged builder set
 generator_binutils = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['binutils'])
+        mode=('problem',), tags=['binutils'])
 mn_binutils = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['binutils@sourceware.org'],
         generators=[generator_binutils])
 c['services'].append(mn_binutils)
 
-# Report for the whole gdb tagged builder set
+# Change report for the whole binutils tagged builder set
+generator_binutils_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['binutils'])
+mn_binutils_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['binutils@sourceware.org'],
+        generators=[generator_binutils_change])
+c['services'].append(mn_binutils_change)
+
+# Problem report for the whole gdb tagged builder set
 generator_gdb = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['gdb'])
+        mode=('problem',), tags=['gdb'])
 mn_gdb = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['gdb-testers@sourceware.org'],
         generators=[generator_gdb])
 c['services'].append(mn_gdb)
 
-# Report for the whole libabigail tagged builder set
+# Change report for the whole gdb tagged builder set
+generator_gdb_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['gdb'])
+mn_gdb_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['gdb-testers@sourceware.org'],
+        generators=[generator_gdb_change])
+c['services'].append(mn_gdb_change)
+
+# Problem report for the whole libabigail tagged builder set
 generator_libabigail = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['libabigail'])
+        mode=('problem',), tags=['libabigail'])
 mn_libabigail = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['libabigail@sourceware.org'],
         generators=[generator_libabigail])
 c['services'].append(mn_libabigail)
 
-# Report for the whole valgrind tagged builder set
+# Changereport for the whole libabigail tagged builder set
+generator_libabigail_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['libabigail'])
+mn_libabigail_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['libabigail@sourceware.org'],
+        generators=[generator_libabigail_change])
+c['services'].append(mn_libabigail_change)
+
+# Problem report for the whole valgrind tagged builder set
 generator_valgrind = reporters.BuildSetStatusGenerator(
-        mode=('change',), tags=['valgrind'])
+        mode=('problem',), tags=['valgrind'])
 mn_valgrind = reporters.MailNotifier(
         fromaddr="builder@sourceware.org",
         sendToInterestedUsers=True,
-        extraRecipients=['valgrind-testresults@lists.sourceforge.net'],
         generators=[generator_valgrind])
 c['services'].append(mn_valgrind)
 
+# Change report for the whole valgrind tagged builder set
+generator_valgrind_change = reporters.BuildSetStatusGenerator(
+        mode=('change',), tags=['valgrind'])
+mn_valgrind_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=False,
+        extraRecipients=['valgrind-testresults@lists.sourceforge.net'],
+        generators=[generator_valgrind_change])
+c['services'].append(mn_valgrind_change)
+
 
 ####### PROJECT IDENTITY
 
-- 
2.30.2