Hi Jan,

On Thu, May 05, 2022 at 09:53:25AM +0200, Jan Beulich wrote:
> On 01.05.2022 18:37, Mark Wielaard wrote:
> > The gas and binutils testsuites seem clean on all builders. But the ld
> > testsuite does see some unexpected failures or passes on some
> > builders. It would be great if we could fix these. If not it might
> > make sense to run the ld testsuite separately.
> > 
> > Note that you can see the used linux kernel, gcc, binutils versions on
> > the workers page:
> > https://builder.sourceware.org/buildbot/#/workers
> > 
> > binutils-fedora-s390x
> > https://builder.sourceware.org/buildbot/#/builders/binutils-fedora-s390x
> > has 2 unexpected ld failures:
> > FAIL: Run pr19719 fun undefined
> > FAIL: pr26580-3
> > 
> > binutils-debian-amd64
> > https://builder.sourceware.org/buildbot/#/builders/binutils-debian-amd64
> > has 2 unexpected ld failures:
> > FAIL: Run p_align-1b with PIE
> > FAIL: Run p_align-1d with -Wl,-z,max-page-size=0x1000 with PIE
> > 
> > binutils-debian-arm64
> > https://builder.sourceware.org/buildbot/#/builders/binutils-debian-arm64
> > has 1 unexpected success:
> > XPASS: Run pr19719 fun undefined
> > 
> > binutils-debian-armhf
> > https://builder.sourceware.org/buildbot/#/builders/binutils-debian-armhf
> > has 8 unexpected ld failures and 7 unexpected successes
> > XPASS: Run pr19719 fun undefined
> > FAIL: Common symbol override ifunc test 1a
> > FAIL: Common symbol override ifunc test 1b
> > FAIL: Run pr18841 with libpr18841b.so
> > FAIL: Run pr18841 with libpr18841c.so
> > FAIL: Run pr18841 with libpr18841bn.so (-z now)
> > FAIL: Run pr18841 with libpr18841cn.so (-z now)
> > FAIL: Run pr23169a
> > FAIL: Run pr23169d
> > XPASS: visibility (hidden_undef) (non PIC)
> > XPASS: visibility (hidden_undef) (non PIC, load offset)
> > XPASS: visibility (hidden_undef) (PIC main, non PIC so)
> > XPASS: visibility (protected_undef) (non PIC)
> > XPASS: visibility (protected_undef) (non PIC, load offset)
> > XPASS: visibility (protected_undef) (PIC main, non PIC so)
> > 
> > The binutils-debian-armhf builder is also the slowest (takes 15
> > minutes). The rest take a few minutes. They should sent email once a
> > new failure occurs (or if one if the currently failing builders starts
> > passing).
> > 
> > There is also one build and check everything builder
> > https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-fedrawhide-x86_64
> > I haven't looked at the test results yet, but they are all stored in
> > the bunsendb.git for later analysis. This builder doesn't sent emails
> > on bad builds. It also takes a very long time to run (from 1 to 7 hours).
> 
> I've received the first email from that build, as it looks. May I please
> ask that this be Cc-ed to individuals (rather than putting them on To: )?
> I view To: as reasonable only if there was breakage which was possible to
> associate with a particular commit at quite high a probability (which
> would require either bisection of there being merely a single commit on
> top of what was tested in the prior run).
> 
> To: otoh quite likely wants to include binutils@ ?

I think the email you received was for the "recovered"
gdb-fedora-s390x builder. This builder failed from the start since the
fix on Friday:
https://builder.sourceware.org/buildbot/#builders/gdb-fedora-s390x

The buildbot assumed everybody who had done a commit since the first
fail should hear the happy news that the build now passes. Although it
is good news, this was probably too many people.

I changed the reporters so only people who committed a change that
caused a build to go from success to failure will get a problem report
email. Then I added a new reporter that just reports to the
mailinglist for any "changed" build (from success to failure or from
failure to success).

That way you should only be on the To: when you most likely caused the
problem. There is one small issue with binutils and gdb because they
share a source repository. You might be included in a report about gdb
or binutils even if you meant to only change one or the other
project. The buildbot knows which directories are part of which
project, but some overlap and it doesn't know that a change it didn't
explicitly tests when the build for one project fails couldn't have
caused that failure.

You can write more complicated reporters in python if you want:
https://docs.buildbot.net/current/manual/configuration/report_generators/
https://docs.buildbot.net/current/manual/configuration/reporters/mail_notifier.html

And as you can see from the attached patch I am probably not a great
python programmer. If someones wants to try to clean the master.cfg
file up a little to remove all the replications, please do!

Cheers,

Mark