Re: ☠ Buildbot (GNU Toolchain): binutils-gdb - failed update (failure) (master)

Re: ☠ Buildbot (GNU Toolchain): binutils-gdb - failed update (failure) (master)

[Mark Wielaard]

Hi Luis,

On Fri, Jul 22, 2022 at 01:52:30AM +0100, Luis Machado wrote:
> On 7/21/22 20:41, builder--- via Binutils wrote:
> > A new failure has been detected on builder binutils-debian-ppc64 while building binutils-gdb.
> > 
> > Full details are available at:
> >      https://builder.sourceware.org/buildbot/#builders/78/builds/373
> > 
> > Build state: failed update (failure)
> > Revision: (unknown)
> > Worker: debian-ppc64
> > Build Reason: (unknown)
> > Blamelist: Andrew Burgess <aburgess@redhat.com>, H.J. Lu <hjl.tools@gmail.com>, Tom de Vries <tdevries@suse.de>, Torbjörn SVENSSON <torbjorn.svensson@foss.st.com>
> > 
> > Steps:
> > 
> > - 0: worker_preparation ( success )
> > 
> > - 1: git checkout ( failure )
> >      Logs:
> >          - stdio: https://builder.sourceware.org/buildbot/#builders/78/builds/373/steps/1/logs/stdio
> > 
> 
> It seems the builder ran out of disk space.

Yes it did (and the ppc64le worker too). We keep all builds, which can
add up. So now we do a make clean at the end to save some space:
https://sourceware.org/pipermail/buildbot/2022q3/000162.html
Both workers have 5GB to 10GB free again now.
Sorry for the trouble.

Cheers,

Mark

Re: ☠ Buildbot (GNU Toolchain): binutils-gdb - failed update (failure) (master)

[Mark Wielaard]

Hi Jan,

On Tue, Aug 16, 2022 at 09:52:39AM +0200, Jan Beulich wrote:
> On 16.08.2022 09:47, builder@sourceware.org wrote:
> > A new failure has been detected on builder binutils-debian-testing-x86_64 while building binutils-gdb.
> > 
> > Full details are available at:
> >     https://builder.sourceware.org/buildbot/#builders/149/builds/233
> > 
> > Build state: failed update (failure)
> > Revision: (unknown)
> > Worker: bbo1-1
> > Build Reason: (unknown)
> > Blamelist: Jan Beulich <jbeulich@suse.com>
> 
> I consider this wrong for ...
> 
> > Steps:
> > 
> > - 0: worker_preparation ( success )
> > 
> > - 1: git checkout ( failure )
> >     Logs:
> >         - stdio: https://builder.sourceware.org/buildbot/#builders/149/builds/233/steps/1/logs/stdio
> 
> ... this kind of failure, which has been happening repeatedly. Correcting this
> would then likely also lead to such mails no longer being sent to individuals,
> but - if at all - to the list only.

Agreed. It doesn't actually happen that often, maybe once in the 200
builds, but we are doing 500 builds a day. So it definitely happens
multiple times a day now :{

It looks like a bug in buildbot, normally transient network issues
should just be retried. But for some reason the get fetch step as we
use it generates a hard error. See also the TODO file. It might be
that a fix is already upstream and we'll get it when upgrading when a
new version is released. If that doesn't happen soon I'll try
backporting that fix and/or do the git fetch updates differently.

Cheers,

Mark