Buildbot failures due to checkout

Buildbot failures due to checkout

[Richard Earnshaw]

It seems that the buildbot is failing regularly due to git checkouts not 
succeeding.  I'm guessing this is because it's using anonymous git 
access and therefore being throttled due to server load.  But it's 
leading to a lot of additional 'spam' on the mailing list.

Is there anything that can be done about this?  Either by nobbling the 
bot to not whinge in this case or by giving the bot more priority on the 
server so that it doesn't get throttled so often.

R.

Re: Buildbot failures due to checkout

[Mark Wielaard]

Hi Richard,

On Thu, 2022-08-25 at 11:13 +0100, Richard Earnshaw via Binutils wrote:
> It seems that the buildbot is failing regularly due to git checkouts not 
> succeeding.  I'm guessing this is because it's using anonymous git 
> access and therefore being throttled due to server load.  But it's 
> leading to a lot of additional 'spam' on the mailing list.
> 
> Is there anything that can be done about this?  Either by nobbling the 
> bot to not whinge in this case or by giving the bot more priority on the 
> server so that it doesn't get throttled so often.

Yeah, my apologies. It is a TODO item:
https://sourceware.org/git/?p=builder.git;a=blob;f=TODO;hb=HEAD

- Git retry step is broken. Looking at the code it only works for
  incremental updates, not for mode='full', method='fresh' which we
  are using. Given we often already use srcdir != builddir we might
  just use incremental mode. Or we could fix the code?

I see there was just the following commit upstream:
git: Do not throw on intial _fetch() failure if retry or clobber
requested
https://github.com/buildbot/buildbot/commit/b2cad8d2840a74455366ddc1db0bcf4fd199125c

Maybe that would fix it. I'll see if I can backport that to our release
or hope there will be a new upstream release soon to upgrade to.

Frank, could we add the worker IPs to some allow list on sourceware?
There are 25+ workers now though (hurrah for success!)
https://builder.sourceware.org/buildbot/#/workers

Cheers,

Mark