Re: Regressions getting more common

[Luis Machado <luis.machado@linaro.org>]

Hi,

On 10/14/20 11:41 AM, Simon Marchi wrote:
> On 2020-10-13 1:05 p.m., Luis Machado via Gdb wrote:
>> Hi,
>>
>> I don't know about other non-x86 architectures, but over the past year I've been noticing more and more regressions being introduced, unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if you have to keep tracking things manually, like we do now.
>>
>> The buildbots worked great for this very purpose, but Sergio has moved on to other duties (thanks for all the work!) and can't maintain it anymore. The builders are still there though, sitting mostly idle.
>> We have a beefy ARM/AArch64 builder, which I can maintain for others to use.
>>
>> We can do better than to declare things OK after a single round of tests under x86, which has been the trend unfortunately.
>>
>> The subject of better CI has come up multiple times on IRC, with sad memories of the gerrit experiment's demise. Now we're left with review by e-mail and no broad testing.
>>
>> I think we need to discuss better validation pre-commit and possible CI solutions for GDB. It is pretty easy to exercise x86, but it doesn't sound fair to other architectures to have to keep cleaning up after things that have only been validated on that architecture.
>>
>> It would be great to establish a roadmap so we can get GDB's testing to today's standards, and maybe revisit the use of more modern patch review tools while at it.
>>
>> What do you think?
> 
> I agree with all of this.
> 
> It all comes down to:
> 
> - human resources: a system like that is not fire and forget, there's
>    always something to look into to make sure it runs smoothly
> - hardware resources: it takes a lot of CPU time, that takes some
>    dedicated machines

Indeed. There will most definitely be some maintenance burden. At least 
from ARM/AArch64's side, I can offer to keep the builders in working 
condition and reasonably updated.

I have the say I'm not terribly skilled with the buildbot 
infrastructure, so I may not be able to help much there. I can learn 
though. So far I've only dealt with Jenkins/Gerrit.

> 
> I think it would be good to reboot the buildbot, but start by focusing
> on what delivers the best "bang for the buck".  I remember workers
> lagging a lot behind, meaning we would get notifications of breakage
> quite a lot after a commit was pushed.  I think it would be good to not
> make the workers build all commits on master.  Either do one build a
> day, or have them constantly build the current master (as a background /
> low priority task).  If there's a regression, then there's a window of
> commits that might be responsible for it.  But in any case, I would see
> having a stable post-commit CI as the first step.

That sounds reasonable to me. That is already a good improvement.

> 
> Then, we can look at having try jobs work again.  I never really liked
> submitting try jobs through the buildbot command line tool, I always
> found there was to little feedback (did my upload work? what jobs did I
> initiate?).  I would receive test results by email and would have a ahrd
> time figuring out which result was for which patch.

I think it lacked more verbosity, or better ways to follow the status of 
that particular build. Like, "here, take this key and use it to monitor 
this particular build request".

> 
> One idea would be to re-use the now abandonned Gerrit instance for this.
> Those who have use Gerrit will probably agree that it integrates quite
> well with CI.  After pushing your patch for review, you can ask the CI
> to test a given patch (often by setting a label on it).  The CI posts a
> comment on the patch with a link to the build, so you can follow it if
> you want.  Once it has ran, the CI posts comments on the patch to say
> that version N of the patch has this result, again with a link to the
> build.

That resembles the workflow I experimented with Jenkins/Gerrit. I quite 
like it and it kept things pretty organized.

But, like I said, I wouldn't know how to configure this properly. But 
I'll ask around at Linaro to see if we can spare someone that can help.

> 
> I would be happy to help take over maintenance of the buildbot master
> from Sergio, but I wouldn't want to be alone in this.

I'm willing to help here. I just need to get more familiar with some 
admin steps so I can get things unstuck when needed.