Regressions getting more common

Regressions getting more common

[Luis Machado]

Hi,

I don't know about other non-x86 architectures, but over the past year 
I've been noticing more and more regressions being introduced, 
unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if 
you have to keep tracking things manually, like we do now.

The buildbots worked great for this very purpose, but Sergio has moved 
on to other duties (thanks for all the work!) and can't maintain it 
anymore. The builders are still there though, sitting mostly idle.
We have a beefy ARM/AArch64 builder, which I can maintain for others to use.

We can do better than to declare things OK after a single round of tests 
under x86, which has been the trend unfortunately.

The subject of better CI has come up multiple times on IRC, with sad 
memories of the gerrit experiment's demise. Now we're left with review 
by e-mail and no broad testing.

I think we need to discuss better validation pre-commit and possible CI 
solutions for GDB. It is pretty easy to exercise x86, but it doesn't 
sound fair to other architectures to have to keep cleaning up after 
things that have only been validated on that architecture.

It would be great to establish a roadmap so we can get GDB's testing to 
today's standards, and maybe revisit the use of more modern patch review 
tools while at it.

What do you think?

Re: Regressions getting more common

[Matt Rice]

On Tue, Oct 13, 2020 at 5:06 PM Luis Machado via Gdb <gdb@sourceware.org> wrote:
>
> Hi,
>
> I don't know about other non-x86 architectures, but over the past year
> I've been noticing more and more regressions being introduced,
> unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if
> you have to keep tracking things manually, like we do now.
>
> The buildbots worked great for this very purpose, but Sergio has moved
> on to other duties (thanks for all the work!) and can't maintain it
> anymore. The builders are still there though, sitting mostly idle.
> We have a beefy ARM/AArch64 builder, which I can maintain for others to use.
>
> We can do better than to declare things OK after a single round of tests
> under x86, which has been the trend unfortunately.
>
> The subject of better CI has come up multiple times on IRC, with sad
> memories of the gerrit experiment's demise. Now we're left with review
> by e-mail and no broad testing.
>

Speaking of gerrit, I noticed that the server side git hooks, used by
https://git-repo.info/
for its repo -upload/git pull-request functionality has landed in git master,
and is slated to be in the next git release...

https://git.kernel.org/pub/scm/git/git.git/commit/?id=6c430a647cb990fc856d328733fa59e1fafadb97

Which seems like an interesting new approach for this stuff.

> I think we need to discuss better validation pre-commit and possible CI
> solutions for GDB. It is pretty easy to exercise x86, but it doesn't
> sound fair to other architectures to have to keep cleaning up after
> things that have only been validated on that architecture.
>
> It would be great to establish a roadmap so we can get GDB's testing to
> today's standards, and maybe revisit the use of more modern patch review
> tools while at it.
>
> What do you think?

Re: Regressions getting more common

[Simon Marchi]

On 2020-10-13 1:05 p.m., Luis Machado via Gdb wrote:
> Hi,
>
> I don't know about other non-x86 architectures, but over the past year I've been noticing more and more regressions being introduced, unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if you have to keep tracking things manually, like we do now.
>
> The buildbots worked great for this very purpose, but Sergio has moved on to other duties (thanks for all the work!) and can't maintain it anymore. The builders are still there though, sitting mostly idle.
> We have a beefy ARM/AArch64 builder, which I can maintain for others to use.
>
> We can do better than to declare things OK after a single round of tests under x86, which has been the trend unfortunately.
>
> The subject of better CI has come up multiple times on IRC, with sad memories of the gerrit experiment's demise. Now we're left with review by e-mail and no broad testing.
>
> I think we need to discuss better validation pre-commit and possible CI solutions for GDB. It is pretty easy to exercise x86, but it doesn't sound fair to other architectures to have to keep cleaning up after things that have only been validated on that architecture.
>
> It would be great to establish a roadmap so we can get GDB's testing to today's standards, and maybe revisit the use of more modern patch review tools while at it.
>
> What do you think?

I agree with all of this.

It all comes down to:

- human resources: a system like that is not fire and forget, there's
  always something to look into to make sure it runs smoothly
- hardware resources: it takes a lot of CPU time, that takes some
  dedicated machines

I think it would be good to reboot the buildbot, but start by focusing
on what delivers the best "bang for the buck".  I remember workers
lagging a lot behind, meaning we would get notifications of breakage
quite a lot after a commit was pushed.  I think it would be good to not
make the workers build all commits on master.  Either do one build a
day, or have them constantly build the current master (as a background /
low priority task).  If there's a regression, then there's a window of
commits that might be responsible for it.  But in any case, I would see
having a stable post-commit CI as the first step.

Then, we can look at having try jobs work again.  I never really liked
submitting try jobs through the buildbot command line tool, I always
found there was to little feedback (did my upload work? what jobs did I
initiate?).  I would receive test results by email and would have a ahrd
time figuring out which result was for which patch.

One idea would be to re-use the now abandonned Gerrit instance for this.
Those who have use Gerrit will probably agree that it integrates quite
well with CI.  After pushing your patch for review, you can ask the CI
to test a given patch (often by setting a label on it).  The CI posts a
comment on the patch with a link to the build, so you can follow it if
you want.  Once it has ran, the CI posts comments on the patch to say
that version N of the patch has this result, again with a link to the
build.

I would be happy to help take over maintenance of the buildbot master
from Sergio, but I wouldn't want to be alone in this.

Simon

Re: Regressions getting more common

[Simon Marchi]

On 2020-10-13 9:13 p.m., Matt Rice via Gdb wrote:
> Speaking of gerrit, I noticed that the server side git hooks, used by
> https://git-repo.info/
> for its repo -upload/git pull-request functionality has landed in git master,
> and is slated to be in the next git release...
>
> https://git.kernel.org/pub/scm/git/git.git/commit/?id=6c430a647cb990fc856d328733fa59e1fafadb97
>
> Which seems like an interesting new approach for this stuff.

Interesting.  If I understand correctly, this allows implementing
something similar to Gerrit's magic branches, where when you push to it,
a script/hook on the  server can do something arbitrary with it?

In any case, I don't think we'll want to develop anything new here, that
wouldn't be a very good way to spend our time.  But perhaps this will
lead to useful tools being developed in the future.

Simon

Re: Regressions getting more common

[Kamil Rytarowski]

[-- Attachment #1.1: Type: text/plain, Size: 2128 bytes --]

Hi Luis,

I talked to Sergio and his ex-boss from Red Hat as I proposed to
maintain the CI infrastructure on behalf of my new company - Moritz
Systems (www.moritz.systems).

Unfortunately RH currently cannot rearrange the budgets, but how about
Linaro? We could spare a dedicated developer full-time or part-time to
monitor the reports and maintain the infrastructure.

We are experienced in maintaining CI infrastructure for years in the BSD
distribution development is a heavy CI workflow (multiple CPUs, multiple
tests, emulators, real machines etc).

Can we discuss it further?

Best regards,

Kamil Rytarowski
CTO, Moritz Systems
www.moritz.systems

On 13.10.2020 19:05, Luis Machado via Gdb wrote:
> Hi,
> 
> I don't know about other non-x86 architectures, but over the past year
> I've been noticing more and more regressions being introduced,
> unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if
> you have to keep tracking things manually, like we do now.
> 
> The buildbots worked great for this very purpose, but Sergio has moved
> on to other duties (thanks for all the work!) and can't maintain it
> anymore. The builders are still there though, sitting mostly idle.
> We have a beefy ARM/AArch64 builder, which I can maintain for others to
> use.
> 
> We can do better than to declare things OK after a single round of tests
> under x86, which has been the trend unfortunately.
> 
> The subject of better CI has come up multiple times on IRC, with sad
> memories of the gerrit experiment's demise. Now we're left with review
> by e-mail and no broad testing.
> 
> I think we need to discuss better validation pre-commit and possible CI
> solutions for GDB. It is pretty easy to exercise x86, but it doesn't
> sound fair to other architectures to have to keep cleaning up after
> things that have only been validated on that architecture.
> 
> It would be great to establish a roadmap so we can get GDB's testing to
> today's standards, and maybe revisit the use of more modern patch review
> tools while at it.
> 
> What do you think?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Regressions getting more common

[Matt Rice]

On Wed, Oct 14, 2020 at 2:46 PM Simon Marchi <simark@simark.ca> wrote:
>
>
> On 2020-10-13 9:13 p.m., Matt Rice via Gdb wrote:
> > Speaking of gerrit, I noticed that the server side git hooks, used by
> > https://git-repo.info/
> > for its repo -upload/git pull-request functionality has landed in git master,
> > and is slated to be in the next git release...
> >
> > https://git.kernel.org/pub/scm/git/git.git/commit/?id=6c430a647cb990fc856d328733fa59e1fafadb97
> >
> > Which seems like an interesting new approach for this stuff.
>
> Interesting.  If I understand correctly, this allows implementing
> something similar to Gerrit's magic branches, where when you push to it,
> a script/hook on the  server can do something arbitrary with it?

More or less, IIUC (haven't not waded into the specific details yet),
I believe it just allows one to separate the permissions to various
refs/branches based on the protocol used to push. And then then you
can do something arbitrary with it.

But because it is integrated at the protocol level, you get the
distributive aspects of git.
which is to me the major distinction.  I.e. so you can set up a mirror
repository downstream from the gdb one which, but accepts
pull-requests.

> In any case, I don't think we'll want to develop anything new here, that
> wouldn't be a very good way to spend our time.  But perhaps this will
> lead to useful tools being developed in the future.

Totally agree, I just happened to be tinkering with it already,
so i'll at least see how my local experiments with it go.


> Simon

Re: Regressions getting more common

[Rainer Orth]

Hi Simon,

> I think it would be good to reboot the buildbot, but start by focusing
> on what delivers the best "bang for the buck".  I remember workers
> lagging a lot behind, meaning we would get notifications of breakage
> quite a lot after a commit was pushed.  I think it would be good to not
> make the workers build all commits on master.  Either do one build a
> day, or have them constantly build the current master (as a background /
> low priority task).  If there's a regression, then there's a window of
> commits that might be responsible for it.  But in any case, I would see
> having a stable post-commit CI as the first step.

maybe we can learn a thing or two from the LLVM buildbots

	https://llvm.org/docs/HowToAddABuilder.html

with the config at

	https://github.com/llvm/llvm-zorg

For one, they do group several commits into one build if the bots lag
too far behind (which is a strict necessity given the enormous size of
LLVM, even if many bots have a smaller scope like only testing clang,
compiler-rt/the sanitizers, ...).  Another possibility is not to run
full builds every time but only under special circumstances (haven't
fully understood which those are, but I'm operating the Solaris
buildbots in that mode to avoid overwhelming them), defaulting to
incremental builds.  Given that GDB build times are reasonably short (a
matter of minutes usually), I'm unsure if this is necessary, though.

> Then, we can look at having try jobs work again.  I never really liked
> submitting try jobs through the buildbot command line tool, I always
> found there was to little feedback (did my upload work? what jobs did I
> initiate?).  I would receive test results by email and would have a ahrd
> time figuring out which result was for which patch.

I have to admit I haven't ever used the try jobs myself.  However, AFAIK
they still do work with the current buildmaster.  Only the jobs
triggered by commits seem to have stopped.

That said, they are certainly useful, especially to allow people to test
their patches on uncommon targets.  I know for a fact that several GDB
developers used my Solaris bots that way.  While the GCC compile farm
could be used as well, this means much more effort for the developers.

However, as a maintainer of two of those bots, I'd be little pleased to
have to maintain and watch over two different packages for this purpose:
the charm of the current buildbot setup is that is serves both (as does,
e.g. the trybot component of the golang farm).

> One idea would be to re-use the now abandonned Gerrit instance for this.
> Those who have use Gerrit will probably agree that it integrates quite
> well with CI.  After pushing your patch for review, you can ask the CI
> to test a given patch (often by setting a label on it).  The CI posts a
> comment on the patch with a link to the build, so you can follow it if
> you want.  Once it has ran, the CI posts comments on the patch to say
> that version N of the patch has this result, again with a link to the
> build.
>
> I would be happy to help take over maintenance of the buildbot master
> from Sergio, but I wouldn't want to be alone in this.

It would be really great if the GDB bots could get started again.
However, I'm unable to help: way too much on my plate already.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: Regressions getting more common

[Sergio Durigan Junior]

On Wednesday, October 14 2020, Simon Marchi wrote:

> Then, we can look at having try jobs work again.  I never really liked
> submitting try jobs through the buildbot command line tool, I always
> found there was to little feedback (did my upload work? what jobs did I
> initiate?).  I would receive test results by email and would have a ahrd
> time figuring out which result was for which patch.
>
> One idea would be to re-use the now abandonned Gerrit instance for this.
> Those who have use Gerrit will probably agree that it integrates quite
> well with CI.  After pushing your patch for review, you can ask the CI
> to test a given patch (often by setting a label on it).  The CI posts a
> comment on the patch with a link to the build, so you can follow it if
> you want.  Once it has ran, the CI posts comments on the patch to say
> that version N of the patch has this result, again with a link to the
> build.

There is another idea, which is simpler to implement, which is to make
the Buildbot monitor user branches with a special naming convention,
like user/sergiodj/try-foobar or some such.  When a new branch showed
up, the Buildbot would then build it, test things, and then send the
results via email to the author of the commit (bonus points if it can
also send an email to the committer).  One of the problems to be solved
(not particular to this solution, but to the Buildbot in general) is to
decide which upstream commit the Buildbot will compare the results
against.  If you're considering running one/two builds per day, this
problem becomes easier to solve.  I had a hard time solving it because I
was running builds for every commit, as you know.

This was next on my TODO list for the Buildbot (along with stop building
separate commits, and start grouping them), but I could not find the
time to implement it.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/

Re: Regressions getting more common

[Luis Machado]

Hi,

On 10/14/20 11:41 AM, Simon Marchi wrote:
> On 2020-10-13 1:05 p.m., Luis Machado via Gdb wrote:
>> Hi,
>>
>> I don't know about other non-x86 architectures, but over the past year I've been noticing more and more regressions being introduced, unnoticed, for ARM/AArch64. This is not good and causes a lot of pain if you have to keep tracking things manually, like we do now.
>>
>> The buildbots worked great for this very purpose, but Sergio has moved on to other duties (thanks for all the work!) and can't maintain it anymore. The builders are still there though, sitting mostly idle.
>> We have a beefy ARM/AArch64 builder, which I can maintain for others to use.
>>
>> We can do better than to declare things OK after a single round of tests under x86, which has been the trend unfortunately.
>>
>> The subject of better CI has come up multiple times on IRC, with sad memories of the gerrit experiment's demise. Now we're left with review by e-mail and no broad testing.
>>
>> I think we need to discuss better validation pre-commit and possible CI solutions for GDB. It is pretty easy to exercise x86, but it doesn't sound fair to other architectures to have to keep cleaning up after things that have only been validated on that architecture.
>>
>> It would be great to establish a roadmap so we can get GDB's testing to today's standards, and maybe revisit the use of more modern patch review tools while at it.
>>
>> What do you think?
> 
> I agree with all of this.
> 
> It all comes down to:
> 
> - human resources: a system like that is not fire and forget, there's
>    always something to look into to make sure it runs smoothly
> - hardware resources: it takes a lot of CPU time, that takes some
>    dedicated machines

Indeed. There will most definitely be some maintenance burden. At least 
from ARM/AArch64's side, I can offer to keep the builders in working 
condition and reasonably updated.

I have the say I'm not terribly skilled with the buildbot 
infrastructure, so I may not be able to help much there. I can learn 
though. So far I've only dealt with Jenkins/Gerrit.

> 
> I think it would be good to reboot the buildbot, but start by focusing
> on what delivers the best "bang for the buck".  I remember workers
> lagging a lot behind, meaning we would get notifications of breakage
> quite a lot after a commit was pushed.  I think it would be good to not
> make the workers build all commits on master.  Either do one build a
> day, or have them constantly build the current master (as a background /
> low priority task).  If there's a regression, then there's a window of
> commits that might be responsible for it.  But in any case, I would see
> having a stable post-commit CI as the first step.

That sounds reasonable to me. That is already a good improvement.

> 
> Then, we can look at having try jobs work again.  I never really liked
> submitting try jobs through the buildbot command line tool, I always
> found there was to little feedback (did my upload work? what jobs did I
> initiate?).  I would receive test results by email and would have a ahrd
> time figuring out which result was for which patch.

I think it lacked more verbosity, or better ways to follow the status of 
that particular build. Like, "here, take this key and use it to monitor 
this particular build request".

> 
> One idea would be to re-use the now abandonned Gerrit instance for this.
> Those who have use Gerrit will probably agree that it integrates quite
> well with CI.  After pushing your patch for review, you can ask the CI
> to test a given patch (often by setting a label on it).  The CI posts a
> comment on the patch with a link to the build, so you can follow it if
> you want.  Once it has ran, the CI posts comments on the patch to say
> that version N of the patch has this result, again with a link to the
> build.

That resembles the workflow I experimented with Jenkins/Gerrit. I quite 
like it and it kept things pretty organized.

But, like I said, I wouldn't know how to configure this properly. But 
I'll ask around at Linaro to see if we can spare someone that can help.

> 
> I would be happy to help take over maintenance of the buildbot master
> from Sergio, but I wouldn't want to be alone in this.

I'm willing to help here. I just need to get more familiar with some 
admin steps so I can get things unstuck when needed.

Re: Regressions getting more common

[Simon Marchi]

On 2020-10-15 8:55 a.m., Luis Machado wrote:
> That resembles the workflow I experimented with Jenkins/Gerrit. I quite 
> like it and it kept things pretty organized.
> 
> But, like I said, I wouldn't know how to configure this properly. But 
> I'll ask around at Linaro to see if we can spare someone that can help.

There is a buildbot change source that watches Gerrit events:

https://docs.buildbot.net/current/manual/configuration/changesources.html#gerritchangesource

So I presume it's not very complicated.  I'd like to tinker with this eventually,
maybe with a local install.

Simon