Buildbot improvements

Buildbot improvements

[Adhemerval Zanella]

Hi DJ,

First thanks for setting up the CL/CI initiative for the glibc buildbots,
it is really helpful.  Carlos has asked to reach you to improve the 
trybots dump log, so it would include additional information the patch 
sender might use to reproduce the issue.

One thing is the configure line, either the config.log from configure or
at least the command itself (similar to what build-many-glibcs.py do).
I think build system information, such as compiler version and other tools,
can be inferred from the make.log.

Another thing is the machine information itself, such the architecture,
the kernel, and other pieces that might help the developer to figure out
the possible cause of the regression.

Another nice thing to have would be to dump a line where one could use
to download the container used to run the tests to reproduce it locally
(it would still be subject to underlying kernel).

Besides this I think you might want to tune the container your are using,
since a recent regression [1] seems to be a limitation of the total number
of threads (I have seem it before running glibc make check insider docker
containers).

https://www.delorie.com/trybots/32bit/2492/nptl-tst-audit-threads.out

Re: Buildbot improvements

[DJ Delorie]

Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:
> First thanks for setting up the CL/CI initiative for the glibc buildbots,
> it is really helpful.  Carlos has asked to reach you to improve the 
> trybots dump log, so it would include additional information the patch 
> sender might use to reproduce the issue.

I think with every patch build failure, there's something new to log for
the user ;-)

> One thing is the configure line, either the config.log from configure or
> at least the command itself (similar to what build-many-glibcs.py do).
> I think build system information, such as compiler version and other tools,
> can be inferred from the make.log.

I can do that

> Another thing is the machine information itself, such the architecture,
> the kernel, and other pieces that might help the developer to figure out
> the possible cause of the regression.

I can do that, somewhere...  Probably stuff a lot of this extra stuff
into make.log.  Or I can wait for pull requests on do-build.sh ;-)

> Another nice thing to have would be to dump a line where one could use
> to download the container used to run the tests to reproduce it locally
> (it would still be subject to underlying kernel).

That one's trickier, for various reasons.  The container *description*
is trivial (it's in git) but the container is rebuilt *often* - every
time there's a git commit, the baseline is rebuilt, and that uses
whatever the latest Fedora baseline is.  For security reasons, we
discard every build container after every build - there's no telling
what evil a patch in a mailing list might have - so those get rebuilt
for every patch.

Plus, a built container is pretty big.  I wouldn't want to have to store
and/or transmit those.  About half a gig per build just for the
build-specific portions, plus whatever the Fedora baseline adds.

So either it's a lot of data to move around, or I can't guarantee the
same contents as was built.  In my case (the 32-bit builder) it's
trivial to just run your own local 32-bit builder.  In the future, we
hope that build bots will be running on vendor-specific hardware that
might be difficult for the public to obtain, and then the reasoning
might differ in favor of storing built containers.  OTOH we expect those
builds to be of interest to those vendors, who have easier access to
those containers anyway ;-)

> Besides this I think you might want to tune the container your are using,
> since a recent regression [1] seems to be a limitation of the total number
> of threads (I have seem it before running glibc make check insider docker
> containers).
>
> https://www.delorie.com/trybots/32bit/2492/nptl-tst-audit-threads.out

The builder's VM has 8 cores, and I don't limit the containers, so
they're 8 cores each.  The only limit I put on the container is to
disable networking, for security reasons.  I'm new to containers, so if
there's something else I need to be doing... please tell ;-)

Re: Buildbot improvements

[Carlos O'Donell]

On 7/13/21 3:15 PM, DJ Delorie wrote:
>> Another nice thing to have would be to dump a line where one could use
>> to download the container used to run the tests to reproduce it locally
>> (it would still be subject to underlying kernel).
> 
> That one's trickier, for various reasons.  The container *description*
> is trivial (it's in git) but the container is rebuilt *often* - every
> time there's a git commit, the baseline is rebuilt, and that uses
> whatever the latest Fedora baseline is.  For security reasons, we
> discard every build container after every build - there's no telling
> what evil a patch in a mailing list might have - so those get rebuilt
> for every patch.
> 
> Plus, a built container is pretty big.  I wouldn't want to have to store
> and/or transmit those.  About half a gig per build just for the
> build-specific portions, plus whatever the Fedora baseline adds.
> 
> So either it's a lot of data to move around, or I can't guarantee the
> same contents as was built.  In my case (the 32-bit builder) it's
> trivial to just run your own local 32-bit builder.  In the future, we
> hope that build bots will be running on vendor-specific hardware that
> might be difficult for the public to obtain, and then the reasoning
> might differ in favor of storing built containers.  OTOH we expect those
> builds to be of interest to those vendors, who have easier access to
> those containers anyway ;-)

... however, we could output a set of instructions that when used in
a script could reconstitute the built container?

This would mean that for each test we ran the developer could use
the instruction steps to reconstitute the container.

I think there is a good reproducibility argument here.

-- 
Cheers,
Carlos.

Re: Buildbot improvements

[DJ Delorie]

"Carlos O'Donell" <carlos@redhat.com> writes:
> ... however, we could output a set of instructions that when used in
> a script could reconstitute the built container?

That's the "I can't guarantee the same contents as was built" case since
the underlying Fedora container might have changed.

But otherwise, yes.  You'd have to insert the right glibc git tree and
patch, but the container descriptions are tiny.

Re: Buildbot improvements

[Adhemerval Zanella]

On 14/07/2021 18:10, DJ Delorie wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
>> ... however, we could output a set of instructions that when used in
>> a script could reconstitute the built container?
> 
> That's the "I can't guarantee the same contents as was built" case since
> the underlying Fedora container might have changed.
> 
> But otherwise, yes.  You'd have to insert the right glibc git tree and
> patch, but the container descriptions are tiny.
> 

I take this builbot setup is used along with Fedora to stress both systems,
but maybe it would better to use a more stable baseline to make it easier
to reproduce.  Either Fedora current or a way to download the required
packages/tools.

Re: Buildbot improvements

[DJ Delorie]

Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:
> I take this builbot setup is used along with Fedora to stress both systems,
> but maybe it would better to use a more stable baseline to make it easier
> to reproduce.  Either Fedora current or a way to download the required
> packages/tools.

I use the published registry.fedoraproject.org/fedora base container.
We chose to use the latest container for security reasons, to make sure
we had all the current security errata etc.  It probably won't change
often enough to be an issue, but it means that I can't guarantee an
exact reproduction of a given trybot container.

Even if the containers were published, the trybot's container includes a
full copy of the glibc git tree at that moment, and we're hoping that
there will eventually be many trybots - not just mine - so consider the
cost of moving all that data around vs just running your own trybot ;-)

Note that I'm not defending my choice here, just explaining it.  I'm
open to new ideas and tweaks.

Re: Buildbot improvements

[Carlos O'Donell]

On 7/15/21 12:11 AM, DJ Delorie wrote:
> Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:
>> I take this builbot setup is used along with Fedora to stress both systems,
>> but maybe it would better to use a more stable baseline to make it easier
>> to reproduce.  Either Fedora current or a way to download the required
>> packages/tools.
> 
> I use the published registry.fedoraproject.org/fedora base container.
> We chose to use the latest container for security reasons, to make sure
> we had all the current security errata etc.  It probably won't change
> often enough to be an issue, but it means that I can't guarantee an
> exact reproduction of a given trybot container.
> 
> Even if the containers were published, the trybot's container includes a
> full copy of the glibc git tree at that moment, and we're hoping that
> there will eventually be many trybots - not just mine - so consider the
> cost of moving all that data around vs just running your own trybot ;-)
> 
> Note that I'm not defending my choice here, just explaining it.  I'm
> open to new ideas and tweaks.

I think a set of instructions that gets you *close enough* to the container
environment and provides host details will be sufficient.

e.g.
podman pull [something]
git clone [something]
git-pw series apply [something]

I agree with Adhemerval that when this fails we'll want any 3rd party
to be able to explain how they got to their setup. I think the host config
is relevant but that just needs to be discussed with the partner.

If we had a "record commands" function and the integrator used it to record
the steps taken then we could just print those out for the container?

-- 
Cheers,
Carlos.