Re: Buildbot improvements

[DJ Delorie <dj@redhat.com>]

Adhemerval Zanella <adhemerval.zanella@linaro.org> writes:
> First thanks for setting up the CL/CI initiative for the glibc buildbots,
> it is really helpful.  Carlos has asked to reach you to improve the 
> trybots dump log, so it would include additional information the patch 
> sender might use to reproduce the issue.

I think with every patch build failure, there's something new to log for
the user ;-)

> One thing is the configure line, either the config.log from configure or
> at least the command itself (similar to what build-many-glibcs.py do).
> I think build system information, such as compiler version and other tools,
> can be inferred from the make.log.

I can do that

> Another thing is the machine information itself, such the architecture,
> the kernel, and other pieces that might help the developer to figure out
> the possible cause of the regression.

I can do that, somewhere...  Probably stuff a lot of this extra stuff
into make.log.  Or I can wait for pull requests on do-build.sh ;-)

> Another nice thing to have would be to dump a line where one could use
> to download the container used to run the tests to reproduce it locally
> (it would still be subject to underlying kernel).

That one's trickier, for various reasons.  The container *description*
is trivial (it's in git) but the container is rebuilt *often* - every
time there's a git commit, the baseline is rebuilt, and that uses
whatever the latest Fedora baseline is.  For security reasons, we
discard every build container after every build - there's no telling
what evil a patch in a mailing list might have - so those get rebuilt
for every patch.

Plus, a built container is pretty big.  I wouldn't want to have to store
and/or transmit those.  About half a gig per build just for the
build-specific portions, plus whatever the Fedora baseline adds.

So either it's a lot of data to move around, or I can't guarantee the
same contents as was built.  In my case (the 32-bit builder) it's
trivial to just run your own local 32-bit builder.  In the future, we
hope that build bots will be running on vendor-specific hardware that
might be difficult for the public to obtain, and then the reasoning
might differ in favor of storing built containers.  OTOH we expect those
builds to be of interest to those vendors, who have easier access to
those containers anyway ;-)

> Besides this I think you might want to tune the container your are using,
> since a recent regression [1] seems to be a limitation of the total number
> of threads (I have seem it before running glibc make check insider docker
> containers).
>
> https://www.delorie.com/trybots/32bit/2492/nptl-tst-audit-threads.out

The builder's VM has 8 cores, and I don't limit the containers, so
they're 8 cores each.  The only limit I put on the container is to
disable networking, for security reasons.  I'm new to containers, so if
there's something else I need to be doing... please tell ;-)