* Buildbot improvements @ 2021-07-13 18:18 Adhemerval Zanella 2021-07-13 19:15 ` DJ Delorie 0 siblings, 1 reply; 7+ messages in thread From: Adhemerval Zanella @ 2021-07-13 18:18 UTC (permalink / raw) To: DJ Delorie; +Cc: Carlos O'Donell, Libc-alpha Hi DJ, First thanks for setting up the CL/CI initiative for the glibc buildbots, it is really helpful. Carlos has asked to reach you to improve the trybots dump log, so it would include additional information the patch sender might use to reproduce the issue. One thing is the configure line, either the config.log from configure or at least the command itself (similar to what build-many-glibcs.py do). I think build system information, such as compiler version and other tools, can be inferred from the make.log. Another thing is the machine information itself, such the architecture, the kernel, and other pieces that might help the developer to figure out the possible cause of the regression. Another nice thing to have would be to dump a line where one could use to download the container used to run the tests to reproduce it locally (it would still be subject to underlying kernel). Besides this I think you might want to tune the container your are using, since a recent regression [1] seems to be a limitation of the total number of threads (I have seem it before running glibc make check insider docker containers). https://www.delorie.com/trybots/32bit/2492/nptl-tst-audit-threads.out ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Buildbot improvements 2021-07-13 18:18 Buildbot improvements Adhemerval Zanella @ 2021-07-13 19:15 ` DJ Delorie 2021-07-14 20:43 ` Carlos O'Donell 0 siblings, 1 reply; 7+ messages in thread From: DJ Delorie @ 2021-07-13 19:15 UTC (permalink / raw) To: Adhemerval Zanella; +Cc: carlos, libc-alpha Adhemerval Zanella <adhemerval.zanella@linaro.org> writes: > First thanks for setting up the CL/CI initiative for the glibc buildbots, > it is really helpful. Carlos has asked to reach you to improve the > trybots dump log, so it would include additional information the patch > sender might use to reproduce the issue. I think with every patch build failure, there's something new to log for the user ;-) > One thing is the configure line, either the config.log from configure or > at least the command itself (similar to what build-many-glibcs.py do). > I think build system information, such as compiler version and other tools, > can be inferred from the make.log. I can do that > Another thing is the machine information itself, such the architecture, > the kernel, and other pieces that might help the developer to figure out > the possible cause of the regression. I can do that, somewhere... Probably stuff a lot of this extra stuff into make.log. Or I can wait for pull requests on do-build.sh ;-) > Another nice thing to have would be to dump a line where one could use > to download the container used to run the tests to reproduce it locally > (it would still be subject to underlying kernel). That one's trickier, for various reasons. The container *description* is trivial (it's in git) but the container is rebuilt *often* - every time there's a git commit, the baseline is rebuilt, and that uses whatever the latest Fedora baseline is. For security reasons, we discard every build container after every build - there's no telling what evil a patch in a mailing list might have - so those get rebuilt for every patch. Plus, a built container is pretty big. I wouldn't want to have to store and/or transmit those. About half a gig per build just for the build-specific portions, plus whatever the Fedora baseline adds. So either it's a lot of data to move around, or I can't guarantee the same contents as was built. In my case (the 32-bit builder) it's trivial to just run your own local 32-bit builder. In the future, we hope that build bots will be running on vendor-specific hardware that might be difficult for the public to obtain, and then the reasoning might differ in favor of storing built containers. OTOH we expect those builds to be of interest to those vendors, who have easier access to those containers anyway ;-) > Besides this I think you might want to tune the container your are using, > since a recent regression [1] seems to be a limitation of the total number > of threads (I have seem it before running glibc make check insider docker > containers). > > https://www.delorie.com/trybots/32bit/2492/nptl-tst-audit-threads.out The builder's VM has 8 cores, and I don't limit the containers, so they're 8 cores each. The only limit I put on the container is to disable networking, for security reasons. I'm new to containers, so if there's something else I need to be doing... please tell ;-) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Buildbot improvements 2021-07-13 19:15 ` DJ Delorie @ 2021-07-14 20:43 ` Carlos O'Donell 2021-07-14 21:10 ` DJ Delorie 0 siblings, 1 reply; 7+ messages in thread From: Carlos O'Donell @ 2021-07-14 20:43 UTC (permalink / raw) To: DJ Delorie, Adhemerval Zanella; +Cc: libc-alpha On 7/13/21 3:15 PM, DJ Delorie wrote: >> Another nice thing to have would be to dump a line where one could use >> to download the container used to run the tests to reproduce it locally >> (it would still be subject to underlying kernel). > > That one's trickier, for various reasons. The container *description* > is trivial (it's in git) but the container is rebuilt *often* - every > time there's a git commit, the baseline is rebuilt, and that uses > whatever the latest Fedora baseline is. For security reasons, we > discard every build container after every build - there's no telling > what evil a patch in a mailing list might have - so those get rebuilt > for every patch. > > Plus, a built container is pretty big. I wouldn't want to have to store > and/or transmit those. About half a gig per build just for the > build-specific portions, plus whatever the Fedora baseline adds. > > So either it's a lot of data to move around, or I can't guarantee the > same contents as was built. In my case (the 32-bit builder) it's > trivial to just run your own local 32-bit builder. In the future, we > hope that build bots will be running on vendor-specific hardware that > might be difficult for the public to obtain, and then the reasoning > might differ in favor of storing built containers. OTOH we expect those > builds to be of interest to those vendors, who have easier access to > those containers anyway ;-) ... however, we could output a set of instructions that when used in a script could reconstitute the built container? This would mean that for each test we ran the developer could use the instruction steps to reconstitute the container. I think there is a good reproducibility argument here. -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Buildbot improvements 2021-07-14 20:43 ` Carlos O'Donell @ 2021-07-14 21:10 ` DJ Delorie 2021-07-14 21:58 ` Adhemerval Zanella 0 siblings, 1 reply; 7+ messages in thread From: DJ Delorie @ 2021-07-14 21:10 UTC (permalink / raw) To: Carlos O'Donell; +Cc: adhemerval.zanella, libc-alpha "Carlos O'Donell" <carlos@redhat.com> writes: > ... however, we could output a set of instructions that when used in > a script could reconstitute the built container? That's the "I can't guarantee the same contents as was built" case since the underlying Fedora container might have changed. But otherwise, yes. You'd have to insert the right glibc git tree and patch, but the container descriptions are tiny. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Buildbot improvements 2021-07-14 21:10 ` DJ Delorie @ 2021-07-14 21:58 ` Adhemerval Zanella 2021-07-15 4:11 ` DJ Delorie 0 siblings, 1 reply; 7+ messages in thread From: Adhemerval Zanella @ 2021-07-14 21:58 UTC (permalink / raw) To: DJ Delorie, Carlos O'Donell; +Cc: libc-alpha On 14/07/2021 18:10, DJ Delorie wrote: > "Carlos O'Donell" <carlos@redhat.com> writes: >> ... however, we could output a set of instructions that when used in >> a script could reconstitute the built container? > > That's the "I can't guarantee the same contents as was built" case since > the underlying Fedora container might have changed. > > But otherwise, yes. You'd have to insert the right glibc git tree and > patch, but the container descriptions are tiny. > I take this builbot setup is used along with Fedora to stress both systems, but maybe it would better to use a more stable baseline to make it easier to reproduce. Either Fedora current or a way to download the required packages/tools. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Buildbot improvements 2021-07-14 21:58 ` Adhemerval Zanella @ 2021-07-15 4:11 ` DJ Delorie 2021-07-15 12:24 ` Carlos O'Donell 0 siblings, 1 reply; 7+ messages in thread From: DJ Delorie @ 2021-07-15 4:11 UTC (permalink / raw) To: Adhemerval Zanella; +Cc: carlos, libc-alpha Adhemerval Zanella <adhemerval.zanella@linaro.org> writes: > I take this builbot setup is used along with Fedora to stress both systems, > but maybe it would better to use a more stable baseline to make it easier > to reproduce. Either Fedora current or a way to download the required > packages/tools. I use the published registry.fedoraproject.org/fedora base container. We chose to use the latest container for security reasons, to make sure we had all the current security errata etc. It probably won't change often enough to be an issue, but it means that I can't guarantee an exact reproduction of a given trybot container. Even if the containers were published, the trybot's container includes a full copy of the glibc git tree at that moment, and we're hoping that there will eventually be many trybots - not just mine - so consider the cost of moving all that data around vs just running your own trybot ;-) Note that I'm not defending my choice here, just explaining it. I'm open to new ideas and tweaks. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Buildbot improvements 2021-07-15 4:11 ` DJ Delorie @ 2021-07-15 12:24 ` Carlos O'Donell 0 siblings, 0 replies; 7+ messages in thread From: Carlos O'Donell @ 2021-07-15 12:24 UTC (permalink / raw) To: DJ Delorie, Adhemerval Zanella; +Cc: libc-alpha On 7/15/21 12:11 AM, DJ Delorie wrote: > Adhemerval Zanella <adhemerval.zanella@linaro.org> writes: >> I take this builbot setup is used along with Fedora to stress both systems, >> but maybe it would better to use a more stable baseline to make it easier >> to reproduce. Either Fedora current or a way to download the required >> packages/tools. > > I use the published registry.fedoraproject.org/fedora base container. > We chose to use the latest container for security reasons, to make sure > we had all the current security errata etc. It probably won't change > often enough to be an issue, but it means that I can't guarantee an > exact reproduction of a given trybot container. > > Even if the containers were published, the trybot's container includes a > full copy of the glibc git tree at that moment, and we're hoping that > there will eventually be many trybots - not just mine - so consider the > cost of moving all that data around vs just running your own trybot ;-) > > Note that I'm not defending my choice here, just explaining it. I'm > open to new ideas and tweaks. I think a set of instructions that gets you *close enough* to the container environment and provides host details will be sufficient. e.g. podman pull [something] git clone [something] git-pw series apply [something] I agree with Adhemerval that when this fails we'll want any 3rd party to be able to explain how they got to their setup. I think the host config is relevant but that just needs to be discussed with the partner. If we had a "record commands" function and the integrator used it to record the steps taken then we could just print those out for the container? -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-07-15 12:24 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-07-13 18:18 Buildbot improvements Adhemerval Zanella 2021-07-13 19:15 ` DJ Delorie 2021-07-14 20:43 ` Carlos O'Donell 2021-07-14 21:10 ` DJ Delorie 2021-07-14 21:58 ` Adhemerval Zanella 2021-07-15 4:11 ` DJ Delorie 2021-07-15 12:24 ` Carlos O'Donell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).