* [RFC] Proposal for hosting GDB CI builds @ 2021-06-30 16:46 Luis Machado 2021-07-01 9:40 ` Rainer Orth ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Luis Machado @ 2021-06-30 16:46 UTC (permalink / raw) To: gdb Hi, This proposal comes as an attempt to revive the GDB CI builds, given the previous one (buildbot) is no longer being maintained by Sergio (thanks a lot for keeping it working for all these years by the way). CI GDB builds are a great help for spotting regressions without having to do the tedious and time-consuming work of running the GDB testsuite for each git revision, for each architecture and reading through hundreds of lines of summaries. If a regression is spotted, then one needs to bisect to find the culprit. This isn't great, specially for architectures without great availability of desktop hardware. Ideally, for each commit, we'd run full builds to validate the state of the tree, but we're not quite there yet. So meanwhile, having some level of automation to get the builds done without manual intervention sounds like a reasonable step forward. From previous IRC conversations, it seems to be a consensus that availability of processing power is not a problem. It is reasonably easy to find hardware to do some builds. The most difficult resource to find is manpower to setup the CI infrastructure and keep it running. With the above said, I've discussed this internally at Linaro and we can spare some manpower to setup and maintain an isolated Linaro-hosted Jenkins instance for GDB CI builds. Linaro can take care of providing builders and build jobs for ARM. Other architectures would be handled by their respective contributors. Those contributors can write jobs and plug builders as needed. Setting up new jobs doesn't require the use of the web interface. It can be done with yaml files in a git repo. It is reasonably simple. You can see an example of the Linaro CI here: https://ci.linaro.org/ Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/ And also the summaries for GDB testsuite runs: https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html Of course, this effort only makes sense if the community is OK with using Jenkins as the CI mechanism and if it actually sees value in having a system like this in place. Thoughts? Questions? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-06-30 16:46 [RFC] Proposal for hosting GDB CI builds Luis Machado @ 2021-07-01 9:40 ` Rainer Orth 2021-07-01 11:48 ` Luis Machado 2021-07-01 11:37 ` Christophe LYON 2021-07-20 15:21 ` Luis Machado 2 siblings, 1 reply; 11+ messages in thread From: Rainer Orth @ 2021-07-01 9:40 UTC (permalink / raw) To: Luis Machado via Gdb Hi Luis, > Linaro can take care of providing builders and build jobs for ARM. Other > architectures would be handled by their respective contributors. Those > contributors can write jobs and plug builders as needed. thanks for coming forward with this: this is very welcome, given how easy it is to miss build failures and other issues especially on not-so-common targets. However, is there any documentation on setting up new builders? I've never dealt with Jenkins before, and from glimpsing over the docs some time ago when Jeff Law talked about extending this GCC builders to a wider range of architectures left me completely at a loss: the whole thing felt like a total moloch with an incredible range of abilities, but little to no guidance on how to start. If the GDB CI wants to extend beyond a Linux-only range of targets, I believe considerable documentation is necessary to make this happen. Besides, I seem to have glimpsed from the Linaro instance that the builders use Docker. Is this a requirement or just a convenience? I'm asking because there's no current Docker port to Solaris (there used to be one based on zones, but it's no longer maintained) and the buildbot-based builders I'm running (for both LLVM and GDB) do fine without. Thanks. Rainer -- ----------------------------------------------------------------------------- Rainer Orth, Center for Biotechnology, Bielefeld University ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-01 9:40 ` Rainer Orth @ 2021-07-01 11:48 ` Luis Machado 2021-07-01 12:06 ` Rainer Orth 0 siblings, 1 reply; 11+ messages in thread From: Luis Machado @ 2021-07-01 11:48 UTC (permalink / raw) To: Rainer Orth, Luis Machado via Gdb Hi Rainer, On 7/1/21 6:40 AM, Rainer Orth wrote: > Hi Luis, > >> Linaro can take care of providing builders and build jobs for ARM. Other >> architectures would be handled by their respective contributors. Those >> contributors can write jobs and plug builders as needed. > > thanks for coming forward with this: this is very welcome, given how > easy it is to miss build failures and other issues especially on > not-so-common targets. > > However, is there any documentation on setting up new builders? I've > never dealt with Jenkins before, and from glimpsing over the docs some > time ago when Jeff Law talked about extending this GCC builders to a > wider range of architectures left me completely at a loss: the whole > thing felt like a total moloch with an incredible range of abilities, > but little to no guidance on how to start. If the GDB CI wants to > extend beyond a Linux-only range of targets, I believe considerable > documentation is necessary to make this happen. I know the feeling. I've been there myself as Jenkins is indeed a very flexible tool. And I don't, by any means, claim to be an expert on dealing with it. Usually what I do is to peak at the documentation and at what others have implemented. Then I work on extending/modifying things to my needs. The GDB build job (currently running for aarch64/armhf/x86_64) I put together is here: https://git.linaro.org/ci/job/configs.git/tree/tcwg-gdb.yaml But I expect we will have to adjust this to make it a bit more generic so other architectures can use it. > > Besides, I seem to have glimpsed from the Linaro instance that the > builders use Docker. Is this a requirement or just a convenience? I'm > asking because there's no current Docker port to Solaris (there used to > be one based on zones, but it's no longer maintained) and the > buildbot-based builders I'm running (for both LLVM and GDB) do fine > without. That is a convenience so we can share hardware resources. It is possible to use real hardware to run the jobs. One may need to adjust the configurations a bit (distro, packages etc), but a job can automate some of that. Details about distros to use and packages to install still need to be investigated/discussed. > > Thanks. > Rainer > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-01 11:48 ` Luis Machado @ 2021-07-01 12:06 ` Rainer Orth 2021-07-01 13:10 ` Luis Machado 0 siblings, 1 reply; 11+ messages in thread From: Rainer Orth @ 2021-07-01 12:06 UTC (permalink / raw) To: Luis Machado; +Cc: Luis Machado via Gdb Hi Luis, >>> Linaro can take care of providing builders and build jobs for ARM. Other >>> architectures would be handled by their respective contributors. Those >>> contributors can write jobs and plug builders as needed. >> thanks for coming forward with this: this is very welcome, given how >> easy it is to miss build failures and other issues especially on >> not-so-common targets. >> However, is there any documentation on setting up new builders? I've >> never dealt with Jenkins before, and from glimpsing over the docs some >> time ago when Jeff Law talked about extending this GCC builders to a >> wider range of architectures left me completely at a loss: the whole >> thing felt like a total moloch with an incredible range of abilities, >> but little to no guidance on how to start. If the GDB CI wants to >> extend beyond a Linux-only range of targets, I believe considerable >> documentation is necessary to make this happen. > > I know the feeling. I've been there myself as Jenkins is indeed a very > flexible tool. And I don't, by any means, claim to be an expert on > dealing with it. > > Usually what I do is to peak at the documentation and at what others have > implemented. Then I work on extending/modifying things to my needs. same here. In the case of the GDB and LLVM buildbots there was some documentation to start from, although I'd still to figure several things out by myself. Fortunately, the buildbot docs aren't that forbidding ;-) > The GDB build job (currently running for aarch64/armhf/x86_64) I put > together is here: > > https://git.linaro.org/ci/job/configs.git/tree/tcwg-gdb.yaml Fine, thanks. > But I expect we will have to adjust this to make it a bit more generic so > other architectures can use it. I'll certainly have look. If nothing else, it's a start and way way easier than having to start from scratch. >> Besides, I seem to have glimpsed from the Linaro instance that the >> builders use Docker. Is this a requirement or just a convenience? I'm >> asking because there's no current Docker port to Solaris (there used to >> be one based on zones, but it's no longer maintained) and the >> buildbot-based builders I'm running (for both LLVM and GDB) do fine >> without. > > That is a convenience so we can share hardware resources. It is possible to > use real hardware to run the jobs. One may need to adjust the For my existing buildbots, I let them run inside Solaris zones (the equivalent of Linux containers) and could use ressource control features to provide additional containment (e.g. cpu, memory use) if need be. > configurations a bit (distro, packages etc), but a job can automate some > of that. Details about distros to use and packages to install still need to > be investigated/discussed. In my case, I start from a configuration matching what I use for manual GDB builds, afterwards keeping the system up to date about once a months. This makes the host somewhat a moving target, but the rate of chance hasn't ever caused problems. Documenting the set of necessary packages is similar to what one needs for manual GDB builds, just a bit more formalized. Rainer -- ----------------------------------------------------------------------------- Rainer Orth, Center for Biotechnology, Bielefeld University ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-01 12:06 ` Rainer Orth @ 2021-07-01 13:10 ` Luis Machado 2021-07-02 9:05 ` Rainer Orth 0 siblings, 1 reply; 11+ messages in thread From: Luis Machado @ 2021-07-01 13:10 UTC (permalink / raw) To: Rainer Orth; +Cc: Luis Machado via Gdb On 7/1/21 9:06 AM, Rainer Orth wrote: > Hi Luis, > >>>> Linaro can take care of providing builders and build jobs for ARM. Other >>>> architectures would be handled by their respective contributors. Those >>>> contributors can write jobs and plug builders as needed. >>> thanks for coming forward with this: this is very welcome, given how >>> easy it is to miss build failures and other issues especially on >>> not-so-common targets. >>> However, is there any documentation on setting up new builders? I've >>> never dealt with Jenkins before, and from glimpsing over the docs some >>> time ago when Jeff Law talked about extending this GCC builders to a >>> wider range of architectures left me completely at a loss: the whole >>> thing felt like a total moloch with an incredible range of abilities, >>> but little to no guidance on how to start. If the GDB CI wants to >>> extend beyond a Linux-only range of targets, I believe considerable >>> documentation is necessary to make this happen. >> >> I know the feeling. I've been there myself as Jenkins is indeed a very >> flexible tool. And I don't, by any means, claim to be an expert on >> dealing with it. >> >> Usually what I do is to peak at the documentation and at what others have >> implemented. Then I work on extending/modifying things to my needs. > > same here. In the case of the GDB and LLVM buildbots there was some > documentation to start from, although I'd still to figure several things > out by myself. Fortunately, the buildbot docs aren't that forbidding ;-) > >> The GDB build job (currently running for aarch64/armhf/x86_64) I put >> together is here: >> >> https://git.linaro.org/ci/job/configs.git/tree/tcwg-gdb.yaml > > Fine, thanks. > >> But I expect we will have to adjust this to make it a bit more generic so >> other architectures can use it. > > I'll certainly have look. If nothing else, it's a start and way way > easier than having to start from scratch. > >>> Besides, I seem to have glimpsed from the Linaro instance that the >>> builders use Docker. Is this a requirement or just a convenience? I'm >>> asking because there's no current Docker port to Solaris (there used to >>> be one based on zones, but it's no longer maintained) and the >>> buildbot-based builders I'm running (for both LLVM and GDB) do fine >>> without. >> >> That is a convenience so we can share hardware resources. It is possible to >> use real hardware to run the jobs. One may need to adjust the > > For my existing buildbots, I let them run inside Solaris zones (the > equivalent of Linux containers) and could use ressource control features > to provide additional containment (e.g. cpu, memory use) if need be. > That's good. One other benefit of using docker images is the consistency between runs. You have control over the exact distro + set of packages that are installed in the image, so you have a better chance of being able to repeat a run. >> configurations a bit (distro, packages etc), but a job can automate some >> of that. Details about distros to use and packages to install still need to >> be investigated/discussed. > > In my case, I start from a configuration matching what I use for manual > GDB builds, afterwards keeping the system up to date about once a > months. This makes the host somewhat a moving target, but the rate of > chance hasn't ever caused problems. I think that's reasonable. Regular updates shouldn't cause breakage to GDB. If they do, that's a sign that something was/got broken anyway. > > Documenting the set of necessary packages is similar to what one needs > for manual GDB builds, just a bit more formalized. We maintain a set of required packages in a separate file. That file gets loaded and processed so we are sure all the dependencies are met. So coming up with a similar non-docker-based mechanism shouldn't be hard. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-01 13:10 ` Luis Machado @ 2021-07-02 9:05 ` Rainer Orth 0 siblings, 0 replies; 11+ messages in thread From: Rainer Orth @ 2021-07-02 9:05 UTC (permalink / raw) To: Luis Machado; +Cc: Luis Machado via Gdb Hi Luis, >>>> Besides, I seem to have glimpsed from the Linaro instance that the >>>> builders use Docker. Is this a requirement or just a convenience? I'm >>>> asking because there's no current Docker port to Solaris (there used to >>>> be one based on zones, but it's no longer maintained) and the >>>> buildbot-based builders I'm running (for both LLVM and GDB) do fine >>>> without. >>> >>> That is a convenience so we can share hardware resources. It is possible to >>> use real hardware to run the jobs. One may need to adjust the >> For my existing buildbots, I let them run inside Solaris zones (the >> equivalent of Linux containers) and could use ressource control features >> to provide additional containment (e.g. cpu, memory use) if need be. >> > > That's good. One other benefit of using docker images is the consistency > between runs. You have control over the exact distro + set of packages > that are installed in the image, so you have a better chance of being able > to repeat a run. understood. You can achieve somethine similar when installing Solaris zones using particular profiles. IIUC that's what had been used for the Solaris port of Docker. >>> configurations a bit (distro, packages etc), but a job can automate some >>> of that. Details about distros to use and packages to install still need to >>> be investigated/discussed. >> In my case, I start from a configuration matching what I use for manual >> GDB builds, afterwards keeping the system up to date about once a >> months. This makes the host somewhat a moving target, but the rate of >> chance hasn't ever caused problems. > > I think that's reasonable. Regular updates shouldn't cause breakage to > GDB. If they do, that's a sign that something was/got broken anyway. Exactly. Besides, I've got good contacts to Solaris engineering in case I cannot figure out what's wrong by myself. >> Documenting the set of necessary packages is similar to what one needs >> for manual GDB builds, just a bit more formalized. > > We maintain a set of required packages in a separate file. That file gets > loaded and processed so we are sure all the dependencies are met. So coming > up with a similar non-docker-based mechanism shouldn't be hard. Agreed. I'll certainly try that once we get there. Rainer -- ----------------------------------------------------------------------------- Rainer Orth, Center for Biotechnology, Bielefeld University ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-06-30 16:46 [RFC] Proposal for hosting GDB CI builds Luis Machado 2021-07-01 9:40 ` Rainer Orth @ 2021-07-01 11:37 ` Christophe LYON 2021-07-01 11:50 ` Luis Machado 2021-07-20 15:21 ` Luis Machado 2 siblings, 1 reply; 11+ messages in thread From: Christophe LYON @ 2021-07-01 11:37 UTC (permalink / raw) To: Luis Machado, gdb Hi Luis, On 30/06/2021 18:46, Luis Machado via Gdb wrote: > Hi, > > This proposal comes as an attempt to revive the GDB CI builds, given > the previous one (buildbot) is no longer being maintained by Sergio > (thanks a lot for keeping it working for all these years by the way). > > CI GDB builds are a great help for spotting regressions without having > to do the tedious and time-consuming work of running the GDB testsuite > for each git revision, for each architecture and reading through > hundreds of lines of summaries. If a regression is spotted, then one > needs to bisect to find the culprit. This isn't great, specially for > architectures without great availability of desktop hardware. > > Ideally, for each commit, we'd run full builds to validate the state > of the tree, but we're not quite there yet. So meanwhile, having some > level of automation to get the builds done without manual intervention > sounds like a reasonable step forward. > > From previous IRC conversations, it seems to be a consensus that > availability of processing power is not a problem. It is reasonably > easy to find hardware to do some builds. The most difficult resource > to find is manpower to setup the CI infrastructure and keep it running. > > With the above said, I've discussed this internally at Linaro and we > can spare some manpower to setup and maintain an isolated > Linaro-hosted Jenkins instance for GDB CI builds. As you could guess, you have a big +1 from me ;-) > > Linaro can take care of providing builders and build jobs for ARM. > Other architectures would be handled by their respective contributors. > Those contributors can write jobs and plug builders as needed. > > Setting up new jobs doesn't require the use of the web interface. It > can be done with yaml files in a git repo. It is reasonably simple. > > You can see an example of the Linaro CI here: https://ci.linaro.org/ > > Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/ > > And also the summaries for GDB testsuite runs: > https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html > > Of course, this effort only makes sense if the community is OK with > using Jenkins as the CI mechanism and if it actually sees value in > having a system like this in place. > > Thoughts? Questions? From the reports you shared above, it seems there are lots of tests with "random" results (or rather lots of differences between two runs). My fear is that reports will soon get ignored if there are too many false regressions. But I hope that making such variations public will help fix the root cause. Thanks, Christophe ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-01 11:37 ` Christophe LYON @ 2021-07-01 11:50 ` Luis Machado 2021-07-01 12:08 ` Rainer Orth 0 siblings, 1 reply; 11+ messages in thread From: Luis Machado @ 2021-07-01 11:50 UTC (permalink / raw) To: Christophe LYON, gdb Hi Christophe, On 7/1/21 8:37 AM, Christophe LYON wrote: > Hi Luis, > > On 30/06/2021 18:46, Luis Machado via Gdb wrote: >> Hi, >> >> This proposal comes as an attempt to revive the GDB CI builds, given >> the previous one (buildbot) is no longer being maintained by Sergio >> (thanks a lot for keeping it working for all these years by the way). >> >> CI GDB builds are a great help for spotting regressions without having >> to do the tedious and time-consuming work of running the GDB testsuite >> for each git revision, for each architecture and reading through >> hundreds of lines of summaries. If a regression is spotted, then one >> needs to bisect to find the culprit. This isn't great, specially for >> architectures without great availability of desktop hardware. >> >> Ideally, for each commit, we'd run full builds to validate the state >> of the tree, but we're not quite there yet. So meanwhile, having some >> level of automation to get the builds done without manual intervention >> sounds like a reasonable step forward. >> >> From previous IRC conversations, it seems to be a consensus that >> availability of processing power is not a problem. It is reasonably >> easy to find hardware to do some builds. The most difficult resource >> to find is manpower to setup the CI infrastructure and keep it running. >> >> With the above said, I've discussed this internally at Linaro and we >> can spare some manpower to setup and maintain an isolated >> Linaro-hosted Jenkins instance for GDB CI builds. > > > As you could guess, you have a big +1 from me ;-) > > >> >> Linaro can take care of providing builders and build jobs for ARM. >> Other architectures would be handled by their respective contributors. >> Those contributors can write jobs and plug builders as needed. >> >> Setting up new jobs doesn't require the use of the web interface. It >> can be done with yaml files in a git repo. It is reasonably simple. >> >> You can see an example of the Linaro CI here: https://ci.linaro.org/ >> >> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/ >> >> And also the summaries for GDB testsuite runs: >> https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html >> >> >> Of course, this effort only makes sense if the community is OK with >> using Jenkins as the CI mechanism and if it actually sees value in >> having a system like this in place. >> >> Thoughts? Questions? > > From the reports you shared above, it seems there are lots of tests > with "random" results (or rather lots of differences between two runs). > My fear is that reports will soon get ignored if there are too many > false regressions. But I hope that making such variations public will > help fix the root cause. Unfortunately that's always been the nature of GDB's testsuite. It has improved over the year, but there are still problematic tests. We will certainly have to put those into an exception list when reporting the summaries, at least at an initial stage. > > Thanks, > > > Christophe > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-01 11:50 ` Luis Machado @ 2021-07-01 12:08 ` Rainer Orth 0 siblings, 0 replies; 11+ messages in thread From: Rainer Orth @ 2021-07-01 12:08 UTC (permalink / raw) To: Luis Machado via Gdb Hi Luis, >> From the reports you shared above, it seems there are lots of tests >> with "random" results (or rather lots of differences between two >> runs). My fear is that reports will soon get ignored if there are too >> many false regressions. But I hope that making such variations public >> will help fix the root cause. > > Unfortunately that's always been the nature of GDB's testsuite. It has > improved over the year, but there are still problematic tests. We will > certainly have to put those into an exception list when reporting the > summaries, at least at an initial stage. on the Solaris side, things are considerably worse: there are currently 2000+ FAILs for a GDB testsuite run, with considerable variation between reruns. I've never got a grip on what's going on, unfortunately, and in the end turned the Solaris GDB buildbots to be build-only so I'd at least catch build failures early. Rainer -- ----------------------------------------------------------------------------- Rainer Orth, Center for Biotechnology, Bielefeld University ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-06-30 16:46 [RFC] Proposal for hosting GDB CI builds Luis Machado 2021-07-01 9:40 ` Rainer Orth 2021-07-01 11:37 ` Christophe LYON @ 2021-07-20 15:21 ` Luis Machado 2021-07-23 20:17 ` Simon Marchi 2 siblings, 1 reply; 11+ messages in thread From: Luis Machado @ 2021-07-20 15:21 UTC (permalink / raw) To: gdb Any other feedback on this? On 6/30/21 1:46 PM, Luis Machado wrote: > Hi, > > This proposal comes as an attempt to revive the GDB CI builds, given the > previous one (buildbot) is no longer being maintained by Sergio (thanks > a lot for keeping it working for all these years by the way). > > CI GDB builds are a great help for spotting regressions without having > to do the tedious and time-consuming work of running the GDB testsuite > for each git revision, for each architecture and reading through > hundreds of lines of summaries. If a regression is spotted, then one > needs to bisect to find the culprit. This isn't great, specially for > architectures without great availability of desktop hardware. > > Ideally, for each commit, we'd run full builds to validate the state of > the tree, but we're not quite there yet. So meanwhile, having some level > of automation to get the builds done without manual intervention sounds > like a reasonable step forward. > > From previous IRC conversations, it seems to be a consensus that > availability of processing power is not a problem. It is reasonably easy > to find hardware to do some builds. The most difficult resource to find > is manpower to setup the CI infrastructure and keep it running. > > With the above said, I've discussed this internally at Linaro and we can > spare some manpower to setup and maintain an isolated Linaro-hosted > Jenkins instance for GDB CI builds. > > Linaro can take care of providing builders and build jobs for ARM. Other > architectures would be handled by their respective contributors. Those > contributors can write jobs and plug builders as needed. > > Setting up new jobs doesn't require the use of the web interface. It can > be done with yaml files in a git repo. It is reasonably simple. > > You can see an example of the Linaro CI here: https://ci.linaro.org/ > > Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/ > > And also the summaries for GDB testsuite runs: > https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html > > > Of course, this effort only makes sense if the community is OK with > using Jenkins as the CI mechanism and if it actually sees value in > having a system like this in place. > > Thoughts? Questions? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Proposal for hosting GDB CI builds 2021-07-20 15:21 ` Luis Machado @ 2021-07-23 20:17 ` Simon Marchi 0 siblings, 0 replies; 11+ messages in thread From: Simon Marchi @ 2021-07-23 20:17 UTC (permalink / raw) To: Luis Machado, gdb On 2021-07-20 11:21 a.m., Luis Machado wrote: > Any other feedback on this? Sorry, I procrastinated replying to this thread for a while - too many things to do. > On 6/30/21 1:46 PM, Luis Machado wrote: >> Hi, >> >> This proposal comes as an attempt to revive the GDB CI builds, given the previous one (buildbot) is no longer being maintained by Sergio (thanks a lot for keeping it working for all these years by the way). >> >> CI GDB builds are a great help for spotting regressions without having to do the tedious and time-consuming work of running the GDB testsuite for each git revision, for each architecture and reading through hundreds of lines of summaries. If a regression is spotted, then one needs to bisect to find the culprit. This isn't great, specially for architectures without great availability of desktop hardware. >> >> Ideally, for each commit, we'd run full builds to validate the state of the tree, but we're not quite there yet. So meanwhile, having some level of automation to get the builds done without manual intervention sounds like a reasonable step forward. >> >> From previous IRC conversations, it seems to be a consensus that availability of processing power is not a problem. It is reasonably easy to find hardware to do some builds. The most difficult resource to find is manpower to setup the CI infrastructure and keep it running. I agree, it's almost a full time job to babysit a CI. >> With the above said, I've discussed this internally at Linaro and we can spare some manpower to setup and maintain an isolated Linaro-hosted Jenkins instance for GDB CI builds. >> >> Linaro can take care of providing builders and build jobs for ARM. Other architectures would be handled by their respective contributors. Those contributors can write jobs and plug builders as needed. >> >> Setting up new jobs doesn't require the use of the web interface. It can be done with yaml files in a git repo. It is reasonably simple. >> >> You can see an example of the Linaro CI here: https://ci.linaro.org/ >> >> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/ >> >> And also the summaries for GDB testsuite runs: https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html The reports look really good. When I looked into setting up a CI, the most difficult / annoying part was analyzing the results. Not only the racy tests that intermittently fail, but also some tests that alternate between PASS and KFAIL (I don't remember which exactly, I think the "many short lived threads" one). The previous buildbot had a racy test analysis to detemine which tests were flaky, and (I suppose) ignore them in the results. I don't know how well that worked. But if that makes it simpler, I'd be ok with an hand-maintained list of known flaky/racy tests that should be ignored when comparing the results. Ignoring some test results is not great, but if it can make the CI's results more trustworthy, I say it's better than letting the racy/flaky tests "contaminate" the results of all the tests that aren't. In the end, we want a CI with results we can trust. With the racy/flaky tests, it often ends up that builds fail but "oh it's just that test failing, it's fine". And this is how regressions can slip in. >> Of course, this effort only makes sense if the community is OK with using Jenkins as the CI mechanism and if it actually sees value in having a system like this in place. I am of course in favor of it. And in my opinion, whoever makes the effort has the final word on how it's done. So if you are used to Jenkins, let's go with Jenkins. Simon ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-07-23 20:17 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-06-30 16:46 [RFC] Proposal for hosting GDB CI builds Luis Machado 2021-07-01 9:40 ` Rainer Orth 2021-07-01 11:48 ` Luis Machado 2021-07-01 12:06 ` Rainer Orth 2021-07-01 13:10 ` Luis Machado 2021-07-02 9:05 ` Rainer Orth 2021-07-01 11:37 ` Christophe LYON 2021-07-01 11:50 ` Luis Machado 2021-07-01 12:08 ` Rainer Orth 2021-07-20 15:21 ` Luis Machado 2021-07-23 20:17 ` Simon Marchi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).