From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.polymtl.ca (smtp.polymtl.ca [132.207.4.11]) by sourceware.org (Postfix) with ESMTPS id 85B08388E800 for ; Fri, 23 Jul 2021 20:17:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 85B08388E800 Received: from simark.ca (simark.ca [158.69.221.121]) (authenticated bits=0) by smtp.polymtl.ca (8.14.7/8.14.7) with ESMTP id 16NKHHI2014136 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Jul 2021 16:17:22 -0400 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp.polymtl.ca 16NKHHI2014136 Received: from [10.0.0.11] (192-222-157-6.qc.cable.ebox.net [192.222.157.6]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by simark.ca (Postfix) with ESMTPSA id 5BEC71E4A3; Fri, 23 Jul 2021 16:17:17 -0400 (EDT) Subject: Re: [RFC] Proposal for hosting GDB CI builds To: Luis Machado , "gdb@sourceware.org" References: <7bfae273-3887-30c8-dc65-94d5b177db56@linaro.org> <519a40f4-5999-f338-227e-464ed7ce23be@linaro.org> From: Simon Marchi Message-ID: <42f9651a-8a44-f961-edc5-e742ac83d7de@polymtl.ca> Date: Fri, 23 Jul 2021 16:17:17 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: <519a40f4-5999-f338-227e-464ed7ce23be@linaro.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Poly-FromMTA: (simark.ca [158.69.221.121]) at Fri, 23 Jul 2021 20:17:17 +0000 X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jul 2021 20:17:25 -0000 On 2021-07-20 11:21 a.m., Luis Machado wrote: > Any other feedback on this? Sorry, I procrastinated replying to this thread for a while - too many things to do. > On 6/30/21 1:46 PM, Luis Machado wrote: >> Hi, >> >> This proposal comes as an attempt to revive the GDB CI builds, given the previous one (buildbot) is no longer being maintained by Sergio (thanks a lot for keeping it working for all these years by the way). >> >> CI GDB builds are a great help for spotting regressions without having to do the tedious and time-consuming work of running the GDB testsuite for each git revision, for each architecture and reading through hundreds of lines of summaries. If a regression is spotted, then one needs to bisect to find the culprit. This isn't great, specially for architectures without great availability of desktop hardware. >> >> Ideally, for each commit, we'd run full builds to validate the state of the tree, but we're not quite there yet. So meanwhile, having some level of automation to get the builds done without manual intervention sounds like a reasonable step forward. >> >> From previous IRC conversations, it seems to be a consensus that availability of processing power is not a problem. It is reasonably easy to find hardware to do some builds. The most difficult resource to find is manpower to setup the CI infrastructure and keep it running. I agree, it's almost a full time job to babysit a CI. >> With the above said, I've discussed this internally at Linaro and we can spare some manpower to setup and maintain an isolated Linaro-hosted Jenkins instance for GDB CI builds. >> >> Linaro can take care of providing builders and build jobs for ARM. Other architectures would be handled by their respective contributors. Those contributors can write jobs and plug builders as needed. >> >> Setting up new jobs doesn't require the use of the web interface. It can be done with yaml files in a git repo. It is reasonably simple. >> >> You can see an example of the Linaro CI here: https://ci.linaro.org/ >> >> Also, a GDB job for aarch64: https://ci.linaro.org/job/tcwg-gdb/ >> >> And also the summaries for GDB testsuite runs: https://ci.linaro.org/job/tcwg-compare-results/13968/artifact/artifacts/logs/0-report.html The reports look really good. When I looked into setting up a CI, the most difficult / annoying part was analyzing the results. Not only the racy tests that intermittently fail, but also some tests that alternate between PASS and KFAIL (I don't remember which exactly, I think the "many short lived threads" one). The previous buildbot had a racy test analysis to detemine which tests were flaky, and (I suppose) ignore them in the results. I don't know how well that worked. But if that makes it simpler, I'd be ok with an hand-maintained list of known flaky/racy tests that should be ignored when comparing the results. Ignoring some test results is not great, but if it can make the CI's results more trustworthy, I say it's better than letting the racy/flaky tests "contaminate" the results of all the tests that aren't. In the end, we want a CI with results we can trust. With the racy/flaky tests, it often ends up that builds fail but "oh it's just that test failing, it's fine". And this is how regressions can slip in. >> Of course, this effort only makes sense if the community is OK with using Jenkins as the CI mechanism and if it actually sees value in having a system like this in place. I am of course in favor of it. And in my opinion, whoever makes the effort has the final word on how it's done. So if you are used to Jenkins, let's go with Jenkins. Simon