From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id B45593858D28 for ; Sat, 30 Apr 2022 19:55:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B45593858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: by gnu.wildebeest.org (Postfix, from userid 1000) id 75DC7302BBED; Sat, 30 Apr 2022 21:55:07 +0200 (CEST) Date: Sat, 30 Apr 2022 21:55:07 +0200 From: Mark Wielaard To: v Cc: Ben Woodard , Ben Woodard via Libabigail Subject: Re: Testing Setup - More Tests and Automation? Message-ID: <20220430195507.GA11996@gnu.wildebeest.org> References: <34E58964-E930-4DF9-87CD-18D4C63DBCEB@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, KAM_SHORT, LIKELY_SPAM_BODY, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Apr 2022 19:55:11 -0000 Hi Vanessa, On Wed, Apr 27, 2022 at 07:57:12PM -0600, v via Libabigail wrote: > Going to give this one more shot in case it works without the image (and I > don't get flagged as spam...) Sorry I missed your email earlier. It might be that your original email was HTML encoded which gets rejected by the mailinglist. > > > I am writing in response to > > https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I > > understand the desire to not add too many tests to make check, but I think > > maybe there is a nice middle ground or compromise? > > > > > > Basically, what I think libabigail needs is more automated testing of > > more cases. Agreed that would be nice. And agreed that the SLOW_TESTS should be tested by default on the buildbot. Ben already made that change and it even caught an issue on ppc64. We might not be advertising the buildbot enough. It recently moved to sourceware and got "badges", see https://builder.sourceware.org/ Maybe the libabigail badges/links should be added to the libabigail homepage. > > I added a simple setup to Ben's repo that does an automated > > build in a container, and since it starts with a base container with the > > deps it's not terribly too slow: > > > > > https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true Note that the logs appear to not be accessible unless you have a github account and are logged in. Could you make them public? > > > In that run we do make check (Ben had added extra tests there) but it > > would also be feasible to clone a testing repository instead. This is what > > I set up for dyninst, because their test suite is a bit of a chonker too > > (note the red clone into external tests). > > > > > > > > https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78 > > > > > > Also, it would be pretty cool to run libabigail on itself to check for > > breaks: > > > > > > https://github.com/woodard/libabigail/actions/runs/2210184567 > > > > > > And (kind of cool) to always run the changes in the PR against a bunch > > of known distro libs (here for Fedora). > > > > > > https://github.com/woodard/libabigail/actions/runs/2208891579 Although I can imagine what this does and that it is useful all these URLs have the same problem, they don't actually show what is happening because the logs are not public. Could you post the scripts and container files somewhere so we can see how to incorporate this? I believe Dodji is working a libabigail-tests project/setup so that larger tests can be added without having them all in the main source repo. > > So, I think what I'm saying is that adding automation to testing > > libabigail would really empower us to catch more bugs. I agree. We do have some automated testing. But what we don't have yet is automated pre-commit testing. I hope to provide that through the buildbot using a try-builder (maybe combined with patchwork). > > Have y'all ever > > considered moving some project stuffs over to GitHub so more people can > > help with libaibgail than just you and Ben? And we can implement a lot of > > automation for the project proper? I gave a talk recently at CU Boulder, > > and the students were really interested in libabigail. If it's readily > > available to contribute to on GitHub, I (and others I bet) would really > > enjoy contributing. I almost didn't write this email because it's a PITA to > > have to join a mailing list, but I think it's important. If you forever > > keep it on sourceware (channeling 1993 they want their web design back!) I > > think in the long run it's more work for you, and probably less fun too. > > > > > > What do you think? Is this a future you could imagine? And how might we > > get there? I think having more automation and more testing would be great. I can certainly imagine a future where we have that. But I hope we can avoid github to get here. Personally I try to avoid using proprietary software to create free software. And just to create an account on github seems to require running a lot of proprietary javascript and accepting 30+ pages of legal text. Note that if you like a "forge" to contribute to sourceware hosted projects then you can use the sourcehut mirror: https://sr.ht/~sourceware/libabigail/ sourcehut also allows you to submit patches through the webinterface (which then sents properly formatted git email patches to this list). Note that you don't have to be subscribed to this list to post (but HTML emails are rejected). I don't know how the libabigail website is generated, but it would be nice to get that also in git so people can update it if they feel they have a better design. Cheers, Mark