Fwd: Testing Setup - More Tests and Automation?

[Ben Woodard <woodard@redhat.com>]

My coworker who did most of the work included in: https://sourceware.org/pipermail/libabigail/2022q2/004292.html <which sadly got scrubbed due to a bug in Fedora 36> asked me to forward this email along to the mailing list because her posts are getting marked as spam.

I personally feel like we need to reconsider https://sourceware.org/pipermail/libabigail/2022q1/004045.html and have checks like this built into the normal “make check” having ENABLE_SLOW_TEST=no by default allows too many bugs to creep in. People don’t use it. As a case in point neither the current trunk or the fixes branch pass all the tests when it is set. 

I will say Vanessa has convinced me of the value of the automated testing. It is great when she does a pull request, I can immediately see not only that it applies but that it continues to pass all the regression tests. I feel like it would be helpful to have more tests included within the make check so that it didn’t depend on me to make sure that something isn’t going wrong.

-ben


> Begin forwarded message:
> 
> From: v <vsochat@gmail.com>
> Subject: Fwd: Testing Setup - More Tests and Automation?
> Date: April 27, 2022 at 4:42:09 PM PDT
> To: woodard@redhat.com
> 
> See my email below - I joined the sourceware list and it blocked me as spam. 😭
> 
> ---------- Forwarded message ---------
> From: v <vsochat@gmail.com>
> Date: Wed, Apr 27, 2022 at 5:40 PM
> Subject: Testing Setup - More Tests and Automation?
> To: <libabigail@sourceware.org>
> 
> 
> Hey Dodji et al,
> 
> I am writing in response to https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I understand the desire to not add too many tests to make check, but I think maybe there is a nice middle ground or compromise?
> 
> Basically, what I think libabigail needs is more automated testing of more cases. I added a simple setup to Ben's repo that does an automated build in a container, and since it starts with a base container with the deps it's not terribly too slow:
> 
> https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true
> 
> In that run we do make check (Ben had added extra tests there) but it would also be feasible to clone a testing repository instead. This is what I set up for dyninst, because their test suite is a bit of a chonker too (note the red clone into external tests). 
> 
> https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> 
> Also, it would be pretty cool to run libabigail on itself to check for breaks:
> 
> https://github.com/woodard/libabigail/actions/runs/2210184567
> 
> And (kind of cool) to always run the changes in the PR against a bunch of known distro libs (here for Fedora).
> 
> https://github.com/woodard/libabigail/actions/runs/2208891579
> 
> So, I think what I'm saying is that adding automation to testing libabigail would really empower us to catch more bugs. Have y'all ever considered moving some project stuffs over to GitHub so more people can help with libaibgail than just you and Ben? And we can implement a lot of automation for the project proper? I gave a talk recently at CU Boulder, and the students were really interested in libabigail. If it's readily available to contribute to on GitHub, I (and others I bet) would really enjoy contributing. I almost didn't write this email because it's a PITA to have to join a mailing list, but I think it's important. If you forever keep it on sourceware (channeling 1993 they want their web design back!) I think in the long run it's more work for you, and probably less fun too.
> 
> What do you think? Is this a future you could imagine? And how might we get there?
> 
> Best,
> 
> Vanessa