Re: Testing Setup - More Tests and Automation?

[Mark Wielaard <mark@klomp.org>]

Hi Vanessa,

On Wed, Apr 27, 2022 at 07:57:12PM -0600, v via Libabigail wrote:
> Going to give this one more shot in case it works without the image (and I
> don't get flagged as spam...)

Sorry I missed your email earlier. It might be that your original
email was HTML encoded which gets rejected by the mailinglist.

> > > I am writing in response to
> > https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I
> > understand the desire to not add too many tests to make check, but I think
> > maybe there is a nice middle ground or compromise?
> > >
> > > Basically, what I think libabigail needs is more automated testing of
> > more cases.

Agreed that would be nice. And agreed that the SLOW_TESTS should be
tested by default on the buildbot. Ben already made that change and it
even caught an issue on ppc64. We might not be advertising the
buildbot enough. It recently moved to sourceware and got "badges", see
https://builder.sourceware.org/ Maybe the libabigail badges/links
should be added to the libabigail homepage.

> > I added a simple setup to Ben's repo that does an automated
> > build in a container, and since it starts with a base container with the
> > deps it's not terribly too slow:
> > >
> > https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true

Note that the logs appear to not be accessible unless you have a
github account and are logged in. Could you make them public?

> > > In that run we do make check (Ben had added extra tests there) but it
> > would also be feasible to clone a testing repository instead. This is what
> > I set up for dyninst, because their test suite is a bit of a chonker too
> > (note the red clone into external tests).
> > >
> > >
> > https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> > >
> > > Also, it would be pretty cool to run libabigail on itself to check for
> > breaks:
> > >
> > > https://github.com/woodard/libabigail/actions/runs/2210184567
> > >
> > > And (kind of cool) to always run the changes in the PR against a bunch
> > of known distro libs (here for Fedora).
> > >
> > > https://github.com/woodard/libabigail/actions/runs/2208891579

Although I can imagine what this does and that it is useful all these
URLs have the same problem, they don't actually show what is happening
because the logs are not public.

Could you post the scripts and container files somewhere so we can see
how to incorporate this? I believe Dodji is working a libabigail-tests
project/setup so that larger tests can be added without having them all
in the main source repo.

> > So, I think what I'm saying is that adding automation to testing
> > libabigail would really empower us to catch more bugs.

I agree. We do have some automated testing. But what we don't have yet
is automated pre-commit testing. I hope to provide that through the
buildbot using a try-builder (maybe combined with patchwork).

> >  Have y'all ever
> > considered moving some project stuffs over to GitHub so more people can
> > help with libaibgail than just you and Ben? And we can implement a lot of
> > automation for the project proper? I gave a talk recently at CU Boulder,
> > and the students were really interested in libabigail. If it's readily
> > available to contribute to on GitHub, I (and others I bet) would really
> > enjoy contributing. I almost didn't write this email because it's a PITA to
> > have to join a mailing list, but I think it's important. If you forever
> > keep it on sourceware (channeling 1993 they want their web design back!) I
> > think in the long run it's more work for you, and probably less fun too.
> > >
> > > What do you think? Is this a future you could imagine? And how might we
> > get there?

I think having more automation and more testing would be great. I can
certainly imagine a future where we have that. But I hope we can avoid
github to get here. Personally I try to avoid using proprietary
software to create free software. And just to create an account on
github seems to require running a lot of proprietary javascript and
accepting 30+ pages of legal text.

Note that if you like a "forge" to contribute to sourceware hosted
projects then you can use the sourcehut mirror:
https://sr.ht/~sourceware/libabigail/
sourcehut also allows you to submit patches through the webinterface
(which then sents properly formatted git email patches to this list).

Note that you don't have to be subscribed to this list to post (but
HTML emails are rejected).

I don't know how the libabigail website is generated, but it would be
nice to get that also in git so people can update it if they feel they
have a better design.

Cheers,

Mark