Re: Testing Setup - More Tests and Automation?

[v <vsochat@gmail.com>]

Going to give this one more shot in case it works without the image (and I
don't get flagged as spam...)

if you didn't see my attached message below Ben's, please unwrap the email
and look! 👇

Also Dodji I heard from a little bird that you are competing in a national
championship this weekend? Kick @ss!! Just forget about work until then
because being a national champion (or even contender) is so much cooler :)

On Wed, Apr 27, 2022 at 6:55 PM Ben Woodard <woodard@redhat.com> wrote:

> My coworker who did most of the work included in:
> https://sourceware.org/pipermail/libabigail/2022q2/004292.html <which
> sadly got scrubbed due to a bug in Fedora 36> asked me to forward this
> email along to the mailing list because her posts are getting marked as
> spam.
>
> I personally feel like we need to reconsider
> https://sourceware.org/pipermail/libabigail/2022q1/004045.html and have
> checks like this built into the normal “make check” having
> ENABLE_SLOW_TEST=no by default allows too many bugs to creep in. People
> don’t use it. As a case in point neither the current trunk or the fixes
> branch pass all the tests when it is set.
>
> I will say Vanessa has convinced me of the value of the automated testing.
> It is great when she does a pull request, I can immediately see not only
> that it applies but that it continues to pass all the regression tests. I
> feel like it would be helpful to have more tests included within the make
> check so that it didn’t depend on me to make sure that something isn’t
> going wrong.
>
> -ben
>
>
> > Begin forwarded message:
> >
> > From: v <vsochat@gmail.com>
> > Subject: Fwd: Testing Setup - More Tests and Automation?
> > Date: April 27, 2022 at 4:42:09 PM PDT
> > To: woodard@redhat.com
> >
> > See my email below - I joined the sourceware list and it blocked me as
> spam. 😭
> >
> > ---------- Forwarded message ---------
> > From: v <vsochat@gmail.com>
> > Date: Wed, Apr 27, 2022 at 5:40 PM
> > Subject: Testing Setup - More Tests and Automation?
> > To: <libabigail@sourceware.org>
> >
> >
> > Hey Dodji et al,
> >
> > I am writing in response to
> https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I
> understand the desire to not add too many tests to make check, but I think
> maybe there is a nice middle ground or compromise?
> >
> > Basically, what I think libabigail needs is more automated testing of
> more cases. I added a simple setup to Ben's repo that does an automated
> build in a container, and since it starts with a base container with the
> deps it's not terribly too slow:
> >
> >
> https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true
> >
> > In that run we do make check (Ben had added extra tests there) but it
> would also be feasible to clone a testing repository instead. This is what
> I set up for dyninst, because their test suite is a bit of a chonker too
> (note the red clone into external tests).
> >
> >
> https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> >
> > Also, it would be pretty cool to run libabigail on itself to check for
> breaks:
> >
> > https://github.com/woodard/libabigail/actions/runs/2210184567
> >
> > And (kind of cool) to always run the changes in the PR against a bunch
> of known distro libs (here for Fedora).
> >
> > https://github.com/woodard/libabigail/actions/runs/2208891579
> >
> > So, I think what I'm saying is that adding automation to testing
> libabigail would really empower us to catch more bugs. Have y'all ever
> considered moving some project stuffs over to GitHub so more people can
> help with libaibgail than just you and Ben? And we can implement a lot of
> automation for the project proper? I gave a talk recently at CU Boulder,
> and the students were really interested in libabigail. If it's readily
> available to contribute to on GitHub, I (and others I bet) would really
> enjoy contributing. I almost didn't write this email because it's a PITA to
> have to join a mailing list, but I think it's important. If you forever
> keep it on sourceware (channeling 1993 they want their web design back!) I
> think in the long run it's more work for you, and probably less fun too.
> >
> > What do you think? Is this a future you could imagine? And how might we
> get there?
> >
> > Best,
> >
> > Vanessa
>
>