Re: Testing Setup - More Tests and Automation?

[v <vsochat@gmail.com>]

Hi Mark,

The links are private because this is Ben's personal repository for his
account - given that libabigail was developed under a proper organization
it would be public. You can look at example OSS projects (e.g., tensorflow
https://github.com/tensorflow/tensorflow/runs/6241307726?check_suite_focus=true)
to get a sense of what the UI looks like.

I understand the desire to not use proprietary software, but if you needed
to move the setup elsewhere that's not hard to do! Modern software
developers use GitHub, and all major companies (including RedHat) grow
strong communities of developers on there. Podman is a good example:
https://github.com/containers/podman. Git is git, so you could develop on
GitHub and keep a backup on sourceware, and have an automated workflow to
do that. In other words, there are ways to have your cake and eat it too.
Let's take a look at Libabigail. From Ben's contributor graph:
https://github.com/woodard/libabigail/graphs/contributors

Or from git:

  1835  Dodji Seketeli <dodji@redhat.com>
   144  Giuliano Procida <gprocida@google.com>
   135  Matthias Maennich <maennich@google.com>
    35  Benjamin Kosnik <bkoz@redhat.com>
    33  Dodji Seketeli <dodji@seketeli.org>
    19  Chenxiong Qi <cqi@redhat.com>
    19  Mark Wielaard <mark@klomp.org>
    19  Ondrej Oprala <ooprala@redhat.com>
    15  Sinny Kumari <sinny@redhat.com>
    13  Thomas Schwinge <thomas@codesourcery.com>
    11  Ondrej Oprala <ondrej.oprala@gmail.com>
     9  Jan Engelhardt <jengelh@inai.de>
     8  Guillermo E. Martinez <guillermo.e.martinez@oracle.com>
     8  tangmeng <tangmeng@uniontech.com>
     7  Ben Woodard <woodard@redhat.com>
     7  Jonathan Wakely <jwakely@redhat.com>
     7  Jose E. Marchesi <jose.marchesi@oracle.com>
     5  Mark Wielaard <mjw@redhat.com>
     4  Sinny Kumari <skumari@redhat.com>
     4  maennich@google.com <maennich@google.com>
     2  Matthias Klose <doko@debian.org>
     1  David Cantrell <dcantrell@redhat.com>
     1  David Seifert <soap@gentoo.org>
     1  Dodji <dodji@ks305400.kimsufi.com>
     1  George Rawlinson <grawlinson@archlinux.org>
     1  Jessica Yu <jeyu@kernel.org>
     1  Randy MacLeod <Randy.MacLeod@windriver.com>
     1  Roland McGrath <roland@hack.frob.com>
     1  Slava Barinov <v.barinov@samsung.com>
     1  Vanessa Sochat <sochat1@llnl.gov>
     1  Xiao Jia <xiaoj@google.com>

You have *eighteen* contributors with over 5 commits in a project that is 8
years old, and that includes repeated individuals using different
alias/email. Out of those 18, it looks like there are maybe 3 that have
contributed meaningfully (and recently), and talking with Ben (and seeing
number of commits), I get a sense that the main developer is Dodji and he
is overwhelmed. So (to me) for a library that is as important as
libabigail, this is problematic. The bus factor is really high. Compare
that to Podman https://github.com/containers/podman/graphs/contributors,
which is also a niche technology / area of work, and the project has only
been around since late 2017 (and I don't know when it went up on GitHub
from perhaps being private at RedHat) and it has almost 400 contributors
(just over 100 with >= 5 commits) and many others that haven't contributed
directly to the code (myself included) but have contributed via issues and
other interaction.

So (to me) this is a problem that needs to be solved, and I don't think
keeping things the way that they are is that solution, regardless of how
you innovate a bit with adding automation. The cost of not changing your
current setup is very high, and I worry about the future of libabigail. As
a developer I want to contribute and champion the work, but in its current
state I find the project feels very closed and not trying bring me in. The
students I spoke to at CU Boulder were inspired, but they will stop their
exploration when they realize it's not easy to ask questions or for help.
In other words, you are 100% losing out in terms of new contributors and
developers by not using these modern practices and services.  That is of
course your choice (and I respect the choices you make generally) but I
strongly suggest you have a conversation with your team about this issue,
and action you can take to address it. I really have enjoyed using
libabigail and I think it can be better in many ways, not just the library
itself but supported tools, automation, and branding, and this burden
cannot just be on one or two developers.

Thank you for engaging with me in this conversation, and fingers crossed
this message is not rejected for too many links, etc. !

Best,

Vanessa

On Sat, Apr 30, 2022 at 1:55 PM Mark Wielaard <mark@klomp.org> wrote:

> Hi Vanessa,
>
> On Wed, Apr 27, 2022 at 07:57:12PM -0600, v via Libabigail wrote:
> > Going to give this one more shot in case it works without the image (and
> I
> > don't get flagged as spam...)
>
> Sorry I missed your email earlier. It might be that your original
> email was HTML encoded which gets rejected by the mailinglist.
>
> > > > I am writing in response to
> > > https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I
> > > understand the desire to not add too many tests to make check, but I
> think
> > > maybe there is a nice middle ground or compromise?
> > > >
> > > > Basically, what I think libabigail needs is more automated testing of
> > > more cases.
>
> Agreed that would be nice. And agreed that the SLOW_TESTS should be
> tested by default on the buildbot. Ben already made that change and it
> even caught an issue on ppc64. We might not be advertising the
> buildbot enough. It recently moved to sourceware and got "badges", see
> https://builder.sourceware.org/ Maybe the libabigail badges/links
> should be added to the libabigail homepage.
>
> > > I added a simple setup to Ben's repo that does an automated
> > > build in a container, and since it starts with a base container with
> the
> > > deps it's not terribly too slow:
> > > >
> > >
> https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true
>
> Note that the logs appear to not be accessible unless you have a
> github account and are logged in. Could you make them public?
>
> > > > In that run we do make check (Ben had added extra tests there) but it
> > > would also be feasible to clone a testing repository instead. This is
> what
> > > I set up for dyninst, because their test suite is a bit of a chonker
> too
> > > (note the red clone into external tests).
> > > >
> > > >
> > >
> https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> > > >
> > > > Also, it would be pretty cool to run libabigail on itself to check
> for
> > > breaks:
> > > >
> > > > https://github.com/woodard/libabigail/actions/runs/2210184567
> > > >
> > > > And (kind of cool) to always run the changes in the PR against a
> bunch
> > > of known distro libs (here for Fedora).
> > > >
> > > > https://github.com/woodard/libabigail/actions/runs/2208891579
>
> Although I can imagine what this does and that it is useful all these
> URLs have the same problem, they don't actually show what is happening
> because the logs are not public.
>
> Could you post the scripts and container files somewhere so we can see
> how to incorporate this? I believe Dodji is working a libabigail-tests
> project/setup so that larger tests can be added without having them all
> in the main source repo.
>
> > > So, I think what I'm saying is that adding automation to testing
> > > libabigail would really empower us to catch more bugs.
>
> I agree. We do have some automated testing. But what we don't have yet
> is automated pre-commit testing. I hope to provide that through the
> buildbot using a try-builder (maybe combined with patchwork).
>
> > >  Have y'all ever
> > > considered moving some project stuffs over to GitHub so more people can
> > > help with libaibgail than just you and Ben? And we can implement a lot
> of
> > > automation for the project proper? I gave a talk recently at CU
> Boulder,
> > > and the students were really interested in libabigail. If it's readily
> > > available to contribute to on GitHub, I (and others I bet) would really
> > > enjoy contributing. I almost didn't write this email because it's a
> PITA to
> > > have to join a mailing list, but I think it's important. If you forever
> > > keep it on sourceware (channeling 1993 they want their web design
> back!) I
> > > think in the long run it's more work for you, and probably less fun
> too.
> > > >
> > > > What do you think? Is this a future you could imagine? And how might
> we
> > > get there?
>
> I think having more automation and more testing would be great. I can
> certainly imagine a future where we have that. But I hope we can avoid
> github to get here. Personally I try to avoid using proprietary
> software to create free software. And just to create an account on
> github seems to require running a lot of proprietary javascript and
> accepting 30+ pages of legal text.
>
> Note that if you like a "forge" to contribute to sourceware hosted
> projects then you can use the sourcehut mirror:
> https://sr.ht/~sourceware/libabigail/
> sourcehut also allows you to submit patches through the webinterface
> (which then sents properly formatted git email patches to this list).
>
> Note that you don't have to be subscribed to this list to post (but
> HTML emails are rejected).
>
> I don't know how the libabigail website is generated, but it would be
> nice to get that also in git so people can update it if they feel they
> have a better design.
>
> Cheers,
>
> Mark
>