Fwd: Testing Setup - More Tests and Automation?

Fwd: Testing Setup - More Tests and Automation?

[Ben Woodard]

My coworker who did most of the work included in: https://sourceware.org/pipermail/libabigail/2022q2/004292.html <which sadly got scrubbed due to a bug in Fedora 36> asked me to forward this email along to the mailing list because her posts are getting marked as spam.

I personally feel like we need to reconsider https://sourceware.org/pipermail/libabigail/2022q1/004045.html and have checks like this built into the normal “make check” having ENABLE_SLOW_TEST=no by default allows too many bugs to creep in. People don’t use it. As a case in point neither the current trunk or the fixes branch pass all the tests when it is set. 

I will say Vanessa has convinced me of the value of the automated testing. It is great when she does a pull request, I can immediately see not only that it applies but that it continues to pass all the regression tests. I feel like it would be helpful to have more tests included within the make check so that it didn’t depend on me to make sure that something isn’t going wrong.

-ben


> Begin forwarded message:
> 
> From: v <vsochat@gmail.com>
> Subject: Fwd: Testing Setup - More Tests and Automation?
> Date: April 27, 2022 at 4:42:09 PM PDT
> To: woodard@redhat.com
> 
> See my email below - I joined the sourceware list and it blocked me as spam. 😭
> 
> ---------- Forwarded message ---------
> From: v <vsochat@gmail.com>
> Date: Wed, Apr 27, 2022 at 5:40 PM
> Subject: Testing Setup - More Tests and Automation?
> To: <libabigail@sourceware.org>
> 
> 
> Hey Dodji et al,
> 
> I am writing in response to https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I understand the desire to not add too many tests to make check, but I think maybe there is a nice middle ground or compromise?
> 
> Basically, what I think libabigail needs is more automated testing of more cases. I added a simple setup to Ben's repo that does an automated build in a container, and since it starts with a base container with the deps it's not terribly too slow:
> 
> https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true
> 
> In that run we do make check (Ben had added extra tests there) but it would also be feasible to clone a testing repository instead. This is what I set up for dyninst, because their test suite is a bit of a chonker too (note the red clone into external tests). 
> 
> https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> 
> Also, it would be pretty cool to run libabigail on itself to check for breaks:
> 
> https://github.com/woodard/libabigail/actions/runs/2210184567
> 
> And (kind of cool) to always run the changes in the PR against a bunch of known distro libs (here for Fedora).
> 
> https://github.com/woodard/libabigail/actions/runs/2208891579
> 
> So, I think what I'm saying is that adding automation to testing libabigail would really empower us to catch more bugs. Have y'all ever considered moving some project stuffs over to GitHub so more people can help with libaibgail than just you and Ben? And we can implement a lot of automation for the project proper? I gave a talk recently at CU Boulder, and the students were really interested in libabigail. If it's readily available to contribute to on GitHub, I (and others I bet) would really enjoy contributing. I almost didn't write this email because it's a PITA to have to join a mailing list, but I think it's important. If you forever keep it on sourceware (channeling 1993 they want their web design back!) I think in the long run it's more work for you, and probably less fun too.
> 
> What do you think? Is this a future you could imagine? And how might we get there?
> 
> Best,
> 
> Vanessa

Re: Testing Setup - More Tests and Automation?

[v]

Going to give this one more shot in case it works without the image (and I
don't get flagged as spam...)

if you didn't see my attached message below Ben's, please unwrap the email
and look! 👇

Also Dodji I heard from a little bird that you are competing in a national
championship this weekend? Kick @ss!! Just forget about work until then
because being a national champion (or even contender) is so much cooler :)

On Wed, Apr 27, 2022 at 6:55 PM Ben Woodard <woodard@redhat.com> wrote:

> My coworker who did most of the work included in:
> https://sourceware.org/pipermail/libabigail/2022q2/004292.html <which
> sadly got scrubbed due to a bug in Fedora 36> asked me to forward this
> email along to the mailing list because her posts are getting marked as
> spam.
>
> I personally feel like we need to reconsider
> https://sourceware.org/pipermail/libabigail/2022q1/004045.html and have
> checks like this built into the normal “make check” having
> ENABLE_SLOW_TEST=no by default allows too many bugs to creep in. People
> don’t use it. As a case in point neither the current trunk or the fixes
> branch pass all the tests when it is set.
>
> I will say Vanessa has convinced me of the value of the automated testing.
> It is great when she does a pull request, I can immediately see not only
> that it applies but that it continues to pass all the regression tests. I
> feel like it would be helpful to have more tests included within the make
> check so that it didn’t depend on me to make sure that something isn’t
> going wrong.
>
> -ben
>
>
> > Begin forwarded message:
> >
> > From: v <vsochat@gmail.com>
> > Subject: Fwd: Testing Setup - More Tests and Automation?
> > Date: April 27, 2022 at 4:42:09 PM PDT
> > To: woodard@redhat.com
> >
> > See my email below - I joined the sourceware list and it blocked me as
> spam. 😭
> >
> > ---------- Forwarded message ---------
> > From: v <vsochat@gmail.com>
> > Date: Wed, Apr 27, 2022 at 5:40 PM
> > Subject: Testing Setup - More Tests and Automation?
> > To: <libabigail@sourceware.org>
> >
> >
> > Hey Dodji et al,
> >
> > I am writing in response to
> https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I
> understand the desire to not add too many tests to make check, but I think
> maybe there is a nice middle ground or compromise?
> >
> > Basically, what I think libabigail needs is more automated testing of
> more cases. I added a simple setup to Ben's repo that does an automated
> build in a container, and since it starts with a base container with the
> deps it's not terribly too slow:
> >
> >
> https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true
> >
> > In that run we do make check (Ben had added extra tests there) but it
> would also be feasible to clone a testing repository instead. This is what
> I set up for dyninst, because their test suite is a bit of a chonker too
> (note the red clone into external tests).
> >
> >
> https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> >
> > Also, it would be pretty cool to run libabigail on itself to check for
> breaks:
> >
> > https://github.com/woodard/libabigail/actions/runs/2210184567
> >
> > And (kind of cool) to always run the changes in the PR against a bunch
> of known distro libs (here for Fedora).
> >
> > https://github.com/woodard/libabigail/actions/runs/2208891579
> >
> > So, I think what I'm saying is that adding automation to testing
> libabigail would really empower us to catch more bugs. Have y'all ever
> considered moving some project stuffs over to GitHub so more people can
> help with libaibgail than just you and Ben? And we can implement a lot of
> automation for the project proper? I gave a talk recently at CU Boulder,
> and the students were really interested in libabigail. If it's readily
> available to contribute to on GitHub, I (and others I bet) would really
> enjoy contributing. I almost didn't write this email because it's a PITA to
> have to join a mailing list, but I think it's important. If you forever
> keep it on sourceware (channeling 1993 they want their web design back!) I
> think in the long run it's more work for you, and probably less fun too.
> >
> > What do you think? Is this a future you could imagine? And how might we
> get there?
> >
> > Best,
> >
> > Vanessa
>
>

Re: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi Vanessa,

On Wed, Apr 27, 2022 at 07:57:12PM -0600, v via Libabigail wrote:
> Going to give this one more shot in case it works without the image (and I
> don't get flagged as spam...)

Sorry I missed your email earlier. It might be that your original
email was HTML encoded which gets rejected by the mailinglist.

> > > I am writing in response to
> > https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I
> > understand the desire to not add too many tests to make check, but I think
> > maybe there is a nice middle ground or compromise?
> > >
> > > Basically, what I think libabigail needs is more automated testing of
> > more cases.

Agreed that would be nice. And agreed that the SLOW_TESTS should be
tested by default on the buildbot. Ben already made that change and it
even caught an issue on ppc64. We might not be advertising the
buildbot enough. It recently moved to sourceware and got "badges", see
https://builder.sourceware.org/ Maybe the libabigail badges/links
should be added to the libabigail homepage.

> > I added a simple setup to Ben's repo that does an automated
> > build in a container, and since it starts with a base container with the
> > deps it's not terribly too slow:
> > >
> > https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true

Note that the logs appear to not be accessible unless you have a
github account and are logged in. Could you make them public?

> > > In that run we do make check (Ben had added extra tests there) but it
> > would also be feasible to clone a testing repository instead. This is what
> > I set up for dyninst, because their test suite is a bit of a chonker too
> > (note the red clone into external tests).
> > >
> > >
> > https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> > >
> > > Also, it would be pretty cool to run libabigail on itself to check for
> > breaks:
> > >
> > > https://github.com/woodard/libabigail/actions/runs/2210184567
> > >
> > > And (kind of cool) to always run the changes in the PR against a bunch
> > of known distro libs (here for Fedora).
> > >
> > > https://github.com/woodard/libabigail/actions/runs/2208891579

Although I can imagine what this does and that it is useful all these
URLs have the same problem, they don't actually show what is happening
because the logs are not public.

Could you post the scripts and container files somewhere so we can see
how to incorporate this? I believe Dodji is working a libabigail-tests
project/setup so that larger tests can be added without having them all
in the main source repo.

> > So, I think what I'm saying is that adding automation to testing
> > libabigail would really empower us to catch more bugs.

I agree. We do have some automated testing. But what we don't have yet
is automated pre-commit testing. I hope to provide that through the
buildbot using a try-builder (maybe combined with patchwork).

> >  Have y'all ever
> > considered moving some project stuffs over to GitHub so more people can
> > help with libaibgail than just you and Ben? And we can implement a lot of
> > automation for the project proper? I gave a talk recently at CU Boulder,
> > and the students were really interested in libabigail. If it's readily
> > available to contribute to on GitHub, I (and others I bet) would really
> > enjoy contributing. I almost didn't write this email because it's a PITA to
> > have to join a mailing list, but I think it's important. If you forever
> > keep it on sourceware (channeling 1993 they want their web design back!) I
> > think in the long run it's more work for you, and probably less fun too.
> > >
> > > What do you think? Is this a future you could imagine? And how might we
> > get there?

I think having more automation and more testing would be great. I can
certainly imagine a future where we have that. But I hope we can avoid
github to get here. Personally I try to avoid using proprietary
software to create free software. And just to create an account on
github seems to require running a lot of proprietary javascript and
accepting 30+ pages of legal text.

Note that if you like a "forge" to contribute to sourceware hosted
projects then you can use the sourcehut mirror:
https://sr.ht/~sourceware/libabigail/
sourcehut also allows you to submit patches through the webinterface
(which then sents properly formatted git email patches to this list).

Note that you don't have to be subscribed to this list to post (but
HTML emails are rejected).

I don't know how the libabigail website is generated, but it would be
nice to get that also in git so people can update it if they feel they
have a better design.

Cheers,

Mark

Re: Testing Setup - More Tests and Automation?

[v]

Hi Mark,

The links are private because this is Ben's personal repository for his
account - given that libabigail was developed under a proper organization
it would be public. You can look at example OSS projects (e.g., tensorflow
https://github.com/tensorflow/tensorflow/runs/6241307726?check_suite_focus=true)
to get a sense of what the UI looks like.

I understand the desire to not use proprietary software, but if you needed
to move the setup elsewhere that's not hard to do! Modern software
developers use GitHub, and all major companies (including RedHat) grow
strong communities of developers on there. Podman is a good example:
https://github.com/containers/podman. Git is git, so you could develop on
GitHub and keep a backup on sourceware, and have an automated workflow to
do that. In other words, there are ways to have your cake and eat it too.
Let's take a look at Libabigail. From Ben's contributor graph:
https://github.com/woodard/libabigail/graphs/contributors

Or from git:

  1835  Dodji Seketeli <dodji@redhat.com>
   144  Giuliano Procida <gprocida@google.com>
   135  Matthias Maennich <maennich@google.com>
    35  Benjamin Kosnik <bkoz@redhat.com>
    33  Dodji Seketeli <dodji@seketeli.org>
    19  Chenxiong Qi <cqi@redhat.com>
    19  Mark Wielaard <mark@klomp.org>
    19  Ondrej Oprala <ooprala@redhat.com>
    15  Sinny Kumari <sinny@redhat.com>
    13  Thomas Schwinge <thomas@codesourcery.com>
    11  Ondrej Oprala <ondrej.oprala@gmail.com>
     9  Jan Engelhardt <jengelh@inai.de>
     8  Guillermo E. Martinez <guillermo.e.martinez@oracle.com>
     8  tangmeng <tangmeng@uniontech.com>
     7  Ben Woodard <woodard@redhat.com>
     7  Jonathan Wakely <jwakely@redhat.com>
     7  Jose E. Marchesi <jose.marchesi@oracle.com>
     5  Mark Wielaard <mjw@redhat.com>
     4  Sinny Kumari <skumari@redhat.com>
     4  maennich@google.com <maennich@google.com>
     2  Matthias Klose <doko@debian.org>
     1  David Cantrell <dcantrell@redhat.com>
     1  David Seifert <soap@gentoo.org>
     1  Dodji <dodji@ks305400.kimsufi.com>
     1  George Rawlinson <grawlinson@archlinux.org>
     1  Jessica Yu <jeyu@kernel.org>
     1  Randy MacLeod <Randy.MacLeod@windriver.com>
     1  Roland McGrath <roland@hack.frob.com>
     1  Slava Barinov <v.barinov@samsung.com>
     1  Vanessa Sochat <sochat1@llnl.gov>
     1  Xiao Jia <xiaoj@google.com>

You have *eighteen* contributors with over 5 commits in a project that is 8
years old, and that includes repeated individuals using different
alias/email. Out of those 18, it looks like there are maybe 3 that have
contributed meaningfully (and recently), and talking with Ben (and seeing
number of commits), I get a sense that the main developer is Dodji and he
is overwhelmed. So (to me) for a library that is as important as
libabigail, this is problematic. The bus factor is really high. Compare
that to Podman https://github.com/containers/podman/graphs/contributors,
which is also a niche technology / area of work, and the project has only
been around since late 2017 (and I don't know when it went up on GitHub
from perhaps being private at RedHat) and it has almost 400 contributors
(just over 100 with >= 5 commits) and many others that haven't contributed
directly to the code (myself included) but have contributed via issues and
other interaction.

So (to me) this is a problem that needs to be solved, and I don't think
keeping things the way that they are is that solution, regardless of how
you innovate a bit with adding automation. The cost of not changing your
current setup is very high, and I worry about the future of libabigail. As
a developer I want to contribute and champion the work, but in its current
state I find the project feels very closed and not trying bring me in. The
students I spoke to at CU Boulder were inspired, but they will stop their
exploration when they realize it's not easy to ask questions or for help.
In other words, you are 100% losing out in terms of new contributors and
developers by not using these modern practices and services.  That is of
course your choice (and I respect the choices you make generally) but I
strongly suggest you have a conversation with your team about this issue,
and action you can take to address it. I really have enjoyed using
libabigail and I think it can be better in many ways, not just the library
itself but supported tools, automation, and branding, and this burden
cannot just be on one or two developers.

Thank you for engaging with me in this conversation, and fingers crossed
this message is not rejected for too many links, etc. !

Best,

Vanessa

On Sat, Apr 30, 2022 at 1:55 PM Mark Wielaard <mark@klomp.org> wrote:

> Hi Vanessa,
>
> On Wed, Apr 27, 2022 at 07:57:12PM -0600, v via Libabigail wrote:
> > Going to give this one more shot in case it works without the image (and
> I
> > don't get flagged as spam...)
>
> Sorry I missed your email earlier. It might be that your original
> email was HTML encoded which gets rejected by the mailinglist.
>
> > > > I am writing in response to
> > > https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I
> > > understand the desire to not add too many tests to make check, but I
> think
> > > maybe there is a nice middle ground or compromise?
> > > >
> > > > Basically, what I think libabigail needs is more automated testing of
> > > more cases.
>
> Agreed that would be nice. And agreed that the SLOW_TESTS should be
> tested by default on the buildbot. Ben already made that change and it
> even caught an issue on ppc64. We might not be advertising the
> buildbot enough. It recently moved to sourceware and got "badges", see
> https://builder.sourceware.org/ Maybe the libabigail badges/links
> should be added to the libabigail homepage.
>
> > > I added a simple setup to Ben's repo that does an automated
> > > build in a container, and since it starts with a base container with
> the
> > > deps it's not terribly too slow:
> > > >
> > >
> https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus=true
>
> Note that the logs appear to not be accessible unless you have a
> github account and are logged in. Could you make them public?
>
> > > > In that run we do make check (Ben had added extra tests there) but it
> > > would also be feasible to clone a testing repository instead. This is
> what
> > > I set up for dyninst, because their test suite is a bit of a chonker
> too
> > > (note the red clone into external tests).
> > > >
> > > >
> > >
> https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=true#step:5:78
> > > >
> > > > Also, it would be pretty cool to run libabigail on itself to check
> for
> > > breaks:
> > > >
> > > > https://github.com/woodard/libabigail/actions/runs/2210184567
> > > >
> > > > And (kind of cool) to always run the changes in the PR against a
> bunch
> > > of known distro libs (here for Fedora).
> > > >
> > > > https://github.com/woodard/libabigail/actions/runs/2208891579
>
> Although I can imagine what this does and that it is useful all these
> URLs have the same problem, they don't actually show what is happening
> because the logs are not public.
>
> Could you post the scripts and container files somewhere so we can see
> how to incorporate this? I believe Dodji is working a libabigail-tests
> project/setup so that larger tests can be added without having them all
> in the main source repo.
>
> > > So, I think what I'm saying is that adding automation to testing
> > > libabigail would really empower us to catch more bugs.
>
> I agree. We do have some automated testing. But what we don't have yet
> is automated pre-commit testing. I hope to provide that through the
> buildbot using a try-builder (maybe combined with patchwork).
>
> > >  Have y'all ever
> > > considered moving some project stuffs over to GitHub so more people can
> > > help with libaibgail than just you and Ben? And we can implement a lot
> of
> > > automation for the project proper? I gave a talk recently at CU
> Boulder,
> > > and the students were really interested in libabigail. If it's readily
> > > available to contribute to on GitHub, I (and others I bet) would really
> > > enjoy contributing. I almost didn't write this email because it's a
> PITA to
> > > have to join a mailing list, but I think it's important. If you forever
> > > keep it on sourceware (channeling 1993 they want their web design
> back!) I
> > > think in the long run it's more work for you, and probably less fun
> too.
> > > >
> > > > What do you think? Is this a future you could imagine? And how might
> we
> > > get there?
>
> I think having more automation and more testing would be great. I can
> certainly imagine a future where we have that. But I hope we can avoid
> github to get here. Personally I try to avoid using proprietary
> software to create free software. And just to create an account on
> github seems to require running a lot of proprietary javascript and
> accepting 30+ pages of legal text.
>
> Note that if you like a "forge" to contribute to sourceware hosted
> projects then you can use the sourcehut mirror:
> https://sr.ht/~sourceware/libabigail/
> sourcehut also allows you to submit patches through the webinterface
> (which then sents properly formatted git email patches to this list).
>
> Note that you don't have to be subscribed to this list to post (but
> HTML emails are rejected).
>
> I don't know how the libabigail website is generated, but it would be
> nice to get that also in git so people can update it if they feel they
> have a better design.
>
> Cheers,
>
> Mark
>

Re: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi Vanessa,

On Sat, Apr 30, 2022 at 02:36:47PM -0600, v wrote:
> The links are private because this is Ben's personal repository for his
> account - given that libabigail was developed under a proper organization
> it would be public. You can look at example OSS projects (e.g., tensorflow
> https://github.com/tensorflow/tensorflow/runs/6241307726?check_suite_focus=true)
> to get a sense of what the UI looks like.

That shows the same thing "Sign in to view logs".

> I understand the desire to not use proprietary software, but if you needed
> to move the setup elsewhere that's not hard to do! Modern software
> developers use GitHub, and all major companies (including RedHat) grow
> strong communities of developers on there. Podman is a good example:
> https://github.com/containers/podman. Git is git, so you could develop on
> GitHub and keep a backup on sourceware, and have an automated workflow to
> do that. In other words, there are ways to have your cake and eat it too.

I have contributed to some projects that use github (see e.g. rpm or
gccrust) but found it a major pita to be honest. Their website
basically doesn't work without javascript. And you have to depend on
people having a github account to do pull requests or accept patches
for you. Maybe it works better if you are able to create an account
and don't mind running all this proprietary javascript. But the legal
language involved in creating one is so large that I cannot even tell
if it is acceptable or not.

I happen to work for Red Hat myself as day job. But not directly
on libabigail, nor on any project that uses github. I know it is
possible to extract some stuff from github, I do keep some mirrors in
my own git setup https://code.wildebeest.org/ so I can sent patches
or pull requests. But I don't find the experience of interacting with
github using projects great.

> You have *eighteen* contributors with over 5 commits in a project that is 8
> years old, and that includes repeated individuals using different
> alias/email. Out of those 18, it looks like there are maybe 3 that have
> contributed meaningfully (and recently), and talking with Ben (and seeing
> number of commits), I get a sense that the main developer is Dodji and he
> is overwhelmed. So (to me) for a library that is as important as
> libabigail, this is problematic. The bus factor is really high.

I agree it would be good to have more active developers. I doubt it
matters whether or not the project has a github webpage. For me
personally I stopped most contributions since the licence
change. Which might or might not be a factor to others. Personally I
like contributing to projects that use strong copyleft.

> Compare
> that to Podman https://github.com/containers/podman/graphs/contributors,
> which is also a niche technology / area of work, and the project has only
> been around since late 2017 (and I don't know when it went up on GitHub
> from perhaps being private at RedHat) and it has almost 400 contributors
> (just over 100 with >= 5 commits) and many others that haven't contributed
> directly to the code (myself included) but have contributed via issues and
> other interaction.

I am not sure that libabigail and podman are comparable projects. But
I get that there are ways to interact with the project on the github
website that you prefer to interacting with the libabigail project?

> So (to me) this is a problem that needs to be solved, and I don't think
> keeping things the way that they are is that solution, regardless of how
> you innovate a bit with adding automation. The cost of not changing your
> current setup is very high, and I worry about the future of libabigail.

So which changes would you like to see? Note that I don't mind a forge
like setup, I just think that github is a pretty bad one. But
something like sourcehut, self-hosted gitlab or pagure would be pretty
cool. I do think adding automated testing is a pretty valuable thing,
especially if contributors can use it as pre-commit gating, which is
why I am working on adding buildbot and container based automation to
sourceware for all project hosted there. So I am a bit surprised you
believe that isn't part of the solution.

> As
> a developer I want to contribute and champion the work, but in its current
> state I find the project feels very closed and not trying bring me in. The
> students I spoke to at CU Boulder were inspired, but they will stop their
> exploration when they realize it's not easy to ask questions or for help.

Is that because they don't use email, bugzilla or irc? Do you think we
should have some kind of web-forum for the project?

Would their opinion change if they realize libabigail is mirrored on
sourcehut already?

> In other words, you are 100% losing out in terms of new contributors and
> developers by not using these modern practices and services.

Which modern practices and services are those exactly?

> That is of
> course your choice (and I respect the choices you make generally) but I
> strongly suggest you have a conversation with your team about this issue,
> and action you can take to address it. I really have enjoyed using
> libabigail and I think it can be better in many ways, not just the library
> itself but supported tools, automation, and branding, and this burden
> cannot just be on one or two developers.

I hope you can help and I hope we can provide you with the tools to do so.

Thanks,

Mark

Re: Testing Setup - More Tests and Automation?

[v]

Hi Mark,

I appreciate you keeping your opinion about GitHub and dislike separate
from an overall evaluation of what is best for the project. There are
things that I don't like too, but I recognize the value for other
developers or (in this case) growing a community.

That shows the same thing "Sign in to view logs".
>
> Yes indeed you need an account! So we can remove our personal biases (I
think sourceware is a PITA) and look at actual data to see where developers
are developing. I think sourceware has ~40 projects
<https://sourceware.org/projects.html>? How many users? Maybe a couple of
hundred? Or maybe a thousand? Well GitHub has over 70 million users and
more than 200 million repos. I suspect you'll find more people on there
than sourceware, but just an educated guess :) Developers aren't going to
go out of their way to come to your project, you have to meet them where
they are at. We can put aside our personal feelings about particular tools
and recognize that.

I have contributed to some projects that use github (see e.g. rpm or
> gccrust) but found it a major pita to be honest. Their website
> basically doesn't work without javascript.


Probably 1% (or fewer) users browse with JavaScript disabled (reference
https://stackoverflow.com/questions/9478737/browser-statistics-on-javascript-disabled).
I only do it when I want to get beyond a paywall  :)  Again, I understand
we all have personal preferences, but sometimes our preferences can get in
the way of what is best for a project or community (or growing one when
it's largely absent).

And you have to depend on
> people having a github account to do pull requests or accept patches
> for you.


Many developers feel differently than you and love GitHub. I can say the
same about libabigail - not only do I need to join an email list (that
doesn't support basic stuff like adding images) the git repository (that is
hard to find) looks like this:
https://sourceware.org/git/gitweb.cgi?p=libabigail.git. I can't easily open
an issue to ask a question, contribute a pull request, or even browse the
source code and derive relationships between things (e.g., GitHub can allow
you to click a function and be taken to the definition and other nice
features). I totally get and respect that you don't have (and do not want)
a GitHub account, but many developers do, and have been using it over a
decade (myself included). I think there is something to say for choosing
the preferences of the majority of people that are potentially developers
for your project?


> Maybe it works better if you are able to create an account
> and don't mind running all this proprietary javascript. But the legal
> language involved in creating one is so large that I cannot even tell
> if it is acceptable or not.
>
> The legal language? I'm not sure I follow here, because your company
(RedHat) has many GitHub users and repos. I am not a lawyer so possibly you
can talk to one at RedHat if you have concerns. But the fact that major
tech companies not just allow but champion their developers to work on
GitHub suggests to me that lawyers have probably already looked and think
it's okay. And for putting code online, generally companies have a process
for making code public, and I suspect you've already done that with
libabigail since it's online.


> I happen to work for Red Hat myself as day job. But not directly
> on libabigail, nor on any project that uses github. I know it is
> possible to extract some stuff from github, I do keep some mirrors in
> my own git setup https://code.wildebeest.org/ so I can sent patches
> or pull requests. But I don't find the experience of interacting with
> github using projects great.
>
>
Again, the majority of developers do find the experience great, and it's
important to separate our personal opinions from what is best for the
project, which is to consider the practices of the bulk majority of
developers (that you'd want to help with your project).


> I agree it would be good to have more active developers. I doubt it
> matters whether or not the project has a github webpage. For me
> personally I stopped most contributions since the licence
> change. Which might or might not be a factor to others. Personally I
> like contributing to projects that use strong copyleft.
>
>
Great! It does matter where the project is, because where it is has
implications for 1. finding it, 2. seeing the branding and feeling that it
is modern and welcoming, and 3. being able to easily open an issue to ask a
question or fork and then contribute. It's not that sourceware is bad, but
that it uses practices that the majority of developers are not familiar
with, and although it's not easily apparent (it's hard to notice absence of
things) but the absence of contributors to your project I do believe is
based on this choice. I think you probably don't have a large enough
developer base (and perhaps I'm the first person to ask for this migration)
but it's fairly common for developers to make this ask, e.g,
https://www.wired.com/story/open-source-all-about-github-now/.

I am not sure that libabigail and podman are comparable projects. But
> I get that there are ways to interact with the project on the github
> website that you prefer to interacting with the libabigail project?
>
> Yes! I can open issues, and I can browse code and click on functions to
navigate to their definition, I can easily fork, make a change, and open a
pull request, click next to any line of code to create a permalink to
reference elsewhere, copy the line, see the git blame or to open an issue,
I can create milestones, discussions, and automate everything from the docs
deployment to linting and building containers (or other artifacts)
alongside the repository, I can reference issues / lines or full sections
in issues or discussions (even from other projects), I can use images /
other media in all places, I can get automatic security alerts based on
dependencies in my repo, I can automate my release process and generate
release notes automatically based on the commit history, and don't even get
me started on how amazing it is to use the API - I can automate everything!
I have entire websites that are generated from data that is updated via a
workflow nightly and rendered without a hitch. And you can add a CNAME to a
repository and get your documentation deployed without a hitch (https works
out of the box!) And deploying documentation to GitHub pages is literally
pushing files to a branch (or subdirectory) and calling it a day. I (and
many others) feel empowered by what we can do, and it's really more of a
question of what we can't do? [reference modern-practices]


> So which changes would you like to see? Note that I don't mind a forge
> like setup, I just think that github is a pretty bad one. But
> something like sourcehut, self-hosted gitlab or pagure would be pretty
> cool.


I think I've expressed in the sections above what I'd like to see. I think
if libabigail is to grow the community of developers you have to go where
they are. Personally speaking, to contribute to libabigail I want to
continue to use all of the above.


> I do think adding automated testing is a pretty valuable thing,
> especially if contributors can use it as pre-commit gating, which is
> why I am working on adding buildbot and container based automation to
> sourceware for all project hosted there. So I am a bit surprised you
> believe that isn't part of the solution.
>
> I made all the workflows / containers for Ben for libabigail in like a day
(you can pull from GitHub packages, e.g.,)

docker run -it ghcr.io/woodard/libabigail:latest

and they are hosted alongside the repository (another amazing feature, out
of the box!). I think it's great you are thinking about this (and it's part
of the automation solution, for sure) but not part of the "grow a community
and get people excited about libabigail." What is your plan for that?


> Is that because they don't use email, bugzilla or irc? Do you think we
> should have some kind of web-forum for the project?
>
> I think you should have the project on GitHub :) Nobody wants another
niche account / service they have to create (speaking for the majority of
developers already on GitHub that don't find it a PITA).


> Would their opinion change if they realize libabigail is mirrored on
> sourcehut already?
>
>
I don't even know what sourcehut is and I'm afraid to look. 😆


> > In other words, you are 100% losing out in terms of new contributors and
> > developers by not using these modern practices and services.
>
> Which modern practices and services are those exactly?
>

See [reference modern-practices] above. That's probably a shortened list.

I hope you can help and I hope we can provide you with the tools to do so.
>
> If I need to do something special for libabigail it's unlikely, but thank
you for the sentiment! If things stay the same the most I'll contribute is
probably working with Ben via his repo, which has been pretty fun :) But
that is because I know Ben and was able to find it, and I suspect others
won't do the same.

Best,

Vanessa

Re: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi Vanessa,

On Sat, Apr 30, 2022 at 04:38:52PM -0600, v via Libabigail wrote:
> I appreciate you keeping your opinion about GitHub and dislike separate
> from an overall evaluation of what is best for the project. There are
> things that I don't like too, but I recognize the value for other
> developers or (in this case) growing a community.

I am aware of my biases and preferences, which might be different from
other developers based on our experiences. And I certainly want to
learn from other experiences. But I do believe software freedom comes
first and there are real negatives to a community for relying on some
proprietary centralized hosting facility. I am happy to learn and
adopt free software replacements to get a similar experience though.

I am a little disappointed you seem to brush away any concerns by
saying github is currently really popular in some groups of
developers. Just because something is popular doesn't mean it is good
or appropriate in all circumstances. It would also be nice if you
could at least acknowledge that there are (possibly better?)
alternative forges which support "modern practices" that aren't
proprietary. Or that you would at least be willing to look at
them. They might not be exactly what you are personally accustomed to,
but if you evaluate them honestly we can at least see whether we can
adapt them. They are free software platforms after all, which we can
improve together.

Cheers,

Mark

Re: Testing Setup - More Tests and Automation?

[v]

My desire is that the project attracts new developers, and from a
standpoint of data and numbers an easy way to do that is using GitHub. My
feelings about software freedom are not even related to this discussion, so
you need not bring up points about me personally. This is my feedback.

I hope y'all can come up with a plan to create a libabigail developer
community, because I don't see it. Take care, I think I've said my piece
and I'm done.

On Sun, May 1, 2022 at 4:42 PM Mark Wielaard <mark@klomp.org> wrote:

> Hi Vanessa,
>
> On Sat, Apr 30, 2022 at 04:38:52PM -0600, v via Libabigail wrote:
> > I appreciate you keeping your opinion about GitHub and dislike separate
> > from an overall evaluation of what is best for the project. There are
> > things that I don't like too, but I recognize the value for other
> > developers or (in this case) growing a community.
>
> I am aware of my biases and preferences, which might be different from
> other developers based on our experiences. And I certainly want to
> learn from other experiences. But I do believe software freedom comes
> first and there are real negatives to a community for relying on some
> proprietary centralized hosting facility. I am happy to learn and
> adopt free software replacements to get a similar experience though.
>
> I am a little disappointed you seem to brush away any concerns by
> saying github is currently really popular in some groups of
> developers. Just because something is popular doesn't mean it is good
> or appropriate in all circumstances. It would also be nice if you
> could at least acknowledge that there are (possibly better?)
> alternative forges which support "modern practices" that aren't
> proprietary. Or that you would at least be willing to look at
> them. They might not be exactly what you are personally accustomed to,
> but if you evaluate them honestly we can at least see whether we can
> adapt them. They are free software platforms after all, which we can
> improve together.
>
> Cheers,
>
> Mark
>

Thinking different (was Re: Testing Setup - More Tests and Automation?)

[Dodji Seketeli]

Hello,

In concrete terms, would it be possible to, say, be able to interact
with those who want it via email (for instance, be able to just post
patches to this mailing list) AND ALSO via (git{hub,lab}) pull requests
(whose notifications would be sent to this list)?

In that case I understand that I would have, indeed, to have an account
on all those platforms, to pull the code.  But maybe I wouldn't be
forcing all of you to do so.

Would that be feasible? How? Or is there a better way forward?

I am really trying to move the needle forward here.

Thanks.

-- 
		Dodji

Re: Thinking different (was Re: Testing Setup - More Tests and Automation?)

[v]

I think we could have a GitHub org for libabigail
https://github.com/libabigail that updated itself nightly with the
sourceware remote, and then take pull requests there that are transformed
into patches and submit by some means. If Ben can walk me through what
happens behind the scenes when he does that I can probably make specific
suggestions. If there is a programmatic way to submit to the list we can
probably get GitHub content there too (the other direction). We can do
everything from posting issues to pull requests (to really whatever content
you want from the list - the GitHub APIs are amazing).

If we pull from sourceware I don't think you'd require an account Dodji,
but if you wanted to try out GitHub and have fun on there with respect to
discussions, issues, I highly recommend it :) Maybe your username would be
https://github.com/masterdodji?

Ben - ping me when you are around and we can talk about these logistics!
Maybe we could write up a concrete plan and present it to Dodji et. al.

Best,

V

On Thu, Jun 9, 2022 at 5:31 AM Dodji Seketeli <dodji@seketeli.org> wrote:

> Hello,
>
> In concrete terms, would it be possible to, say, be able to interact
> with those who want it via email (for instance, be able to just post
> patches to this mailing list) AND ALSO via (git{hub,lab}) pull requests
> (whose notifications would be sent to this list)?
>
> In that case I understand that I would have, indeed, to have an account
> on all those platforms, to pull the code.  But maybe I wouldn't be
> forcing all of you to do so.
>
> Would that be feasible? How? Or is there a better way forward?
>
> I am really trying to move the needle forward here.
>
> Thanks.
>
> --
>                 Dodji
>

Re: Thinking different (was Re: Testing Setup - More Tests and Automation?)

[Dodji Seketeli]

Hello V,

v <vsochat@gmail.com> a écrit:

> I think we could have a GitHub org for libabigail
> https://github.com/libabigail that updated itself nightly with the
> sourceware remote, and then take pull requests there that are transformed
> into patches and submit by some means.

Whoah ...

Could the comments/discussions happening during the patch review be
forwarded to the mailing list as well?

> If Ben can walk me through what happens behind the scenes when he does
> that I can probably make specific suggestions. If there is a
> programmatic way to submit to the list we can probably get GitHub
> content there too (the other direction).

Oh, you mean, have the discussions that happen on the mailing about a
particular pull request be forwared to github as well?


> We can do everything from posting issues to pull requests (to really
> whatever content you want from the list - the GitHub APIs are
> amazing).

Whoah ...

> If we pull from sourceware I don't think you'd require an account Dodji,
> but if you wanted to try out GitHub and have fun on there with respect to
> discussions, issues, I highly recommend it :) Maybe your username would be
> https://github.com/masterdodji?

As weird as it sounds, I already have github account!  Of course, I
forgot what my username there was.  It's simply is 'dodjiseketeli' ;-)p

> Ben - ping me when you are around and we can talk about these logistics!
> Maybe we could write up a concrete plan and present it to Dodji et. al.

Thanks so much.

[...]

Cheers,

-- 
		Dodji

Re: Testing Setup - More Tests and Automation?

[Dodji Seketeli]

Hello Vanessa,

v via Libabigail <libabigail@sourceware.org> a écrit:

[...]

First of all, let me say that for what it's worth, I support your effort
of building a stronger community at 1000%, at very least.

Now, my take is that, in a very pragmatic way, we just need to figure
out a way to get "there".

[...]

> I understand the desire to not use proprietary software, but if you needed
> to move the setup elsewhere that's not hard to do! Modern software
> developers use GitHub, and all major companies (including RedHat) grow
> strong communities of developers on there. Podman is a good example:
> https://github.com/containers/podman. Git is git, so you could develop on
> GitHub and keep a backup on sourceware, and have an automated workflow to
> do that. In other words, there are ways to have your cake and eat it too.
> Let's take a look at Libabigail. From Ben's contributor graph:
> https://github.com/woodard/libabigail/graphs/contributors
>
> Or from git:
>
>   1835  Dodji Seketeli <dodji@redhat.com>
>    144  Giuliano Procida <gprocida@google.com>
>    135  Matthias Maennich <maennich@google.com>
>     35  Benjamin Kosnik <bkoz@redhat.com>
>     33  Dodji Seketeli <dodji@seketeli.org>
>     19  Chenxiong Qi <cqi@redhat.com>
>     19  Mark Wielaard <mark@klomp.org>
>     19  Ondrej Oprala <ooprala@redhat.com>
>     15  Sinny Kumari <sinny@redhat.com>
>     13  Thomas Schwinge <thomas@codesourcery.com>
>     11  Ondrej Oprala <ondrej.oprala@gmail.com>
>      9  Jan Engelhardt <jengelh@inai.de>
>      8  Guillermo E. Martinez <guillermo.e.martinez@oracle.com>
>      8  tangmeng <tangmeng@uniontech.com>
>      7  Ben Woodard <woodard@redhat.com>
>      7  Jonathan Wakely <jwakely@redhat.com>
>      7  Jose E. Marchesi <jose.marchesi@oracle.com>
>      5  Mark Wielaard <mjw@redhat.com>
>      4  Sinny Kumari <skumari@redhat.com>
>      4  maennich@google.com <maennich@google.com>
>      2  Matthias Klose <doko@debian.org>
>      1  David Cantrell <dcantrell@redhat.com>
>      1  David Seifert <soap@gentoo.org>
>      1  Dodji <dodji@ks305400.kimsufi.com>
>      1  George Rawlinson <grawlinson@archlinux.org>
>      1  Jessica Yu <jeyu@kernel.org>
>      1  Randy MacLeod <Randy.MacLeod@windriver.com>
>      1  Roland McGrath <roland@hack.frob.com>
>      1  Slava Barinov <v.barinov@samsung.com>
>      1  Vanessa Sochat <sochat1@llnl.gov>
>      1  Xiao Jia <xiaoj@google.com>
>
> You have *eighteen* contributors with over 5 commits in a project that is 8
> years old, and that includes repeated individuals using different
> alias/email. Out of those 18, it looks like there are maybe 3 that have
> contributed meaningfully (and recently), and talking with Ben (and seeing
> number of commits), I get a sense that the main developer is Dodji and he
> is overwhelmed. So (to me) for a library that is as important as
> libabigail, this is problematic. The bus factor is really high. Compare
> that to Podman https://github.com/containers/podman/graphs/contributors,
> which is also a niche technology / area of work, and the project has only
> been around since late 2017 (and I don't know when it went up on GitHub
> from perhaps being private at RedHat) and it has almost 400 contributors
> (just over 100 with >= 5 commits) and many others that haven't contributed
> directly to the code (myself included) but have contributed via issues and
> other interaction.
>
> So (to me) this is a problem that needs to be solved, and I don't think
> keeping things the way that they are is that solution, regardless of how
> you innovate a bit with adding automation. The cost of not changing your
> current setup is very high, and I worry about the future of libabigail. As
> a developer I want to contribute and champion the work, but in its current
> state I find the project feels very closed and not trying bring me in. The
> students I spoke to at CU Boulder were inspired, but they will stop their
> exploration when they realize it's not easy to ask questions or for help.
> In other words, you are 100% losing out in terms of new contributors and
> developers by not using these modern practices and services.  That is of
> course your choice (and I respect the choices you make generally) but I
> strongly suggest you have a conversation with your team about this issue,
> and action you can take to address it. I really have enjoyed using
> libabigail and I think it can be better in many ways, not just the library
> itself but supported tools, automation, and branding, and this burden
> cannot just be on one or two developers.
>
> Thank you for engaging with me in this conversation, and fingers crossed
> this message is not rejected for too many links, etc. !

What else can I say, really :-)  This is so eloquently put.  There is
not much I can add to this.

Thank you so much and let's get there together.

Cheers,

-- 
		Dodji

Re: Testing Setup - More Tests and Automation?

[Dodji Seketeli]

Mark Wielaard <mark@klomp.org> a écrit:


> I don't know how the libabigail website is generated, but it would be
> nice to get that also in git so people can update it if they feel they
> have a better design.

It's generated by "make doc" and it's in $builddir/doc/website.  It's in
git, thus.

And it definitely needs a revamp, that's for sure.

[...]

Cheers,

-- 
		Dodji

Re: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi Dodji,

On Thu, 2022-06-09 at 13:11 +0200, Dodji Seketeli wrote:
> Mark Wielaard <mark@klomp.org> a écrit:
> > I don't know how the libabigail website is generated, but it would be
> > nice to get that also in git so people can update it if they feel they
> > have a better design.
> 
> It's generated by "make doc" and it's in $builddir/doc/website.  It's in
> git, thus.

Is there a special way to configure to get this populated?
When I run make doc it does generate several docs, but the website dir
is (nearly) empty:

$ ls doc/*
doc/Makefile  doc/Makefile.am  doc/Makefile.in  doc/suppr-doc.txt

doc/api:
doxygen-warnings.txt  html  html.doxy  libabigail.doxy

doc/html:
bc_s.png     dynsections.js    menu.js    splitbar.png  tabs.css
bdwn.png     folderclosed.png  nav_f.png  sync_off.png  tab_s.png
closed.png   folderopen.png    nav_g.png  sync_on.png
doc.png      index.html        nav_h.png  tab_a.png
doxygen.css  jquery.js         open.png   tab_b.png
doxygen.png  menudata.js       search     tab_h.png

doc/latex:
doxygen.sty  Makefile  refman.tex

doc/manuals:
abicompat.rst   conf.py            kmidiff.rst              Makefile.am
abidiff.rst     doctrees           libabigail-concepts.rst  Makefile.in
abidw.rst       fedabipkgdiff.rst  libabigail-overview.rst  man
abilint.rst     html               libabigail-tools.rst     texinfo
abipkgdiff.rst  index.rst          Makefile

doc/vizualization:
graph  layout

doc/website:
libabigail-website.doxy  mainpage.txt

Is there a followup step needed?

We could automate updating the website through builder.sourceware.org
if there is a script to update it.

Cheers,

Mark

Re: Fwd: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi,

On Wed, Apr 27, 2022 at 05:55:07PM -0700, Ben Woodard via Libabigail wrote:
> I personally feel like we need to reconsider
> https://sourceware.org/pipermail/libabigail/2022q1/004045.html and
> have checks like this built into the normal “make check” having
> ENABLE_SLOW_TEST=no by default allows too many bugs to creep
> in. People don’t use it. As a case in point neither the current
> trunk or the fixes branch pass all the tests when it is set.

We could/should probably add ENABLE_SLOW_TEST=yes to the buildbot.
The builder recently got moved to the general sourceware GNU Toolchain
https://builder.sourceware.org/ with configuration/sources here
https://sourceware.org/git/builder.git

The builder doesn't do containers builds nor pre-commit builds, but
that is on the TODO list.

Cheers,

Mark

Re: Fwd: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi Ben,

On Thu, Apr 28, 2022 at 10:06:59AM +0200, Mark Wielaard wrote:
> On Wed, Apr 27, 2022 at 05:55:07PM -0700, Ben Woodard via Libabigail wrote:
> > I personally feel like we need to reconsider
> > https://sourceware.org/pipermail/libabigail/2022q1/004045.html and
> > have checks like this built into the normal “make check” having
> > ENABLE_SLOW_TEST=no by default allows too many bugs to creep
> > in. People don’t use it. As a case in point neither the current
> > trunk or the fixes branch pass all the tests when it is set.
> 
> We could/should probably add ENABLE_SLOW_TEST=yes to the buildbot.
> The builder recently got moved to the general sourceware GNU Toolchain
> https://builder.sourceware.org/ with configuration/sources here
> https://sourceware.org/git/builder.git

I see you already did that.
And it caught an issue on fedora-ppc64 and debian-ppc64
https://builder.sourceware.org/buildbot/#/builders/libabigail-fedora-ppc64
https://builder.sourceware.org/buildbot/#/builders/libabigail-debian-ppc64

FAIL: runtestslowselfcompare.sh
===============================
+ abidw=../tools/abidw
+ objdir=../src/.libs
+ echo ENABLE_SLOW_TEST=yes
ENABLE_SLOW_TEST=yes
+ test xyes '!=' x
++ ../tools/abidw --abidiff ../src/.libs/libabigail.so
lt-abidw: abg-symtab-reader.cc:557: void abigail::symtab_reader::symtab::update_function_entry_address_symbol_map(Elf*, GElf_Sym*, const elf_symbol_sptr&): Assertion `__abg_cond__' failed.
FAIL runtestslowselfcompare.sh (exit status: 134)

It didn't fail any other builder (so it isn't power specific, because
fedora-ppc64le is green, and it isn't big endian specific, because
fedora-s390x is green).

But it might have to do with ppc64 function descriptors, which are
ppc64 specific.

Note that it didn't sent email because this is the first build since
the migration, so the builder doesn't know yet this is unusual. It
will only sent email for new failures.

BTW. I see almost all builders do a full distcheck. Which takes a long
time on some. I like to reduce that to just one (fast) builder, since
I don't think that will catch any distro or arch specific issues and
now that we have more project making use of the GNU Toolchain buildbot
on sourceware we should try to not do unnecessary work.

That said, if there are other tests, environment variables or
configure flags that could be used to catch more issues we could
enable them (on one or more builders). As long as it doesn't take more
than 10 minutes to run the extra tests on a particular builder.

Cheers,

Mark

Re: Fwd: Testing Setup - More Tests and Automation?

[Mark Wielaard]

Hi,

On Fri, Apr 29, 2022 at 11:13:04PM +0200, Mark Wielaard wrote:
> And it caught an issue on fedora-ppc64 and debian-ppc64
> https://builder.sourceware.org/buildbot/#/builders/libabigail-fedora-ppc64
> https://builder.sourceware.org/buildbot/#/builders/libabigail-debian-ppc64
> 
> FAIL: runtestslowselfcompare.sh
> ===============================
> + abidw=../tools/abidw
> + objdir=../src/.libs
> + echo ENABLE_SLOW_TEST=yes
> ENABLE_SLOW_TEST=yes
> + test xyes '!=' x
> ++ ../tools/abidw --abidiff ../src/.libs/libabigail.so
> lt-abidw: abg-symtab-reader.cc:557: void abigail::symtab_reader::symtab::update_function_entry_address_symbol_map(Elf*, GElf_Sym*, const elf_symbol_sptr&): Assertion `__abg_cond__' failed.
> FAIL runtestslowselfcompare.sh (exit status: 134)
> 
> It didn't fail any other builder (so it isn't power specific, because
> fedora-ppc64le is green, and it isn't big endian specific, because
> fedora-s390x is green).
> 
> But it might have to do with ppc64 function descriptors, which are
> ppc64 specific.

It indeed is. The issue is in update_function_entry_address_symbol_map
which is only called for ppc64 ELFv1 binaries.

src/.libs/libabigail.so contains two identical symbols:

  468: 0000000000427498    604 FUNC    GLOBAL DEFAULT       21 _ZN7abigail10comparison12pointer_diffC2ESt10shared_ptrINS_2ir16pointer_type_defEES5_S2_INS0_4diffEES2_INS0_12diff_contextEE
 7362: 0000000000427498    604 FUNC    GLOBAL DEFAULT       21 _ZN7abigail10comparison12pointer_diffC2ESt10shared_ptrINS_2ir16pointer_type_defEES5_S2_INS0_4diffEES2_INS0_12diff_contextEE

But those are not seen as aliases. And the assert says:

      ABG_ASSERT(two_symbols_alias
                 || symbol_is_foo_and_prev_symbol_is_dot_foo);

I don't understand the aliasing code well enough to understand why
yet.

Cheers,

Mark

Re: Testing Setup - More Tests and Automation?

[Ben Woodard]

> On Apr 29, 2022, at 2:13 PM, Mark Wielaard <mark@klomp.org> wrote:
> 
> That said, if there are other tests, environment variables or
> configure flags that could be used to catch more issues we could
> enable them (on one or more builders). As long as it doesn't take more
> than 10 minutes to run the extra tests on a particular builder.

 I don’t know of any specific configure flags or environmental variables which could be added. I believe that some additional regression tests are needed. 

I have a WIP patch that I haven’t gotten working yet. It amounts to:

diff --git a/tests/runtestfedabipkgdiff.py.in b/tests/runtestfedabipkgdiff.py.in
index dd4e041d..3ce6878d 100755
--- a/tests/runtestfedabipkgdiff.py.in
+++ b/tests/runtestfedabipkgdiff.py.in
@@ -80,6 +80,35 @@ FEDABIPKGDIFF_TEST_SPECS = [
     (['--self-compare', '-a', '--from', 'fc23', 'dbus-glib'],
      'data/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-report-0.txt',
      'output/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'aspell'],
+     'data/test-fedabipkgdiff/test8-self-compare-from-fc36-aspell-report-0.txt',
+     'output/test-fedabipkgdiff/test8-self-compare-from-fc36-aspell-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'dyninst'],
+     'data/test-fedabipkgdiff/test9-self-compare-from-fc36-dyninst-report-0.txt',
+     'output/test-fedabipkgdiff/test9-self-compare-from-fc36-dyninst-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'libabigail'],
+     'data/test-fedabipkgdiff/test10-self-compare-from-fc36-libabigail-report-0.txt',
+     'output/test-fedabipkgdiff/test10-self-compare-from-fc36-libabigail-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'hdf5'],
+     'data/test-fedabipkgdiff/test11-self-compare-from-fc36-hdf5-report-0.txt',
+     'output/test-fedabipkgdiff/test11-self-compare-from-fc36-hdf5-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'gcc'],
+     'data/test-fedabipkgdiff/test12-self-compare-from-fc36-gcc-report-0.txt',
+     'output/test-fedabipkgdiff/test12-self-compare-from-fc36-gcc-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'openmpi'],
+     'data/test-fedabipkgdiff/test13-self-compare-from-fc36-openmpi-report-0.txt',
+     'output/test-fedabipkgdiff/test13-self-compare-from-fc36-openmpi-report-0.txt'),
+
+    (['--self-compare', '-a', '--from', 'fc36', 'protobuf'],
+     'data/test-fedabipkgdiff/test14-self-compare-from-fc36-protobuf-report-0.txt',
+     'output/test-fedabipkgdiff/test14-self-compare-from-fc36-protobuf-report-0.txt'),
+
 ]

Along with the expected results in the data directory. These are the ones which most frequently seem to break. Other than getting the patch to actually work, I believe that for it to be acceptable to Dodji it would need to be lumped under the slow tests and I haven’t figured out how to do that yet. Python is not my best language.

So if we could have one of the builders run something like:

for i in aspell dyninst libabigail hdf5 gcc openmpi proj protobuf; do
   fedabipkgdiff --self-compare -a --from fc36 $i >/dev/null
  if [ $? != 0 ];then
    exit 1
  fi
done

Then I think that we’d catch many of the problems that I have been seeing.

Notes:
- Having libabigail in the list may be redundant 
- GCC is in there because libstdc++ tends to break things. I haven’t seen a problem with gcc itself.
- When sharing the diff, I realized that I forgot to add ‘proj’
- Note that these are all C++. libabigail rarely fails on C these days. However, the HPC apps that I care most about are all C++ and Fortran.

-ben