Re: Fwd: Testing Setup - More Tests and Automation?

[Mark Wielaard <mark@klomp.org>]

Hi Ben,

On Thu, Apr 28, 2022 at 10:06:59AM +0200, Mark Wielaard wrote:
> On Wed, Apr 27, 2022 at 05:55:07PM -0700, Ben Woodard via Libabigail wrote:
> > I personally feel like we need to reconsider
> > https://sourceware.org/pipermail/libabigail/2022q1/004045.html and
> > have checks like this built into the normal “make check” having
> > ENABLE_SLOW_TEST=no by default allows too many bugs to creep
> > in. People don’t use it. As a case in point neither the current
> > trunk or the fixes branch pass all the tests when it is set.
> 
> We could/should probably add ENABLE_SLOW_TEST=yes to the buildbot.
> The builder recently got moved to the general sourceware GNU Toolchain
> https://builder.sourceware.org/ with configuration/sources here
> https://sourceware.org/git/builder.git

I see you already did that.
And it caught an issue on fedora-ppc64 and debian-ppc64
https://builder.sourceware.org/buildbot/#/builders/libabigail-fedora-ppc64
https://builder.sourceware.org/buildbot/#/builders/libabigail-debian-ppc64

FAIL: runtestslowselfcompare.sh
===============================
+ abidw=../tools/abidw
+ objdir=../src/.libs
+ echo ENABLE_SLOW_TEST=yes
ENABLE_SLOW_TEST=yes
+ test xyes '!=' x
++ ../tools/abidw --abidiff ../src/.libs/libabigail.so
lt-abidw: abg-symtab-reader.cc:557: void abigail::symtab_reader::symtab::update_function_entry_address_symbol_map(Elf*, GElf_Sym*, const elf_symbol_sptr&): Assertion `__abg_cond__' failed.
FAIL runtestslowselfcompare.sh (exit status: 134)

It didn't fail any other builder (so it isn't power specific, because
fedora-ppc64le is green, and it isn't big endian specific, because
fedora-s390x is green).

But it might have to do with ppc64 function descriptors, which are
ppc64 specific.

Note that it didn't sent email because this is the first build since
the migration, so the builder doesn't know yet this is unusual. It
will only sent email for new failures.

BTW. I see almost all builders do a full distcheck. Which takes a long
time on some. I like to reduce that to just one (fast) builder, since
I don't think that will catch any distro or arch specific issues and
now that we have more project making use of the GNU Toolchain buildbot
on sourceware we should try to not do unnecessary work.

That said, if there are other tests, environment variables or
configure flags that could be used to catch more issues we could
enable them (on one or more builders). As long as it doesn't take more
than 10 minutes to run the extra tests on a particular builder.

Cheers,

Mark