From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id BF3453858C2C for ; Fri, 29 Apr 2022 21:13:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org BF3453858C2C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: by gnu.wildebeest.org (Postfix, from userid 1000) id CA94A302BBED; Fri, 29 Apr 2022 23:13:04 +0200 (CEST) Date: Fri, 29 Apr 2022 23:13:04 +0200 From: Mark Wielaard To: Ben Woodard Cc: Ben Woodard via Libabigail Subject: Re: Fwd: Testing Setup - More Tests and Automation? Message-ID: <20220429211304.GC7305@gnu.wildebeest.org> References: <34E58964-E930-4DF9-87CD-18D4C63DBCEB@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2022 21:13:07 -0000 Hi Ben, On Thu, Apr 28, 2022 at 10:06:59AM +0200, Mark Wielaard wrote: > On Wed, Apr 27, 2022 at 05:55:07PM -0700, Ben Woodard via Libabigail wrote: > > I personally feel like we need to reconsider > > https://sourceware.org/pipermail/libabigail/2022q1/004045.html and > > have checks like this built into the normal “make check” having > > ENABLE_SLOW_TEST=no by default allows too many bugs to creep > > in. People don’t use it. As a case in point neither the current > > trunk or the fixes branch pass all the tests when it is set. > > We could/should probably add ENABLE_SLOW_TEST=yes to the buildbot. > The builder recently got moved to the general sourceware GNU Toolchain > https://builder.sourceware.org/ with configuration/sources here > https://sourceware.org/git/builder.git I see you already did that. And it caught an issue on fedora-ppc64 and debian-ppc64 https://builder.sourceware.org/buildbot/#/builders/libabigail-fedora-ppc64 https://builder.sourceware.org/buildbot/#/builders/libabigail-debian-ppc64 FAIL: runtestslowselfcompare.sh =============================== + abidw=../tools/abidw + objdir=../src/.libs + echo ENABLE_SLOW_TEST=yes ENABLE_SLOW_TEST=yes + test xyes '!=' x ++ ../tools/abidw --abidiff ../src/.libs/libabigail.so lt-abidw: abg-symtab-reader.cc:557: void abigail::symtab_reader::symtab::update_function_entry_address_symbol_map(Elf*, GElf_Sym*, const elf_symbol_sptr&): Assertion `__abg_cond__' failed. FAIL runtestslowselfcompare.sh (exit status: 134) It didn't fail any other builder (so it isn't power specific, because fedora-ppc64le is green, and it isn't big endian specific, because fedora-s390x is green). But it might have to do with ppc64 function descriptors, which are ppc64 specific. Note that it didn't sent email because this is the first build since the migration, so the builder doesn't know yet this is unusual. It will only sent email for new failures. BTW. I see almost all builders do a full distcheck. Which takes a long time on some. I like to reduce that to just one (fast) builder, since I don't think that will catch any distro or arch specific issues and now that we have more project making use of the GNU Toolchain buildbot on sourceware we should try to not do unnecessary work. That said, if there are other tests, environment variables or configure flags that could be used to catch more issues we could enable them (on one or more builders). As long as it doesn't take more than 10 minutes to run the extra tests on a particular builder. Cheers, Mark