Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

Hi,

On Mon, Sep 06, 2021 at 05:07:50PM +0200, Mark Wielaard wrote:
> On Sat, 2021-09-04 at 01:47 +0200, Mark Wielaard wrote:
> > On Fri, Sep 03, 2021 at 02:08:40PM +0000, 
> > buildbot@builder.wildebeest.org wrote:
> > > The Buildbot has detected a new failure on builder elfutils-debian-
> > > armhf while building elfutils.
> > > Full details are available at:
> > >     https://builder.wildebeest.org/buildbot/#builders/15/builds/603
> > > 
> > > Buildbot URL: https://builder.wildebeest.org/buildbot/
> > > 
> > > Worker for this Build: debian-armhf
> > > 
> > > Build Reason: <unknown>
> > > Blamelist: Mark Wielaard <mark@klomp.org>
> > > 
> > > BUILD FAILED: failed test (failure)
> > 
> > So some of the builders did pass without any failures. But some had
> > errors, and not all the same ones. I couldn't replicate locally
> > (everything was green) both with make check and make distcheck.
> > 
> > But looking at the tests I noticed not all used a clean new cache and
> > some used the default database.
> > 
> > So hopefully the attached patch makes things a bit more stable by
> > explicitly setting the DEBUGINFOD_CACHE_PATH and invoke debuginfod
> > with a fresh sqlite database.
> 
> Sadly we still have failures even with this patch:
> https://builder.wildebeest.org/buildbot/#/changes/2567
> 
> The good news is that elfutils-fedora-x86_64, elfutils-fedora-ppc64le,
> elfutils-fedora-s390x, elfutils-debian-i386, elfutils-fedora-ppc64 and
> elfutils-debian-armhf are fully green, but...
> 
> On elfutils-centos-x86_64 make check fails with:
> 
> FAIL: run-debuginfod-tmp-home.sh
> FAIL: run-debuginfod-writable.sh
> 
> On elfutils-debian-arm64 make check fails with:
> 
> FAIL: run-debuginfod-federation-sqlite.sh
> 
> On elfutils-centos-x86_64 make check first timed out (!?!)
> then it failed with:
> 
> FAIL: run-debuginfod-tmp-home.sh
> FAIL: run-debuginfod-writable.sh
> 
> On elfutils-debian-amd64 make check failed with:
> 
> FAIL: run-debuginfod-archive-groom.sh
> 
> The failures still look somewhat random (although less random than
> before). I don't yet know why.

This doesn't really explain the above failures, but there were still
two tests that didn't use their own debuginfod client cache. Also one
set a useless, but confusing DEBUGINFOD_URLS.

Still looking for explanations of the other FAILS (which I cannot
reproduce locally).

Cheers,

Mark

[-- Attachment #2: 0001-tests-Set-DEBUGINFOD_CACHE_PATH-for-run-debuginfod-f.patch --]
[-- Type: text/x-diff, Size: 2926 bytes --]

From 44a36e39fc7df2a3e2f9f278f79f65233d774a48 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Mon, 6 Sep 2021 21:48:38 +0200
Subject: [PATCH] tests: Set DEBUGINFOD_CACHE_PATH for
 run-debuginfod-{file,query-retry}.sh

---
 tests/ChangeLog                     | 7 +++++++
 tests/run-debuginfod-file.sh        | 6 +++---
 tests/run-debuginfod-query-retry.sh | 2 +-
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 178697bb..c1760877 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,10 @@
+2021-09-06  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-file.sh: Set DEBUGINFOD_CACHE_PATH. Export
+	correct DEBUGINFOD_URLS.
+	* run-debuginfod-query-retry.sh: Add DEBUGINFOD_CACHE_PATH
+	to env.
+
 2021-09-03  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-000-permission.sh: Set DEBUGINFOD_CACHE_PATH
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
index 22c956dd..341bbc68 100755
--- a/tests/run-debuginfod-file.sh
+++ b/tests/run-debuginfod-file.sh
@@ -22,7 +22,7 @@
 set -x
 unset VALGRIND_CMD
 
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/   # or without trailing /
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
 
 # Test fetching a file using file:// . No debuginfod server needs to be run for
 # this test.
@@ -30,11 +30,11 @@ local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/sourc
 mkdir -p ${local_dir}
 echo "int main() { return 0; }" > ${local_dir}/main.c
 # first test that is doesn't work, when no DEBUGINFOD_URLS is set
-DEBUGINFOD_URLS=""
+export DEBUGINFOD_URLS=""
 testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
 
 # Now test is with proper DEBUGINFOD_URLS
-DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+export DEBUGINFOD_URLS="file://${PWD}/mocktree/"
 filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
 cmp $filename ${local_dir}/main.c
 
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
index 3c5542d5..c9192510 100755
--- a/tests/run-debuginfod-query-retry.sh
+++ b/tests/run-debuginfod-query-retry.sh
@@ -24,7 +24,7 @@ unset VALGRIND_CMD
 
 ########################################################################
 # set up tests for retrying failed queries.
-retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache \
         ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
         | grep -c 'Retry failed query'`
 if [ $retry_attempts -ne 10 ]; then
-- 
2.32.0