Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

Hi,

On Fri, Aug 27, 2021 at 11:19:14PM +0000, buildbot@builder.wildebeest.org wrote:
> The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils.
> Full details are available at:
>     https://builder.wildebeest.org/buildbot/#builders/2/builds/803
> 
> Buildbot URL: https://builder.wildebeest.org/buildbot/
> 
> Worker for this Build: debian-amd64
> 
> Build Reason: <unknown>
> Blamelist: Di Chen <dichen@redhat.com>
> 
> BUILD FAILED: failed test (failure)

This took me a while to figure out because it does not consistently
fail.  It turns out that sharing a database between two debuginfod
instances (especially if they forward queries to each other) is not a
good idea.

Hopefully fixed with the attached patch.

At first I had hoped that using -d :memory: would work. But debuginfod
doesn't really support in memory databases like that. It opens the
database twice in read/write and read-only mode. But :memory: is
special and isn't always a new database, so the read-only copy doesn't
see what is in the read/write copy.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Use-fresh-separate-databases-for-debuginfd-for.patch --]
[-- Type: text/x-diff, Size: 2123 bytes --]

From 17a9b1303e533c13aac6550844bdd68c669091bf Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 28 Aug 2021 15:54:18 +0200
Subject: [PATCH] tests: Use fresh separate databases for debuginfd
 forwarded-ttl-limit

Sharing the database between the two debuginfod instances that forward
queries to each other causes issues. Make both debuginfod instances
use a new fresh database.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog              | 5 +++++
 tests/run-debuginfod-find.sh | 7 +++++--
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 29c48b97..cbd1c227 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2021-08-28  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-find.sh: Use clean, separate databases for
+	forwarded-ttl-limit tests.
+
 2021-08-20  Di Chen  <dichen@redhat.com>
 
 	* run-debuginfod-find.sh: Add test for X-Forwarded-For hops limit.
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
index 7e12dd7f..5d38d625 100755
--- a/tests/run-debuginfod-find.sh
+++ b/tests/run-debuginfod-find.sh
@@ -824,10 +824,13 @@ done
 tempfiles vlog$PORT4 vlog$PORT5
 errfiles vlog$PORT4 vlog$PORT5
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
+# Give each debuginfd its own clean database.
+tempfiles db.$PORT4.sql db.$PORT5.sql
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT4.sql --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
 PID5=$!
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT5.sql --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
 PID6=$!
 
 wait_ready $PORT4 'ready' 1
-- 
2.32.0