Re: Buildbot failure in Wildebeest Builder on whole buildset

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

Hi,

On Tue, Sep 07, 2021 at 11:34:16AM +0300, Dmitry V. Levin wrote:
> On Tue, Sep 07, 2021 at 02:12:18AM +0200, Mark Wielaard wrote:
> [...]
> > Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle
> > 
> > At startup the debuginfod server does a scan and groom cycle. Make
> > sure to wait for that before triggering a new one with SIGUSR1 or
> > SIGURST2.
> 
> Did you mean SIGUSR2?

Sigh. Yes, that was a typo. Sorry.

It seems this did help with the debuginfod testcases. But we were
still seeing some failures on some of the buildbots because it wasn't
early enough. We really should wait for the initial scan (and possibly
groom) cycle to be done before we make any changes to the scan
directories. Otherwise only part of them might be scanned (or groomed)
at the start (especially because some tests populate the scan dirs and
then remove some files, which aren't supported, again).

The attached (pushed) fixes that hopefully.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Make-sure-to-wait-for-initial-scan-and-groom-c.patch --]
[-- Type: text/x-diff, Size: 8771 bytes --]

From 5ce85e424907d82797f5efe59a8416ec537ebd9e Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Tue, 7 Sep 2021 23:56:42 +0200
Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle
 earlier

At startup the debuginfod server does a scan and groom cycle. Make
sure to wait for that before making any changes to the scan dirs.  And
not just right before triggering a new one with SIGUSR1 for scan or
SIGURS2 for groom.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog                          | 12 ++++++++++++
 tests/run-debuginfod-archive-groom.sh    | 10 +++++++---
 tests/run-debuginfod-archive-rename.sh   | 11 +++++++----
 tests/run-debuginfod-artifact-running.sh | 12 +++++++-----
 tests/run-debuginfod-dlopen.sh           |  7 ++++---
 tests/run-debuginfod-extraction.sh       |  6 ++++--
 tests/run-debuginfod-federation-link.sh  |  3 +--
 7 files changed, 42 insertions(+), 19 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index 61080c52..42989d19 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,15 @@
+2021-09-07  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-archive-groom.sh: Wait for initial scan and groom
+	cycle before making any changes to the scan dirs.
+	* run-debuginfod-archive-rename.sh: Likewise.
+	* run-debuginfod-artifact-running.sh: Wait for initial scan cycle
+	before making any changes to the scan dirs.
+	* run-debuginfod-dlopen.sh: Likewise.
+	* run-debuginfod-extraction.sh: Likewise.
+	* run-debuginfod-federation-link.sh: Don't wait twice for the initial
+	scan.
+
 2021-09-06  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-archive-groom.sh: Wait for initial scan and groom
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
index 1e7bd67a..7813ee28 100755
--- a/tests/run-debuginfod-archive-groom.sh
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -46,6 +46,13 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# wait till the initial scan is done before triggering a new one
+# and before dropping new file into the scan dirs
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+# Same for the initial groom cycle, we don't want it to be done
+# half way initializing the file setup
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+
 # Build a non-stripped binary
 echo "int main() { return 0; }" > ${PWD}/F/prog.c
 gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
@@ -58,8 +65,6 @@ if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
 
-# wait till the initial scan is done before triggering a new one
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Now there should be 1 files in the index
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
@@ -140,7 +145,6 @@ RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
 # debuginfod has forgotten them, but remembers others
 rm -r R/debuginfod-rpms/rhel6/*
 
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 kill -USR2 $PID1  # groom cycle
 ## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
 wait_ready $PORT1 'thread_work_total{role="groom"}' 2
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
index 4fc1b441..5369949b 100755
--- a/tests/run-debuginfod-archive-rename.sh
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -44,13 +44,18 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# wait till the initial scan is done before triggering a new one
+# and before dropping new file into the scan dirs
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+# Same for the initial groom cycle, we don't want it to be done
+# half way initializing the file setup
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+
 cp -rvp ${abs_srcdir}/debuginfod-rpms R
 if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
 fi
 
-# Make sure the initial scan is done
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Now there should be 1 files in the index
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
@@ -65,8 +70,6 @@ SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
 # there are two copies of the same buildid in the index, one for the
 # no-longer-existing file name, and one under the new name.
 
-# Make sure the initial groom cycle has been done
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
 # run a groom cycle to force server to drop its fdcache
 kill -USR2 $PID1  # groom cycle
 wait_ready $PORT1 'thread_work_total{role="groom"}' 2
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
index 4eae0200..51fa9c0a 100755
--- a/tests/run-debuginfod-artifact-running.sh
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -42,6 +42,13 @@ env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d
 PID1=$!
 tempfiles vlog$PORT1
 errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# And the initial scan should have been done before moving
+# files under the scan dirs.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 mv prog F
 mv prog.debug F
 tempfiles prog/F
@@ -49,11 +56,6 @@ tempfiles prog/F
 # Be patient when run on a busy machine things might take a bit.
 export DEBUGINFOD_TIMEOUT=10
 
-# Server must become ready
-wait_ready $PORT1 'ready' 1
-# And the initial scan should have been done
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-
 kill -USR1 $PID1
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
index 5f33394a..39ee5190 100755
--- a/tests/run-debuginfod-dlopen.sh
+++ b/tests/run-debuginfod-dlopen.sh
@@ -51,6 +51,10 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# Make sure the initial scan has finished.
+# Before moving files under the scan dirs.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 # We use -t0 and -g0 here to turn off time-based scanning & grooming.
 # For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
 
@@ -71,9 +75,6 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
 mv p+r%o\$g F
 mv p+r%o\$g.debug F
 
-# Make sure the initial scan has finished.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-
 kill -USR1 $PID1
 # Wait till both files are in the index.
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
diff --git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
index 4750f184..06f60e78 100755
--- a/tests/run-debuginfod-extraction.sh
+++ b/tests/run-debuginfod-extraction.sh
@@ -47,6 +47,10 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom
 ps -q $PID1 -e -L -o '%p %c %a' | grep scan
 ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
 
+# Make sure the initial scan has finished before copying the new files in
+# We might remove some, which we don't want to be accidentially scanned.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
 cp -rvp ${abs_srcdir}/debuginfod-rpms R
 if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
     rm -vrf R/debuginfod-rpms/fedora31
@@ -54,8 +58,6 @@ fi
 
 cp -rvp ${abs_srcdir}/debuginfod-tars Z
 
-# Make sure the initial scan has finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Wait till both files are in the index and scan/index fully finished
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
index 1aef7174..050bcbcf 100755
--- a/tests/run-debuginfod-federation-link.sh
+++ b/tests/run-debuginfod-federation-link.sh
@@ -43,6 +43,7 @@ tempfiles vlog$PORT1
 errfiles vlog$PORT1
 
 wait_ready $PORT1 'ready' 1
+# Make sure initial scan was done
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
 wait_ready $PORT1 'thread_busy{role="scan"}' 0
@@ -63,8 +64,6 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
 mv prog F
 mv prog.debug F
 
-# Make sure initial scan was done
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
 kill -USR1 $PID1
 # Wait till both files are in the index.
 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
-- 
2.32.0