From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (wildebeest.demon.nl [212.238.236.112]) by sourceware.org (Postfix) with ESMTPS id C83E83858023 for ; Tue, 7 Sep 2021 22:04:27 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C83E83858023 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from reform (deer0x11.wildebeest.org [172.31.17.147]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 461A3301FE93; Wed, 8 Sep 2021 00:04:26 +0200 (CEST) Received: by reform (Postfix, from userid 1000) id F2C9E2E807A6; Wed, 8 Sep 2021 00:04:25 +0200 (CEST) Date: Wed, 8 Sep 2021 00:04:25 +0200 From: Mark Wielaard To: "Dmitry V. Levin" Cc: elfutils-devel@sourceware.org Subject: Re: Buildbot failure in Wildebeest Builder on whole buildset Message-ID: References: <20210903140840.A291B8026A2@builder.wildebeest.org> <31a5a4ff4a158f1e794a28defc30eb14423208ac.camel@klomp.org> <20210907083416.GA27352@altlinux.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="cNckHbPrbJ1TshU9" Content-Disposition: inline In-Reply-To: <20210907083416.GA27352@altlinux.org> X-Spam-Status: No, score=-11.1 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2021 22:04:32 -0000 --cNckHbPrbJ1TshU9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, On Tue, Sep 07, 2021 at 11:34:16AM +0300, Dmitry V. Levin wrote: > On Tue, Sep 07, 2021 at 02:12:18AM +0200, Mark Wielaard wrote: > [...] > > Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle > > > > At startup the debuginfod server does a scan and groom cycle. Make > > sure to wait for that before triggering a new one with SIGUSR1 or > > SIGURST2. > > Did you mean SIGUSR2? Sigh. Yes, that was a typo. Sorry. It seems this did help with the debuginfod testcases. But we were still seeing some failures on some of the buildbots because it wasn't early enough. We really should wait for the initial scan (and possibly groom) cycle to be done before we make any changes to the scan directories. Otherwise only part of them might be scanned (or groomed) at the start (especially because some tests populate the scan dirs and then remove some files, which aren't supported, again). The attached (pushed) fixes that hopefully. Cheers, Mark --cNckHbPrbJ1TshU9 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-tests-Make-sure-to-wait-for-initial-scan-and-groom-c.patch" >From 5ce85e424907d82797f5efe59a8416ec537ebd9e Mon Sep 17 00:00:00 2001 From: Mark Wielaard Date: Tue, 7 Sep 2021 23:56:42 +0200 Subject: [PATCH] tests: Make sure to wait for initial scan and groom cycle earlier At startup the debuginfod server does a scan and groom cycle. Make sure to wait for that before making any changes to the scan dirs. And not just right before triggering a new one with SIGUSR1 for scan or SIGURS2 for groom. Signed-off-by: Mark Wielaard --- tests/ChangeLog | 12 ++++++++++++ tests/run-debuginfod-archive-groom.sh | 10 +++++++--- tests/run-debuginfod-archive-rename.sh | 11 +++++++---- tests/run-debuginfod-artifact-running.sh | 12 +++++++----- tests/run-debuginfod-dlopen.sh | 7 ++++--- tests/run-debuginfod-extraction.sh | 6 ++++-- tests/run-debuginfod-federation-link.sh | 3 +-- 7 files changed, 42 insertions(+), 19 deletions(-) diff --git a/tests/ChangeLog b/tests/ChangeLog index 61080c52..42989d19 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -1,3 +1,15 @@ +2021-09-07 Mark Wielaard + + * run-debuginfod-archive-groom.sh: Wait for initial scan and groom + cycle before making any changes to the scan dirs. + * run-debuginfod-archive-rename.sh: Likewise. + * run-debuginfod-artifact-running.sh: Wait for initial scan cycle + before making any changes to the scan dirs. + * run-debuginfod-dlopen.sh: Likewise. + * run-debuginfod-extraction.sh: Likewise. + * run-debuginfod-federation-link.sh: Don't wait twice for the initial + scan. + 2021-09-06 Mark Wielaard * run-debuginfod-archive-groom.sh: Wait for initial scan and groom diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh index 1e7bd67a..7813ee28 100755 --- a/tests/run-debuginfod-archive-groom.sh +++ b/tests/run-debuginfod-archive-groom.sh @@ -46,6 +46,13 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom ps -q $PID1 -e -L -o '%p %c %a' | grep scan ps -q $PID1 -e -L -o '%p %c %a' | grep traverse +# wait till the initial scan is done before triggering a new one +# and before dropping new file into the scan dirs +wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 +# Same for the initial groom cycle, we don't want it to be done +# half way initializing the file setup +wait_ready $PORT1 'thread_work_total{role="groom"}' 1 + # Build a non-stripped binary echo "int main() { return 0; }" > ${PWD}/F/prog.c gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c @@ -58,8 +65,6 @@ if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones rm -vrf R/debuginfod-rpms/fedora31 fi -# wait till the initial scan is done before triggering a new one -wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 kill -USR1 $PID1 # Now there should be 1 files in the index wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 @@ -140,7 +145,6 @@ RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir # debuginfod has forgotten them, but remembers others rm -r R/debuginfod-rpms/rhel6/* -wait_ready $PORT1 'thread_work_total{role="groom"}' 1 kill -USR2 $PID1 # groom cycle ## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2 wait_ready $PORT1 'thread_work_total{role="groom"}' 2 diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh index 4fc1b441..5369949b 100755 --- a/tests/run-debuginfod-archive-rename.sh +++ b/tests/run-debuginfod-archive-rename.sh @@ -44,13 +44,18 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom ps -q $PID1 -e -L -o '%p %c %a' | grep scan ps -q $PID1 -e -L -o '%p %c %a' | grep traverse +# wait till the initial scan is done before triggering a new one +# and before dropping new file into the scan dirs +wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 +# Same for the initial groom cycle, we don't want it to be done +# half way initializing the file setup +wait_ready $PORT1 'thread_work_total{role="groom"}' 1 + cp -rvp ${abs_srcdir}/debuginfod-rpms R if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones rm -vrf R/debuginfod-rpms/fedora31 fi -# Make sure the initial scan is done -wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 kill -USR1 $PID1 # Now there should be 1 files in the index wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 @@ -65,8 +70,6 @@ SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1 # there are two copies of the same buildid in the index, one for the # no-longer-existing file name, and one under the new name. -# Make sure the initial groom cycle has been done -wait_ready $PORT1 'thread_work_total{role="groom"}' 1 # run a groom cycle to force server to drop its fdcache kill -USR2 $PID1 # groom cycle wait_ready $PORT1 'thread_work_total{role="groom"}' 2 diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh index 4eae0200..51fa9c0a 100755 --- a/tests/run-debuginfod-artifact-running.sh +++ b/tests/run-debuginfod-artifact-running.sh @@ -42,6 +42,13 @@ env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d PID1=$! tempfiles vlog$PORT1 errfiles vlog$PORT1 + +# Server must become ready +wait_ready $PORT1 'ready' 1 +# And the initial scan should have been done before moving +# files under the scan dirs. +wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 + mv prog F mv prog.debug F tempfiles prog/F @@ -49,11 +56,6 @@ tempfiles prog/F # Be patient when run on a busy machine things might take a bit. export DEBUGINFOD_TIMEOUT=10 -# Server must become ready -wait_ready $PORT1 'ready' 1 -# And the initial scan should have been done -wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 - kill -USR1 $PID1 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0 diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh index 5f33394a..39ee5190 100755 --- a/tests/run-debuginfod-dlopen.sh +++ b/tests/run-debuginfod-dlopen.sh @@ -51,6 +51,10 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom ps -q $PID1 -e -L -o '%p %c %a' | grep scan ps -q $PID1 -e -L -o '%p %c %a' | grep traverse +# Make sure the initial scan has finished. +# Before moving files under the scan dirs. +wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 + # We use -t0 and -g0 here to turn off time-based scanning & grooming. # For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process. @@ -71,9 +75,6 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \ mv p+r%o\$g F mv p+r%o\$g.debug F -# Make sure the initial scan has finished. -wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 - kill -USR1 $PID1 # Wait till both files are in the index. wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 diff --git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh index 4750f184..06f60e78 100755 --- a/tests/run-debuginfod-extraction.sh +++ b/tests/run-debuginfod-extraction.sh @@ -47,6 +47,10 @@ ps -q $PID1 -e -L -o '%p %c %a' | grep groom ps -q $PID1 -e -L -o '%p %c %a' | grep scan ps -q $PID1 -e -L -o '%p %c %a' | grep traverse +# Make sure the initial scan has finished before copying the new files in +# We might remove some, which we don't want to be accidentially scanned. +wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 + cp -rvp ${abs_srcdir}/debuginfod-rpms R if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones rm -vrf R/debuginfod-rpms/fedora31 @@ -54,8 +58,6 @@ fi cp -rvp ${abs_srcdir}/debuginfod-tars Z -# Make sure the initial scan has finished -wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 kill -USR1 $PID1 # Wait till both files are in the index and scan/index fully finished wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh index 1aef7174..050bcbcf 100755 --- a/tests/run-debuginfod-federation-link.sh +++ b/tests/run-debuginfod-federation-link.sh @@ -43,6 +43,7 @@ tempfiles vlog$PORT1 errfiles vlog$PORT1 wait_ready $PORT1 'ready' 1 +# Make sure initial scan was done wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0 wait_ready $PORT1 'thread_busy{role="scan"}' 0 @@ -63,8 +64,6 @@ BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \ mv prog F mv prog.debug F -# Make sure initial scan was done -wait_ready $PORT1 'thread_work_total{role="traverse"}' 1 kill -USR1 $PID1 # Wait till both files are in the index. wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 -- 2.32.0 --cNckHbPrbJ1TshU9--