From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id AFC5B3856DE7 for ; Fri, 27 May 2022 16:20:50 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org AFC5B3856DE7 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from tarox.wildebeest.org (83-87-18-245.cable.dynamic.v4.ziggo.nl [83.87.18.245]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 1CA85302FB8A; Fri, 27 May 2022 18:20:47 +0200 (CEST) Received: by tarox.wildebeest.org (Postfix, from userid 1000) id 366434000CBC; Fri, 27 May 2022 18:20:47 +0200 (CEST) Message-ID: Subject: Re: [Bug debuginfod/28577] Make run-...-fd-prefetch-caches.sh test something From: Mark Wielaard To: Noah Sanci , elfutils-devel@sourceware.org Date: Fri, 27 May 2022 18:20:47 +0200 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Evolution 3.28.5 (3.28.5-10.el7) Mime-Version: 1.0 X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_BADIPHTTP, KAM_DMARC_STATUS, KAM_LOTSOFHASH, NUMERIC_HTTP_ADDR, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 May 2022 16:20:53 -0000 Hi Noah, On Wed, 2022-05-25 at 14:44 -0400, Noah Sanci via Elfutils-devel wrote: > PR28577: Make run-debuginfod-fd-prefetch-caches.sh test > something >=20 > Update to the run-debuginfod-fd-prefetch to make the test more sound. Thanks. And according to Evgeny this makes the test pass for him: https://sourceware.org/bugzilla/show_bug.cgi?id=3D29180#c2 But I find it a little hard to review, it would have been helpful to get a bit more comments/documentation what exactly is being tested now. > diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run- > debuginfod-fd-prefetch-caches.sh > index 66a7a083..50ffc4ac 100755 > --- a/tests/run-debuginfod-fd-prefetch-caches.sh > +++ b/tests/run-debuginfod-fd-prefetch-caches.sh > @@ -21,22 +21,22 @@ > # for test case debugging, uncomment: > set -x > unset VALGRIND_CMD > +mkdir R Z OK, some boilerplate. > -mkdir R > -cp -rvp ${abs_srcdir}/debuginfod-rpms R > -if [ "$zstd" =3D "false" ]; then # nuke the zstd fedora 31 ones > - rm -vrf R/debuginfod-rpms/fedora31 > -fi This gets moved to below. > -FDCACHE_MBS=3D$((1/1024/1024)) > +FDCACHE_MBS=3D1 > FDCACHE_FDS=3D1 > -PREFETCH_MBS=3D$((1/1024/1024)) > -PREFETCH_FDS=3D2 > +PREFETCH_MBS=3D1 > +PREFETCH_FDS=3D1 > PREFETCH=3D1 OK, new presets. > # This variable is essential and ensures no time-race for claiming > ports occurs > # set base to a unique multiple of 100 not used in any other 'run- > debuginfod-*' test > base=3D8800 > get_ports > +cp -rvp ${abs_srcdir}/debuginfod-rpms R > +if [ "$zstd" =3D "false" ]; then # nuke the zstd fedora 31 ones > + rm -vrf R/debuginfod-rpms/fedora31 > +fi > +cp -rvp ${abs_srcdir}/debuginfod-tars Z OK, put rpms under R, tars under Z. =20 > DB=3D${PWD}/.debuginfod_tmp.sqlite > tempfiles $DB > @@ -47,88 +47,68 @@ export DEBUGINFOD_CACHE_PATH=3D${PWD}/.client_cache > ###### > rm -rf $DEBUGINFOD_CACHE_PATH > rm -rf $DB > + > # Set mb values to ensure the caches certainly have enough space > # To store the test files > env LD_LIBRARY_PATH=3D$ldpath DEBUGINFOD_URLS=3D ${abs_builddir}/../debu= ginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \ > - --fdcache-fds=3D$FDCACHE_FDS --fdcache-prefetch-fds=3D$PREFETCH_FDS = --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \ > - --fdcache-mbs=3D50 --fdcache-prefetch-mbs=3D50 \ > - --fdcache-prefetch=3D$PREFETCH > vlog$PORT1 2>&1 & > + --fdcache-fds=3D$FDCACHE_FDS --fdcache-prefetch-fds=3D$PREFETCH_FDS = -vvvvv -g 0 -t 0 -R R \ > + -Z .tar.bz2=3Dbzcat Z --fdcache-mbs=3D50 --fdcache-prefetch-mbs=3D= 100 \ > + --fdcache-prefetch=3D$PREFETCH -F > vlog$PORT1 2>&1 & Now run with --fdcache-fds=3D1 --fdcache-mbs=3D50 and -fdcache-prefetch-fds=3D1 fdcache-prefetch=3D1 -fdcache-prefetch-mbs=3D100 It is these last 3 we want to test. > PID1=3D$! > tempfiles vlog$PORT1 > errfiles vlog$PORT1 > + > # Server must become ready > wait_ready $PORT1 'ready' 1 > ######################################################################## > -cp -rvp ${abs_srcdir}/debuginfod-rpms R > -if [ "$zstd" =3D "false" ]; then # nuke the zstd fedora 31 ones > - rm -vrf R/debuginfod-rpms/fedora31 > -fi > kill -USR1 $PID1 > wait_ready $PORT1 'thread_work_total{role=3D"traverse"}' 1 > +wait_ready $PORT1 'thread_work_pending{role=3D"scan"}' 0 > +wait_ready $PORT1 'thread_busy{role=3D"scan"}' 0 OK, boilerplate to wait for initial scan. > # All rpms need to be in the index, except the dummy permission-000 one > rpms=3D$(find R -name \*rpm | grep -v nothing | wc -l) > wait_ready $PORT1 'scanned_files_total{source=3D".rpm archive"}' $rpms > -kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug-= >dwz->srefs > -# Wait till both files are in the index and scan/index fully finished > +kill -USR1 $PID1=20 > wait_ready $PORT1 'thread_work_total{role=3D"traverse"}' 2 > -export DEBUGINFOD_URLS=3D"http://127.0.0.1:"$PORT1 > +wait_ready $PORT1 'thread_work_pending{role=3D"scan"}' 0 > +wait_ready $PORT1 'thread_busy{role=3D"scan"}' 0 Do another scan and wait for it to be ready. > -archive_hits=3D5 > -SHA=3Df4a1a8062be998ae93b8f1cd744a398c6de6dbb1 > -for i in $archive_hits > -do > - archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/h= ello2-1.0-2.x86_64/hello.c $SHA > -done > +export DEBUGINFOD_URLS=3Dhttp://127.0.0.1:$PORT1/ > +# load prefetch cache > +testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo cee13b2= ea505a7f37bd20d271c6bc7e5f8d2dfcb OK, which particular file is this? > metrics=3D$(curl http://127.0.0.1:$PORT1/metrics) > -regex=3D"fdcache_op_count\{op=3D\"enqueue\"\} ([0-9]+).*fdcache_op_count= \{op=3D\"evict\"\} ([0-9]+).*" > -enqueue=3D0 > -if [[ $metrics =3D~ $regex ]] > -then > - enqueue=3D${BASH_REMATCH[1]} > - evict=3D${BASH_REMATCH[2]} > +regex=3D"fdcache_prefetch_count ([0-9])+" > +# Check to see if prefetch cache is loaded > +if [[ $metrics =3D~ $regex ]]; then > + if [[ ${BASH_REMATCH[1]} -ne 1 ]]; then > + err > + fi Why the -ne 1 ? What number are we expecting and why? > else > - err > -fi > -# This is an ad-hoc test than only works when FDCACHE_FDS < "archive_tes= t"s above (5) > -# otherwise there will be no eviction > -if [[ $(( $enqueue-$FDCACHE_FDS )) -ne $evict ]] > -then > - err > -fi > -# Test prefetch cache > -archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hel= lo-1.0/hello.c $SHA > + err > +fi=20 > +# Ensure that searching for the source prefetched finds it in the prefet= ch cache > +testrun ${abs_top_builddir}/debuginfod/debuginfod-find source cee13b2ea5= 05a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c > metrics=3D$(curl http://127.0.0.1:$PORT1/metrics) > -regex=3D"fdcache_prefetch_count ([0-9])+" > -pf_count=3D0 > -for i in $FDCACHE_FDS > -do > - if [[ $metrics =3D~ $regex ]]; then > - > - if [[ $(( $pf_count+$PREFETCH )) -ne ${BASH_REMATCH[1]} ]]; then > - err > - else > - pf_count=3D${BASH_REMATCH[1]} > - fi > - > - else > +regex=3D"fdcache_op_count\{op=3D\"prefetch_access\"\} ([0-9])+" > +if [[ $metrics =3D~ $regex ]]; then > + if [[ ${BASH_REMATCH[1]} -ne 1 ]]; then > err > fi > -done > +else > + err > +fi Again -ne 1? Shouldn't it be different from the above now? =20 > kill $PID1 > wait $PID1 > PID1=3D0 > =20 > - > ######### > -# Test mb limit on fd and prefetch cache > +# Test mb limit on fd cache > ######### > - > rm -rf $DEBUGINFOD_CACHE_PATH > rm -rf $DB > env LD_LIBRARY_PATH=3D$ldpath DEBUGINFOD_URLS=3D ${abs_builddir}/../debu= ginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \ > - --fdcache-mbs=3D1 --fdcache-prefetch-mbs=3D1 --fdcache-mintmp 0 -vvv= vv -g 0 -t 0 -R R \ > - > vlog2$PORT1 2>&1 & > + --fdcache-mbs=3D$FDCACHE_MBS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R = R > vlog2$PORT1 2>&1 & > PID1=3D$! OK start a fresh debuginfod with -fdcache-mbs=3D1 --fdcache-mintmp 0 Is the fdcache-mintmp 0 meant to clear the cache immediately? > tempfiles vlog2$PORT1 > errfiles vlog2$PORT1 > @@ -139,26 +119,26 @@ wait_ready $PORT1 'thread_work_total{role=3D"traver= se"}' 1 > # All rpms need to be in the index, except the dummy permission-000 one > rpms=3D$(find R -name \*rpm | grep -v nothing | wc -l) > wait_ready $PORT1 'scanned_files_total{source=3D".rpm archive"}' $rpms > -kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug-= >dwz->srefs > -# Wait till both files are in the index and scan/index fully finished > +kill -USR1 $PID1 > wait_ready $PORT1 'thread_work_total{role=3D"traverse"}' 2 > +wait_ready $PORT1 'thread_work_pending{role=3D"scan"}' 0 > +wait_ready $PORT1 'thread_busy{role=3D"scan"}' 0 OK, boilerplate again for triggering and waiting for a new scan to complete. > -archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hel= lo2-1.0-2.x86_64/hello.c $SHA > +# This many archives cause the fd cache to be loaded with just over 1mb = of > +# files. So we expect the 1mb cap off > +SHA=3Df4a1a8062be998ae93b8f1cd744a398c6de6dbb1 > archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hel= lo-1.0/hello.c $SHA > archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hel= lo2-1.0-2.x86_64/hello.c $SHA > archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hel= lo2-1.0-2.x86_64/hello.c $SHA > archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hel= lo-1.0/hello.c $SHA > archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hel= lo-1.0/hello.c $SHA > archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hel= lo-1.0/hello.c $SHA > -archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hel= lo-1.0/hello.c $SHA OK, test that the hello.c source file is found in the scanned archives, that the fdcache is used and that the sha1sum checks out. > metrics=3D$(curl http://127.0.0.1:$PORT1/metrics) > -regex=3D"fdcache_bytes ([0-9]+).*fdcache_prefetch_bytes ([0-9]+)" > +regex=3D"fdcache_bytes ([0-9]+)" > mb=3D$((1024*1024)) > if [[ $metrics =3D~ $regex ]]; then > fdbytes=3D${BASH_REMATCH[1]} > - pfbytes=3D${BASH_REMATCH[2]} > - if [ $fdbytes -gt $mb ] || [ $pfbytes -gt $mb ]; then > + if [ $fdbytes -gt $mb ] ; then > err > fi OK, check that the prefetch cache isn't bigger than 10MB. Why that number? Thanks, Mark