Re: [Bug debuginfod/28577] Make run-...-fd-prefetch-caches.sh test something

[Mark Wielaard <mark@klomp.org>]

Hi Noah,

On Wed, 2022-05-25 at 14:44 -0400, Noah Sanci via Elfutils-devel wrote:
> PR28577: Make run-debuginfod-fd-prefetch-caches.sh test
>              something
> 
> Update to the run-debuginfod-fd-prefetch to make the test more sound.

Thanks. And according to Evgeny this makes the test pass for him:
https://sourceware.org/bugzilla/show_bug.cgi?id=29180#c2

But I find it a little hard to review, it would have been helpful to
get a bit more comments/documentation what exactly is being tested now.

> diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-
> debuginfod-fd-prefetch-caches.sh
> index 66a7a083..50ffc4ac 100755
> --- a/tests/run-debuginfod-fd-prefetch-caches.sh
> +++ b/tests/run-debuginfod-fd-prefetch-caches.sh
> @@ -21,22 +21,22 @@
>  # for test case debugging, uncomment:
>  set -x
>  unset VALGRIND_CMD
> +mkdir R Z

OK, some boilerplate.

> -mkdir R
> -cp -rvp ${abs_srcdir}/debuginfod-rpms R
> -if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
> -    rm -vrf R/debuginfod-rpms/fedora31
> -fi

This gets moved to below.

> -FDCACHE_MBS=$((1/1024/1024))
> +FDCACHE_MBS=1
>  FDCACHE_FDS=1
> -PREFETCH_MBS=$((1/1024/1024))
> -PREFETCH_FDS=2
> +PREFETCH_MBS=1
> +PREFETCH_FDS=1
>  PREFETCH=1

OK, new presets.

>  # This variable is essential and ensures no time-race for claiming
> ports occurs
>  # set base to a unique multiple of 100 not used in any other 'run-
> debuginfod-*' test
>  base=8800
>  get_ports
> +cp -rvp ${abs_srcdir}/debuginfod-rpms R
> +if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
> +    rm -vrf R/debuginfod-rpms/fedora31
> +fi
> +cp -rvp ${abs_srcdir}/debuginfod-tars Z

OK, put rpms under R, tars under Z.
 
>  DB=${PWD}/.debuginfod_tmp.sqlite
>  tempfiles $DB
> @@ -47,88 +47,68 @@ export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
>  ######
>  rm -rf $DEBUGINFOD_CACHE_PATH
>  rm -rf $DB
> +
>  # Set mb values to ensure the caches certainly have enough space
>  # To store the test files
>  env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \
> -    --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \
> -    --fdcache-mbs=50  --fdcache-prefetch-mbs=50 \
> -    --fdcache-prefetch=$PREFETCH  > vlog$PORT1 2>&1 &
> +    --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-fds=$PREFETCH_FDS  -vvvvv -g 0 -t 0 -R R \
> +    -Z .tar.bz2=bzcat Z  --fdcache-mbs=50  --fdcache-prefetch-mbs=100 \
> +    --fdcache-prefetch=$PREFETCH -F > vlog$PORT1 2>&1 &

Now run with --fdcache-fds=1 --fdcache-mbs=50 and
-fdcache-prefetch-fds=1 fdcache-prefetch=1 -fdcache-prefetch-mbs=100

It is these last 3 we want to test.

>  PID1=$!
>  tempfiles vlog$PORT1
>  errfiles vlog$PORT1
> +
>  # Server must become ready
>  wait_ready $PORT1 'ready' 1
>  ########################################################################
> -cp -rvp ${abs_srcdir}/debuginfod-rpms R
> -if [ "$zstd" = "false" ]; then  # nuke the zstd fedora 31 ones
> -    rm -vrf R/debuginfod-rpms/fedora31
> -fi
>  kill -USR1 $PID1
>  wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
> +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
> +wait_ready $PORT1 'thread_busy{role="scan"}' 0

OK, boilerplate to wait for initial scan.

>  # All rpms need to be in the index, except the dummy permission-000 one
>  rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
>  wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
> -kill -USR1 $PID1  # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
> -# Wait till both files are in the index and scan/index fully finished
> +kill -USR1 $PID1 
>  wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
> -export DEBUGINFOD_URLS="http://127.0.0.1:"$PORT1
> +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
> +wait_ready $PORT1 'thread_busy{role="scan"}' 0

Do another scan and wait for it to be ready.

> -archive_hits=5
> -SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
> -for i in $archive_hits
> -do
> -  archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
> -done
> +export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/
> +# load prefetch cache
> +testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb

OK, which particular file is this?

>  metrics=$(curl http://127.0.0.1:$PORT1/metrics)
> -regex="fdcache_op_count\{op=\"enqueue\"\} ([0-9]+).*fdcache_op_count\{op=\"evict\"\} ([0-9]+).*"
> -enqueue=0
> -if [[ $metrics =~ $regex ]]
> -then
> -   enqueue=${BASH_REMATCH[1]}
> -   evict=${BASH_REMATCH[2]}
> +regex="fdcache_prefetch_count ([0-9])+"
> +# Check to see if prefetch cache is loaded
> +if [[ $metrics =~ $regex ]]; then
> +  if [[  ${BASH_REMATCH[1]} -ne 1 ]]; then
> +    err
> +  fi

Why the -ne 1 ? What number are we expecting and why?

>  else
> -   err
> -fi
> -# This is an ad-hoc test than only works when FDCACHE_FDS < "archive_test"s above (5)
> -# otherwise there will be no eviction
> -if [[ $(( $enqueue-$FDCACHE_FDS )) -ne $evict ]]
> -then
> -   err
> -fi
> -# Test prefetch cache
> -archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
> +  err
> +fi 
> +# Ensure that searching for the source prefetched finds it in the prefetch cache
> +testrun ${abs_top_builddir}/debuginfod/debuginfod-find source cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c
>  metrics=$(curl http://127.0.0.1:$PORT1/metrics)
> -regex="fdcache_prefetch_count ([0-9])+"
> -pf_count=0
> -for i in $FDCACHE_FDS
> -do
> -  if [[ $metrics =~ $regex ]]; then
> -
> -    if [[ $(( $pf_count+$PREFETCH )) -ne ${BASH_REMATCH[1]} ]]; then
> -      err
> -    else
> -      pf_count=${BASH_REMATCH[1]}
> -    fi
> -
> -  else
> +regex="fdcache_op_count\{op=\"prefetch_access\"\} ([0-9])+"
> +if [[ $metrics =~ $regex ]]; then
> +  if [[  ${BASH_REMATCH[1]} -ne 1 ]]; then
>      err
>    fi
> -done
> +else
> +  err
> +fi

Again -ne 1? Shouldn't it be different from the above now?
 
>  kill $PID1
>  wait $PID1
>  PID1=0
>  
> -
>  #########
> -# Test mb limit on fd and prefetch cache
> +# Test mb limit on fd cache
>  #########
> -
>  rm -rf $DEBUGINFOD_CACHE_PATH
>  rm -rf $DB
>  env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 -d $DB \
> -    --fdcache-mbs=1 --fdcache-prefetch-mbs=1 --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R \
> -    > vlog2$PORT1 2>&1 &
> +    --fdcache-mbs=$FDCACHE_MBS --fdcache-mintmp 0 -vvvvv -g 0 -t 0 -R R > vlog2$PORT1 2>&1 &
>  PID1=$!

OK start a fresh debuginfod with -fdcache-mbs=1 --fdcache-mintmp 0
Is the fdcache-mintmp 0 meant to clear the cache immediately?

>  tempfiles vlog2$PORT1
>  errfiles vlog2$PORT1
> @@ -139,26 +119,26 @@ wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
>  # All rpms need to be in the index, except the dummy permission-000 one
>  rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
>  wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
> -kill -USR1 $PID1  # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
> -# Wait till both files are in the index and scan/index fully finished
> +kill -USR1 $PID1
>  wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
> +wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
> +wait_ready $PORT1 'thread_busy{role="scan"}' 0

OK, boilerplate again for triggering and waiting for a new scan to
complete.

> -archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
> +# This many archives cause the fd cache to be loaded with just over 1mb of
> +# files. So we expect the 1mb cap off
> +SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
>  archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
>  archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
>  archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
>  archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
>  archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
>  archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
> -archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA

OK, test that the hello.c source file is found in the scanned archives,
that the fdcache is used and that the sha1sum checks out.

>  metrics=$(curl http://127.0.0.1:$PORT1/metrics)
> -regex="fdcache_bytes ([0-9]+).*fdcache_prefetch_bytes ([0-9]+)"
> +regex="fdcache_bytes ([0-9]+)"
>  mb=$((1024*1024))
>  if [[ $metrics =~ $regex ]]; then
>    fdbytes=${BASH_REMATCH[1]}
> -  pfbytes=${BASH_REMATCH[2]}
> -  if [ $fdbytes -gt $mb ] || [ $pfbytes -gt $mb ]; then
> +  if [ $fdbytes -gt $mb ] ; then
>      err
>    fi

OK, check that the prefetch cache isn't bigger than 10MB. Why that
number?

Thanks,

Mark