* debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests
@ 2021-08-23 14:01 Noah Sanci
2021-08-23 19:31 ` Noah Sanci
0 siblings, 1 reply; 6+ messages in thread
From: Noah Sanci @ 2021-08-23 14:01 UTC (permalink / raw)
To: elfutils-devel
[-- Attachment #1: Type: text/plain, Size: 807 bytes --]
Hello,
tests/run-debuginfod-find.sh was a massive test script with many broadly
varying tests. This caused the test script to fail when any number of
things went wrong and because of its intertwined nature, detecting the
source of a failure could be difficult. The size of the test script
also meant many unrelated tests were run making the testing process
unnecessarily lengthy.
This patch fractures tests/run-debuginfod-find.sh into smaller, more
manageable individual test script files. This ensures that when failure
occurs, a programmer can easily determine where their patch went
wrong. It also allows programmers to specify exactly which tests to
run, making testing more efficient. Redundancies are also reduced by
placing code in tests/test-subr.sh.
Please find the patch attached.
Noah Sanci
[-- Attachment #2: 0001-debuginfod-Fracture-tests-run-debuginfod-find.sh-int.patch --]
[-- Type: text/x-patch, Size: 127307 bytes --]
From b8547b94a2b2b1fac63d9634b4dd3f519e2700cd Mon Sep 17 00:00:00 2001
From: Noah Sanci <nsanci@redhat.com>
Date: Tue, 10 Aug 2021 11:21:35 -0400
Subject: [PATCH] debuginfod: Fracture tests/run-debuginfod-find.sh into
specific tests
tests/run-debuginfod-find.sh was a massive test script with many broadly
varying tests. This caused the test script to fail when any number of
things went wrong and because of its intertwined nature, detecting the
source of a failure could be difficult. The size of the test script
also meant many unrelated tests were run making the testing process
unnecessarily lengthy.
This patch fractures tests/run-debuginfod-find.sh into smaller, more
manageable individual test script files. This ensures that when failure
occurs, a programmer can easily determine where their patch went
wrong. It also allows programmers to specify exactly which tests to
run, making testing more efficient. Redundancies are also reduced by
placing code in tests/test-subr.sh.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
---
ChangeLog | 3 +
Makefile.am | 2 +-
tests/ChangeLog | 26 +
tests/Makefile.am | 33 +-
tests/run-debuginfod-000-permission.sh | 82 +++
tests/run-debuginfod-archive-groom.sh | 151 ++++
tests/run-debuginfod-archive-rename.sh | 92 +++
tests/run-debuginfod-archive-test.sh | 80 ++
tests/run-debuginfod-artifact-running.sh | 118 +++
tests/run-debuginfod-dlopen.sh | 80 ++
tests/run-debuginfod-duplicate-urls.sh | 47 ++
| 98 +++
tests/run-debuginfod-fd-prefetch-caches.sh | 55 ++
tests/run-debuginfod-federation-link.sh | 158 ++++
tests/run-debuginfod-federation-metrics.sh | 211 ++++++
tests/run-debuginfod-federation-sqlite.sh | 199 +++++
tests/run-debuginfod-file.sh | 38 +
tests/run-debuginfod-find.sh | 807 ---------------------
tests/run-debuginfod-malformed.sh | 106 +++
tests/run-debuginfod-no-urls.sh | 37 +
tests/run-debuginfod-query-retry.sh | 33 +
tests/run-debuginfod-regex.sh | 94 +++
tests/run-debuginfod-sizetime.sh | 74 ++
tests/run-debuginfod-tmp-home.sh | 121 +++
tests/run-debuginfod-writable.sh | 81 +++
tests/test-subr.sh | 132 +++-
26 files changed, 2146 insertions(+), 812 deletions(-)
create mode 100755 tests/run-debuginfod-000-permission.sh
create mode 100755 tests/run-debuginfod-archive-groom.sh
create mode 100755 tests/run-debuginfod-archive-rename.sh
create mode 100755 tests/run-debuginfod-archive-test.sh
create mode 100755 tests/run-debuginfod-artifact-running.sh
create mode 100755 tests/run-debuginfod-dlopen.sh
create mode 100755 tests/run-debuginfod-duplicate-urls.sh
create mode 100755 tests/run-debuginfod-extraction.sh
create mode 100755 tests/run-debuginfod-fd-prefetch-caches.sh
create mode 100755 tests/run-debuginfod-federation-link.sh
create mode 100755 tests/run-debuginfod-federation-metrics.sh
create mode 100755 tests/run-debuginfod-federation-sqlite.sh
create mode 100755 tests/run-debuginfod-file.sh
delete mode 100755 tests/run-debuginfod-find.sh
create mode 100755 tests/run-debuginfod-malformed.sh
create mode 100755 tests/run-debuginfod-no-urls.sh
create mode 100755 tests/run-debuginfod-query-retry.sh
create mode 100755 tests/run-debuginfod-regex.sh
create mode 100755 tests/run-debuginfod-sizetime.sh
create mode 100755 tests/run-debuginfod-tmp-home.sh
create mode 100755 tests/run-debuginfod-writable.sh
diff --git a/ChangeLog b/ChangeLog
index 12b8f403..0aeac175 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,6 @@
+2021-08-23 Noah Sanci <nsanci@redhat.com
+ * ChangeLog: removed --sign flag on rpmbuild
+
2021-07-28 Mark Wielaard <mark@klomp.org>
* configure.ac (AC_CHECK_DECLS): Add reallocarray check.
diff --git a/Makefile.am b/Makefile.am
index 9c47afa9..3dfefa9f 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -45,7 +45,7 @@ distcheck-hook:
chmod -R u+w $(distdir)
rpm: dist
- rpmbuild -ts --sign elfutils-@PACKAGE_VERSION@.tar.bz2
+ rpmbuild -ts elfutils-@PACKAGE_VERSION@.tar.bz2
if GCOV
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 3bfd1ca2..22c50773 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,29 @@
+2021-08-20 Noah Sanci <nsanci@redhat.com>
+
+ * debuginfod-find.sh: Separated file into
+ run-debuginfod-000-permission.sh,
+ run-debuginfod-archive-groom.sh,
+ run-debuginfod-archive-rename.sh,
+ run-debuginfod-archive-test.sh,
+ run-debuginfod-artifact-running.sh,
+ run-debuginfod-dlopen.sh,
+ run-debuginfod-duplicate-urls.sh,
+ run-debuginfod-extraction.sh,
+ run-debuginfod-fd-prefetch-caches.sh,
+ run-debuginfod-federation-link.sh,
+ run-debuginfod-federation-metrics.sh,
+ run-debuginfod-federation-sqlite.sh,
+ run-debuginfod-file.sh,
+ run-debuginfod-malformed.sh,
+ run-debuginfod-no-urls.sh,
+ run-debuginfod-query-retry.sh,
+ run-debuginfod-regex.sh,
+ run-debuginfod-sizetime.sh,
+ run-debuginfod-tmp-home.sh, and
+ run-debuginfod-writable.sh
+ * tests/Makefile.am: Added the above new files to the test suite
+ * tests/test-subr.sh: Added some general functions for above tests
+
2021-08-04 Mark Wielaard <mark@klomp.org>
PR28190
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 429649f4..46ffe301 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -211,7 +211,16 @@ if DEBUGINFOD
check_PROGRAMS += debuginfod_build_id_find
# With the dummy delegation doesn't work
if !DUMMY_LIBDEBUGINFOD
-TESTS += run-debuginfod-find.sh
+TESTS += run-debuginfod-dlopen.sh run-debuginfod-artifact-running.sh \
+ run-debuginfod-fd-prefetch-caches.sh run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh run-debuginfod-extraction.sh \
+ run-debuginfod-archive-groom.sh run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-federation-link.sh run-debuginfod-federation-metrics.sh
endif
endif
@@ -474,8 +483,26 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-elfclassify.sh run-elfclassify-self.sh \
run-disasm-riscv64.sh \
testfile-riscv64-dis1.o.bz2 testfile-riscv64-dis1.expect.bz2 \
- run-debuginfod-find.sh \
- debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-dlopen.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-federation-sqlite.sh \
+ debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debugsource-1.0-2.x86_64.rpm \
diff --git a/tests/run-debuginfod-000-permission.sh b/tests/run-debuginfod-000-permission.sh
new file mode 100755
index 00000000..2eb4c51d
--- /dev/null
+++ b/tests/run-debuginfod-000-permission.sh
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+# PR25628
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# The query is designed to fail, while the 000-permission file should be created.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
+ err
+fi
+
+if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
+ err
+fi
+
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" != "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} has changed."
+ err
+fi
+
+# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
+sleep 1
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" == "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
new file mode 100755
index 00000000..1ab0206d
--- /dev/null
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -0,0 +1,151 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+# find an unused port number
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Be patient when run on a busy machine things might take a bit.
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Build a non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/F/prog.c
+gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a ${PWD}/F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+tempfiles ${PWD}/F/prog ${PWD}/F/prog.c
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+tempfiles vlog3 $DB
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
+
+# Drop some of the artifacts, run a groom cycle; confirm that
+# debuginfod has forgotten them, but remembers others
+rm -r R/debuginfod-rpms/rhel6/*
+kill -USR2 $PID1 # groom cycle
+## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# Expect 4 rpms containing 2 buildids to be deleted by the groom
+wait_ready $PORT1 'groomed_total{decision="stale"}' 4
+
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# this is one of the buildids from the groom-deleted rpms
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
+
+# but this one was not deleted so should be still around
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID || true
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
new file mode 100755
index 00000000..54a513a9
--- /dev/null
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DEBUGINFOD_CACHE_PATH $DB
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+
+########################################################################
+## PR26810: Now rename some files in the R directory, then rescan, so
+# there are two copies of the same buildid in the index, one for the
+# no-longer-existing file name, and one under the new name.
+
+# run a groom cycle to force server to drop its fdcache
+kill -USR2 $PID1 # groom cycle
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
+# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
+mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# retest rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+
+egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0;
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
new file mode 100755
index 00000000..89772f1f
--- /dev/null
+++ b/tests/run-debuginfod-archive-test.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+# find an unused port number
+mkdir R
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -t0 -g0 -v R > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+# arch
+#archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
new file mode 100755
index 00000000..59d5a27e
--- /dev/null
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+get_ports
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+echo 'int main(int argc, char * argv){ return 0; }' > ${PWD}/prog.c
+gcc -Wl,--build-id -g -o prog ${PWD}/prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d $DB -F -p $PORT1 -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+mv prog F
+mv prog.debug F
+tempfiles prog/F
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+kill -USR1 $PID1
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Add artifacts to the search paths and test whether debuginfod finds them while already running.
+# Build another, non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/prog2.c
+tempfiles prog2.c
+gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
+#testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
+mv prog2 F
+#mv prog2.debug F
+tempfiles F/prog2 F/prog2.debug
+
+kill -USR1 $PID1
+# Now there should be 3 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/prog.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/prog`
+cmp $filename F/prog
+
+# raw source filename
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../prog.c`
+cmp $filename ${PWD}/prog.c
+
+# and also the canonicalized one
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/prog.c`
+cmp $filename ${PWD}/prog.c
+
+# Rerun same tests for the prog2 binary
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
+cmp $filename F/prog2
+grep -q Progress vlog
+grep -q Downloaded.from vlog
+tempfiles vlog
+filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
+cmp $filename F/prog2
+grep -q 'Downloading.*http' vlog2
+tempfiles vlog2
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
+cmp $filename ${PWD}/prog2.c
+
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
new file mode 100755
index 00000000..6c7103e4
--- /dev/null
+++ b/tests/run-debuginfod-dlopen.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+mkdir F
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a'
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether elfutils, via the debuginfod client library dlopen hooks,
+# is able to fetch debuginfo from the local debuginfod.
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-duplicate-urls.sh b/tests/run-debuginfod-duplicate-urls.sh
new file mode 100755
index 00000000..f0142e5b
--- /dev/null
+++ b/tests/run-debuginfod-duplicate-urls.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+########################################################################
+## PR27983
+# Ensure no duplicate urls are used in when querying servers for files
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:7999" \
+ LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -v executable 0 > vlog1 2>&1 || true
+tempfiles vlog1
+if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog1 ) -ne 2 ]; then
+ echo "Duplicate servers remain";
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
--git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
new file mode 100755
index 00000000..7ff509d4
--- /dev/null
+++ b/tests/run-debuginfod-extraction.sh
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+mkdir R Z
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
+tb2=$(find Z -name \*tar.bz2 | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh
new file mode 100755
index 00000000..059866de
--- /dev/null
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+type curl 2>/dev/null || (echo "need curl"; exit 77)
+
+FDCACHE_FDS=100
+FDCACHE_MBS=100
+PREFETCH_FDS=100
+PREFETCH_MBS=100
+get_ports
+
+echo $PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 \
+ --fdcache-mbs=$FDCACHE_MDS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS \
+ --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -v -F F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+grep 'fdcache fds ' vlog$PORT1 #$FDCACHE_FDS
+grep 'fdcache mbs ' vlog$PORT1 #$FDCACHE_MBS
+grep 'prefetch fds ' vlog$PORT1 #$PREFETCH_FDS
+grep 'prefetch mbs ' vlog$PORT1 #$PREFETCH_MBS
+# search the vlog to find what metric counts should be and check the correct metrics
+# were incrimented
+wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
new file mode 100755
index 00000000..38df2a68
--- /dev/null
+++ b/tests/run-debuginfod-federation-link.sh
@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+
+kill -USR1 $PID2
+# Wait till both files are in the index.
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/run-debuginfod-federation-metrics.sh b/tests/run-debuginfod-federation-metrics.sh
new file mode 100755
index 00000000..a9d49846
--- /dev/null
+++ b/tests/run-debuginfod-federation-metrics.sh
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+# Clean old dirictories
+mkdir D L F
+
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+get_ports
+
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+########################################################################
+# Fetch some metrics
+curl -s http://127.0.0.1:$PORT1/badapi
+curl -s http://127.0.0.1:$PORT1/metrics
+curl -s http://127.0.0.1:$PORT2/metrics
+curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
+curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# Confirm that some debuginfod client pools are being used
+curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
+
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+exit 0
+
diff --git a/tests/run-debuginfod-federation-sqlite.sh b/tests/run-debuginfod-federation-sqlite.sh
new file mode 100755
index 00000000..15be522b
--- /dev/null
+++ b/tests/run-debuginfod-federation-sqlite.sh
@@ -0,0 +1,199 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D F > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+########################################################################
+# Corrupt the sqlite database and get debuginfod to trip across its errors
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
+dd if=/dev/zero of=$DB bs=1 count=1
+
+# trigger some random activity that's Sure to get sqlite3 upset
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -USR2 $PID1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
+# Run the tests again without the servers running. The target file should
+# be found in the cache.
+
+kill -INT $PID1 $PID2
+wait $PID1 $PID2
+PID1=0
+PID2=0
+tempfiles .debuginfod_*
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# check out the debuginfod logs for the new style status lines
+cat vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
+
+exit 0
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
new file mode 100755
index 00000000..435c2156
--- /dev/null
+++ b/tests/run-debuginfod-file.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Test fetching a file using file:// . No debuginfod server needs to be run for
+# this test.
+local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
+mkdir -p ${local_dir}
+echo "int main() { return 0; }" > ${local_dir}/main.c
+# first test that is doesn't work, when no DEBUGINFOD_URLS is set
+DEBUGINFOD_URLS=""
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
+
+# Now test is with proper DEBUGINFOD_URLS
+DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
+cmp $filename ${local_dir}/main.c
+
+exit 0
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
deleted file mode 100755
index 991d1dc5..00000000
--- a/tests/run-debuginfod-find.sh
+++ /dev/null
@@ -1,807 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (C) 2019-2021 Red Hat, Inc.
-# This file is part of elfutils.
-#
-# This file is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 3 of the License, or
-# (at your option) any later version.
-#
-# elfutils is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-. $srcdir/test-subr.sh # includes set -e
-
-type curl 2>/dev/null || (echo "need curl"; exit 77)
-type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
-type cpio 2>/dev/null || (echo "need cpio"; exit 77)
-type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
-bsdtar --version | grep -q zstd && zstd=true || zstd=false
-echo "zstd=$zstd bsdtar=`bsdtar --version`"
-
-# for test case debugging, uncomment:
-set -x
-VERBOSE=-vvv
-
-DB=${PWD}/.debuginfod_tmp.sqlite
-tempfiles $DB
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
-
-PID1=0
-PID2=0
-PID3=0
-PID4=0
-
-cleanup()
-{
- if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
- if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
- if [ $PID3 -ne 0 ]; then kill $PID3; wait $PID3; fi
- if [ $PID4 -ne 0 ]; then kill $PID4; wait $PID4; fi
- rm -rf F R D L Z ${PWD}/foobar ${PWD}/mocktree ${PWD}/.client_cache* ${PWD}/tmp*
- exit_cleanup
-}
-
-# clean up trash if we were aborted early
-trap cleanup 0 1 2 3 5 9 15
-
-errfiles_list=
-err() {
- echo ERROR REPORTS
- for ports in $PORT1 $PORT2 $PORT3
- do
- echo ERROR REPORT $port metrics
- curl -s http://127.0.0.1:$port/metrics
- echo
- done
- for x in $errfiles_list
- do
- echo ERROR REPORT "$x"
- cat $x
- echo
- done
- false # trigger set -e
-}
-trap err ERR
-
-errfiles() {
- errfiles_list="$errfiles_list $*"
-}
-
-
-# find an unused port number
-while true; do
- PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT1" || break
-done
-
-# We want to run debuginfod in the background. We also want to start
-# it with the same check/installcheck-sensitive LD_LIBRARY_PATH stuff
-# that the testrun alias sets. But: we if we just use
-# testrun .../debuginfod
-# it runs in a subshell, with different pid, so not helpful.
-#
-# So we gather the LD_LIBRARY_PATH with this cunning trick:
-ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
-
-mkdir F R L D Z
-# not tempfiles F R L D Z - they are directories which we clean up manually
-ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
-
-wait_ready()
-{
- port=$1;
- what=$2;
- value=$3;
- timeout=20;
-
- echo "Wait $timeout seconds on $port for metric $what to change to $value"
- while [ $timeout -gt 0 ]; do
- mvalue="$(curl -s http://127.0.0.1:$port/metrics \
- | grep "$what" | awk '{print $NF}')"
- if [ -z "$mvalue" ]; then mvalue=0; fi
- echo "metric $what: $mvalue"
- if [ "$mvalue" -eq "$value" ]; then
- break;
- fi
- sleep 0.5;
- ((timeout--));
- done;
-
- if [ $timeout -eq 0 ]; then
- echo "metric $what never changed to $value on port $port"
- err
- fi
-}
-
-FDCACHE_FDS=50
-FDCACHE_MBS=190
-PREFETCH_FDS=10
-PREFETCH_MBS=120
-# create a bogus .rpm file to evoke a metric-visible error
-# Use a cyclic symlink instead of chmod 000 to make sure even root
-# would see an error (running the testsuite under root is NOT encouraged).
-ln -s R/nothing.rpm R/nothing.rpm
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -d $DB -p $PORT1 -t0 -g0 --fdcache-mbs=$FDCACHE_MBS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -Z .tar.xz -Z .tar.bz2=bzcat -v R F Z L > vlog$PORT1 2>&1 &
-PID1=$!
-tempfiles vlog$PORT1
-errfiles vlog$PORT1
-# Server must become ready
-wait_ready $PORT1 'ready' 1
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
-
-# Be patient when run on a busy machine things might take a bit.
-export DEBUGINFOD_TIMEOUT=10
-
-# Check thread comm names
-ps -q $PID1 -e -L -o '%p %c %a' | grep groom
-ps -q $PID1 -e -L -o '%p %c %a' | grep scan
-ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
-
-# We use -t0 and -g0 here to turn off time-based scanning & grooming.
-# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
-
-########################################################################
-
-# Compile a simple program, strip its debuginfo and save the build-id.
-# Also move the debuginfo into another directory so that elfutils
-# cannot find it without debuginfod.
-echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
-tempfiles p+r%o\$g.c
-# Create a subdirectory to confound source path names
-mkdir foobar
-gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
-testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
-
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-mv p+r%o\$g F
-mv p+r%o\$g.debug F
-kill -USR1 $PID1
-# Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-########################################################################
-
-# Test whether elfutils, via the debuginfod client library dlopen hooks,
-# is able to fetch debuginfo from the local debuginfod.
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-########################################################################
-## PR27892
-# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
-# code
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-tempfiles find-vlog$PORT1
-# wait for the server to fail the same number of times the query is retried.
-wait_ready $PORT1 'http_responses_after_you_milliseconds_count{code="406"}' 1
-# ensure all reporting is functional
-grep 'serving file '$(realpath ${PWD})'/F/p+r%o\$g.debug' vlog$PORT1
-grep 'File too large' vlog$PORT1
-grep 'using max size 1B' find-vlog$PORT1
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxsize check"
- err
-fi
-
-# Ensure DEBUGINFOD_MAXTIME is functional
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:8002/" DEBUGINFOD_MAXTIME=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-grep 'using max time' find-vlog$PORT1
-# Ensure p+r%o\$g.debug is NOT cached
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxtime check"
- err
-fi
-########################################################################
-# PR25628
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# The query is designed to fail, while the 000-permission file should be created.
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
- err
-fi
-
-if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
- err
-fi
-
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" != "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} has changed."
- err
-fi
-
-# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
-sleep 1
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" == "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
- err
-fi
-########################################################################
-
-# Test whether debuginfod-find is able to fetch those files.
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
-cmp $filename F/p+r%o\$g.debug
-if [ -w $filename ]; then
- echo "cache file writable, boo"
- err
-fi
-
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/p+r%o\\$g`
-cmp $filename F/p+r%o\$g
-
-# raw source filename
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-# and also the canonicalized one
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-
-########################################################################
-
-# Test whether the cache default locations are correct
-
-mkdir tmphome
-
-# $HOME/.cache should be created.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $HOME/.cache should be found.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $XDG_CACHE_HOME should take priority over $HOME.cache.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpxdg/"
- err
-fi
-
-# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
-# priority over $HOME/.cache, $XDG_CACHE_HOME.
-cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
-# ||true is for tolerating errors, such a valgrind or something else
-# leaving 000-perm files in there
-
-# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
-mkdir tmphome/.debuginfod_client_cache/deadbeef
-echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
-cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-
-# $DEBUGINFO_CACHE_PATH should take priority over all else.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpcache/"
- err
-fi
-
-########################################################################
-
-# Add artifacts to the search paths and test whether debuginfod finds them while already running.
-
-# Build another, non-stripped binary
-echo "int main() { return 0; }" > ${PWD}/prog2.c
-tempfiles prog2.c
-gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
-BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
-
-mv prog2 F
-kill -USR1 $PID1
-# Now there should be 3 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-cp $DB $DB.backup
-tempfiles $DB.backup
-# Rerun same tests for the prog2 binary
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
-cmp $filename F/prog2
-cat vlog
-grep -q Progress vlog
-grep -q Downloaded.from vlog
-tempfiles vlog
-filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
-cmp $filename F/prog2
-cat vlog2
-grep -q 'Downloading.*http' vlog2
-tempfiles vlog2
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
-cmp $filename ${PWD}/prog2.c
-
-cp -rvp ${abs_srcdir}/debuginfod-rpms R
-if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
- rm -vrf R/debuginfod-rpms/fedora31
-fi
-
-cp -rvp ${abs_srcdir}/debuginfod-tars Z
-kill -USR1 $PID1
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# All rpms need to be in the index, except the dummy permission-000 one
-rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
-txz=$(find Z -name \*tar.xz | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
-tb2=$(find Z -name \*tar.bz2 | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
-
-kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 5
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# Expect all source files found in the rpms (they are all called hello.c :)
-# We will need to extract all rpms (in their own directory) and could all
-# sources referenced in the .debug files.
-mkdir extracted
-cd extracted
-subdir=0;
-newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
-for i in $newrpms; do
- subdir=$[$subdir+1];
- mkdir $subdir;
- cd $subdir;
- ls -lah ../$i
- rpm2cpio ../$i | cpio -ivd;
- cd ..;
-done
-sourcefiles=$(find -name \*\\.debug \
- | env LD_LIBRARY_PATH=$ldpath xargs \
- ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
- | grep mtime: | wc --lines)
-cd ..
-rm -rf extracted
-
-wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
-
-########################################################################
-# PR27983 ensure no duplicate urls are used in when querying servers for files
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http:127.0.0.1:7999" \
- LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -v executable $BUILDID2 > vlog4 2>&1 || true
-tempfiles vlog4
-if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog4 ) -ne 2 ]; then
- echo "Duplicated servers remain";
- err
-fi
-########################################################################
-# Run a bank of queries against the debuginfod-rpms / debuginfod-debs test cases
-
-archive_test() {
- __BUILDID=$1
- __SOURCEPATH=$2
- __SOURCESHA1=$3
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- # check that timestamps are plausible - older than the near-present (tmpdir mtime)
- test $filename -ot `pwd`
-
- # run again to assure that fdcache is being enjoyed
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- if test "x$__SOURCEPATH" != "x"; then
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
- hash=`cat $filename | sha1sum | awk '{print $1}'`
- test $__SOURCESHA1 = $hash
- test $filename -ot `pwd`
- fi
-}
-
-
-# common source file sha1
-SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
-# fedora31
-if [ $zstd = true ]; then
- # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
- # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
- archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
- archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
-fi
-# fedora30
-archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-# rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-# rhel6
-archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
-# arch
-archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
-
-RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir, for a later test
-
-
-########################################################################
-
-# Drop some of the artifacts, run a groom cycle; confirm that
-# debuginfod has forgotten them, but remembers others
-
-rm -r R/debuginfod-rpms/rhel6/*
-kill -USR2 $PID1 # groom cycle
-# 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
-wait_ready $PORT1 'thread_work_total{role="groom"}' 2
-# Expect 4 rpms containing 2 buildids to be deleted by the groom
-wait_ready $PORT1 'groomed_total{decision="stale"}' 4
-
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# this is one of the buildids from the groom-deleted rpms
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
-# but this one was not deleted so should be still around
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2
-
-########################################################################
-
-# PR26810: Now rename some files in the R directory, then rescan, so
-# there are two copies of the same buildid in the index, one for the
-# no-longer-existing file name, and one under the new name.
-
-# run a groom cycle to force server to drop its fdcache
-kill -USR2 $PID1 # groom cycle
-wait_ready $PORT1 'thread_work_total{role="groom"}' 3
-# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
-mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 6
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 7
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 8
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# retest rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-
-egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
-
-########################################################################
-## PR25978
-# Ensure that the fdcache options are working.
-grep "prefetch fds" vlog$PORT1
-grep "prefetch mbs" vlog$PORT1
-grep "fdcache fds" vlog$PORT1
-grep "fdcache mbs" vlog$PORT1
-# search the vlog to find what metric counts should be and check the correct metrics
-# were incrimented
-wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
-########################################################################
-
-# Federation mode
-
-# find another unused port
-while true; do
- PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT2" || break
-done
-
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
-mkdir -p $DEBUGINFOD_CACHE_PATH
-# NB: inherits the DEBUGINFOD_URLS to the first server
-# NB: run in -L symlink-following mode for the L subdir
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d ${DB}_2 -p $PORT2 -L L D > vlog$PORT2 2>&1 &
-PID2=$!
-tempfiles vlog$PORT2
-errfiles vlog$PORT2
-tempfiles ${DB}_2
-wait_ready $PORT2 'ready' 1
-wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
-wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
-wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
-
-# have clients contact the new server
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-
-if type bsdtar 2>/dev/null; then
- # copy in the deb files
- cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
- kill -USR1 $PID2
- wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
- wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
- wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
- # All debs need to be in the index
- debs=$(find D -name \*.deb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
- ddebs=$(find D -name \*.ddeb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
-
- # ubuntu
- archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
-fi
-
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# send a request to stress XFF and User-Agent federation relay;
-# we'll grep for the two patterns in vlog$PORT1
-curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
-
-grep UA:TESTCURL vlog$PORT1
-grep XFF:TESTXFF vlog$PORT1
-
-
-# confirm that first server can't resolve symlinked info in L/ but second can
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
-file L/foo
-file -L L/foo
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test again with scheme free url
-export DEBUGINFOD_URLS=127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test parallel queries in client
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
-mkdir -p $DEBUGINFOD_CACHE_PATH
-export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-########################################################################
-
-# Fetch some metrics
-curl -s http://127.0.0.1:$PORT1/badapi
-curl -s http://127.0.0.1:$PORT1/metrics
-curl -s http://127.0.0.1:$PORT2/metrics
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*fdcache'
-curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
-
-# And generate a few errors into the second debuginfod's logs, for analysis just below
-curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
-# NB: this error is used to seed the 404 failure for the survive-404 tests
-
-# Confirm bad artifact types are rejected without leaving trace
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
-(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
-
-# DISABLE VALGRIND checking because valgrind might use debuginfod client
-# requests itself, causing confusion about who put what in the cache.
-# It stays disabled till the end of this test.
-unset VALGRIND_CMD
-
-# Confirm that reused curl connections survive 404 errors.
-# The rm's force an uncached fetch
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# Confirm that some debuginfod client pools are being used
-curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
-
-# Trigger a flood of requests against the same archive content file.
-# Use a file that hasn't been previously extracted in to make it
-# likely that even this test debuginfod will experience concurrency
-# and impose some "after-you" delays.
-(for i in `seq 100`; do
- curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
- done;
- wait)
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
-# If we could guarantee some minimum number of seconds of CPU time, we
-# could assert that the after_you metrics show some nonzero amount of
-# waiting. A few hundred ms is typical on this developer's workstation.
-
-
-########################################################################
-# Corrupt the sqlite database and get debuginfod to trip across its errors
-
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
-ls -al $DB
-dd if=/dev/zero of=$DB bs=1 count=1
-ls -al $DB
-# trigger some random activity that's Sure to get sqlite3 upset
-kill -USR1 $PID1
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 9
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -USR2 $PID1
-wait_ready $PORT1 'thread_work_total{role="groom"}' 4
-curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
-
-########################################################################
-
-# Run the tests again without the servers running. The target file should
-# be found in the cache.
-
-kill -INT $PID1 $PID2
-wait $PID1 $PID2
-PID1=0
-PID2=0
-tempfiles .debuginfod_*
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-# check out the debuginfod logs for the new style status lines
-# cat vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
-
-########################################################################
-
-# Add some files to the cache that do not fit its naming format.
-# They should survive cache cleaning.
-mkdir $DEBUGINFOD_CACHE_PATH/malformed
-touch $DEBUGINFOD_CACHE_PATH/malformed0
-touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
-
-# A valid format for an empty buildid subdirectory
-mkdir $DEBUGINFOD_CACHE_PATH/00000000
-touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
-
-# Trigger a cache clean and run the tests again. The clients should be unable to
-# find the target.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
-echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID2 && false || true
-
-if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
- || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
- echo "unrelated files did not survive cache cleaning"
- err
-fi
-
-if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
- echo "failed to rmdir old cache dir"
- err
-fi
-
-# Test debuginfod without a path list; reuse $PORT1
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
-PID3=$!
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -int $PID3
-wait $PID3
-PID3=0
-
-########################################################################
-# Test fetching a file using file:// . No debuginfod server needs to be run for
-# this test.
-local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
-mkdir -p ${local_dir}
-echo "int main() { return 0; }" > ${local_dir}/main.c
-
-# first test that is doesn't work, when no DEBUGINFOD_URLS is set
-DEBUGINFOD_URLS=""
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
-
-# Now test is with proper DEBUGINFOD_URLS
-DEBUGINFOD_URLS="file://${PWD}/mocktree/"
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
-cmp $filename ${local_dir}/main.c
-
-########################################################################
-## PR27711
-# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
-while true; do
- PORT3=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT3" || break
-done
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT3/" ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT3 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d $DB.backup > vlog$PORT3 2>&1 &
-PID4=$!
-wait_ready $PORT3 'ready' 1
-tempfiles vlog$PORT3
-errfiles vlog$PORT3
-
-kill -USR2 $PID4
-wait_ready $PORT3 'thread_work_total{role="groom"}' 1
-wait_ready $PORT3 'groom{statistic="archive d/e"}' 0
-wait_ready $PORT3 'groom{statistic="archive sdef"}' 0
-wait_ready $PORT3 'groom{statistic="archive sref"}' 0
-wait_ready $PORT3 'groom{statistic="buildids"}' 0
-wait_ready $PORT3 'groom{statistic="file d/e"}' 0
-wait_ready $PORT3 'groom{statistic="file s"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (#)"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (mb)"}' 0
-
-kill $PID4
-wait $PID4
-PID4=0
-
-########################################################################
-# set up tests for retrying failed queries.
-retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
- | grep -c 'Retry failed query'`
-if [ $retry_attempts -ne 10 ]; then
- echo "retry mechanism failed."
- exit 1;
- fi
-
-exit 0
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
new file mode 100755
index 00000000..8a2d9c46
--- /dev/null
+++ b/tests/run-debuginfod-malformed.sh
@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+# find an unused port number
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Add some files to the cache that do not fit its naming format.
+# They should survive cache cleaning.
+mkdir $DEBUGINFOD_CACHE_PATH/malformed
+touch $DEBUGINFOD_CACHE_PATH/malformed0
+touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
+
+# A valid format for an empty buildid subdirectory
+mkdir $DEBUGINFOD_CACHE_PATH/00000000
+touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
+
+# Trigger a cache clean and run the tests again. The clients should be unable to
+# find the target.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
+echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+
+if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
+ || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
+ echo "unrelated files did not survive cache cleaning"
+ err
+fi
+
+if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
+ echo "failed to rmdir old cache dir"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-no-urls.sh b/tests/run-debuginfod-no-urls.sh
new file mode 100755
index 00000000..75d116e8
--- /dev/null
+++ b/tests/run-debuginfod-no-urls.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+########################################################################
+# Test debuginfod without a path list
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
+PID1=$!
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -int $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
new file mode 100755
index 00000000..33fb338b
--- /dev/null
+++ b/tests/run-debuginfod-query-retry.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+########################################################################
+# set up tests for retrying failed queries.
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
+ | grep -c 'Retry failed query'`
+if [ $retry_attempts -ne 10 ]; then
+ echo "retry mechanism failed."
+ exit 1;
+fi
+
+exit 0;
diff --git a/tests/run-debuginfod-regex.sh b/tests/run-debuginfod-regex.sh
new file mode 100755
index 00000000..5bc05d1d
--- /dev/null
+++ b/tests/run-debuginfod-regex.sh
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+get_ports
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+
+mv prog F
+mv prog.debug F
+tempfiles F/prog.debug F/prog
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+cp ${DB} ${DB}.backup
+tempfiles ${DB}.backup
+
+kill $PID1
+wait $PID1
+PID1=0
+
+#######################################################################
+## PR27711
+# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
+#
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -p $PORT2 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d ${DB}.backup > vlog$PORT2 2>&1 &
+
+#reuse PID1
+PID1=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+# Server must become ready
+wait_ready $PORT2 'ready' 1
+
+kill -USR2 $PID1
+wait_ready $PORT2 'thread_work_total{role="groom"}' 1
+wait_ready $PORT2 'groom{statistic="archive d/e"}' 0
+wait_ready $PORT2 'groom{statistic="archive sdef"}' 0
+wait_ready $PORT2 'groom{statistic="archive sref"}' 0
+wait_ready $PORT2 'groom{statistic="buildids"}' 0
+wait_ready $PORT2 'groom{statistic="file d/e"}' 0
+wait_ready $PORT2 'groom{statistic="file s"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (#)"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (mb)"}' 0
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
+
diff --git a/tests/run-debuginfod-sizetime.sh b/tests/run-debuginfod-sizetime.sh
new file mode 100755
index 00000000..e7d6324e
--- /dev/null
+++ b/tests/run-debuginfod-sizetime.sh
@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+set -x
+
+get_ports
+
+echo "int main() { return 0; }" > ${PWD}/prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+## PR27892
+# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
+# code
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v executable ${PWD}/prog 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+errfiles find-vlog$PORT1
+echo "Checking maxsize"
+grep "using max size 1B" find-vlog$PORT1
+echo "Checking maxsize"
+grep 'serving file '$(realpath ${PWD})'/prog' vlog$PORT1
+echo "Checking maxsize"
+grep 'File too large' vlog$PORT1
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxsize check"
+ err
+fi
+# Ensure no file is downloaded for longer than DEBUGINFOD_MAXTIME
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXTIME=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo ${PWD}/prog.debug 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+grep 'using max time' find-vlog$PORT1
+# Ensure p+r%o\$g.debug is NOT cached
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxtime check"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
new file mode 100755
index 00000000..120325d2
--- /dev/null
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether the cache default locations are correct
+mkdir tmphome
+
+# $HOME/.cache should be created.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+
+# $HOME/.cache should be found.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+# $XDG_CACHE_HOME should take priority over $HOME.cache.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpxdg/"
+ err
+fi
+
+# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
+# priority over $HOME/.cache, $XDG_CACHE_HOME.
+cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
+# ||true is for tolerating errors, such a valgrind or something else
+# leaving 000-perm files in there
+
+# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
+mkdir tmphome/.debuginfod_client_cache/deadbeef
+echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
+cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+
+# $DEBUGINFO_CACHE_PATH should take priority over all else.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpcache/"
+ err
+fi
+rm -rf ${PWD}/tmphome/ ${PWD}/tmpxdg ${PWD}/tmpcache
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
new file mode 100755
index 00000000..a63db3ff
--- /dev/null
+++ b/tests/run-debuginfod-writable.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+mkdir F
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+#######################################################################
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/p+r%o\$g.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/test-subr.sh b/tests/test-subr.sh
index 411e5f28..7965a68d 100644
--- a/tests/test-subr.sh
+++ b/tests/test-subr.sh
@@ -27,6 +27,13 @@ test_dir="test-$$"
mkdir -p "$test_dir"
cd "$test_dir"
+type curl 2>/dev/null || (echo "need curl"; exit 77)
+type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
+type cpio 2>/dev/null || (echo "need cpio"; exit 77)
+type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
+bsdtar --version | grep -q zstd && zstd=true || zstd=false
+echo "zstd=$zstd bsdtar=`bsdtar --version`"
+
#LC_ALL=C
#export LC_ALL
@@ -35,7 +42,11 @@ remove_files=
# Tests that trap EXIT (0) themselves should call this explicitly.
exit_cleanup()
{
- rm -f $remove_files; cd ..; rmdir $test_dir
+ rm -rf ${PWD}/.client_cache*
+ rm -f $remove_files;
+ ls -a ${PWD}
+ cd ..;
+ rmdir $test_dir
}
trap exit_cleanup 0
@@ -212,3 +223,122 @@ testrun_on_self_quiet()
# Only exit if something failed
if test $exit_status != 0; then exit $exit_status; fi
}
+
+cleanup()
+{
+ if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
+ if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
+ rm -rf ${PWD}/.client_cache F R D L Z ${PWD}/foobar ${PWD}/mocktree
+ exit_cleanup
+}
+trap cleanup 0 1 2 3 5 9 15
+errfiles_list=
+err() {
+ echo ERROR REPORTS
+ for ports in $PORT1 $PORT2
+ do
+ echo ERROR REPORT $port metrics
+ curl -s http://127.0.0.1:$port/metrics
+ echo
+ done
+ for x in $errfiles_list
+ do
+ echo ERROR REPORT "$x"
+ cat $x
+ echo
+ done
+ false # trigger set -e
+}
+trap err ERR
+
+errfiles() {
+ errfiles_list="$errfiles_list $*"
+}
+
+getport() {
+ # find an unused port number
+ while true; do
+ port=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$port" || break
+ done
+}
+
+wait_ready()
+{
+ port=$1;
+ what=$2;
+ value=$3;
+ timeout=20;
+
+ echo "Wait $timeout seconds on $port for metric $what to change to $value"
+ while [ $timeout -gt 0 ]; do
+ mvalue="$(curl -s http://127.0.0.1:$port/metrics \
+ | grep "$what" | awk '{print $NF}')"
+ if [ -z "$mvalue" ]; then mvalue=0; fi
+ echo "metric $what: $mvalue"
+ if [ "$mvalue" -eq "$value" ]; then
+ break;
+ fi
+ sleep 0.5;
+ ((timeout--));
+ done;
+
+ if [ $timeout -eq 0 ]; then
+ echo "metric $what never changed to $value on port $port"
+ err
+ fi
+}
+
+archive_test() {
+ __BUILDID=$1
+ __SOURCEPATH=$2
+ __SOURCESHA1=$3
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ # check that timestamps are plausible - older than the near-present (tmpdir mtime)
+ test $filename -ot `pwd`
+
+ # run again to assure that fdcache is being enjoyed
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ if test "x$__SOURCEPATH" != "x"; then
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
+ hash=`cat $filename | sha1sum | awk '{print $1}'`
+ test $__SOURCESHA1 = $hash
+ test $filename -ot `pwd`
+ fi
+}
+
+get_ports() {
+ while true; do
+ PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$PORT1" || break
+ done
+# Some tests will use two servers, so assign the second var
+ while true; do
+ PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
+ done
+
+}
+
+VERBOSE=-vvv
+# We gather the LD_LIBRARY_PATH with this cunning trick:
+ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
+PORT1=0
+PORT2=0
+PID1=0
+PID2=0
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests
2021-08-23 14:01 debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests Noah Sanci
@ 2021-08-23 19:31 ` Noah Sanci
2021-09-02 11:29 ` Mark Wielaard
2021-09-02 13:42 ` Mark Wielaard
0 siblings, 2 replies; 6+ messages in thread
From: Noah Sanci @ 2021-08-23 19:31 UTC (permalink / raw)
To: elfutils-devel
[-- Attachment #1: Type: text/plain, Size: 104 bytes --]
Hello,
Here is an updated patch that unsets VALGRIND_CMD for all the new
debuginfod tests.
Noah Sanci
[-- Attachment #2: 0001-debuginfod-Fracture-tests-run-debuginfod-find.sh-int.patch --]
[-- Type: text/x-patch, Size: 127461 bytes --]
From c3a232d3d4d5f4c7768f41a09957b5edacacbc25 Mon Sep 17 00:00:00 2001
From: Noah Sanci <nsanci@redhat.com>
Date: Tue, 10 Aug 2021 11:21:35 -0400
Subject: [PATCH] debuginfod: Fracture tests/run-debuginfod-find.sh into
specific tests
tests/run-debuginfod-find.sh was a massive test script with many broadly
varying tests. This caused the test script to fail when any number of
things went wrong and because of its intertwined nature, detecting the
source of a failure could be difficult. The size of the test script
also meant many unrelated tests were run making the testing process
unnecessarily lengthy.
This patch fractures tests/run-debuginfod-find.sh into smaller, more
manageable individual test script files. This ensures that when failure
occurs, a programmer can easily determine where their patch went
wrong. It also allows programmers to specify exactly which tests to
run, making testing more efficient. Redundancies are also reduced by
placing code in tests/test-subr.sh.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
---
ChangeLog | 3 +
Makefile.am | 2 +-
tests/ChangeLog | 26 +
tests/Makefile.am | 33 +-
tests/run-debuginfod-000-permission.sh | 82 +++
tests/run-debuginfod-archive-groom.sh | 151 ++++
tests/run-debuginfod-archive-rename.sh | 93 +++
tests/run-debuginfod-archive-test.sh | 81 +++
tests/run-debuginfod-artifact-running.sh | 120 +++
tests/run-debuginfod-dlopen.sh | 81 +++
tests/run-debuginfod-duplicate-urls.sh | 49 ++
| 99 +++
tests/run-debuginfod-fd-prefetch-caches.sh | 54 ++
tests/run-debuginfod-federation-link.sh | 159 ++++
tests/run-debuginfod-federation-metrics.sh | 212 ++++++
tests/run-debuginfod-federation-sqlite.sh | 200 +++++
tests/run-debuginfod-file.sh | 39 +
tests/run-debuginfod-find.sh | 807 ---------------------
tests/run-debuginfod-malformed.sh | 107 +++
tests/run-debuginfod-no-urls.sh | 38 +
tests/run-debuginfod-query-retry.sh | 34 +
tests/run-debuginfod-regex.sh | 95 +++
tests/run-debuginfod-sizetime.sh | 75 ++
tests/run-debuginfod-tmp-home.sh | 122 ++++
tests/run-debuginfod-writable.sh | 82 +++
tests/test-subr.sh | 124 +++-
26 files changed, 2156 insertions(+), 812 deletions(-)
create mode 100755 tests/run-debuginfod-000-permission.sh
create mode 100755 tests/run-debuginfod-archive-groom.sh
create mode 100755 tests/run-debuginfod-archive-rename.sh
create mode 100755 tests/run-debuginfod-archive-test.sh
create mode 100755 tests/run-debuginfod-artifact-running.sh
create mode 100755 tests/run-debuginfod-dlopen.sh
create mode 100755 tests/run-debuginfod-duplicate-urls.sh
create mode 100755 tests/run-debuginfod-extraction.sh
create mode 100755 tests/run-debuginfod-fd-prefetch-caches.sh
create mode 100755 tests/run-debuginfod-federation-link.sh
create mode 100755 tests/run-debuginfod-federation-metrics.sh
create mode 100755 tests/run-debuginfod-federation-sqlite.sh
create mode 100755 tests/run-debuginfod-file.sh
delete mode 100755 tests/run-debuginfod-find.sh
create mode 100755 tests/run-debuginfod-malformed.sh
create mode 100755 tests/run-debuginfod-no-urls.sh
create mode 100755 tests/run-debuginfod-query-retry.sh
create mode 100755 tests/run-debuginfod-regex.sh
create mode 100755 tests/run-debuginfod-sizetime.sh
create mode 100755 tests/run-debuginfod-tmp-home.sh
create mode 100755 tests/run-debuginfod-writable.sh
diff --git a/ChangeLog b/ChangeLog
index 12b8f403..0aeac175 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,6 @@
+2021-08-23 Noah Sanci <nsanci@redhat.com
+ * ChangeLog: removed --sign flag on rpmbuild
+
2021-07-28 Mark Wielaard <mark@klomp.org>
* configure.ac (AC_CHECK_DECLS): Add reallocarray check.
diff --git a/Makefile.am b/Makefile.am
index 9c47afa9..3dfefa9f 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -45,7 +45,7 @@ distcheck-hook:
chmod -R u+w $(distdir)
rpm: dist
- rpmbuild -ts --sign elfutils-@PACKAGE_VERSION@.tar.bz2
+ rpmbuild -ts elfutils-@PACKAGE_VERSION@.tar.bz2
if GCOV
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 3bfd1ca2..22c50773 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,29 @@
+2021-08-20 Noah Sanci <nsanci@redhat.com>
+
+ * debuginfod-find.sh: Separated file into
+ run-debuginfod-000-permission.sh,
+ run-debuginfod-archive-groom.sh,
+ run-debuginfod-archive-rename.sh,
+ run-debuginfod-archive-test.sh,
+ run-debuginfod-artifact-running.sh,
+ run-debuginfod-dlopen.sh,
+ run-debuginfod-duplicate-urls.sh,
+ run-debuginfod-extraction.sh,
+ run-debuginfod-fd-prefetch-caches.sh,
+ run-debuginfod-federation-link.sh,
+ run-debuginfod-federation-metrics.sh,
+ run-debuginfod-federation-sqlite.sh,
+ run-debuginfod-file.sh,
+ run-debuginfod-malformed.sh,
+ run-debuginfod-no-urls.sh,
+ run-debuginfod-query-retry.sh,
+ run-debuginfod-regex.sh,
+ run-debuginfod-sizetime.sh,
+ run-debuginfod-tmp-home.sh, and
+ run-debuginfod-writable.sh
+ * tests/Makefile.am: Added the above new files to the test suite
+ * tests/test-subr.sh: Added some general functions for above tests
+
2021-08-04 Mark Wielaard <mark@klomp.org>
PR28190
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 429649f4..46ffe301 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -211,7 +211,16 @@ if DEBUGINFOD
check_PROGRAMS += debuginfod_build_id_find
# With the dummy delegation doesn't work
if !DUMMY_LIBDEBUGINFOD
-TESTS += run-debuginfod-find.sh
+TESTS += run-debuginfod-dlopen.sh run-debuginfod-artifact-running.sh \
+ run-debuginfod-fd-prefetch-caches.sh run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh run-debuginfod-extraction.sh \
+ run-debuginfod-archive-groom.sh run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-federation-link.sh run-debuginfod-federation-metrics.sh
endif
endif
@@ -474,8 +483,26 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-elfclassify.sh run-elfclassify-self.sh \
run-disasm-riscv64.sh \
testfile-riscv64-dis1.o.bz2 testfile-riscv64-dis1.expect.bz2 \
- run-debuginfod-find.sh \
- debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-dlopen.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-federation-sqlite.sh \
+ debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debugsource-1.0-2.x86_64.rpm \
diff --git a/tests/run-debuginfod-000-permission.sh b/tests/run-debuginfod-000-permission.sh
new file mode 100755
index 00000000..308cdf6d
--- /dev/null
+++ b/tests/run-debuginfod-000-permission.sh
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+# PR25628
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# The query is designed to fail, while the 000-permission file should be created.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
+ err
+fi
+
+if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
+ err
+fi
+
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" != "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} has changed."
+ err
+fi
+
+# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
+sleep 1
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" == "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
new file mode 100755
index 00000000..9afe3000
--- /dev/null
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -0,0 +1,151 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Be patient when run on a busy machine things might take a bit.
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Build a non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/F/prog.c
+gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a ${PWD}/F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+tempfiles ${PWD}/F/prog ${PWD}/F/prog.c
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+tempfiles vlog3 $DB
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
+
+# Drop some of the artifacts, run a groom cycle; confirm that
+# debuginfod has forgotten them, but remembers others
+rm -r R/debuginfod-rpms/rhel6/*
+kill -USR2 $PID1 # groom cycle
+## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# Expect 4 rpms containing 2 buildids to be deleted by the groom
+wait_ready $PORT1 'groomed_total{decision="stale"}' 4
+
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# this is one of the buildids from the groom-deleted rpms
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
+
+# but this one was not deleted so should be still around
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID || true
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
new file mode 100755
index 00000000..38143c46
--- /dev/null
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DEBUGINFOD_CACHE_PATH $DB
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+
+########################################################################
+## PR26810: Now rename some files in the R directory, then rescan, so
+# there are two copies of the same buildid in the index, one for the
+# no-longer-existing file name, and one under the new name.
+
+# run a groom cycle to force server to drop its fdcache
+kill -USR2 $PID1 # groom cycle
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
+# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
+mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# retest rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+
+egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0;
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
new file mode 100755
index 00000000..cc76b6a1
--- /dev/null
+++ b/tests/run-debuginfod-archive-test.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# find an unused port number
+mkdir R
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -t0 -g0 -v R > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+# arch
+#archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
new file mode 100755
index 00000000..95ae0a3c
--- /dev/null
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+echo 'int main(int argc, char * argv){ return 0; }' > ${PWD}/prog.c
+gcc -Wl,--build-id -g -o prog ${PWD}/prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d $DB -F -p $PORT1 -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+mv prog F
+mv prog.debug F
+tempfiles prog/F
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+kill -USR1 $PID1
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Add artifacts to the search paths and test whether debuginfod finds them while already running.
+# Build another, non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/prog2.c
+tempfiles prog2.c
+gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
+#testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
+mv prog2 F
+#mv prog2.debug F
+tempfiles F/prog2 F/prog2.debug
+
+kill -USR1 $PID1
+# Now there should be 3 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/prog.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/prog`
+cmp $filename F/prog
+
+# raw source filename
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../prog.c`
+cmp $filename ${PWD}/prog.c
+
+# and also the canonicalized one
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/prog.c`
+cmp $filename ${PWD}/prog.c
+
+# Rerun same tests for the prog2 binary
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
+cmp $filename F/prog2
+grep -q Progress vlog
+grep -q Downloaded.from vlog
+tempfiles vlog
+filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
+cmp $filename F/prog2
+grep -q 'Downloading.*http' vlog2
+tempfiles vlog2
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
+cmp $filename ${PWD}/prog2.c
+
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
new file mode 100755
index 00000000..e87eb5e2
--- /dev/null
+++ b/tests/run-debuginfod-dlopen.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a'
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether elfutils, via the debuginfod client library dlopen hooks,
+# is able to fetch debuginfo from the local debuginfod.
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-duplicate-urls.sh b/tests/run-debuginfod-duplicate-urls.sh
new file mode 100755
index 00000000..5ecbb206
--- /dev/null
+++ b/tests/run-debuginfod-duplicate-urls.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+########################################################################
+## PR27983
+# Ensure no duplicate urls are used in when querying servers for files
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:7999" \
+ LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -vvv executable 0 > vlog1 2>&1 || true
+tempfiles vlog1
+cat vlog1
+if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog1 ) -ne 2 ]; then
+ echo "Duplicate servers remain";
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
--git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
new file mode 100755
index 00000000..c9d3b053
--- /dev/null
+++ b/tests/run-debuginfod-extraction.sh
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir R Z
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
+tb2=$(find Z -name \*tar.bz2 | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh
new file mode 100755
index 00000000..978643a3
--- /dev/null
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+FDCACHE_FDS=100
+FDCACHE_MBS=100
+PREFETCH_FDS=100
+PREFETCH_MBS=100
+get_ports
+
+echo $PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 \
+ --fdcache-mbs=$FDCACHE_MDS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS \
+ --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -v -F F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+grep 'fdcache fds ' vlog$PORT1 #$FDCACHE_FDS
+grep 'fdcache mbs ' vlog$PORT1 #$FDCACHE_MBS
+grep 'prefetch fds ' vlog$PORT1 #$PREFETCH_FDS
+grep 'prefetch mbs ' vlog$PORT1 #$PREFETCH_MBS
+# search the vlog to find what metric counts should be and check the correct metrics
+# were incrimented
+wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
new file mode 100755
index 00000000..20319519
--- /dev/null
+++ b/tests/run-debuginfod-federation-link.sh
@@ -0,0 +1,159 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+
+kill -USR1 $PID2
+# Wait till both files are in the index.
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/run-debuginfod-federation-metrics.sh b/tests/run-debuginfod-federation-metrics.sh
new file mode 100755
index 00000000..234a71b0
--- /dev/null
+++ b/tests/run-debuginfod-federation-metrics.sh
@@ -0,0 +1,212 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+# Clean old dirictories
+mkdir D L F
+
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+get_ports
+
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+########################################################################
+# Fetch some metrics
+curl -s http://127.0.0.1:$PORT1/badapi
+curl -s http://127.0.0.1:$PORT1/metrics
+curl -s http://127.0.0.1:$PORT2/metrics
+curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
+curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# Confirm that some debuginfod client pools are being used
+curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
+
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+exit 0
+
diff --git a/tests/run-debuginfod-federation-sqlite.sh b/tests/run-debuginfod-federation-sqlite.sh
new file mode 100755
index 00000000..e7e00544
--- /dev/null
+++ b/tests/run-debuginfod-federation-sqlite.sh
@@ -0,0 +1,200 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D F > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+########################################################################
+# Corrupt the sqlite database and get debuginfod to trip across its errors
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
+dd if=/dev/zero of=$DB bs=1 count=1
+
+# trigger some random activity that's Sure to get sqlite3 upset
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -USR2 $PID1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
+# Run the tests again without the servers running. The target file should
+# be found in the cache.
+
+kill -INT $PID1 $PID2
+wait $PID1 $PID2
+PID1=0
+PID2=0
+tempfiles .debuginfod_*
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# check out the debuginfod logs for the new style status lines
+cat vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
+
+exit 0
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
new file mode 100755
index 00000000..3187c92b
--- /dev/null
+++ b/tests/run-debuginfod-file.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Test fetching a file using file:// . No debuginfod server needs to be run for
+# this test.
+local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
+mkdir -p ${local_dir}
+echo "int main() { return 0; }" > ${local_dir}/main.c
+# first test that is doesn't work, when no DEBUGINFOD_URLS is set
+DEBUGINFOD_URLS=""
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
+
+# Now test is with proper DEBUGINFOD_URLS
+DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
+cmp $filename ${local_dir}/main.c
+
+exit 0
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
deleted file mode 100755
index 991d1dc5..00000000
--- a/tests/run-debuginfod-find.sh
+++ /dev/null
@@ -1,807 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (C) 2019-2021 Red Hat, Inc.
-# This file is part of elfutils.
-#
-# This file is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 3 of the License, or
-# (at your option) any later version.
-#
-# elfutils is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-. $srcdir/test-subr.sh # includes set -e
-
-type curl 2>/dev/null || (echo "need curl"; exit 77)
-type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
-type cpio 2>/dev/null || (echo "need cpio"; exit 77)
-type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
-bsdtar --version | grep -q zstd && zstd=true || zstd=false
-echo "zstd=$zstd bsdtar=`bsdtar --version`"
-
-# for test case debugging, uncomment:
-set -x
-VERBOSE=-vvv
-
-DB=${PWD}/.debuginfod_tmp.sqlite
-tempfiles $DB
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
-
-PID1=0
-PID2=0
-PID3=0
-PID4=0
-
-cleanup()
-{
- if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
- if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
- if [ $PID3 -ne 0 ]; then kill $PID3; wait $PID3; fi
- if [ $PID4 -ne 0 ]; then kill $PID4; wait $PID4; fi
- rm -rf F R D L Z ${PWD}/foobar ${PWD}/mocktree ${PWD}/.client_cache* ${PWD}/tmp*
- exit_cleanup
-}
-
-# clean up trash if we were aborted early
-trap cleanup 0 1 2 3 5 9 15
-
-errfiles_list=
-err() {
- echo ERROR REPORTS
- for ports in $PORT1 $PORT2 $PORT3
- do
- echo ERROR REPORT $port metrics
- curl -s http://127.0.0.1:$port/metrics
- echo
- done
- for x in $errfiles_list
- do
- echo ERROR REPORT "$x"
- cat $x
- echo
- done
- false # trigger set -e
-}
-trap err ERR
-
-errfiles() {
- errfiles_list="$errfiles_list $*"
-}
-
-
-# find an unused port number
-while true; do
- PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT1" || break
-done
-
-# We want to run debuginfod in the background. We also want to start
-# it with the same check/installcheck-sensitive LD_LIBRARY_PATH stuff
-# that the testrun alias sets. But: we if we just use
-# testrun .../debuginfod
-# it runs in a subshell, with different pid, so not helpful.
-#
-# So we gather the LD_LIBRARY_PATH with this cunning trick:
-ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
-
-mkdir F R L D Z
-# not tempfiles F R L D Z - they are directories which we clean up manually
-ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
-
-wait_ready()
-{
- port=$1;
- what=$2;
- value=$3;
- timeout=20;
-
- echo "Wait $timeout seconds on $port for metric $what to change to $value"
- while [ $timeout -gt 0 ]; do
- mvalue="$(curl -s http://127.0.0.1:$port/metrics \
- | grep "$what" | awk '{print $NF}')"
- if [ -z "$mvalue" ]; then mvalue=0; fi
- echo "metric $what: $mvalue"
- if [ "$mvalue" -eq "$value" ]; then
- break;
- fi
- sleep 0.5;
- ((timeout--));
- done;
-
- if [ $timeout -eq 0 ]; then
- echo "metric $what never changed to $value on port $port"
- err
- fi
-}
-
-FDCACHE_FDS=50
-FDCACHE_MBS=190
-PREFETCH_FDS=10
-PREFETCH_MBS=120
-# create a bogus .rpm file to evoke a metric-visible error
-# Use a cyclic symlink instead of chmod 000 to make sure even root
-# would see an error (running the testsuite under root is NOT encouraged).
-ln -s R/nothing.rpm R/nothing.rpm
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -d $DB -p $PORT1 -t0 -g0 --fdcache-mbs=$FDCACHE_MBS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -Z .tar.xz -Z .tar.bz2=bzcat -v R F Z L > vlog$PORT1 2>&1 &
-PID1=$!
-tempfiles vlog$PORT1
-errfiles vlog$PORT1
-# Server must become ready
-wait_ready $PORT1 'ready' 1
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
-
-# Be patient when run on a busy machine things might take a bit.
-export DEBUGINFOD_TIMEOUT=10
-
-# Check thread comm names
-ps -q $PID1 -e -L -o '%p %c %a' | grep groom
-ps -q $PID1 -e -L -o '%p %c %a' | grep scan
-ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
-
-# We use -t0 and -g0 here to turn off time-based scanning & grooming.
-# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
-
-########################################################################
-
-# Compile a simple program, strip its debuginfo and save the build-id.
-# Also move the debuginfo into another directory so that elfutils
-# cannot find it without debuginfod.
-echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
-tempfiles p+r%o\$g.c
-# Create a subdirectory to confound source path names
-mkdir foobar
-gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
-testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
-
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-mv p+r%o\$g F
-mv p+r%o\$g.debug F
-kill -USR1 $PID1
-# Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-########################################################################
-
-# Test whether elfutils, via the debuginfod client library dlopen hooks,
-# is able to fetch debuginfo from the local debuginfod.
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-########################################################################
-## PR27892
-# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
-# code
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-tempfiles find-vlog$PORT1
-# wait for the server to fail the same number of times the query is retried.
-wait_ready $PORT1 'http_responses_after_you_milliseconds_count{code="406"}' 1
-# ensure all reporting is functional
-grep 'serving file '$(realpath ${PWD})'/F/p+r%o\$g.debug' vlog$PORT1
-grep 'File too large' vlog$PORT1
-grep 'using max size 1B' find-vlog$PORT1
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxsize check"
- err
-fi
-
-# Ensure DEBUGINFOD_MAXTIME is functional
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:8002/" DEBUGINFOD_MAXTIME=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-grep 'using max time' find-vlog$PORT1
-# Ensure p+r%o\$g.debug is NOT cached
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxtime check"
- err
-fi
-########################################################################
-# PR25628
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# The query is designed to fail, while the 000-permission file should be created.
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
- err
-fi
-
-if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
- err
-fi
-
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" != "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} has changed."
- err
-fi
-
-# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
-sleep 1
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" == "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
- err
-fi
-########################################################################
-
-# Test whether debuginfod-find is able to fetch those files.
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
-cmp $filename F/p+r%o\$g.debug
-if [ -w $filename ]; then
- echo "cache file writable, boo"
- err
-fi
-
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/p+r%o\\$g`
-cmp $filename F/p+r%o\$g
-
-# raw source filename
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-# and also the canonicalized one
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-
-########################################################################
-
-# Test whether the cache default locations are correct
-
-mkdir tmphome
-
-# $HOME/.cache should be created.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $HOME/.cache should be found.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $XDG_CACHE_HOME should take priority over $HOME.cache.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpxdg/"
- err
-fi
-
-# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
-# priority over $HOME/.cache, $XDG_CACHE_HOME.
-cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
-# ||true is for tolerating errors, such a valgrind or something else
-# leaving 000-perm files in there
-
-# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
-mkdir tmphome/.debuginfod_client_cache/deadbeef
-echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
-cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-
-# $DEBUGINFO_CACHE_PATH should take priority over all else.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpcache/"
- err
-fi
-
-########################################################################
-
-# Add artifacts to the search paths and test whether debuginfod finds them while already running.
-
-# Build another, non-stripped binary
-echo "int main() { return 0; }" > ${PWD}/prog2.c
-tempfiles prog2.c
-gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
-BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
-
-mv prog2 F
-kill -USR1 $PID1
-# Now there should be 3 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-cp $DB $DB.backup
-tempfiles $DB.backup
-# Rerun same tests for the prog2 binary
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
-cmp $filename F/prog2
-cat vlog
-grep -q Progress vlog
-grep -q Downloaded.from vlog
-tempfiles vlog
-filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
-cmp $filename F/prog2
-cat vlog2
-grep -q 'Downloading.*http' vlog2
-tempfiles vlog2
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
-cmp $filename ${PWD}/prog2.c
-
-cp -rvp ${abs_srcdir}/debuginfod-rpms R
-if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
- rm -vrf R/debuginfod-rpms/fedora31
-fi
-
-cp -rvp ${abs_srcdir}/debuginfod-tars Z
-kill -USR1 $PID1
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# All rpms need to be in the index, except the dummy permission-000 one
-rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
-txz=$(find Z -name \*tar.xz | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
-tb2=$(find Z -name \*tar.bz2 | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
-
-kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 5
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# Expect all source files found in the rpms (they are all called hello.c :)
-# We will need to extract all rpms (in their own directory) and could all
-# sources referenced in the .debug files.
-mkdir extracted
-cd extracted
-subdir=0;
-newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
-for i in $newrpms; do
- subdir=$[$subdir+1];
- mkdir $subdir;
- cd $subdir;
- ls -lah ../$i
- rpm2cpio ../$i | cpio -ivd;
- cd ..;
-done
-sourcefiles=$(find -name \*\\.debug \
- | env LD_LIBRARY_PATH=$ldpath xargs \
- ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
- | grep mtime: | wc --lines)
-cd ..
-rm -rf extracted
-
-wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
-
-########################################################################
-# PR27983 ensure no duplicate urls are used in when querying servers for files
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http:127.0.0.1:7999" \
- LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -v executable $BUILDID2 > vlog4 2>&1 || true
-tempfiles vlog4
-if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog4 ) -ne 2 ]; then
- echo "Duplicated servers remain";
- err
-fi
-########################################################################
-# Run a bank of queries against the debuginfod-rpms / debuginfod-debs test cases
-
-archive_test() {
- __BUILDID=$1
- __SOURCEPATH=$2
- __SOURCESHA1=$3
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- # check that timestamps are plausible - older than the near-present (tmpdir mtime)
- test $filename -ot `pwd`
-
- # run again to assure that fdcache is being enjoyed
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- if test "x$__SOURCEPATH" != "x"; then
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
- hash=`cat $filename | sha1sum | awk '{print $1}'`
- test $__SOURCESHA1 = $hash
- test $filename -ot `pwd`
- fi
-}
-
-
-# common source file sha1
-SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
-# fedora31
-if [ $zstd = true ]; then
- # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
- # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
- archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
- archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
-fi
-# fedora30
-archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-# rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-# rhel6
-archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
-# arch
-archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
-
-RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir, for a later test
-
-
-########################################################################
-
-# Drop some of the artifacts, run a groom cycle; confirm that
-# debuginfod has forgotten them, but remembers others
-
-rm -r R/debuginfod-rpms/rhel6/*
-kill -USR2 $PID1 # groom cycle
-# 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
-wait_ready $PORT1 'thread_work_total{role="groom"}' 2
-# Expect 4 rpms containing 2 buildids to be deleted by the groom
-wait_ready $PORT1 'groomed_total{decision="stale"}' 4
-
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# this is one of the buildids from the groom-deleted rpms
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
-# but this one was not deleted so should be still around
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2
-
-########################################################################
-
-# PR26810: Now rename some files in the R directory, then rescan, so
-# there are two copies of the same buildid in the index, one for the
-# no-longer-existing file name, and one under the new name.
-
-# run a groom cycle to force server to drop its fdcache
-kill -USR2 $PID1 # groom cycle
-wait_ready $PORT1 'thread_work_total{role="groom"}' 3
-# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
-mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 6
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 7
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 8
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# retest rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-
-egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
-
-########################################################################
-## PR25978
-# Ensure that the fdcache options are working.
-grep "prefetch fds" vlog$PORT1
-grep "prefetch mbs" vlog$PORT1
-grep "fdcache fds" vlog$PORT1
-grep "fdcache mbs" vlog$PORT1
-# search the vlog to find what metric counts should be and check the correct metrics
-# were incrimented
-wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
-########################################################################
-
-# Federation mode
-
-# find another unused port
-while true; do
- PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT2" || break
-done
-
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
-mkdir -p $DEBUGINFOD_CACHE_PATH
-# NB: inherits the DEBUGINFOD_URLS to the first server
-# NB: run in -L symlink-following mode for the L subdir
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d ${DB}_2 -p $PORT2 -L L D > vlog$PORT2 2>&1 &
-PID2=$!
-tempfiles vlog$PORT2
-errfiles vlog$PORT2
-tempfiles ${DB}_2
-wait_ready $PORT2 'ready' 1
-wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
-wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
-wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
-
-# have clients contact the new server
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-
-if type bsdtar 2>/dev/null; then
- # copy in the deb files
- cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
- kill -USR1 $PID2
- wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
- wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
- wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
- # All debs need to be in the index
- debs=$(find D -name \*.deb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
- ddebs=$(find D -name \*.ddeb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
-
- # ubuntu
- archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
-fi
-
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# send a request to stress XFF and User-Agent federation relay;
-# we'll grep for the two patterns in vlog$PORT1
-curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
-
-grep UA:TESTCURL vlog$PORT1
-grep XFF:TESTXFF vlog$PORT1
-
-
-# confirm that first server can't resolve symlinked info in L/ but second can
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
-file L/foo
-file -L L/foo
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test again with scheme free url
-export DEBUGINFOD_URLS=127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test parallel queries in client
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
-mkdir -p $DEBUGINFOD_CACHE_PATH
-export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-########################################################################
-
-# Fetch some metrics
-curl -s http://127.0.0.1:$PORT1/badapi
-curl -s http://127.0.0.1:$PORT1/metrics
-curl -s http://127.0.0.1:$PORT2/metrics
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*fdcache'
-curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
-
-# And generate a few errors into the second debuginfod's logs, for analysis just below
-curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
-# NB: this error is used to seed the 404 failure for the survive-404 tests
-
-# Confirm bad artifact types are rejected without leaving trace
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
-(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
-
-# DISABLE VALGRIND checking because valgrind might use debuginfod client
-# requests itself, causing confusion about who put what in the cache.
-# It stays disabled till the end of this test.
-unset VALGRIND_CMD
-
-# Confirm that reused curl connections survive 404 errors.
-# The rm's force an uncached fetch
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# Confirm that some debuginfod client pools are being used
-curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
-
-# Trigger a flood of requests against the same archive content file.
-# Use a file that hasn't been previously extracted in to make it
-# likely that even this test debuginfod will experience concurrency
-# and impose some "after-you" delays.
-(for i in `seq 100`; do
- curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
- done;
- wait)
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
-# If we could guarantee some minimum number of seconds of CPU time, we
-# could assert that the after_you metrics show some nonzero amount of
-# waiting. A few hundred ms is typical on this developer's workstation.
-
-
-########################################################################
-# Corrupt the sqlite database and get debuginfod to trip across its errors
-
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
-ls -al $DB
-dd if=/dev/zero of=$DB bs=1 count=1
-ls -al $DB
-# trigger some random activity that's Sure to get sqlite3 upset
-kill -USR1 $PID1
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 9
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -USR2 $PID1
-wait_ready $PORT1 'thread_work_total{role="groom"}' 4
-curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
-
-########################################################################
-
-# Run the tests again without the servers running. The target file should
-# be found in the cache.
-
-kill -INT $PID1 $PID2
-wait $PID1 $PID2
-PID1=0
-PID2=0
-tempfiles .debuginfod_*
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-# check out the debuginfod logs for the new style status lines
-# cat vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
-
-########################################################################
-
-# Add some files to the cache that do not fit its naming format.
-# They should survive cache cleaning.
-mkdir $DEBUGINFOD_CACHE_PATH/malformed
-touch $DEBUGINFOD_CACHE_PATH/malformed0
-touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
-
-# A valid format for an empty buildid subdirectory
-mkdir $DEBUGINFOD_CACHE_PATH/00000000
-touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
-
-# Trigger a cache clean and run the tests again. The clients should be unable to
-# find the target.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
-echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID2 && false || true
-
-if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
- || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
- echo "unrelated files did not survive cache cleaning"
- err
-fi
-
-if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
- echo "failed to rmdir old cache dir"
- err
-fi
-
-# Test debuginfod without a path list; reuse $PORT1
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
-PID3=$!
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -int $PID3
-wait $PID3
-PID3=0
-
-########################################################################
-# Test fetching a file using file:// . No debuginfod server needs to be run for
-# this test.
-local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
-mkdir -p ${local_dir}
-echo "int main() { return 0; }" > ${local_dir}/main.c
-
-# first test that is doesn't work, when no DEBUGINFOD_URLS is set
-DEBUGINFOD_URLS=""
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
-
-# Now test is with proper DEBUGINFOD_URLS
-DEBUGINFOD_URLS="file://${PWD}/mocktree/"
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
-cmp $filename ${local_dir}/main.c
-
-########################################################################
-## PR27711
-# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
-while true; do
- PORT3=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT3" || break
-done
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT3/" ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT3 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d $DB.backup > vlog$PORT3 2>&1 &
-PID4=$!
-wait_ready $PORT3 'ready' 1
-tempfiles vlog$PORT3
-errfiles vlog$PORT3
-
-kill -USR2 $PID4
-wait_ready $PORT3 'thread_work_total{role="groom"}' 1
-wait_ready $PORT3 'groom{statistic="archive d/e"}' 0
-wait_ready $PORT3 'groom{statistic="archive sdef"}' 0
-wait_ready $PORT3 'groom{statistic="archive sref"}' 0
-wait_ready $PORT3 'groom{statistic="buildids"}' 0
-wait_ready $PORT3 'groom{statistic="file d/e"}' 0
-wait_ready $PORT3 'groom{statistic="file s"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (#)"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (mb)"}' 0
-
-kill $PID4
-wait $PID4
-PID4=0
-
-########################################################################
-# set up tests for retrying failed queries.
-retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
- | grep -c 'Retry failed query'`
-if [ $retry_attempts -ne 10 ]; then
- echo "retry mechanism failed."
- exit 1;
- fi
-
-exit 0
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
new file mode 100755
index 00000000..b420a45b
--- /dev/null
+++ b/tests/run-debuginfod-malformed.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# find an unused port number
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Add some files to the cache that do not fit its naming format.
+# They should survive cache cleaning.
+mkdir $DEBUGINFOD_CACHE_PATH/malformed
+touch $DEBUGINFOD_CACHE_PATH/malformed0
+touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
+
+# A valid format for an empty buildid subdirectory
+mkdir $DEBUGINFOD_CACHE_PATH/00000000
+touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
+
+# Trigger a cache clean and run the tests again. The clients should be unable to
+# find the target.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
+echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+
+if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
+ || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
+ echo "unrelated files did not survive cache cleaning"
+ err
+fi
+
+if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
+ echo "failed to rmdir old cache dir"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-no-urls.sh b/tests/run-debuginfod-no-urls.sh
new file mode 100755
index 00000000..53c29f4c
--- /dev/null
+++ b/tests/run-debuginfod-no-urls.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# Test debuginfod without a path list
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
+PID1=$!
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -int $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
new file mode 100755
index 00000000..918c6801
--- /dev/null
+++ b/tests/run-debuginfod-query-retry.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# set up tests for retrying failed queries.
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
+ | grep -c 'Retry failed query'`
+if [ $retry_attempts -ne 10 ]; then
+ echo "retry mechanism failed."
+ exit 1;
+fi
+
+exit 0;
diff --git a/tests/run-debuginfod-regex.sh b/tests/run-debuginfod-regex.sh
new file mode 100755
index 00000000..e36d8f5f
--- /dev/null
+++ b/tests/run-debuginfod-regex.sh
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+get_ports
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+
+mv prog F
+mv prog.debug F
+tempfiles F/prog.debug F/prog
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+cp ${DB} ${DB}.backup
+tempfiles ${DB}.backup
+
+kill $PID1
+wait $PID1
+PID1=0
+
+#######################################################################
+## PR27711
+# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
+#
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -p $PORT2 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d ${DB}.backup > vlog$PORT2 2>&1 &
+
+#reuse PID1
+PID1=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+# Server must become ready
+wait_ready $PORT2 'ready' 1
+
+kill -USR2 $PID1
+wait_ready $PORT2 'thread_work_total{role="groom"}' 1
+wait_ready $PORT2 'groom{statistic="archive d/e"}' 0
+wait_ready $PORT2 'groom{statistic="archive sdef"}' 0
+wait_ready $PORT2 'groom{statistic="archive sref"}' 0
+wait_ready $PORT2 'groom{statistic="buildids"}' 0
+wait_ready $PORT2 'groom{statistic="file d/e"}' 0
+wait_ready $PORT2 'groom{statistic="file s"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (#)"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (mb)"}' 0
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
+
diff --git a/tests/run-debuginfod-sizetime.sh b/tests/run-debuginfod-sizetime.sh
new file mode 100755
index 00000000..edf4cce5
--- /dev/null
+++ b/tests/run-debuginfod-sizetime.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+set -x
+unset VALGRIND_CMD
+
+get_ports
+
+echo "int main() { return 0; }" > ${PWD}/prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+## PR27892
+# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
+# code
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v executable ${PWD}/prog 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+errfiles find-vlog$PORT1
+echo "Checking maxsize"
+grep "using max size 1B" find-vlog$PORT1
+echo "Checking maxsize"
+grep 'serving file '$(realpath ${PWD})'/prog' vlog$PORT1
+echo "Checking maxsize"
+grep 'File too large' vlog$PORT1
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxsize check"
+ err
+fi
+# Ensure no file is downloaded for longer than DEBUGINFOD_MAXTIME
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXTIME=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo ${PWD}/prog.debug 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+grep 'using max time' find-vlog$PORT1
+# Ensure p+r%o\$g.debug is NOT cached
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxtime check"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
new file mode 100755
index 00000000..0f2f2d53
--- /dev/null
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether the cache default locations are correct
+mkdir tmphome
+
+# $HOME/.cache should be created.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+
+# $HOME/.cache should be found.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+# $XDG_CACHE_HOME should take priority over $HOME.cache.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpxdg/"
+ err
+fi
+
+# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
+# priority over $HOME/.cache, $XDG_CACHE_HOME.
+cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
+# ||true is for tolerating errors, such a valgrind or something else
+# leaving 000-perm files in there
+
+# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
+mkdir tmphome/.debuginfod_client_cache/deadbeef
+echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
+cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+
+# $DEBUGINFO_CACHE_PATH should take priority over all else.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpcache/"
+ err
+fi
+rm -rf ${PWD}/tmphome/ ${PWD}/tmpxdg ${PWD}/tmpcache
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
new file mode 100755
index 00000000..edec9357
--- /dev/null
+++ b/tests/run-debuginfod-writable.sh
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+#######################################################################
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/p+r%o\$g.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/test-subr.sh b/tests/test-subr.sh
index 411e5f28..41e95e31 100644
--- a/tests/test-subr.sh
+++ b/tests/test-subr.sh
@@ -27,6 +27,13 @@ test_dir="test-$$"
mkdir -p "$test_dir"
cd "$test_dir"
+type curl 2>/dev/null || (echo "need curl"; exit 77)
+type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
+type cpio 2>/dev/null || (echo "need cpio"; exit 77)
+type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
+bsdtar --version | grep -q zstd && zstd=true || zstd=false
+echo "zstd=$zstd bsdtar=`bsdtar --version`"
+
#LC_ALL=C
#export LC_ALL
@@ -35,7 +42,11 @@ remove_files=
# Tests that trap EXIT (0) themselves should call this explicitly.
exit_cleanup()
{
- rm -f $remove_files; cd ..; rmdir $test_dir
+ rm -rf ${PWD}/.client_cache*
+ rm -f $remove_files;
+ ls -a ${PWD}
+ cd ..;
+ rmdir $test_dir
}
trap exit_cleanup 0
@@ -212,3 +223,114 @@ testrun_on_self_quiet()
# Only exit if something failed
if test $exit_status != 0; then exit $exit_status; fi
}
+
+cleanup()
+{
+ if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
+ if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
+ rm -rf ${PWD}/.client_cache F R D L Z ${PWD}/foobar ${PWD}/mocktree
+ exit_cleanup
+}
+trap cleanup 0 1 2 3 5 9 15
+errfiles_list=
+err() {
+ echo ERROR REPORTS
+ for ports in $PORT1 $PORT2
+ do
+ echo ERROR REPORT $port metrics
+ curl -s http://127.0.0.1:$port/metrics
+ echo
+ done
+ for x in $errfiles_list
+ do
+ echo ERROR REPORT "$x"
+ cat $x
+ echo
+ done
+ false # trigger set -e
+}
+trap err ERR
+
+errfiles() {
+ errfiles_list="$errfiles_list $*"
+}
+
+wait_ready()
+{
+ port=$1;
+ what=$2;
+ value=$3;
+ timeout=20;
+
+ echo "Wait $timeout seconds on $port for metric $what to change to $value"
+ while [ $timeout -gt 0 ]; do
+ mvalue="$(curl -s http://127.0.0.1:$port/metrics \
+ | grep "$what" | awk '{print $NF}')"
+ if [ -z "$mvalue" ]; then mvalue=0; fi
+ echo "metric $what: $mvalue"
+ if [ "$mvalue" -eq "$value" ]; then
+ break;
+ fi
+ sleep 0.5;
+ ((timeout--));
+ done;
+
+ if [ $timeout -eq 0 ]; then
+ echo "metric $what never changed to $value on port $port"
+ err
+ fi
+}
+
+archive_test() {
+ __BUILDID=$1
+ __SOURCEPATH=$2
+ __SOURCESHA1=$3
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ # check that timestamps are plausible - older than the near-present (tmpdir mtime)
+ test $filename -ot `pwd`
+
+ # run again to assure that fdcache is being enjoyed
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ if test "x$__SOURCEPATH" != "x"; then
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
+ hash=`cat $filename | sha1sum | awk '{print $1}'`
+ test $__SOURCESHA1 = $hash
+ test $filename -ot `pwd`
+ fi
+}
+
+get_ports() {
+ while true; do
+ PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$PORT1" || break
+ done
+# Some tests will use two servers, so assign the second var
+ while true; do
+ PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
+ done
+
+}
+
+VERBOSE=-vvv
+# We gather the LD_LIBRARY_PATH with this cunning trick:
+ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
+PORT1=0
+PORT2=0
+PID1=0
+PID2=0
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests
2021-08-23 19:31 ` Noah Sanci
@ 2021-09-02 11:29 ` Mark Wielaard
2021-09-02 13:42 ` Mark Wielaard
1 sibling, 0 replies; 6+ messages in thread
From: Mark Wielaard @ 2021-09-02 11:29 UTC (permalink / raw)
To: Noah Sanci, elfutils-devel
Hi Noah,
I am excited about this patch, but wondering about this part:
> diff --git a/ChangeLog b/ChangeLog
> index 12b8f403..0aeac175 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,3 +1,6 @@
> +2021-08-23 Noah Sanci <nsanci@redhat.com
> + * ChangeLog: removed --sign flag on rpmbuild
> +
> 2021-07-28 Mark Wielaard <mark@klomp.org>
>
> * configure.ac (AC_CHECK_DECLS): Add reallocarray check.
> diff --git a/Makefile.am b/Makefile.am
> index 9c47afa9..3dfefa9f 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -45,7 +45,7 @@ distcheck-hook:
> chmod -R u+w $(distdir)
>
> rpm: dist
> - rpmbuild -ts --sign elfutils-@PACKAGE_VERSION@.tar.bz2
> + rpmbuild -ts elfutils-@PACKAGE_VERSION@.tar.bz2
>
> if GCOV
This might be a ok thing to do. I am not sure who is actually using the
rpm target (we release src.tar.bz files, not srpms). But it seems
unrelated to the rest of the patch.
Cheers,
Mark
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests
2021-08-23 19:31 ` Noah Sanci
2021-09-02 11:29 ` Mark Wielaard
@ 2021-09-02 13:42 ` Mark Wielaard
2021-09-02 18:59 ` Noah Sanci
1 sibling, 1 reply; 6+ messages in thread
From: Mark Wielaard @ 2021-09-02 13:42 UTC (permalink / raw)
To: Noah Sanci, elfutils-devel
[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]
Hi Noah,
I really like this split up of test cases.
I added the new x-forward-ttl testcase to it.
run-debuginfod-x-forwarded-for.sh
Two tests run-debuginfod-federation-metrics.sh and run-debuginfod-
federation-sqlite.sh called get_ports twice to get an "invalid" port
number to set DEBUGINFOD_URLS. I changed this to:
export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
The only concern I have is with get_ports. There is a race condition
selecting the random port when lots of debuginfod tests are running. It
could hand out ports that are not yet used to multiple tests.
I only managed to trigger this once, with make check -j256. So it
probably isn't a big race, but getting random failures is bad.
I don't have a good solution for this though. Maybe we could assign an
unique range to each testcase? e.g.
diff --git a/tests/test-subr.sh b/tests/test-subr.sh
index 41e95e31..880d6590 100644
--- a/tests/test-subr.sh
+++ b/tests/test-subr.sh
@@ -314,14 +314,17 @@ archive_test() {
fi
}
+# takes one argument a "base" port number, this should be unique between
+# tests to select (two) random port number between $base..$base+100.
get_ports() {
+ base=$1
while true; do
- PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
+ PORT1=`expr '(' $RANDOM % 100 ')' + $base`
ss -atn | fgrep ":$PORT1" || break
done
# Some tests will use two servers, so assign the second var
while true; do
- PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
+ PORT2=`expr '(' $RANDOM % 100 ')' + $base`
ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
done
And then add a unique base to each get_port call (8000, 8100, 8200,
...). We would to be careful to never assign the same unique range to
each test though. Which requires the programmer to check all other
tests. But maybe that isn't such a big deal? git grep ^get_port tests
would give you all the currently used port ranges...
Ideas?
Attached the slightly updated patch, including the new run-debuginfod-
x-forwarded-for.sh, also on
https://code.wildebeest.org/git/user/mjw/elfutils/commit/?h=debuginfod-fracture-tests
Cheers,
Mark
[-- Attachment #2: 0001-debuginfod-Fracture-tests-run-debuginfod-find.sh-int.patch --]
[-- Type: text/x-patch, Size: 130805 bytes --]
From dec807fc3d521faf9ec14e93598da6c3a6e80865 Mon Sep 17 00:00:00 2001
From: Noah Sanci <nsanci@redhat.com>
Date: Tue, 10 Aug 2021 11:21:35 -0400
Subject: [PATCH] debuginfod: Fracture tests/run-debuginfod-find.sh into
specific tests
tests/run-debuginfod-find.sh was a massive test script with many broadly
varying tests. This caused the test script to fail when any number of
things went wrong and because of its intertwined nature, detecting the
source of a failure could be difficult. The size of the test script
also meant many unrelated tests were run making the testing process
unnecessarily lengthy.
This patch fractures tests/run-debuginfod-find.sh into smaller, more
manageable individual test script files. This ensures that when failure
occurs, a programmer can easily determine where their patch went
wrong. It also allows programmers to specify exactly which tests to
run, making testing more efficient. Redundancies are also reduced by
placing code in tests/test-subr.sh.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
---
tests/ChangeLog | 27 +
tests/Makefile.am | 43 +-
tests/run-debuginfod-000-permission.sh | 82 ++
tests/run-debuginfod-archive-groom.sh | 151 ++++
tests/run-debuginfod-archive-rename.sh | 93 +++
tests/run-debuginfod-archive-test.sh | 81 ++
tests/run-debuginfod-artifact-running.sh | 120 +++
tests/run-debuginfod-dlopen.sh | 81 ++
tests/run-debuginfod-duplicate-urls.sh | 49 ++
| 99 +++
tests/run-debuginfod-fd-prefetch-caches.sh | 54 ++
tests/run-debuginfod-federation-link.sh | 159 ++++
tests/run-debuginfod-federation-metrics.sh | 211 +++++
tests/run-debuginfod-federation-sqlite.sh | 199 +++++
tests/run-debuginfod-file.sh | 39 +
tests/run-debuginfod-find.sh | 852 ---------------------
tests/run-debuginfod-malformed.sh | 107 +++
tests/run-debuginfod-no-urls.sh | 38 +
tests/run-debuginfod-query-retry.sh | 34 +
tests/run-debuginfod-regex.sh | 95 +++
tests/run-debuginfod-sizetime.sh | 75 ++
tests/run-debuginfod-tmp-home.sh | 122 +++
tests/run-debuginfod-writable.sh | 82 ++
tests/run-debuginfod-x-forwarded-for.sh | 60 ++
tests/test-subr.sh | 124 ++-
25 files changed, 2222 insertions(+), 855 deletions(-)
create mode 100755 tests/run-debuginfod-000-permission.sh
create mode 100755 tests/run-debuginfod-archive-groom.sh
create mode 100755 tests/run-debuginfod-archive-rename.sh
create mode 100755 tests/run-debuginfod-archive-test.sh
create mode 100755 tests/run-debuginfod-artifact-running.sh
create mode 100755 tests/run-debuginfod-dlopen.sh
create mode 100755 tests/run-debuginfod-duplicate-urls.sh
create mode 100755 tests/run-debuginfod-extraction.sh
create mode 100755 tests/run-debuginfod-fd-prefetch-caches.sh
create mode 100755 tests/run-debuginfod-federation-link.sh
create mode 100755 tests/run-debuginfod-federation-metrics.sh
create mode 100755 tests/run-debuginfod-federation-sqlite.sh
create mode 100755 tests/run-debuginfod-file.sh
delete mode 100755 tests/run-debuginfod-find.sh
create mode 100755 tests/run-debuginfod-malformed.sh
create mode 100755 tests/run-debuginfod-no-urls.sh
create mode 100755 tests/run-debuginfod-query-retry.sh
create mode 100755 tests/run-debuginfod-regex.sh
create mode 100755 tests/run-debuginfod-sizetime.sh
create mode 100755 tests/run-debuginfod-tmp-home.sh
create mode 100755 tests/run-debuginfod-writable.sh
create mode 100755 tests/run-debuginfod-x-forwarded-for.sh
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 3fa7c2e4..29d0278d 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,30 @@
+2021-08-20 Noah Sanci <nsanci@redhat.com>
+
+ * debuginfod-find.sh: Separated file into
+ run-debuginfod-000-permission.sh,
+ run-debuginfod-archive-groom.sh,
+ run-debuginfod-archive-rename.sh,
+ run-debuginfod-archive-test.sh,
+ run-debuginfod-artifact-running.sh,
+ run-debuginfod-dlopen.sh,
+ run-debuginfod-duplicate-urls.sh,
+ run-debuginfod-extraction.sh,
+ run-debuginfod-fd-prefetch-caches.sh,
+ run-debuginfod-federation-link.sh,
+ run-debuginfod-federation-metrics.sh,
+ run-debuginfod-federation-sqlite.sh,
+ run-debuginfod-file.sh,
+ run-debuginfod-malformed.sh,
+ run-debuginfod-no-urls.sh,
+ run-debuginfod-query-retry.sh,
+ run-debuginfod-regex.sh,
+ run-debuginfod-sizetime.sh,
+ run-debuginfod-tmp-home.sh, and
+ run-debuginfod-writable.sh
+ run-debuginfod-x-forwarded-for.sh
+ * tests/Makefile.am: Added the above new files to the test suite
+ * tests/test-subr.sh: Added some general functions for above tests
+
2021-08-28 Mark Wielaard <mark@klomp.org>
* run-debuginfod-find.sh: Use ":memory:" for the
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 429649f4..d51f7281 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -211,7 +211,27 @@ if DEBUGINFOD
check_PROGRAMS += debuginfod_build_id_find
# With the dummy delegation doesn't work
if !DUMMY_LIBDEBUGINFOD
-TESTS += run-debuginfod-find.sh
+TESTS += run-debuginfod-dlopen.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh \
+ run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-x-forwarded-for.sh
endif
endif
@@ -474,7 +494,26 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-elfclassify.sh run-elfclassify-self.sh \
run-disasm-riscv64.sh \
testfile-riscv64-dis1.o.bz2 testfile-riscv64-dis1.expect.bz2 \
- run-debuginfod-find.sh \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-dlopen.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-x-forwarded-for.sh \
debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
diff --git a/tests/run-debuginfod-000-permission.sh b/tests/run-debuginfod-000-permission.sh
new file mode 100755
index 00000000..308cdf6d
--- /dev/null
+++ b/tests/run-debuginfod-000-permission.sh
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+# PR25628
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# The query is designed to fail, while the 000-permission file should be created.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
+ err
+fi
+
+if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
+ err
+fi
+
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" != "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} has changed."
+ err
+fi
+
+# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
+sleep 1
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" == "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
new file mode 100755
index 00000000..9afe3000
--- /dev/null
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -0,0 +1,151 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Be patient when run on a busy machine things might take a bit.
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Build a non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/F/prog.c
+gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a ${PWD}/F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+tempfiles ${PWD}/F/prog ${PWD}/F/prog.c
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+tempfiles vlog3 $DB
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
+
+# Drop some of the artifacts, run a groom cycle; confirm that
+# debuginfod has forgotten them, but remembers others
+rm -r R/debuginfod-rpms/rhel6/*
+kill -USR2 $PID1 # groom cycle
+## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# Expect 4 rpms containing 2 buildids to be deleted by the groom
+wait_ready $PORT1 'groomed_total{decision="stale"}' 4
+
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# this is one of the buildids from the groom-deleted rpms
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
+
+# but this one was not deleted so should be still around
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID || true
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
new file mode 100755
index 00000000..38143c46
--- /dev/null
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DEBUGINFOD_CACHE_PATH $DB
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+
+########################################################################
+## PR26810: Now rename some files in the R directory, then rescan, so
+# there are two copies of the same buildid in the index, one for the
+# no-longer-existing file name, and one under the new name.
+
+# run a groom cycle to force server to drop its fdcache
+kill -USR2 $PID1 # groom cycle
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
+# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
+mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# retest rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+
+egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0;
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
new file mode 100755
index 00000000..cc76b6a1
--- /dev/null
+++ b/tests/run-debuginfod-archive-test.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# find an unused port number
+mkdir R
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -t0 -g0 -v R > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+# arch
+#archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
new file mode 100755
index 00000000..95ae0a3c
--- /dev/null
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+echo 'int main(int argc, char * argv){ return 0; }' > ${PWD}/prog.c
+gcc -Wl,--build-id -g -o prog ${PWD}/prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d $DB -F -p $PORT1 -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+mv prog F
+mv prog.debug F
+tempfiles prog/F
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+kill -USR1 $PID1
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Add artifacts to the search paths and test whether debuginfod finds them while already running.
+# Build another, non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/prog2.c
+tempfiles prog2.c
+gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
+#testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
+mv prog2 F
+#mv prog2.debug F
+tempfiles F/prog2 F/prog2.debug
+
+kill -USR1 $PID1
+# Now there should be 3 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/prog.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/prog`
+cmp $filename F/prog
+
+# raw source filename
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../prog.c`
+cmp $filename ${PWD}/prog.c
+
+# and also the canonicalized one
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/prog.c`
+cmp $filename ${PWD}/prog.c
+
+# Rerun same tests for the prog2 binary
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
+cmp $filename F/prog2
+grep -q Progress vlog
+grep -q Downloaded.from vlog
+tempfiles vlog
+filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
+cmp $filename F/prog2
+grep -q 'Downloading.*http' vlog2
+tempfiles vlog2
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
+cmp $filename ${PWD}/prog2.c
+
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
new file mode 100755
index 00000000..e87eb5e2
--- /dev/null
+++ b/tests/run-debuginfod-dlopen.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a'
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether elfutils, via the debuginfod client library dlopen hooks,
+# is able to fetch debuginfo from the local debuginfod.
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-duplicate-urls.sh b/tests/run-debuginfod-duplicate-urls.sh
new file mode 100755
index 00000000..5ecbb206
--- /dev/null
+++ b/tests/run-debuginfod-duplicate-urls.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+########################################################################
+## PR27983
+# Ensure no duplicate urls are used in when querying servers for files
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:7999" \
+ LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -vvv executable 0 > vlog1 2>&1 || true
+tempfiles vlog1
+cat vlog1
+if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog1 ) -ne 2 ]; then
+ echo "Duplicate servers remain";
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
--git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
new file mode 100755
index 00000000..c9d3b053
--- /dev/null
+++ b/tests/run-debuginfod-extraction.sh
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir R Z
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
+tb2=$(find Z -name \*tar.bz2 | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh
new file mode 100755
index 00000000..978643a3
--- /dev/null
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+FDCACHE_FDS=100
+FDCACHE_MBS=100
+PREFETCH_FDS=100
+PREFETCH_MBS=100
+get_ports
+
+echo $PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 \
+ --fdcache-mbs=$FDCACHE_MDS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS \
+ --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -v -F F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+grep 'fdcache fds ' vlog$PORT1 #$FDCACHE_FDS
+grep 'fdcache mbs ' vlog$PORT1 #$FDCACHE_MBS
+grep 'prefetch fds ' vlog$PORT1 #$PREFETCH_FDS
+grep 'prefetch mbs ' vlog$PORT1 #$PREFETCH_MBS
+# search the vlog to find what metric counts should be and check the correct metrics
+# were incrimented
+wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
new file mode 100755
index 00000000..20319519
--- /dev/null
+++ b/tests/run-debuginfod-federation-link.sh
@@ -0,0 +1,159 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+
+kill -USR1 $PID2
+# Wait till both files are in the index.
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/run-debuginfod-federation-metrics.sh b/tests/run-debuginfod-federation-metrics.sh
new file mode 100755
index 00000000..defaa8cc
--- /dev/null
+++ b/tests/run-debuginfod-federation-metrics.sh
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
+tempfiles $DB
+# Clean old dirictories
+mkdir D L F
+
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+get_ports
+
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+########################################################################
+# Fetch some metrics
+curl -s http://127.0.0.1:$PORT1/badapi
+curl -s http://127.0.0.1:$PORT1/metrics
+curl -s http://127.0.0.1:$PORT2/metrics
+curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
+curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# Confirm that some debuginfod client pools are being used
+curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
+
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+exit 0
+
diff --git a/tests/run-debuginfod-federation-sqlite.sh b/tests/run-debuginfod-federation-sqlite.sh
new file mode 100755
index 00000000..573cbc83
--- /dev/null
+++ b/tests/run-debuginfod-federation-sqlite.sh
@@ -0,0 +1,199 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D F > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+########################################################################
+# Corrupt the sqlite database and get debuginfod to trip across its errors
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
+dd if=/dev/zero of=$DB bs=1 count=1
+
+# trigger some random activity that's Sure to get sqlite3 upset
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -USR2 $PID1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
+# Run the tests again without the servers running. The target file should
+# be found in the cache.
+
+kill -INT $PID1 $PID2
+wait $PID1 $PID2
+PID1=0
+PID2=0
+tempfiles .debuginfod_*
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# check out the debuginfod logs for the new style status lines
+cat vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
+
+exit 0
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
new file mode 100755
index 00000000..3187c92b
--- /dev/null
+++ b/tests/run-debuginfod-file.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Test fetching a file using file:// . No debuginfod server needs to be run for
+# this test.
+local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
+mkdir -p ${local_dir}
+echo "int main() { return 0; }" > ${local_dir}/main.c
+# first test that is doesn't work, when no DEBUGINFOD_URLS is set
+DEBUGINFOD_URLS=""
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
+
+# Now test is with proper DEBUGINFOD_URLS
+DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
+cmp $filename ${local_dir}/main.c
+
+exit 0
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
deleted file mode 100755
index 7515b7cd..00000000
--- a/tests/run-debuginfod-find.sh
+++ /dev/null
@@ -1,852 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (C) 2019-2021 Red Hat, Inc.
-# This file is part of elfutils.
-#
-# This file is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 3 of the License, or
-# (at your option) any later version.
-#
-# elfutils is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-. $srcdir/test-subr.sh # includes set -e
-
-type curl 2>/dev/null || (echo "need curl"; exit 77)
-type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
-type cpio 2>/dev/null || (echo "need cpio"; exit 77)
-type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
-bsdtar --version | grep -q zstd && zstd=true || zstd=false
-echo "zstd=$zstd bsdtar=`bsdtar --version`"
-
-# for test case debugging, uncomment:
-set -x
-VERBOSE=-vvv
-
-DB=${PWD}/.debuginfod_tmp.sqlite
-tempfiles $DB
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
-
-PID1=0
-PID2=0
-PID3=0
-PID4=0
-PID5=0
-PID6=0
-
-cleanup()
-{
- if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
- if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
- if [ $PID3 -ne 0 ]; then kill $PID3; wait $PID3; fi
- if [ $PID4 -ne 0 ]; then kill $PID4; wait $PID4; fi
- if [ $PID5 -ne 0 ]; then kill $PID5; wait $PID5; fi
- if [ $PID6 -ne 0 ]; then kill $PID6; wait $PID6; fi
- rm -rf F R D L Z ${PWD}/foobar ${PWD}/mocktree ${PWD}/.client_cache* ${PWD}/tmp*
- exit_cleanup
-}
-
-# clean up trash if we were aborted early
-trap cleanup 0 1 2 3 5 9 15
-
-errfiles_list=
-err() {
- echo ERROR REPORTS
- for ports in $PORT1 $PORT2 $PORT3 $PORT4 $PORT5
- do
- echo ERROR REPORT $port metrics
- curl -s http://127.0.0.1:$port/metrics
- echo
- done
- for x in $errfiles_list
- do
- echo ERROR REPORT "$x"
- cat $x
- echo
- done
- false # trigger set -e
-}
-trap err ERR
-
-errfiles() {
- errfiles_list="$errfiles_list $*"
-}
-
-
-# find an unused port number
-while true; do
- PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT1" || break
-done
-
-# We want to run debuginfod in the background. We also want to start
-# it with the same check/installcheck-sensitive LD_LIBRARY_PATH stuff
-# that the testrun alias sets. But: we if we just use
-# testrun .../debuginfod
-# it runs in a subshell, with different pid, so not helpful.
-#
-# So we gather the LD_LIBRARY_PATH with this cunning trick:
-ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
-
-mkdir F R L D Z
-# not tempfiles F R L D Z - they are directories which we clean up manually
-ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
-
-wait_ready()
-{
- port=$1;
- what=$2;
- value=$3;
- timeout=20;
-
- echo "Wait $timeout seconds on $port for metric $what to change to $value"
- while [ $timeout -gt 0 ]; do
- mvalue="$(curl -s http://127.0.0.1:$port/metrics \
- | grep "$what" | awk '{print $NF}')"
- if [ -z "$mvalue" ]; then mvalue=0; fi
- echo "metric $what: $mvalue"
- if [ "$mvalue" -eq "$value" ]; then
- break;
- fi
- sleep 0.5;
- ((timeout--));
- done;
-
- if [ $timeout -eq 0 ]; then
- echo "metric $what never changed to $value on port $port"
- err
- fi
-}
-
-FDCACHE_FDS=50
-FDCACHE_MBS=190
-PREFETCH_FDS=10
-PREFETCH_MBS=120
-# create a bogus .rpm file to evoke a metric-visible error
-# Use a cyclic symlink instead of chmod 000 to make sure even root
-# would see an error (running the testsuite under root is NOT encouraged).
-ln -s R/nothing.rpm R/nothing.rpm
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -d $DB -p $PORT1 -t0 -g0 --fdcache-mbs=$FDCACHE_MBS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -Z .tar.xz -Z .tar.bz2=bzcat -v R F Z L > vlog$PORT1 2>&1 &
-PID1=$!
-tempfiles vlog$PORT1
-errfiles vlog$PORT1
-# Server must become ready
-wait_ready $PORT1 'ready' 1
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
-
-# Be patient when run on a busy machine things might take a bit.
-export DEBUGINFOD_TIMEOUT=10
-
-# Check thread comm names
-ps -q $PID1 -e -L -o '%p %c %a' | grep groom
-ps -q $PID1 -e -L -o '%p %c %a' | grep scan
-ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
-
-# We use -t0 and -g0 here to turn off time-based scanning & grooming.
-# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
-
-########################################################################
-
-# Compile a simple program, strip its debuginfo and save the build-id.
-# Also move the debuginfo into another directory so that elfutils
-# cannot find it without debuginfod.
-echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
-tempfiles p+r%o\$g.c
-# Create a subdirectory to confound source path names
-mkdir foobar
-gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
-testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
-
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-mv p+r%o\$g F
-mv p+r%o\$g.debug F
-kill -USR1 $PID1
-# Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-########################################################################
-
-# Test whether elfutils, via the debuginfod client library dlopen hooks,
-# is able to fetch debuginfo from the local debuginfod.
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-########################################################################
-## PR27892
-# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
-# code
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-tempfiles find-vlog$PORT1
-# wait for the server to fail the same number of times the query is retried.
-wait_ready $PORT1 'http_responses_after_you_milliseconds_count{code="406"}' 1
-# ensure all reporting is functional
-grep 'serving file '$(realpath ${PWD})'/F/p+r%o\$g.debug' vlog$PORT1
-grep 'File too large' vlog$PORT1
-grep 'using max size 1B' find-vlog$PORT1
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxsize check"
- err
-fi
-
-# Ensure DEBUGINFOD_MAXTIME is functional
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:8002/" DEBUGINFOD_MAXTIME=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-grep 'using max time' find-vlog$PORT1
-# Ensure p+r%o\$g.debug is NOT cached
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxtime check"
- err
-fi
-########################################################################
-# PR25628
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# The query is designed to fail, while the 000-permission file should be created.
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
- err
-fi
-
-if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
- err
-fi
-
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" != "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} has changed."
- err
-fi
-
-# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
-sleep 1
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" == "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
- err
-fi
-########################################################################
-
-# Test whether debuginfod-find is able to fetch those files.
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
-cmp $filename F/p+r%o\$g.debug
-if [ -w $filename ]; then
- echo "cache file writable, boo"
- err
-fi
-
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/p+r%o\\$g`
-cmp $filename F/p+r%o\$g
-
-# raw source filename
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-# and also the canonicalized one
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-
-########################################################################
-
-# Test whether the cache default locations are correct
-
-mkdir tmphome
-
-# $HOME/.cache should be created.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $HOME/.cache should be found.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $XDG_CACHE_HOME should take priority over $HOME.cache.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpxdg/"
- err
-fi
-
-# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
-# priority over $HOME/.cache, $XDG_CACHE_HOME.
-cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
-# ||true is for tolerating errors, such a valgrind or something else
-# leaving 000-perm files in there
-
-# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
-mkdir tmphome/.debuginfod_client_cache/deadbeef
-echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
-cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-
-# $DEBUGINFO_CACHE_PATH should take priority over all else.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpcache/"
- err
-fi
-
-########################################################################
-
-# Add artifacts to the search paths and test whether debuginfod finds them while already running.
-
-# Build another, non-stripped binary
-echo "int main() { return 0; }" > ${PWD}/prog2.c
-tempfiles prog2.c
-gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
-BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
-
-mv prog2 F
-kill -USR1 $PID1
-# Now there should be 3 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-cp $DB $DB.backup
-tempfiles $DB.backup
-# Rerun same tests for the prog2 binary
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
-cmp $filename F/prog2
-cat vlog
-grep -q Progress vlog
-grep -q Downloaded.from vlog
-tempfiles vlog
-filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
-cmp $filename F/prog2
-cat vlog2
-grep -q 'Downloading.*http' vlog2
-tempfiles vlog2
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
-cmp $filename ${PWD}/prog2.c
-
-cp -rvp ${abs_srcdir}/debuginfod-rpms R
-if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
- rm -vrf R/debuginfod-rpms/fedora31
-fi
-
-cp -rvp ${abs_srcdir}/debuginfod-tars Z
-kill -USR1 $PID1
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# All rpms need to be in the index, except the dummy permission-000 one
-rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
-txz=$(find Z -name \*tar.xz | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
-tb2=$(find Z -name \*tar.bz2 | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
-
-kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 5
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# Expect all source files found in the rpms (they are all called hello.c :)
-# We will need to extract all rpms (in their own directory) and could all
-# sources referenced in the .debug files.
-mkdir extracted
-cd extracted
-subdir=0;
-newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
-for i in $newrpms; do
- subdir=$[$subdir+1];
- mkdir $subdir;
- cd $subdir;
- ls -lah ../$i
- rpm2cpio ../$i | cpio -ivd;
- cd ..;
-done
-sourcefiles=$(find -name \*\\.debug \
- | env LD_LIBRARY_PATH=$ldpath xargs \
- ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
- | grep mtime: | wc --lines)
-cd ..
-rm -rf extracted
-
-wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
-
-########################################################################
-# PR27983 ensure no duplicate urls are used in when querying servers for files
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http:127.0.0.1:7999" \
- LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -v executable $BUILDID2 > vlog4 2>&1 || true
-tempfiles vlog4
-if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog4 ) -ne 2 ]; then
- echo "Duplicated servers remain";
- err
-fi
-########################################################################
-# Run a bank of queries against the debuginfod-rpms / debuginfod-debs test cases
-
-archive_test() {
- __BUILDID=$1
- __SOURCEPATH=$2
- __SOURCESHA1=$3
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- # check that timestamps are plausible - older than the near-present (tmpdir mtime)
- test $filename -ot `pwd`
-
- # run again to assure that fdcache is being enjoyed
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- if test "x$__SOURCEPATH" != "x"; then
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
- hash=`cat $filename | sha1sum | awk '{print $1}'`
- test $__SOURCESHA1 = $hash
- test $filename -ot `pwd`
- fi
-}
-
-
-# common source file sha1
-SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
-# fedora31
-if [ $zstd = true ]; then
- # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
- # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
- archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
- archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
-fi
-# fedora30
-archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-# rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-# rhel6
-archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
-# arch
-archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
-
-RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir, for a later test
-
-
-########################################################################
-
-# Drop some of the artifacts, run a groom cycle; confirm that
-# debuginfod has forgotten them, but remembers others
-
-rm -r R/debuginfod-rpms/rhel6/*
-kill -USR2 $PID1 # groom cycle
-# 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
-wait_ready $PORT1 'thread_work_total{role="groom"}' 2
-# Expect 4 rpms containing 2 buildids to be deleted by the groom
-wait_ready $PORT1 'groomed_total{decision="stale"}' 4
-
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# this is one of the buildids from the groom-deleted rpms
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
-# but this one was not deleted so should be still around
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2
-
-########################################################################
-
-# PR26810: Now rename some files in the R directory, then rescan, so
-# there are two copies of the same buildid in the index, one for the
-# no-longer-existing file name, and one under the new name.
-
-# run a groom cycle to force server to drop its fdcache
-kill -USR2 $PID1 # groom cycle
-wait_ready $PORT1 'thread_work_total{role="groom"}' 3
-# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
-mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 6
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 7
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 8
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# retest rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-
-egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
-
-########################################################################
-## PR25978
-# Ensure that the fdcache options are working.
-grep "prefetch fds" vlog$PORT1
-grep "prefetch mbs" vlog$PORT1
-grep "fdcache fds" vlog$PORT1
-grep "fdcache mbs" vlog$PORT1
-# search the vlog to find what metric counts should be and check the correct metrics
-# were incrimented
-wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
-########################################################################
-
-# Federation mode
-
-# find another unused port
-while true; do
- PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT2" || break
-done
-
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
-mkdir -p $DEBUGINFOD_CACHE_PATH
-# NB: inherits the DEBUGINFOD_URLS to the first server
-# NB: run in -L symlink-following mode for the L subdir
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d ${DB}_2 -p $PORT2 -L L D > vlog$PORT2 2>&1 &
-PID2=$!
-tempfiles vlog$PORT2
-errfiles vlog$PORT2
-tempfiles ${DB}_2
-wait_ready $PORT2 'ready' 1
-wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
-wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
-wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
-
-# have clients contact the new server
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-
-if type bsdtar 2>/dev/null; then
- # copy in the deb files
- cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
- kill -USR1 $PID2
- wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
- wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
- wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
- # All debs need to be in the index
- debs=$(find D -name \*.deb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
- ddebs=$(find D -name \*.ddeb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
-
- # ubuntu
- archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
-fi
-
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# send a request to stress XFF and User-Agent federation relay;
-# we'll grep for the two patterns in vlog$PORT1
-curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
-
-grep UA:TESTCURL vlog$PORT1
-grep XFF:TESTXFF vlog$PORT1
-
-
-# confirm that first server can't resolve symlinked info in L/ but second can
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
-file L/foo
-file -L L/foo
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test again with scheme free url
-export DEBUGINFOD_URLS=127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test parallel queries in client
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
-mkdir -p $DEBUGINFOD_CACHE_PATH
-export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-########################################################################
-
-# Fetch some metrics
-curl -s http://127.0.0.1:$PORT1/badapi
-curl -s http://127.0.0.1:$PORT1/metrics
-curl -s http://127.0.0.1:$PORT2/metrics
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*fdcache'
-curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
-
-# And generate a few errors into the second debuginfod's logs, for analysis just below
-curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
-# NB: this error is used to seed the 404 failure for the survive-404 tests
-
-# Confirm bad artifact types are rejected without leaving trace
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
-(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
-
-# DISABLE VALGRIND checking because valgrind might use debuginfod client
-# requests itself, causing confusion about who put what in the cache.
-# It stays disabled till the end of this test.
-unset VALGRIND_CMD
-
-# Confirm that reused curl connections survive 404 errors.
-# The rm's force an uncached fetch
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# Confirm that some debuginfod client pools are being used
-curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
-
-# Trigger a flood of requests against the same archive content file.
-# Use a file that hasn't been previously extracted in to make it
-# likely that even this test debuginfod will experience concurrency
-# and impose some "after-you" delays.
-(for i in `seq 100`; do
- curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
- done;
- wait)
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
-# If we could guarantee some minimum number of seconds of CPU time, we
-# could assert that the after_you metrics show some nonzero amount of
-# waiting. A few hundred ms is typical on this developer's workstation.
-
-
-########################################################################
-# Corrupt the sqlite database and get debuginfod to trip across its errors
-
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
-ls -al $DB
-dd if=/dev/zero of=$DB bs=1 count=1
-ls -al $DB
-# trigger some random activity that's Sure to get sqlite3 upset
-kill -USR1 $PID1
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 9
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -USR2 $PID1
-wait_ready $PORT1 'thread_work_total{role="groom"}' 4
-curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
-
-########################################################################
-
-# Run the tests again without the servers running. The target file should
-# be found in the cache.
-
-kill -INT $PID1 $PID2
-wait $PID1 $PID2
-PID1=0
-PID2=0
-tempfiles .debuginfod_*
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-# check out the debuginfod logs for the new style status lines
-# cat vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
-
-########################################################################
-
-# Add some files to the cache that do not fit its naming format.
-# They should survive cache cleaning.
-mkdir $DEBUGINFOD_CACHE_PATH/malformed
-touch $DEBUGINFOD_CACHE_PATH/malformed0
-touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
-
-# A valid format for an empty buildid subdirectory
-mkdir $DEBUGINFOD_CACHE_PATH/00000000
-touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
-
-# Trigger a cache clean and run the tests again. The clients should be unable to
-# find the target.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
-echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID2 && false || true
-
-if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
- || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
- echo "unrelated files did not survive cache cleaning"
- err
-fi
-
-if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
- echo "failed to rmdir old cache dir"
- err
-fi
-
-# Test debuginfod without a path list; reuse $PORT1
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
-PID3=$!
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -int $PID3
-wait $PID3
-PID3=0
-
-########################################################################
-# Test fetching a file using file:// . No debuginfod server needs to be run for
-# this test.
-local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
-mkdir -p ${local_dir}
-echo "int main() { return 0; }" > ${local_dir}/main.c
-
-# first test that is doesn't work, when no DEBUGINFOD_URLS is set
-DEBUGINFOD_URLS=""
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
-
-# Now test is with proper DEBUGINFOD_URLS
-DEBUGINFOD_URLS="file://${PWD}/mocktree/"
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
-cmp $filename ${local_dir}/main.c
-
-########################################################################
-## PR27711
-# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
-while true; do
- PORT3=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT3" || break
-done
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT3/" ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT3 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d $DB.backup > vlog$PORT3 2>&1 &
-PID4=$!
-wait_ready $PORT3 'ready' 1
-tempfiles vlog$PORT3
-errfiles vlog$PORT3
-
-kill -USR2 $PID4
-wait_ready $PORT3 'thread_work_total{role="groom"}' 1
-wait_ready $PORT3 'groom{statistic="archive d/e"}' 0
-wait_ready $PORT3 'groom{statistic="archive sdef"}' 0
-wait_ready $PORT3 'groom{statistic="archive sref"}' 0
-wait_ready $PORT3 'groom{statistic="buildids"}' 0
-wait_ready $PORT3 'groom{statistic="file d/e"}' 0
-wait_ready $PORT3 'groom{statistic="file s"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (#)"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (mb)"}' 0
-
-kill $PID4
-wait $PID4
-PID4=0
-
-########################################################################
-# set up tests for retrying failed queries.
-retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
- | grep -c 'Retry failed query'`
-if [ $retry_attempts -ne 10 ]; then
- echo "retry mechanism failed."
- exit 1;
- fi
-
-########################################################################
-
-# Test when debuginfod hitting X-Forwarded-For hops limit.
-# This test will start two servers (as a loop) with two different hop limits.
-
-while true; do
- PORT4=`expr '(' $RANDOM % 1000 ')' + 9000`
- PORT5=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep -e ":$PORT4" -e ":$PORT5"|| break
-done
-
-# Make sure the vlogs are cleaned up after the test
-# and that they are printed on error.
-tempfiles vlog$PORT4 vlog$PORT5
-errfiles vlog$PORT4 vlog$PORT5
-
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
-PID5=$!
-
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
-PID6=$!
-
-wait_ready $PORT4 'ready' 1
-wait_ready $PORT5 'ready' 1
-
-export DEBUGINFOD_URLS="http://127.0.0.1:$PORT4/"
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-
-# Use a different buildid to avoid using same cache.
-export DEBUGINFOD_URLS="http://127.0.0.1:$PORT5/"
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 11234567 || true
-
-grep "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT4
-grep -v "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT5 | grep "not found" vlog$PORT5
-
-kill $PID5 $PID6
-wait $PID5 $PID6
-
-PID5=0
-PID6=0
-
-exit 0
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
new file mode 100755
index 00000000..b420a45b
--- /dev/null
+++ b/tests/run-debuginfod-malformed.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# find an unused port number
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Add some files to the cache that do not fit its naming format.
+# They should survive cache cleaning.
+mkdir $DEBUGINFOD_CACHE_PATH/malformed
+touch $DEBUGINFOD_CACHE_PATH/malformed0
+touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
+
+# A valid format for an empty buildid subdirectory
+mkdir $DEBUGINFOD_CACHE_PATH/00000000
+touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
+
+# Trigger a cache clean and run the tests again. The clients should be unable to
+# find the target.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
+echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+
+if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
+ || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
+ echo "unrelated files did not survive cache cleaning"
+ err
+fi
+
+if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
+ echo "failed to rmdir old cache dir"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-no-urls.sh b/tests/run-debuginfod-no-urls.sh
new file mode 100755
index 00000000..53c29f4c
--- /dev/null
+++ b/tests/run-debuginfod-no-urls.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# Test debuginfod without a path list
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
+PID1=$!
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -int $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
new file mode 100755
index 00000000..918c6801
--- /dev/null
+++ b/tests/run-debuginfod-query-retry.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# set up tests for retrying failed queries.
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
+ | grep -c 'Retry failed query'`
+if [ $retry_attempts -ne 10 ]; then
+ echo "retry mechanism failed."
+ exit 1;
+fi
+
+exit 0;
diff --git a/tests/run-debuginfod-regex.sh b/tests/run-debuginfod-regex.sh
new file mode 100755
index 00000000..e36d8f5f
--- /dev/null
+++ b/tests/run-debuginfod-regex.sh
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+get_ports
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+
+mv prog F
+mv prog.debug F
+tempfiles F/prog.debug F/prog
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+cp ${DB} ${DB}.backup
+tempfiles ${DB}.backup
+
+kill $PID1
+wait $PID1
+PID1=0
+
+#######################################################################
+## PR27711
+# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
+#
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -p $PORT2 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d ${DB}.backup > vlog$PORT2 2>&1 &
+
+#reuse PID1
+PID1=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+# Server must become ready
+wait_ready $PORT2 'ready' 1
+
+kill -USR2 $PID1
+wait_ready $PORT2 'thread_work_total{role="groom"}' 1
+wait_ready $PORT2 'groom{statistic="archive d/e"}' 0
+wait_ready $PORT2 'groom{statistic="archive sdef"}' 0
+wait_ready $PORT2 'groom{statistic="archive sref"}' 0
+wait_ready $PORT2 'groom{statistic="buildids"}' 0
+wait_ready $PORT2 'groom{statistic="file d/e"}' 0
+wait_ready $PORT2 'groom{statistic="file s"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (#)"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (mb)"}' 0
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
+
diff --git a/tests/run-debuginfod-sizetime.sh b/tests/run-debuginfod-sizetime.sh
new file mode 100755
index 00000000..edf4cce5
--- /dev/null
+++ b/tests/run-debuginfod-sizetime.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+set -x
+unset VALGRIND_CMD
+
+get_ports
+
+echo "int main() { return 0; }" > ${PWD}/prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+## PR27892
+# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
+# code
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v executable ${PWD}/prog 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+errfiles find-vlog$PORT1
+echo "Checking maxsize"
+grep "using max size 1B" find-vlog$PORT1
+echo "Checking maxsize"
+grep 'serving file '$(realpath ${PWD})'/prog' vlog$PORT1
+echo "Checking maxsize"
+grep 'File too large' vlog$PORT1
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxsize check"
+ err
+fi
+# Ensure no file is downloaded for longer than DEBUGINFOD_MAXTIME
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXTIME=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo ${PWD}/prog.debug 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+grep 'using max time' find-vlog$PORT1
+# Ensure p+r%o\$g.debug is NOT cached
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxtime check"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
new file mode 100755
index 00000000..0f2f2d53
--- /dev/null
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether the cache default locations are correct
+mkdir tmphome
+
+# $HOME/.cache should be created.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+
+# $HOME/.cache should be found.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+# $XDG_CACHE_HOME should take priority over $HOME.cache.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpxdg/"
+ err
+fi
+
+# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
+# priority over $HOME/.cache, $XDG_CACHE_HOME.
+cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
+# ||true is for tolerating errors, such a valgrind or something else
+# leaving 000-perm files in there
+
+# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
+mkdir tmphome/.debuginfod_client_cache/deadbeef
+echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
+cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+
+# $DEBUGINFO_CACHE_PATH should take priority over all else.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpcache/"
+ err
+fi
+rm -rf ${PWD}/tmphome/ ${PWD}/tmpxdg ${PWD}/tmpcache
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
new file mode 100755
index 00000000..edec9357
--- /dev/null
+++ b/tests/run-debuginfod-writable.sh
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+#######################################################################
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/p+r%o\$g.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-x-forwarded-for.sh b/tests/run-debuginfod-x-forwarded-for.sh
new file mode 100755
index 00000000..d379e410
--- /dev/null
+++ b/tests/run-debuginfod-x-forwarded-for.sh
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+get_ports
+
+# Test when debuginfod hitting X-Forwarded-For hops limit.
+# This test will start two servers (as a loop) with two different hop limits.
+
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT2 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT1 > vlog$PORT1 2>&1 &
+PID1=$!
+
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT2 > vlog$PORT2 2>&1 &
+PID2=$!
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+
+# Use a different buildid to avoid using same cache.
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT2/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 11234567 || true
+
+grep "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT1
+grep -v "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT2 | grep "not found" vlog$PORT2
+
+kill $PID1 $PID2
+wait $PID1 $PID2
+
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/test-subr.sh b/tests/test-subr.sh
index 411e5f28..41e95e31 100644
--- a/tests/test-subr.sh
+++ b/tests/test-subr.sh
@@ -27,6 +27,13 @@ test_dir="test-$$"
mkdir -p "$test_dir"
cd "$test_dir"
+type curl 2>/dev/null || (echo "need curl"; exit 77)
+type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
+type cpio 2>/dev/null || (echo "need cpio"; exit 77)
+type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
+bsdtar --version | grep -q zstd && zstd=true || zstd=false
+echo "zstd=$zstd bsdtar=`bsdtar --version`"
+
#LC_ALL=C
#export LC_ALL
@@ -35,7 +42,11 @@ remove_files=
# Tests that trap EXIT (0) themselves should call this explicitly.
exit_cleanup()
{
- rm -f $remove_files; cd ..; rmdir $test_dir
+ rm -rf ${PWD}/.client_cache*
+ rm -f $remove_files;
+ ls -a ${PWD}
+ cd ..;
+ rmdir $test_dir
}
trap exit_cleanup 0
@@ -212,3 +223,114 @@ testrun_on_self_quiet()
# Only exit if something failed
if test $exit_status != 0; then exit $exit_status; fi
}
+
+cleanup()
+{
+ if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
+ if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
+ rm -rf ${PWD}/.client_cache F R D L Z ${PWD}/foobar ${PWD}/mocktree
+ exit_cleanup
+}
+trap cleanup 0 1 2 3 5 9 15
+errfiles_list=
+err() {
+ echo ERROR REPORTS
+ for ports in $PORT1 $PORT2
+ do
+ echo ERROR REPORT $port metrics
+ curl -s http://127.0.0.1:$port/metrics
+ echo
+ done
+ for x in $errfiles_list
+ do
+ echo ERROR REPORT "$x"
+ cat $x
+ echo
+ done
+ false # trigger set -e
+}
+trap err ERR
+
+errfiles() {
+ errfiles_list="$errfiles_list $*"
+}
+
+wait_ready()
+{
+ port=$1;
+ what=$2;
+ value=$3;
+ timeout=20;
+
+ echo "Wait $timeout seconds on $port for metric $what to change to $value"
+ while [ $timeout -gt 0 ]; do
+ mvalue="$(curl -s http://127.0.0.1:$port/metrics \
+ | grep "$what" | awk '{print $NF}')"
+ if [ -z "$mvalue" ]; then mvalue=0; fi
+ echo "metric $what: $mvalue"
+ if [ "$mvalue" -eq "$value" ]; then
+ break;
+ fi
+ sleep 0.5;
+ ((timeout--));
+ done;
+
+ if [ $timeout -eq 0 ]; then
+ echo "metric $what never changed to $value on port $port"
+ err
+ fi
+}
+
+archive_test() {
+ __BUILDID=$1
+ __SOURCEPATH=$2
+ __SOURCESHA1=$3
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ # check that timestamps are plausible - older than the near-present (tmpdir mtime)
+ test $filename -ot `pwd`
+
+ # run again to assure that fdcache is being enjoyed
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ if test "x$__SOURCEPATH" != "x"; then
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
+ hash=`cat $filename | sha1sum | awk '{print $1}'`
+ test $__SOURCESHA1 = $hash
+ test $filename -ot `pwd`
+ fi
+}
+
+get_ports() {
+ while true; do
+ PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$PORT1" || break
+ done
+# Some tests will use two servers, so assign the second var
+ while true; do
+ PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
+ ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
+ done
+
+}
+
+VERBOSE=-vvv
+# We gather the LD_LIBRARY_PATH with this cunning trick:
+ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
+PORT1=0
+PORT2=0
+PID1=0
+PID2=0
--
2.18.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests
2021-09-02 13:42 ` Mark Wielaard
@ 2021-09-02 18:59 ` Noah Sanci
2021-09-03 8:45 ` Mark Wielaard
0 siblings, 1 reply; 6+ messages in thread
From: Noah Sanci @ 2021-09-02 18:59 UTC (permalink / raw)
To: Mark Wielaard; +Cc: elfutils-devel
[-- Attachment #1: Type: text/plain, Size: 745 bytes --]
Hello,
As for the rpm/Makefile.am stuff, I think that was a temporary fix for
an odd error
I got while testing. I just forgot to remove it and see if it would
break again. It has
been removed and makes now.
As for the 'base' idea, I gave it some thought and believe this is the
best solution
and I have implemented it.
> And then add a unique base to each get_port call (8000, 8100, 8200,
> ...). We would to be careful to never assign the same unique range to
> each test though. Which requires the programmer to check all other
> tests. But maybe that isn't such a big deal? git grep ^get_port tests
> would give you all the currently used port ranges...
>
> Ideas?
>
Agreed, implemented.
The patch also includes x-forwarded
-Noah Sanci
[-- Attachment #2: 0001-debuginfod-Fracture-tests-run-debuginfod-find.sh-int.patch --]
[-- Type: text/x-patch, Size: 106190 bytes --]
From 0583d98836cd7492f88579e5aa154699069e97b8 Mon Sep 17 00:00:00 2001
From: Noah Sanci <nsanci@redhat.com>
Date: Tue, 10 Aug 2021 11:21:35 -0400
Subject: [PATCH] debuginfod: Fracture tests/run-debuginfod-find.sh into
specific tests
tests/run-debuginfod-find.sh was a massive test script with many broadly
varying tests. This caused the test script to fail when any number of
things went wrong and because of its intertwined nature, detecting the
source of a failure could be difficult. The size of the test script
also meant many unrelated tests were run making the testing process
unnecessarily lengthy.
This patch fractures tests/run-debuginfod-find.sh into smaller, more
manageable individual test script files. This ensures that when failure
occurs, a programmer can easily determine where their patch went
wrong. It also allows programmers to specify exactly which tests to
run, making testing more efficient. Redundancies are also reduced by
placing code in tests/test-subr.sh.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
---
tests/ChangeLog | 29 +++
tests/Makefile.am | 46 ++++-
tests/run-debuginfod-000-permission.sh | 85 ++++++++
tests/run-debuginfod-X-FORWARDED.sh | 60 ++++++
tests/run-debuginfod-archive-groom.sh | 153 +++++++++++++++
tests/run-debuginfod-archive-rename.sh | 95 +++++++++
tests/run-debuginfod-archive-test.sh | 83 ++++++++
tests/run-debuginfod-artifact-running.sh | 122 ++++++++++++
tests/run-debuginfod-dlopen.sh | 84 ++++++++
tests/run-debuginfod-duplicate-urls.sh | 51 +++++
| 101 ++++++++++
tests/run-debuginfod-fd-prefetch-caches.sh | 57 ++++++
tests/run-debuginfod-federation-link.sh | 161 ++++++++++++++++
tests/run-debuginfod-federation-metrics.sh | 213 +++++++++++++++++++++
tests/run-debuginfod-federation-sqlite.sh | 201 +++++++++++++++++++
tests/run-debuginfod-file.sh | 39 ++++
tests/run-debuginfod-malformed.sh | 110 +++++++++++
tests/run-debuginfod-no-urls.sh | 41 ++++
tests/run-debuginfod-query-retry.sh | 34 ++++
tests/run-debuginfod-regex.sh | 97 ++++++++++
| 118 ++++++++++++
tests/run-debuginfod-sizetime.sh | 77 ++++++++
tests/run-debuginfod-tmp-home.sh | 124 ++++++++++++
tests/run-debuginfod-writable.sh | 84 ++++++++
tests/run-debuginfod-x-forwarded-for.sh | 62 ++++++
tests/test-subr.sh | 124 +++++++++++-
26 files changed, 2448 insertions(+), 3 deletions(-)
create mode 100755 tests/run-debuginfod-000-permission.sh
create mode 100644 tests/run-debuginfod-X-FORWARDED.sh
create mode 100755 tests/run-debuginfod-archive-groom.sh
create mode 100755 tests/run-debuginfod-archive-rename.sh
create mode 100755 tests/run-debuginfod-archive-test.sh
create mode 100755 tests/run-debuginfod-artifact-running.sh
create mode 100755 tests/run-debuginfod-dlopen.sh
create mode 100755 tests/run-debuginfod-duplicate-urls.sh
create mode 100755 tests/run-debuginfod-extraction.sh
create mode 100755 tests/run-debuginfod-fd-prefetch-caches.sh
create mode 100755 tests/run-debuginfod-federation-link.sh
create mode 100755 tests/run-debuginfod-federation-metrics.sh
create mode 100755 tests/run-debuginfod-federation-sqlite.sh
create mode 100755 tests/run-debuginfod-file.sh
create mode 100755 tests/run-debuginfod-malformed.sh
create mode 100755 tests/run-debuginfod-no-urls.sh
create mode 100755 tests/run-debuginfod-query-retry.sh
create mode 100755 tests/run-debuginfod-regex.sh
create mode 100755 tests/run-debuginfod-response-headers.sh
create mode 100755 tests/run-debuginfod-sizetime.sh
create mode 100755 tests/run-debuginfod-tmp-home.sh
create mode 100755 tests/run-debuginfod-writable.sh
create mode 100755 tests/run-debuginfod-x-forwarded-for.sh
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 3fa7c2e4..db5b25d1 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -12,6 +12,35 @@
* run-debuginfod-find.sh: Add test for X-Forwarded-For hops limit.
+2021-08-20 Noah Sanci <nsanci@redhat.com>
+
+ * debuginfod-find.sh: Separated file into
+ run-debuginfod-000-permission.sh,
+ run-debuginfod-archive-groom.sh,
+ run-debuginfod-archive-rename.sh,
+ run-debuginfod-archive-test.sh,
+ run-debuginfod-artifact-running.sh,
+ run-debuginfod-dlopen.sh,
+ run-debuginfod-duplicate-urls.sh,
+ run-debuginfod-extraction.sh,
+ run-debuginfod-fd-prefetch-caches.sh,
+ run-debuginfod-federation-link.sh,
+ run-debuginfod-federation-metrics.sh,
+ run-debuginfod-federation-sqlite.sh,
+ run-debuginfod-file.sh,
+ run-debuginfod-malformed.sh,
+ run-debuginfod-no-urls.sh,
+ run-debuginfod-query-retry.sh,
+ run-debuginfod-regex.sh,
+ run-debuginfod-sizetime.sh,
+ run-debuginfod-tmp-home.sh,
+ run-debuginfod-x-forwarded.sh
+ run-debuginfod-writable.sh, and
+ run-debuginfod-response-headers.sh. All files use the $base variable
+ to find ports.
+ * tests/Makefile.am: Added the above new files to the test suite
+ * tests/test-subr.sh: Added some general functions for above tests
+
2021-08-04 Mark Wielaard <mark@klomp.org>
PR28190
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 429649f4..a9adeb4f 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -211,7 +211,28 @@ if DEBUGINFOD
check_PROGRAMS += debuginfod_build_id_find
# With the dummy delegation doesn't work
if !DUMMY_LIBDEBUGINFOD
-TESTS += run-debuginfod-find.sh
+TESTS += run-debuginfod-dlopen.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh \
+ run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-x-forwarded-for.sh \
+ run-debuginfod-response-headers.sh
endif
endif
@@ -474,7 +495,28 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-elfclassify.sh run-elfclassify-self.sh \
run-disasm-riscv64.sh \
testfile-riscv64-dis1.o.bz2 testfile-riscv64-dis1.expect.bz2 \
- run-debuginfod-find.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-response-headers.sh \
+ run-debuginfod-x-forwarded-for.sh \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-dlopen.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh \
debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
diff --git a/tests/run-debuginfod-000-permission.sh b/tests/run-debuginfod-000-permission.sh
new file mode 100755
index 00000000..5bbb1df0
--- /dev/null
+++ b/tests/run-debuginfod-000-permission.sh
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8000
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+# PR25628
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# The query is designed to fail, while the 000-permission file should be created.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
+ err
+fi
+
+if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
+ err
+fi
+
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" != "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} has changed."
+ err
+fi
+
+# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
+sleep 1
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" == "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-X-FORWARDED.sh b/tests/run-debuginfod-X-FORWARDED.sh
new file mode 100644
index 00000000..a67c5646
--- /dev/null
+++ b/tests/run-debuginfod-X-FORWARDED.sh
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=10000
+get_ports
+
+# Make sure the vlogs are cleaned up after the test
+# and that they are printed on error.
+tempfiles vlog$PORT1 vlog$PORT2
+errfiles vlog$PORT1 vlog$PORT2
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT1 > vlog$PORT1 2>&1 &
+PID1=$!
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT2 > vlog$PORT2 2>&1 &
+PID2=$!
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+
+# Use a different buildid to avoid using same cache.
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT2/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 11234567 || true
+
+grep "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT1
+grep -v "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT2 | grep "not found" vlog$PORT2
+
+kill $PID1 $PID2
+wait $PID1 $PID2
+
+PID1=0
+PID2=0
+
+exit 0
+
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
new file mode 100755
index 00000000..c84e9d74
--- /dev/null
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -0,0 +1,153 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8100
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Be patient when run on a busy machine things might take a bit.
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Build a non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/F/prog.c
+gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a ${PWD}/F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+tempfiles ${PWD}/F/prog ${PWD}/F/prog.c
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+tempfiles vlog3 $DB
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
+
+# Drop some of the artifacts, run a groom cycle; confirm that
+# debuginfod has forgotten them, but remembers others
+rm -r R/debuginfod-rpms/rhel6/*
+kill -USR2 $PID1 # groom cycle
+## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# Expect 4 rpms containing 2 buildids to be deleted by the groom
+wait_ready $PORT1 'groomed_total{decision="stale"}' 4
+
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# this is one of the buildids from the groom-deleted rpms
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
+
+# but this one was not deleted so should be still around
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID || true
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
new file mode 100755
index 00000000..2f6ee2dd
--- /dev/null
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8200
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DEBUGINFOD_CACHE_PATH $DB
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+
+########################################################################
+## PR26810: Now rename some files in the R directory, then rescan, so
+# there are two copies of the same buildid in the index, one for the
+# no-longer-existing file name, and one under the new name.
+
+# run a groom cycle to force server to drop its fdcache
+kill -USR2 $PID1 # groom cycle
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
+# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
+mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# retest rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+
+egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0;
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
new file mode 100755
index 00000000..cffe95ba
--- /dev/null
+++ b/tests/run-debuginfod-archive-test.sh
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# find an unused port number
+mkdir R
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8300
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -t0 -g0 -v R > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+# arch
+#archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
new file mode 100755
index 00000000..89dee994
--- /dev/null
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8400
+get_ports
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+echo 'int main(int argc, char * argv){ return 0; }' > ${PWD}/prog.c
+gcc -Wl,--build-id -g -o prog ${PWD}/prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d $DB -F -p $PORT1 -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+mv prog F
+mv prog.debug F
+tempfiles prog/F
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+kill -USR1 $PID1
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Add artifacts to the search paths and test whether debuginfod finds them while already running.
+# Build another, non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/prog2.c
+tempfiles prog2.c
+gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
+#testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
+mv prog2 F
+#mv prog2.debug F
+tempfiles F/prog2 F/prog2.debug
+
+kill -USR1 $PID1
+# Now there should be 3 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/prog.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/prog`
+cmp $filename F/prog
+
+# raw source filename
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../prog.c`
+cmp $filename ${PWD}/prog.c
+
+# and also the canonicalized one
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/prog.c`
+cmp $filename ${PWD}/prog.c
+
+# Rerun same tests for the prog2 binary
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
+cmp $filename F/prog2
+grep -q Progress vlog
+grep -q Downloaded.from vlog
+tempfiles vlog
+filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
+cmp $filename F/prog2
+grep -q 'Downloading.*http' vlog2
+tempfiles vlog2
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
+cmp $filename ${PWD}/prog2.c
+
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
new file mode 100755
index 00000000..803f4f44
--- /dev/null
+++ b/tests/run-debuginfod-dlopen.sh
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8500
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a'
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether elfutils, via the debuginfod client library dlopen hooks,
+# is able to fetch debuginfo from the local debuginfod.
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-duplicate-urls.sh b/tests/run-debuginfod-duplicate-urls.sh
new file mode 100755
index 00000000..3c5bdfc6
--- /dev/null
+++ b/tests/run-debuginfod-duplicate-urls.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8600
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+########################################################################
+## PR27983
+# Ensure no duplicate urls are used in when querying servers for files
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:7999" \
+ LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -vvv executable 0 > vlog1 2>&1 || true
+tempfiles vlog1
+cat vlog1
+if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog1 ) -ne 2 ]; then
+ echo "Duplicate servers remain";
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
--git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
new file mode 100755
index 00000000..499d7e00
--- /dev/null
+++ b/tests/run-debuginfod-extraction.sh
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir R Z
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8700
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
+tb2=$(find Z -name \*tar.bz2 | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh
new file mode 100755
index 00000000..d3e77dc2
--- /dev/null
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+FDCACHE_FDS=100
+FDCACHE_MBS=100
+PREFETCH_FDS=100
+PREFETCH_MBS=100
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8800
+get_ports
+
+echo $PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 \
+ --fdcache-mbs=$FDCACHE_MDS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS \
+ --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -v -F F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+grep 'fdcache fds ' vlog$PORT1 #$FDCACHE_FDS
+grep 'fdcache mbs ' vlog$PORT1 #$FDCACHE_MBS
+grep 'prefetch fds ' vlog$PORT1 #$PREFETCH_FDS
+grep 'prefetch mbs ' vlog$PORT1 #$PREFETCH_MBS
+# search the vlog to find what metric counts should be and check the correct metrics
+# were incrimented
+wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
new file mode 100755
index 00000000..9f0bd6fa
--- /dev/null
+++ b/tests/run-debuginfod-federation-link.sh
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8900
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+
+kill -USR1 $PID2
+# Wait till both files are in the index.
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/run-debuginfod-federation-metrics.sh b/tests/run-debuginfod-federation-metrics.sh
new file mode 100755
index 00000000..6ad927c9
--- /dev/null
+++ b/tests/run-debuginfod-federation-metrics.sh
@@ -0,0 +1,213 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
+tempfiles $DB
+# Clean old dirictories
+mkdir D L F
+
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9000
+get_ports
+
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+########################################################################
+# Fetch some metrics
+curl -s http://127.0.0.1:$PORT1/badapi
+curl -s http://127.0.0.1:$PORT1/metrics
+curl -s http://127.0.0.1:$PORT2/metrics
+curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
+curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# Confirm that some debuginfod client pools are being used
+curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
+
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+exit 0
+
diff --git a/tests/run-debuginfod-federation-sqlite.sh b/tests/run-debuginfod-federation-sqlite.sh
new file mode 100755
index 00000000..40d52f6e
--- /dev/null
+++ b/tests/run-debuginfod-federation-sqlite.sh
@@ -0,0 +1,201 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9100
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D F > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+########################################################################
+# Corrupt the sqlite database and get debuginfod to trip across its errors
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
+dd if=/dev/zero of=$DB bs=1 count=1
+
+# trigger some random activity that's Sure to get sqlite3 upset
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -USR2 $PID1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
+# Run the tests again without the servers running. The target file should
+# be found in the cache.
+
+kill -INT $PID1 $PID2
+wait $PID1 $PID2
+PID1=0
+PID2=0
+tempfiles .debuginfod_*
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# check out the debuginfod logs for the new style status lines
+cat vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
+
+exit 0
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
new file mode 100755
index 00000000..3187c92b
--- /dev/null
+++ b/tests/run-debuginfod-file.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Test fetching a file using file:// . No debuginfod server needs to be run for
+# this test.
+local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
+mkdir -p ${local_dir}
+echo "int main() { return 0; }" > ${local_dir}/main.c
+# first test that is doesn't work, when no DEBUGINFOD_URLS is set
+DEBUGINFOD_URLS=""
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
+
+# Now test is with proper DEBUGINFOD_URLS
+DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
+cmp $filename ${local_dir}/main.c
+
+exit 0
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
new file mode 100755
index 00000000..468b4e1e
--- /dev/null
+++ b/tests/run-debuginfod-malformed.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9200
+# find an unused port number
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Add some files to the cache that do not fit its naming format.
+# They should survive cache cleaning.
+mkdir $DEBUGINFOD_CACHE_PATH/malformed
+touch $DEBUGINFOD_CACHE_PATH/malformed0
+touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
+
+# A valid format for an empty buildid subdirectory
+mkdir $DEBUGINFOD_CACHE_PATH/00000000
+touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
+
+# Trigger a cache clean and run the tests again. The clients should be unable to
+# find the target.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
+echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+
+if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
+ || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
+ echo "unrelated files did not survive cache cleaning"
+ err
+fi
+
+if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
+ echo "failed to rmdir old cache dir"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-no-urls.sh b/tests/run-debuginfod-no-urls.sh
new file mode 100755
index 00000000..f08aee71
--- /dev/null
+++ b/tests/run-debuginfod-no-urls.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# Test debuginfod without a path list
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9300
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
+PID1=$!
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -int $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
new file mode 100755
index 00000000..918c6801
--- /dev/null
+++ b/tests/run-debuginfod-query-retry.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# set up tests for retrying failed queries.
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
+ | grep -c 'Retry failed query'`
+if [ $retry_attempts -ne 10 ]; then
+ echo "retry mechanism failed."
+ exit 1;
+fi
+
+exit 0;
diff --git a/tests/run-debuginfod-regex.sh b/tests/run-debuginfod-regex.sh
new file mode 100755
index 00000000..113a582c
--- /dev/null
+++ b/tests/run-debuginfod-regex.sh
@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9400
+get_ports
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+
+mv prog F
+mv prog.debug F
+tempfiles F/prog.debug F/prog
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+cp ${DB} ${DB}.backup
+tempfiles ${DB}.backup
+
+kill $PID1
+wait $PID1
+PID1=0
+
+#######################################################################
+## PR27711
+# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
+#
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -p $PORT2 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d ${DB}.backup > vlog$PORT2 2>&1 &
+
+#reuse PID1
+PID1=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+# Server must become ready
+wait_ready $PORT2 'ready' 1
+
+kill -USR2 $PID1
+wait_ready $PORT2 'thread_work_total{role="groom"}' 1
+wait_ready $PORT2 'groom{statistic="archive d/e"}' 0
+wait_ready $PORT2 'groom{statistic="archive sdef"}' 0
+wait_ready $PORT2 'groom{statistic="archive sref"}' 0
+wait_ready $PORT2 'groom{statistic="buildids"}' 0
+wait_ready $PORT2 'groom{statistic="file d/e"}' 0
+wait_ready $PORT2 'groom{statistic="file s"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (#)"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (mb)"}' 0
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
+
--git a/tests/run-debuginfod-response-headers.sh b/tests/run-debuginfod-response-headers.sh
new file mode 100755
index 00000000..feb179ce
--- /dev/null
+++ b/tests/run-debuginfod-response-headers.sh
@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9500
+get_ports
+mkdir F R
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -d $DB -p $PORT1 -t0 -g0 -v R F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+## PR27277
+# Make a simple request to the debuginfod server and check debuginfod-find's vlog to see if
+# the custom HTTP headers are received.
+rm -rf $DEBUGINFOD_CACHE_PATH
+env DEBUGINFOD_URLS="http://127.0.0.1:"$PORT1 LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find\
+ -vvv executable F/prog > vlog-find$PORT1.1 2>&1
+tempfiles vlog-find$PORT1.1
+grep 'Content-Length: ' vlog-find$PORT1.1
+grep 'Connection: ' vlog-find$PORT1.1
+grep 'Cache-Control: ' vlog-find$PORT1.1
+grep 'X-FILE: ' vlog-find$PORT1.1
+grep 'X-FILE-SIZE: ' vlog-find$PORT1.1
+
+# Check to see if an executable file located in an archive prints the file's description and archive
+env DEBUGINFOD_URLS="http://127.0.0.1:"$PORT1 LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find\
+ -vvv executable c36708a78618d597dee15d0dc989f093ca5f9120 > vlog-find$PORT1.2 2>&1
+tempfiles vlog-find$PORT1.2
+grep 'Content-Length: ' vlog-find$PORT1.2
+grep 'Connection: ' vlog-find$PORT1.2
+grep 'Cache-Control: ' vlog-find$PORT1.2
+grep 'X-FILE: ' vlog-find$PORT1.2
+grep 'X-FILE-SIZE: ' vlog-find$PORT1.2
+grep 'X-ARCHIVE: ' vlog-find$PORT1.2
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-sizetime.sh b/tests/run-debuginfod-sizetime.sh
new file mode 100755
index 00000000..10941289
--- /dev/null
+++ b/tests/run-debuginfod-sizetime.sh
@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9600
+get_ports
+
+echo "int main() { return 0; }" > ${PWD}/prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+## PR27892
+# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
+# code
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v executable ${PWD}/prog 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+errfiles find-vlog$PORT1
+echo "Checking maxsize"
+grep "using max size 1B" find-vlog$PORT1
+echo "Checking maxsize"
+grep 'serving file '$(realpath ${PWD})'/prog' vlog$PORT1
+echo "Checking maxsize"
+grep 'File too large' vlog$PORT1
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxsize check"
+ err
+fi
+# Ensure no file is downloaded for longer than DEBUGINFOD_MAXTIME
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXTIME=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo ${PWD}/prog.debug 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+grep 'using max time' find-vlog$PORT1
+# Ensure p+r%o\$g.debug is NOT cached
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxtime check"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
new file mode 100755
index 00000000..85a20da7
--- /dev/null
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9700
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether the cache default locations are correct
+mkdir tmphome
+
+# $HOME/.cache should be created.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+
+# $HOME/.cache should be found.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+# $XDG_CACHE_HOME should take priority over $HOME.cache.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpxdg/"
+ err
+fi
+
+# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
+# priority over $HOME/.cache, $XDG_CACHE_HOME.
+cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
+# ||true is for tolerating errors, such a valgrind or something else
+# leaving 000-perm files in there
+
+# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
+mkdir tmphome/.debuginfod_client_cache/deadbeef
+echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
+cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+
+# $DEBUGINFO_CACHE_PATH should take priority over all else.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpcache/"
+ err
+fi
+rm -rf ${PWD}/tmphome/ ${PWD}/tmpxdg ${PWD}/tmpcache
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
new file mode 100755
index 00000000..25ee0df7
--- /dev/null
+++ b/tests/run-debuginfod-writable.sh
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9800
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+#######################################################################
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/p+r%o\$g.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-x-forwarded-for.sh b/tests/run-debuginfod-x-forwarded-for.sh
new file mode 100755
index 00000000..3beb782e
--- /dev/null
+++ b/tests/run-debuginfod-x-forwarded-for.sh
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/test-subr.sh # includes set -e
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9900
+get_ports
+
+# Test when debuginfod hitting X-Forwarded-For hops limit.
+# This test will start two servers (as a loop) with two different hop limits.
+
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT2 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT1 > vlog$PORT1 2>&1 &
+PID1=$!
+
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT2 > vlog$PORT2 2>&1 &
+PID2=$!
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+
+# Use a different buildid to avoid using same cache.
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT2/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 11234567 || true
+
+grep "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT1
+grep -v "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT2 | grep "not found" vlog$PORT2
+
+kill $PID1 $PID2
+wait $PID1 $PID2
+
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/test-subr.sh b/tests/test-subr.sh
index 411e5f28..95dcbb26 100644
--- a/tests/test-subr.sh
+++ b/tests/test-subr.sh
@@ -27,6 +27,13 @@ test_dir="test-$$"
mkdir -p "$test_dir"
cd "$test_dir"
+type curl 2>/dev/null || (echo "need curl"; exit 77)
+type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
+type cpio 2>/dev/null || (echo "need cpio"; exit 77)
+type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
+bsdtar --version | grep -q zstd && zstd=true || zstd=false
+echo "zstd=$zstd bsdtar=`bsdtar --version`"
+
#LC_ALL=C
#export LC_ALL
@@ -35,7 +42,11 @@ remove_files=
# Tests that trap EXIT (0) themselves should call this explicitly.
exit_cleanup()
{
- rm -f $remove_files; cd ..; rmdir $test_dir
+ rm -rf ${PWD}/.client_cache*
+ rm -f $remove_files;
+ ls -a ${PWD}
+ cd ..;
+ rmdir $test_dir
}
trap exit_cleanup 0
@@ -212,3 +223,114 @@ testrun_on_self_quiet()
# Only exit if something failed
if test $exit_status != 0; then exit $exit_status; fi
}
+
+cleanup()
+{
+ if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
+ if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
+ rm -rf ${PWD}/.client_cache F R D L Z ${PWD}/foobar ${PWD}/mocktree
+ exit_cleanup
+}
+trap cleanup 0 1 2 3 5 9 15
+errfiles_list=
+err() {
+ echo ERROR REPORTS
+ for ports in $PORT1 $PORT2
+ do
+ echo ERROR REPORT $port metrics
+ curl -s http://127.0.0.1:$port/metrics
+ echo
+ done
+ for x in $errfiles_list
+ do
+ echo ERROR REPORT "$x"
+ cat $x
+ echo
+ done
+ false # trigger set -e
+}
+trap err ERR
+
+errfiles() {
+ errfiles_list="$errfiles_list $*"
+}
+
+wait_ready()
+{
+ port=$1;
+ what=$2;
+ value=$3;
+ timeout=20;
+
+ echo "Wait $timeout seconds on $port for metric $what to change to $value"
+ while [ $timeout -gt 0 ]; do
+ mvalue="$(curl -s http://127.0.0.1:$port/metrics \
+ | grep "$what" | awk '{print $NF}')"
+ if [ -z "$mvalue" ]; then mvalue=0; fi
+ echo "metric $what: $mvalue"
+ if [ "$mvalue" -eq "$value" ]; then
+ break;
+ fi
+ sleep 0.5;
+ ((timeout--));
+ done;
+
+ if [ $timeout -eq 0 ]; then
+ echo "metric $what never changed to $value on port $port"
+ err
+ fi
+}
+
+archive_test() {
+ __BUILDID=$1
+ __SOURCEPATH=$2
+ __SOURCESHA1=$3
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ # check that timestamps are plausible - older than the near-present (tmpdir mtime)
+ test $filename -ot `pwd`
+
+ # run again to assure that fdcache is being enjoyed
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ if test "x$__SOURCEPATH" != "x"; then
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
+ hash=`cat $filename | sha1sum | awk '{print $1}'`
+ test $__SOURCESHA1 = $hash
+ test $filename -ot `pwd`
+ fi
+}
+
+get_ports() {
+ while true; do
+ PORT1=`expr '(' $RANDOM % 100 ')' + $base`
+ ss -atn | fgrep ":$PORT1" || break
+ done
+# Some tests will use two servers, so assign the second var
+ while true; do
+ PORT2=`expr '(' $RANDOM % 100 ')' + $base`
+ ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
+ done
+
+}
+
+VERBOSE=-vvv
+# We gather the LD_LIBRARY_PATH with this cunning trick:
+ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
+PORT1=0
+PORT2=0
+PID1=0
+PID2=0
--
2.31.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests
2021-09-02 18:59 ` Noah Sanci
@ 2021-09-03 8:45 ` Mark Wielaard
0 siblings, 0 replies; 6+ messages in thread
From: Mark Wielaard @ 2021-09-03 8:45 UTC (permalink / raw)
To: Noah Sanci; +Cc: elfutils-devel
[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]
Hi Noah,
On Thu, Sep 02, 2021 at 02:59:06PM -0400, Noah Sanci wrote:
> As for the rpm/Makefile.am stuff, I think that was a temporary fix for
> an odd error I got while testing. I just forgot to remove it and see if
> it would break again. It has been removed and makes now.
>
> As for the 'base' idea, I gave it some thought and believe this is the
> best solution and I have implemented it.
Great, thanks.
BTW. You don't have to use globals in shell, shell function can take arguments.
They are just not named, so you then have to use $1, $2, etc to use them.
But this works too.
> The patch also includes x-forwarded
It even contained it twice :)
I removed the not really used copy run-debuginfod-X-FORWARDED.sh.
I also removed run-debuginfod-response-headers.sh for now since it
depends on a patch that isn't in yet.
Finally when testing on a system where /bin/sh isn't bash I noticed
that trap err ERR isn't posix shell. Sigh.
I moved the helpers into their own tests/debuginfod-subr.sh which is
then only called from the run-debuginfod-*.sh scripts (which are all
explicitly bash scripts). This does have the advantage that the extra
checks are only run when testing debuginfod and not any of the other
tests.
Attached the variant that I pushed. I hope I didn't mess anything up.
Thanks so much for splitting these tests, it is really great to have
individual test cases.
Cheers,
Mark
[-- Attachment #2: 0001-debuginfod-Fracture-tests-run-debuginfod-find.sh-int.patch --]
[-- Type: text/x-diff, Size: 134950 bytes --]
From ef856762088bcd85ac9121b129dba0c6910369a2 Mon Sep 17 00:00:00 2001
From: Noah Sanci <nsanci@redhat.com>
Date: Tue, 10 Aug 2021 11:21:35 -0400
Subject: [PATCH] debuginfod: Fracture tests/run-debuginfod-find.sh into
specific tests
tests/run-debuginfod-find.sh was a massive test script with many broadly
varying tests. This caused the test script to fail when any number of
things went wrong and because of its intertwined nature, detecting the
source of a failure could be difficult. The size of the test script
also meant many unrelated tests were run making the testing process
unnecessarily lengthy.
This patch fractures tests/run-debuginfod-find.sh into smaller, more
manageable individual test script files. This ensures that when failure
occurs, a programmer can easily determine where their patch went
wrong. It also allows programmers to specify exactly which tests to
run, making testing more efficient. Redundancies are also reduced by
placing code in tests/debuginfod-subr.sh.
Signed-off-by: Noah Sanci <nsanci@redhat.com>
---
tests/ChangeLog | 28 +
tests/Makefile.am | 44 +-
tests/debuginfod-subr.sh | 149 ++++
tests/run-debuginfod-000-permission.sh | 85 ++
tests/run-debuginfod-archive-groom.sh | 154 ++++
tests/run-debuginfod-archive-rename.sh | 96 +++
tests/run-debuginfod-archive-test.sh | 83 ++
tests/run-debuginfod-artifact-running.sh | 122 +++
tests/run-debuginfod-dlopen.sh | 84 ++
tests/run-debuginfod-duplicate-urls.sh | 52 ++
| 101 +++
tests/run-debuginfod-fd-prefetch-caches.sh | 58 ++
tests/run-debuginfod-federation-link.sh | 161 ++++
tests/run-debuginfod-federation-metrics.sh | 213 ++++++
tests/run-debuginfod-federation-sqlite.sh | 202 +++++
tests/run-debuginfod-file.sh | 41 +
tests/run-debuginfod-find.sh | 852 ---------------------
tests/run-debuginfod-malformed.sh | 110 +++
tests/run-debuginfod-no-urls.sh | 41 +
tests/run-debuginfod-query-retry.sh | 35 +
tests/run-debuginfod-regex.sh | 98 +++
tests/run-debuginfod-sizetime.sh | 78 ++
tests/run-debuginfod-tmp-home.sh | 125 +++
tests/run-debuginfod-writable.sh | 84 ++
tests/run-debuginfod-x-forwarded-for.sh | 63 ++
25 files changed, 2305 insertions(+), 854 deletions(-)
create mode 100755 tests/debuginfod-subr.sh
create mode 100755 tests/run-debuginfod-000-permission.sh
create mode 100755 tests/run-debuginfod-archive-groom.sh
create mode 100755 tests/run-debuginfod-archive-rename.sh
create mode 100755 tests/run-debuginfod-archive-test.sh
create mode 100755 tests/run-debuginfod-artifact-running.sh
create mode 100755 tests/run-debuginfod-dlopen.sh
create mode 100755 tests/run-debuginfod-duplicate-urls.sh
create mode 100755 tests/run-debuginfod-extraction.sh
create mode 100755 tests/run-debuginfod-fd-prefetch-caches.sh
create mode 100755 tests/run-debuginfod-federation-link.sh
create mode 100755 tests/run-debuginfod-federation-metrics.sh
create mode 100755 tests/run-debuginfod-federation-sqlite.sh
create mode 100755 tests/run-debuginfod-file.sh
delete mode 100755 tests/run-debuginfod-find.sh
create mode 100755 tests/run-debuginfod-malformed.sh
create mode 100755 tests/run-debuginfod-no-urls.sh
create mode 100755 tests/run-debuginfod-query-retry.sh
create mode 100755 tests/run-debuginfod-regex.sh
create mode 100755 tests/run-debuginfod-sizetime.sh
create mode 100755 tests/run-debuginfod-tmp-home.sh
create mode 100755 tests/run-debuginfod-writable.sh
create mode 100755 tests/run-debuginfod-x-forwarded-for.sh
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 3fa7c2e4..0529f641 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -12,6 +12,34 @@
* run-debuginfod-find.sh: Add test for X-Forwarded-For hops limit.
+2021-08-20 Noah Sanci <nsanci@redhat.com>
+
+ * debuginfod-find.sh: Separated file into
+ run-debuginfod-000-permission.sh,
+ run-debuginfod-archive-groom.sh,
+ run-debuginfod-archive-rename.sh,
+ run-debuginfod-archive-test.sh,
+ run-debuginfod-artifact-running.sh,
+ run-debuginfod-dlopen.sh,
+ run-debuginfod-duplicate-urls.sh,
+ run-debuginfod-extraction.sh,
+ run-debuginfod-fd-prefetch-caches.sh,
+ run-debuginfod-federation-link.sh,
+ run-debuginfod-federation-metrics.sh,
+ run-debuginfod-federation-sqlite.sh,
+ run-debuginfod-file.sh,
+ run-debuginfod-malformed.sh,
+ run-debuginfod-no-urls.sh,
+ run-debuginfod-query-retry.sh,
+ run-debuginfod-regex.sh,
+ run-debuginfod-sizetime.sh,
+ run-debuginfod-tmp-home.sh,
+ run-debuginfod-x-forwarded.sh
+ and run-debuginfod-writable.sh.
+ All files source debuginfod-subr.sh and use the $base variable to find ports.
+ * tests/Makefile.am: Added the above new files to the test suite
+ * tests/debuginfod-subr.sh: Added some general functions for above tests
+
2021-08-04 Mark Wielaard <mark@klomp.org>
PR28190
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 429649f4..3c302e72 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -211,7 +211,27 @@ if DEBUGINFOD
check_PROGRAMS += debuginfod_build_id_find
# With the dummy delegation doesn't work
if !DUMMY_LIBDEBUGINFOD
-TESTS += run-debuginfod-find.sh
+TESTS += run-debuginfod-dlopen.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh \
+ run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-x-forwarded-for.sh
endif
endif
@@ -474,7 +494,27 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-elfclassify.sh run-elfclassify-self.sh \
run-disasm-riscv64.sh \
testfile-riscv64-dis1.o.bz2 testfile-riscv64-dis1.expect.bz2 \
- run-debuginfod-find.sh \
+ run-debuginfod-extraction.sh \
+ run-debuginfod-federation-link.sh \
+ run-debuginfod-federation-metrics.sh \
+ run-debuginfod-artifact-running.sh \
+ run-debuginfod-federation-sqlite.sh \
+ run-debuginfod-x-forwarded-for.sh \
+ run-debuginfod-fd-prefetch-caches.sh \
+ run-debuginfod-regex.sh \
+ run-debuginfod-duplicate-urls.sh \
+ run-debuginfod-file.sh \
+ run-debuginfod-sizetime.sh \
+ run-debuginfod-dlopen.sh \
+ run-debuginfod-malformed.sh \
+ run-debuginfod-000-permission.sh \
+ run-debuginfod-tmp-home.sh \
+ run-debuginfod-writable.sh \
+ run-debuginfod-no-urls.sh \
+ run-debuginfod-query-retry.sh \
+ run-debuginfod-archive-groom.sh \
+ run-debuginfod-archive-rename.sh \
+ run-debuginfod-archive-test.sh \
debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
diff --git a/tests/debuginfod-subr.sh b/tests/debuginfod-subr.sh
new file mode 100755
index 00000000..3222a2b0
--- /dev/null
+++ b/tests/debuginfod-subr.sh
@@ -0,0 +1,149 @@
+# Copyright (C) 2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# sourced from run-debuginfod-*.sh tests (must be bash scripts)
+
+. $srcdir/test-subr.sh # includes set -e
+
+type curl 2>/dev/null || (echo "need curl"; exit 77)
+type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
+type cpio 2>/dev/null || (echo "need cpio"; exit 77)
+type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
+bsdtar --version | grep -q zstd && zstd=true || zstd=false
+echo "zstd=$zstd bsdtar=`bsdtar --version`"
+
+cleanup()
+{
+ if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
+ if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
+ rm -rf F R D L Z ${PWD}/foobar ${PWD}/mocktree ${PWD}/.client_cache* ${PWD}/tmp*
+ exit_cleanup
+}
+
+# clean up trash if we were aborted early
+trap cleanup 0 1 2 3 5 9 15
+
+errfiles_list=
+err() {
+ echo ERROR REPORTS
+ for ports in $PORT1 $PORT2
+ do
+ echo ERROR REPORT $port metrics
+ curl -s http://127.0.0.1:$port/metrics
+ echo
+ done
+ for x in $errfiles_list
+ do
+ echo ERROR REPORT "$x"
+ cat $x
+ echo
+ done
+ false # trigger set -e
+}
+trap err ERR
+
+errfiles() {
+ errfiles_list="$errfiles_list $*"
+}
+
+# We want to run debuginfod in the background. We also want to start
+# it with the same check/installcheck-sensitive LD_LIBRARY_PATH stuff
+# that the testrun alias sets. But: we if we just use
+# testrun .../debuginfod
+# it runs in a subshell, with different pid, so not helpful.
+#
+# So we gather the LD_LIBRARY_PATH with this cunning trick:
+ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
+
+wait_ready()
+{
+ port=$1;
+ what=$2;
+ value=$3;
+ timeout=20;
+
+ echo "Wait $timeout seconds on $port for metric $what to change to $value"
+ while [ $timeout -gt 0 ]; do
+ mvalue="$(curl -s http://127.0.0.1:$port/metrics \
+ | grep "$what" | awk '{print $NF}')"
+ if [ -z "$mvalue" ]; then mvalue=0; fi
+ echo "metric $what: $mvalue"
+ if [ "$mvalue" -eq "$value" ]; then
+ break;
+ fi
+ sleep 0.5;
+ ((timeout--));
+ done;
+
+ if [ $timeout -eq 0 ]; then
+ echo "metric $what never changed to $value on port $port"
+ err
+ fi
+}
+
+archive_test() {
+ __BUILDID=$1
+ __SOURCEPATH=$2
+ __SOURCESHA1=$3
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ # check that timestamps are plausible - older than the near-present (tmpdir mtime)
+ test $filename -ot `pwd`
+
+ # run again to assure that fdcache is being enjoyed
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
+ buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
+ test $__BUILDID = $buildid
+ test $filename -ot `pwd`
+
+ if test "x$__SOURCEPATH" != "x"; then
+ filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
+ hash=`cat $filename | sha1sum | awk '{print $1}'`
+ test $__SOURCESHA1 = $hash
+ test $filename -ot `pwd`
+ fi
+}
+
+get_ports() {
+ while true; do
+ PORT1=`expr '(' $RANDOM % 100 ')' + $base`
+ ss -atn | fgrep ":$PORT1" || break
+ done
+# Some tests will use two servers, so assign the second var
+ while true; do
+ PORT2=`expr '(' $RANDOM % 100 ')' + $base`
+ ss -atn | fgrep ":$PORT2" && $PORT1 -ne $PORT2 || break
+ done
+
+}
+
+VERBOSE=-vvv
+# We gather the LD_LIBRARY_PATH with this cunning trick:
+ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
+PORT1=0
+PORT2=0
+PID1=0
+PID2=0
diff --git a/tests/run-debuginfod-000-permission.sh b/tests/run-debuginfod-000-permission.sh
new file mode 100755
index 00000000..28e54385
--- /dev/null
+++ b/tests/run-debuginfod-000-permission.sh
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8000
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+# PR25628
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# The query is designed to fail, while the 000-permission file should be created.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
+ err
+fi
+
+if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
+ echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
+ err
+fi
+
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" != "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} has changed."
+ err
+fi
+
+# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
+sleep 1
+bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
+if [ "$bytecount_before" == "$bytecount_after" ]; then
+ echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-groom.sh b/tests/run-debuginfod-archive-groom.sh
new file mode 100755
index 00000000..0ee056ff
--- /dev/null
+++ b/tests/run-debuginfod-archive-groom.sh
@@ -0,0 +1,154 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8100
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Be patient when run on a busy machine things might take a bit.
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Build a non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/F/prog.c
+gcc -Wl,--build-id -g -o ${PWD}/F/prog ${PWD}/F/prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a ${PWD}/F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+tempfiles ${PWD}/F/prog ${PWD}/F/prog.c
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+tempfiles vlog3 $DB
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir
+
+# Drop some of the artifacts, run a groom cycle; confirm that
+# debuginfod has forgotten them, but remembers others
+rm -r R/debuginfod-rpms/rhel6/*
+kill -USR2 $PID1 # groom cycle
+## 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# Expect 4 rpms containing 2 buildids to be deleted by the groom
+wait_ready $PORT1 'groomed_total{decision="stale"}' 4
+
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+
+# this is one of the buildids from the groom-deleted rpms
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
+
+# but this one was not deleted so should be still around
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID || true
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-archive-rename.sh b/tests/run-debuginfod-archive-rename.sh
new file mode 100755
index 00000000..38697eee
--- /dev/null
+++ b/tests/run-debuginfod-archive-rename.sh
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8200
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DEBUGINFOD_CACHE_PATH $DB
+# Clean old dirictories
+mkdir R ${PWD}/F
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -F -R -d $DB -p $PORT1 -t0 -g0 -v R ${PWD}/F > vlog$PORT1 2>&1 &
+
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Now there should be 1 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+
+########################################################################
+## PR26810: Now rename some files in the R directory, then rescan, so
+# there are two copies of the same buildid in the index, one for the
+# no-longer-existing file name, and one under the new name.
+
+# run a groom cycle to force server to drop its fdcache
+kill -USR2 $PID1 # groom cycle
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
+# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
+mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
+kill -USR1 $PID1 # scan cycle
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# retest rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+
+egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0;
diff --git a/tests/run-debuginfod-archive-test.sh b/tests/run-debuginfod-archive-test.sh
new file mode 100755
index 00000000..bc500540
--- /dev/null
+++ b/tests/run-debuginfod-archive-test.sh
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# find an unused port number
+mkdir R
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8300
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -R -p $PORT1 -t0 -g0 -v R > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# common source file sha1
+SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
+# fedora31
+if [ $zstd = true ]; then
+ # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
+ # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
+ archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
+ archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
+fi
+# fedora30
+archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
+# rhel7
+archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
+# rhel6
+archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
+archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
+# arch
+#archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-artifact-running.sh b/tests/run-debuginfod-artifact-running.sh
new file mode 100755
index 00000000..e96cb966
--- /dev/null
+++ b/tests/run-debuginfod-artifact-running.sh
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8400
+get_ports
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+echo 'int main(int argc, char * argv){ return 0; }' > ${PWD}/prog.c
+gcc -Wl,--build-id -g -o prog ${PWD}/prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d $DB -F -p $PORT1 -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+mv prog F
+mv prog.debug F
+tempfiles prog/F
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+kill -USR1 $PID1
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# Add artifacts to the search paths and test whether debuginfod finds them while already running.
+# Build another, non-stripped binary
+echo "int main() { return 0; }" > ${PWD}/prog2.c
+tempfiles prog2.c
+gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
+#testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
+mv prog2 F
+#mv prog2.debug F
+tempfiles F/prog2 F/prog2.debug
+
+kill -USR1 $PID1
+# Now there should be 3 files in the index
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/prog.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/prog`
+cmp $filename F/prog
+
+# raw source filename
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../prog.c`
+cmp $filename ${PWD}/prog.c
+
+# and also the canonicalized one
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/prog.c`
+cmp $filename ${PWD}/prog.c
+
+# Rerun same tests for the prog2 binary
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
+cmp $filename F/prog2
+grep -q Progress vlog
+grep -q Downloaded.from vlog
+tempfiles vlog
+filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
+cmp $filename F/prog2
+grep -q 'Downloading.*http' vlog2
+tempfiles vlog2
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
+cmp $filename ${PWD}/prog2.c
+
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-dlopen.sh b/tests/run-debuginfod-dlopen.sh
new file mode 100755
index 00000000..6476612c
--- /dev/null
+++ b/tests/run-debuginfod-dlopen.sh
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8500
+get_ports
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod -F -R $VERBOSE -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a'
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether elfutils, via the debuginfod client library dlopen hooks,
+# is able to fetch debuginfo from the local debuginfod.
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-duplicate-urls.sh b/tests/run-debuginfod-duplicate-urls.sh
new file mode 100755
index 00000000..b76b39a3
--- /dev/null
+++ b/tests/run-debuginfod-duplicate-urls.sh
@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8600
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+########################################################################
+## PR27983
+# Ensure no duplicate urls are used in when querying servers for files
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:7999" \
+ LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -vvv executable 0 > vlog1 2>&1 || true
+tempfiles vlog1
+cat vlog1
+if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog1 ) -ne 2 ]; then
+ echo "Duplicate servers remain";
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
--git a/tests/run-debuginfod-extraction.sh b/tests/run-debuginfod-extraction.sh
new file mode 100755
index 00000000..7c534d09
--- /dev/null
+++ b/tests/run-debuginfod-extraction.sh
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir R Z
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8700
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v R Z > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
+ rm -vrf R/debuginfod-rpms/fedora31
+fi
+
+cp -rvp ${abs_srcdir}/debuginfod-tars Z
+
+kill -USR1 $PID1
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# All rpms need to be in the index, except the dummy permission-000 one
+rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
+txz=$(find Z -name \*tar.xz | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
+tb2=$(find Z -name \*tar.bz2 | wc -l)
+wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
+
+kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
+# Wait till both files are in the index and scan/index fully finished
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+# Expect all source files found in the rpms (they are all called hello.c :)
+# We will need to extract all rpms (in their own directory) and could all
+# sources referenced in the .debug files.
+mkdir extracted
+cd extracted
+subdir=0;
+newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
+for i in $newrpms; do
+ subdir=$[$subdir+1];
+ mkdir $subdir;
+ cd $subdir;
+ ls -lah ../$i
+ rpm2cpio ../$i | cpio -ivd;
+ cd ..;
+done
+sourcefiles=$(find -name \*\\.debug \
+ | env LD_LIBRARY_PATH=$ldpath xargs \
+ ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
+ | grep mtime: | wc --lines)
+cd ..
+rm -rf extracted
+
+wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-fd-prefetch-caches.sh b/tests/run-debuginfod-fd-prefetch-caches.sh
new file mode 100755
index 00000000..08b32923
--- /dev/null
+++ b/tests/run-debuginfod-fd-prefetch-caches.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+FDCACHE_FDS=100
+FDCACHE_MBS=100
+PREFETCH_FDS=100
+PREFETCH_MBS=100
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8800
+get_ports
+
+echo $PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT1 \
+ --fdcache-mbs=$FDCACHE_MDS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS \
+ --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -v -F F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+
+grep 'fdcache fds ' vlog$PORT1 #$FDCACHE_FDS
+grep 'fdcache mbs ' vlog$PORT1 #$FDCACHE_MBS
+grep 'prefetch fds ' vlog$PORT1 #$PREFETCH_FDS
+grep 'prefetch mbs ' vlog$PORT1 #$PREFETCH_MBS
+# search the vlog to find what metric counts should be and check the correct metrics
+# were incrimented
+wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
+wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-federation-link.sh b/tests/run-debuginfod-federation-link.sh
new file mode 100755
index 00000000..ae5d4381
--- /dev/null
+++ b/tests/run-debuginfod-federation-link.sh
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT1
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=8900
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+
+kill -USR1 $PID2
+# Wait till both files are in the index.
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+
+exit 0
diff --git a/tests/run-debuginfod-federation-metrics.sh b/tests/run-debuginfod-federation-metrics.sh
new file mode 100755
index 00000000..9998a04a
--- /dev/null
+++ b/tests/run-debuginfod-federation-metrics.sh
@@ -0,0 +1,213 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
+tempfiles $DB
+# Clean old dirictories
+mkdir D L F
+
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9000
+get_ports
+
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT1 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT1 'thread_busy{role="http-metrics"}' 1
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+
+wait_ready $PORT2 'ready' 1
+wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
+wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
+
+# have clients contact the new server
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+########################################################################
+# Fetch some metrics
+curl -s http://127.0.0.1:$PORT1/badapi
+curl -s http://127.0.0.1:$PORT1/metrics
+curl -s http://127.0.0.1:$PORT2/metrics
+curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
+curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# Confirm that some debuginfod client pools are being used
+curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
+
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+kill $PID1
+kill $PID2
+wait $PID1
+wait $PID2
+PID1=0
+PID2=0
+exit 0
+
diff --git a/tests/run-debuginfod-federation-sqlite.sh b/tests/run-debuginfod-federation-sqlite.sh
new file mode 100755
index 00000000..d10a3385
--- /dev/null
+++ b/tests/run-debuginfod-federation-sqlite.sh
@@ -0,0 +1,202 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+export DEBUGINFOD_TIMEOUT=10
+export DEBUGINFOD_URLS='http://127.0.0.1:0' # Note invalid
+tempfiles $DB
+
+# Clean old dirictories
+mkdir D L F
+# not tempfiles F R L D Z - they are directories which we clean up manually
+ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
+
+########################################################################
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv prog F
+mv prog.debug F
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9100
+get_ports
+# Launch server which will be unable to follow symlinks
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB} -F -U -t0 -g0 -p $PORT1 L D F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+
+wait_ready $PORT1 'ready' 1
+
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# NB: run in -L symlink-following mode for the L subdir
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d ${DB}_2 -F -U -p $PORT2 -L L D F > vlog$PORT2 2>&1 &
+PID2=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+tempfiles ${DB}_2
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS='http://127.0.0.1:'$PORT2
+if type bsdtar 2>/dev/null; then
+ # copy in the deb files
+ cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
+ kill -USR1 $PID2
+ wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
+ wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
+ wait_ready $PORT2 'thread_busy{role="scan"}' 0
+
+ # All debs need to be in the index
+ debs=$(find D -name \*.deb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
+ ddebs=$(find D -name \*.ddeb | wc -l)
+ wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
+
+ # ubuntu
+ archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
+fi
+
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# send a request to stress XFF and User-Agent federation relay;
+# we'll grep for the two patterns in vlog$PORT1
+curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
+
+grep UA:TESTCURL vlog$PORT1
+grep XFF:TESTXFF vlog$PORT1
+
+# confirm that first server can't resolve symlinked info in L/ but second can
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
+file L/foo
+file -L L/foo
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+
+# test again with scheme free url
+export DEBUGINFOD_URLS=127.0.0.1:$PORT1
+rm -rf $DEBUGINFOD_CACHE_PATH
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
+export DEBUGINFOD_URLS=127.0.0.1:$PORT2
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# test parallel queries in client
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
+mkdir -p $DEBUGINFOD_CACHE_PATH
+export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# And generate a few errors into the second debuginfod's logs, for analysis just below
+curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
+# NB: this error is used to seed the 404 failure for the survive-404 tests
+
+# Confirm bad artifact types are rejected without leaving trace
+curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
+(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
+
+# DISABLE VALGRIND checking because valgrind might use debuginfod client
+# requests itself, causing confusion about who put what in the cache.
+# It stays disabled till the end of this test.
+unset VALGRIND_CMD
+
+# Confirm that reused curl connections survive 404 errors.
+# The rm's force an uncached fetch
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+# Trigger a flood of requests against the same archive content file.
+# Use a file that hasn't been previously extracted in to make it
+# likely that even this test debuginfod will experience concurrency
+# and impose some "after-you" delays.
+(for i in `seq 100`; do
+ curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
+ done;
+ wait)
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
+# If we could guarantee some minimum number of seconds of CPU time, we
+# could assert that the after_you metrics show some nonzero amount of
+# waiting. A few hundred ms is typical on this developer's workstation.
+
+########################################################################
+# Corrupt the sqlite database and get debuginfod to trip across its errors
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
+dd if=/dev/zero of=$DB bs=1 count=1
+
+# trigger some random activity that's Sure to get sqlite3 upset
+kill -USR1 $PID1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -USR2 $PID1
+wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
+curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
+# Run the tests again without the servers running. The target file should
+# be found in the cache.
+
+kill -INT $PID1 $PID2
+wait $PID1 $PID2
+PID1=0
+PID2=0
+tempfiles .debuginfod_*
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1
+
+# check out the debuginfod logs for the new style status lines
+cat vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
+grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
+
+exit 0
diff --git a/tests/run-debuginfod-file.sh b/tests/run-debuginfod-file.sh
new file mode 100755
index 00000000..22c956dd
--- /dev/null
+++ b/tests/run-debuginfod-file.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Test fetching a file using file:// . No debuginfod server needs to be run for
+# this test.
+local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
+mkdir -p ${local_dir}
+echo "int main() { return 0; }" > ${local_dir}/main.c
+# first test that is doesn't work, when no DEBUGINFOD_URLS is set
+DEBUGINFOD_URLS=""
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
+
+# Now test is with proper DEBUGINFOD_URLS
+DEBUGINFOD_URLS="file://${PWD}/mocktree/"
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
+cmp $filename ${local_dir}/main.c
+
+exit 0
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
deleted file mode 100755
index 7515b7cd..00000000
--- a/tests/run-debuginfod-find.sh
+++ /dev/null
@@ -1,852 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (C) 2019-2021 Red Hat, Inc.
-# This file is part of elfutils.
-#
-# This file is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 3 of the License, or
-# (at your option) any later version.
-#
-# elfutils is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-. $srcdir/test-subr.sh # includes set -e
-
-type curl 2>/dev/null || (echo "need curl"; exit 77)
-type rpm2cpio 2>/dev/null || (echo "need rpm2cpio"; exit 77)
-type cpio 2>/dev/null || (echo "need cpio"; exit 77)
-type bzcat 2>/dev/null || (echo "need bzcat"; exit 77)
-bsdtar --version | grep -q zstd && zstd=true || zstd=false
-echo "zstd=$zstd bsdtar=`bsdtar --version`"
-
-# for test case debugging, uncomment:
-set -x
-VERBOSE=-vvv
-
-DB=${PWD}/.debuginfod_tmp.sqlite
-tempfiles $DB
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
-
-PID1=0
-PID2=0
-PID3=0
-PID4=0
-PID5=0
-PID6=0
-
-cleanup()
-{
- if [ $PID1 -ne 0 ]; then kill $PID1; wait $PID1; fi
- if [ $PID2 -ne 0 ]; then kill $PID2; wait $PID2; fi
- if [ $PID3 -ne 0 ]; then kill $PID3; wait $PID3; fi
- if [ $PID4 -ne 0 ]; then kill $PID4; wait $PID4; fi
- if [ $PID5 -ne 0 ]; then kill $PID5; wait $PID5; fi
- if [ $PID6 -ne 0 ]; then kill $PID6; wait $PID6; fi
- rm -rf F R D L Z ${PWD}/foobar ${PWD}/mocktree ${PWD}/.client_cache* ${PWD}/tmp*
- exit_cleanup
-}
-
-# clean up trash if we were aborted early
-trap cleanup 0 1 2 3 5 9 15
-
-errfiles_list=
-err() {
- echo ERROR REPORTS
- for ports in $PORT1 $PORT2 $PORT3 $PORT4 $PORT5
- do
- echo ERROR REPORT $port metrics
- curl -s http://127.0.0.1:$port/metrics
- echo
- done
- for x in $errfiles_list
- do
- echo ERROR REPORT "$x"
- cat $x
- echo
- done
- false # trigger set -e
-}
-trap err ERR
-
-errfiles() {
- errfiles_list="$errfiles_list $*"
-}
-
-
-# find an unused port number
-while true; do
- PORT1=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT1" || break
-done
-
-# We want to run debuginfod in the background. We also want to start
-# it with the same check/installcheck-sensitive LD_LIBRARY_PATH stuff
-# that the testrun alias sets. But: we if we just use
-# testrun .../debuginfod
-# it runs in a subshell, with different pid, so not helpful.
-#
-# So we gather the LD_LIBRARY_PATH with this cunning trick:
-ldpath=`testrun sh -c 'echo $LD_LIBRARY_PATH'`
-
-mkdir F R L D Z
-# not tempfiles F R L D Z - they are directories which we clean up manually
-ln -s ${abs_builddir}/dwfllines L/foo # any program not used elsewhere in this test
-
-wait_ready()
-{
- port=$1;
- what=$2;
- value=$3;
- timeout=20;
-
- echo "Wait $timeout seconds on $port for metric $what to change to $value"
- while [ $timeout -gt 0 ]; do
- mvalue="$(curl -s http://127.0.0.1:$port/metrics \
- | grep "$what" | awk '{print $NF}')"
- if [ -z "$mvalue" ]; then mvalue=0; fi
- echo "metric $what: $mvalue"
- if [ "$mvalue" -eq "$value" ]; then
- break;
- fi
- sleep 0.5;
- ((timeout--));
- done;
-
- if [ $timeout -eq 0 ]; then
- echo "metric $what never changed to $value on port $port"
- err
- fi
-}
-
-FDCACHE_FDS=50
-FDCACHE_MBS=190
-PREFETCH_FDS=10
-PREFETCH_MBS=120
-# create a bogus .rpm file to evoke a metric-visible error
-# Use a cyclic symlink instead of chmod 000 to make sure even root
-# would see an error (running the testsuite under root is NOT encouraged).
-ln -s R/nothing.rpm R/nothing.rpm
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -R -d $DB -p $PORT1 -t0 -g0 --fdcache-mbs=$FDCACHE_MBS --fdcache-fds=$FDCACHE_FDS --fdcache-prefetch-mbs=$PREFETCH_MBS --fdcache-prefetch-fds=$PREFETCH_FDS --fdcache-mintmp 0 -Z .tar.xz -Z .tar.bz2=bzcat -v R F Z L > vlog$PORT1 2>&1 &
-PID1=$!
-tempfiles vlog$PORT1
-errfiles vlog$PORT1
-# Server must become ready
-wait_ready $PORT1 'ready' 1
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
-
-# Be patient when run on a busy machine things might take a bit.
-export DEBUGINFOD_TIMEOUT=10
-
-# Check thread comm names
-ps -q $PID1 -e -L -o '%p %c %a' | grep groom
-ps -q $PID1 -e -L -o '%p %c %a' | grep scan
-ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
-
-# We use -t0 and -g0 here to turn off time-based scanning & grooming.
-# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
-
-########################################################################
-
-# Compile a simple program, strip its debuginfo and save the build-id.
-# Also move the debuginfo into another directory so that elfutils
-# cannot find it without debuginfod.
-echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
-tempfiles p+r%o\$g.c
-# Create a subdirectory to confound source path names
-mkdir foobar
-gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
-testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
-
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-mv p+r%o\$g F
-mv p+r%o\$g.debug F
-kill -USR1 $PID1
-# Wait till both files are in the index.
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-########################################################################
-
-# Test whether elfutils, via the debuginfod client library dlopen hooks,
-# is able to fetch debuginfo from the local debuginfod.
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-########################################################################
-## PR27892
-# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
-# code
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-tempfiles find-vlog$PORT1
-# wait for the server to fail the same number of times the query is retried.
-wait_ready $PORT1 'http_responses_after_you_milliseconds_count{code="406"}' 1
-# ensure all reporting is functional
-grep 'serving file '$(realpath ${PWD})'/F/p+r%o\$g.debug' vlog$PORT1
-grep 'File too large' vlog$PORT1
-grep 'using max size 1B' find-vlog$PORT1
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxsize check"
- err
-fi
-
-# Ensure DEBUGINFOD_MAXTIME is functional
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:8002/" DEBUGINFOD_MAXTIME=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo F/p+r%o\$g.debug 2> find-vlog$PORT1 || true
-grep 'using max time' find-vlog$PORT1
-# Ensure p+r%o\$g.debug is NOT cached
-if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
- echo "File cached after maxtime check"
- err
-fi
-########################################################################
-# PR25628
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# The query is designed to fail, while the 000-permission file should be created.
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-if [ ! -f $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "could not find cache in $DEBUGINFOD_CACHE_PATH"
- err
-fi
-
-if [ -r $DEBUGINFOD_CACHE_PATH/01234567/debuginfo ]; then
- echo "The cache $DEBUGINFOD_CACHE_PATH/01234567/debuginfo is readable"
- err
-fi
-
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" != "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} has changed."
- err
-fi
-
-# set cache_miss_s to 0 and sleep 1 to make the mtime expire.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_miss_s
-sleep 1
-bytecount_before=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-bytecount_after=`curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count{code="404"}'`
-if [ "$bytecount_before" == "$bytecount_after" ]; then
- echo "http_responses_transfer_bytes_count{code="404"} should be incremented."
- err
-fi
-########################################################################
-
-# Test whether debuginfod-find is able to fetch those files.
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
-cmp $filename F/p+r%o\$g.debug
-if [ -w $filename ]; then
- echo "cache file writable, boo"
- err
-fi
-
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable F/p+r%o\\$g`
-cmp $filename F/p+r%o\$g
-
-# raw source filename
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/foobar///./../p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-# and also the canonicalized one
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID ${PWD}/p+r%o\\$g.c`
-cmp $filename ${PWD}/p+r%o\$g.c
-
-
-########################################################################
-
-# Test whether the cache default locations are correct
-
-mkdir tmphome
-
-# $HOME/.cache should be created.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $HOME/.cache should be found.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
-if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
- echo "could not find cache in $PWD/tmphome/.cache"
- err
-fi
-
-# $XDG_CACHE_HOME should take priority over $HOME.cache.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpxdg/"
- err
-fi
-
-# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
-# priority over $HOME/.cache, $XDG_CACHE_HOME.
-cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
-# ||true is for tolerating errors, such a valgrind or something else
-# leaving 000-perm files in there
-
-# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
-mkdir tmphome/.debuginfod_client_cache/deadbeef
-echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
-cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
-
-# $DEBUGINFO_CACHE_PATH should take priority over all else.
-testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
- echo "could not find cache in $PWD/tmpcache/"
- err
-fi
-
-########################################################################
-
-# Add artifacts to the search paths and test whether debuginfod finds them while already running.
-
-# Build another, non-stripped binary
-echo "int main() { return 0; }" > ${PWD}/prog2.c
-tempfiles prog2.c
-gcc -Wl,--build-id -g -o prog2 ${PWD}/prog2.c
-BUILDID2=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a prog2 | grep 'Build ID' | cut -d ' ' -f 7`
-
-mv prog2 F
-kill -USR1 $PID1
-# Now there should be 3 files in the index
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 3
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-cp $DB $DB.backup
-tempfiles $DB.backup
-# Rerun same tests for the prog2 binary
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo $BUILDID2 2>vlog`
-cmp $filename F/prog2
-cat vlog
-grep -q Progress vlog
-grep -q Downloaded.from vlog
-tempfiles vlog
-filename=`testrun env DEBUGINFOD_PROGRESS=1 ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2 2>vlog2`
-cmp $filename F/prog2
-cat vlog2
-grep -q 'Downloading.*http' vlog2
-tempfiles vlog2
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $BUILDID2 ${PWD}/prog2.c`
-cmp $filename ${PWD}/prog2.c
-
-cp -rvp ${abs_srcdir}/debuginfod-rpms R
-if [ "$zstd" = "false" ]; then # nuke the zstd fedora 31 ones
- rm -vrf R/debuginfod-rpms/fedora31
-fi
-
-cp -rvp ${abs_srcdir}/debuginfod-tars Z
-kill -USR1 $PID1
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 4
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# All rpms need to be in the index, except the dummy permission-000 one
-rpms=$(find R -name \*rpm | grep -v nothing | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".rpm archive"}' $rpms
-txz=$(find Z -name \*tar.xz | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.xz archive"}' $txz
-tb2=$(find Z -name \*tar.bz2 | wc -l)
-wait_ready $PORT1 'scanned_files_total{source=".tar.bz2 archive"}' $tb2
-
-kill -USR1 $PID1 # two hits of SIGUSR1 may be needed to resolve .debug->dwz->srefs
-# Wait till both files are in the index and scan/index fully finished
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 5
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# Expect all source files found in the rpms (they are all called hello.c :)
-# We will need to extract all rpms (in their own directory) and could all
-# sources referenced in the .debug files.
-mkdir extracted
-cd extracted
-subdir=0;
-newrpms=$(find ../R -name \*\.rpm | grep -v nothing)
-for i in $newrpms; do
- subdir=$[$subdir+1];
- mkdir $subdir;
- cd $subdir;
- ls -lah ../$i
- rpm2cpio ../$i | cpio -ivd;
- cd ..;
-done
-sourcefiles=$(find -name \*\\.debug \
- | env LD_LIBRARY_PATH=$ldpath xargs \
- ${abs_top_builddir}/src/readelf --debug-dump=decodedline \
- | grep mtime: | wc --lines)
-cd ..
-rm -rf extracted
-
-wait_ready $PORT1 'found_sourcerefs_total{source=".rpm archive"}' $sourcefiles
-
-########################################################################
-# PR27983 ensure no duplicate urls are used in when querying servers for files
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-env DEBUGINFOD_URLS="http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http://127.0.0.1:$PORT1 http:127.0.0.1:7999" \
- LD_LIBRARY_PATH=$ldpath ${abs_top_builddir}/debuginfod/debuginfod-find -v executable $BUILDID2 > vlog4 2>&1 || true
-tempfiles vlog4
-if [ $( grep -c 'duplicate url: http://127.0.0.1:'$PORT1'.*' vlog4 ) -ne 2 ]; then
- echo "Duplicated servers remain";
- err
-fi
-########################################################################
-# Run a bank of queries against the debuginfod-rpms / debuginfod-debs test cases
-
-archive_test() {
- __BUILDID=$1
- __SOURCEPATH=$2
- __SOURCESHA1=$3
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- # check that timestamps are plausible - older than the near-present (tmpdir mtime)
- test $filename -ot `pwd`
-
- # run again to assure that fdcache is being enjoyed
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $__BUILDID`
- buildid=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a $filename | grep 'Build ID' | cut -d ' ' -f 7`
- test $__BUILDID = $buildid
- test $filename -ot `pwd`
-
- if test "x$__SOURCEPATH" != "x"; then
- filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source $__BUILDID $__SOURCEPATH`
- hash=`cat $filename | sha1sum | awk '{print $1}'`
- test $__SOURCESHA1 = $hash
- test $filename -ot `pwd`
- fi
-}
-
-
-# common source file sha1
-SHA=f4a1a8062be998ae93b8f1cd744a398c6de6dbb1
-# fedora31
-if [ $zstd = true ]; then
- # fedora31 uses zstd compression on rpms, older rpm2cpio/libarchive can't handle it
- # and we're not using the fancy -Z '.rpm=(rpm2cpio|zstdcat)<' workaround in this testsuite
- archive_test 420e9e3308971f4b817cc5bf83928b41a6909d88 /usr/src/debug/hello3-1.0-2.x86_64/foobar////./../hello.c $SHA
- archive_test 87c08d12c78174f1082b7c888b3238219b0eb265 /usr/src/debug/hello3-1.0-2.x86_64///foobar/./..//hello.c $SHA
-fi
-# fedora30
-archive_test c36708a78618d597dee15d0dc989f093ca5f9120 /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-archive_test 41a236eb667c362a1c4196018cc4581e09722b1b /usr/src/debug/hello2-1.0-2.x86_64/hello.c $SHA
-# rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-# rhel6
-archive_test bbbf92ebee5228310e398609c23c2d7d53f6e2f9 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test d44d42cbd7d915bc938c81333a21e355a6022fb7 /usr/src/debug/hello-1.0/hello.c $SHA
-# arch
-archive_test cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb /usr/src/debug/hello.c 7a1334e086b97e5f124003a6cfb3ed792d10cdf4
-
-RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in rhel6/ subdir, for a later test
-
-
-########################################################################
-
-# Drop some of the artifacts, run a groom cycle; confirm that
-# debuginfod has forgotten them, but remembers others
-
-rm -r R/debuginfod-rpms/rhel6/*
-kill -USR2 $PID1 # groom cycle
-# 1 groom cycle already took place at/soon-after startup, so -USR2 makes 2
-wait_ready $PORT1 'thread_work_total{role="groom"}' 2
-# Expect 4 rpms containing 2 buildids to be deleted by the groom
-wait_ready $PORT1 'groomed_total{decision="stale"}' 4
-
-rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
-
-# this is one of the buildids from the groom-deleted rpms
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID && false || true
-# but this one was not deleted so should be still around
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID2
-
-########################################################################
-
-# PR26810: Now rename some files in the R directory, then rescan, so
-# there are two copies of the same buildid in the index, one for the
-# no-longer-existing file name, and one under the new name.
-
-# run a groom cycle to force server to drop its fdcache
-kill -USR2 $PID1 # groom cycle
-wait_ready $PORT1 'thread_work_total{role="groom"}' 3
-# move it around a couple of times to make it likely to hit a nonexistent entry during iteration
-mv R/debuginfod-rpms/rhel7 R/debuginfod-rpms/rhel7renamed
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 6
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed R/debuginfod-rpms/rhel7renamed2
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 7
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-mv R/debuginfod-rpms/rhel7renamed2 R/debuginfod-rpms/rhel7renamed3
-kill -USR1 $PID1 # scan cycle
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 8
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-
-# retest rhel7
-archive_test bc1febfd03ca05e030f0d205f7659db29f8a4b30 /usr/src/debug/hello-1.0/hello.c $SHA
-archive_test f0aa15b8aba4f3c28cac3c2a73801fefa644a9f2 /usr/src/debug/hello-1.0/hello.c $SHA
-
-egrep '(libc.error.*rhel7)|(bc1febfd03ca)|(f0aa15b8aba)' vlog$PORT1
-
-########################################################################
-## PR25978
-# Ensure that the fdcache options are working.
-grep "prefetch fds" vlog$PORT1
-grep "prefetch mbs" vlog$PORT1
-grep "fdcache fds" vlog$PORT1
-grep "fdcache mbs" vlog$PORT1
-# search the vlog to find what metric counts should be and check the correct metrics
-# were incrimented
-wait_ready $PORT1 'fdcache_op_count{op="enqueue"}' $( grep -c 'interned.*front=1' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="evict"}' $( grep -c 'evicted a=.*' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_enqueue"}' $( grep -c 'interned.*front=0' vlog$PORT1 )
-wait_ready $PORT1 'fdcache_op_count{op="prefetch_evict"}' $( grep -c 'evicted from prefetch a=.*front=0' vlog$PORT1 || true )
-########################################################################
-
-# Federation mode
-
-# find another unused port
-while true; do
- PORT2=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT2" || break
-done
-
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache2
-mkdir -p $DEBUGINFOD_CACHE_PATH
-# NB: inherits the DEBUGINFOD_URLS to the first server
-# NB: run in -L symlink-following mode for the L subdir
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d ${DB}_2 -p $PORT2 -L L D > vlog$PORT2 2>&1 &
-PID2=$!
-tempfiles vlog$PORT2
-errfiles vlog$PORT2
-tempfiles ${DB}_2
-wait_ready $PORT2 'ready' 1
-wait_ready $PORT2 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
-wait_ready $PORT2 'thread_busy{role="http-buildid"}' 0
-wait_ready $PORT2 'thread_busy{role="http-metrics"}' 1
-
-# have clients contact the new server
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-
-if type bsdtar 2>/dev/null; then
- # copy in the deb files
- cp -rvp ${abs_srcdir}/debuginfod-debs/*deb D
- kill -USR1 $PID2
- wait_ready $PORT2 'thread_work_total{role="traverse"}' 2
- wait_ready $PORT2 'thread_work_pending{role="scan"}' 0
- wait_ready $PORT2 'thread_busy{role="scan"}' 0
-
- # All debs need to be in the index
- debs=$(find D -name \*.deb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".deb archive"}' `expr $debs`
- ddebs=$(find D -name \*.ddeb | wc -l)
- wait_ready $PORT2 'scanned_files_total{source=".ddeb archive"}' `expr $ddebs`
-
- # ubuntu
- archive_test f17a29b5a25bd4960531d82aa6b07c8abe84fa66 "" ""
-fi
-
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# send a request to stress XFF and User-Agent federation relay;
-# we'll grep for the two patterns in vlog$PORT1
-curl -s -H 'User-Agent: TESTCURL' -H 'X-Forwarded-For: TESTXFF' $DEBUGINFOD_URLS/buildid/deaddeadbeef00000000/debuginfo -o /dev/null || true
-
-grep UA:TESTCURL vlog$PORT1
-grep XFF:TESTXFF vlog$PORT1
-
-
-# confirm that first server can't resolve symlinked info in L/ but second can
-BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
- -a L/foo | grep 'Build ID' | cut -d ' ' -f 7`
-file L/foo
-file -L L/foo
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=http://127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test again with scheme free url
-export DEBUGINFOD_URLS=127.0.0.1:$PORT1
-rm -rf $DEBUGINFOD_CACHE_PATH
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo # drop 000-perm negative-hit file
-export DEBUGINFOD_URLS=127.0.0.1:$PORT2
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# test parallel queries in client
-export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache3
-mkdir -p $DEBUGINFOD_CACHE_PATH
-export DEBUGINFOD_URLS="BAD http://127.0.0.1:$PORT1 127.0.0.1:$PORT1 http://127.0.0.1:$PORT2 DNE"
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-########################################################################
-
-# Fetch some metrics
-curl -s http://127.0.0.1:$PORT1/badapi
-curl -s http://127.0.0.1:$PORT1/metrics
-curl -s http://127.0.0.1:$PORT2/metrics
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*error'
-curl -s http://127.0.0.1:$PORT1/metrics | grep -q 'http_responses_total.*result.*fdcache'
-curl -s http://127.0.0.1:$PORT2/metrics | grep -q 'http_responses_total.*result.*upstream'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_duration_milliseconds_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_transfer_bytes_sum'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'fdcache_'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'traversed_total'
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'scanned_bytes_total'
-
-# And generate a few errors into the second debuginfod's logs, for analysis just below
-curl -s http://127.0.0.1:$PORT2/badapi > /dev/null || true
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/debuginfo > /dev/null || true
-# NB: this error is used to seed the 404 failure for the survive-404 tests
-
-# Confirm bad artifact types are rejected without leaving trace
-curl -s http://127.0.0.1:$PORT2/buildid/deadbeef/badtype > /dev/null || true
-(curl -s http://127.0.0.1:$PORT2/metrics | grep 'badtype') && false
-
-# DISABLE VALGRIND checking because valgrind might use debuginfod client
-# requests itself, causing confusion about who put what in the cache.
-# It stays disabled till the end of this test.
-unset VALGRIND_CMD
-
-# Confirm that reused curl connections survive 404 errors.
-# The rm's force an uncached fetch
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-rm -f $DEBUGINFOD_CACHE_PATH/$BUILDID/debuginfo .client_cache*/$BUILDID/debuginfo
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
-
-# Confirm that some debuginfod client pools are being used
-curl -s http://127.0.0.1:$PORT2/metrics | grep 'dc_pool_op.*reuse'
-
-# Trigger a flood of requests against the same archive content file.
-# Use a file that hasn't been previously extracted in to make it
-# likely that even this test debuginfod will experience concurrency
-# and impose some "after-you" delays.
-(for i in `seq 100`; do
- curl -s http://127.0.0.1:$PORT1/buildid/87c08d12c78174f1082b7c888b3238219b0eb265/executable >/dev/null &
- done;
- wait)
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'http_responses_after_you.*'
-# If we could guarantee some minimum number of seconds of CPU time, we
-# could assert that the after_you metrics show some nonzero amount of
-# waiting. A few hundred ms is typical on this developer's workstation.
-
-
-########################################################################
-# Corrupt the sqlite database and get debuginfod to trip across its errors
-
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset'
-ls -al $DB
-dd if=/dev/zero of=$DB bs=1 count=1
-ls -al $DB
-# trigger some random activity that's Sure to get sqlite3 upset
-kill -USR1 $PID1
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 9
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -USR2 $PID1
-wait_ready $PORT1 'thread_work_total{role="groom"}' 4
-curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true
-curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite'
-
-########################################################################
-
-# Run the tests again without the servers running. The target file should
-# be found in the cache.
-
-kill -INT $PID1 $PID2
-wait $PID1 $PID2
-PID1=0
-PID2=0
-tempfiles .debuginfod_*
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog2 1
-
-# check out the debuginfod logs for the new style status lines
-# cat vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/.* 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /metrics 200 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /badapi 503 ' vlog$PORT2
-grep -q 'UA:.*XFF:.*GET /buildid/deadbeef.* 404 ' vlog$PORT2
-
-########################################################################
-
-# Add some files to the cache that do not fit its naming format.
-# They should survive cache cleaning.
-mkdir $DEBUGINFOD_CACHE_PATH/malformed
-touch $DEBUGINFOD_CACHE_PATH/malformed0
-touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
-
-# A valid format for an empty buildid subdirectory
-mkdir $DEBUGINFOD_CACHE_PATH/00000000
-touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
-
-# Trigger a cache clean and run the tests again. The clients should be unable to
-# find the target.
-echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
-echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
-
-testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
-
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID2 && false || true
-
-if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
- || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
- echo "unrelated files did not survive cache cleaning"
- err
-fi
-
-if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
- echo "failed to rmdir old cache dir"
- err
-fi
-
-# Test debuginfod without a path list; reuse $PORT1
-env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
-PID3=$!
-wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
-wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
-wait_ready $PORT1 'thread_busy{role="scan"}' 0
-kill -int $PID3
-wait $PID3
-PID3=0
-
-########################################################################
-# Test fetching a file using file:// . No debuginfod server needs to be run for
-# this test.
-local_dir=${PWD}/mocktree/buildid/aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd/source/my/path
-mkdir -p ${local_dir}
-echo "int main() { return 0; }" > ${local_dir}/main.c
-
-# first test that is doesn't work, when no DEBUGINFOD_URLS is set
-DEBUGINFOD_URLS=""
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c && false || true
-
-# Now test is with proper DEBUGINFOD_URLS
-DEBUGINFOD_URLS="file://${PWD}/mocktree/"
-filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find source aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd /my/path/main.c`
-cmp $filename ${local_dir}/main.c
-
-########################################################################
-## PR27711
-# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
-while true; do
- PORT3=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep ":$PORT3" || break
-done
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT3/" ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -p $PORT3 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d $DB.backup > vlog$PORT3 2>&1 &
-PID4=$!
-wait_ready $PORT3 'ready' 1
-tempfiles vlog$PORT3
-errfiles vlog$PORT3
-
-kill -USR2 $PID4
-wait_ready $PORT3 'thread_work_total{role="groom"}' 1
-wait_ready $PORT3 'groom{statistic="archive d/e"}' 0
-wait_ready $PORT3 'groom{statistic="archive sdef"}' 0
-wait_ready $PORT3 'groom{statistic="archive sref"}' 0
-wait_ready $PORT3 'groom{statistic="buildids"}' 0
-wait_ready $PORT3 'groom{statistic="file d/e"}' 0
-wait_ready $PORT3 'groom{statistic="file s"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (#)"}' 0
-wait_ready $PORT3 'groom{statistic="files scanned (mb)"}' 0
-
-kill $PID4
-wait $PID4
-PID4=0
-
-########################################################################
-# set up tests for retrying failed queries.
-retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
- ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
- | grep -c 'Retry failed query'`
-if [ $retry_attempts -ne 10 ]; then
- echo "retry mechanism failed."
- exit 1;
- fi
-
-########################################################################
-
-# Test when debuginfod hitting X-Forwarded-For hops limit.
-# This test will start two servers (as a loop) with two different hop limits.
-
-while true; do
- PORT4=`expr '(' $RANDOM % 1000 ')' + 9000`
- PORT5=`expr '(' $RANDOM % 1000 ')' + 9000`
- ss -atn | fgrep -e ":$PORT4" -e ":$PORT5"|| break
-done
-
-# Make sure the vlogs are cleaned up after the test
-# and that they are printed on error.
-tempfiles vlog$PORT4 vlog$PORT5
-errfiles vlog$PORT4 vlog$PORT5
-
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
-PID5=$!
-
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
-PID6=$!
-
-wait_ready $PORT4 'ready' 1
-wait_ready $PORT5 'ready' 1
-
-export DEBUGINFOD_URLS="http://127.0.0.1:$PORT4/"
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
-
-# Use a different buildid to avoid using same cache.
-export DEBUGINFOD_URLS="http://127.0.0.1:$PORT5/"
-testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 11234567 || true
-
-grep "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT4
-grep -v "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT5 | grep "not found" vlog$PORT5
-
-kill $PID5 $PID6
-wait $PID5 $PID6
-
-PID5=0
-PID6=0
-
-exit 0
diff --git a/tests/run-debuginfod-malformed.sh b/tests/run-debuginfod-malformed.sh
new file mode 100755
index 00000000..78b3b7fc
--- /dev/null
+++ b/tests/run-debuginfod-malformed.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9200
+# find an unused port number
+get_ports
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+# Add some files to the cache that do not fit its naming format.
+# They should survive cache cleaning.
+mkdir $DEBUGINFOD_CACHE_PATH/malformed
+touch $DEBUGINFOD_CACHE_PATH/malformed0
+touch $DEBUGINFOD_CACHE_PATH/malformed/malformed1
+
+# A valid format for an empty buildid subdirectory
+mkdir $DEBUGINFOD_CACHE_PATH/00000000
+touch -d '1970-01-01' $DEBUGINFOD_CACHE_PATH/00000000 # old enough to guarantee nukage
+
+# Trigger a cache clean and run the tests again. The clients should be unable to
+# find the target.
+echo 0 > $DEBUGINFOD_CACHE_PATH/cache_clean_interval_s
+echo 0 > $DEBUGINFOD_CACHE_PATH/max_unused_age_s
+
+testrun ${abs_builddir}/debuginfod_build_id_find -e F/p+r%o\$g 1
+
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID && false || true
+
+if [ ! -f $DEBUGINFOD_CACHE_PATH/malformed0 ] \
+ || [ ! -f $DEBUGINFOD_CACHE_PATH/malformed/malformed1 ]; then
+ echo "unrelated files did not survive cache cleaning"
+ err
+fi
+
+if [ -d $DEBUGINFOD_CACHE_PATH/00000000 ]; then
+ echo "failed to rmdir old cache dir"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-no-urls.sh b/tests/run-debuginfod-no-urls.sh
new file mode 100755
index 00000000..7e3ffef6
--- /dev/null
+++ b/tests/run-debuginfod-no-urls.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# Test debuginfod without a path list
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9300
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -U -d :memory: -p $PORT1 -L -F &
+PID1=$!
+
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+kill -int $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-query-retry.sh b/tests/run-debuginfod-query-retry.sh
new file mode 100755
index 00000000..3c5542d5
--- /dev/null
+++ b/tests/run-debuginfod-query-retry.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+########################################################################
+# set up tests for retrying failed queries.
+retry_attempts=`(testrun env DEBUGINFOD_URLS=http://255.255.255.255/JUNKJUNK DEBUGINFOD_RETRY_LIMIT=10 DEBUGINFOD_VERBOSE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo /bin/ls || true) 2>&1 >/dev/null \
+ | grep -c 'Retry failed query'`
+if [ $retry_attempts -ne 10 ]; then
+ echo "retry mechanism failed."
+ exit 1;
+fi
+
+exit 0;
diff --git a/tests/run-debuginfod-regex.sh b/tests/run-debuginfod-regex.sh
new file mode 100755
index 00000000..32ddf6ff
--- /dev/null
+++ b/tests/run-debuginfod-regex.sh
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9400
+get_ports
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -F -p $PORT1 -t0 -g0 -d ${DB} F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+
+mv prog F
+mv prog.debug F
+tempfiles F/prog.debug F/prog
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+cp ${DB} ${DB}.backup
+tempfiles ${DB}.backup
+
+kill $PID1
+wait $PID1
+PID1=0
+
+#######################################################################
+## PR27711
+# Test to ensure that the --include="^$" --exclude=".*" options remove all files from a database backup
+#
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod \
+ $VERBOSE -p $PORT2 -t0 -g0 --regex-groom --include="^$" --exclude=".*" -d ${DB}.backup > vlog$PORT2 2>&1 &
+
+#reuse PID1
+PID1=$!
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+# Server must become ready
+wait_ready $PORT2 'ready' 1
+
+kill -USR2 $PID1
+wait_ready $PORT2 'thread_work_total{role="groom"}' 1
+wait_ready $PORT2 'groom{statistic="archive d/e"}' 0
+wait_ready $PORT2 'groom{statistic="archive sdef"}' 0
+wait_ready $PORT2 'groom{statistic="archive sref"}' 0
+wait_ready $PORT2 'groom{statistic="buildids"}' 0
+wait_ready $PORT2 'groom{statistic="file d/e"}' 0
+wait_ready $PORT2 'groom{statistic="file s"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (#)"}' 0
+wait_ready $PORT2 'groom{statistic="files scanned (mb)"}' 0
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
+
diff --git a/tests/run-debuginfod-sizetime.sh b/tests/run-debuginfod-sizetime.sh
new file mode 100755
index 00000000..068b548b
--- /dev/null
+++ b/tests/run-debuginfod-sizetime.sh
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+set -x
+unset VALGRIND_CMD
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9600
+get_ports
+
+echo "int main() { return 0; }" > ${PWD}/prog.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o prog ${PWD}/foobar///./../prog.c
+testrun ${abs_top_builddir}/src/strip -g -f prog.debug ${PWD}/prog
+tempfiles prog prog.debug prog.c
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 -t0 -g0 ${PWD} > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+## PR27892
+# Ensure DEBUGINFOD_MAXSIZE is functional and sends back the correct http
+# code
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_RETRY_LIMIT=1 DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXSIZE=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v executable ${PWD}/prog 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+errfiles find-vlog$PORT1
+echo "Checking maxsize"
+grep "using max size 1B" find-vlog$PORT1
+echo "Checking maxsize"
+grep 'serving file '$(realpath ${PWD})'/prog' vlog$PORT1
+echo "Checking maxsize"
+grep 'File too large' vlog$PORT1
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxsize check"
+ err
+fi
+# Ensure no file is downloaded for longer than DEBUGINFOD_MAXTIME
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/" DEBUGINFOD_MAXTIME=1 \
+ ${abs_top_builddir}/debuginfod/debuginfod-find -v debuginfo ${PWD}/prog.debug 2> find-vlog$PORT1 || true
+tempfiles find-vlog$PORT1
+grep 'using max time' find-vlog$PORT1
+# Ensure p+r%o\$g.debug is NOT cached
+if [ -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID} ]; then
+ echo "File cached after maxtime check"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0;
diff --git a/tests/run-debuginfod-tmp-home.sh b/tests/run-debuginfod-tmp-home.sh
new file mode 100755
index 00000000..81986198
--- /dev/null
+++ b/tests/run-debuginfod-tmp-home.sh
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+mkdir F
+mkdir -p $DEBUGINFOD_CACHE_PATH
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9700
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Test whether the cache default locations are correct
+mkdir tmphome
+
+# $HOME/.cache should be created.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+
+# $HOME/.cache should be found.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find executable $BUILDID
+if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/executable ]; then
+ echo "could not find cache in $PWD/tmphome/.cache"
+ err
+fi
+# $XDG_CACHE_HOME should take priority over $HOME.cache.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpxdg/debuginfod_client/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpxdg/"
+ err
+fi
+
+# A cache at the old default location ($HOME/.debuginfod_client_cache) should take
+# priority over $HOME/.cache, $XDG_CACHE_HOME.
+cp -vr $DEBUGINFOD_CACHE_PATH tmphome/.debuginfod_client_cache || true
+# ||true is for tolerating errors, such a valgrind or something else
+# leaving 000-perm files in there
+
+# Add a file that doesn't exist in $HOME/.cache, $XDG_CACHE_HOME.
+mkdir tmphome/.debuginfod_client_cache/deadbeef
+echo ELF... > tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+filename=`testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH= ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo deadbeef`
+cmp $filename tmphome/.debuginfod_client_cache/deadbeef/debuginfo
+
+# $DEBUGINFO_CACHE_PATH should take priority over all else.
+testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=$PWD/tmpcache ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
+if [ ! -f $PWD/tmpcache/$BUILDID/debuginfo ]; then
+ echo "could not find cache in $PWD/tmpcache/"
+ err
+fi
+rm -rf ${PWD}/tmphome/ ${PWD}/tmpxdg ${PWD}/tmpcache
+
+kill $PID1
+wait $PID1
+PID1=0
+exit 0
diff --git a/tests/run-debuginfod-writable.sh b/tests/run-debuginfod-writable.sh
new file mode 100755
index 00000000..440a5666
--- /dev/null
+++ b/tests/run-debuginfod-writable.sh
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+mkdir F
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9800
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -F -p $PORT1 \
+ -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1/ # or without trailing /
+
+# Be patient when run on a busy machine things might take a bit.
+export DEBUGINFOD_TIMEOUT=10
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/p+r%o\$g.c
+tempfiles p+r%o\$g.c
+# Create a subdirectory to confound source path names
+mkdir foobar
+gcc -Wl,--build-id -g -o p+r%o\$g ${PWD}/foobar///./../p+r%o\$g.c
+testrun ${abs_top_builddir}/src/strip -g -f p+r%o\$g.debug ${PWD}/p+r%o\$g
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a p+r%o\\$g | grep 'Build ID' | cut -d ' ' -f 7`
+
+mv p+r%o\$g F
+mv p+r%o\$g.debug F
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+#######################################################################
+# Test whether debuginfod-find is able to fetch those files.
+rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
+filename=`testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID`
+cmp $filename F/p+r%o\$g.debug
+if [ -w $filename ]; then
+ echo "cache file writable, boo"
+ err
+fi
+
+kill $PID1
+wait $PID1
+PID1=0
+
+exit 0
diff --git a/tests/run-debuginfod-x-forwarded-for.sh b/tests/run-debuginfod-x-forwarded-for.sh
new file mode 100755
index 00000000..5b756b22
--- /dev/null
+++ b/tests/run-debuginfod-x-forwarded-for.sh
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=9900
+get_ports
+
+# Test when debuginfod hitting X-Forwarded-For hops limit.
+# This test will start two servers (as a loop) with two different hop limits.
+
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT2 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT1 > vlog$PORT1 2>&1 &
+PID1=$!
+
+tempfiles vlog$PORT2
+errfiles vlog$PORT2
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT1 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT2 > vlog$PORT2 2>&1 &
+PID2=$!
+
+wait_ready $PORT1 'ready' 1
+wait_ready $PORT2 'ready' 1
+
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT1/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 01234567 || true
+
+# Use a different buildid to avoid using same cache.
+export DEBUGINFOD_URLS="http://127.0.0.1:$PORT2/"
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo 11234567 || true
+
+grep "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT1
+grep -v "forwared-ttl-limit reached and will not query the upstream servers" vlog$PORT2 | grep "not found" vlog$PORT2
+
+kill $PID1 $PID2
+wait $PID1 $PID2
+
+PID1=0
+PID2=0
+
+exit 0
--
2.32.0
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-03 8:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-23 14:01 debuginfod: Fracture tests/run-debuginfod-find.sh into specific tests Noah Sanci
2021-08-23 19:31 ` Noah Sanci
2021-09-02 11:29 ` Mark Wielaard
2021-09-02 13:42 ` Mark Wielaard
2021-09-02 18:59 ` Noah Sanci
2021-09-03 8:45 ` Mark Wielaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).