Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset)

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

Hi Frank,

On Sat, Aug 28, 2021 at 10:41:30AM -0400, Frank Ch. Eigler via Elfutils-devel wrote:
> > At first I had hoped that using -d :memory: would work. But debuginfod
> > doesn't really support in memory databases like that. [...]
> 
> Yes, that functionality was broken at the point you identifed.  We could
> bring it back by hacking debuginfod's command line parser to map
>     -d :memory:
> to  -d file::memory:?cache=shared
> 
> (https://sqlite.org/inmemorydb.html)

That is really cool and imho really useful for quick testing of debuginfod.
What do you think of the attached patch to implement this?

Thanks,

Mark

[-- Attachment #2: 0001-debuginfod-Turn-d-memory-into-d-file-memory-cache-sh.patch --]
[-- Type: text/x-diff, Size: 4893 bytes --]

From 61554a292f753032490c349fedf47c4fb461e3ea Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 28 Aug 2021 20:25:56 +0200
Subject: [PATCH] debuginfod: Turn -d ":memory:" into -d
 "file::memory:?cache=shared"

debuginfod opens the database twice, once in read/wrote and once in
read-only mode. This means the magic ":memory:" in-memory database
cannot be used as is because the two connections don't really share
the underlying database. Fix this by turning ":memory:" into
":file::memory:?cache=shared" which makes the in-memory database
shared. See https://sqlite.org/inmemorydb.html

Document this in debuginfod.8 and make some tests use -d :memory:

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 debuginfod/ChangeLog         | 5 +++++
 debuginfod/debuginfod.cxx    | 9 ++++++++-
 doc/ChangeLog                | 4 ++++
 doc/debuginfod.8             | 6 ++++--
 tests/ChangeLog              | 5 +++++
 tests/run-debuginfod-find.sh | 7 ++-----
 6 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 395af94f..c5459823 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,8 @@
+2021-08-28  Mark Wielaard  <mjw@redhat.com>
+
+	* debuginfod.cxx (parse_opt): Turn the -d arg ":memory:" into
+	"file::memory:?cache=shared" for the db_path.
+
 2021-08-20  Di Chen  <dichen@redhat.com>
 
 	* debuginfod.cxx (options): Add ARGP_KEY_FORWARDED_TTL_LIMIT.
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 6e182a84..3269f657 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -483,7 +483,14 @@ parse_opt (int key, char *arg,
   switch (key)
     {
     case 'v': verbose ++; break;
-    case 'd': db_path = string(arg); break;
+    case 'd':
+      /* When using the in-memory database make sure it is shareable,
+	 so we can open it twice as read/write and read-only.  */
+      if (strcmp (arg, ":memory:") == 0)
+	db_path = "file::memory:?cache=shared";
+      else
+	db_path = string(arg);
+      break;
     case 'p': http_port = (unsigned) atoi(arg);
       if (http_port == 0 || http_port > 65535)
         argp_failure(state, 1, EINVAL, "port number");
diff --git a/doc/ChangeLog b/doc/ChangeLog
index d5f34f0f..ada48383 100644
--- a/doc/ChangeLog
+++ b/doc/ChangeLog
@@ -1,3 +1,7 @@
+2021-08-28  Di Chen  <dichen@redhat.com>
+
+	* debuginfod.8 (-d): Document ":memory:" as in-memory database.
+
 2021-08-20  Di Chen  <dichen@redhat.com>
 
 	* debuginfod.8: Add --forwarded-ttl-limit=NUM documentation.
diff --git a/doc/debuginfod.8 b/doc/debuginfod.8
index 5b0d793c..f9a418d1 100644
--- a/doc/debuginfod.8
+++ b/doc/debuginfod.8
@@ -117,8 +117,10 @@ file is disposable in the sense that a later rescan will repopulate
 data.  It will contain absolute file path names, so it may not be
 portable across machines.  It may be frequently read/written, so it
 should be on a fast filesystem.  It should not be shared across
-machines or users, to maximize sqlite locking performance.  The
-default database file is \%$HOME/.debuginfod.sqlite.
+machines or users, to maximize sqlite locking performance.  For quick
+testing the magic string ":memory:" can be used to use an one-time
+memory-only database.  The default database file is
+\%$HOME/.debuginfod.sqlite.
 
 .TP
 .B "\-D SQL" "\-\-ddl=SQL"
diff --git a/tests/ChangeLog b/tests/ChangeLog
index cbd1c227..3fa7c2e4 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2021-08-28  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-find.sh: Use ":memory:" for the
+	forwarded-ttl-limit tests.
+
 2021-08-28  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-find.sh: Use clean, separate databases for
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
index 5d38d625..7515b7cd 100755
--- a/tests/run-debuginfod-find.sh
+++ b/tests/run-debuginfod-find.sh
@@ -824,13 +824,10 @@ done
 tempfiles vlog$PORT4 vlog$PORT5
 errfiles vlog$PORT4 vlog$PORT5
 
-# Give each debuginfd its own clean database.
-tempfiles db.$PORT4.sql db.$PORT5.sql
-
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT4.sql --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 &
 PID5=$!
 
-env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT5.sql --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 &
 PID6=$!
 
 wait_ready $PORT4 'ready' 1
-- 
2.32.0