From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (wildebeest.demon.nl [212.238.236.112]) by sourceware.org (Postfix) with ESMTPS id 7853F3858435 for ; Sat, 28 Aug 2021 18:36:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7853F3858435 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from reform (deer0x0b.wildebeest.org [172.31.17.141]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 7C5FE3000ADE; Sat, 28 Aug 2021 20:36:36 +0200 (CEST) Received: by reform (Postfix, from userid 1000) id 1000C2E80EBC; Sat, 28 Aug 2021 20:36:36 +0200 (CEST) Date: Sat, 28 Aug 2021 20:36:36 +0200 From: Mark Wielaard To: "Frank Ch. Eigler" Cc: elfutils-devel@sourceware.org Subject: Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset) Message-ID: References: <20210827231914.54F2E817A9B@builder.wildebeest.org> <20210828144130.GA679@redhat.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8DS5Dx5H3O/uJ7UY" Content-Disposition: inline In-Reply-To: <20210828144130.GA679@redhat.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_BADIPHTTP, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Aug 2021 18:36:53 -0000 --8DS5Dx5H3O/uJ7UY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Frank, On Sat, Aug 28, 2021 at 10:41:30AM -0400, Frank Ch. Eigler via Elfutils-devel wrote: > > At first I had hoped that using -d :memory: would work. But debuginfod > > doesn't really support in memory databases like that. [...] > > Yes, that functionality was broken at the point you identifed. We could > bring it back by hacking debuginfod's command line parser to map > -d :memory: > to -d file::memory:?cache=shared > > (https://sqlite.org/inmemorydb.html) That is really cool and imho really useful for quick testing of debuginfod. What do you think of the attached patch to implement this? Thanks, Mark --8DS5Dx5H3O/uJ7UY Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-debuginfod-Turn-d-memory-into-d-file-memory-cache-sh.patch" >From 61554a292f753032490c349fedf47c4fb461e3ea Mon Sep 17 00:00:00 2001 From: Mark Wielaard Date: Sat, 28 Aug 2021 20:25:56 +0200 Subject: [PATCH] debuginfod: Turn -d ":memory:" into -d "file::memory:?cache=shared" debuginfod opens the database twice, once in read/wrote and once in read-only mode. This means the magic ":memory:" in-memory database cannot be used as is because the two connections don't really share the underlying database. Fix this by turning ":memory:" into ":file::memory:?cache=shared" which makes the in-memory database shared. See https://sqlite.org/inmemorydb.html Document this in debuginfod.8 and make some tests use -d :memory: Signed-off-by: Mark Wielaard --- debuginfod/ChangeLog | 5 +++++ debuginfod/debuginfod.cxx | 9 ++++++++- doc/ChangeLog | 4 ++++ doc/debuginfod.8 | 6 ++++-- tests/ChangeLog | 5 +++++ tests/run-debuginfod-find.sh | 7 ++----- 6 files changed, 28 insertions(+), 8 deletions(-) diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog index 395af94f..c5459823 100644 --- a/debuginfod/ChangeLog +++ b/debuginfod/ChangeLog @@ -1,3 +1,8 @@ +2021-08-28 Mark Wielaard + + * debuginfod.cxx (parse_opt): Turn the -d arg ":memory:" into + "file::memory:?cache=shared" for the db_path. + 2021-08-20 Di Chen * debuginfod.cxx (options): Add ARGP_KEY_FORWARDED_TTL_LIMIT. diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 6e182a84..3269f657 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -483,7 +483,14 @@ parse_opt (int key, char *arg, switch (key) { case 'v': verbose ++; break; - case 'd': db_path = string(arg); break; + case 'd': + /* When using the in-memory database make sure it is shareable, + so we can open it twice as read/write and read-only. */ + if (strcmp (arg, ":memory:") == 0) + db_path = "file::memory:?cache=shared"; + else + db_path = string(arg); + break; case 'p': http_port = (unsigned) atoi(arg); if (http_port == 0 || http_port > 65535) argp_failure(state, 1, EINVAL, "port number"); diff --git a/doc/ChangeLog b/doc/ChangeLog index d5f34f0f..ada48383 100644 --- a/doc/ChangeLog +++ b/doc/ChangeLog @@ -1,3 +1,7 @@ +2021-08-28 Di Chen + + * debuginfod.8 (-d): Document ":memory:" as in-memory database. + 2021-08-20 Di Chen * debuginfod.8: Add --forwarded-ttl-limit=NUM documentation. diff --git a/doc/debuginfod.8 b/doc/debuginfod.8 index 5b0d793c..f9a418d1 100644 --- a/doc/debuginfod.8 +++ b/doc/debuginfod.8 @@ -117,8 +117,10 @@ file is disposable in the sense that a later rescan will repopulate data. It will contain absolute file path names, so it may not be portable across machines. It may be frequently read/written, so it should be on a fast filesystem. It should not be shared across -machines or users, to maximize sqlite locking performance. The -default database file is \%$HOME/.debuginfod.sqlite. +machines or users, to maximize sqlite locking performance. For quick +testing the magic string ":memory:" can be used to use an one-time +memory-only database. The default database file is +\%$HOME/.debuginfod.sqlite. .TP .B "\-D SQL" "\-\-ddl=SQL" diff --git a/tests/ChangeLog b/tests/ChangeLog index cbd1c227..3fa7c2e4 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -1,3 +1,8 @@ +2021-08-28 Mark Wielaard + + * run-debuginfod-find.sh: Use ":memory:" for the + forwarded-ttl-limit tests. + 2021-08-28 Mark Wielaard * run-debuginfod-find.sh: Use clean, separate databases for diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh index 5d38d625..7515b7cd 100755 --- a/tests/run-debuginfod-find.sh +++ b/tests/run-debuginfod-find.sh @@ -824,13 +824,10 @@ done tempfiles vlog$PORT4 vlog$PORT5 errfiles vlog$PORT4 vlog$PORT5 -# Give each debuginfd its own clean database. -tempfiles db.$PORT4.sql db.$PORT5.sql - -env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT4.sql --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 & +env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 & PID5=$! -env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT5.sql --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 & +env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 & PID6=$! wait_ready $PORT4 'ready' 1 -- 2.32.0 --8DS5Dx5H3O/uJ7UY--