* Buildbot failure in Wildebeest Builder on whole buildset @ 2021-08-27 23:19 buildbot 2021-08-28 14:01 ` Mark Wielaard 0 siblings, 1 reply; 5+ messages in thread From: buildbot @ 2021-08-27 23:19 UTC (permalink / raw) To: elfutils-devel The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils. Full details are available at: https://builder.wildebeest.org/buildbot/#builders/2/builds/803 Buildbot URL: https://builder.wildebeest.org/buildbot/ Worker for this Build: debian-amd64 Build Reason: <unknown> Blamelist: Di Chen <dichen@redhat.com> BUILD FAILED: failed test (failure) Sincerely, -The BuildbotThe Buildbot has detected a new failure on builder elfutils-fedora-x86_64 while building elfutils. Full details are available at: https://builder.wildebeest.org/buildbot/#builders/3/builds/807 Buildbot URL: https://builder.wildebeest.org/buildbot/ Worker for this Build: fedora-x86_64 Build Reason: <unknown> Blamelist: Di Chen <dichen@redhat.com> BUILD FAILED: failed test (failure) Sincerely, -The BuildbotThe Buildbot has detected a new failure on builder elfutils-debian-arm64 while building elfutils. Full details are available at: https://builder.wildebeest.org/buildbot/#builders/45/builds/239 Buildbot URL: https://builder.wildebeest.org/buildbot/ Worker for this Build: debian-arm64 Build Reason: <unknown> Blamelist: Di Chen <dichen@redhat.com> BUILD FAILED: failed test (failure) Sincerely, -The Buildbot ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Buildbot failure in Wildebeest Builder on whole buildset 2021-08-27 23:19 Buildbot failure in Wildebeest Builder on whole buildset buildbot @ 2021-08-28 14:01 ` Mark Wielaard 2021-08-28 14:41 ` Frank Ch. Eigler 0 siblings, 1 reply; 5+ messages in thread From: Mark Wielaard @ 2021-08-28 14:01 UTC (permalink / raw) To: buildbot; +Cc: elfutils-devel [-- Attachment #1: Type: text/plain, Size: 1085 bytes --] Hi, On Fri, Aug 27, 2021 at 11:19:14PM +0000, buildbot@builder.wildebeest.org wrote: > The Buildbot has detected a new failure on builder elfutils-debian-amd64 while building elfutils. > Full details are available at: > https://builder.wildebeest.org/buildbot/#builders/2/builds/803 > > Buildbot URL: https://builder.wildebeest.org/buildbot/ > > Worker for this Build: debian-amd64 > > Build Reason: <unknown> > Blamelist: Di Chen <dichen@redhat.com> > > BUILD FAILED: failed test (failure) This took me a while to figure out because it does not consistently fail. It turns out that sharing a database between two debuginfod instances (especially if they forward queries to each other) is not a good idea. Hopefully fixed with the attached patch. At first I had hoped that using -d :memory: would work. But debuginfod doesn't really support in memory databases like that. It opens the database twice in read/write and read-only mode. But :memory: is special and isn't always a new database, so the read-only copy doesn't see what is in the read/write copy. Cheers, Mark [-- Attachment #2: 0001-tests-Use-fresh-separate-databases-for-debuginfd-for.patch --] [-- Type: text/x-diff, Size: 2123 bytes --] From 17a9b1303e533c13aac6550844bdd68c669091bf Mon Sep 17 00:00:00 2001 From: Mark Wielaard <mark@klomp.org> Date: Sat, 28 Aug 2021 15:54:18 +0200 Subject: [PATCH] tests: Use fresh separate databases for debuginfd forwarded-ttl-limit Sharing the database between the two debuginfod instances that forward queries to each other causes issues. Make both debuginfod instances use a new fresh database. Signed-off-by: Mark Wielaard <mark@klomp.org> --- tests/ChangeLog | 5 +++++ tests/run-debuginfod-find.sh | 7 +++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/tests/ChangeLog b/tests/ChangeLog index 29c48b97..cbd1c227 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -1,3 +1,8 @@ +2021-08-28 Mark Wielaard <mark@klomp.org> + + * run-debuginfod-find.sh: Use clean, separate databases for + forwarded-ttl-limit tests. + 2021-08-20 Di Chen <dichen@redhat.com> * run-debuginfod-find.sh: Add test for X-Forwarded-For hops limit. diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh index 7e12dd7f..5d38d625 100755 --- a/tests/run-debuginfod-find.sh +++ b/tests/run-debuginfod-find.sh @@ -824,10 +824,13 @@ done tempfiles vlog$PORT4 vlog$PORT5 errfiles vlog$PORT4 vlog$PORT5 -env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 & +# Give each debuginfd its own clean database. +tempfiles db.$PORT4.sql db.$PORT5.sql + +env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT4.sql --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 & PID5=$! -env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 & +env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT5.sql --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 & PID6=$! wait_ready $PORT4 'ready' 1 -- 2.32.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Buildbot failure in Wildebeest Builder on whole buildset 2021-08-28 14:01 ` Mark Wielaard @ 2021-08-28 14:41 ` Frank Ch. Eigler 2021-08-28 18:36 ` Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset) Mark Wielaard 0 siblings, 1 reply; 5+ messages in thread From: Frank Ch. Eigler @ 2021-08-28 14:41 UTC (permalink / raw) To: Mark Wielaard; +Cc: elfutils-devel Hi - > This took me a while to figure out because it does not consistently > fail. It turns out that sharing a database between two debuginfod > instances (especially if they forward queries to each other) is not a > good idea. That's correct. Each test case invocation of debuginfod should be run with its own -d $database. > Hopefully fixed with the attached patch. > At first I had hoped that using -d :memory: would work. But debuginfod > doesn't really support in memory databases like that. [...] Yes, that functionality was broken at the point you identifed. We could bring it back by hacking debuginfod's command line parser to map -d :memory: to -d file::memory:?cache=shared (https://sqlite.org/inmemorydb.html) - FChE ^ permalink raw reply [flat|nested] 5+ messages in thread
* Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset) 2021-08-28 14:41 ` Frank Ch. Eigler @ 2021-08-28 18:36 ` Mark Wielaard 2021-08-28 19:23 ` Frank Ch. Eigler 0 siblings, 1 reply; 5+ messages in thread From: Mark Wielaard @ 2021-08-28 18:36 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: elfutils-devel [-- Attachment #1: Type: text/plain, Size: 626 bytes --] Hi Frank, On Sat, Aug 28, 2021 at 10:41:30AM -0400, Frank Ch. Eigler via Elfutils-devel wrote: > > At first I had hoped that using -d :memory: would work. But debuginfod > > doesn't really support in memory databases like that. [...] > > Yes, that functionality was broken at the point you identifed. We could > bring it back by hacking debuginfod's command line parser to map > -d :memory: > to -d file::memory:?cache=shared > > (https://sqlite.org/inmemorydb.html) That is really cool and imho really useful for quick testing of debuginfod. What do you think of the attached patch to implement this? Thanks, Mark [-- Attachment #2: 0001-debuginfod-Turn-d-memory-into-d-file-memory-cache-sh.patch --] [-- Type: text/x-diff, Size: 4893 bytes --] From 61554a292f753032490c349fedf47c4fb461e3ea Mon Sep 17 00:00:00 2001 From: Mark Wielaard <mark@klomp.org> Date: Sat, 28 Aug 2021 20:25:56 +0200 Subject: [PATCH] debuginfod: Turn -d ":memory:" into -d "file::memory:?cache=shared" debuginfod opens the database twice, once in read/wrote and once in read-only mode. This means the magic ":memory:" in-memory database cannot be used as is because the two connections don't really share the underlying database. Fix this by turning ":memory:" into ":file::memory:?cache=shared" which makes the in-memory database shared. See https://sqlite.org/inmemorydb.html Document this in debuginfod.8 and make some tests use -d :memory: Signed-off-by: Mark Wielaard <mark@klomp.org> --- debuginfod/ChangeLog | 5 +++++ debuginfod/debuginfod.cxx | 9 ++++++++- doc/ChangeLog | 4 ++++ doc/debuginfod.8 | 6 ++++-- tests/ChangeLog | 5 +++++ tests/run-debuginfod-find.sh | 7 ++----- 6 files changed, 28 insertions(+), 8 deletions(-) diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog index 395af94f..c5459823 100644 --- a/debuginfod/ChangeLog +++ b/debuginfod/ChangeLog @@ -1,3 +1,8 @@ +2021-08-28 Mark Wielaard <mjw@redhat.com> + + * debuginfod.cxx (parse_opt): Turn the -d arg ":memory:" into + "file::memory:?cache=shared" for the db_path. + 2021-08-20 Di Chen <dichen@redhat.com> * debuginfod.cxx (options): Add ARGP_KEY_FORWARDED_TTL_LIMIT. diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 6e182a84..3269f657 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -483,7 +483,14 @@ parse_opt (int key, char *arg, switch (key) { case 'v': verbose ++; break; - case 'd': db_path = string(arg); break; + case 'd': + /* When using the in-memory database make sure it is shareable, + so we can open it twice as read/write and read-only. */ + if (strcmp (arg, ":memory:") == 0) + db_path = "file::memory:?cache=shared"; + else + db_path = string(arg); + break; case 'p': http_port = (unsigned) atoi(arg); if (http_port == 0 || http_port > 65535) argp_failure(state, 1, EINVAL, "port number"); diff --git a/doc/ChangeLog b/doc/ChangeLog index d5f34f0f..ada48383 100644 --- a/doc/ChangeLog +++ b/doc/ChangeLog @@ -1,3 +1,7 @@ +2021-08-28 Di Chen <dichen@redhat.com> + + * debuginfod.8 (-d): Document ":memory:" as in-memory database. + 2021-08-20 Di Chen <dichen@redhat.com> * debuginfod.8: Add --forwarded-ttl-limit=NUM documentation. diff --git a/doc/debuginfod.8 b/doc/debuginfod.8 index 5b0d793c..f9a418d1 100644 --- a/doc/debuginfod.8 +++ b/doc/debuginfod.8 @@ -117,8 +117,10 @@ file is disposable in the sense that a later rescan will repopulate data. It will contain absolute file path names, so it may not be portable across machines. It may be frequently read/written, so it should be on a fast filesystem. It should not be shared across -machines or users, to maximize sqlite locking performance. The -default database file is \%$HOME/.debuginfod.sqlite. +machines or users, to maximize sqlite locking performance. For quick +testing the magic string ":memory:" can be used to use an one-time +memory-only database. The default database file is +\%$HOME/.debuginfod.sqlite. .TP .B "\-D SQL" "\-\-ddl=SQL" diff --git a/tests/ChangeLog b/tests/ChangeLog index cbd1c227..3fa7c2e4 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -1,3 +1,8 @@ +2021-08-28 Mark Wielaard <mark@klomp.org> + + * run-debuginfod-find.sh: Use ":memory:" for the + forwarded-ttl-limit tests. + 2021-08-28 Mark Wielaard <mark@klomp.org> * run-debuginfod-find.sh: Use clean, separate databases for diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh index 5d38d625..7515b7cd 100755 --- a/tests/run-debuginfod-find.sh +++ b/tests/run-debuginfod-find.sh @@ -824,13 +824,10 @@ done tempfiles vlog$PORT4 vlog$PORT5 errfiles vlog$PORT4 vlog$PORT5 -# Give each debuginfd its own clean database. -tempfiles db.$PORT4.sql db.$PORT5.sql - -env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT4.sql --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 & +env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT5 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 0 -p $PORT4 > vlog$PORT4 2>&1 & PID5=$! -env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d db.$PORT5.sql --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 & +env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS=http://127.0.0.1:$PORT4 ${abs_builddir}/../debuginfod/debuginfod $VERBOSE -d :memory: --forwarded-ttl-limit 1 -p $PORT5 > vlog$PORT5 2>&1 & PID6=$! wait_ready $PORT4 'ready' 1 -- 2.32.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset) 2021-08-28 18:36 ` Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset) Mark Wielaard @ 2021-08-28 19:23 ` Frank Ch. Eigler 0 siblings, 0 replies; 5+ messages in thread From: Frank Ch. Eigler @ 2021-08-28 19:23 UTC (permalink / raw) To: Mark Wielaard; +Cc: elfutils-devel Hi - > That is really cool and imho really useful for quick testing of debuginfod. > What do you think of the attached patch to implement this? chef's kiss - FChE ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-08-28 19:23 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-27 23:19 Buildbot failure in Wildebeest Builder on whole buildset buildbot 2021-08-28 14:01 ` Mark Wielaard 2021-08-28 14:41 ` Frank Ch. Eigler 2021-08-28 18:36 ` Using in-memory database (Was: Buildbot failure in Wildebeest Builder on whole buildset) Mark Wielaard 2021-08-28 19:23 ` Frank Ch. Eigler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).