From: "Frank Ch. Eigler" <fche@redhat.com>
To: elfutils-devel@sourceware.org
Subject: patch rfc: PR28661: debuginfod thread-pool
Date: Wed, 8 Dec 2021 22:55:41 -0500 [thread overview]
Message-ID: <20211209035541.GA21366@redhat.com> (raw)
Hi -
While I think this patch itself is fine, and works around the
libmicrohttpd bug that motivated it, I don't know how to test it
seriously in the testsuite. (We can certainly try few -C options for
parsing & operability.) The error edge cases only appear to occur
under very high load and task limits such as those imposed by systemd
cgroups.
Author: Frank Ch. Eigler <fche@redhat.com>
Date: Wed Dec 8 22:41:47 2021 -0500
PR28661: debuginfo connection thread pool support
Add an option -C, which activates libmicrohttpd's thread-pool mode for
handling incoming http connections. Add libmicrohttpd error-logging
callback function so as to receive indication of its internal errors,
and relay counts to our metrics. Some of these internal errors tipped
us off to a microhttpd bug that thread pooling works around. Document
in debuginfod.8 page. Hand-tested against "ulimit -u NNN" shells.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 21d0721ef080..125e7d816f51 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,11 @@
+2021-12-08 Frank Ch. Eigler <fche@redhat.com>
+
+ * debuginfod.cxx (connection_pool): New global.
+ (parse_opt): Parse & check -C option to set it.
+ (error_cb): New callback for libmicrohttpd error counting.
+ (main): Activate MHD_OPTION_THREAD_POOL_SIZE if appropriate.
+ Activate error_cb.
+
2021-12-01 Mark Wielaard <mark@klomp.org>
* debuginfod-client.c (debuginfod_query_server): Free tmp_url on
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 0d3f02978ee2..bb0c6cd65670 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -352,7 +352,9 @@ static const struct argp_option options[] =
{ "rescan-time", 't', "SECONDS", 0, "Number of seconds to wait between rescans, 0=disable.", 0 },
{ "groom-time", 'g', "SECONDS", 0, "Number of seconds to wait between database grooming, 0=disable.", 0 },
{ "maxigroom", 'G', NULL, 0, "Run a complete database groom/shrink pass at startup.", 0 },
- { "concurrency", 'c', "NUM", 0, "Limit scanning thread concurrency to NUM.", 0 },
+ { "concurrency", 'c', "NUM", 0, "Limit scanning thread concurrency to NUM, default=#CPUs.", 0 },
+ { "connection-pool", 'C', "NUM", OPTION_ARG_OPTIONAL,
+ "Use webapi connection pool with NUM threads, default=unlim.", 0 },
{ "include", 'I', "REGEX", 0, "Include files matching REGEX, default=all.", 0 },
{ "exclude", 'X', "REGEX", 0, "Exclude files matching REGEX, default=none.", 0 },
{ "port", 'p', "NUM", 0, "HTTP port to listen on, default 8002.", 0 },
@@ -411,6 +413,7 @@ static unsigned rescan_s = 300;
static unsigned groom_s = 86400;
static bool maxigroom = false;
static unsigned concurrency = std::thread::hardware_concurrency() ?: 1;
+static int connection_pool = 0;
static set<string> source_paths;
static bool scan_files = false;
static map<string,string> scan_archives;
@@ -557,6 +560,16 @@ parse_opt (int key, char *arg,
concurrency = (unsigned) atoi(arg);
if (concurrency < 1) concurrency = 1;
break;
+ case 'C':
+ if (arg)
+ {
+ connection_pool = atoi(arg);
+ if (connection_pool < 2)
+ argp_failure(state, 1, EINVAL, "-C NUM minimum 2");
+ }
+ else // arg not given
+ connection_pool = std::thread::hardware_concurrency() * 2 ?: 2;
+ break;
case 'I':
// NB: no problem with unconditional free here - an earlier failed regcomp would exit program
if (passive_p)
@@ -3684,7 +3698,13 @@ sigusr2_handler (int /* sig */)
}
-
+static void // error logging callback from libmicrohttpd internals
+error_cb (void *arg, const char *fmt, va_list ap)
+{
+ (void) arg;
+ inc_metric("error_count","libmicrohttpd",fmt);
+ (void) vdprintf (STDERR_FILENO, fmt, ap);
+}
// A user-defined sqlite function, to score the sharedness of the
@@ -3829,7 +3849,7 @@ main (int argc, char *argv[])
// Start httpd server threads. Separate pool for IPv4 and IPv6, in
// case the host only has one protocol stack.
- MHD_Daemon *d4 = MHD_start_daemon (MHD_USE_THREAD_PER_CONNECTION
+ MHD_Daemon *d4 = MHD_start_daemon ((connection_pool ? 0 : MHD_USE_THREAD_PER_CONNECTION)
#if MHD_VERSION >= 0x00095300
| MHD_USE_INTERNAL_POLLING_THREAD
#else
@@ -3839,8 +3859,11 @@ main (int argc, char *argv[])
http_port,
NULL, NULL, /* default accept policy */
handler_cb, NULL, /* handler callback */
+ MHD_OPTION_EXTERNAL_LOGGER, error_cb, NULL,
+ (connection_pool ? MHD_OPTION_THREAD_POOL_SIZE : MHD_OPTION_END),
+ (connection_pool ? (int)connection_pool : MHD_OPTION_END),
MHD_OPTION_END);
- MHD_Daemon *d6 = MHD_start_daemon (MHD_USE_THREAD_PER_CONNECTION
+ MHD_Daemon *d6 = MHD_start_daemon ((connection_pool ? 0 : MHD_USE_THREAD_PER_CONNECTION)
#if MHD_VERSION >= 0x00095300
| MHD_USE_INTERNAL_POLLING_THREAD
#else
@@ -3851,6 +3874,9 @@ main (int argc, char *argv[])
http_port,
NULL, NULL, /* default accept policy */
handler_cb, NULL, /* handler callback */
+ MHD_OPTION_EXTERNAL_LOGGER, error_cb, NULL,
+ (connection_pool ? MHD_OPTION_THREAD_POOL_SIZE : MHD_OPTION_END),
+ (connection_pool ? (int)connection_pool : MHD_OPTION_END),
MHD_OPTION_END);
if (d4 == NULL && d6 == NULL) // neither ipv4 nor ipv6? boo
@@ -3904,6 +3930,8 @@ main (int argc, char *argv[])
if (! passive_p)
obatched(clog) << "search concurrency " << concurrency << endl;
+ obatched(clog) << "webapi connection pool " << connection_pool
+ << (connection_pool ? "" : " (unlimited)") << endl;
if (! passive_p)
obatched(clog) << "rescan time " << rescan_s << endl;
obatched(clog) << "fdcache fds " << fdcache_fds << endl;
diff --git a/doc/ChangeLog b/doc/ChangeLog
index 7a73fa107bf3..f25bcd2ee2b3 100644
--- a/doc/ChangeLog
+++ b/doc/ChangeLog
@@ -1,3 +1,8 @@
+2021-12-08 Frank Ch. Eigler <fche@redhat.com>
+
+ PR28661
+ * debuginfod.8 (-C): Document new flag.
+
2021-11-05 Frank Ch. Eigler <fche@redhat.com>
PR28430
diff --git a/doc/debuginfod.8 b/doc/debuginfod.8
index 1e56f6568322..ee8e4078e5b5 100644
--- a/doc/debuginfod.8
+++ b/doc/debuginfod.8
@@ -205,6 +205,23 @@ This important for controlling CPU-intensive operations like parsing
an ELF file and especially decompressing archives. The default is the
number of processors on the system; the minimum is 1.
+.TP
+.B "\-C" "\-C=NUM" "\-\-connection\-pool" "\-\-connection\-pool=NUM"
+Set the size of the pool of threads serving webapi queries. The
+following table summarizes the interpretaton of this option and its
+optional NUM parameter.
+.TS
+l l.
+no option clone new thread for every request, no fixed pool
+\-C use a fixed thread pool sized automatically
+\-C=NUM use a fixed thread pool sized NUM, minimum 2
+.TE
+
+The first mode is useful for friendly bursty traffic. The second mode
+is a simple and safe configuration based on the number of processors.
+The third mode is suitable for tuned load-limiting configurations
+facing unruly traffic.
+
.TP
.B "\-L"
Traverse symbolic links encountered during traversal of the PATHs,
next reply other threads:[~2021-12-09 3:55 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-09 3:55 Frank Ch. Eigler [this message]
2021-12-09 16:47 ` Mark Wielaard
2021-12-09 16:54 ` Frank Ch. Eigler
2021-12-10 0:10 ` Frank Ch. Eigler
2021-12-10 17:44 ` Mark Wielaard
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211209035541.GA21366@redhat.com \
--to=fche@redhat.com \
--cc=elfutils-devel@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).