From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id D05C83858C56 for ; Sat, 23 Apr 2022 20:19:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D05C83858C56 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from reform (deer0x0b.wildebeest.org [172.31.17.141]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id C835430003C8; Sat, 23 Apr 2022 22:19:40 +0200 (CEST) Received: by reform (Postfix, from userid 1000) id DB6E32E822B8; Sat, 23 Apr 2022 22:19:39 +0200 (CEST) Date: Sat, 23 Apr 2022 22:19:39 +0200 From: Mark Wielaard To: elfutils-devel@sourceware.org Cc: "Frank Ch. Eigler" , Thomas Fitzsimmons Subject: =?utf-8?Q?run-debuginfod-webapi-concurrenc?= =?utf-8?B?eS5zaCAoV2FzOiDimKAgQnVpbGRibw==?= =?utf-8?Q?t?= (GNU Toolchain): elfutils - failed test (failure)) (master) Message-ID: References: <20220423011953.1D7713858430@sourceware.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Apr 2022 20:19:46 -0000 Hi, On Sat, Apr 23, 2022 at 03:31:04AM +0200, Mark Wielaard wrote: > On Sat, Apr 23, 2022 at 01:19:53AM +0000, builder--- via Elfutils-devel wrote: > > A new failure has been detected on builder elfutils-debian-ppc64 while building elfutils. > > > > Full details are available at: > > https://builder.sourceware.org/buildbot/#builders/63/builds/4 > > > > Build state: failed test (failure) > > Revision: 7b046b7c060acc32c00748ee66ac350f77bc6571 > > Worker: debian-ppc64 > > Build Reason: (unknown) > > Blamelist: наб via Elfutils-devel > > > > Steps: > > [...] > > - 7: make check ( failure ) > > Logs: > > - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/stdio > > - test-suite.log: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/test-suite_log > > Gah. It is run-debuginfod-webapi-concurrency.sh again with > error_count{libmicrohttpd="Server reached connection limit. > Closing inbound connection.\n"} 9 > > I guess 100 parallel lookups really is too much. I am going to lower > it to 64. And that didn't help. As Frank already explained in https://sourceware.org/bugzilla/show_bug.cgi?id=28708#c11 The issue isn't the parallel lookups, but the concurrent threadpool. The total number of connections accepted gets divided by the number of threads in the daemon threadpool. So one way to fix this is to lower the max number of -C THREADS:: diff --git a/tests/run-debuginfod-webapi-concurrency.sh b/tests/run-debuginfod-webapi-concurrency.sh index 4928f6d0..12b460a8 100755 --- a/tests/run-debuginfod-webapi-concurrency.sh +++ b/tests/run-debuginfod-webapi-concurrency.sh @@ -33,7 +33,7 @@ cp -rvp ${abs_srcdir}/debuginfod-tars Z tempfiles Z -for Cnum in "" "-C" "-C10" "-C100" +for Cnum in "" "-C" "-C10" "-C32" do env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE $Cnum -d :memory: -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v --fdcache-fds=0 --fdcache-prefetch-fds=0 Z >> vlog$PORT1 2>&1 & PID1=$! This seem to make the test pass even on a system (s390x 2 vcpus, f36) where it occasionally FAILs. But there is another way to prevent the "Server reached connection limit. Closing inbound connection." Pass the MHD_USE_ITC flag to MHD_start_daemon: diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 9c0217f6..67124771 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -3910,6 +3910,7 @@ main (int argc, char *argv[]) | MHD_USE_EPOLL #endif | MHD_USE_DUAL_STACK + | MHD_USE_ITC | MHD_USE_DEBUG, /* report errors to stderr */ http_port, NULL, NULL, /* default accept policy */ This has my preference since it will make sure connections are still accepted (by the OS) instead of being immediately closed once the connection limit is reached: If the connection limit is reached, MHD’s behavior depends a bit on other options. If MHD_USE_ITC was given, MHD will stop accepting connections on the listen socket. This will cause the operating system to queue connections (up to the listen() limit) above the connection limit. Those connections will be held until MHD is done processing at least one of the active connections. If MHD_USE_ITC is not set, then MHD will continue to accept() and immediately close() these connections. But it has some more consequences which I think are OK, but am not 100% sure of: MHD_USE_ITC Force MHD to use a signal inter-thread communication channel to notify the event loop (of threads) of our shutdown and other events. This is required if an application uses MHD_USE_INTERNAL_POLLING_THREAD and then performs MHD_quiesce_daemon (which eliminates our ability to signal termination via the listen socket). In these modes, MHD_quiesce_daemon will fail if this option was not set. Also, use of this option is automatic (as in, you do not even have to specify it), if MHD_USE_NO_LISTEN_SOCKET is specified. In "external" select mode, this option is always simply ignored. Using this option also guarantees that MHD will not call shutdown() on the listen socket, which means a parent process can continue to use the socket. Opinions on just changing the testcasse to use -C32 as max, or to add MHD_USE_ITC to the MHD_start_daemon flags? Cheers, Mark