☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)

☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)

[builder]

A new failure has been detected on builder elfutils-debian-ppc64 while building elfutils.

Full details are available at:
    https://builder.sourceware.org/buildbot/#builders/63/builds/4

Build state: failed test (failure)
Revision: 7b046b7c060acc32c00748ee66ac350f77bc6571
Worker: debian-ppc64
Build Reason: (unknown)
Blamelist: наб via Elfutils-devel <elfutils-devel@sourceware.org>

Steps:

- 0: worker_preparation ( success )

- 1: set package name ( success )

- 2: git checkout ( success )
    Logs:
        - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/2/logs/stdio

- 3: autoreconf ( success )
    Logs:
        - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/3/logs/stdio

- 4: configure ( success )
    Logs:
        - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/4/logs/stdio

- 5: get version ( success )
    Logs:
        - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/5/logs/stdio
        - property changes: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/5/logs/property_changes

- 6: make ( warnings )
    Logs:
        - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/6/logs/stdio
        - warnings (3): https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/6/logs/warnings__3_

- 7: make check ( failure )
    Logs:
        - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/stdio
        - test-suite.log: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/test-suite_log

Re: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On Sat, Apr 23, 2022 at 01:19:53AM +0000, builder--- via Elfutils-devel wrote:
> A new failure has been detected on builder elfutils-debian-ppc64 while building elfutils.
> 
> Full details are available at:
>     https://builder.sourceware.org/buildbot/#builders/63/builds/4
> 
> Build state: failed test (failure)
> Revision: 7b046b7c060acc32c00748ee66ac350f77bc6571
> Worker: debian-ppc64
> Build Reason: (unknown)
> Blamelist: наб via Elfutils-devel <elfutils-devel@sourceware.org>
> 
> Steps:
> [...]
> - 7: make check ( failure )
>     Logs:
>         - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/stdio
>         - test-suite.log: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/test-suite_log

Gah. It is run-debuginfod-webapi-concurrency.sh again with
error_count{libmicrohttpd="Server reached connection limit.
Closing inbound connection.\n"} 9

I guess 100 parallel lookups really is too much. I am going to lower
it to 64.

Cheers,

Mark

[-- Attachment #2: 0001-tests-Lower-parallel-lookups-in-run-debuginfod-webap.patch --]
[-- Type: text/x-diff, Size: 1851 bytes --]

From 318807e7f968fd70b80408e3df029c04365c47d8 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 23 Apr 2022 03:20:17 +0200
Subject: [PATCH] tests: Lower parallel lookups in
 run-debuginfod-webapi-concurrency.sh

With 100 parallel lookups we sometimes see:
Server reached connection limit. Closing inbound connection.

Lower parallel lookups to 64

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 tests/ChangeLog                            | 4 ++++
 tests/run-debuginfod-webapi-concurrency.sh | 7 ++++---
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/tests/ChangeLog b/tests/ChangeLog
index c734b260..2286f53f 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,7 @@
+2022-04-23  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-webapi-concurrency.sh: Lower parallel lookups.
+
 2022-03-01  Di Chen <dichen@redhat.com>
 
 	* alldts.c (dtflags): Put DT_NULL last.
diff --git a/tests/run-debuginfod-webapi-concurrency.sh b/tests/run-debuginfod-webapi-concurrency.sh
index 402a6307..4928f6d0 100755
--- a/tests/run-debuginfod-webapi-concurrency.sh
+++ b/tests/run-debuginfod-webapi-concurrency.sh
@@ -47,12 +47,13 @@ do
     wait_ready $PORT1 'thread_busy{role="scan"}' 0
 
     # Do a bunch of lookups in parallel
-    for jobs in `seq 100`; do
+    lookup_nr=64
+    for jobs in `seq $lookup_nr`; do
         curl -s http://localhost:$PORT1/buildid/cee13b2ea505a7f37bd20d271c6bc7e5f8d2dfcb/debuginfo > /dev/null &
     done
 
-    # all 100 curls should succeed
-    wait_ready $PORT1 'http_responses_transfer_bytes_count{code="200",type="debuginfo"}' 100
+    # all curls should succeed
+    wait_ready $PORT1 'http_responses_transfer_bytes_count{code="200",type="debuginfo"}' $lookup_nr
     
     (sleep 5;
      curl -s http://localhost:$PORT1/metrics | egrep 'error|responses';
-- 
2.30.2

run-debuginfod-webapi-concurrency.sh (Was: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure)) (master)

[Mark Wielaard]

Hi,

On Sat, Apr 23, 2022 at 03:31:04AM +0200, Mark Wielaard wrote:
> On Sat, Apr 23, 2022 at 01:19:53AM +0000, builder--- via Elfutils-devel wrote:
> > A new failure has been detected on builder elfutils-debian-ppc64 while building elfutils.
> > 
> > Full details are available at:
> >     https://builder.sourceware.org/buildbot/#builders/63/builds/4
> > 
> > Build state: failed test (failure)
> > Revision: 7b046b7c060acc32c00748ee66ac350f77bc6571
> > Worker: debian-ppc64
> > Build Reason: (unknown)
> > Blamelist: наб via Elfutils-devel <elfutils-devel@sourceware.org>
> > 
> > Steps:
> > [...]
> > - 7: make check ( failure )
> >     Logs:
> >         - stdio: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/stdio
> >         - test-suite.log: https://builder.sourceware.org/buildbot/#builders/63/builds/4/steps/7/logs/test-suite_log
> 
> Gah. It is run-debuginfod-webapi-concurrency.sh again with
> error_count{libmicrohttpd="Server reached connection limit.
> Closing inbound connection.\n"} 9
> 
> I guess 100 parallel lookups really is too much. I am going to lower
> it to 64.

And that didn't help. As Frank already explained in
https://sourceware.org/bugzilla/show_bug.cgi?id=28708#c11

The issue isn't the parallel lookups, but the concurrent threadpool.
The total number of connections accepted gets divided by the number of
threads in the daemon threadpool.

So one way to fix this is to lower the max number of -C THREADS::

diff --git a/tests/run-debuginfod-webapi-concurrency.sh b/tests/run-debuginfod-webapi-concurrency.sh
index 4928f6d0..12b460a8 100755
--- a/tests/run-debuginfod-webapi-concurrency.sh
+++ b/tests/run-debuginfod-webapi-concurrency.sh
@@ -33,7 +33,7 @@ cp -rvp ${abs_srcdir}/debuginfod-tars Z
 tempfiles Z
 
 
-for Cnum in "" "-C" "-C10" "-C100"
+for Cnum in "" "-C" "-C10" "-C32"
 do
     env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../debuginfod/debuginfod $VERBOSE $Cnum -d :memory: -Z .tar.xz -Z .tar.bz2=bzcat -p $PORT1 -t0 -g0 -v --fdcache-fds=0 --fdcache-prefetch-fds=0 Z >> vlog$PORT1 2>&1 &
     PID1=$!

This seem to make the test pass even on a system (s390x 2 vcpus, f36)
where it occasionally FAILs.

But there is another way to prevent the "Server reached connection
limit. Closing inbound connection." Pass the MHD_USE_ITC flag to
MHD_start_daemon:

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 9c0217f6..67124771 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3910,6 +3910,7 @@ main (int argc, char *argv[])
                                      | MHD_USE_EPOLL
 #endif
                                      | MHD_USE_DUAL_STACK
+                                     | MHD_USE_ITC
                                      | MHD_USE_DEBUG, /* report errors to stderr */
                                      http_port,
                                      NULL, NULL, /* default accept policy */

This has my preference since it will make sure connections are still
accepted (by the OS) instead of being immediately closed once the
connection limit is reached:

    If the connection limit is reached, MHD’s behavior depends a bit
    on other options. If MHD_USE_ITC was given, MHD will stop
    accepting connections on the listen socket. This will cause the
    operating system to queue connections (up to the listen() limit)
    above the connection limit. Those connections will be held until
    MHD is done processing at least one of the active connections. If
    MHD_USE_ITC is not set, then MHD will continue to accept() and
    immediately close() these connections.

But it has some more consequences which I think are OK, but am not
100% sure of:

MHD_USE_ITC

    Force MHD to use a signal inter-thread communication channel to
    notify the event loop (of threads) of our shutdown and other
    events. This is required if an application uses
    MHD_USE_INTERNAL_POLLING_THREAD and then performs
    MHD_quiesce_daemon (which eliminates our ability to signal
    termination via the listen socket). In these modes,
    MHD_quiesce_daemon will fail if this option was not set. Also, use
    of this option is automatic (as in, you do not even have to
    specify it), if MHD_USE_NO_LISTEN_SOCKET is specified. In
    "external" select mode, this option is always simply ignored.

    Using this option also guarantees that MHD will not call
    shutdown() on the listen socket, which means a parent process can
    continue to use the socket.

Opinions on just changing the testcasse to use -C32 as max, or to add
MHD_USE_ITC to the MHD_start_daemon flags?

Cheers,

Mark

Re: run-debuginfod-webapi-concurrency.sh (Was: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure)) (master)

[Frank Ch. Eigler]

Hi -

> But there is another way to prevent the "Server reached connection
> limit. Closing inbound connection." Pass the MHD_USE_ITC flag to
> MHD_start_daemon:

Yeah, that looked promising to me too.  When I was last working on
this, that would have been my next thing to try.  I can't think of
a relevant downside, so let's try it.  (Add a #ifdef guard around
that macro, for older libmicrohttpd, like rhel7 methinks.)

- FChE

Re: run-debuginfod-webapi-concurrency.sh

[Thomas Fitzsimmons]

Hi,

"Frank Ch. Eigler" <fche@redhat.com> writes:

>> But there is another way to prevent the "Server reached connection
>> limit. Closing inbound connection." Pass the MHD_USE_ITC flag to
>> MHD_start_daemon:
>
> Yeah, that looked promising to me too.  When I was last working on
> this, that would have been my next thing to try.  I can't think of
> a relevant downside, so let's try it.  (Add a #ifdef guard around
> that macro, for older libmicrohttpd, like rhel7 methinks.)

On debian-ppc64, with and without the MHD_USE_ITC patch, I ran the test
20 times in a shell loop.  With MHD_USE_ITC, I got 20 passes, without
it, 9 passes and 11 failures.

With the patch applied, a full "make check" succeeded.

Thomas

Re: run-debuginfod-webapi-concurrency.sh

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Hi,

On Sat, Apr 23, 2022 at 10:56:01PM -0400, Thomas Fitzsimmons wrote:
> "Frank Ch. Eigler" <fche@redhat.com> writes:
> >> But there is another way to prevent the "Server reached connection
> >> limit. Closing inbound connection." Pass the MHD_USE_ITC flag to
> >> MHD_start_daemon:
> >
> > Yeah, that looked promising to me too.  When I was last working on
> > this, that would have been my next thing to try.  I can't think of
> > a relevant downside, so let's try it.  (Add a #ifdef guard around
> > that macro, for older libmicrohttpd, like rhel7 methinks.)
> 
> On debian-ppc64, with and without the MHD_USE_ITC patch, I ran the test
> 20 times in a shell loop.  With MHD_USE_ITC, I got 20 passes, without
> it, 9 passes and 11 failures.
> 
> With the patch applied, a full "make check" succeeded.

Nice. Thanks for the feedback and testing. I pushed the attached.

Cheers,

Mark

[-- Attachment #2: 0001-debuginfod-Use-MHD_USE_ITC-in-MHD_start_daemon-flags.patch --]
[-- Type: text/x-diff, Size: 3290 bytes --]

From 3bcf887340fd47d0d8a3671cc45abe2989d1fd6c Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sun, 24 Apr 2022 12:16:58 +0200
Subject: [PATCH] debuginfod: Use MHD_USE_ITC in MHD_start_daemon flags
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This prevents the "Server reached connection limit. Closing inbound
connection." issue we have been seeing in the
run-debuginfod-webapi-concurrency.sh testcase. From the manual:

    If the connection limit is reached, MHD’s behavior depends a bit
    on other options. If MHD_USE_ITC was given, MHD will stop
    accepting connections on the listen socket. This will cause the
    operating system to queue connections (up to the listen() limit)
    above the connection limit. Those connections will be held until
    MHD is done processing at least one of the active connections. If
    MHD_USE_ITC is not set, then MHD will continue to accept() and
    immediately close() these connections.

https://sourceware.org/bugzilla/show_bug.cgi?id=28708

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 debuginfod/ChangeLog                       | 4 ++++
 debuginfod/debuginfod.cxx                  | 3 +++
 tests/ChangeLog                            | 4 ++++
 tests/run-debuginfod-webapi-concurrency.sh | 4 +++-
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index d6f7b282..0f1bca6f 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,7 @@
+2022-04-24  Mark Wielaard  <mark@klomp.org>
+
+	* debuginfod.cxx (main): Add MHD_USE_ITC to MHD_start_daemon flags.
+
 2022-04-13  Aaron Merey  <amerey@redhat.com>
 
 	* debuginfod-client.c (debuginfod_query_server):
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 9c0217f6..adca8208 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3910,6 +3910,9 @@ main (int argc, char *argv[])
                                      | MHD_USE_EPOLL
 #endif
                                      | MHD_USE_DUAL_STACK
+#if MHD_VERSION >= 0x00095200
+                                     | MHD_USE_ITC
+#endif
                                      | MHD_USE_DEBUG, /* report errors to stderr */
                                      http_port,
                                      NULL, NULL, /* default accept policy */
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 2286f53f..44b8df88 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,7 @@
+2022-04-24  Mark Wielaard  <mark@klomp.org>
+
+	* run-debuginfod-webapi-concurrency.sh: Fix PR number in xfail.
+
 2022-04-23  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-webapi-concurrency.sh: Lower parallel lookups.
diff --git a/tests/run-debuginfod-webapi-concurrency.sh b/tests/run-debuginfod-webapi-concurrency.sh
index 4928f6d0..47dcadcc 100755
--- a/tests/run-debuginfod-webapi-concurrency.sh
+++ b/tests/run-debuginfod-webapi-concurrency.sh
@@ -62,6 +62,8 @@ do
     PID1=0
 done
 
-xfail "grep Server.reached.connection vlog$PORT1" # PR18661
+# Note this xfail comes too late, the above wait_ready for
+# http_responses_transfer_bytes_count will have failed.
+xfail "grep Server.reached.connection vlog$PORT1" # PR28661
 
 exit 0
-- 
2.30.2

Re: run-debuginfod-webapi-concurrency.sh

[Frank Ch. Eigler]

Hi -

> [...]
> I think that might be Frank's local experimentation. I also got email
> about my workers being unavailable. If you run a buildbot locally it
> will not see any workers connect and after an hour it will try to
> notify the owners.

Sorry about that.  After this noisy testing, I pushed to the
sourceware builder a configuration change for elfutils builds that
automatically upload testsuite results (both make check and make
distcheck) into the local bunsen testsuite database for future
analysis.

- FChE