Re: ☠ Buildbot (GNU Toolchain): elfutils - failed test (failure) (master)

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 4288 bytes --]

On Sat, 2022-05-28 at 11:43 +0200, Mark Wielaard wrote:
> On Sat, May 28, 2022 at 11:35:19AM +0200, Mark Wielaard wrote:
> > Retrying with ulimit -c unlimited...
> > 
> > O! It failed again, now with core
> > 
> > (gdb) where
> > #0  0x00007f4d528be387 in __GI_raise (sig=sig@entry=6) at
> > ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> > #1  0x00007f4d528bfa78 in __GI_abort () at abort.c:90
> > #2  0x00007f4d53dfae1f in mhd_panic_std (cls=<optimized out>,
> > file=<optimized out>, 
> >     line=<optimized out>, reason=<optimized out>) at daemon.c:117
> > #3  0x00007f4d53dfc69e in MHD_cleanup_connections (
> > daemon=daemon@entry=0xdf38e0) at daemon.c:1826
> > #4  0x00007f4d53dfdf60 in MHD_run_from_select (daemon=daemon@entry=
> > 0xdf38e0, 
> >     read_fd_set=read_fd_set@entry=0x7f4d4c153c10, 
> > write_fd_set=write_fd_set@entry=0x7f4d4c153c90, 
> >     except_fd_set=except_fd_set@entry=0x7f4d4c153d10) at
> > daemon.c:2014
> > #5  0x00007f4d53dfe1e9 in MHD_select (daemon=daemon@entry=0xdf38e0,
> > may_block=may_block@entry=1)
> >     at daemon.c:2109
> > #6  0x00007f4d53dfe3b2 in MHD_select_thread (cls=0xdf38e0) at
> > daemon.c:2632
> > #7  0x00007f4d53681ea5 in start_thread (arg=0x7f4d4c154700) at
> > pthread_create.c:307
> > #8  0x00007f4d52986b0d in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> > 
> > (gdb) 
> > #3  0x00007f4d53dfc69e in MHD_cleanup_connections (
> > daemon=daemon@entry=0xdf38e0) at daemon.c:1826
> > 1826		    MHD_PANIC ("close failed\n");
> > (gdb) list
> > 1821		{
> > 1822	#ifdef WINDOWS
> > 1823		  SHUTDOWN (pos->socket_fd, SHUT_WR);
> > 1824	#endif
> > 1825		  if (0 != CLOSE (pos->socket_fd))
> > 1826		    MHD_PANIC ("close failed\n");
> > 1827		}
> > 1828	      if (NULL != pos->addr)
> > 1829		free (pos->addr);
> > 1830	      free (pos);
> > 
> > Hohum. That does look like a bug in libmicrohttpd. Or do we
> > manipulate
> > the socket_fd in any way?
> 
> And in case it is useful for analysis:
> 
> (gdb) print *pos
> $2 = {nextE = 0x0, prevE = 0x0, next = 0x0, prev = 0x0, nextX = 0x0,
> prevX = 0x0, daemon = 0xdf38e0, 
>   headers_received = 0x0, headers_received_tail = 0x0, response =
> 0x0, pool = 0x7f4d242f2450, 
>   client_context = 0x0, method = 0x0, url = 0x0, version = 0x0, 
>   read_buffer = 0x7f4d242f2480 "\240\355\v$M\177", write_buffer =
> 0x0, last = 0x0, colon = 0x0, 
>   addr = 0x7f4d242fa490, pid = 139968370415360, read_buffer_size =
> 16384, read_buffer_offset = 0, 
>   write_buffer_size = 0, write_buffer_send_offset = 0,
> write_buffer_append_offset = 0, 
>   remaining_upload_size = 0, response_write_position = 0,
> continue_message_write_offset = 0, 
>   addr_len = 28, last_activity = 599293, connection_timeout = 0,
> client_aware = 0, socket_fd = 96, 
>   read_closed = 1, thread_joined = 0, in_idle = 0, epoll_state =
> MHD_EPOLL_STATE_UNREADY, 
>   state = MHD_CONNECTION_CLOSED, event_loop_info =
> MHD_EVENT_LOOP_INFO_CLEANUP, responseCode = 0, 
>   response_unready = 0, have_chunked_upload = 0, current_chunk_size =
> 0, current_chunk_offset = 0, 
>   read_handler = 0x7f4d53df7680 <MHD_connection_handle_read>, 
>   write_handler = 0x7f4d53df78d0 <MHD_connection_handle_write>, 
>   idle_handler = 0x7f4d53df83a0 <MHD_connection_handle_idle>, 
>   recv_cls = 0x7f4d53dfa030 <recv_param_adapter>, send_cls =
> 0x7f4d53df9ec0 <send_param_adapter>, 
>   tls_session = 0x0, protocol = 0, cipher = 0, tls_read_ready = 0,
> suspended = 0, resuming = 0}
> 
> $ rpm -q libmicrohttpd
> libmicrohttpd-0.9.33-2.el7.x86_64

I tried to debug this a bit. But the code is really different from any
more recent version of libmicrohttpd. Somewhere libmicrohttpd looses
track of the state of the file descriptor and then it just aborts in
cleanup. I couldn't find a way to work around it. The version is so old
that it doesn't actually use any of the modern flags (MHD_USE_EPOLL,
MHD_USE_INTERNAL_POLLING_THREAD and MHD_USE_ITC are all not available).

Even reducing the number of fetches in run-debuginfod-federation-
metrics.sh doesn't prevent it from crashing.

I gave up and just wrote this ugly version based patch to not run the
testcase on such old libmicrohttpd.

Cheers,

Mark


[-- Attachment #2: Type: text/x-patch, Size: 3410 bytes --]

From 8cbadd86c147eeaec4344ac65ad00ea96ae1451a Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Thu, 2 Jun 2022 17:36:39 +0200
Subject: [PATCH] tests: Disable run-debuginfod-federation-metrics.sh for old
 libmicrohttpd

On really old libmicrohttpd the run-debuginfod-federation-metrics.sh
test will crash debuginfod after too many file descriptors have been
used. libmicrohttpd looses track of the state and aborts instead of
producing an error. Just disable the testcase on these very old versions.

Signed-off-by: Mark Wielaard <mark@klomp.org>
---
 ChangeLog         | 5 +++++
 configure.ac      | 2 ++
 tests/ChangeLog   | 5 +++++
 tests/Makefile.am | 6 +++++-
 4 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index f1a14b5c..c5e43e8c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2022-06-02  Mark Wielaard  <mark@klomp.org>
+
+	* configure.ac (OLD_LIBMICROHTTPD): New AM_CONDITIONAL based on
+	libmicrohttpd < 0.9.51.
+
 2022-05-02  Mark Wielaard  <mark@klomp.org>
 
 	* Makefile.am (AM_DISTCHECK_CONFIGURE_FLAGS): Remove
diff --git a/configure.ac b/configure.ac
index 11d1cf82..03b67a9d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -783,6 +783,7 @@ AS_IF([test "x$enable_debuginfod" != "xno"], [
     enable_debuginfod=yes # presume success
     PKG_PROG_PKG_CONFIG
     PKG_CHECK_MODULES([libmicrohttpd],[libmicrohttpd >= 0.9.33],[],[enable_debuginfod=no])
+    PKG_CHECK_MODULES([oldlibmicrohttpd],[libmicrohttpd < 0.9.51],[old_libmicrohttpd=yes],[old_libmicrohttpd=no])
     PKG_CHECK_MODULES([sqlite3],[sqlite3 >= 3.7.17],[],[enable_debuginfod=no])
     PKG_CHECK_MODULES([libarchive],[libarchive >= 3.1.2],[],[enable_debuginfod=no])
     if test "x$enable_debuginfod" = "xno"; then
@@ -792,6 +793,7 @@ AS_IF([test "x$enable_debuginfod" != "xno"], [
 
 AS_IF([test "x$enable_debuginfod" != "xno"],AC_DEFINE([ENABLE_DEBUGINFOD],[1],[Build debuginfod]))
 AM_CONDITIONAL([DEBUGINFOD],[test "x$enable_debuginfod" = "xyes"])
+AM_CONDITIONAL([OLD_LIBMICROHTTPD],[test "x$old_libmicrohttpd" = "xyes"])
 
 dnl for /etc/profile.d/elfutils.{csh,sh}
 default_debuginfod_urls=""
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 44b8df88..fb956925 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2022-06-02  Mark Wielaard  <mark@klomp.org>
+
+	* Makefile.am (TESTS): Add run-debuginfod-federation-metrics.sh
+	only when OLD_LIBMICROHTTPD conditional is not set.
+
 2022-04-24  Mark Wielaard  <mark@klomp.org>
 
 	* run-debuginfod-webapi-concurrency.sh: Fix PR number in xfail.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 84c3950a..d30b07c4 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -238,13 +238,17 @@ TESTS += run-debuginfod-dlopen.sh \
 	 run-debuginfod-archive-test.sh \
 	 run-debuginfod-federation-sqlite.sh \
 	 run-debuginfod-federation-link.sh \
-	 run-debuginfod-federation-metrics.sh \
          run-debuginfod-percent-escape.sh \
          run-debuginfod-x-forwarded-for.sh \
          run-debuginfod-response-headers.sh \
          run-debuginfod-extraction-passive.sh \
 	 run-debuginfod-webapi-concurrency.sh
 endif
+if !OLD_LIBMICROHTTPD
+# Will crash on too old libmicrohttpd
+# Too many open file descriptors confuses libmicrohttpd < 0.9.51
+TESTS += run-debuginfod-federation-metrics.sh
+endif
 endif
 
 EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
-- 
2.18.4