On Sat, 2022-05-28 at 11:43 +0200, Mark Wielaard wrote:
> On Sat, May 28, 2022 at 11:35:19AM +0200, Mark Wielaard wrote:
> > Retrying with ulimit -c unlimited...
> > 
> > O! It failed again, now with core
> > 
> > (gdb) where
> > #0  0x00007f4d528be387 in __GI_raise (sig=sig@entry=6) at
> > ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> > #1  0x00007f4d528bfa78 in __GI_abort () at abort.c:90
> > #2  0x00007f4d53dfae1f in mhd_panic_std (cls=<optimized out>,
> > file=<optimized out>, 
> >     line=<optimized out>, reason=<optimized out>) at daemon.c:117
> > #3  0x00007f4d53dfc69e in MHD_cleanup_connections (
> > daemon=daemon@entry=0xdf38e0) at daemon.c:1826
> > #4  0x00007f4d53dfdf60 in MHD_run_from_select (daemon=daemon@entry=
> > 0xdf38e0, 
> >     read_fd_set=read_fd_set@entry=0x7f4d4c153c10, 
> > write_fd_set=write_fd_set@entry=0x7f4d4c153c90, 
> >     except_fd_set=except_fd_set@entry=0x7f4d4c153d10) at
> > daemon.c:2014
> > #5  0x00007f4d53dfe1e9 in MHD_select (daemon=daemon@entry=0xdf38e0,
> > may_block=may_block@entry=1)
> >     at daemon.c:2109
> > #6  0x00007f4d53dfe3b2 in MHD_select_thread (cls=0xdf38e0) at
> > daemon.c:2632
> > #7  0x00007f4d53681ea5 in start_thread (arg=0x7f4d4c154700) at
> > pthread_create.c:307
> > #8  0x00007f4d52986b0d in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> > 
> > (gdb) 
> > #3  0x00007f4d53dfc69e in MHD_cleanup_connections (
> > daemon=daemon@entry=0xdf38e0) at daemon.c:1826
> > 1826		    MHD_PANIC ("close failed\n");
> > (gdb) list
> > 1821		{
> > 1822	#ifdef WINDOWS
> > 1823		  SHUTDOWN (pos->socket_fd, SHUT_WR);
> > 1824	#endif
> > 1825		  if (0 != CLOSE (pos->socket_fd))
> > 1826		    MHD_PANIC ("close failed\n");
> > 1827		}
> > 1828	      if (NULL != pos->addr)
> > 1829		free (pos->addr);
> > 1830	      free (pos);
> > 
> > Hohum. That does look like a bug in libmicrohttpd. Or do we
> > manipulate
> > the socket_fd in any way?
> 
> And in case it is useful for analysis:
> 
> (gdb) print *pos
> $2 = {nextE = 0x0, prevE = 0x0, next = 0x0, prev = 0x0, nextX = 0x0,
> prevX = 0x0, daemon = 0xdf38e0, 
>   headers_received = 0x0, headers_received_tail = 0x0, response =
> 0x0, pool = 0x7f4d242f2450, 
>   client_context = 0x0, method = 0x0, url = 0x0, version = 0x0, 
>   read_buffer = 0x7f4d242f2480 "\240\355\v$M\177", write_buffer =
> 0x0, last = 0x0, colon = 0x0, 
>   addr = 0x7f4d242fa490, pid = 139968370415360, read_buffer_size =
> 16384, read_buffer_offset = 0, 
>   write_buffer_size = 0, write_buffer_send_offset = 0,
> write_buffer_append_offset = 0, 
>   remaining_upload_size = 0, response_write_position = 0,
> continue_message_write_offset = 0, 
>   addr_len = 28, last_activity = 599293, connection_timeout = 0,
> client_aware = 0, socket_fd = 96, 
>   read_closed = 1, thread_joined = 0, in_idle = 0, epoll_state =
> MHD_EPOLL_STATE_UNREADY, 
>   state = MHD_CONNECTION_CLOSED, event_loop_info =
> MHD_EVENT_LOOP_INFO_CLEANUP, responseCode = 0, 
>   response_unready = 0, have_chunked_upload = 0, current_chunk_size =
> 0, current_chunk_offset = 0, 
>   read_handler = 0x7f4d53df7680 <MHD_connection_handle_read>, 
>   write_handler = 0x7f4d53df78d0 <MHD_connection_handle_write>, 
>   idle_handler = 0x7f4d53df83a0 <MHD_connection_handle_idle>, 
>   recv_cls = 0x7f4d53dfa030 <recv_param_adapter>, send_cls =
> 0x7f4d53df9ec0 <send_param_adapter>, 
>   tls_session = 0x0, protocol = 0, cipher = 0, tls_read_ready = 0,
> suspended = 0, resuming = 0}
> 
> $ rpm -q libmicrohttpd
> libmicrohttpd-0.9.33-2.el7.x86_64

I tried to debug this a bit. But the code is really different from any
more recent version of libmicrohttpd. Somewhere libmicrohttpd looses
track of the state of the file descriptor and then it just aborts in
cleanup. I couldn't find a way to work around it. The version is so old
that it doesn't actually use any of the modern flags (MHD_USE_EPOLL,
MHD_USE_INTERNAL_POLLING_THREAD and MHD_USE_ITC are all not available).

Even reducing the number of fetches in run-debuginfod-federation-
metrics.sh doesn't prevent it from crashing.

I gave up and just wrote this ugly version based patch to not run the
testcase on such old libmicrohttpd.

Cheers,

Mark