From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id 5E4F23857348 for ; Thu, 2 Jun 2022 15:45:06 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 5E4F23857348 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from tarox.wildebeest.org (83-87-18-245.cable.dynamic.v4.ziggo.nl [83.87.18.245]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id B1356300BF33; Thu, 2 Jun 2022 17:45:01 +0200 (CEST) Received: by tarox.wildebeest.org (Postfix, from userid 1000) id 00DCA404414C; Thu, 2 Jun 2022 17:44:57 +0200 (CEST) Message-ID: <76b0116e6acf15cb21ee13609f2a773119674e6e.camel@klomp.org> Subject: Re: =?UTF-8?Q?=E2=98=A0?= Buildbot (GNU Toolchain): elfutils - failed test (failure) (master) From: Mark Wielaard To: elfutils-devel@sourceware.org Date: Thu, 02 Jun 2022 17:44:57 +0200 In-Reply-To: References: <20220528091530.AD8583857373@sourceware.org> Content-Type: multipart/mixed; boundary="=-eLGwJVjKyGGjgzNaCUTo" X-Mailer: Evolution 3.28.5 (3.28.5-10.el7) Mime-Version: 1.0 X-Spam-Status: No, score=-11.2 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2022 15:45:09 -0000 --=-eLGwJVjKyGGjgzNaCUTo Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, 2022-05-28 at 11:43 +0200, Mark Wielaard wrote: > On Sat, May 28, 2022 at 11:35:19AM +0200, Mark Wielaard wrote: > > Retrying with ulimit -c unlimited... > >=20 > > O! It failed again, now with core > >=20 > > (gdb) where > > #0 0x00007f4d528be387 in __GI_raise (sig=3Dsig@entry=3D6) at > > ../nptl/sysdeps/unix/sysv/linux/raise.c:55 > > #1 0x00007f4d528bfa78 in __GI_abort () at abort.c:90 > > #2 0x00007f4d53dfae1f in mhd_panic_std (cls=3D, > > file=3D,=20 > > line=3D, reason=3D) at daemon.c:117 > > #3 0x00007f4d53dfc69e in MHD_cleanup_connections ( > > daemon=3Ddaemon@entry=3D0xdf38e0) at daemon.c:1826 > > #4 0x00007f4d53dfdf60 in MHD_run_from_select (daemon=3Ddaemon@entry=3D > > 0xdf38e0,=20 > > read_fd_set=3Dread_fd_set@entry=3D0x7f4d4c153c10,=20 > > write_fd_set=3Dwrite_fd_set@entry=3D0x7f4d4c153c90,=20 > > except_fd_set=3Dexcept_fd_set@entry=3D0x7f4d4c153d10) at > > daemon.c:2014 > > #5 0x00007f4d53dfe1e9 in MHD_select (daemon=3Ddaemon@entry=3D0xdf38e0, > > may_block=3Dmay_block@entry=3D1) > > at daemon.c:2109 > > #6 0x00007f4d53dfe3b2 in MHD_select_thread (cls=3D0xdf38e0) at > > daemon.c:2632 > > #7 0x00007f4d53681ea5 in start_thread (arg=3D0x7f4d4c154700) at > > pthread_create.c:307 > > #8 0x00007f4d52986b0d in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > >=20 > > (gdb)=20 > > #3 0x00007f4d53dfc69e in MHD_cleanup_connections ( > > daemon=3Ddaemon@entry=3D0xdf38e0) at daemon.c:1826 > > 1826 MHD_PANIC ("close failed\n"); > > (gdb) list > > 1821 { > > 1822 #ifdef WINDOWS > > 1823 SHUTDOWN (pos->socket_fd, SHUT_WR); > > 1824 #endif > > 1825 if (0 !=3D CLOSE (pos->socket_fd)) > > 1826 MHD_PANIC ("close failed\n"); > > 1827 } > > 1828 if (NULL !=3D pos->addr) > > 1829 free (pos->addr); > > 1830 free (pos); > >=20 > > Hohum. That does look like a bug in libmicrohttpd. Or do we > > manipulate > > the socket_fd in any way? >=20 > And in case it is useful for analysis: >=20 > (gdb) print *pos > $2 =3D {nextE =3D 0x0, prevE =3D 0x0, next =3D 0x0, prev =3D 0x0, nextX = =3D 0x0, > prevX =3D 0x0, daemon =3D 0xdf38e0,=20 > headers_received =3D 0x0, headers_received_tail =3D 0x0, response =3D > 0x0, pool =3D 0x7f4d242f2450,=20 > client_context =3D 0x0, method =3D 0x0, url =3D 0x0, version =3D 0x0,= =20 > read_buffer =3D 0x7f4d242f2480 "\240\355\v$M\177", write_buffer =3D > 0x0, last =3D 0x0, colon =3D 0x0,=20 > addr =3D 0x7f4d242fa490, pid =3D 139968370415360, read_buffer_size =3D > 16384, read_buffer_offset =3D 0,=20 > write_buffer_size =3D 0, write_buffer_send_offset =3D 0, > write_buffer_append_offset =3D 0,=20 > remaining_upload_size =3D 0, response_write_position =3D 0, > continue_message_write_offset =3D 0,=20 > addr_len =3D 28, last_activity =3D 599293, connection_timeout =3D 0, > client_aware =3D 0, socket_fd =3D 96,=20 > read_closed =3D 1, thread_joined =3D 0, in_idle =3D 0, epoll_state =3D > MHD_EPOLL_STATE_UNREADY,=20 > state =3D MHD_CONNECTION_CLOSED, event_loop_info =3D > MHD_EVENT_LOOP_INFO_CLEANUP, responseCode =3D 0,=20 > response_unready =3D 0, have_chunked_upload =3D 0, current_chunk_size = =3D > 0, current_chunk_offset =3D 0,=20 > read_handler =3D 0x7f4d53df7680 ,=20 > write_handler =3D 0x7f4d53df78d0 ,=20 > idle_handler =3D 0x7f4d53df83a0 ,=20 > recv_cls =3D 0x7f4d53dfa030 , send_cls =3D > 0x7f4d53df9ec0 ,=20 > tls_session =3D 0x0, protocol =3D 0, cipher =3D 0, tls_read_ready =3D 0= , > suspended =3D 0, resuming =3D 0} >=20 > $ rpm -q libmicrohttpd > libmicrohttpd-0.9.33-2.el7.x86_64 I tried to debug this a bit. But the code is really different from any more recent version of libmicrohttpd. Somewhere libmicrohttpd looses track of the state of the file descriptor and then it just aborts in cleanup. I couldn't find a way to work around it. The version is so old that it doesn't actually use any of the modern flags (MHD_USE_EPOLL, MHD_USE_INTERNAL_POLLING_THREAD and MHD_USE_ITC are all not available). Even reducing the number of fetches in run-debuginfod-federation- metrics.sh doesn't prevent it from crashing. I gave up and just wrote this ugly version based patch to not run the testcase on such old libmicrohttpd. Cheers, Mark --=-eLGwJVjKyGGjgzNaCUTo Content-Description: Content-Disposition: inline; filename*0=0001-tests-Disable-run-debuginfod-federation-metrics.sh-f.pat; filename*1=ch Content-Transfer-Encoding: base64 Content-Type: text/x-patch; name="0001-tests-Disable-run-debuginfod-federation-metrics.sh-f.patch"; charset="UTF-8" RnJvbSA4Y2JhZGQ4NmMxNDdlZWFlYzQzNDRhYzY1YWQwMGVhOTZhZTE0NTFhIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXJrIFdpZWxhYXJkIDxtYXJrQGtsb21wLm9yZz4KRGF0ZTog VGh1LCAyIEp1biAyMDIyIDE3OjM2OjM5ICswMjAwClN1YmplY3Q6IFtQQVRDSF0gdGVzdHM6IERp c2FibGUgcnVuLWRlYnVnaW5mb2QtZmVkZXJhdGlvbi1tZXRyaWNzLnNoIGZvciBvbGQKIGxpYm1p Y3JvaHR0cGQKCk9uIHJlYWxseSBvbGQgbGlibWljcm9odHRwZCB0aGUgcnVuLWRlYnVnaW5mb2Qt ZmVkZXJhdGlvbi1tZXRyaWNzLnNoCnRlc3Qgd2lsbCBjcmFzaCBkZWJ1Z2luZm9kIGFmdGVyIHRv byBtYW55IGZpbGUgZGVzY3JpcHRvcnMgaGF2ZSBiZWVuCnVzZWQuIGxpYm1pY3JvaHR0cGQgbG9v c2VzIHRyYWNrIG9mIHRoZSBzdGF0ZSBhbmQgYWJvcnRzIGluc3RlYWQgb2YKcHJvZHVjaW5nIGFu IGVycm9yLiBKdXN0IGRpc2FibGUgdGhlIHRlc3RjYXNlIG9uIHRoZXNlIHZlcnkgb2xkIHZlcnNp b25zLgoKU2lnbmVkLW9mZi1ieTogTWFyayBXaWVsYWFyZCA8bWFya0BrbG9tcC5vcmc+Ci0tLQog Q2hhbmdlTG9nICAgICAgICAgfCA1ICsrKysrCiBjb25maWd1cmUuYWMgICAgICB8IDIgKysKIHRl c3RzL0NoYW5nZUxvZyAgIHwgNSArKysrKwogdGVzdHMvTWFrZWZpbGUuYW0gfCA2ICsrKysrLQog NCBmaWxlcyBjaGFuZ2VkLCAxNyBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9uKC0pCgpkaWZmIC0t Z2l0IGEvQ2hhbmdlTG9nIGIvQ2hhbmdlTG9nCmluZGV4IGYxYTE0YjVjLi5jNWU0M2U4YyAxMDA2 NDQKLS0tIGEvQ2hhbmdlTG9nCisrKyBiL0NoYW5nZUxvZwpAQCAtMSwzICsxLDggQEAKKzIwMjIt MDYtMDIgIE1hcmsgV2llbGFhcmQgIDxtYXJrQGtsb21wLm9yZz4KKworCSogY29uZmlndXJlLmFj IChPTERfTElCTUlDUk9IVFRQRCk6IE5ldyBBTV9DT05ESVRJT05BTCBiYXNlZCBvbgorCWxpYm1p Y3JvaHR0cGQgPCAwLjkuNTEuCisKIDIwMjItMDUtMDIgIE1hcmsgV2llbGFhcmQgIDxtYXJrQGts b21wLm9yZz4KIAogCSogTWFrZWZpbGUuYW0gKEFNX0RJU1RDSEVDS19DT05GSUdVUkVfRkxBR1Mp OiBSZW1vdmUKZGlmZiAtLWdpdCBhL2NvbmZpZ3VyZS5hYyBiL2NvbmZpZ3VyZS5hYwppbmRleCAx MWQxY2Y4Mi4uMDNiNjdhOWQgMTAwNjQ0Ci0tLSBhL2NvbmZpZ3VyZS5hYworKysgYi9jb25maWd1 cmUuYWMKQEAgLTc4Myw2ICs3ODMsNyBAQCBBU19JRihbdGVzdCAieCRlbmFibGVfZGVidWdpbmZv ZCIgIT0gInhubyJdLCBbCiAgICAgZW5hYmxlX2RlYnVnaW5mb2Q9eWVzICMgcHJlc3VtZSBzdWNj ZXNzCiAgICAgUEtHX1BST0dfUEtHX0NPTkZJRwogICAgIFBLR19DSEVDS19NT0RVTEVTKFtsaWJt aWNyb2h0dHBkXSxbbGlibWljcm9odHRwZCA+PSAwLjkuMzNdLFtdLFtlbmFibGVfZGVidWdpbmZv ZD1ub10pCisgICAgUEtHX0NIRUNLX01PRFVMRVMoW29sZGxpYm1pY3JvaHR0cGRdLFtsaWJtaWNy b2h0dHBkIDwgMC45LjUxXSxbb2xkX2xpYm1pY3JvaHR0cGQ9eWVzXSxbb2xkX2xpYm1pY3JvaHR0 cGQ9bm9dKQogICAgIFBLR19DSEVDS19NT0RVTEVTKFtzcWxpdGUzXSxbc3FsaXRlMyA+PSAzLjcu MTddLFtdLFtlbmFibGVfZGVidWdpbmZvZD1ub10pCiAgICAgUEtHX0NIRUNLX01PRFVMRVMoW2xp YmFyY2hpdmVdLFtsaWJhcmNoaXZlID49IDMuMS4yXSxbXSxbZW5hYmxlX2RlYnVnaW5mb2Q9bm9d KQogICAgIGlmIHRlc3QgIngkZW5hYmxlX2RlYnVnaW5mb2QiID0gInhubyI7IHRoZW4KQEAgLTc5 Miw2ICs3OTMsNyBAQCBBU19JRihbdGVzdCAieCRlbmFibGVfZGVidWdpbmZvZCIgIT0gInhubyJd LCBbCiAKIEFTX0lGKFt0ZXN0ICJ4JGVuYWJsZV9kZWJ1Z2luZm9kIiAhPSAieG5vIl0sQUNfREVG SU5FKFtFTkFCTEVfREVCVUdJTkZPRF0sWzFdLFtCdWlsZCBkZWJ1Z2luZm9kXSkpCiBBTV9DT05E SVRJT05BTChbREVCVUdJTkZPRF0sW3Rlc3QgIngkZW5hYmxlX2RlYnVnaW5mb2QiID0gInh5ZXMi XSkKK0FNX0NPTkRJVElPTkFMKFtPTERfTElCTUlDUk9IVFRQRF0sW3Rlc3QgIngkb2xkX2xpYm1p Y3JvaHR0cGQiID0gInh5ZXMiXSkKIAogZG5sIGZvciAvZXRjL3Byb2ZpbGUuZC9lbGZ1dGlscy57 Y3NoLHNofQogZGVmYXVsdF9kZWJ1Z2luZm9kX3VybHM9IiIKZGlmZiAtLWdpdCBhL3Rlc3RzL0No YW5nZUxvZyBiL3Rlc3RzL0NoYW5nZUxvZwppbmRleCA0NGI4ZGY4OC4uZmI5NTY5MjUgMTAwNjQ0 Ci0tLSBhL3Rlc3RzL0NoYW5nZUxvZworKysgYi90ZXN0cy9DaGFuZ2VMb2cKQEAgLTEsMyArMSw4 IEBACisyMDIyLTA2LTAyICBNYXJrIFdpZWxhYXJkICA8bWFya0BrbG9tcC5vcmc+CisKKwkqIE1h a2VmaWxlLmFtIChURVNUUyk6IEFkZCBydW4tZGVidWdpbmZvZC1mZWRlcmF0aW9uLW1ldHJpY3Mu c2gKKwlvbmx5IHdoZW4gT0xEX0xJQk1JQ1JPSFRUUEQgY29uZGl0aW9uYWwgaXMgbm90IHNldC4K KwogMjAyMi0wNC0yNCAgTWFyayBXaWVsYWFyZCAgPG1hcmtAa2xvbXAub3JnPgogCiAJKiBydW4t ZGVidWdpbmZvZC13ZWJhcGktY29uY3VycmVuY3kuc2g6IEZpeCBQUiBudW1iZXIgaW4geGZhaWwu CmRpZmYgLS1naXQgYS90ZXN0cy9NYWtlZmlsZS5hbSBiL3Rlc3RzL01ha2VmaWxlLmFtCmluZGV4 IDg0YzM5NTBhLi5kMzBiMDdjNCAxMDA2NDQKLS0tIGEvdGVzdHMvTWFrZWZpbGUuYW0KKysrIGIv dGVzdHMvTWFrZWZpbGUuYW0KQEAgLTIzOCwxMyArMjM4LDE3IEBAIFRFU1RTICs9IHJ1bi1kZWJ1 Z2luZm9kLWRsb3Blbi5zaCBcCiAJIHJ1bi1kZWJ1Z2luZm9kLWFyY2hpdmUtdGVzdC5zaCBcCiAJ IHJ1bi1kZWJ1Z2luZm9kLWZlZGVyYXRpb24tc3FsaXRlLnNoIFwKIAkgcnVuLWRlYnVnaW5mb2Qt ZmVkZXJhdGlvbi1saW5rLnNoIFwKLQkgcnVuLWRlYnVnaW5mb2QtZmVkZXJhdGlvbi1tZXRyaWNz LnNoIFwKICAgICAgICAgIHJ1bi1kZWJ1Z2luZm9kLXBlcmNlbnQtZXNjYXBlLnNoIFwKICAgICAg ICAgIHJ1bi1kZWJ1Z2luZm9kLXgtZm9yd2FyZGVkLWZvci5zaCBcCiAgICAgICAgICBydW4tZGVi dWdpbmZvZC1yZXNwb25zZS1oZWFkZXJzLnNoIFwKICAgICAgICAgIHJ1bi1kZWJ1Z2luZm9kLWV4 dHJhY3Rpb24tcGFzc2l2ZS5zaCBcCiAJIHJ1bi1kZWJ1Z2luZm9kLXdlYmFwaS1jb25jdXJyZW5j eS5zaAogZW5kaWYKK2lmICFPTERfTElCTUlDUk9IVFRQRAorIyBXaWxsIGNyYXNoIG9uIHRvbyBv bGQgbGlibWljcm9odHRwZAorIyBUb28gbWFueSBvcGVuIGZpbGUgZGVzY3JpcHRvcnMgY29uZnVz ZXMgbGlibWljcm9odHRwZCA8IDAuOS41MQorVEVTVFMgKz0gcnVuLWRlYnVnaW5mb2QtZmVkZXJh dGlvbi1tZXRyaWNzLnNoCitlbmRpZgogZW5kaWYKIAogRVhUUkFfRElTVCA9IHJ1bi1hcmV4dHJh Y3Quc2ggcnVuLWFyc3ltdGVzdC5zaCBydW4tYXIuc2ggXAotLSAKMi4xOC40Cgo= --=-eLGwJVjKyGGjgzNaCUTo--