From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id B340F3858D35 for ; Thu, 22 Jun 2023 10:59:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B340F3858D35 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from r6.localdomain (82-217-174-174.cable.dynamic.v4.ziggo.nl [82.217.174.174]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 6C7F33023089; Thu, 22 Jun 2023 12:59:13 +0200 (CEST) Received: by r6.localdomain (Postfix, from userid 1000) id 80B51340162; Thu, 22 Jun 2023 12:59:12 +0200 (CEST) Message-ID: <9a242e2d1fda9a3eb9972278e8c624f95431ab7f.camel@klomp.org> Subject: Re: Performance issue with systemd-coredump and container process linking 2000 shared libraries. From: Mark Wielaard To: "Frank Ch. Eigler" , Romain GEISSLER Cc: "elfutils-devel@sourceware.org" , Francois RIGAULT Date: Thu, 22 Jun 2023 12:59:12 +0200 In-Reply-To: <20230622024044.GI5772@redhat.com> References: <20230614133920.10-1-romain.geissler@amadeus.com> <20230620213701.GN24233@gnu.wildebeest.org> <754003F4-3708-4C9C-AA30-EB76DAECF059@amadeus.com> <20230622024044.GI5772@redhat.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.3 (3.48.3-1.fc38) MIME-Version: 1.0 X-Spam-Status: No, score=-3028.1 required=5.0 tests=BAYES_00,JMQ_SPF_NEUTRAL,KAM_DMARC_STATUS,RCVD_IN_BARRACUDACENTRAL,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi Frank, On Wed, 2023-06-21 at 22:40 -0400, Frank Ch. Eigler wrote: > For an application that processes these elf/dwarf files sequentially, > queries for each synthetic solib are going to result in 2000 https-404 > transactions, sans debuginfod caching.=C2=A0 If you're lucky (reusing a > dwfl object), elfutils may be able to reuse a long-lived https > connection to a server, otherwise a new https connection might have to > be spun up for each.=C2=A0 But even with reuse, we're talking about 2000 > pingponging messages.=C2=A0 That will take a handful of minutes of elapse= d > time just by itself. >=20 > If the calling code made these queries in parallel batches, it would > be much faster overall. I have been thinking about this too. But don't know of a good solution that doesn't negate the (iterative) lazy model that Dwfl uses. libdwfl tries to do the least amount of work possible so that you don't "pay" for looking up extra information for Dwfl_Modules (libraries) you don't care about. So for the use case of getting a backtrace of a particular thread in that core file (Dwfl) you only fetch/load the register notes and stack memory from the core file. Then, only if you translate those stack addresses to symbols, only for those modules associated with those stack addresses it will try to fetch/load the symbol tables, which might involve resolving build-ids to Elf or separate Dwarf debug files. This is normally an iterative process and for something like generating a backtrace it often involves just a handful of Dwfl_Modules (libraries), not all 2000. In this case this falls down a bit since the application creates a Dwfl from a core file and then requests information (the elf file) from all Dwfl_Modules, so it can get at the package note description for each of them. As Romain noted it would be really nice if elfutils/libdwfl had a way to get at the package note description (just like it has for getting the build-id, which is another loaded ELF note). So such a function would know it doesn't need to get the full ELF file. Maybe another solution might be an "get me everything for this Dwfl, all symbol tables, all elf files, all separate Dwarf debug files, etc." function so an application can "pay upfront" for not having to fetch each item lazily? Such a function could then do a "parallel/batched fetch" through debuginfod. Cheers, Mark