Re: Performance issue with systemd-coredump and container process linking 2000 shared libraries.

[Mark Wielaard <mark@klomp.org>]

Hi Frank,

On Wed, 2023-06-21 at 22:40 -0400, Frank Ch. Eigler wrote:
> For an application that processes these elf/dwarf files sequentially,
> queries for each synthetic solib are going to result in 2000 https-404
> transactions, sans debuginfod caching.  If you're lucky (reusing a
> dwfl object), elfutils may be able to reuse a long-lived https
> connection to a server, otherwise a new https connection might have to
> be spun up for each.  But even with reuse, we're talking about 2000
> pingponging messages.  That will take a handful of minutes of elapsed
> time just by itself.
> 
> If the calling code made these queries in parallel batches, it would
> be much faster overall.

I have been thinking about this too. But don't know of a good solution
that doesn't negate the (iterative) lazy model that Dwfl uses.

libdwfl tries to do the least amount of work possible so that you don't
"pay" for looking up extra information for Dwfl_Modules (libraries) you
don't care about. So for the use case of getting a backtrace of a
particular thread in that core file (Dwfl) you only fetch/load the
register notes and stack memory from the core file. Then, only if you
translate those stack addresses to symbols, only for those modules
associated with those stack addresses it will try to fetch/load the
symbol tables, which might involve resolving build-ids to Elf or
separate Dwarf debug files. This is normally an iterative process and
for something like generating a backtrace it often involves just a
handful of Dwfl_Modules (libraries), not all 2000.

In this case this falls down a bit since the application creates a Dwfl
from a core file and then requests information (the elf file) from all
Dwfl_Modules, so it can get at the package note description for each of
them. As Romain noted it would be really nice if elfutils/libdwfl had a
way to get at the package note description (just like it has for
getting the build-id, which is another loaded ELF note). So such a
function would know it doesn't need to get the full ELF file.

Maybe another solution might be an "get me everything for this Dwfl,
all symbol tables, all elf files, all separate Dwarf debug files, etc."
function so an application can "pay upfront" for not having to fetch
each item lazily? Such a function could then do a "parallel/batched
fetch" through debuginfod.

Cheers,

Mark