parallel downloads of multiple debuginfo files

parallel downloads of multiple debuginfo files

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Hey all,

one more debuginfod question: Would it be possible to extend the API to allow 
downloading of multiple debug info files in parallel?

The `debuginfod_find_*` API currently only supports looking at multiple server 
urls in parallel. I would like to ask multiple files in parallel.

The use case would be profiling tools like the ones I'm working on 
(perfparser, heaptrack, ...). There, you usually have a list of libraries that 
you know you'll hit sooner or later. Thus it would be more efficient to start 
downloading all debug information files in parallel directly. At least 
compared to the current status quo with dwfl where we would iteratively find 
individual debug information files, each time blocking the process.

Parallel downloads from a single server would definitely increase the load 
there though, when done naively. Have you thought about adding support for 
something like HTTP/3's ability to download mutiple assets over a single 
connection (maybe even in parallel)?

Thanks
-- 
Milian Wolff
mail@milianw.de
http://milianw.de

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: parallel downloads of multiple debuginfo files

[Frank Ch. Eigler]

Hi -

> one more debuginfod question: Would it be possible to extend the API
> to allow downloading of multiple debug info files in parallel?  The
> `debuginfod_find_*` API currently only supports looking at multiple
> server urls in parallel. I would like to ask multiple files in
> parallel.

Spin up N threads, do one set of debuginfod_begin/_find/_end ops in each.
Bob is your uncle!

- FChE

Re: parallel downloads of multiple debuginfo files

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

On Freitag, 8. April 2022 22:54:35 CEST Frank Ch. Eigler wrote:
> Hi -
> 
> > one more debuginfod question: Would it be possible to extend the API
> > to allow downloading of multiple debug info files in parallel?  The
> > `debuginfod_find_*` API currently only supports looking at multiple
> > server urls in parallel. I would like to ask multiple files in
> > parallel.
> 
> Spin up N threads, do one set of debuginfod_begin/_find/_end ops in each.
> Bob is your uncle!

OK, got it.

But once again - isn't this a problem that everyone using dwfl is going to 
encounter? Should we not try to find a general solution to this problem and 
fix it for all consumers of the API?

Thanks

-- 
Milian Wolff
mail@milianw.de
http://milianw.de

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: parallel downloads of multiple debuginfo files

[Frank Ch. Eigler]

Hi -

> But once again - isn't this a problem that everyone using dwfl is going to 
> encounter? Should we not try to find a general solution to this problem and 
> fix it for all consumers of the API?

I suspect not many apps are going to have a complete list of files
they know they'll need a priori.  A live profiler tends to find out
about target files gradually.  A debugger uses debuginfo to help
enumerate other debuginfo.  DWZ files can only be even known of when
the base DWARF file is fetched.  So yeah I'm not sure there exists a
widespread general problem for which multithreading is not a workable
solution.

- FChE

Re: parallel downloads of multiple debuginfo files

[Mark Wielaard]

Hi,

On Fri, 2022-04-08 at 17:31 -0400, Frank Ch. Eigler via Elfutils-devel
wrote:
> Hi -
> 
> > But once again - isn't this a problem that everyone using dwfl is
> > going to 
> > encounter? Should we not try to find a general solution to this
> > problem and 
> > fix it for all consumers of the API?
> 
> I suspect not many apps are going to have a complete list of files
> they know they'll need a priori.  A live profiler tends to find out
> about target files gradually.  A debugger uses debuginfo to help
> enumerate other debuginfo.  DWZ files can only be even known of when
> the base DWARF file is fetched.

A "two stage" profiler like perf, which first just records build-ids,
addresses and stack dumps, then afterwards does address/symbol/dwarf
resolution and backtracing, can know before the second stage all the
needed debuginfo for all the build-ids recorded. And a debugger can be
"greedy" and try to make sure all debuginfo for a core file or live
process is available up front. I believe gdb for example does try to
generate symbol tables for everything on startup in a background thread
so everything is available quickly once the user is really starting to
debug.

>   So yeah I'm not sure there exists a
> widespread general problem for which multithreading is not a workable
> solution.

That said, I am not sure multithreading is a real solution for the
above scenarios. Doing parallel downloads is often not that efficient.
It is better to reuse the same debuginfo_client handle to download
things serially, that way you can reuse the existing connection instead
of having to set it up for each download separately.

Cheers,

Mark