* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
@ 2021-03-31 14:58 ` fche at redhat dot com
2021-03-31 15:00 ` fche at redhat dot com
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: fche at redhat dot com @ 2021-03-31 14:58 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |fche at redhat dot com
--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Tom, can you break down for me what debuginfod or client side change you think
we need, from your gdb trace?
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
2021-03-31 14:58 ` [Bug debuginfod/27673] " fche at redhat dot com
@ 2021-03-31 15:00 ` fche at redhat dot com
2021-03-31 15:33 ` vries at gcc dot gnu.org
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: fche at redhat dot com @ 2021-03-31 15:00 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |WAITING
--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
The subject line "handle sources ... from same buildid ... more efficiently"
suggests a caching opportunity. The debuginfod --fdcache-prefetch parameter is
probably the gadget of interest here. The opensuse operators could experiment
with a much larger than default figure. (You can test it locally by running
debuinfod for yourself, against a local set of relevant -debug* RPMs.)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
2021-03-31 14:58 ` [Bug debuginfod/27673] " fche at redhat dot com
2021-03-31 15:00 ` fche at redhat dot com
@ 2021-03-31 15:33 ` vries at gcc dot gnu.org
2021-03-31 16:03 ` fche at redhat dot com
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: vries at gcc dot gnu.org @ 2021-03-31 15:33 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
Tom de Vries <vries at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|WAITING |UNCONFIRMED
Ever confirmed|1 |0
--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Frank Ch. Eigler from comment #2)
> The subject line "handle sources ... from same buildid ... more efficiently"
> suggests a caching opportunity. The debuginfod --fdcache-prefetch parameter
> is probably the gadget of interest here. The opensuse operators could
> experiment with a much larger than default figure. (You can test it locally
> by running debuinfod for yourself, against a local set of relevant -debug*
> RPMs.)
The time it takes for the client to see the response of the server to the
request consist of:
- time for request to travel to server (latency)
- time for server to react to request
- time for answer to travel back to client (latency again)
I've looked at the documentation of the option fdcache-prefetch, and AFAIU this
improves "time for server to react to request".
Instead, the focus of this PR is to hide latency (since that's where the
majority of the time is spent) by using more bandwidth.
Proposal a:
debuginfod:
- when receiving a source request and ENOENT, send as reply the list of
available files for the buildid
client:
- when receiving a list of available files for a buildid, store it and
use it to reply to source requests related to the buildid. That is,
if the file is not in the list, reply with -2. Otherwise, send a
request to debuginfod, and expect it to succeed.
Proposal b:
debuginfod:
- when receiving a source request, send a package with the sources
for that buildid to the client.
client:
- when receiving a package with the sources for a buildid, store them
and use them to reply to source requests related to the buildid.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
` (2 preceding siblings ...)
2021-03-31 15:33 ` vries at gcc dot gnu.org
@ 2021-03-31 16:03 ` fche at redhat dot com
2021-03-31 16:41 ` vries at gcc dot gnu.org
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: fche at redhat dot com @ 2021-03-31 16:03 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> ---
> The time it takes for the client to see the response of the server to the
> request consist of:
> - time for request to travel to server (latency)
> - time for server to react to request
> - time for answer to travel back to client (latency again)
>
> I've looked at the documentation of the option fdcache-prefetch, and AFAIU
> this improves "time for server to react to request".
That's correct.
> debuginfod:
> - when receiving a source request and ENOENT, send as reply the list of
> available files for the buildid
>
> client:
> - when receiving a list of available files for a buildid, store it and
> use it to reply to source requests related to the buildid. That is,
> if the file is not in the list, reply with -2. Otherwise, send a
> request to debuginfod, and expect it to succeed.
Interesting. A more first-class solution could be a new webapi to
enumerate source files: a "/buildid/HEXCODE/sourcelist" query that
returns a structured piece of data. This can be computed by debuginfod
fairly rapidly. The client could cache that and use it to drive a
negative-cache hit on a subsequent source query.
> Proposal b:
>
> debuginfod:
> - when receiving a source request, send a package with the sources
> for that buildid to the client.
>
> client:
> - when receiving a package with the sources for a buildid, store them
> and use them to reply to source requests related to the buildid.
So this could be a "/buildid/HEXCODE/sources" query that returns a tarball of
all sources related to a given buildid. This is challenging in principle
because sources may not live in a single upstream package we can just relay
verbatim. debuginfod may have to assemble a new one on the fly, kind of like
gitweb's 'archive' buttons ... which are disabled by default for performance
reasons. Worth a consideration I guess, but risky to deploy.
By the way, a client also has another option: querying in parallel. If it
knows all interesting file names, it can fork N threads and make N concurrent
requests to debuginfod. The poor server may get larger bursts of load but
total elapsed time should be better.
And another option: if connection establishment / teardown are a bit part of
the problem - and they can be with TLS - we could teach the client code to
activate as much curl level http-keepalive as possible. So as long as a single
debuginfod_client object were reused, it could avoid the TCP/TLS handshakes.
(It MIGHT already be doing that.)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
` (3 preceding siblings ...)
2021-03-31 16:03 ` fche at redhat dot com
@ 2021-03-31 16:41 ` vries at gcc dot gnu.org
2021-04-01 6:06 ` vries at gcc dot gnu.org
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: vries at gcc dot gnu.org @ 2021-03-31 16:41 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
--- Comment #5 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Frank Ch. Eigler from comment #4)
> > debuginfod:
> > - when receiving a source request and ENOENT, send as reply the list of
> > available files for the buildid
> >
> > client:
> > - when receiving a list of available files for a buildid, store it and
> > use it to reply to source requests related to the buildid. That is,
> > if the file is not in the list, reply with -2. Otherwise, send a
> > request to debuginfod, and expect it to succeed.
>
> Interesting. A more first-class solution could be a new webapi to
> enumerate source files: a "/buildid/HEXCODE/sourcelist" query that
> returns a structured piece of data. This can be computed by debuginfod
> fairly rapidly. The client could cache that and use it to drive a
> negative-cache hit on a subsequent source query.
Agreed, it would be good to have this kind of functionality available more
generically.
Then the client (preferably the elfutils client itself using some setting,
otherwise the part using the elfutils client) could upon the first source
request send a sourcelist request in parallel, to have the latencies of the two
requests overlap.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
` (4 preceding siblings ...)
2021-03-31 16:41 ` vries at gcc dot gnu.org
@ 2021-04-01 6:06 ` vries at gcc dot gnu.org
2021-12-03 21:52 ` fche at redhat dot com
2022-01-19 23:50 ` fche at redhat dot com
7 siblings, 0 replies; 9+ messages in thread
From: vries at gcc dot gnu.org @ 2021-04-01 6:06 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
Tom de Vries <vries at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Blocks| |27678
Referenced Bugs:
https://sourceware.org/bugzilla/show_bug.cgi?id=27678
[Bug 27678] [gdb/symtab] Use debuginfod sourcelist api to test for file
existence
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
` (5 preceding siblings ...)
2021-04-01 6:06 ` vries at gcc dot gnu.org
@ 2021-12-03 21:52 ` fche at redhat dot com
2022-01-19 23:50 ` fche at redhat dot com
7 siblings, 0 replies; 9+ messages in thread
From: fche at redhat dot com @ 2021-12-03 21:52 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |WAITING
Ever confirmed|0 |1
Last reconfirmed| |2021-12-03
--- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> ---
Tom, could you run a test on more current client & server version code? gdb &
libdebuginfod save latency with connection reuse, and the server saves latency
with aggressive (configurable) archive prefetching. I'd expect much better
results now, without further webapi/architecture changes.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently
2021-03-31 14:45 [Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently vries at gcc dot gnu.org
` (6 preceding siblings ...)
2021-12-03 21:52 ` fche at redhat dot com
@ 2022-01-19 23:50 ` fche at redhat dot com
7 siblings, 0 replies; 9+ messages in thread
From: fche at redhat dot com @ 2022-01-19 23:50 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=27673
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|WAITING |RESOLVED
Resolution|--- |WONTFIX
--- Comment #7 from Frank Ch. Eigler <fche at redhat dot com> ---
For now, shelving this idea until a clearer need arises.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread