[Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently

[Bug debuginfod/27673] New: [debuginfod] Handle source requests for same buildid more efficiently

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

            Bug ID: 27673
           Summary: [debuginfod] Handle source requests for same buildid
                    more efficiently
           Product: elfutils
           Version: unspecified
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: debuginfod
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
                CC: elfutils-devel at sourceware dot org
  Target Milestone: ---

Consider the following debuginfod usage session from gdb (reported in gdb
PR27667):
...
$ time gdb -batch -q ./mi-sym-info -ex start -ex "interpreter-exec mi
111-symbol-info-functions" > OUTPUT
TRYING DEBUGINFO FOR: /data/gdb_versions/devel/system-supplied DSO at
0x7ffff7fca000
BUILDID: e77a560007595d823e2c5a34697c0b0ae1ffc767
TOOK: 0.000084 seconds
RESULT fd: 16
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/config/i386/sfp-exceptions.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.601374 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.486450 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.493712 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.487850 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.488300 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/libgcc2.c
BUILDID: 7076274fc469c32fa8f4201f467447da3ad2b6be
TOOK: 0.493761 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/addtf3.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.486202 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/divtf3.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.485941 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/eqtf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.477989 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/extenddftf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.487699 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/fixtfdi.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.488412 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/fixtfsi.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.496897 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/floatditf.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.492576 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/floatsitf.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.477747 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/floatunsitf.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.480452 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/getf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.487295 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/letf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.479365 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/multf3.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.486071 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/subtf3.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.478588 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/trunctfdf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.479030 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/trunctfsf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.485941 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/trunctfxf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.493175 seconds
RESULT fd: -2
TRYING SOURCE:
/home/abuild/rpmbuild/BUILD/gcc-10.2.1+git1331/obj-x86_64-suse-linux/x86_64-suse-linux/libgcc/../../../libgcc/soft-fp/unordtf2.c
BUILDID: d34928e1b5c7bde5e269e47349ade7369a3efa83
TOOK: 0.706890 seconds
RESULT fd: -2
TRYING SOURCE:
/usr/src/debug/kernel-default-5.11.6-1.4.x86_64/linux-5.11/linux-obj/../arch/x86/entry/vdso/../../../../lib/vdso/gettimeofday.c
BUILDID: e77a560007595d823e2c5a34697c0b0ae1ffc767
TOOK: 0.000152 seconds
RESULT fd: 17
TRYING SOURCE:
/usr/src/debug/kernel-default-5.11.6-1.4.x86_64/linux-5.11/linux-obj/../arch/x86/entry/vdso/vclock_gettime.c
BUILDID: e77a560007595d823e2c5a34697c0b0ae1ffc767
TOOK: 0.000094 seconds
RESULT fd: 17
TRYING SOURCE:
/usr/src/debug/kernel-default-5.11.6-1.4.x86_64/linux-5.11/linux-obj/../arch/x86/entry/vdso/vgetcpu.c
BUILDID: e77a560007595d823e2c5a34697c0b0ae1ffc767
TOOK: 0.000117 seconds
RESULT fd: 17
TRYING SOURCE:
/usr/src/debug/kernel-default-5.11.6-1.4.x86_64/linux-5.11/linux-obj/../arch/x86/include/asm/vdso/gettimeofday.h
BUILDID: e77a560007595d823e2c5a34697c0b0ae1ffc767
TOOK: 0.000061 seconds
RESULT fd: 17
TRYING SOURCE:
/usr/src/debug/glibc-2.33-4.1.x86_64/io/../sysdeps/unix/sysv/linux/wordsize-64/../fstatvfs.c
BUILDID: 7076274fc469c32fa8f4201f467447da3ad2b6be
TOOK: 0.000068 seconds
RESULT fd: 17
TRYING SOURCE:
/usr/src/debug/glibc-2.33-4.1.x86_64/io/../sysdeps/unix/sysv/linux/wordsize-64/../statvfs.c
BUILDID: 7076274fc469c32fa8f4201f467447da3ad2b6be
TOOK: 0.000054 seconds
RESULT fd: 17
TRYING SOURCE: /home/abuild/rpmbuild/BUILD/glibc-2.33/csu/elf-init.c
BUILDID: 00ef641a70731291619bdc1ae4badda60c853386
TOOK: 0.492776 seconds
RESULT fd: -2

real    0m17.807s
user    0m6.203s
sys     0m0.137s
...

If the results are found (fd >= 0), they are quick because they're in the
cache.

There's a PR open about caching negative results, PR25628. That would help, at
least the second time.

But there's another observation to be made: there's a number of files for which
-2 fd is returned with the same buildid.  If the server would reply to an
ENOENT source request with a list of files contained in the buildid, then the
client would have the information to answer that request, as well as subsequent
requests using the same buildid.  AFAIU, this ties in somewhat with PR27277. 

Taking the idea further, a more bandwith-hungry variant would be to answer any
source request with the source archive, after which any source query for that
buildid can be dealt with by the client.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Tom, can you break down for me what debuginfod or client side change you think
we need, from your gdb trace?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
The subject line "handle sources ... from same buildid ... more efficiently"
suggests a caching opportunity.  The debuginfod --fdcache-prefetch parameter is
probably the gadget of interest here.  The opensuse operators could experiment
with a much larger than default figure.  (You can test it locally by running
debuinfod for yourself, against a local set of relevant -debug* RPMs.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |UNCONFIRMED
     Ever confirmed|1                           |0

--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Frank Ch. Eigler from comment #2)
> The subject line "handle sources ... from same buildid ... more efficiently"
> suggests a caching opportunity.  The debuginfod --fdcache-prefetch parameter
> is probably the gadget of interest here.  The opensuse operators could
> experiment with a much larger than default figure.  (You can test it locally
> by running debuinfod for yourself, against a local set of relevant -debug*
> RPMs.)

The time it takes for the client to see the response of the server to the
request consist of:
- time for request to travel to server (latency)
- time for server to react to request
- time for answer to travel back to client (latency again)

I've looked at the documentation of the option fdcache-prefetch, and AFAIU this
improves "time for server to react to request".

Instead, the focus of this PR is to hide latency (since that's where the
majority of the time is spent) by using more bandwidth.

Proposal a:

debuginfod:
- when receiving a source request and ENOENT, send as reply the list of
  available files for the buildid

client:
- when receiving a list of available files for a buildid, store it and
  use it to reply to source requests related to the buildid. That is,
  if the file is not in the list, reply with -2.  Otherwise, send a
  request to debuginfod, and expect it to succeed.

Proposal b:

debuginfod:
- when receiving a source request, send a package with the sources
  for that buildid to the client.

client:
- when receiving a package with the sources for a buildid, store them
  and use them to reply to source requests related to the buildid.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> ---
> The time it takes for the client to see the response of the server to the
> request consist of:
> - time for request to travel to server (latency)
> - time for server to react to request
> - time for answer to travel back to client (latency again)
> 
> I've looked at the documentation of the option fdcache-prefetch, and AFAIU
> this improves "time for server to react to request".

That's correct.


> debuginfod:
> - when receiving a source request and ENOENT, send as reply the list of
>   available files for the buildid
> 
> client:
> - when receiving a list of available files for a buildid, store it and
>   use it to reply to source requests related to the buildid. That is,
>   if the file is not in the list, reply with -2.  Otherwise, send a
>   request to debuginfod, and expect it to succeed.

Interesting.  A more first-class solution could be a new webapi to
enumerate source files: a "/buildid/HEXCODE/sourcelist" query that
returns a structured piece of data.  This can be computed by debuginfod
fairly rapidly.  The client could cache that and use it to drive a
negative-cache hit on a subsequent source query.


> Proposal b:
> 
> debuginfod:
> - when receiving a source request, send a package with the sources
>   for that buildid to the client.
> 
> client:
> - when receiving a package with the sources for a buildid, store them
>   and use them to reply to source requests related to the buildid.

So this could be a "/buildid/HEXCODE/sources" query that returns a tarball of
all sources related to a given buildid.  This is challenging in principle
because sources may not live in a single upstream package we can just relay
verbatim.  debuginfod may have to assemble a new one on the fly, kind of like
gitweb's 'archive' buttons ... which are disabled by default for performance
reasons.  Worth a consideration I guess, but risky to deploy.


By the way, a client also has another option: querying in parallel.  If it
knows all interesting file names, it can fork N threads and make N concurrent
requests to debuginfod.  The poor server may get larger bursts of load but
total elapsed time should be better.


And another option: if connection establishment / teardown are a bit part of
the problem - and they can be with TLS - we could teach the client code to
activate as much curl level http-keepalive as possible.  So as long as a single
debuginfod_client object were reused, it could avoid the TCP/TLS handshakes. 
(It MIGHT already be doing that.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

--- Comment #5 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Frank Ch. Eigler from comment #4)
> > debuginfod:
> > - when receiving a source request and ENOENT, send as reply the list of
> >   available files for the buildid
> > 
> > client:
> > - when receiving a list of available files for a buildid, store it and
> >   use it to reply to source requests related to the buildid. That is,
> >   if the file is not in the list, reply with -2.  Otherwise, send a
> >   request to debuginfod, and expect it to succeed.
> 
> Interesting.  A more first-class solution could be a new webapi to
> enumerate source files: a "/buildid/HEXCODE/sourcelist" query that
> returns a structured piece of data.  This can be computed by debuginfod
> fairly rapidly.  The client could cache that and use it to drive a
> negative-cache hit on a subsequent source query.

Agreed, it would be good to have this kind of functionality available more
generically.

Then the client (preferably the elfutils client itself using some setting,
otherwise the part using the elfutils client) could upon the first source
request send a sourcelist request in parallel, to have the latencies of the two
requests overlap.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |27678


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=27678
[Bug 27678] [gdb/symtab] Use debuginfod sourcelist api to test for file
existence
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-12-03

--- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> ---
Tom, could you run a test on more current client & server version code?  gdb &
libdebuginfod save latency with connection reuse, and the server saves latency
with aggressive (configurable) archive prefetching.  I'd expect much better
results now, without further webapi/architecture changes.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/27673] [debuginfod] Handle source requests for same buildid more efficiently

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27673

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |WONTFIX

--- Comment #7 from Frank Ch. Eigler <fche at redhat dot com> ---
For now, shelving this idea until a clearer need arises.

-- 
You are receiving this mail because:
You are on the CC list for the bug.