[Bug debuginfod/29478] New: Slow query of a debuginfo in a big rpm with very many files

[Bug debuginfod/29478] New: Slow query of a debuginfo in a big rpm with very many files

[mliska at suse dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=29478

            Bug ID: 29478
           Summary: Slow query of a debuginfo in a big rpm with very many
                    files
           Product: elfutils
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: debuginfod
          Assignee: unassigned at sourceware dot org
          Reporter: mliska at suse dot cz
                CC: elfutils-devel at sourceware dot org
  Target Milestone: ---

Consider a very common test case where you debug a binary and the binary
depends on linux-vdso.so.1, well, actually each Linux binary depends on it.

$ file /usr/lib/modules/5.19.0-1-default/vdso/vdso64.so
/usr/lib/modules/5.19.0-1-default/vdso/vdso64.so: ELF 64-bit LSB shared object,
x86-64, version 1 (SYSV), static-pie linked,
BuildID[sha1]=6c5e8eefd4b928a12d9eec149ba8eab49b3febcc, stripped

it belongs to the following debuginfo package:

$ du -hs ~/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm 
1.6G    /home/marxin/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm

and we (openSUSE) use bzip2 compression for it (mainly due to compatibility
with older system where one can install a recent kernel).

Anyway it contains quite many files:

$ rpm -qpl ~/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm | wc -l
15906

and it occupies 5.5 GB when extracted.

$ DEBUGINFOD_VERBOSE=1 gdb `which htop`
GNU gdb (GDB; openSUSE Tumbleweed) 12.1
...
url 0
https://debuginfod.opensuse.org/buildid/6c5e8eefd4b928a12d9eec149ba8eab49b3febcc/debuginfo
query 1 urls in parallel
server response HTTP response code said error
url 0 The requested URL returned error: 504
not found No such file or directory (err=-2)

So as seen, there's a timeout of the request. Apparently, libarchive
uncompresses the content of each entry:

perf confirms that:

    23.14%  libbz2.so.1.0.6   [.] BZ2_bzDecompress
     4.86%  libbz2.so.1.0.6   [.] 0x0000000000015f62
     2.21%  [kernel]          [k] virtnet_poll
     1.86%  [kernel]          [k] __raw_callee_save___pv_queued_spin_unlock

and I can reproduce it locally with:

$ cat /tmp/demo.c
#include <archive.h>
#include <archive_entry.h>
#include <stdlib.h>

int main()
{
  struct archive *a;
  struct archive_entry *entry;
  int r;

  a = archive_read_new();
  archive_read_support_filter_all(a);
  archive_read_support_format_all(a);
  r = archive_read_open_filename(a,
"/home/marxin/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm", 10240);
// Note 1
  if (r != ARCHIVE_OK)
    exit(1);
  while (archive_read_next_header(a, &entry) == ARCHIVE_OK) {
    printf("%s\n",archive_entry_pathname(entry));
    archive_read_data_skip(a);  // Note 2
  }
  r = archive_read_free(a);  // Note 3
  if (r != ARCHIVE_OK)
    exit(1);
}

Note rpm -qpl finishes very quickly:

$ time rpm -qpl ~/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm
>/dev/null

real    0m0.064s
user    0m0.060s
sys     0m0.004s

If I see correctly, the function handle_buildid_r_match skips all entries
different from b_source1, but it's really slow
due to mentioned libarchive slownes.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/29478] Slow query of a debuginfo in a big rpm with very many files

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29478

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |NOTABUG

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Unfortunately, we're not aware of an easy way around this.  There is no way to
seek in the compressed files to an arbitrary index, so one is stuck reading
from the beginning to the file of interest.

I think that, unless the kernel packaging changes, one needs to rely on
(trigger!) debuginfod to fetch & cache highly likely files such as the vdso*. 
Some work is underway to allow this automation, but until then, make sure your
debuginfod server has a large enough fdcache, and you don't groom too
frequently (which clears out the fdcache).

See also https://bugzilla.redhat.com/show_bug.cgi?id=1970578

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/29478] Slow query of a debuginfo in a big rpm with very many files

[mliska at suse dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=29478

--- Comment #2 from Martin Liska <mliska at suse dot cz> ---
Thanks for the reply, after reading of the RH bug, I can see you faced the very
same problem as me ;)

-- 
You are receiving this mail because:
You are on the CC list for the bug.