[Bug debuginfod/29098] set default prefetch limits to >0

[Bug debuginfod/29098] set default prefetch limits to >0

[Noah Sanci]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

debuginfod/debuginfod.cxx:
- Added default value to fdcache_prefetch_mds and
  fdcache_prefetch_fds when -R is specified. Defaults   to one half of
  fdcache's values for those respective   variables. These values are only set
  when -R is specified

Note: the ratio of prefetch:fdcache file descriptors and MBs
          is set to 1:2 since the fdcache seems to be used less
          than the standard fdcache.

[-- Attachment #2: 0001-PR29098-debuginfod-set-default-prefetch-limits-to-0.patch --]
[-- Type: text/x-patch, Size: 1531 bytes --]

From caa420f97225c5cc31c3fa0650cc7e25af3b4e91 Mon Sep 17 00:00:00 2001
From: Noah Sanci <nsanci@redhat.com>
Date: Mon, 9 May 2022 13:15:04 -0400
Subject: [PATCH] PR29098: debuginfod - set default prefetch limits to >0
 debuginfod/debuginfod.cxx: - Added default value to fdcache_prefetch_mds and 
  fdcache_prefetch_fds when -R is specified. Defaults   to one half of
 fdcache's values for those respective   variables. These values are only set
 when -R is specified

Signed-off-by: Noah Sanci <nsanci@redhat.com>
---
 debuginfod/debuginfod.cxx | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 4aaf41c0..b2fb2afb 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3826,6 +3826,18 @@ main (int argc, char *argv[])
       error (EXIT_FAILURE, 0,
              "unexpected argument: %s", argv[remaining]);
 
+  // Make the prefetch cache spaces smaller than the normal
+  // fd cache if rpm scanning is on. This is to not waste memory
+  // since the prefetch cache isn't used when -R isn't specified
+  // Set to 1/2 arbitrarily
+  if ( scan_archives[".rpm"] == "cat" )
+    {
+      if ( fdcache_prefetch_fds == 0 )
+        fdcache_prefetch_fds = fdcache_fds * .5;
+      if ( fdcache_prefetch_mbs == 0 )
+        fdcache_prefetch_mbs = fdcache_mbs * .5;
+    }
+
   if (scan_archives.size()==0 && !scan_files && source_paths.size()>0)
     obatched(clog) << "warning: without -F -R -U -Z, ignoring PATHs" << endl;
 
-- 
2.31.1

Re: [Bug debuginfod/29098] set default prefetch limits to >0

[Frank Ch. Eigler]

Hi -

> +  // Make the prefetch cache spaces smaller than the normal
> +  // fd cache if rpm scanning is on. This is to not waste memory
> +  // since the prefetch cache isn't used when -R isn't specified
> +  // Set to 1/2 arbitrarily
> +  if ( scan_archives[".rpm"] == "cat" )
> +    {
> +      if ( fdcache_prefetch_fds == 0 )
> +        fdcache_prefetch_fds = fdcache_fds * .5;
> +      if ( fdcache_prefetch_mbs == 0 )
> +        fdcache_prefetch_mbs = fdcache_mbs * .5;
> +    }

Yeah, something like that.  It turns out that space for the prefetch
cache is not used at all if RPM (and don't forget about other archive
types!) is not in effect.  So it's harmless to set those defaults
unconditionally.  How about a simpler:

     if ( fdcache_prefetch_fds == 0 )
       fdcache_prefetch_fds = fdcache_fds * .5;
     if ( fdcache_prefetch_mbs == 0 )
       fdcache_prefetch_mbs = fdcache_mbs * .5;

- FChE

Re: [Bug debuginfod/29098] set default prefetch limits to >0

[Frank Ch. Eigler]

Hi -

Thanks, committing this, with corrected comments and changelog entries
and a bit of man page cleanup.


diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 619ebd8c9202..026908c85000 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,7 @@
+2022-05-09  Noah Sanci  <nsanci@redhat.com>
+
+	* debuginfod.cxx (main): Set nonzero defaults for fdcache.
+
 2022-05-04  Frank Ch. Eigler <fche@redhat.com>
 	    Mark Wielaard  <mark@klomp.org>
 
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 4aaf41c0886e..fde4e194b526 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3826,6 +3826,13 @@ main (int argc, char *argv[])
       error (EXIT_FAILURE, 0,
              "unexpected argument: %s", argv[remaining]);
 
+  // Make the prefetch cache spaces a fraction of the main fdcache if
+  // unspecified.
+  if (fdcache_prefetch_fds == 0)
+    fdcache_prefetch_fds = fdcache_fds / 2;
+  if (fdcache_prefetch_mbs == 0)
+    fdcache_prefetch_mbs = fdcache_mbs / 2;
+
   if (scan_archives.size()==0 && !scan_files && source_paths.size()>0)
     obatched(clog) << "warning: without -F -R -U -Z, ignoring PATHs" << endl;
 
diff --git a/doc/ChangeLog b/doc/ChangeLog
index 303e3dc05dc5..cb754d04ba3f 100644
--- a/doc/ChangeLog
+++ b/doc/ChangeLog
@@ -1,3 +1,7 @@
+2022-05-09  Frank Ch. Eigler  <fche@redhat.com>
+
+	* debuginfod.8: Tweak prefetch descriptions.
+
 2022-01-31  Frank Ch. Eigler  <fche@redhat.com>
 
 	* debuginfod-client-config.7: Elaborate DEBUGINFOD_URLS.
diff --git a/doc/debuginfod.8 b/doc/debuginfod.8
index ee8e4078e5b5..95b827e9cc35 100644
--- a/doc/debuginfod.8
+++ b/doc/debuginfod.8
@@ -232,34 +232,36 @@ loops in the symbolic directory tree might lead to \fIinfinite
 traversal\fP.
 
 .TP
-.B "\-\-fdcache\-fds=NUM"  "\-\-fdcache\-mbs=MB"  "\-\-fdcache\-prefetch=NUM2"
+.B "\-\-fdcache\-fds=NUM"  "\-\-fdcache\-mbs=MB"
 Configure limits on a cache that keeps recently extracted files from
 archives.  Up to NUM requested files and up to a total of MB megabytes
 will be kept extracted, in order to avoid having to decompress their
-archives over and over again.  In addition, up to NUM2 other files
-from an archive may be prefetched into the cache before they are even
-requested.  The default NUM, NUM2, and MB values depend on the
-concurrency of the system, and on the available disk space on the
+archives over and over again. The default NUM and MB values depend on
+the concurrency of the system, and on the available disk space on the
 $TMPDIR or \fB/tmp\fP filesystem.  This is because that is where the
-most recently used extracted files are kept.  Grooming cleans this
+most recently used extracted files are kept.  Grooming cleans out this
 cache.
 
 .TP
 .B "\-\-fdcache\-\-prefetch\-fds=NUM"  "\-\-fdcache\-\-prefetch\-mbs=MB"
-Configure how many file descriptors (fds) and megabytes (mbs) are
-allocated to the prefetch fdcache. If unspecified, values of
-\fB\-\-prefetch\-fds\fP and \fB\-\-prefetch\-mbs\fP depend
-on concurrency of the system and on the available disk space on
-the $TMPDIR. Allocating more to the prefetch cache will improve
-performance in environments where different parts of several large
-archives are being accessed.
+.B "\-\-fdcache\-prefetch=NUM2"
+
+In addition to the main fdcache, up to NUM2 other files from an
+archive may be prefetched into another cache before they are even
+requested.  Configure how many file descriptors (fds) and megabytes
+(mbs) are allocated to the prefetch fdcache. If unspecified, these
+values depend on concurrency of the system and on the available disk
+space on the $TMPDIR.  Allocating more to the prefetch cache will
+improve performance in environments where different parts of several
+large archives are being accessed.  This cache is also cleaned out
+during grooming.
 
 .TP
 .B "\-\-fdcache\-mintmp=NUM"
-Configure a disk space threshold for emergency flushing of the cache.
-The filesystem holding the cache is checked periodically.  If the
-available space falls below the given percentage, the cache is
-flushed, and the fdcache will stay disabled until the next groom
+Configure a disk space threshold for emergency flushing of the caches.
+The filesystem holding the caches is checked periodically.  If the
+available space falls below the given percentage, the caches are
+flushed, and the fdcaches will stay disabled until the next groom
 cycle.  This mechanism, along a few associated /metrics on the webapi,
 are intended to give an operator notice about storage scarcity - which
 can translate to RAM scarcity if the disk happens to be on a RAM