From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 3B78C3856DE2 for ; Mon, 9 May 2022 23:42:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3B78C3856DE2 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-595-DQugCU4eP1KSnUBfdIIsmQ-1; Mon, 09 May 2022 19:42:54 -0400 X-MC-Unique: DQugCU4eP1KSnUBfdIIsmQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 90D1B8015BA for ; Mon, 9 May 2022 23:42:54 +0000 (UTC) Received: from redhat.com (unknown [10.2.16.15]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 77306C27E89 for ; Mon, 9 May 2022 23:42:54 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.94.2) (envelope-from ) id 1noD1h-0002aG-Mz for elfutils-devel@sourceware.org; Mon, 09 May 2022 19:42:53 -0400 Resent-From: "Frank Ch. Eigler" Resent-Date: Mon, 9 May 2022 19:42:53 -0400 Resent-Message-ID: <20220509234253.GH20072@redhat.com> Resent-To: elfutils-devel@sourceware.org Date: Mon, 9 May 2022 19:40:28 -0400 From: "Frank Ch. Eigler" To: Noah Sanci Cc: elfutils-devel@redhat.com Subject: Re: [Bug debuginfod/29098] set default prefetch limits to >0 Message-ID: <20220509234027.GG20072@redhat.com> References: <20220509183520.GE20072@redhat.com> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_NUMSUBJECT, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2022 23:42:58 -0000 Hi - Thanks, committing this, with corrected comments and changelog entries and a bit of man page cleanup. diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog index 619ebd8c9202..026908c85000 100644 --- a/debuginfod/ChangeLog +++ b/debuginfod/ChangeLog @@ -1,3 +1,7 @@ +2022-05-09 Noah Sanci + + * debuginfod.cxx (main): Set nonzero defaults for fdcache. + 2022-05-04 Frank Ch. Eigler Mark Wielaard diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 4aaf41c0886e..fde4e194b526 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -3826,6 +3826,13 @@ main (int argc, char *argv[]) error (EXIT_FAILURE, 0, "unexpected argument: %s", argv[remaining]); + // Make the prefetch cache spaces a fraction of the main fdcache if + // unspecified. + if (fdcache_prefetch_fds == 0) + fdcache_prefetch_fds = fdcache_fds / 2; + if (fdcache_prefetch_mbs == 0) + fdcache_prefetch_mbs = fdcache_mbs / 2; + if (scan_archives.size()==0 && !scan_files && source_paths.size()>0) obatched(clog) << "warning: without -F -R -U -Z, ignoring PATHs" << endl; diff --git a/doc/ChangeLog b/doc/ChangeLog index 303e3dc05dc5..cb754d04ba3f 100644 --- a/doc/ChangeLog +++ b/doc/ChangeLog @@ -1,3 +1,7 @@ +2022-05-09 Frank Ch. Eigler + + * debuginfod.8: Tweak prefetch descriptions. + 2022-01-31 Frank Ch. Eigler * debuginfod-client-config.7: Elaborate DEBUGINFOD_URLS. diff --git a/doc/debuginfod.8 b/doc/debuginfod.8 index ee8e4078e5b5..95b827e9cc35 100644 --- a/doc/debuginfod.8 +++ b/doc/debuginfod.8 @@ -232,34 +232,36 @@ loops in the symbolic directory tree might lead to \fIinfinite traversal\fP. .TP -.B "\-\-fdcache\-fds=NUM" "\-\-fdcache\-mbs=MB" "\-\-fdcache\-prefetch=NUM2" +.B "\-\-fdcache\-fds=NUM" "\-\-fdcache\-mbs=MB" Configure limits on a cache that keeps recently extracted files from archives. Up to NUM requested files and up to a total of MB megabytes will be kept extracted, in order to avoid having to decompress their -archives over and over again. In addition, up to NUM2 other files -from an archive may be prefetched into the cache before they are even -requested. The default NUM, NUM2, and MB values depend on the -concurrency of the system, and on the available disk space on the +archives over and over again. The default NUM and MB values depend on +the concurrency of the system, and on the available disk space on the $TMPDIR or \fB/tmp\fP filesystem. This is because that is where the -most recently used extracted files are kept. Grooming cleans this +most recently used extracted files are kept. Grooming cleans out this cache. .TP .B "\-\-fdcache\-\-prefetch\-fds=NUM" "\-\-fdcache\-\-prefetch\-mbs=MB" -Configure how many file descriptors (fds) and megabytes (mbs) are -allocated to the prefetch fdcache. If unspecified, values of -\fB\-\-prefetch\-fds\fP and \fB\-\-prefetch\-mbs\fP depend -on concurrency of the system and on the available disk space on -the $TMPDIR. Allocating more to the prefetch cache will improve -performance in environments where different parts of several large -archives are being accessed. +.B "\-\-fdcache\-prefetch=NUM2" + +In addition to the main fdcache, up to NUM2 other files from an +archive may be prefetched into another cache before they are even +requested. Configure how many file descriptors (fds) and megabytes +(mbs) are allocated to the prefetch fdcache. If unspecified, these +values depend on concurrency of the system and on the available disk +space on the $TMPDIR. Allocating more to the prefetch cache will +improve performance in environments where different parts of several +large archives are being accessed. This cache is also cleaned out +during grooming. .TP .B "\-\-fdcache\-mintmp=NUM" -Configure a disk space threshold for emergency flushing of the cache. -The filesystem holding the cache is checked periodically. If the -available space falls below the given percentage, the cache is -flushed, and the fdcache will stay disabled until the next groom +Configure a disk space threshold for emergency flushing of the caches. +The filesystem holding the caches is checked periodically. If the +available space falls below the given percentage, the caches are +flushed, and the fdcaches will stay disabled until the next groom cycle. This mechanism, along a few associated /metrics on the webapi, are intended to give an operator notice about storage scarcity - which can translate to RAM scarcity if the disk happens to be on a RAM