* [Bug debuginfod/28514] New: limit grooming time for many stale files
@ 2021-10-28 19:06 fche at redhat dot com
2021-11-05 12:17 ` [Bug debuginfod/28514] " fche at redhat dot com
0 siblings, 1 reply; 2+ messages in thread
From: fche at redhat dot com @ 2021-10-28 19:06 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=28514
Bug ID: 28514
Summary: limit grooming time for many stale files
Product: elfutils
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: debuginfod
Assignee: unassigned at sourceware dot org
Reporter: fche at redhat dot com
CC: elfutils-devel at sourceware dot org
Target Milestone: ---
When a big debuginfod server starts grooming, and starts finding stale data
(archives or files being removed), its self-cleaning efforts can take a long
time. It's been observed to take O(seconds) to do a single sqlite query. In
the metrics, see the sqlite3_milliseconds_count...{"nuke..."} ones. And the
groom() function will check every file for staleness, until interrupted by a
SIGUSR1, so that O(50000) stale files could take a whole day.
During all this time, the server can service buildid requests, so it's not that
bad, but it cannot scan for new files.
We should investigate whether a more time-bounded groom operation could serve
about as well. We could limit groom to a certain percentage of time, like 1
hr/day, then abort. (We'd have to traverse the file list in some stateful or
random way in order not to just recheck the same ones over and over.) The
post-loop cleanup ops ("nuke orphan buildids" ... end of function) are
relatively quick and not worth worrying about at this time.
Alternately, there may be a way to accelerate the individual nuke queries,
maybe with more indexes, at the cost of more storage.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 2+ messages in thread
* [Bug debuginfod/28514] limit grooming time for many stale files
2021-10-28 19:06 [Bug debuginfod/28514] New: limit grooming time for many stale files fche at redhat dot com
@ 2021-11-05 12:17 ` fche at redhat dot com
0 siblings, 0 replies; 2+ messages in thread
From: fche at redhat dot com @ 2021-11-05 12:17 UTC (permalink / raw)
To: elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=28514
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
commit c1e8c8c6b25cb2b5c16553609f19a9ed5dd4e146
Author: Frank Ch. Eigler <fche@redhat.com>
Date: Thu Nov 4 13:08:35 2021 -0400
PR28514: debuginfod: limit groom operation times
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-11-05 12:17 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-28 19:06 [Bug debuginfod/28514] New: limit grooming time for many stale files fche at redhat dot com
2021-11-05 12:17 ` [Bug debuginfod/28514] " fche at redhat dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).