From: "fche at redhat dot com" <sourceware-bugzilla@sourceware.org>
To: elfutils-devel@sourceware.org
Subject: [Bug debuginfod/28514] New: limit grooming time for many stale files
Date: Thu, 28 Oct 2021 19:06:55 +0000 [thread overview]
Message-ID: <bug-28514-10460@http.sourceware.org/bugzilla/> (raw)
https://sourceware.org/bugzilla/show_bug.cgi?id=28514
Bug ID: 28514
Summary: limit grooming time for many stale files
Product: elfutils
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: debuginfod
Assignee: unassigned at sourceware dot org
Reporter: fche at redhat dot com
CC: elfutils-devel at sourceware dot org
Target Milestone: ---
When a big debuginfod server starts grooming, and starts finding stale data
(archives or files being removed), its self-cleaning efforts can take a long
time. It's been observed to take O(seconds) to do a single sqlite query. In
the metrics, see the sqlite3_milliseconds_count...{"nuke..."} ones. And the
groom() function will check every file for staleness, until interrupted by a
SIGUSR1, so that O(50000) stale files could take a whole day.
During all this time, the server can service buildid requests, so it's not that
bad, but it cannot scan for new files.
We should investigate whether a more time-bounded groom operation could serve
about as well. We could limit groom to a certain percentage of time, like 1
hr/day, then abort. (We'd have to traverse the file list in some stateful or
random way in order not to just recheck the same ones over and over.) The
post-loop cleanup ops ("nuke orphan buildids" ... end of function) are
relatively quick and not worth worrying about at this time.
Alternately, there may be a way to accelerate the individual nuke queries,
maybe with more indexes, at the cost of more storage.
--
You are receiving this mail because:
You are on the CC list for the bug.
next reply other threads:[~2021-10-28 19:06 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-28 19:06 fche at redhat dot com [this message]
2021-11-05 12:17 ` [Bug debuginfod/28514] " fche at redhat dot com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bug-28514-10460@http.sourceware.org/bugzilla/ \
--to=sourceware-bugzilla@sourceware.org \
--cc=elfutils-devel@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).