From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 91187 invoked by alias); 24 Feb 2020 22:29:55 -0000 Mailing-List: contact elfutils-devel-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Post: List-Help: List-Subscribe: Sender: elfutils-devel-owner@sourceware.org Received: (qmail 91177 invoked by uid 89); 24 Feb 2020 22:29:54 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Checked: by ClamAV 0.100.3 on sourceware.org X-Virus-Found: No X-Spam-SWARE-Status: No, score=-16.6 required=5.0 tests=AWL,BAYES_00,GIT_PATCH_0,GIT_PATCH_1,GIT_PATCH_2,GIT_PATCH_3,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=ham version=3.3.1 spammy=H*r:4.92.3, baking, from X-Spam-Status: No, score=-16.6 required=5.0 tests=AWL,BAYES_00,GIT_PATCH_0,GIT_PATCH_1,GIT_PATCH_2,GIT_PATCH_3,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on sourceware.org X-Spam-Level: X-HELO: us-smtp-delivery-1.mimecast.com Received: from us-smtp-2.mimecast.com (HELO us-smtp-delivery-1.mimecast.com) (207.211.31.81) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 24 Feb 2020 22:29:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582583391; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=h4IBZ4MQ+ZngkzEFkWsrLKUbagwslKBfCNnM2YyfzJ8=; b=hjKpJn2K4QX1weVUh+yhkn2EZPQGUJrcUxNmtG68M50iydiqAlOfFzzW+kkrUoE3k02lRv F03jDdTi5fyadT5lt0d/tBCdxthqWXx+HU4g5/SLq+EXucfKBQjHn0DsUC7ydUxmKqb0Xf hkPfEMsRTFrYSrR3Qvuo/wb7Q3LMzvg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-185-Zh9Xr9xUNguWCwXWTmzV5Q-1; Mon, 24 Feb 2020 17:29:49 -0500 X-MC-Unique: Zh9Xr9xUNguWCwXWTmzV5Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4E9C7107ACCC for ; Mon, 24 Feb 2020 22:29:48 +0000 (UTC) Received: from redhat.com (ovpn-116-36.phx2.redhat.com [10.3.116.36]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 172285D9E5 for ; Mon, 24 Feb 2020 22:29:48 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.92.3) (envelope-from ) id 1j6MEU-0007Cs-RR for elfutils-devel@sourceware.org; Mon, 24 Feb 2020 17:29:46 -0500 Date: Mon, 24 Feb 2020 22:29:00 -0000 From: "Frank Ch. Eigler" To: elfutils-devel@sourceware.org Subject: patch to commit soon: PR25375 debuginfod prefetching Message-ID: <20200224222946.GC26763@redhat.com> MIME-Version: 1.0 User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-IsSubscribed: yes X-SW-Source: 2020-q1/txt/msg00130.txt Hi - This patch has been baking on my public servers awhile and can make a huge difference in performance. It's not something immediately obvious how or whether to test, as it's a pure performance improvement. Planning to push shortly. commit ae8b89716116a6df124f8700f77460b5e97c12c4 (origin/fche/pr25375) Author: Frank Ch. Eigler Date: Sat Jan 25 18:43:07 2020 -0500 PR25375: fdcache prefetching to reduce repeated archive decompression diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx index 0acd70e4a916..7f432d7d9c41 100644 --- a/debuginfod/debuginfod.cxx +++ b/debuginfod/debuginfod.cxx @@ -355,6 +355,8 @@ static const struct argp_option options[] =3D { "fdcache-fds", ARGP_KEY_FDCACHE_FDS, "NUM", 0, "Maximum number of arc= hive files to keep in fdcache.", 0 }, #define ARGP_KEY_FDCACHE_MBS 0x1002 { "fdcache-mbs", ARGP_KEY_FDCACHE_MBS, "MB", 0, "Maximum total size of = archive file fdcache.", 0 }, +#define ARGP_KEY_FDCACHE_PREFETCH 0x1003 + { "fdcache-prefetch", ARGP_KEY_FDCACHE_PREFETCH, "NUM", 0, "Number of a= rchive files to prefetch into fdcache.", 0 }, { NULL, 0, NULL, 0, NULL, 0 } }; =20 @@ -394,6 +396,7 @@ static regex_t file_exclude_regex; static bool traverse_logical; static long fdcache_fds; static long fdcache_mbs; +static long fdcache_prefetch; static string tmpdir; =20 static void set_metric(const string& key, int64_t value); @@ -476,6 +479,9 @@ parse_opt (int key, char *arg, case ARGP_KEY_FDCACHE_MBS: fdcache_mbs =3D atol (arg); break; + case ARGP_KEY_FDCACHE_PREFETCH: + fdcache_prefetch =3D atol (arg); + break; case ARGP_KEY_ARG: source_paths.insert(string(arg)); break; @@ -975,14 +981,14 @@ class libarchive_fdcache string archive; string entry; string fd; - long fd_size_mb; // rounded up megabytes + double fd_size_mb; // slightly rounded up megabytes }; deque lru; // @head: most recently used long max_fds; long max_mbs; =20 public: - void intern(const string& a, const string& b, string fd, off_t sz) + void intern(const string& a, const string& b, string fd, off_t sz, bool = front_p) { { unique_lock lock(fdcache_lock); @@ -995,11 +1001,15 @@ class libarchive_fdcache break; // must not continue iterating } } - long mb =3D ((sz+1023)/1024+1023)/1024; + double mb =3D (sz+65535)/1048576.0; // round up to 64K block fdcache_entry n =3D { a, b, fd, mb }; - lru.push_front(n); + if (front_p) + lru.push_front(n); + else + lru.push_back(n); if (verbose > 3) - obatched(clog) << "fdcache interned a=3D" << a << " b=3D" << b << " = fd=3D" << fd << " mb=3D" << mb << endl; + obatched(clog) << "fdcache interned a=3D" << a << " b=3D" << b + << " fd=3D" << fd << " mb=3D" << mb << " front=3D" <<= front_p << endl; } =20 this->limit(max_fds, max_mbs); // age cache if required @@ -1022,6 +1032,17 @@ class libarchive_fdcache return -1; } =20 + int probe(const string& a, const string& b) // just a cache residency ch= eck - don't modify LRU state, don't open + { + unique_lock lock(fdcache_lock); + for (auto i =3D lru.begin(); i < lru.end(); i++) + { + if (i->archive =3D=3D a && i->entry =3D=3D b) + return true; + } + return false; + } + void clear(const string& a, const string& b) { unique_lock lock(fdcache_lock); @@ -1047,7 +1068,7 @@ class libarchive_fdcache this->max_mbs =3D maxmbs; =20 long total_fd =3D 0; - long total_mb =3D 0; + double total_mb =3D 0.0; for (auto i =3D lru.begin(); i < lru.end(); i++) { // accumulate totals from most recently used one going backward @@ -1117,6 +1138,7 @@ handle_buildid_r_match (int64_t b_mtime, return 0; } =20 + // check for a match in the fdcache first int fd =3D fdcache.lookup(b_source0, b_source1); while (fd >=3D 0) // got one!; NB: this is really an if() with a possibl= e branch out to the end { @@ -1152,6 +1174,7 @@ handle_buildid_r_match (int64_t b_mtime, // NB: see, we never go around the 'loop' more than once } =20 + // no match ... grumble, must process the archive string archive_decoder =3D "/dev/null"; string archive_extension =3D ""; for (auto&& arch : scan_archives) @@ -1196,8 +1219,19 @@ handle_buildid_r_match (int64_t b_mtime, if (rc !=3D ARCHIVE_OK) throw archive_exception(a, "cannot open archive from pipe"); =20 - while(1) // parse cpio archive entries + // archive traversal is in three stages: + // 1) skip entries whose names do not match the requested one + // 2) extract the matching entry name (set r =3D result) + // 3) extract some number of prefetched entries (just into fdcache) + // 4) abort any further processing + struct MHD_Response* r =3D 0; // will set in stage 2 + unsigned prefetch_count =3D fdcache_prefetch; // will decrement in stage= 3 + + while(r =3D=3D 0 || prefetch_count > 0) // stage 1, 2, or 3 { + if (interrupted) + break; + struct archive_entry *e; rc =3D archive_read_next_header (a, &e); if (rc !=3D ARCHIVE_OK) @@ -1229,18 +1263,32 @@ handle_buildid_r_match (int64_t b_mtime, throw archive_exception(a, "cannot extract file"); } =20 + if (r !=3D 0) // stage 3 + { + // NB: now we know we have a complete reusable file; make fdcache + // responsible for unlinking it later. + fdcache.intern(b_source0, not_source1, + tmppath, archive_entry_size(e), + false); // prefetched ones go to back of lru + prefetch_count --; + close (fd); // we're not saving this fd to make a mhd-response f= rom! + continue; + } + // NB: now we know we have a complete reusable file; make fdcache // responsible for unlinking it later. - fdcache.intern(b_source0, b_source1, tmppath, archive_entry_size(e)); + fdcache.intern(b_source0, b_source1, + tmppath, archive_entry_size(e), + true); // requested ones go to the front of lru =20 inc_metric ("http_responses_total","result",archive_extension + " ar= chive"); - struct MHD_Response* r =3D MHD_create_response_from_fd (archive_entr= y_size(e), fd); + r =3D MHD_create_response_from_fd (archive_entry_size(e), fd); if (r =3D=3D 0) { if (verbose) obatched(clog) << "cannot create fd-response for " << b_source= 0 << endl; close(fd); - break; // assume no chance of better luck around another iterati= on + break; // assume no chance of better luck around another iterati= on; no other copies of same file } else { @@ -1251,12 +1299,12 @@ handle_buildid_r_match (int64_t b_mtime, /* libmicrohttpd will close it. */ if (result_fd) *result_fd =3D fd; - return r; + continue; } } =20 // XXX: rpm/file not found: delete this R entry? - return 0; + return r; } =20 =20 @@ -2809,7 +2857,8 @@ main (int argc, char *argv[]) fdcache_mbs =3D 1024; // 1 gigabyte else fdcache_mbs =3D sfs.f_bavail * sfs.f_bsize / 1024 / 1024 / 4; // 25% o= f free space - fdcache_fds =3D concurrency * 2; + fdcache_prefetch =3D 64; // guesstimate storage is this much less costly= than re-decompression + fdcache_fds =3D (concurrency + fdcache_prefetch) * 2; =20 /* Parse and process arguments. */ int remaining; @@ -2943,6 +2992,7 @@ main (int argc, char *argv[]) obatched(clog) << "rescan time " << rescan_s << endl; obatched(clog) << "fdcache fds " << fdcache_fds << endl; obatched(clog) << "fdcache mbs " << fdcache_mbs << endl; + obatched(clog) << "fdcache prefetch " << fdcache_prefetch << endl; obatched(clog) << "fdcache tmpdir " << tmpdir << endl; obatched(clog) << "groom time " << groom_s << endl; if (scan_archives.size()>0) diff --git a/doc/debuginfod.8 b/doc/debuginfod.8 index ca844aedcfdc..ad4b2d993c8a 100644 --- a/doc/debuginfod.8 +++ b/doc/debuginfod.8 @@ -193,14 +193,17 @@ loops in the symbolic directory tree might lead to \f= Iinfinite traversal\fP. =20 .TP -.B "\-\-fdcache-fds=3DNUM" "\-\-fdcache-mbs=3DMB" +.B "\-\-fdcache\-fds=3DNUM" "\-\-fdcache\-mbs=3DMB" "\-\-fdcache-prefetc= h=3DNUM2" Configure limits on a cache that keeps recently extracted files from -archives. Up to NUM files and up to a total of MB megabytes will be -kept extracted, in order to avoid having to decompress their archives -again. The default NUM and MB values depend on the concurrency of the -system, and on the available disk space on the $TMPDIR or \fB/tmp\fP -filesystem. This is because that is where the most recently used -extracted files are kept. Grooming cleans this cache. +archives. Up to NUM requested files and up to a total of MB megabytes +will be kept extracted, in order to avoid having to decompress their +archives over and over again. In addition, up to NUM2 other files +from an archive may be prefetched into the cache before they are even +requested. The default NUM, NUM2, and MB values depend on the +concurrency of the system, and on the available disk space on the +$TMPDIR or \fB/tmp\fP filesystem. This is because that is where the +most recently used extracted files are kept. Grooming cleans this +cache. =20 .TP .B "\-v"