From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 02E5C3858D28 for ; Fri, 8 Apr 2022 20:05:31 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 02E5C3858D28 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-158-7HRjIOfNMSOsfqc5GJo_sA-1; Fri, 08 Apr 2022 16:05:28 -0400 X-MC-Unique: 7HRjIOfNMSOsfqc5GJo_sA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 456CC85A5BE; Fri, 8 Apr 2022 20:05:28 +0000 (UTC) Received: from redhat.com (unknown [10.2.16.64]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 350B0815B; Fri, 8 Apr 2022 20:05:28 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.94.2) (envelope-from ) id 1ncurH-00017u-Bd; Fri, 08 Apr 2022 16:05:27 -0400 Date: Fri, 8 Apr 2022 16:05:27 -0400 From: "Frank Ch. Eigler" To: Milian Wolff Cc: elfutils-devel@sourceware.org Subject: Re: caching failed lookups of debuginfo? Message-ID: <20220408200527.GC23295@redhat.com> References: <4448277.fIUe8AKecr@milian-workstation> MIME-Version: 1.0 In-Reply-To: <4448277.fIUe8AKecr@milian-workstation> User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2022 20:05:33 -0000 Hi - > another debuginfod related question, but unrelated to the other thread I > started earlier today. In a work branch I have ported my heaptrack profiler > over to elfutils. I have then run the analyzer that uses elfutils (and thus > debuginfod internally via dwfl) on a recorded data file to have it download > all the debug info files it can find. Nice. > These negative lookups are not cached. Meaning rerunning the same process > using dwfl and debuginfod on the same data would always incur a significant > slowdown, as we would again and again try to look for something that's not > there. The lookups take roughly ~200ms for me to realize the data is not on > the server. That's not correct, as of elfutils 0.184 (commit 5f72c51a7e5c0), with some more recent tweaks in (commit 7d64173fb11c6). - FChE > What's worse, I'm seeing multiple lookups for the same buildid *within the > same process*. I.e.: > > ``` > export DEBUGINFOD_VERBOSE=1 > ./heaptrack_interpret ... |& egrep "^url 0 https" | sort | uniq -c | sort > ... > 6 url 0 https://debuginfod.archlinux.org/buildid/ > 7f4b16b4b407cbae2d7118d6f99610e29a18a56a/debuginfo > 8 url 0 https://debuginfod.archlinux.org/buildid/ > c09c6f50f6bcec73c64a0b4be77eadb8f7202410/debuginfo > 14 url 0 https://debuginfod.archlinux.org/buildid/ > 85766e9d8458b16e9c7ce6e07c712c02b8471dbc/debuginfo > ``` > > Here, we are paying roughly `14 * 0.2s = 2.8s` just for a single library. > > Can we find a way to improve this situation somehow generically? I would > personally even be OK with caching the 404 error case locally for some time > (say, one hour or one day or ...). Then at least we would at most pay this > cost once per library, and not multiple times. And rerunning the analysis a > second time would become much faster again. > > Was there a deliberate decision against caching negative server side lookups? > > Thanks > -- > Milian Wolff > mail@milianw.de > http://milianw.de