From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id DBA8A3858D1E for ; Tue, 20 Jun 2023 21:37:02 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DBA8A3858D1E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: by gnu.wildebeest.org (Postfix, from userid 1000) id D65CD313AC85; Tue, 20 Jun 2023 23:37:01 +0200 (CEST) Date: Tue, 20 Jun 2023 23:37:01 +0200 From: Mark Wielaard To: Romain Geissler , elfutils-devel@sourceware.org Subject: Re: Performance issue with systemd-coredump and container process linking 2000 shared libraries. Message-ID: <20230620213701.GN24233@gnu.wildebeest.org> References: <20230614133920.10-1-romain.geissler@amadeus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-3029.7 required=5.0 tests=BAYES_00,JMQ_SPF_NEUTRAL,KAM_DMARC_STATUS,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi, On Mon, Jun 19, 2023 at 05:08:50PM +0200, Mark Wielaard wrote: > Just to let you know I am looking at this. But haven't made much > progress in understanding it yet. Thanks so much for the reproducer. I > have been able to see the (very slow) parsing of the core file with it. > > $ time ./mimic-systemd-coredump > [...] > real 3m35.965s > user 0m0.722s > sys 0m0.345s > > Note however that a lot of time is "missing". > And in fact running it again is fast!?! > > $ time ./mimic-systemd-coredump > real 0m0.327s > user 0m0.272s > sys 0m0.050s > > This is because of the kernel inode/dentry cache. > If I do $ echo 2 | sudo tee /proc/sys/vm/drop_caches > before running ./mimic-systemd-coredump it is always slow. > > I'll try to figure out what we do to make it so hard for the kernel to > do these lookups. So I made a mistake here. Since I was testing on fedora 38 which has DEBUGINFOD_URLS set. Without DEBUGINFOD_URLS set there is no big slowdown. Do you have the DEBUGINFOD_URLS environment variable set? The real sd-coredump will not have DEBUGINFOD_URLS set (I hope). Thanks, Mark