From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mr4.vodafonemail.de (mr4.vodafonemail.de [145.253.228.164]) by sourceware.org (Postfix) with ESMTPS id 1630D386103A for ; Sun, 18 Feb 2024 19:51:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1630D386103A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=nexgo.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=nexgo.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1630D386103A Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=145.253.228.164 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1708285916; cv=none; b=XzdtAYkG0z/kfHfAwAAdcoNy9973uEW27oWUR5uM9K4BagcatVDPdw0XKztfWO+FVZeCFAlEb0lKbhb75n3qyK90dZ3ofk1fsXkjEcRdmrQ7kkLA81cGUrS9OTB/t7jo2cpcxS6xEYl26hIulCDSXPkIVyubF7iIPY+pFZHX6M4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1708285916; c=relaxed/simple; bh=RuFHDUJuDMvE4MF0u70v9261wyQC0pWL59cRF3Fi5Jg=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=oy2GpufIAHPOTIrN4BXVsYBPYqaaq7Yc24Vb2fmD+1fEmk/75IL0xEf1e7x8u3nxaVO++jNVxy/ASnddnlMoJz7ONTzTamFxHi1RPP0TRCLh7pKCaDtEJtKr3a0/vNRqvSgy6qB8zof1gAMVeZtAo8pNvMWeY2Di+cMbrkjqBmk= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nexgo.de; s=vfde-mb-mr2-23sep; t=1708285912; bh=835X/ryPLtSNb4Zu8ImpyhFySMpU4Kupd97w3GTdPs4=; h=From:To:Subject:Date:Message-ID:User-Agent:Content-Type:From; b=Lbc3mam/TF8G6R7JRBZo8nXH7xV9ezaBQhjlNuVSWLdXdX3vS17Syftiai5/An9Yw bSe66Kjh7IwQ81HQQ5DnaN96HNzcYM4OJegr9pYTqRzaeI2M/tgN6P0p+5g2qWif7h koXiwQuGwggD3Oz5kVGkSJdUStwhTnjAkXA57Vhw= Received: from smtp.vodafone.de (unknown [10.0.0.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mr4.vodafonemail.de (Postfix) with ESMTPS id 4TdGX84931z1xvF for ; Sun, 18 Feb 2024 19:51:52 +0000 (UTC) Received: from Gerda (p54a0c6f4.dip0.t-ipconnect.de [84.160.198.244]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.vodafone.de (Postfix) with ESMTPSA id 4TdGX42LtYz9s3q for ; Sun, 18 Feb 2024 19:51:45 +0000 (UTC) From: ASSI To: cygwin-apps@cygwin.com Subject: [cygport] enabling a replacement for "objdump -d -l" Date: Sun, 18 Feb 2024 20:51:41 +0100 Message-ID: <87a5nx5z5e.fsf@Gerda.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-purgate-type: clean X-purgate: clean X-purgate-size: 5536 X-purgate-ID: 155817::1708285908-F1FFDBDE-B5978A45/0/0 X-Spam-Status: No, score=-3036.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,KAM_SHORT,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Cygport uses "objdump -d -l" to extract the list of source files that need to be copied into the debuginfo package. This operation triggers some O(N=C2=B2) or even higher complexity and in addition has been getting slower in recent binutils releases due to more and more information being put into the object files. For gcc-11 extracting the debug source files takes up to 45 minutes per executable (up from about 15 minutes until 2.39) and for gcc-13 (with about 1.5 times the number of lines to extract) it is already taking more than two hours. So if you just package gcc-13 using a single thread you'd be looking on the order of 20 hours wall clock time, which is unacceptable. The deassembly implied by the "-d" (which is not the part that has the superlinear complexity btw, but produces a baseline of 2 hours single thread runtime all by itself) is also unnecessary to extract just the filenames of the source files as we throw away the location information anyway and so I've written a small parser that works on the DWARF dump instead (which can be produced in linear time with a very small scaling factor, so practically constant time even for very large executables). Unfortunately binutils does not yet offer a machine readable format for these dumps, but parsing the text is not too difficult even though the format is undocumented. The DWARF-5 documentation isn't the most enjoyable read, but it was helpful enough to figure it all out. I've also integrated the filtering of unrelated source file information (from system headers and external libraries). The end result is the same runtime as before on small object files, a factor up to 100 speedup for medium sized object files and speedups in the several thousands range for large sized ones (or a total single-thread runtime of less than 20 seconds for gcc-13). dwarf-parse.-pl --8<---------------cut here---------------start------------->8--- #!perl -w use common::sense; use List::Util qw( sum ); my $filter =3D shift @ARGV or die "not enough arguments"; my $obj =3D shift @ARGV or die "not enough arguments"; my @objdump =3D qw( /usr/bin/objdump -WNl ); open my $DWARF, "-|", @objdump, $obj or die "can't invoke objdump\n$!"; my ( @dirs, @files, %fn, %rn ); while (<$DWARF>) { if (/^ The Directory Table/../^$/) { if (/^ \d+/) { =09=20=20=20=20 my ( $entry, $dir ) =3D m/^ (\d+)\t.+: (.+)$/; $dir =3D "$dirs[0]/$dir" if ($dir =3D~ m:\A[^/]:); push @dirs, $dir; } } if (/^ The File Name Table/../^$/) { if (/^ \d+/) { my ( $idx, $fn, undef ) =3D m/^ \d+\t(\d+)\t.+: (.+)$/; $rn{"$dirs[$idx]/$fn"}++; push @files, "$dirs[$idx]/$fn"; } } if (my $rc =3D /^ Line Number Statements/../^ Offset:/) { $fn{"$files[0]"}++ if ($rc =3D=3D 1); $fn{"$files[$1]"}++ if m/ Set File Name to entry (\d+) in the File Name Ta= ble/; @files =3D () if ($rc =3D~ m/E0$/); @dirs =3D () if ($rc =3D~ m/E0$/); } if (/^ No Line Number Statements./../^$/) { @files =3D (); @dirs =3D (); } } foreach my $fn (grep m:^$filter:, sort keys %fn) { say sprintf "%s", $fn; } say STDERR sprintf "\tLNS: %6d (%6d locations) <=3D> FNT: %6d ( %6d locatio= ns)", 0+grep( m:^$filter:, keys %fn ), sum( values %fn ), 0+grep( m:^$filter:, keys %rn ), sum( values %rn ) if (0); close $DWARF or die "failed to close objdump\n$!"; --8<---------------cut here---------------end--------------->8--- Integration into cygport is made configurable via a variable to be set in .cygportrc for instance in order to easily revert back to the original objdump invocation if necessary. I've been producing packages with that setup for a while now and have not noticed any errors. In principle the new parser actually produces more complete output as there can be multiple line number statements and hence source files per location, but objdump only lists one of them in the disassembly (at least sometimes). In practise I haven't found a package until now where the final list (after filtering) is different. https://repo.or.cz/cygport/rpm-style.git/commitdiff_plain/7ab8b26aaefb8a6ce= 050a196ddc97ce416ebe7a9 --8<---------------cut here---------------start------------->8--- lib/src_postinst.cygpart: use DWARF_PARSE optionally instead of objdump -dl --- diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart index f06004e4..3dd6e893 100644 --- a/lib/src_postinst.cygpart +++ b/lib/src_postinst.cygpart @@ -1096,7 +1096,12 @@ __prepstrip_one() { else dbg=3D"/usr/lib/debug/${exe}.dbg"; =20 - lines=3D$(${objdump} -d -l "${exe}" 2>/dev/null | sed -ne "s|.*\(/usr/sr= c/debug/${PF}/.*\):[0-9]*$|\1|gp" | sort -u | tee -a ${T}/.dbgsrc.out.${oxt= } | wc -l); + if defined DWARF_PARSE + then + lines=3D$(${DWARF_PARSE} /usr/src/debug/${PF}/ "${exe}" | tee -a ${T}/.= dbgsrc.out.${oxt} | wc -l); + else + lines=3D$(${objdump} -d -l "${exe}" 2>/dev/null | sed -ne "s|.*\(/usr/s= rc/debug/${PF}/.*\):[0-9]*$|\1|gp" | sort -u | tee -a ${T}/.dbgsrc.out.${ox= t} | wc -l); + fi =20 # we expect --add-gnu-debuglink to fail if a # .gnu_debuglink section already exists (e.g. binutils, --8<---------------cut here---------------end--------------->8--- Regards, Achim. --=20 +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves