From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id 92E403858018; Wed, 10 Mar 2021 10:19:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 92E403858018 From: "vries at gcc dot gnu.org" To: dwz@sourceware.org Subject: [Bug default/25951] support for parallel processing? Date: Wed, 10 Mar 2021 10:19:44 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: dwz X-Bugzilla-Component: default X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: enhancement X-Bugzilla-Who: vries at gcc dot gnu.org X-Bugzilla-Status: NEW X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: nobody at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: dwz@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Dwz mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2021 10:19:44 -0000 https://sourceware.org/bugzilla/show_bug.cgi?id=3D25951 --- Comment #1 from Tom de Vries --- Created attachment 13297 --> https://sourceware.org/bugzilla/attachment.cgi?id=3D13297&action=3Ded= it Demonstrator patch This demonstrator patch implements a simple form of multithreading, which o= nly works without: - multifile (-m) - hardlink (-h) - low-mem limit 0 (-l0) If a file hits the low-mem limit during the parallel phase, it's rerun in low-mem mode after the parallel phase. It passes the test-suite. There is only one thread-sanitizer warning left,= for multiple assignment of dwz_oom to obstack_alloc_failed_handler. I did a build of the libreoffice package on openSUSE with dwz disabled, harvested the resulting .debug files (in total 175 files, 685MB), and did a= dwz run (without multifile) using those files. With master: ... maxmem: 714956 real: 17.77 user: 15.76 system: 0.50 ... With the patch on top of master: ... maxmem: 1106516 real: 10.37 user: 20.59 system: 1.46 ... So, the trade off is as expected: faster realtime, but higher peak memory. DWZ though contains the low-mem mode to keep memory usage in check, such th= at dwz can be used on 32-bit systems, with still relatively large files. So t= he trade off on those systems may not be advantageous. We could fix this by n= ot enabling parallel processing on such systems. OTOH, we could also spawn processes instead of threads. That means the per-process peak memory does not increase. It would also mean less messy c= ode changes (not having to use __thread all over the place). An initial version that wouldn't deal with multifile (like this demonstrator patch) wouldn't need much changes. A version that would support multifile would need a switch to indicate the location of the dwz.debug_info etc file= s.=20 So, something like: ... $ dwz -m 3 1 2 create temp dir /tmp/abcdef spawn dwz 1 --multifile-dir /tmp/abcdef spawn dwz 2 --multifile-dir /tmp/abcdef wait for 2 spawned processes to finish ... spawned dwz 1 - compressing spawned dwz 2 - compressing spawned dwz 1 - multifile write (using dir /tmp/abcdef) spawned dwz 2 - multifile write (using dir /tmp/abcdef) spawned dwz 1 - done spawned dwz 2 - done waiting done multifile optimize (using files in /tmp/abcdef) multifile read multifile finalize 1 multifile finalize 2 ... --=20 You are receiving this mail because: You are on the CC list for the bug.=