[Bug default/25951] support for parallel processing?

["vries at gcc dot gnu.org" <sourceware-bugzilla@sourceware.org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=25951

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
Created attachment 13297
  --> https://sourceware.org/bugzilla/attachment.cgi?id=13297&action=edit
Demonstrator patch

This demonstrator patch implements a simple form of multithreading, which only
works without:
- multifile (-m)
- hardlink (-h)
- low-mem limit 0 (-l0)

If a file hits the low-mem limit during the parallel phase, it's rerun in
low-mem mode after the parallel phase.

It passes the test-suite.  There is only one thread-sanitizer warning left, for
multiple assignment of dwz_oom to obstack_alloc_failed_handler.

I did a build of the libreoffice package on openSUSE with dwz disabled,
harvested the resulting .debug files (in total 175 files, 685MB), and did a dwz
run (without multifile) using those files.

With master:
...
maxmem: 714956
real: 17.77
user: 15.76
system: 0.50
...

With the patch on top of master:
...
maxmem: 1106516
real: 10.37
user: 20.59
system: 1.46
...

So, the trade off is as expected: faster realtime, but higher peak memory.

DWZ though contains the low-mem mode to keep memory usage in check, such that
dwz can be used on 32-bit systems, with still relatively large files.  So the
trade off on those systems may not be advantageous.  We could fix this by not
enabling parallel processing on such systems.

OTOH, we could also spawn processes instead of threads.  That means the
per-process peak memory does not increase.  It would also mean less messy code
changes (not having to use __thread all over the place).

An initial version that wouldn't deal with multifile (like this demonstrator
patch) wouldn't need much changes.  A version that would support multifile
would need a switch to indicate the location of the dwz.debug_info etc files. 
So, something like:
...
$ dwz -m 3 1 2
 create temp dir /tmp/abcdef
 spawn dwz 1 --multifile-dir /tmp/abcdef
 spawn dwz 2 --multifile-dir /tmp/abcdef
 wait for 2 spawned processes to finish ...
 spawned dwz 1 - compressing
 spawned dwz 2 - compressing
 spawned dwz 1 - multifile write (using dir /tmp/abcdef)
 spawned dwz 2 - multifile write (using dir /tmp/abcdef)
 spawned dwz 1 - done
 spawned dwz 2 - done
 waiting done
 multifile optimize (using files in /tmp/abcdef)
 multifile read
 multifile finalize 1
 multifile finalize 2
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.