From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by sourceware.org (Postfix) with ESMTPS id 7FFF9385801A for ; Sat, 27 Mar 2021 12:57:50 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 7FFF9385801A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=tdevries@suse.de X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 9D381AE15; Sat, 27 Mar 2021 12:57:49 +0000 (UTC) Date: Sat, 27 Mar 2021 13:57:48 +0100 From: Tom de Vries To: dwz@sourceware.org, jakub@redhat.com, mark@klomp.org Subject: [committed] Do finalize-multifile in parallel Message-ID: <20210327125746.GA18400@delia> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: dwz@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Dwz mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Mar 2021 12:57:51 -0000 Hi, Run the finalize-multifile phase in parallel. Using an experiment, we can see the effect: ... $ for j in $(seq 1 2); do \ rm -f 1 2 3; \ cp debug/cc1 1; cp 1 2; \ echo "j: $j"; \ ../measure/time.sh ./dwz -lnone -m 3 1 2 -j$j; \ done j: 1 maxmem: 1295068 real: 26.38 user: 25.38 system: 0.77 j: 2 maxmem: 1297724 real: 23.55 user: 25.81 system: 0.88 ... Committed to trunk. Thanks, - Tom Do finalize-multifile in parallel 2021-03-27 Tom de Vries * dwz.c (dwz_files_1): Run finalize-multifile phase in parallel, if max_forks > 0. --- dwz.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 62 insertions(+), 14 deletions(-) diff --git a/dwz.c b/dwz.c index 3cc0a26..eeae750 100644 --- a/dwz.c +++ b/dwz.c @@ -16608,21 +16608,69 @@ dwz_files_1 (int nr_files, char *files[], bool hardlink, goto cleanup; } - for (i = 0; i < nr_files; i++) + if (max_forks > 1) { - dw_cu_ref cu; - file = files[i]; - if (stats_p) - init_stats (file); - multifile_mode = MULTIFILE_MODE_FI; - /* Don't process again files that couldn't - be processed successfully. Also skip hard links. */ - if (resa[i].res == -1 || resa[i].res == -2 - || resa[i].skip_multifile) - continue; - for (cu = alt_first_cu; cu; cu = cu->cu_next) - alt_clear_dups (cu->cu_die); - ret |= dwz (file, NULL, &resa[i]); + pid_t pids[nr_files]; + int nr_forks = 0; + for (i = 0; i < nr_files; i++) + pids[i] = 0; + for (i = 0; i < nr_files; i++) + { + file = files[i]; + struct file_result *res = &resa[i]; + multifile_mode = MULTIFILE_MODE_FI; + /* Don't process again files that couldn't + be processed successfully. Also skip hard links. */ + if (resa[i].res == -1 || resa[i].res == -2 + || resa[i].skip_multifile) + continue; + + if (nr_forks == max_forks) + { + int thisret = wait_child_exit (-1, pids, i, res); + if (thisret == 1) + ret = 1; + nr_forks--; + } + + pid_t fork_res = fork (); + assert (fork_res != -1); + if (fork_res == 0) + { + int thisret = dwz (file, NULL, res); + return encode_child_exit_status (thisret, res); + } + else + { + pids[i] = fork_res; + nr_forks++; + } + } + if (nr_forks > 0) + { + int thisret = wait_children_exit (pids, nr_files, resa); + if (thisret == 1) + ret = 1; + } + } + else + { + for (i = 0; i < nr_files; i++) + { + dw_cu_ref cu; + file = files[i]; + if (stats_p) + init_stats (file); + multifile_mode = MULTIFILE_MODE_FI; + /* Don't process again files that couldn't + be processed successfully. Also skip hard links. */ + if (resa[i].res == -1 || resa[i].res == -2 + || resa[i].skip_multifile) + continue; + for (cu = alt_first_cu; cu; cu = cu->cu_next) + alt_clear_dups (cu->cu_die); + ret |= dwz (file, NULL, &resa[i]); + } } if (hardlink)