From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by sourceware.org (Postfix) with ESMTPS id 205FC3857C44; Fri, 21 May 2021 08:43:06 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 205FC3857C44 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mliska@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id CAEDFAC11; Fri, 21 May 2021 08:43:04 +0000 (UTC) From: =?UTF-8?Q?Martin_Li=c5=a1ka?= Subject: Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) To: Richard Biener Cc: Segher Boessenkool , Andrew Pinski , GCC Mailing List , GCC Patches References: <20210511145904.GM10366@gate.crashing.org> <20210512121248.GU10366@gate.crashing.org> <5f13a740-5eff-886f-2b29-52a305fdf3b1@suse.cz> Message-ID: <03febb0f-16fa-4048-6680-438a63b11dcf@suse.cz> Date: Fri, 21 May 2021 10:43:04 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/mixed; boundary="------------4716C71B54C5901606B70CEB" Content-Language: en-US X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, NICE_REPLY_A, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_SBL, URIBL_SBL_A autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 May 2021 08:43:09 -0000 This is a multi-part message in MIME format. --------------4716C71B54C5901606B70CEB Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 5/20/21 2:54 PM, Richard Biener wrote: > On Thu, May 20, 2021 at 2:34 PM Martin Liška wrote: >> >> Hello. >> >> I've got a patch candidate that leverages partial linking for a couple of selected object files. >> >> I'm sending make all-host- jX results for my machine: >> >> before: 3m18s (user 32m52s) >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg >> >> after: 2m57m (user 35m) >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg >> >> One can utilize it with: >> make -j16 all-host PARTIAL_LTO=1 >> >> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? >> One can also tweak --param=lto-partitions=16 param value. >> >> Thoughts? > > You're LTO linking multiple objects here - that's almost as if you > were doing this > for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the > libbackend.a rule do a similar partial link trick. Yeah, apart from that one can't likely do partial linking for an archive: $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped compilation terminated. while ld.bfd immediately finishes. > > That gets you half of a LTO bootstrap then. > > So why did you go from applying this per-file to multiple files? Does $(LINKER) > have a proper rule to pick up a jobserver? > > When upstreaming in any form you probably have to gate it on bootstrap-lto > being not active. Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? One more thing I face is broken dependency: $ make clean && make -j32 PARTIAL_LTO=1 g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto In file included from ./tm.h:26, from /home/marxin/Programming/gcc/gcc/backend.h:28, from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, from generic-match.c:4: /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory 2286 | #include "insn-attr-common.h" | ^~~~~~~~~~~~~~~~~~~~ compilation terminated. make: *** [Makefile:2678: generic-match-lto.o] Error 1 make: *** Waiting for unfinished jobs.... In file included from ./tm.h:26, from /home/marxin/Programming/gcc/gcc/backend.h:28, from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, from gimple-match.c:4: /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory 2286 | #include "insn-attr-common.h" | ^~~~~~~~~~~~~~~~~~~~ I explicitly added: gimple-match.o: gimple-match.c $(generated_files) generic-match.o: generic-match.c $(generated_files) But it's not obeyed. Martin > > Richard. > >> Thanks, >> Martin --------------4716C71B54C5901606B70CEB Content-Type: text/x-patch; charset=UTF-8; name="0001-Try-LTO-partial-linking.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-Try-LTO-partial-linking.patch" >From b209c7aea76ceb8eadcf5843f30299fc8041a579 Mon Sep 17 00:00:00 2001 From: Martin Liska Date: Thu, 20 May 2021 14:29:35 +0200 Subject: [PATCH] Try LTO partial linking. --- gcc/Makefile.in | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 1164554e6d6..decfea7412b 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -220,7 +220,9 @@ libgcov-util.o-warn = -Wno-error libgcov-driver-tool.o-warn = -Wno-error libgcov-merge-tool.o-warn = -Wno-error gimple-match.o-warn = -Wno-unused +gimple-match-lto.o-warn = -Wno-unused generic-match.o-warn = -Wno-unused +generic-match-lto.o-warn = -Wno-unused dfp.o-warn = -Wno-strict-aliasing # All warnings have to be shut off in stage1 if the compiler used then @@ -1282,12 +1284,10 @@ ANALYZER_OBJS = \ # will build them sooner, because they are large and otherwise tend to be # the last objects to finish building. OBJS = \ - gimple-match.o \ - generic-match.o \ + common-base.a \ insn-attrtab.o \ insn-automata.o \ insn-dfatab.o \ - insn-emit.o \ insn-extract.o \ insn-latencytab.o \ insn-modes.o \ @@ -1295,7 +1295,6 @@ OBJS = \ insn-output.o \ insn-peep.o \ insn-preds.o \ - insn-recog.o \ insn-enums.o \ ggc-page.o \ adjust-alignment.o \ @@ -2627,6 +2626,32 @@ s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd generic-match.c $(STAMP) s-match +gimple-match.o: gimple-match.c $(generated_files) +generic-match.o: generic-match.c $(generated_files) + +ifdef PARTIAL_LTO +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r +LTO_FLAGS = -flto + +gimple-match-lto.o: gimple-match.c + $(COMPILE) $< $(LTO_FLAGS) +generic-match-lto.o: generic-match.c + $(COMPILE) $< $(LTO_FLAGS) +insn-recog-lto.o: insn-recog.c + $(COMPILE) $< $(LTO_FLAGS) +insn-emit-lto.o: insn-emit.c + $(COMPILE) $< $(LTO_FLAGS) + +common-base.a: gimple-match-lto.o generic-match-lto.o insn-recog-lto.o insn-emit-lto.o + -rm -rf $@ + $(LINKER) $^ $(LTO_LINKER_FLAGS) -o common-base.o + $(AR) $(AR_FLAGS)T $@ common-base.o +else +common-base.a: gimple-match.o generic-match.o insn-recog.o insn-emit.o + -rm -rf $@ + $(AR) $(AR_FLAGS)T $@ $^ +endif + GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \ $(host_xm_file_list) \ $(tm_file_list) $(HASHTAB_H) $(SPLAY_TREE_H) $(srcdir)/bitmap.h \ -- 2.31.1 --------------4716C71B54C5901606B70CEB--