* [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) [not found] ` <20210512121248.GU10366@gate.crashing.org> @ 2021-05-20 12:34 ` Martin Liška 2021-05-20 12:54 ` Richard Biener 0 siblings, 1 reply; 21+ messages in thread From: Martin Liška @ 2021-05-20 12:34 UTC (permalink / raw) To: Segher Boessenkool, Richard Biener, Andrew Pinski Cc: GCC Mailing List, GCC Patches [-- Attachment #1: Type: text/plain, Size: 726 bytes --] Hello. I've got a patch candidate that leverages partial linking for a couple of selected object files. I'm sending make all-host- jX results for my machine: before: 3m18s (user 32m52s) https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg after: 2m57m (user 35m) https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg One can utilize it with: make -j16 all-host PARTIAL_LTO=1 @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? One can also tweak --param=lto-partitions=16 param value. Thoughts? Thanks, Martin [-- Attachment #2: 0001-Try-LTO-partial-linking.patch --] [-- Type: text/x-patch, Size: 2367 bytes --] From 85228e612610c0e4b0324f6bebc84ef7c0211c4a Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Thu, 20 May 2021 14:29:35 +0200 Subject: [PATCH] Try LTO partial linking. --- gcc/Makefile.in | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 1164554e6d6..f76bcea66f5 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -220,7 +220,9 @@ libgcov-util.o-warn = -Wno-error libgcov-driver-tool.o-warn = -Wno-error libgcov-merge-tool.o-warn = -Wno-error gimple-match.o-warn = -Wno-unused +gimple-match-lto.o-warn = -Wno-unused generic-match.o-warn = -Wno-unused +generic-match-lto.o-warn = -Wno-unused dfp.o-warn = -Wno-strict-aliasing # All warnings have to be shut off in stage1 if the compiler used then @@ -1282,12 +1284,10 @@ ANALYZER_OBJS = \ # will build them sooner, because they are large and otherwise tend to be # the last objects to finish building. OBJS = \ - gimple-match.o \ - generic-match.o \ + common-base.a \ insn-attrtab.o \ insn-automata.o \ insn-dfatab.o \ - insn-emit.o \ insn-extract.o \ insn-latencytab.o \ insn-modes.o \ @@ -1295,7 +1295,6 @@ OBJS = \ insn-output.o \ insn-peep.o \ insn-preds.o \ - insn-recog.o \ insn-enums.o \ ggc-page.o \ adjust-alignment.o \ @@ -2627,6 +2626,29 @@ s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd generic-match.c $(STAMP) s-match +ifdef PARTIAL_LTO +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r +LTO_FLAGS = -flto + +gimple-match-lto.o: gimple-match.c $(TARGET_H) + $(COMPILE) $< $(LTO_FLAGS) +generic-match-lto.o: generic-match.c $(TARGET_H) + $(COMPILE) $< $(LTO_FLAGS) +insn-recog-lto.o: insn-recog.c + $(COMPILE) $< $(LTO_FLAGS) +insn-emit-lto.o: insn-emit.c + $(COMPILE) $< $(LTO_FLAGS) + +common-base.a: gimple-match-lto.o generic-match-lto.o insn-recog-lto.o insn-emit-lto.o + -rm -rf $@ + $(LINKER) $^ $(LTO_LINKER_FLAGS) -o common-base.o + $(AR) $(AR_FLAGS)T $@ common-base.o +else +common-base.a: gimple-match.o generic-match.o insn-recog.o insn-emit.o + -rm -rf $@ + $(AR) $(AR_FLAGS)T $@ $^ +endif + GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \ $(host_xm_file_list) \ $(tm_file_list) $(HASHTAB_H) $(SPLAY_TREE_H) $(srcdir)/bitmap.h \ -- 2.31.1 ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 12:34 ` [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) Martin Liška @ 2021-05-20 12:54 ` Richard Biener 2021-05-20 13:06 ` Martin Liška 2021-05-21 8:43 ` Martin Liška 0 siblings, 2 replies; 21+ messages in thread From: Richard Biener @ 2021-05-20 12:54 UTC (permalink / raw) To: Martin Liška Cc: Segher Boessenkool, Andrew Pinski, GCC Mailing List, GCC Patches On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: > > Hello. > > I've got a patch candidate that leverages partial linking for a couple of selected object files. > > I'm sending make all-host- jX results for my machine: > > before: 3m18s (user 32m52s) > https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg > > after: 2m57m (user 35m) > https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg > > One can utilize it with: > make -j16 all-host PARTIAL_LTO=1 > > @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? > One can also tweak --param=lto-partitions=16 param value. > > Thoughts? You're LTO linking multiple objects here - that's almost as if you were doing this for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the libbackend.a rule do a similar partial link trick. That gets you half of a LTO bootstrap then. So why did you go from applying this per-file to multiple files? Does $(LINKER) have a proper rule to pick up a jobserver? When upstreaming in any form you probably have to gate it on bootstrap-lto being not active. Richard. > Thanks, > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 12:54 ` Richard Biener @ 2021-05-20 13:06 ` Martin Liška 2021-05-20 13:16 ` Richard Biener 2021-05-21 8:43 ` Martin Liška 1 sibling, 1 reply; 21+ messages in thread From: Martin Liška @ 2021-05-20 13:06 UTC (permalink / raw) To: Richard Biener Cc: Segher Boessenkool, Andrew Pinski, GCC Mailing List, GCC Patches On 5/20/21 2:54 PM, Richard Biener wrote: > So why did you go from applying this per-file to multiple files? When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols: /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]' /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]' /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]' Any idea what was I doing wrong? Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 13:06 ` Martin Liška @ 2021-05-20 13:16 ` Richard Biener 2021-05-20 13:22 ` Richard Biener 2021-06-12 15:55 ` Giuliano Belinassi 0 siblings, 2 replies; 21+ messages in thread From: Richard Biener @ 2021-05-20 13:16 UTC (permalink / raw) To: Martin Liška, Jan Hubicka Cc: Segher Boessenkool, Andrew Pinski, GCC Mailing List, GCC Patches On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote: > > On 5/20/21 2:54 PM, Richard Biener wrote: > > So why did you go from applying this per-file to multiple files? > > When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols: > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]' > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]' > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]' > > Any idea what was I doing wrong? Nothing in particular I think - you're just hitting the issue that LTO produces new symbols and that those can obviously clash. Giuliano hit the very same issue. When not doing partial links those internal symbols pose no problem, but with -r -flinker-output=nolto-rel and re-linking the produced objects they obviously do. ELF has no solution for this though, but I think we could strip those from the partially linked object - if WPA would give us a list of objects the link step could postprocess the object with objcopy or maybe a custom linker script could do the trick as well. So your workaround is to only ever have a single LTO produced object file participating in the final links ;) Richard. > > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 13:16 ` Richard Biener @ 2021-05-20 13:22 ` Richard Biener 2021-05-20 15:55 ` Jan Hubicka 2021-06-12 15:55 ` Giuliano Belinassi 1 sibling, 1 reply; 21+ messages in thread From: Richard Biener @ 2021-05-20 13:22 UTC (permalink / raw) To: Martin Liška, Jan Hubicka Cc: Segher Boessenkool, Andrew Pinski, GCC Mailing List, GCC Patches On Thu, May 20, 2021 at 3:16 PM Richard Biener <richard.guenther@gmail.com> wrote: > > On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote: > > > > On 5/20/21 2:54 PM, Richard Biener wrote: > > > So why did you go from applying this per-file to multiple files? > > > > When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols: > > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]' > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]' > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]' > > > > Any idea what was I doing wrong? > > Nothing in particular I think - you're just hitting the issue that LTO > produces new symbols and that those > can obviously clash. Giuliano hit the very same issue. When not > doing partial links those internal > symbols pose no problem, but with -r -flinker-output=nolto-rel and > re-linking the produced objects > they obviously do. ELF has no solution for this though, but I think > we could strip those from the > partially linked object - if WPA would give us a list of objects the > link step could postprocess > the object with objcopy or maybe a custom linker script could do the > trick as well. Oh, and the "best" solution would be to avoid involving the linker when doing -r -flinker-output=nolto-rel but instead have the assembler produce the single object from the multiple LTRANS assembly snippets which could then use local labels instead of symbols for these. > So your workaround is to only ever have a single LTO produced object > file participating in the > final links ;) > > Richard. > > > > > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 13:22 ` Richard Biener @ 2021-05-20 15:55 ` Jan Hubicka 2021-05-21 8:29 ` Martin Liška 0 siblings, 1 reply; 21+ messages in thread From: Jan Hubicka @ 2021-05-20 15:55 UTC (permalink / raw) To: Richard Biener Cc: Martin Liška, GCC Mailing List, GCC Patches, Segher Boessenkool > On Thu, May 20, 2021 at 3:16 PM Richard Biener > <richard.guenther@gmail.com> wrote: > > > > On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote: > > > > > > On 5/20/21 2:54 PM, Richard Biener wrote: > > > > So why did you go from applying this per-file to multiple files? > > > > > > When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols: > > > > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]' > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]' > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]' > > > > > > Any idea what was I doing wrong? > > > > Nothing in particular I think - you're just hitting the issue that LTO > > produces new symbols and that those > > can obviously clash. Giuliano hit the very same issue. When not > > doing partial links those internal > > symbols pose no problem, but with -r -flinker-output=nolto-rel and > > re-linking the produced objects > > they obviously do. ELF has no solution for this though, but I think > > we could strip those from the > > partially linked object - if WPA would give us a list of objects the > > link step could postprocess > > the object with objcopy or maybe a custom linker script could do the > > trick as well. > > Oh, and the "best" solution would be to avoid involving the linker > when doing -r -flinker-output=nolto-rel but instead have the assembler > produce the single object from the multiple LTRANS assembly snippets > which could then use local labels instead of symbols for these. Quick solution is to also modify partitioner to use the local symbol names when doing incremental linking (those mixing in source code and random seeds) to avoid clashes. Honza > > > So your workaround is to only ever have a single LTO produced object > > file participating in the > > final links ;) > > > > Richard. > > > > > > > > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 15:55 ` Jan Hubicka @ 2021-05-21 8:29 ` Martin Liška 2021-06-01 7:31 ` Martin Liška ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Martin Liška @ 2021-05-21 8:29 UTC (permalink / raw) To: Jan Hubicka, Richard Biener Cc: GCC Mailing List, GCC Patches, Segher Boessenkool [-- Attachment #1: Type: text/plain, Size: 385 bytes --] On 5/20/21 5:55 PM, Jan Hubicka wrote: > Quick solution is to also modify partitioner to use the local symbol > names when doing incremental linking (those mixing in source code and > random seeds) to avoid clashes. Good hint. I added hash based on object file name (I don't want to handle proper string escaping) and -frandom-seed. What do you think about the patch? Thanks, Martin [-- Attachment #2: 0001-LTO-add-lto_priv-suffixfor-LTO_LINKER_OUTPUT_NOLTORE.patch --] [-- Type: text/x-patch, Size: 1829 bytes --] From 372d2944571906932fd1419bfc51a949d67b857e Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Fri, 21 May 2021 10:25:49 +0200 Subject: [PATCH] LTO: add lto_priv suffixfor LTO_LINKER_OUTPUT_NOLTOREL. gcc/lto/ChangeLog: * lto-partition.c (privatize_symbol_name_1): Add random suffix based on hash of the object file and -frandom-seed. --- gcc/lto/lto-partition.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/gcc/lto/lto-partition.c b/gcc/lto/lto-partition.c index 15761ac9eb5..fef48c869a2 100644 --- a/gcc/lto/lto-partition.c +++ b/gcc/lto/lto-partition.c @@ -35,6 +35,7 @@ along with GCC; see the file COPYING3. If not see #include "ipa-fnsummary.h" #include "lto-partition.h" #include "sreal.h" +#include "toplev.h" vec<ltrans_partition> ltrans_partitions; @@ -941,9 +942,23 @@ privatize_symbol_name_1 (symtab_node *node, tree decl) name = maybe_rewrite_identifier (name); unsigned &clone_number = lto_clone_numbers->get_or_insert (name); - symtab->change_decl_assembler_name (decl, - clone_function_name ( - name, "lto_priv", clone_number)); + + char *suffix = NULL; + if (flag_lto_linker_output == LTO_LINKER_OUTPUT_NOLTOREL) + { + hashval_t fnhash = 0; + if (node->lto_file_data != NULL) + fnhash = htab_hash_string (node->lto_file_data->file_name); + suffix = XNEWVEC (char, 128); + char sep = symbol_table::symbol_suffix_separator (); + sprintf (suffix, "lto_priv%c%u%c%" PRIu64, sep, fnhash, sep, + (unsigned HOST_WIDE_INT)get_random_seed (false)); + } + + tree clone + = clone_function_name (name, suffix ? suffix : "lto_priv", clone_number); + symtab->change_decl_assembler_name (decl, clone); + free (suffix); clone_number++; if (node->lto_file_data) -- 2.31.1 ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-21 8:29 ` Martin Liška @ 2021-06-01 7:31 ` Martin Liška 2021-06-23 13:53 ` Martin Liška 2021-08-22 13:09 ` Jan Hubicka 2 siblings, 0 replies; 21+ messages in thread From: Martin Liška @ 2021-06-01 7:31 UTC (permalink / raw) To: Jan Hubicka, Richard Biener Cc: GCC Mailing List, GCC Patches, Segher Boessenkool PING^1 On 5/21/21 10:29 AM, Martin Liška wrote: > On 5/20/21 5:55 PM, Jan Hubicka wrote: >> Quick solution is to also modify partitioner to use the local symbol >> names when doing incremental linking (those mixing in source code and >> random seeds) to avoid clashes. > > Good hint. I added hash based on object file name (I don't want to handle > proper string escaping) and -frandom-seed. > > What do you think about the patch? > Thanks, > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-21 8:29 ` Martin Liška 2021-06-01 7:31 ` Martin Liška @ 2021-06-23 13:53 ` Martin Liška 2021-08-16 13:58 ` Martin Liška 2021-08-22 13:09 ` Jan Hubicka 2 siblings, 1 reply; 21+ messages in thread From: Martin Liška @ 2021-06-23 13:53 UTC (permalink / raw) To: Jan Hubicka, Richard Biener Cc: GCC Mailing List, GCC Patches, Segher Boessenkool On 5/21/21 10:29 AM, Martin Liška wrote: > On 5/20/21 5:55 PM, Jan Hubicka wrote: >> Quick solution is to also modify partitioner to use the local symbol >> names when doing incremental linking (those mixing in source code and >> random seeds) to avoid clashes. > > Good hint. I added hash based on object file name (I don't want to handle > proper string escaping) and -frandom-seed. > > What do you think about the patch? > Thanks, > Martin @Honza: Can you please take a look at this patch? Cheers, Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-06-23 13:53 ` Martin Liška @ 2021-08-16 13:58 ` Martin Liška 2021-08-20 14:54 ` Martin Liška 0 siblings, 1 reply; 21+ messages in thread From: Martin Liška @ 2021-08-16 13:58 UTC (permalink / raw) To: Jan Hubicka, Richard Biener Cc: GCC Mailing List, GCC Patches, Segher Boessenkool PING^2 @Honza: Can you please review the change? Martin On 6/23/21 3:53 PM, Martin Liška wrote: > On 5/21/21 10:29 AM, Martin Liška wrote: >> On 5/20/21 5:55 PM, Jan Hubicka wrote: >>> Quick solution is to also modify partitioner to use the local symbol >>> names when doing incremental linking (those mixing in source code and >>> random seeds) to avoid clashes. >> >> Good hint. I added hash based on object file name (I don't want to handle >> proper string escaping) and -frandom-seed. >> >> What do you think about the patch? >> Thanks, >> Martin > > @Honza: Can you please take a look at this patch? > > Cheers, > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-08-16 13:58 ` Martin Liška @ 2021-08-20 14:54 ` Martin Liška 0 siblings, 0 replies; 21+ messages in thread From: Martin Liška @ 2021-08-20 14:54 UTC (permalink / raw) To: Jan Hubicka, Richard Biener Cc: GCC Mailing List, GCC Patches, Segher Boessenkool, giuliano.belinassi@usp.br >> Giuliano Belinassi On 8/16/21 3:58 PM, Martin Liška wrote: > PING^2 > > @Honza: Can you please review the change? I've tested the patch and apparently it's not enough for {gimple,generic}-match.o not clashing in symbol names. Apparently there are more IPA clones that collide. Leaving that for now. Martin > > Martin > > On 6/23/21 3:53 PM, Martin Liška wrote: >> On 5/21/21 10:29 AM, Martin Liška wrote: >>> On 5/20/21 5:55 PM, Jan Hubicka wrote: >>>> Quick solution is to also modify partitioner to use the local symbol >>>> names when doing incremental linking (those mixing in source code and >>>> random seeds) to avoid clashes. >>> >>> Good hint. I added hash based on object file name (I don't want to handle >>> proper string escaping) and -frandom-seed. >>> >>> What do you think about the patch? >>> Thanks, >>> Martin >> >> @Honza: Can you please take a look at this patch? >> >> Cheers, >> Martin > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-21 8:29 ` Martin Liška 2021-06-01 7:31 ` Martin Liška 2021-06-23 13:53 ` Martin Liška @ 2021-08-22 13:09 ` Jan Hubicka 2 siblings, 0 replies; 21+ messages in thread From: Jan Hubicka @ 2021-08-22 13:09 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, GCC Mailing List, GCC Patches, Segher Boessenkool > Good hint. I added hash based on object file name (I don't want to handle > proper string escaping) and -frandom-seed. > > What do you think about the patch? Sorry for taking so long - I remember I was sending reply earlier but it seems I only wrote it and never sent. > Thanks, > Martin > From 372d2944571906932fd1419bfc51a949d67b857e Mon Sep 17 00:00:00 2001 > From: Martin Liska <mliska@suse.cz> > Date: Fri, 21 May 2021 10:25:49 +0200 > Subject: [PATCH] LTO: add lto_priv suffixfor LTO_LINKER_OUTPUT_NOLTOREL. > > gcc/lto/ChangeLog: > > * lto-partition.c (privatize_symbol_name_1): Add random suffix > based on hash of the object file and -frandom-seed. > --- > gcc/lto/lto-partition.c | 21 ++++++++++++++++++--- > 1 file changed, 18 insertions(+), 3 deletions(-) > > diff --git a/gcc/lto/lto-partition.c b/gcc/lto/lto-partition.c > index 15761ac9eb5..fef48c869a2 100644 > --- a/gcc/lto/lto-partition.c > +++ b/gcc/lto/lto-partition.c > @@ -35,6 +35,7 @@ along with GCC; see the file COPYING3. If not see > #include "ipa-fnsummary.h" > #include "lto-partition.h" > #include "sreal.h" > +#include "toplev.h" > > vec<ltrans_partition> ltrans_partitions; > > @@ -941,9 +942,23 @@ privatize_symbol_name_1 (symtab_node *node, tree decl) > > name = maybe_rewrite_identifier (name); > unsigned &clone_number = lto_clone_numbers->get_or_insert (name); > - symtab->change_decl_assembler_name (decl, > - clone_function_name ( > - name, "lto_priv", clone_number)); > + > + char *suffix = NULL; > + if (flag_lto_linker_output == LTO_LINKER_OUTPUT_NOLTOREL) > + { > + hashval_t fnhash = 0; > + if (node->lto_file_data != NULL) > + fnhash = htab_hash_string (node->lto_file_data->file_name); > + suffix = XNEWVEC (char, 128); > + char sep = symbol_table::symbol_suffix_separator (); > + sprintf (suffix, "lto_priv%c%u%c%" PRIu64, sep, fnhash, sep, > + (unsigned HOST_WIDE_INT)get_random_seed (false)); We have get_file_function_name which does similar work but also working without random seeds. Perhaps we can reuse it here: use get_file_function_name once and use it as prefix or compute hash from it. The logic to get unique symbol name is not completely easy and it would be better to not duplciate it. Patch is OK with that change (and indeed it is bugfix - even if it is relatively little used partial linking of LTO objects into non-LTO should be supported and working). Honza > + } > + > + tree clone > + = clone_function_name (name, suffix ? suffix : "lto_priv", clone_number); > + symtab->change_decl_assembler_name (decl, clone); > + free (suffix); > clone_number++; > > if (node->lto_file_data) > -- > 2.31.1 > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 13:16 ` Richard Biener 2021-05-20 13:22 ` Richard Biener @ 2021-06-12 15:55 ` Giuliano Belinassi 1 sibling, 0 replies; 21+ messages in thread From: Giuliano Belinassi @ 2021-06-12 15:55 UTC (permalink / raw) To: Richard Biener, Martin Liška, Jan Hubicka Cc: GCC Mailing List, GCC Patches, Segher Boessenkool Hi, all. Please CC me when I am mentioned into a mail. On Thu, 2021-05-20 at 15:16 +0200, Richard Biener via Gcc wrote: > On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote: > > > > On 5/20/21 2:54 PM, Richard Biener wrote: > > > So why did you go from applying this per-file to multiple files? > > > > When I did per-file for {gimple,generic}-match.c I hit the > > following issue with lto.priv symbols: > > > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse- > > linux/bin/ld: error: libbackend.a(generic-match.o): multiple > > definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone > > .lto_priv.0]' > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse- > > linux/bin/ld: libbackend.a(gimple-match.o): previous definition > > here > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse- > > linux/bin/ld: error: libbackend.a(generic-match.o): multiple > > definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone > > .part.0] [clone .lto_priv.0]' > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse- > > linux/bin/ld: libbackend.a(gimple-match.o): previous definition > > here > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse- > > linux/bin/ld: error: libbackend.a(generic-match.o): multiple > > definition of 'vec<constructor_elt, va_gc, > > vl_embed>::operator[](unsigned int) [clone .part.0] [clone > > .lto_priv.0]' > > > > Any idea what was I doing wrong? > > Nothing in particular I think - you're just hitting the issue that > LTO > produces new symbols and that those > can obviously clash. Giuliano hit the very same issue. When not > doing partial links those internal > symbols pose no problem, but with -r -flinker-output=nolto-rel and > re-linking the produced objects > they obviously do. ELF has no solution for this though, but I think > we could strip those from the > partially linked object - if WPA would give us a list of objects the > link step could postprocess > the object with objcopy or maybe a custom linker script could do the > trick as well. I've "fixed" this issue in my branch by mangling any promoted to public symbol. I've also disabled the "ipa-split" pass in the paper branch because of some created symbols which I was not able to fix in time. Perhaps this goes away if you disable it. Perhaps we should work on getting the autopar branch merged into trunk. There are several issues which must be fixed and I don't think it will be ready for this next release. The main ones that I remember from the top of my head: 1- Fix the driver to use SPEC language for the multiple required calls to `as`, instead of injecting code for that directly on the `void execute()` function. 2- Merge my custom partitioner for using the default LTO partitioner. The default LTO partitioner were hitting the assertions about COMDAT being split into multiple partitions. 3- Fix the issue with the ipa-split pass. Perhaps we should further explore avoiding partial linking altogether and concat the assembler files. Thank you, Giuliano. > > So your workaround is to only ever have a single LTO produced object > file participating in the > final links ;) > > Richard. > > > > > Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-20 12:54 ` Richard Biener 2021-05-20 13:06 ` Martin Liška @ 2021-05-21 8:43 ` Martin Liška 2021-05-21 12:35 ` David Edelsohn 2021-06-01 7:33 ` Martin Liška 1 sibling, 2 replies; 21+ messages in thread From: Martin Liška @ 2021-05-21 8:43 UTC (permalink / raw) To: Richard Biener Cc: Segher Boessenkool, Andrew Pinski, GCC Mailing List, GCC Patches [-- Attachment #1: Type: text/plain, Size: 4739 bytes --] On 5/20/21 2:54 PM, Richard Biener wrote: > On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: >> >> Hello. >> >> I've got a patch candidate that leverages partial linking for a couple of selected object files. >> >> I'm sending make all-host- jX results for my machine: >> >> before: 3m18s (user 32m52s) >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg >> >> after: 2m57m (user 35m) >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg >> >> One can utilize it with: >> make -j16 all-host PARTIAL_LTO=1 >> >> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? >> One can also tweak --param=lto-partitions=16 param value. >> >> Thoughts? > > You're LTO linking multiple objects here - that's almost as if you > were doing this > for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the > libbackend.a rule do a similar partial link trick. Yeah, apart from that one can't likely do partial linking for an archive: $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped compilation terminated. while ld.bfd immediately finishes. > > That gets you half of a LTO bootstrap then. > > So why did you go from applying this per-file to multiple files? Does $(LINKER) > have a proper rule to pick up a jobserver? > > When upstreaming in any form you probably have to gate it on bootstrap-lto > being not active. Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? One more thing I face is broken dependency: $ make clean && make -j32 PARTIAL_LTO=1 g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto In file included from ./tm.h:26, from /home/marxin/Programming/gcc/gcc/backend.h:28, from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, from generic-match.c:4: /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory 2286 | #include "insn-attr-common.h" | ^~~~~~~~~~~~~~~~~~~~ compilation terminated. make: *** [Makefile:2678: generic-match-lto.o] Error 1 make: *** Waiting for unfinished jobs.... In file included from ./tm.h:26, from /home/marxin/Programming/gcc/gcc/backend.h:28, from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, from gimple-match.c:4: /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory 2286 | #include "insn-attr-common.h" | ^~~~~~~~~~~~~~~~~~~~ I explicitly added: gimple-match.o: gimple-match.c $(generated_files) generic-match.o: generic-match.c $(generated_files) But it's not obeyed. Martin > > Richard. > >> Thanks, >> Martin [-- Attachment #2: 0001-Try-LTO-partial-linking.patch --] [-- Type: text/x-patch, Size: 2452 bytes --] From b209c7aea76ceb8eadcf5843f30299fc8041a579 Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Thu, 20 May 2021 14:29:35 +0200 Subject: [PATCH] Try LTO partial linking. --- gcc/Makefile.in | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 1164554e6d6..decfea7412b 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -220,7 +220,9 @@ libgcov-util.o-warn = -Wno-error libgcov-driver-tool.o-warn = -Wno-error libgcov-merge-tool.o-warn = -Wno-error gimple-match.o-warn = -Wno-unused +gimple-match-lto.o-warn = -Wno-unused generic-match.o-warn = -Wno-unused +generic-match-lto.o-warn = -Wno-unused dfp.o-warn = -Wno-strict-aliasing # All warnings have to be shut off in stage1 if the compiler used then @@ -1282,12 +1284,10 @@ ANALYZER_OBJS = \ # will build them sooner, because they are large and otherwise tend to be # the last objects to finish building. OBJS = \ - gimple-match.o \ - generic-match.o \ + common-base.a \ insn-attrtab.o \ insn-automata.o \ insn-dfatab.o \ - insn-emit.o \ insn-extract.o \ insn-latencytab.o \ insn-modes.o \ @@ -1295,7 +1295,6 @@ OBJS = \ insn-output.o \ insn-peep.o \ insn-preds.o \ - insn-recog.o \ insn-enums.o \ ggc-page.o \ adjust-alignment.o \ @@ -2627,6 +2626,32 @@ s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd generic-match.c $(STAMP) s-match +gimple-match.o: gimple-match.c $(generated_files) +generic-match.o: generic-match.c $(generated_files) + +ifdef PARTIAL_LTO +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r +LTO_FLAGS = -flto + +gimple-match-lto.o: gimple-match.c + $(COMPILE) $< $(LTO_FLAGS) +generic-match-lto.o: generic-match.c + $(COMPILE) $< $(LTO_FLAGS) +insn-recog-lto.o: insn-recog.c + $(COMPILE) $< $(LTO_FLAGS) +insn-emit-lto.o: insn-emit.c + $(COMPILE) $< $(LTO_FLAGS) + +common-base.a: gimple-match-lto.o generic-match-lto.o insn-recog-lto.o insn-emit-lto.o + -rm -rf $@ + $(LINKER) $^ $(LTO_LINKER_FLAGS) -o common-base.o + $(AR) $(AR_FLAGS)T $@ common-base.o +else +common-base.a: gimple-match.o generic-match.o insn-recog.o insn-emit.o + -rm -rf $@ + $(AR) $(AR_FLAGS)T $@ $^ +endif + GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \ $(host_xm_file_list) \ $(tm_file_list) $(HASHTAB_H) $(SPLAY_TREE_H) $(srcdir)/bitmap.h \ -- 2.31.1 ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-21 8:43 ` Martin Liška @ 2021-05-21 12:35 ` David Edelsohn 2021-05-24 8:07 ` Martin Liška 2021-06-01 7:33 ` Martin Liška 1 sibling, 1 reply; 21+ messages in thread From: David Edelsohn @ 2021-05-21 12:35 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, GCC Mailing List, GCC Patches, Segher Boessenkool On Fri, May 21, 2021 at 5:25 AM Martin Liška <mliska@suse.cz> wrote: > > On 5/20/21 2:54 PM, Richard Biener wrote: > > On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> Hello. > >> > >> I've got a patch candidate that leverages partial linking for a couple of selected object files. > >> > >> I'm sending make all-host- jX results for my machine: > >> > >> before: 3m18s (user 32m52s) > >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg > >> > >> after: 2m57m (user 35m) > >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg > >> > >> One can utilize it with: > >> make -j16 all-host PARTIAL_LTO=1 > >> > >> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? > >> One can also tweak --param=lto-partitions=16 param value. > >> > >> Thoughts? > > > > You're LTO linking multiple objects here - that's almost as if you > > were doing this > > for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the > > libbackend.a rule do a similar partial link trick. > > Yeah, apart from that one can't likely do partial linking for an archive: > > $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a > collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped > compilation terminated. > > while ld.bfd immediately finishes. > > > > > That gets you half of a LTO bootstrap then. > > > > So why did you go from applying this per-file to multiple files? Does $(LINKER) > > have a proper rule to pick up a jobserver? > > > > When upstreaming in any form you probably have to gate it on bootstrap-lto > > being not active. > > Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? > > One more thing I face is broken dependency: > $ make clean && make -j32 PARTIAL_LTO=1 > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto > > In file included from ./tm.h:26, > from /home/marxin/Programming/gcc/gcc/backend.h:28, > from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, > from generic-match.c:4: > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > 2286 | #include "insn-attr-common.h" > | ^~~~~~~~~~~~~~~~~~~~ > compilation terminated. > make: *** [Makefile:2678: generic-match-lto.o] Error 1 > make: *** Waiting for unfinished jobs.... > > In file included from ./tm.h:26, > from /home/marxin/Programming/gcc/gcc/backend.h:28, > from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, > from gimple-match.c:4: > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > 2286 | #include "insn-attr-common.h" > | ^~~~~~~~~~~~~~~~~~~~ > > I explicitly added: > gimple-match.o: gimple-match.c $(generated_files) > generic-match.o: generic-match.c $(generated_files) > > But it's not obeyed. Please remember that not all targets support LTO so a fallback to a non-partial-LTO build needs to be provided and automatically invoked for those targets. Thanks, David ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-21 12:35 ` David Edelsohn @ 2021-05-24 8:07 ` Martin Liška 0 siblings, 0 replies; 21+ messages in thread From: Martin Liška @ 2021-05-24 8:07 UTC (permalink / raw) To: David Edelsohn Cc: Richard Biener, GCC Mailing List, GCC Patches, Segher Boessenkool On 5/21/21 2:35 PM, David Edelsohn wrote: > Please remember that not all targets support LTO so a fallback to a > non-partial-LTO build needs to be provided and automatically invoked > for those targets. Sure, for now it's definitely going to be a opt-in, enabled by something like: make PARTIAL_LTO=1. Thanks, Martin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-05-21 8:43 ` Martin Liška 2021-05-21 12:35 ` David Edelsohn @ 2021-06-01 7:33 ` Martin Liška 2021-06-01 7:42 ` Richard Biener 1 sibling, 1 reply; 21+ messages in thread From: Martin Liška @ 2021-06-01 7:33 UTC (permalink / raw) To: Richard Biener; +Cc: GCC Mailing List, GCC Patches, Segher Boessenkool @Richi: Can you please reply to this email? On 5/21/21 10:43 AM, Martin Liška wrote: > On 5/20/21 2:54 PM, Richard Biener wrote: >> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: >>> >>> Hello. >>> >>> I've got a patch candidate that leverages partial linking for a couple of selected object files. >>> >>> I'm sending make all-host- jX results for my machine: >>> >>> before: 3m18s (user 32m52s) >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg >>> >>> after: 2m57m (user 35m) >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg >>> >>> One can utilize it with: >>> make -j16 all-host PARTIAL_LTO=1 >>> >>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? >>> One can also tweak --param=lto-partitions=16 param value. >>> >>> Thoughts? >> >> You're LTO linking multiple objects here - that's almost as if you >> were doing this >> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the >> libbackend.a rule do a similar partial link trick. > > Yeah, apart from that one can't likely do partial linking for an archive: > > $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a > collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped > compilation terminated. > > while ld.bfd immediately finishes. > >> >> That gets you half of a LTO bootstrap then. >> >> So why did you go from applying this per-file to multiple files? Does $(LINKER) >> have a proper rule to pick up a jobserver? >> >> When upstreaming in any form you probably have to gate it on bootstrap-lto >> being not active. > > Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? > > One more thing I face is broken dependency: > $ make clean && make -j32 PARTIAL_LTO=1 > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto > > In file included from ./tm.h:26, > from /home/marxin/Programming/gcc/gcc/backend.h:28, > from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, > from generic-match.c:4: > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > 2286 | #include "insn-attr-common.h" > | ^~~~~~~~~~~~~~~~~~~~ > compilation terminated. > make: *** [Makefile:2678: generic-match-lto.o] Error 1 > make: *** Waiting for unfinished jobs.... > > In file included from ./tm.h:26, > from /home/marxin/Programming/gcc/gcc/backend.h:28, > from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, > from gimple-match.c:4: > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > 2286 | #include "insn-attr-common.h" > | ^~~~~~~~~~~~~~~~~~~~ > > I explicitly added: > gimple-match.o: gimple-match.c $(generated_files) > generic-match.o: generic-match.c $(generated_files) > > But it's not obeyed. > > Martin > >> >> Richard. >> >>> Thanks, >>> Martin > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-06-01 7:33 ` Martin Liška @ 2021-06-01 7:42 ` Richard Biener 2021-06-01 11:25 ` Martin Liška 0 siblings, 1 reply; 21+ messages in thread From: Richard Biener @ 2021-06-01 7:42 UTC (permalink / raw) To: Martin Liška; +Cc: GCC Mailing List, GCC Patches, Segher Boessenkool On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote: > > @Richi: Can you please reply to this email? Not sure what I should add here? Honza suggested to mangle the promoted symbol names. I don't really like the idea to compile multiple TUs into one object. Also +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r why hard-code to 16 partitions? You're side-stepping the driver diagnostic by doing compile & link separately, but in the end we're going to want sth like Giulianos -fparallel-compile that works transparently from within the driver, so the "manual" operation should try to follow that or alternatively a driver-only wrapper around the "manual" processing could be added whose implementation can be optimized later. Why do you use -flto=auto? There should be a jobserver active. > On 5/21/21 10:43 AM, Martin Liška wrote: > > On 5/20/21 2:54 PM, Richard Biener wrote: > >> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: > >>> > >>> Hello. > >>> > >>> I've got a patch candidate that leverages partial linking for a couple of selected object files. > >>> > >>> I'm sending make all-host- jX results for my machine: > >>> > >>> before: 3m18s (user 32m52s) > >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg > >>> > >>> after: 2m57m (user 35m) > >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg > >>> > >>> One can utilize it with: > >>> make -j16 all-host PARTIAL_LTO=1 > >>> > >>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? > >>> One can also tweak --param=lto-partitions=16 param value. > >>> > >>> Thoughts? > >> > >> You're LTO linking multiple objects here - that's almost as if you > >> were doing this > >> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the > >> libbackend.a rule do a similar partial link trick. > > > > Yeah, apart from that one can't likely do partial linking for an archive: > > > > $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a > > collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped > > compilation terminated. > > > > while ld.bfd immediately finishes. > > > >> > >> That gets you half of a LTO bootstrap then. > >> > >> So why did you go from applying this per-file to multiple files? Does $(LINKER) > >> have a proper rule to pick up a jobserver? > >> > >> When upstreaming in any form you probably have to gate it on bootstrap-lto > >> being not active. > > > > Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? > > > > One more thing I face is broken dependency: > > $ make clean && make -j32 PARTIAL_LTO=1 > > > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto > > > > In file included from ./tm.h:26, > > from /home/marxin/Programming/gcc/gcc/backend.h:28, > > from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, > > from generic-match.c:4: > > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > > 2286 | #include "insn-attr-common.h" > > | ^~~~~~~~~~~~~~~~~~~~ > > compilation terminated. > > make: *** [Makefile:2678: generic-match-lto.o] Error 1 > > make: *** Waiting for unfinished jobs.... > > > > In file included from ./tm.h:26, > > from /home/marxin/Programming/gcc/gcc/backend.h:28, > > from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, > > from gimple-match.c:4: > > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > > 2286 | #include "insn-attr-common.h" > > | ^~~~~~~~~~~~~~~~~~~~ > > > > I explicitly added: > > gimple-match.o: gimple-match.c $(generated_files) > > generic-match.o: generic-match.c $(generated_files) > > > > But it's not obeyed. > > > > Martin > > > >> > >> Richard. > >> > >>> Thanks, > >>> Martin > > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-06-01 7:42 ` Richard Biener @ 2021-06-01 11:25 ` Martin Liška 2021-06-01 13:19 ` Richard Biener 0 siblings, 1 reply; 21+ messages in thread From: Martin Liška @ 2021-06-01 11:25 UTC (permalink / raw) To: Richard Biener; +Cc: GCC Mailing List, GCC Patches, Segher Boessenkool On 6/1/21 9:42 AM, Richard Biener wrote: > On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote: >> >> @Richi: Can you please reply to this email? > > Not sure what I should add here? Honza suggested to mangle the > promoted symbol names. Sure and I sent a patch for that. > I don't > really like the idea to compile multiple TUs into one object. Also What's problematic is that we'll have to wait for one another release to make it useful (if you don't want to build the current master with a snapshot compiler). > > +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 > -flinker-output=nolto-rel -r > > why hard-code to 16 partitions? You're side-stepping the driver > diagnostic by doing > compile & link separately, but in the end we're going to want sth like Giulianos > -fparallel-compile that works transparently from within the driver, so > the "manual" > operation should try to follow that or alternatively a driver-only > wrapper around the > "manual" processing could be added whose implementation can be optimized later. All right. Do you want me refreshing his -fparallel-compile option introduction? > > Why do you use -flto=auto? There should be a jobserver active. Yes, that should not be needed. Martin > >> On 5/21/21 10:43 AM, Martin Liška wrote: >>> On 5/20/21 2:54 PM, Richard Biener wrote: >>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: >>>>> >>>>> Hello. >>>>> >>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files. >>>>> >>>>> I'm sending make all-host- jX results for my machine: >>>>> >>>>> before: 3m18s (user 32m52s) >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg >>>>> >>>>> after: 2m57m (user 35m) >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg >>>>> >>>>> One can utilize it with: >>>>> make -j16 all-host PARTIAL_LTO=1 >>>>> >>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? >>>>> One can also tweak --param=lto-partitions=16 param value. >>>>> >>>>> Thoughts? >>>> >>>> You're LTO linking multiple objects here - that's almost as if you >>>> were doing this >>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the >>>> libbackend.a rule do a similar partial link trick. >>> >>> Yeah, apart from that one can't likely do partial linking for an archive: >>> >>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a >>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped >>> compilation terminated. >>> >>> while ld.bfd immediately finishes. >>> >>>> >>>> That gets you half of a LTO bootstrap then. >>>> >>>> So why did you go from applying this per-file to multiple files? Does $(LINKER) >>>> have a proper rule to pick up a jobserver? >>>> >>>> When upstreaming in any form you probably have to gate it on bootstrap-lto >>>> being not active. >>> >>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? >>> >>> One more thing I face is broken dependency: >>> $ make clean && make -j32 PARTIAL_LTO=1 >>> >>> g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto >>> g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto >>> >>> In file included from ./tm.h:26, >>> from /home/marxin/Programming/gcc/gcc/backend.h:28, >>> from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, >>> from generic-match.c:4: >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory >>> 2286 | #include "insn-attr-common.h" >>> | ^~~~~~~~~~~~~~~~~~~~ >>> compilation terminated. >>> make: *** [Makefile:2678: generic-match-lto.o] Error 1 >>> make: *** Waiting for unfinished jobs.... >>> >>> In file included from ./tm.h:26, >>> from /home/marxin/Programming/gcc/gcc/backend.h:28, >>> from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, >>> from gimple-match.c:4: >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory >>> 2286 | #include "insn-attr-common.h" >>> | ^~~~~~~~~~~~~~~~~~~~ >>> >>> I explicitly added: >>> gimple-match.o: gimple-match.c $(generated_files) >>> generic-match.o: generic-match.c $(generated_files) >>> >>> But it's not obeyed. >>> >>> Martin >>> >>>> >>>> Richard. >>>> >>>>> Thanks, >>>>> Martin >>> >> ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-06-01 11:25 ` Martin Liška @ 2021-06-01 13:19 ` Richard Biener 2021-08-20 14:57 ` Martin Liška 0 siblings, 1 reply; 21+ messages in thread From: Richard Biener @ 2021-06-01 13:19 UTC (permalink / raw) To: Martin Liška; +Cc: GCC Mailing List, GCC Patches, Segher Boessenkool On Tue, Jun 1, 2021 at 1:25 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/1/21 9:42 AM, Richard Biener wrote: > > On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote: > >> > >> @Richi: Can you please reply to this email? > > > > Not sure what I should add here? Honza suggested to mangle the > > promoted symbol names. > > Sure and I sent a patch for that. > > > I don't > > really like the idea to compile multiple TUs into one object. Also > > What's problematic is that we'll have to wait for one another release to make it useful > (if you don't want to build the current master with a snapshot compiler). IMHO it's a bugfix. Note that I'm not sure what the intent of the change is. If it is to speedup bootstrap then using LTO bootstrap would do the trick as well (and better) if we'd simply process all of libbackend.a this way (and thus avoid re-linking that once for each frontend). If it is to speedup dev (re-)builds then dragging in more files will make it build longer since for example insn-recog.c may be unchanged but gimple-match.c not. > > +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 > > -flinker-output=nolto-rel -r > > > > why hard-code to 16 partitions? You're side-stepping the driver > > diagnostic by doing > > compile & link separately, but in the end we're going to want sth like Giulianos > > -fparallel-compile that works transparently from within the driver, so > > the "manual" > > operation should try to follow that or alternatively a driver-only > > wrapper around the > > "manual" processing could be added whose implementation can be optimized later. > > All right. Do you want me refreshing his -fparallel-compile option introduction? I'm not sure if we've arrived at mergeable state - but if it's reasonably possible to hide s/-fparallel-compile/-flto -r -flinker-output=nolto-rel/ split into compile & link parts (avoiding the diagnostic on -flinker-output) in the driver then I think that's a very reasonable first step (after fixing the symbol privatization issue). The GSOC project then was to elide the IL streaming from the high-level operation. Richard, > > > > Why do you use -flto=auto? There should be a jobserver active. > > Yes, that should not be needed. > > Martin > > > > >> On 5/21/21 10:43 AM, Martin Liška wrote: > >>> On 5/20/21 2:54 PM, Richard Biener wrote: > >>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: > >>>>> > >>>>> Hello. > >>>>> > >>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files. > >>>>> > >>>>> I'm sending make all-host- jX results for my machine: > >>>>> > >>>>> before: 3m18s (user 32m52s) > >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg > >>>>> > >>>>> after: 2m57m (user 35m) > >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg > >>>>> > >>>>> One can utilize it with: > >>>>> make -j16 all-host PARTIAL_LTO=1 > >>>>> > >>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? > >>>>> One can also tweak --param=lto-partitions=16 param value. > >>>>> > >>>>> Thoughts? > >>>> > >>>> You're LTO linking multiple objects here - that's almost as if you > >>>> were doing this > >>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the > >>>> libbackend.a rule do a similar partial link trick. > >>> > >>> Yeah, apart from that one can't likely do partial linking for an archive: > >>> > >>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a > >>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped > >>> compilation terminated. > >>> > >>> while ld.bfd immediately finishes. > >>> > >>>> > >>>> That gets you half of a LTO bootstrap then. > >>>> > >>>> So why did you go from applying this per-file to multiple files? Does $(LINKER) > >>>> have a proper rule to pick up a jobserver? > >>>> > >>>> When upstreaming in any form you probably have to gate it on bootstrap-lto > >>>> being not active. > >>> > >>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? > >>> > >>> One more thing I face is broken dependency: > >>> $ make clean && make -j32 PARTIAL_LTO=1 > >>> > >>> g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto > >>> g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto > >>> > >>> In file included from ./tm.h:26, > >>> from /home/marxin/Programming/gcc/gcc/backend.h:28, > >>> from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, > >>> from generic-match.c:4: > >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > >>> 2286 | #include "insn-attr-common.h" > >>> | ^~~~~~~~~~~~~~~~~~~~ > >>> compilation terminated. > >>> make: *** [Makefile:2678: generic-match-lto.o] Error 1 > >>> make: *** Waiting for unfinished jobs.... > >>> > >>> In file included from ./tm.h:26, > >>> from /home/marxin/Programming/gcc/gcc/backend.h:28, > >>> from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, > >>> from gimple-match.c:4: > >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory > >>> 2286 | #include "insn-attr-common.h" > >>> | ^~~~~~~~~~~~~~~~~~~~ > >>> > >>> I explicitly added: > >>> gimple-match.o: gimple-match.c $(generated_files) > >>> generic-match.o: generic-match.c $(generated_files) > >>> > >>> But it's not obeyed. > >>> > >>> Martin > >>> > >>>> > >>>> Richard. > >>>> > >>>>> Thanks, > >>>>> Martin > >>> > >> > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) 2021-06-01 13:19 ` Richard Biener @ 2021-08-20 14:57 ` Martin Liška 0 siblings, 0 replies; 21+ messages in thread From: Martin Liška @ 2021-08-20 14:57 UTC (permalink / raw) To: Richard Biener Cc: GCC Mailing List, GCC Patches, Segher Boessenkool, Giuliano Belinassi On 6/1/21 3:19 PM, Richard Biener wrote: > On Tue, Jun 1, 2021 at 1:25 PM Martin Liška <mliska@suse.cz> wrote: >> >> On 6/1/21 9:42 AM, Richard Biener wrote: >>> On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote: >>>> >>>> @Richi: Can you please reply to this email? >>> >>> Not sure what I should add here? Honza suggested to mangle the >>> promoted symbol names. >> >> Sure and I sent a patch for that. >> >>> I don't >>> really like the idea to compile multiple TUs into one object. Also >> >> What's problematic is that we'll have to wait for one another release to make it useful >> (if you don't want to build the current master with a snapshot compiler). > > IMHO it's a bugfix. Note that I'm not sure what the intent of the change is. > If it is to speedup bootstrap then using LTO bootstrap would do the trick > as well (and better) if we'd simply process all of libbackend.a this way > (and thus avoid re-linking that once for each frontend). When building a GCC package, we intentionally re-link them with all FEs. > If it is to speedup > dev (re-)builds then dragging in more files will make it build longer since > for example insn-recog.c may be unchanged but gimple-match.c not. Yes, the original motivation was a speed up of a dev. build and yes, the shown example is problematic. Right now, I'm leaving that as I'm not interested enough in the parallel build of a simple source file. Martin > >>> +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 >>> -flinker-output=nolto-rel -r >>> >>> why hard-code to 16 partitions? You're side-stepping the driver >>> diagnostic by doing >>> compile & link separately, but in the end we're going to want sth like Giulianos >>> -fparallel-compile that works transparently from within the driver, so >>> the "manual" >>> operation should try to follow that or alternatively a driver-only >>> wrapper around the >>> "manual" processing could be added whose implementation can be optimized later. >> >> All right. Do you want me refreshing his -fparallel-compile option introduction? > > I'm not sure if we've arrived at mergeable state - but if it's > reasonably possible > to hide s/-fparallel-compile/-flto -r -flinker-output=nolto-rel/ split > into compile & link > parts (avoiding the diagnostic on -flinker-output) in the driver then > I think that's > a very reasonable first step (after fixing the symbol privatization issue). The > GSOC project then was to elide the IL streaming from the high-level operation. > > Richard, > >>> >>> Why do you use -flto=auto? There should be a jobserver active. >> >> Yes, that should not be needed. >> >> Martin >> >>> >>>> On 5/21/21 10:43 AM, Martin Liška wrote: >>>>> On 5/20/21 2:54 PM, Richard Biener wrote: >>>>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote: >>>>>>> >>>>>>> Hello. >>>>>>> >>>>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files. >>>>>>> >>>>>>> I'm sending make all-host- jX results for my machine: >>>>>>> >>>>>>> before: 3m18s (user 32m52s) >>>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg >>>>>>> >>>>>>> after: 2m57m (user 35m) >>>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg >>>>>>> >>>>>>> One can utilize it with: >>>>>>> make -j16 all-host PARTIAL_LTO=1 >>>>>>> >>>>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap? >>>>>>> One can also tweak --param=lto-partitions=16 param value. >>>>>>> >>>>>>> Thoughts? >>>>>> >>>>>> You're LTO linking multiple objects here - that's almost as if you >>>>>> were doing this >>>>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the >>>>>> libbackend.a rule do a similar partial link trick. >>>>> >>>>> Yeah, apart from that one can't likely do partial linking for an archive: >>>>> >>>>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a >>>>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped >>>>> compilation terminated. >>>>> >>>>> while ld.bfd immediately finishes. >>>>> >>>>>> >>>>>> That gets you half of a LTO bootstrap then. >>>>>> >>>>>> So why did you go from applying this per-file to multiple files? Does $(LINKER) >>>>>> have a proper rule to pick up a jobserver? >>>>>> >>>>>> When upstreaming in any form you probably have to gate it on bootstrap-lto >>>>>> being not active. >>>>> >>>>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right? >>>>> >>>>> One more thing I face is broken dependency: >>>>> $ make clean && make -j32 PARTIAL_LTO=1 >>>>> >>>>> g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto >>>>> g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto >>>>> >>>>> In file included from ./tm.h:26, >>>>> from /home/marxin/Programming/gcc/gcc/backend.h:28, >>>>> from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23, >>>>> from generic-match.c:4: >>>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory >>>>> 2286 | #include "insn-attr-common.h" >>>>> | ^~~~~~~~~~~~~~~~~~~~ >>>>> compilation terminated. >>>>> make: *** [Makefile:2678: generic-match-lto.o] Error 1 >>>>> make: *** Waiting for unfinished jobs.... >>>>> >>>>> In file included from ./tm.h:26, >>>>> from /home/marxin/Programming/gcc/gcc/backend.h:28, >>>>> from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23, >>>>> from gimple-match.c:4: >>>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory >>>>> 2286 | #include "insn-attr-common.h" >>>>> | ^~~~~~~~~~~~~~~~~~~~ >>>>> >>>>> I explicitly added: >>>>> gimple-match.o: gimple-match.c $(generated_files) >>>>> generic-match.o: generic-match.c $(generated_files) >>>>> >>>>> But it's not obeyed. >>>>> >>>>> Martin >>>>> >>>>>> >>>>>> Richard. >>>>>> >>>>>>> Thanks, >>>>>>> Martin >>>>> >>>> >> ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2021-08-22 13:09 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CA+=Sn1=+tuqG7qu5C6_atWzJp+af1P6oVPvABHxqNxiUmJb1dg@mail.gmail.com>
[not found] ` <CAFiYyc1DC1W==r37RhVX9epv1F5XSQhr1XeZEKCk1QOBObSKaQ@mail.gmail.com>
[not found] ` <20210511145904.GM10366@gate.crashing.org>
[not found] ` <CAFiYyc12G6qAeCevEH0-oE4kX7KN1gafBHN4UNYnQJWQVujffg@mail.gmail.com>
[not found] ` <20210512121248.GU10366@gate.crashing.org>
2021-05-20 12:34 ` [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) Martin Liška
2021-05-20 12:54 ` Richard Biener
2021-05-20 13:06 ` Martin Liška
2021-05-20 13:16 ` Richard Biener
2021-05-20 13:22 ` Richard Biener
2021-05-20 15:55 ` Jan Hubicka
2021-05-21 8:29 ` Martin Liška
2021-06-01 7:31 ` Martin Liška
2021-06-23 13:53 ` Martin Liška
2021-08-16 13:58 ` Martin Liška
2021-08-20 14:54 ` Martin Liška
2021-08-22 13:09 ` Jan Hubicka
2021-06-12 15:55 ` Giuliano Belinassi
2021-05-21 8:43 ` Martin Liška
2021-05-21 12:35 ` David Edelsohn
2021-05-24 8:07 ` Martin Liška
2021-06-01 7:33 ` Martin Liška
2021-06-01 7:42 ` Richard Biener
2021-06-01 11:25 ` Martin Liška
2021-06-01 13:19 ` Richard Biener
2021-08-20 14:57 ` Martin Liška
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).