Speed of compiling gimple-match.c

Speed of compiling gimple-match.c

[Andrew Pinski]

Hi all,
  I noticed my (highly, -j24) parallel build of GCC is serialized on
compiling gimple-match.c.  Has anyone looked into splitting this
generated file into multiple files?

Thanks,
Andrew Pinski

Re: Speed of compiling gimple-match.c

[Richard Biener]

On Mon, May 3, 2021 at 11:10 PM Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
>
> Hi all,
>   I noticed my (highly, -j24) parallel build of GCC is serialized on
> compiling gimple-match.c.  Has anyone looked into splitting this
> generated file into multiple files?

There were threads about this in the past, yes.  There's the
possibility to use LTO for this as well (also mentioned in those
threads).  Note it's not easy to split in a meaningful way in
genmatch.c

Richard.

> Thanks,
> Andrew Pinski

Re: Speed of compiling gimple-match.c

[Thomas Schwinge]

Hi!

On 2021-05-03T13:18:35-0700, Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
>   I noticed my (highly, -j24) parallel build of GCC is serialized on
> compiling gimple-match.c.  Has anyone looked into splitting this
> generated file into multiple files?

There is <https://gcc.gnu.org/PR84402> "[meta] GCC build system:
parallelism bottleneck".

Relatedly, there is Jojo's "genemit.c (main): split insn-emit.c for
compiling parallelly",
<http://mid.mail-archive.com/20201104015315.81416-1-jiejie_rong@c-sky.com>,
etc.

And, I'm aware of the following, somewhat related, too: Giuliano's
"[GSoC] Automatic Parallel Compilation Viability",
<http://mid.mail-archive.com/20200828203232.du364uy7vawjavym@smtp.gmail.com>,
etc.


Grüße
 Thomas
-----------------
Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 München Registergericht München HRB 106955, Geschäftsführer: Thomas Heurung, Frank Thürauf

Re: Speed of compiling gimple-match.c

[JojoR]

Jojo R
在 2021年5月4日 +0800 PM4:58，Thomas Schwinge <thomas@codesourcery.com>，写道：
> Hi!
>
> On 2021-05-03T13:18:35-0700, Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
> > I noticed my (highly, -j24) parallel build of GCC is serialized on
> > compiling gimple-match.c. Has anyone looked into splitting this
> > generated file into multiple files?
>
> There is <https://gcc.gnu.org/PR84402> "[meta] GCC build system:
> parallelism bottleneck".
>
> Relatedly, there is Jojo's "genemit.c (main): split insn-emit.c for
> compiling parallelly",
> <http://mid.mail-archive.com/20201104015315.81416-1-jiejie_rong@c-sky.com>,
> etc.
This is useful for generated large files :)
>
> And, I'm aware of the following, somewhat related, too: Giuliano's
> "[GSoC] Automatic Parallel Compilation Viability",
> <http://mid.mail-archive.com/20200828203232.du364uy7vawjavym@smtp.gmail.com>,
> etc.
>
>
> Grüße
> Thomas
> -----------------
> Mentor Graphics (Deutschland) GmbH, Arnulfstrasse 201, 80634 München Registergericht München HRB 106955, Geschäftsführer: Thomas Heurung, Frank Thürauf

Re: Speed of compiling gimple-match.c

[Segher Boessenkool]

On Tue, May 04, 2021 at 10:40:38AM +0200, Richard Biener via Gcc wrote:
> On Mon, May 3, 2021 at 11:10 PM Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
> >   I noticed my (highly, -j24) parallel build of GCC is serialized on
> > compiling gimple-match.c.  Has anyone looked into splitting this
> > generated file into multiple files?
> 
> There were threads about this in the past, yes.  There's the
> possibility to use LTO for this as well (also mentioned in those
> threads).  Note it's not easy to split in a meaningful way in
> genmatch.c

But it will have to be handled at least somewhat soon: on not huge
parallelism (-j120 for example) building *-match.c takes longer than
building everything else in gcc/ together (wallclock time), and it is a
huge part of regstrap time (bigger than running all of the testsuite!)


Segher

Re: Speed of compiling gimple-match.c

[Richard Biener]

On Tue, May 11, 2021 at 5:01 PM Segher Boessenkool
<segher@kernel.crashing.org> wrote:
>
> On Tue, May 04, 2021 at 10:40:38AM +0200, Richard Biener via Gcc wrote:
> > On Mon, May 3, 2021 at 11:10 PM Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
> > >   I noticed my (highly, -j24) parallel build of GCC is serialized on
> > > compiling gimple-match.c.  Has anyone looked into splitting this
> > > generated file into multiple files?
> >
> > There were threads about this in the past, yes.  There's the
> > possibility to use LTO for this as well (also mentioned in those
> > threads).  Note it's not easy to split in a meaningful way in
> > genmatch.c
>
> But it will have to be handled at least somewhat soon: on not huge
> parallelism (-j120 for example) building *-match.c takes longer than
> building everything else in gcc/ together (wallclock time), and it is a
> huge part of regstrap time (bigger than running all of the testsuite!)

I would classify -j120 as "huge parallelism" ;)  Testing time still
dominates my builds (with -j24) where bootstrap takes ~20 mins
but testing another 40.

Is it building stage2 gimple-match.c that you are worried about?
(it's built using the -O0 compiled stage1 compiler - but we at
least should end up using -fno-checking for this build)

Maybe you can do some experiments - like add
-fno-inline-functions-called-once and change
genmatch.c:3766 to split out single uses as well
(should decrease function sizes).

There's the option to make all functions external in
gimple-match.c so splitting the file at arbitrary points
will be possible (directly from genmatch), we'll need
some internal header with all declarations then
as well or alternatively some clever logic in
genmatch to only externalize functions needed from
mutliple split files.

That said - ideas to reduce the size of the generated
code are welcome as well.

There's also pattern ordering in match.pd that can
make a difference because we're honoring
first-match and thus have to re-start matching from
outermost on conflicts (most of the time the actual
oder in match.pd is just random).  If you add -v
to genmatch then you'll see

/home/rguenther/src/gcc3/gcc/match.pd:6092:10 warning: failed to merge
decision tree node
   (cmp (op@3 @0 INTEGER_CST@1) INTEGER_CST@2)
         ^
/home/rguenther/src/gcc3/gcc/match.pd:4263:11 warning: with the following
    (cmp (op @0 REAL_CST@1) REAL_CST@2)
          ^
/home/rguenther/src/gcc3/gcc/match.pd:5164:6 warning: because of the
following which serves as ordering barrier
 (eq @0 integer_onep)
     ^

that means that the simple (eq @0 integer_onep) should match after
4263 but before 6092
(only the latter will actually match the same - the former has
REAL_CST@2 but 5164
uses a predicate integer_onep).  This causes us to emit three switch
(code){ case EQ_EXPR: }
instead of one.

There might be legitimate cases of such order constraints but most of them
are spurious.  "Fixing" them will also make the matching process faster, but
it's quite some legwork where moving a pattern can fix one occurance but
result in new others.

For me building stage3 gimple-match.o (on a fully loaded system.. :/) is

95.05user 0.42system 1:35.51elapsed 99%CPU (0avgtext+0avgdata
929400maxresident)k
0inputs+0outputs (0major+393349minor)pagefaults 0swaps

and when I use -Wno-error -flto=24 -flinker-output=nolto-rel -r

139.95user 1.79system 0:25.92elapsed 546%CPU (0avgtext+0avgdata
538852maxresident)k
0inputs+0outputs (0major+1139679minor)pagefaults 0swaps

the issue of course is that we can't use this for the stage1 build
(unless we detect working
GCC LTO in the host compiler setup).  I suppose those measures show the lower
bound of what should be possible with splitting up the file (LTO
splits to 128 pieces),
so for me it's a 4x speedup in wallclock time despite the overhead of
LTO which is
quite noticable.  -fno-checking also makes a dramatic difference for me.

Richard.

>
> Segher

Re: Speed of compiling gimple-match.c

[Andrew Pinski]

On Wed, May 12, 2021 at 1:19 AM Richard Biener
<richard.guenther@gmail.com> wrote:
>
> On Tue, May 11, 2021 at 5:01 PM Segher Boessenkool
> <segher@kernel.crashing.org> wrote:
> >
> > On Tue, May 04, 2021 at 10:40:38AM +0200, Richard Biener via Gcc wrote:
> > > On Mon, May 3, 2021 at 11:10 PM Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
> > > >   I noticed my (highly, -j24) parallel build of GCC is serialized on
> > > > compiling gimple-match.c.  Has anyone looked into splitting this
> > > > generated file into multiple files?
> > >
> > > There were threads about this in the past, yes.  There's the
> > > possibility to use LTO for this as well (also mentioned in those
> > > threads).  Note it's not easy to split in a meaningful way in
> > > genmatch.c
> >
> > But it will have to be handled at least somewhat soon: on not huge
> > parallelism (-j120 for example) building *-match.c takes longer than
> > building everything else in gcc/ together (wallclock time), and it is a
> > huge part of regstrap time (bigger than running all of the testsuite!)
>
> I would classify -j120 as "huge parallelism" ;)  Testing time still
> dominates my builds (with -j24) where bootstrap takes ~20 mins
> but testing another 40.

For me, it is around 1 hour bootstrapping and 1 hour testing.

> Is it building stage2 gimple-match.c that you are worried about?
> (it's built using the -O0 compiled stage1 compiler - but we at
> least should end up using -fno-checking for this build)

Yes.  It takes on the machine I was using 15 minutes to compile
gimple-match.c, dominating the whole time for bootstrapping.
Everything else was done in 1-3 minutes max even.
This is on an aarch64 machine with 24 cores (not threads).

Thanks,
Andrew Pinski

>
> Maybe you can do some experiments - like add
> -fno-inline-functions-called-once and change
> genmatch.c:3766 to split out single uses as well
> (should decrease function sizes).
>
> There's the option to make all functions external in
> gimple-match.c so splitting the file at arbitrary points
> will be possible (directly from genmatch), we'll need
> some internal header with all declarations then
> as well or alternatively some clever logic in
> genmatch to only externalize functions needed from
> mutliple split files.
>
> That said - ideas to reduce the size of the generated
> code are welcome as well.
>
> There's also pattern ordering in match.pd that can
> make a difference because we're honoring
> first-match and thus have to re-start matching from
> outermost on conflicts (most of the time the actual
> oder in match.pd is just random).  If you add -v
> to genmatch then you'll see
>
> /home/rguenther/src/gcc3/gcc/match.pd:6092:10 warning: failed to merge
> decision tree node
>    (cmp (op@3 @0 INTEGER_CST@1) INTEGER_CST@2)
>          ^
> /home/rguenther/src/gcc3/gcc/match.pd:4263:11 warning: with the following
>     (cmp (op @0 REAL_CST@1) REAL_CST@2)
>           ^
> /home/rguenther/src/gcc3/gcc/match.pd:5164:6 warning: because of the
> following which serves as ordering barrier
>  (eq @0 integer_onep)
>      ^
>
> that means that the simple (eq @0 integer_onep) should match after
> 4263 but before 6092
> (only the latter will actually match the same - the former has
> REAL_CST@2 but 5164
> uses a predicate integer_onep).  This causes us to emit three switch
> (code){ case EQ_EXPR: }
> instead of one.
>
> There might be legitimate cases of such order constraints but most of them
> are spurious.  "Fixing" them will also make the matching process faster, but
> it's quite some legwork where moving a pattern can fix one occurance but
> result in new others.
>
> For me building stage3 gimple-match.o (on a fully loaded system.. :/) is
>
> 95.05user 0.42system 1:35.51elapsed 99%CPU (0avgtext+0avgdata
> 929400maxresident)k
> 0inputs+0outputs (0major+393349minor)pagefaults 0swaps
>
> and when I use -Wno-error -flto=24 -flinker-output=nolto-rel -r
>
> 139.95user 1.79system 0:25.92elapsed 546%CPU (0avgtext+0avgdata
> 538852maxresident)k
> 0inputs+0outputs (0major+1139679minor)pagefaults 0swaps
>
> the issue of course is that we can't use this for the stage1 build
> (unless we detect working
> GCC LTO in the host compiler setup).  I suppose those measures show the lower
> bound of what should be possible with splitting up the file (LTO
> splits to 128 pieces),
> so for me it's a 4x speedup in wallclock time despite the overhead of
> LTO which is
> quite noticable.  -fno-checking also makes a dramatic difference for me.
>
> Richard.
>
> >
> > Segher

Re: Speed of compiling gimple-match.c

[Richard Biener]

On Wed, May 12, 2021 at 11:03 AM Andrew Pinski <pinskia@gmail.com> wrote:
>
> On Wed, May 12, 2021 at 1:19 AM Richard Biener
> <richard.guenther@gmail.com> wrote:
> >
> > On Tue, May 11, 2021 at 5:01 PM Segher Boessenkool
> > <segher@kernel.crashing.org> wrote:
> > >
> > > On Tue, May 04, 2021 at 10:40:38AM +0200, Richard Biener via Gcc wrote:
> > > > On Mon, May 3, 2021 at 11:10 PM Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
> > > > >   I noticed my (highly, -j24) parallel build of GCC is serialized on
> > > > > compiling gimple-match.c.  Has anyone looked into splitting this
> > > > > generated file into multiple files?
> > > >
> > > > There were threads about this in the past, yes.  There's the
> > > > possibility to use LTO for this as well (also mentioned in those
> > > > threads).  Note it's not easy to split in a meaningful way in
> > > > genmatch.c
> > >
> > > But it will have to be handled at least somewhat soon: on not huge
> > > parallelism (-j120 for example) building *-match.c takes longer than
> > > building everything else in gcc/ together (wallclock time), and it is a
> > > huge part of regstrap time (bigger than running all of the testsuite!)
> >
> > I would classify -j120 as "huge parallelism" ;)  Testing time still
> > dominates my builds (with -j24) where bootstrap takes ~20 mins
> > but testing another 40.
>
> For me, it is around 1 hour bootstrapping and 1 hour testing.
>
> > Is it building stage2 gimple-match.c that you are worried about?
> > (it's built using the -O0 compiled stage1 compiler - but we at
> > least should end up using -fno-checking for this build)
>
> Yes.  It takes on the machine I was using 15 minutes to compile
> gimple-match.c, dominating the whole time for bootstrapping.
> Everything else was done in 1-3 minutes max even.
> This is on an aarch64 machine with 24 cores (not threads).

I'm usually using STAGE1_CFLAGS="-O2" to speed up the
"useless" part of the bootstrap cycle...

> Thanks,
> Andrew Pinski
>
> >
> > Maybe you can do some experiments - like add
> > -fno-inline-functions-called-once and change
> > genmatch.c:3766 to split out single uses as well
> > (should decrease function sizes).
> >
> > There's the option to make all functions external in
> > gimple-match.c so splitting the file at arbitrary points
> > will be possible (directly from genmatch), we'll need
> > some internal header with all declarations then
> > as well or alternatively some clever logic in
> > genmatch to only externalize functions needed from
> > mutliple split files.
> >
> > That said - ideas to reduce the size of the generated
> > code are welcome as well.
> >
> > There's also pattern ordering in match.pd that can
> > make a difference because we're honoring
> > first-match and thus have to re-start matching from
> > outermost on conflicts (most of the time the actual
> > oder in match.pd is just random).  If you add -v
> > to genmatch then you'll see
> >
> > /home/rguenther/src/gcc3/gcc/match.pd:6092:10 warning: failed to merge
> > decision tree node
> >    (cmp (op@3 @0 INTEGER_CST@1) INTEGER_CST@2)
> >          ^
> > /home/rguenther/src/gcc3/gcc/match.pd:4263:11 warning: with the following
> >     (cmp (op @0 REAL_CST@1) REAL_CST@2)
> >           ^
> > /home/rguenther/src/gcc3/gcc/match.pd:5164:6 warning: because of the
> > following which serves as ordering barrier
> >  (eq @0 integer_onep)
> >      ^
> >
> > that means that the simple (eq @0 integer_onep) should match after
> > 4263 but before 6092
> > (only the latter will actually match the same - the former has
> > REAL_CST@2 but 5164
> > uses a predicate integer_onep).  This causes us to emit three switch
> > (code){ case EQ_EXPR: }
> > instead of one.
> >
> > There might be legitimate cases of such order constraints but most of them
> > are spurious.  "Fixing" them will also make the matching process faster, but
> > it's quite some legwork where moving a pattern can fix one occurance but
> > result in new others.
> >
> > For me building stage3 gimple-match.o (on a fully loaded system.. :/) is
> >
> > 95.05user 0.42system 1:35.51elapsed 99%CPU (0avgtext+0avgdata
> > 929400maxresident)k
> > 0inputs+0outputs (0major+393349minor)pagefaults 0swaps
> >
> > and when I use -Wno-error -flto=24 -flinker-output=nolto-rel -r
> >
> > 139.95user 1.79system 0:25.92elapsed 546%CPU (0avgtext+0avgdata
> > 538852maxresident)k
> > 0inputs+0outputs (0major+1139679minor)pagefaults 0swaps
> >
> > the issue of course is that we can't use this for the stage1 build
> > (unless we detect working
> > GCC LTO in the host compiler setup).  I suppose those measures show the lower
> > bound of what should be possible with splitting up the file (LTO
> > splits to 128 pieces),
> > so for me it's a 4x speedup in wallclock time despite the overhead of
> > LTO which is
> > quite noticable.  -fno-checking also makes a dramatic difference for me.
> >
> > Richard.
> >
> > >
> > > Segher

Re: Speed of compiling gimple-match.c

[Segher Boessenkool]

On Wed, May 12, 2021 at 10:19:17AM +0200, Richard Biener wrote:
> On Tue, May 11, 2021 at 5:01 PM Segher Boessenkool
> <segher@kernel.crashing.org> wrote:
> >
> > On Tue, May 04, 2021 at 10:40:38AM +0200, Richard Biener via Gcc wrote:
> > > On Mon, May 3, 2021 at 11:10 PM Andrew Pinski via Gcc <gcc@gcc.gnu.org> wrote:
> > > >   I noticed my (highly, -j24) parallel build of GCC is serialized on
> > > > compiling gimple-match.c.  Has anyone looked into splitting this
> > > > generated file into multiple files?
> > >
> > > There were threads about this in the past, yes.  There's the
> > > possibility to use LTO for this as well (also mentioned in those
> > > threads).  Note it's not easy to split in a meaningful way in
> > > genmatch.c
> >
> > But it will have to be handled at least somewhat soon: on not huge
> > parallelism (-j120 for example) building *-match.c takes longer than
> > building everything else in gcc/ together (wallclock time), and it is a
> > huge part of regstrap time (bigger than running all of the testsuite!)
> 
> I would classify -j120 as "huge parallelism" ;)  Testing time still
> dominates my builds (with -j24) where bootstrap takes ~20 mins
> but testing another 40.

The issue is that the usual regstrap has become something like 2x slower
over five or so years.  Hardware doesn't keep up (per thread), so the
only way to keep acceptable turnaround times is to up the thread count.
And then the few tasks that take way more time than everything else will
dominate.

> Is it building stage2 gimple-match.c that you are worried about?
> (it's built using the -O0 compiled stage1 compiler - but we at
> least should end up using -fno-checking for this build)

The *-match builds in other stages are noticeably slow as well.  But
yes, stage2 is by far slowest.

> Maybe you can do some experiments - like add
> -fno-inline-functions-called-once and change
> genmatch.c:3766 to split out single uses as well
> (should decrease function sizes).
> 
> There's the option to make all functions external in
> gimple-match.c so splitting the file at arbitrary points
> will be possible (directly from genmatch), we'll need
> some internal header with all declarations then
> as well or alternatively some clever logic in
> genmatch to only externalize functions needed from
> mutliple split files.
> 
> That said - ideas to reduce the size of the generated
> code are welcome as well.

I don't currently have good plans for this.  I'm just remarking that it
increasingly is the biggest bootstrap time problem.  It always has been,
but it has become significantly worse the past few releases.

[ snip a lot I cannot reply to right now ]

> -fno-checking also makes a dramatic difference for me.

Checking quite often finds serious problems, so that is not a real
option, neither during development nor while just testing
configurations.  I even like to enable rtl,tree checking even though
that costs a factor of three or so build time.  Disabling all checking
most of the time is counter-productive imo.


Segher

[PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

Hello.

I've got a patch candidate that leverages partial linking for a couple of selected object files.

I'm sending make all-host- jX results for my machine:

before: 3m18s (user 32m52s)
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg

after: 2m57m (user 35m)
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg

One can utilize it with:
make -j16 all-host PARTIAL_LTO=1

@Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
One can also tweak --param=lto-partitions=16 param value.

Thoughts?
Thanks,
Martin

[-- Attachment #2: 0001-Try-LTO-partial-linking.patch --]
[-- Type: text/x-patch, Size: 2367 bytes --]

From 85228e612610c0e4b0324f6bebc84ef7c0211c4a Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Thu, 20 May 2021 14:29:35 +0200
Subject: [PATCH] Try LTO partial linking.

---
 gcc/Makefile.in | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 1164554e6d6..f76bcea66f5 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -220,7 +220,9 @@ libgcov-util.o-warn = -Wno-error
 libgcov-driver-tool.o-warn = -Wno-error
 libgcov-merge-tool.o-warn = -Wno-error
 gimple-match.o-warn = -Wno-unused
+gimple-match-lto.o-warn = -Wno-unused
 generic-match.o-warn = -Wno-unused
+generic-match-lto.o-warn = -Wno-unused
 dfp.o-warn = -Wno-strict-aliasing
 
 # All warnings have to be shut off in stage1 if the compiler used then
@@ -1282,12 +1284,10 @@ ANALYZER_OBJS = \
 # will build them sooner, because they are large and otherwise tend to be
 # the last objects to finish building.
 OBJS = \
-	gimple-match.o \
-	generic-match.o \
+	common-base.a \
 	insn-attrtab.o \
 	insn-automata.o \
 	insn-dfatab.o \
-	insn-emit.o \
 	insn-extract.o \
 	insn-latencytab.o \
 	insn-modes.o \
@@ -1295,7 +1295,6 @@ OBJS = \
 	insn-output.o \
 	insn-peep.o \
 	insn-preds.o \
-	insn-recog.o \
 	insn-enums.o \
 	ggc-page.o \
 	adjust-alignment.o \
@@ -2627,6 +2626,29 @@ s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd
 	    					generic-match.c
 	$(STAMP) s-match
 
+ifdef PARTIAL_LTO
+LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r
+LTO_FLAGS = -flto
+
+gimple-match-lto.o: gimple-match.c $(TARGET_H)
+	$(COMPILE) $< $(LTO_FLAGS)
+generic-match-lto.o: generic-match.c $(TARGET_H)
+	$(COMPILE) $< $(LTO_FLAGS)
+insn-recog-lto.o: insn-recog.c
+	$(COMPILE) $< $(LTO_FLAGS)
+insn-emit-lto.o: insn-emit.c
+	$(COMPILE) $< $(LTO_FLAGS)
+
+common-base.a: gimple-match-lto.o generic-match-lto.o insn-recog-lto.o insn-emit-lto.o
+	-rm -rf $@
+	$(LINKER) $^ $(LTO_LINKER_FLAGS) -o common-base.o
+	$(AR) $(AR_FLAGS)T $@ common-base.o
+else
+common-base.a: gimple-match.o generic-match.o insn-recog.o insn-emit.o
+	-rm -rf $@
+	$(AR) $(AR_FLAGS)T $@ $^
+endif
+
 GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \
   $(host_xm_file_list) \
   $(tm_file_list) $(HASHTAB_H) $(SPLAY_TREE_H) $(srcdir)/bitmap.h \
-- 
2.31.1

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Richard Biener]

On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>
> Hello.
>
> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>
> I'm sending make all-host- jX results for my machine:
>
> before: 3m18s (user 32m52s)
> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>
> after: 2m57m (user 35m)
> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>
> One can utilize it with:
> make -j16 all-host PARTIAL_LTO=1
>
> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
> One can also tweak --param=lto-partitions=16 param value.
>
> Thoughts?

You're LTO linking multiple objects here - that's almost as if you
were doing this
for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
libbackend.a rule do a similar partial link trick.

That gets you half of a LTO bootstrap then.

So why did you go from applying this per-file to multiple files?  Does $(LINKER)
have a proper rule to pick up a jobserver?

When upstreaming in any form you probably have to gate it on bootstrap-lto
being not active.

Richard.

> Thanks,
> Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 5/20/21 2:54 PM, Richard Biener wrote:
> So why did you go from applying this per-file to multiple files?

When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols:

/usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
/usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
/usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
/usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
/usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]'

Any idea what was I doing wrong?

Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Richard Biener]

On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 5/20/21 2:54 PM, Richard Biener wrote:
> > So why did you go from applying this per-file to multiple files?
>
> When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols:
>
> /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
> /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
> /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
> /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
> /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]'
>
> Any idea what was I doing wrong?

Nothing in particular I think - you're just hitting the issue that LTO
produces new symbols and that those
can obviously clash.  Giuliano hit the very same issue.  When not
doing partial links those internal
symbols pose no problem, but with -r -flinker-output=nolto-rel and
re-linking the produced objects
they obviously do.  ELF has no solution for this though, but I think
we could strip those from the
partially linked object - if WPA would give us a list of objects the
link step could postprocess
the object with objcopy or maybe a custom linker script could do the
trick as well.

So your workaround is to only ever have a single LTO produced object
file participating in the
final links ;)

Richard.

>
> Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Richard Biener]

On Thu, May 20, 2021 at 3:16 PM Richard Biener
<richard.guenther@gmail.com> wrote:
>
> On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote:
> >
> > On 5/20/21 2:54 PM, Richard Biener wrote:
> > > So why did you go from applying this per-file to multiple files?
> >
> > When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols:
> >
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]'
> >
> > Any idea what was I doing wrong?
>
> Nothing in particular I think - you're just hitting the issue that LTO
> produces new symbols and that those
> can obviously clash.  Giuliano hit the very same issue.  When not
> doing partial links those internal
> symbols pose no problem, but with -r -flinker-output=nolto-rel and
> re-linking the produced objects
> they obviously do.  ELF has no solution for this though, but I think
> we could strip those from the
> partially linked object - if WPA would give us a list of objects the
> link step could postprocess
> the object with objcopy or maybe a custom linker script could do the
> trick as well.

Oh, and the "best" solution would be to avoid involving the linker
when doing -r -flinker-output=nolto-rel but instead have the assembler
produce the single object from the multiple LTRANS assembly snippets
which could then use local labels instead of symbols for these.

> So your workaround is to only ever have a single LTO produced object
> file participating in the
> final links ;)
>
> Richard.
>
> >
> > Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Jan Hubicka]

> On Thu, May 20, 2021 at 3:16 PM Richard Biener
> <richard.guenther@gmail.com> wrote:
> >
> > On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote:
> > >
> > > On 5/20/21 2:54 PM, Richard Biener wrote:
> > > > So why did you go from applying this per-file to multiple files?
> > >
> > > When I did per-file for {gimple,generic}-match.c I hit the following issue with lto.priv symbols:
> > >
> > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
> > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
> > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone .part.0] [clone .lto_priv.0]'
> > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: libbackend.a(gimple-match.o): previous definition here
> > > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-linux/bin/ld: error: libbackend.a(generic-match.o): multiple definition of 'vec<constructor_elt, va_gc, vl_embed>::operator[](unsigned int) [clone .part.0] [clone .lto_priv.0]'
> > >
> > > Any idea what was I doing wrong?
> >
> > Nothing in particular I think - you're just hitting the issue that LTO
> > produces new symbols and that those
> > can obviously clash.  Giuliano hit the very same issue.  When not
> > doing partial links those internal
> > symbols pose no problem, but with -r -flinker-output=nolto-rel and
> > re-linking the produced objects
> > they obviously do.  ELF has no solution for this though, but I think
> > we could strip those from the
> > partially linked object - if WPA would give us a list of objects the
> > link step could postprocess
> > the object with objcopy or maybe a custom linker script could do the
> > trick as well.
> 
> Oh, and the "best" solution would be to avoid involving the linker
> when doing -r -flinker-output=nolto-rel but instead have the assembler
> produce the single object from the multiple LTRANS assembly snippets
> which could then use local labels instead of symbols for these.

Quick solution is to also modify partitioner to use the local symbol
names when doing incremental linking (those mixing in source code and
random seeds) to avoid clashes.

Honza
> 
> > So your workaround is to only ever have a single LTO produced object
> > file participating in the
> > final links ;)
> >
> > Richard.
> >
> > >
> > > Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

On 5/20/21 5:55 PM, Jan Hubicka wrote:
> Quick solution is to also modify partitioner to use the local symbol
> names when doing incremental linking (those mixing in source code and
> random seeds) to avoid clashes.

Good hint. I added hash based on object file name (I don't want to handle
proper string escaping) and -frandom-seed.

What do you think about the patch?
Thanks,
Martin

[-- Attachment #2: 0001-LTO-add-lto_priv-suffixfor-LTO_LINKER_OUTPUT_NOLTORE.patch --]
[-- Type: text/x-patch, Size: 1829 bytes --]

From 372d2944571906932fd1419bfc51a949d67b857e Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Fri, 21 May 2021 10:25:49 +0200
Subject: [PATCH] LTO: add lto_priv suffixfor LTO_LINKER_OUTPUT_NOLTOREL.

gcc/lto/ChangeLog:

	* lto-partition.c (privatize_symbol_name_1): Add random suffix
	based on hash of the object file and -frandom-seed.
---
 gcc/lto/lto-partition.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/gcc/lto/lto-partition.c b/gcc/lto/lto-partition.c
index 15761ac9eb5..fef48c869a2 100644
--- a/gcc/lto/lto-partition.c
+++ b/gcc/lto/lto-partition.c
@@ -35,6 +35,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "ipa-fnsummary.h"
 #include "lto-partition.h"
 #include "sreal.h"
+#include "toplev.h"
 
 vec<ltrans_partition> ltrans_partitions;
 
@@ -941,9 +942,23 @@ privatize_symbol_name_1 (symtab_node *node, tree decl)
 
   name = maybe_rewrite_identifier (name);
   unsigned &clone_number = lto_clone_numbers->get_or_insert (name);
-  symtab->change_decl_assembler_name (decl,
-				      clone_function_name (
-					  name, "lto_priv", clone_number));
+
+  char *suffix = NULL;
+  if (flag_lto_linker_output == LTO_LINKER_OUTPUT_NOLTOREL)
+    {
+      hashval_t fnhash = 0;
+      if (node->lto_file_data != NULL)
+	fnhash = htab_hash_string (node->lto_file_data->file_name);
+      suffix = XNEWVEC (char, 128);
+      char sep = symbol_table::symbol_suffix_separator ();
+      sprintf (suffix, "lto_priv%c%u%c%" PRIu64, sep, fnhash, sep,
+	       (unsigned HOST_WIDE_INT)get_random_seed (false));
+    }
+
+  tree clone
+    = clone_function_name (name, suffix ? suffix : "lto_priv", clone_number);
+  symtab->change_decl_assembler_name (decl, clone);
+  free (suffix);
   clone_number++;
 
   if (node->lto_file_data)
-- 
2.31.1

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 4739 bytes --]

On 5/20/21 2:54 PM, Richard Biener wrote:
> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> Hello.
>>
>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>>
>> I'm sending make all-host- jX results for my machine:
>>
>> before: 3m18s (user 32m52s)
>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>>
>> after: 2m57m (user 35m)
>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>>
>> One can utilize it with:
>> make -j16 all-host PARTIAL_LTO=1
>>
>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
>> One can also tweak --param=lto-partitions=16 param value.
>>
>> Thoughts?
> 
> You're LTO linking multiple objects here - that's almost as if you
> were doing this
> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
> libbackend.a rule do a similar partial link trick.

Yeah, apart from that one can't likely do partial linking for an archive:

$ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.

while ld.bfd immediately finishes.

> 
> That gets you half of a LTO bootstrap then.
> 
> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
> have a proper rule to pick up a jobserver?
> 
> When upstreaming in any form you probably have to gate it on bootstrap-lto
> being not active.

Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?

One more thing I face is broken dependency:
$ make clean && make -j32 PARTIAL_LTO=1

g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto

In file included from ./tm.h:26,
                  from /home/marxin/Programming/gcc/gcc/backend.h:28,
                  from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
                  from generic-match.c:4:
/home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
  2286 | #include "insn-attr-common.h"
       |          ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:2678: generic-match-lto.o] Error 1
make: *** Waiting for unfinished jobs....

In file included from ./tm.h:26,
                  from /home/marxin/Programming/gcc/gcc/backend.h:28,
                  from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
                  from gimple-match.c:4:
/home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
  2286 | #include "insn-attr-common.h"
       |          ^~~~~~~~~~~~~~~~~~~~

I explicitly added:
gimple-match.o: gimple-match.c $(generated_files)
generic-match.o: generic-match.c $(generated_files)

But it's not obeyed.

Martin

> 
> Richard.
> 
>> Thanks,
>> Martin


[-- Attachment #2: 0001-Try-LTO-partial-linking.patch --]
[-- Type: text/x-patch, Size: 2452 bytes --]

From b209c7aea76ceb8eadcf5843f30299fc8041a579 Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Thu, 20 May 2021 14:29:35 +0200
Subject: [PATCH] Try LTO partial linking.

---
 gcc/Makefile.in | 33 +++++++++++++++++++++++++++++----
 1 file changed, 29 insertions(+), 4 deletions(-)

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 1164554e6d6..decfea7412b 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -220,7 +220,9 @@ libgcov-util.o-warn = -Wno-error
 libgcov-driver-tool.o-warn = -Wno-error
 libgcov-merge-tool.o-warn = -Wno-error
 gimple-match.o-warn = -Wno-unused
+gimple-match-lto.o-warn = -Wno-unused
 generic-match.o-warn = -Wno-unused
+generic-match-lto.o-warn = -Wno-unused
 dfp.o-warn = -Wno-strict-aliasing
 
 # All warnings have to be shut off in stage1 if the compiler used then
@@ -1282,12 +1284,10 @@ ANALYZER_OBJS = \
 # will build them sooner, because they are large and otherwise tend to be
 # the last objects to finish building.
 OBJS = \
-	gimple-match.o \
-	generic-match.o \
+	common-base.a \
 	insn-attrtab.o \
 	insn-automata.o \
 	insn-dfatab.o \
-	insn-emit.o \
 	insn-extract.o \
 	insn-latencytab.o \
 	insn-modes.o \
@@ -1295,7 +1295,6 @@ OBJS = \
 	insn-output.o \
 	insn-peep.o \
 	insn-preds.o \
-	insn-recog.o \
 	insn-enums.o \
 	ggc-page.o \
 	adjust-alignment.o \
@@ -2627,6 +2626,32 @@ s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd
 	    					generic-match.c
 	$(STAMP) s-match
 
+gimple-match.o: gimple-match.c $(generated_files)
+generic-match.o: generic-match.c $(generated_files)
+
+ifdef PARTIAL_LTO
+LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r
+LTO_FLAGS = -flto
+
+gimple-match-lto.o: gimple-match.c
+	$(COMPILE) $< $(LTO_FLAGS)
+generic-match-lto.o: generic-match.c
+	$(COMPILE) $< $(LTO_FLAGS)
+insn-recog-lto.o: insn-recog.c
+	$(COMPILE) $< $(LTO_FLAGS)
+insn-emit-lto.o: insn-emit.c
+	$(COMPILE) $< $(LTO_FLAGS)
+
+common-base.a: gimple-match-lto.o generic-match-lto.o insn-recog-lto.o insn-emit-lto.o
+	-rm -rf $@
+	$(LINKER) $^ $(LTO_LINKER_FLAGS) -o common-base.o
+	$(AR) $(AR_FLAGS)T $@ common-base.o
+else
+common-base.a: gimple-match.o generic-match.o insn-recog.o insn-emit.o
+	-rm -rf $@
+	$(AR) $(AR_FLAGS)T $@ $^
+endif
+
 GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \
   $(host_xm_file_list) \
   $(tm_file_list) $(HASHTAB_H) $(SPLAY_TREE_H) $(srcdir)/bitmap.h \
-- 
2.31.1

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[David Edelsohn]

On Fri, May 21, 2021 at 5:25 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 5/20/21 2:54 PM, Richard Biener wrote:
> > On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> Hello.
> >>
> >> I've got a patch candidate that leverages partial linking for a couple of selected object files.
> >>
> >> I'm sending make all-host- jX results for my machine:
> >>
> >> before: 3m18s (user 32m52s)
> >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
> >>
> >> after: 2m57m (user 35m)
> >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
> >>
> >> One can utilize it with:
> >> make -j16 all-host PARTIAL_LTO=1
> >>
> >> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
> >> One can also tweak --param=lto-partitions=16 param value.
> >>
> >> Thoughts?
> >
> > You're LTO linking multiple objects here - that's almost as if you
> > were doing this
> > for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
> > libbackend.a rule do a similar partial link trick.
>
> Yeah, apart from that one can't likely do partial linking for an archive:
>
> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
> compilation terminated.
>
> while ld.bfd immediately finishes.
>
> >
> > That gets you half of a LTO bootstrap then.
> >
> > So why did you go from applying this per-file to multiple files?  Does $(LINKER)
> > have a proper rule to pick up a jobserver?
> >
> > When upstreaming in any form you probably have to gate it on bootstrap-lto
> > being not active.
>
> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
>
> One more thing I face is broken dependency:
> $ make clean && make -j32 PARTIAL_LTO=1
>
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
>
> In file included from ./tm.h:26,
>                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>                   from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>                   from generic-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>        |          ^~~~~~~~~~~~~~~~~~~~
> compilation terminated.
> make: *** [Makefile:2678: generic-match-lto.o] Error 1
> make: *** Waiting for unfinished jobs....
>
> In file included from ./tm.h:26,
>                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>                   from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>                   from gimple-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>        |          ^~~~~~~~~~~~~~~~~~~~
>
> I explicitly added:
> gimple-match.o: gimple-match.c $(generated_files)
> generic-match.o: generic-match.c $(generated_files)
>
> But it's not obeyed.

Please remember that not all targets support LTO so a fallback to a
non-partial-LTO build needs to be provided and automatically invoked
for those targets.

Thanks, David

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 5/21/21 2:35 PM, David Edelsohn wrote:
> Please remember that not all targets support LTO so a fallback to a
> non-partial-LTO build needs to be provided and automatically invoked
> for those targets.

Sure, for now it's definitely going to be a opt-in, enabled by something like:
make PARTIAL_LTO=1.

Thanks,
Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

PING^1

On 5/21/21 10:29 AM, Martin Liška wrote:
> On 5/20/21 5:55 PM, Jan Hubicka wrote:
>> Quick solution is to also modify partitioner to use the local symbol
>> names when doing incremental linking (those mixing in source code and
>> random seeds) to avoid clashes.
> 
> Good hint. I added hash based on object file name (I don't want to handle
> proper string escaping) and -frandom-seed.
> 
> What do you think about the patch?
> Thanks,
> Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

@Richi: Can you please reply to this email?

On 5/21/21 10:43 AM, Martin Liška wrote:
> On 5/20/21 2:54 PM, Richard Biener wrote:
>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>>>
>>> Hello.
>>>
>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>>>
>>> I'm sending make all-host- jX results for my machine:
>>>
>>> before: 3m18s (user 32m52s)
>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>>>
>>> after: 2m57m (user 35m)
>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>>>
>>> One can utilize it with:
>>> make -j16 all-host PARTIAL_LTO=1
>>>
>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
>>> One can also tweak --param=lto-partitions=16 param value.
>>>
>>> Thoughts?
>>
>> You're LTO linking multiple objects here - that's almost as if you
>> were doing this
>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
>> libbackend.a rule do a similar partial link trick.
> 
> Yeah, apart from that one can't likely do partial linking for an archive:
> 
> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
> compilation terminated.
> 
> while ld.bfd immediately finishes.
> 
>>
>> That gets you half of a LTO bootstrap then.
>>
>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
>> have a proper rule to pick up a jobserver?
>>
>> When upstreaming in any form you probably have to gate it on bootstrap-lto
>> being not active.
> 
> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
> 
> One more thing I face is broken dependency:
> $ make clean && make -j32 PARTIAL_LTO=1
> 
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
> 
> In file included from ./tm.h:26,
>                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>                   from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>                   from generic-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>        |          ^~~~~~~~~~~~~~~~~~~~
> compilation terminated.
> make: *** [Makefile:2678: generic-match-lto.o] Error 1
> make: *** Waiting for unfinished jobs....
> 
> In file included from ./tm.h:26,
>                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>                   from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>                   from gimple-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>        |          ^~~~~~~~~~~~~~~~~~~~
> 
> I explicitly added:
> gimple-match.o: gimple-match.c $(generated_files)
> generic-match.o: generic-match.c $(generated_files)
> 
> But it's not obeyed.
> 
> Martin
> 
>>
>> Richard.
>>
>>> Thanks,
>>> Martin
> 

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Richard Biener]

On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote:
>
> @Richi: Can you please reply to this email?

Not sure what I should add here?  Honza suggested to mangle the
promoted symbol names.  I don't
really like the idea to compile multiple TUs into one object.  Also

+LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16
-flinker-output=nolto-rel -r

why hard-code to 16 partitions?  You're side-stepping the driver
diagnostic by doing
compile & link separately, but in the end we're going to want sth like Giulianos
-fparallel-compile that works transparently from within the driver, so
the "manual"
operation should try to follow that or alternatively a driver-only
wrapper around the
"manual" processing could be added whose implementation can be optimized later.

Why do you use -flto=auto?  There should be a jobserver active.

> On 5/21/21 10:43 AM, Martin Liška wrote:
> > On 5/20/21 2:54 PM, Richard Biener wrote:
> >> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
> >>>
> >>> Hello.
> >>>
> >>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
> >>>
> >>> I'm sending make all-host- jX results for my machine:
> >>>
> >>> before: 3m18s (user 32m52s)
> >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
> >>>
> >>> after: 2m57m (user 35m)
> >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
> >>>
> >>> One can utilize it with:
> >>> make -j16 all-host PARTIAL_LTO=1
> >>>
> >>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
> >>> One can also tweak --param=lto-partitions=16 param value.
> >>>
> >>> Thoughts?
> >>
> >> You're LTO linking multiple objects here - that's almost as if you
> >> were doing this
> >> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
> >> libbackend.a rule do a similar partial link trick.
> >
> > Yeah, apart from that one can't likely do partial linking for an archive:
> >
> > $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
> > collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
> > compilation terminated.
> >
> > while ld.bfd immediately finishes.
> >
> >>
> >> That gets you half of a LTO bootstrap then.
> >>
> >> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
> >> have a proper rule to pick up a jobserver?
> >>
> >> When upstreaming in any form you probably have to gate it on bootstrap-lto
> >> being not active.
> >
> > Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
> >
> > One more thing I face is broken dependency:
> > $ make clean && make -j32 PARTIAL_LTO=1
> >
> > g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
> > g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
> >
> > In file included from ./tm.h:26,
> >                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
> >                   from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
> >                   from generic-match.c:4:
> > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
> >   2286 | #include "insn-attr-common.h"
> >        |          ^~~~~~~~~~~~~~~~~~~~
> > compilation terminated.
> > make: *** [Makefile:2678: generic-match-lto.o] Error 1
> > make: *** Waiting for unfinished jobs....
> >
> > In file included from ./tm.h:26,
> >                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
> >                   from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
> >                   from gimple-match.c:4:
> > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
> >   2286 | #include "insn-attr-common.h"
> >        |          ^~~~~~~~~~~~~~~~~~~~
> >
> > I explicitly added:
> > gimple-match.o: gimple-match.c $(generated_files)
> > generic-match.o: generic-match.c $(generated_files)
> >
> > But it's not obeyed.
> >
> > Martin
> >
> >>
> >> Richard.
> >>
> >>> Thanks,
> >>> Martin
> >
>

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 6/1/21 9:42 AM, Richard Biener wrote:
> On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote:
>>
>> @Richi: Can you please reply to this email?
> 
> Not sure what I should add here?  Honza suggested to mangle the
> promoted symbol names.

Sure and I sent a patch for that.

> I don't
> really like the idea to compile multiple TUs into one object.  Also

What's problematic is that we'll have to wait for one another release to make it useful
(if you don't want to build the current master with a snapshot compiler).

> 
> +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16
> -flinker-output=nolto-rel -r
> 
> why hard-code to 16 partitions?  You're side-stepping the driver
> diagnostic by doing
> compile & link separately, but in the end we're going to want sth like Giulianos
> -fparallel-compile that works transparently from within the driver, so
> the "manual"
> operation should try to follow that or alternatively a driver-only
> wrapper around the
> "manual" processing could be added whose implementation can be optimized later.

All right. Do you want me refreshing his -fparallel-compile option introduction?

> 
> Why do you use -flto=auto?  There should be a jobserver active.

Yes, that should not be needed.

Martin

> 
>> On 5/21/21 10:43 AM, Martin Liška wrote:
>>> On 5/20/21 2:54 PM, Richard Biener wrote:
>>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>
>>>>> Hello.
>>>>>
>>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>>>>>
>>>>> I'm sending make all-host- jX results for my machine:
>>>>>
>>>>> before: 3m18s (user 32m52s)
>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>>>>>
>>>>> after: 2m57m (user 35m)
>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>>>>>
>>>>> One can utilize it with:
>>>>> make -j16 all-host PARTIAL_LTO=1
>>>>>
>>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
>>>>> One can also tweak --param=lto-partitions=16 param value.
>>>>>
>>>>> Thoughts?
>>>>
>>>> You're LTO linking multiple objects here - that's almost as if you
>>>> were doing this
>>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
>>>> libbackend.a rule do a similar partial link trick.
>>>
>>> Yeah, apart from that one can't likely do partial linking for an archive:
>>>
>>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
>>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
>>> compilation terminated.
>>>
>>> while ld.bfd immediately finishes.
>>>
>>>>
>>>> That gets you half of a LTO bootstrap then.
>>>>
>>>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
>>>> have a proper rule to pick up a jobserver?
>>>>
>>>> When upstreaming in any form you probably have to gate it on bootstrap-lto
>>>> being not active.
>>>
>>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
>>>
>>> One more thing I face is broken dependency:
>>> $ make clean && make -j32 PARTIAL_LTO=1
>>>
>>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
>>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
>>>
>>> In file included from ./tm.h:26,
>>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
>>>                    from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>>>                    from generic-match.c:4:
>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>>>    2286 | #include "insn-attr-common.h"
>>>         |          ^~~~~~~~~~~~~~~~~~~~
>>> compilation terminated.
>>> make: *** [Makefile:2678: generic-match-lto.o] Error 1
>>> make: *** Waiting for unfinished jobs....
>>>
>>> In file included from ./tm.h:26,
>>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
>>>                    from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>>>                    from gimple-match.c:4:
>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>>>    2286 | #include "insn-attr-common.h"
>>>         |          ^~~~~~~~~~~~~~~~~~~~
>>>
>>> I explicitly added:
>>> gimple-match.o: gimple-match.c $(generated_files)
>>> generic-match.o: generic-match.c $(generated_files)
>>>
>>> But it's not obeyed.
>>>
>>> Martin
>>>
>>>>
>>>> Richard.
>>>>
>>>>> Thanks,
>>>>> Martin
>>>
>>

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Richard Biener]

On Tue, Jun 1, 2021 at 1:25 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/1/21 9:42 AM, Richard Biener wrote:
> > On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> @Richi: Can you please reply to this email?
> >
> > Not sure what I should add here?  Honza suggested to mangle the
> > promoted symbol names.
>
> Sure and I sent a patch for that.
>
> > I don't
> > really like the idea to compile multiple TUs into one object.  Also
>
> What's problematic is that we'll have to wait for one another release to make it useful
> (if you don't want to build the current master with a snapshot compiler).

IMHO it's a bugfix.  Note that I'm not sure what the intent of the change is.
If it is to speedup bootstrap then using LTO bootstrap would do the trick
as well (and better) if we'd simply process all of libbackend.a this way
(and thus avoid re-linking that once for each frontend).  If it is to speedup
dev (re-)builds then dragging in more files will make it build longer since
for example insn-recog.c may be unchanged but gimple-match.c not.

> > +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16
> > -flinker-output=nolto-rel -r
> >
> > why hard-code to 16 partitions?  You're side-stepping the driver
> > diagnostic by doing
> > compile & link separately, but in the end we're going to want sth like Giulianos
> > -fparallel-compile that works transparently from within the driver, so
> > the "manual"
> > operation should try to follow that or alternatively a driver-only
> > wrapper around the
> > "manual" processing could be added whose implementation can be optimized later.
>
> All right. Do you want me refreshing his -fparallel-compile option introduction?

I'm not sure if we've arrived at mergeable state - but if it's
reasonably possible
to hide s/-fparallel-compile/-flto -r -flinker-output=nolto-rel/ split
into compile & link
parts (avoiding the diagnostic on -flinker-output) in the driver then
I think that's
a very reasonable first step (after fixing the symbol privatization issue).  The
GSOC project then was to elide the IL streaming from the high-level operation.

Richard,

> >
> > Why do you use -flto=auto?  There should be a jobserver active.
>
> Yes, that should not be needed.
>
> Martin
>
> >
> >> On 5/21/21 10:43 AM, Martin Liška wrote:
> >>> On 5/20/21 2:54 PM, Richard Biener wrote:
> >>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>
> >>>>> Hello.
> >>>>>
> >>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
> >>>>>
> >>>>> I'm sending make all-host- jX results for my machine:
> >>>>>
> >>>>> before: 3m18s (user 32m52s)
> >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
> >>>>>
> >>>>> after: 2m57m (user 35m)
> >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
> >>>>>
> >>>>> One can utilize it with:
> >>>>> make -j16 all-host PARTIAL_LTO=1
> >>>>>
> >>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
> >>>>> One can also tweak --param=lto-partitions=16 param value.
> >>>>>
> >>>>> Thoughts?
> >>>>
> >>>> You're LTO linking multiple objects here - that's almost as if you
> >>>> were doing this
> >>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
> >>>> libbackend.a rule do a similar partial link trick.
> >>>
> >>> Yeah, apart from that one can't likely do partial linking for an archive:
> >>>
> >>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
> >>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
> >>> compilation terminated.
> >>>
> >>> while ld.bfd immediately finishes.
> >>>
> >>>>
> >>>> That gets you half of a LTO bootstrap then.
> >>>>
> >>>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
> >>>> have a proper rule to pick up a jobserver?
> >>>>
> >>>> When upstreaming in any form you probably have to gate it on bootstrap-lto
> >>>> being not active.
> >>>
> >>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
> >>>
> >>> One more thing I face is broken dependency:
> >>> $ make clean && make -j32 PARTIAL_LTO=1
> >>>
> >>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
> >>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
> >>>
> >>> In file included from ./tm.h:26,
> >>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
> >>>                    from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
> >>>                    from generic-match.c:4:
> >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
> >>>    2286 | #include "insn-attr-common.h"
> >>>         |          ^~~~~~~~~~~~~~~~~~~~
> >>> compilation terminated.
> >>> make: *** [Makefile:2678: generic-match-lto.o] Error 1
> >>> make: *** Waiting for unfinished jobs....
> >>>
> >>> In file included from ./tm.h:26,
> >>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
> >>>                    from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
> >>>                    from gimple-match.c:4:
> >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
> >>>    2286 | #include "insn-attr-common.h"
> >>>         |          ^~~~~~~~~~~~~~~~~~~~
> >>>
> >>> I explicitly added:
> >>> gimple-match.o: gimple-match.c $(generated_files)
> >>> generic-match.o: generic-match.c $(generated_files)
> >>>
> >>> But it's not obeyed.
> >>>
> >>> Martin
> >>>
> >>>>
> >>>> Richard.
> >>>>
> >>>>> Thanks,
> >>>>> Martin
> >>>
> >>
>

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Giuliano Belinassi]

Hi, all.

Please CC me when I am mentioned into a mail.

On Thu, 2021-05-20 at 15:16 +0200, Richard Biener via Gcc wrote:
> On Thu, May 20, 2021 at 3:06 PM Martin Liška <mliska@suse.cz> wrote:
> > 
> > On 5/20/21 2:54 PM, Richard Biener wrote:
> > > So why did you go from applying this per-file to multiple files?
> > 
> > When I did per-file for {gimple,generic}-match.c I hit the
> > following issue with lto.priv symbols:
> > 
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-
> > linux/bin/ld: error: libbackend.a(generic-match.o): multiple
> > definition of 'wi::to_wide(tree_node const*) [clone .part.0] [clone
> > .lto_priv.0]'
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-
> > linux/bin/ld: libbackend.a(gimple-match.o): previous definition
> > here
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-
> > linux/bin/ld: error: libbackend.a(generic-match.o): multiple
> > definition of 'TYPE_VECTOR_SUBPARTS(tree_node const*) [clone
> > .part.0] [clone .lto_priv.0]'
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-
> > linux/bin/ld: libbackend.a(gimple-match.o): previous definition
> > here
> > /usr/lib64/gcc/x86_64-suse-linux/10/../../../../x86_64-suse-
> > linux/bin/ld: error: libbackend.a(generic-match.o): multiple
> > definition of 'vec<constructor_elt, va_gc,
> > vl_embed>::operator[](unsigned int) [clone .part.0] [clone
> > .lto_priv.0]'
> > 
> > Any idea what was I doing wrong?
> 
> Nothing in particular I think - you're just hitting the issue that
> LTO
> produces new symbols and that those
> can obviously clash.  Giuliano hit the very same issue.  When not
> doing partial links those internal
> symbols pose no problem, but with -r -flinker-output=nolto-rel and
> re-linking the produced objects
> they obviously do.  ELF has no solution for this though, but I think
> we could strip those from the
> partially linked object - if WPA would give us a list of objects the
> link step could postprocess
> the object with objcopy or maybe a custom linker script could do the
> trick as well.

I've "fixed" this issue in my branch by mangling any promoted to public
symbol. I've also disabled the "ipa-split" pass in the paper branch
because of some created symbols which I was not able to fix in time.
Perhaps this goes away if you disable it.

Perhaps we should work on getting the autopar branch merged into trunk.
There are several issues which must be fixed and I don't think it will
be ready for this next release. The main ones that I remember from the
top of my head:

1- Fix the driver to use SPEC language for the multiple required calls
to `as`, instead of injecting code for that directly on the `void
execute()` function.

2- Merge my custom partitioner for using the default LTO partitioner.
The default LTO partitioner were hitting the assertions about COMDAT
being split into multiple partitions.

3- Fix the issue with the ipa-split pass.

Perhaps we should further explore avoiding partial linking altogether
and concat the assembler files.

Thank you,
Giuliano.

> 
> So your workaround is to only ever have a single LTO produced object
> file participating in the
> final links ;)
> 
> Richard.
> 
> > 
> > Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 5/21/21 10:29 AM, Martin Liška wrote:
> On 5/20/21 5:55 PM, Jan Hubicka wrote:
>> Quick solution is to also modify partitioner to use the local symbol
>> names when doing incremental linking (those mixing in source code and
>> random seeds) to avoid clashes.
> 
> Good hint. I added hash based on object file name (I don't want to handle
> proper string escaping) and -frandom-seed.
> 
> What do you think about the patch?
> Thanks,
> Martin

@Honza: Can you please take a look at this patch?

Cheers,
Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

PING^2

@Honza: Can you please review the change?

Martin

On 6/23/21 3:53 PM, Martin Liška wrote:
> On 5/21/21 10:29 AM, Martin Liška wrote:
>> On 5/20/21 5:55 PM, Jan Hubicka wrote:
>>> Quick solution is to also modify partitioner to use the local symbol
>>> names when doing incremental linking (those mixing in source code and
>>> random seeds) to avoid clashes.
>>
>> Good hint. I added hash based on object file name (I don't want to handle
>> proper string escaping) and -frandom-seed.
>>
>> What do you think about the patch?
>> Thanks,
>> Martin
> 
> @Honza: Can you please take a look at this patch?
> 
> Cheers,
> Martin

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 8/16/21 3:58 PM, Martin Liška wrote:
> PING^2
> 
> @Honza: Can you please review the change?

I've tested the patch and apparently it's not enough for {gimple,generic}-match.o not clashing
in symbol names. Apparently there are more IPA clones that collide.

Leaving that for now.

Martin

> 
> Martin
> 
> On 6/23/21 3:53 PM, Martin Liška wrote:
>> On 5/21/21 10:29 AM, Martin Liška wrote:
>>> On 5/20/21 5:55 PM, Jan Hubicka wrote:
>>>> Quick solution is to also modify partitioner to use the local symbol
>>>> names when doing incremental linking (those mixing in source code and
>>>> random seeds) to avoid clashes.
>>>
>>> Good hint. I added hash based on object file name (I don't want to handle
>>> proper string escaping) and -frandom-seed.
>>>
>>> What do you think about the patch?
>>> Thanks,
>>> Martin
>>
>> @Honza: Can you please take a look at this patch?
>>
>> Cheers,
>> Martin
> 

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 6/1/21 3:19 PM, Richard Biener wrote:
> On Tue, Jun 1, 2021 at 1:25 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/1/21 9:42 AM, Richard Biener wrote:
>>> On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> @Richi: Can you please reply to this email?
>>>
>>> Not sure what I should add here?  Honza suggested to mangle the
>>> promoted symbol names.
>>
>> Sure and I sent a patch for that.
>>
>>> I don't
>>> really like the idea to compile multiple TUs into one object.  Also
>>
>> What's problematic is that we'll have to wait for one another release to make it useful
>> (if you don't want to build the current master with a snapshot compiler).
> 
> IMHO it's a bugfix.  Note that I'm not sure what the intent of the change is.
> If it is to speedup bootstrap then using LTO bootstrap would do the trick
> as well (and better) if we'd simply process all of libbackend.a this way
> (and thus avoid re-linking that once for each frontend).

When building a GCC package, we intentionally re-link them with all FEs.

> If it is to speedup
> dev (re-)builds then dragging in more files will make it build longer since
> for example insn-recog.c may be unchanged but gimple-match.c not.

Yes, the original motivation was a speed up of a dev. build and yes, the shown
example is problematic. Right now, I'm leaving that as I'm not interested enough
in the parallel build of a simple source file.

Martin

> 
>>> +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16
>>> -flinker-output=nolto-rel -r
>>>
>>> why hard-code to 16 partitions?  You're side-stepping the driver
>>> diagnostic by doing
>>> compile & link separately, but in the end we're going to want sth like Giulianos
>>> -fparallel-compile that works transparently from within the driver, so
>>> the "manual"
>>> operation should try to follow that or alternatively a driver-only
>>> wrapper around the
>>> "manual" processing could be added whose implementation can be optimized later.
>>
>> All right. Do you want me refreshing his -fparallel-compile option introduction?
> 
> I'm not sure if we've arrived at mergeable state - but if it's
> reasonably possible
> to hide s/-fparallel-compile/-flto -r -flinker-output=nolto-rel/ split
> into compile & link
> parts (avoiding the diagnostic on -flinker-output) in the driver then
> I think that's
> a very reasonable first step (after fixing the symbol privatization issue).  The
> GSOC project then was to elide the IL streaming from the high-level operation.
> 
> Richard,
> 
>>>
>>> Why do you use -flto=auto?  There should be a jobserver active.
>>
>> Yes, that should not be needed.
>>
>> Martin
>>
>>>
>>>> On 5/21/21 10:43 AM, Martin Liška wrote:
>>>>> On 5/20/21 2:54 PM, Richard Biener wrote:
>>>>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>>
>>>>>>> Hello.
>>>>>>>
>>>>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>>>>>>>
>>>>>>> I'm sending make all-host- jX results for my machine:
>>>>>>>
>>>>>>> before: 3m18s (user 32m52s)
>>>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>>>>>>>
>>>>>>> after: 2m57m (user 35m)
>>>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>>>>>>>
>>>>>>> One can utilize it with:
>>>>>>> make -j16 all-host PARTIAL_LTO=1
>>>>>>>
>>>>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
>>>>>>> One can also tweak --param=lto-partitions=16 param value.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>
>>>>>> You're LTO linking multiple objects here - that's almost as if you
>>>>>> were doing this
>>>>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
>>>>>> libbackend.a rule do a similar partial link trick.
>>>>>
>>>>> Yeah, apart from that one can't likely do partial linking for an archive:
>>>>>
>>>>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
>>>>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
>>>>> compilation terminated.
>>>>>
>>>>> while ld.bfd immediately finishes.
>>>>>
>>>>>>
>>>>>> That gets you half of a LTO bootstrap then.
>>>>>>
>>>>>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
>>>>>> have a proper rule to pick up a jobserver?
>>>>>>
>>>>>> When upstreaming in any form you probably have to gate it on bootstrap-lto
>>>>>> being not active.
>>>>>
>>>>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
>>>>>
>>>>> One more thing I face is broken dependency:
>>>>> $ make clean && make -j32 PARTIAL_LTO=1
>>>>>
>>>>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
>>>>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
>>>>>
>>>>> In file included from ./tm.h:26,
>>>>>                     from /home/marxin/Programming/gcc/gcc/backend.h:28,
>>>>>                     from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>>>>>                     from generic-match.c:4:
>>>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>>>>>     2286 | #include "insn-attr-common.h"
>>>>>          |          ^~~~~~~~~~~~~~~~~~~~
>>>>> compilation terminated.
>>>>> make: *** [Makefile:2678: generic-match-lto.o] Error 1
>>>>> make: *** Waiting for unfinished jobs....
>>>>>
>>>>> In file included from ./tm.h:26,
>>>>>                     from /home/marxin/Programming/gcc/gcc/backend.h:28,
>>>>>                     from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>>>>>                     from gimple-match.c:4:
>>>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>>>>>     2286 | #include "insn-attr-common.h"
>>>>>          |          ^~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> I explicitly added:
>>>>> gimple-match.o: gimple-match.c $(generated_files)
>>>>> generic-match.o: generic-match.c $(generated_files)
>>>>>
>>>>> But it's not obeyed.
>>>>>
>>>>> Martin
>>>>>
>>>>>>
>>>>>> Richard.
>>>>>>
>>>>>>> Thanks,
>>>>>>> Martin
>>>>>
>>>>
>>

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Jan Hubicka]

> Good hint. I added hash based on object file name (I don't want to handle
> proper string escaping) and -frandom-seed.
> 
> What do you think about the patch?
Sorry for taking so long - I remember I was sending reply earlier but it
seems I only wrote it and never sent.
> Thanks,
> Martin

> From 372d2944571906932fd1419bfc51a949d67b857e Mon Sep 17 00:00:00 2001
> From: Martin Liska <mliska@suse.cz>
> Date: Fri, 21 May 2021 10:25:49 +0200
> Subject: [PATCH] LTO: add lto_priv suffixfor LTO_LINKER_OUTPUT_NOLTOREL.
> 
> gcc/lto/ChangeLog:
> 
> 	* lto-partition.c (privatize_symbol_name_1): Add random suffix
> 	based on hash of the object file and -frandom-seed.
> ---
>  gcc/lto/lto-partition.c | 21 ++++++++++++++++++---
>  1 file changed, 18 insertions(+), 3 deletions(-)
> 
> diff --git a/gcc/lto/lto-partition.c b/gcc/lto/lto-partition.c
> index 15761ac9eb5..fef48c869a2 100644
> --- a/gcc/lto/lto-partition.c
> +++ b/gcc/lto/lto-partition.c
> @@ -35,6 +35,7 @@ along with GCC; see the file COPYING3.  If not see
>  #include "ipa-fnsummary.h"
>  #include "lto-partition.h"
>  #include "sreal.h"
> +#include "toplev.h"
>  
>  vec<ltrans_partition> ltrans_partitions;
>  
> @@ -941,9 +942,23 @@ privatize_symbol_name_1 (symtab_node *node, tree decl)
>  
>    name = maybe_rewrite_identifier (name);
>    unsigned &clone_number = lto_clone_numbers->get_or_insert (name);
> -  symtab->change_decl_assembler_name (decl,
> -				      clone_function_name (
> -					  name, "lto_priv", clone_number));
> +
> +  char *suffix = NULL;
> +  if (flag_lto_linker_output == LTO_LINKER_OUTPUT_NOLTOREL)
> +    {
> +      hashval_t fnhash = 0;
> +      if (node->lto_file_data != NULL)
> +	fnhash = htab_hash_string (node->lto_file_data->file_name);
> +      suffix = XNEWVEC (char, 128);
> +      char sep = symbol_table::symbol_suffix_separator ();
> +      sprintf (suffix, "lto_priv%c%u%c%" PRIu64, sep, fnhash, sep,
> +	       (unsigned HOST_WIDE_INT)get_random_seed (false));

We have get_file_function_name which does similar work but also working
without random seeds.  Perhaps we can reuse it here: use
get_file_function_name once and use it as prefix or compute hash from
it.

The logic to get unique symbol name is not completely easy and it would
be better to not duplciate it.  Patch is OK with that change
(and indeed it is bugfix - even if it is relatively little used partial
linking of LTO objects into non-LTO should be supported and working).
Honza
> +    }
> +
> +  tree clone
> +    = clone_function_name (name, suffix ? suffix : "lto_priv", clone_number);
> +  symtab->change_decl_assembler_name (decl, clone);
> +  free (suffix);
>    clone_number++;
>  
>    if (node->lto_file_data)
> -- 
> 2.31.1
>