From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by sourceware.org (Postfix) with ESMTPS id 1AAE13846078; Tue, 1 Jun 2021 07:43:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 1AAE13846078 Received: by mail-ed1-x52b.google.com with SMTP id s6so16068002edu.10; Tue, 01 Jun 2021 00:43:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=gYulJbekGz7EkI4r7Y3Ycs1q4kLxkfUDew5c9Hm27qk=; b=MGg7eiNbHTIBMmoStx3mqTKufcF1Hxekq7OalPo1p5kgiMQ2Qy6wmzazSCuZpmjGKd n0HvfJIL4EgDzmmEXghtyCV4K0SMMQDBC+RNJVDed6Eakt1FfsbmzhyLcXevw7cpD8Fc tuK9MaOS22MQdELf1gBjP6lvkczw6yjOxOrKZyOtZpXdc3gTSQJRjMrLpMyPjb8Idn66 mxW/Amcq0SP951yC2TteSp2OOGX2JF7M1EVYQop6tMJ5qLiJFhHwHTv8xodWJW1jhZxJ h2OeHJtgl24LyQIpEZWzRi+wg705sWkNSwqtmTeZ652rkKI6jNetkGtfn5WmXV48Mv5a 7Vig== X-Gm-Message-State: AOAM531qQ78SbBcrpiuLt/MF2NuL/fKB/wvan2fUpDgbHVdH67/JqQ+1 EyXIYBmlWGLncBnUmmO7gEowOrVOOUYQv7SiTQo= X-Google-Smtp-Source: ABdhPJwb06ugbKFX49QH88oLb2GPt5e5NBJ4xok4bVwIb5ZK8CPRUBkI+1nYR5lKDa4ofZ8m1C/b7n578jK+WNg+Fzw= X-Received: by 2002:aa7:cf06:: with SMTP id a6mr31445822edy.138.1622533381953; Tue, 01 Jun 2021 00:43:01 -0700 (PDT) MIME-Version: 1.0 References: <20210511145904.GM10366@gate.crashing.org> <20210512121248.GU10366@gate.crashing.org> <5f13a740-5eff-886f-2b29-52a305fdf3b1@suse.cz> <03febb0f-16fa-4048-6680-438a63b11dcf@suse.cz> <5114b886-6b32-90ac-38ac-ac549950f8dc@suse.cz> In-Reply-To: <5114b886-6b32-90ac-38ac-ac549950f8dc@suse.cz> From: Richard Biener Date: Tue, 1 Jun 2021 09:42:51 +0200 Message-ID: Subject: Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) To: =?UTF-8?Q?Martin_Li=C5=A1ka?= Cc: GCC Mailing List , GCC Patches , Segher Boessenkool Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_SBL, URIBL_SBL_A autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jun 2021 07:43:04 -0000 On Tue, Jun 1, 2021 at 9:33 AM Martin Li=C5=A1ka wrote: > > @Richi: Can you please reply to this email? Not sure what I should add here? Honza suggested to mangle the promoted symbol names. I don't really like the idea to compile multiple TUs into one object. Also +LTO_LINKER_FLAGS =3D -flto=3Dauto --param=3Dlto-partitions=3D16 -flinker-output=3Dnolto-rel -r why hard-code to 16 partitions? You're side-stepping the driver diagnostic by doing compile & link separately, but in the end we're going to want sth like Giul= ianos -fparallel-compile that works transparently from within the driver, so the "manual" operation should try to follow that or alternatively a driver-only wrapper around the "manual" processing could be added whose implementation can be optimized la= ter. Why do you use -flto=3Dauto? There should be a jobserver active. > On 5/21/21 10:43 AM, Martin Li=C5=A1ka wrote: > > On 5/20/21 2:54 PM, Richard Biener wrote: > >> On Thu, May 20, 2021 at 2:34 PM Martin Li=C5=A1ka wro= te: > >>> > >>> Hello. > >>> > >>> I've got a patch candidate that leverages partial linking for a coupl= e of selected object files. > >>> > >>> I'm sending make all-host- jX results for my machine: > >>> > >>> before: 3m18s (user 32m52s) > >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77= dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg > >>> > >>> after: 2m57m (user 35m) > >>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77= dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto= .svg > >>> > >>> One can utilize it with: > >>> make -j16 all-host PARTIAL_LTO=3D1 > >>> > >>> @Segher, Andrew: Can you please measure time improvement for your slo= w bootstrap? > >>> One can also tweak --param=3Dlto-partitions=3D16 param value. > >>> > >>> Thoughts? > >> > >> You're LTO linking multiple objects here - that's almost as if you > >> were doing this > >> for the whole of libbackend.a ... so $(OBJS)_CLFAGS +=3D -flto and in = the > >> libbackend.a rule do a similar partial link trick. > > > > Yeah, apart from that one can't likely do partial linking for an archiv= e: > > > > $ g++ -no-pie -flto=3Dauto --param=3Dlto-partitions=3D16 -flinker-outpu= t=3Dnolto-rel -r libbackend.a > > collect2: fatal error: ld terminated with signal 11 [Segmentation fault= ], core dumped > > compilation terminated. > > > > while ld.bfd immediately finishes. > > > >> > >> That gets you half of a LTO bootstrap then. > >> > >> So why did you go from applying this per-file to multiple files? Does= $(LINKER) > >> have a proper rule to pick up a jobserver? > >> > >> When upstreaming in any form you probably have to gate it on bootstrap= -lto > >> being not active. > > > > Sure, that's reasonable, we can likely detect a -flto option in $(COMPI= LE), right? > > > > One more thing I face is broken dependency: > > $ make clean && make -j32 PARTIAL_LTO=3D1 > > > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exception= s -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-st= rings -Wcast-qual -Wno-error=3Dformat-diag -Wmissing-format-attribute -Wove= rloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overleng= th-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/P= rogramming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Prog= ramming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/inc= lude -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programm= ing/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnum= ber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrac= e -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimpl= e-match-lto.TPo gimple-match.c -flto > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exception= s -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-st= rings -Wcast-qual -Wno-error=3Dformat-diag -Wmissing-format-attribute -Wove= rloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overleng= th-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/P= rogramming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Prog= ramming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/inc= lude -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programm= ing/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnum= ber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrac= e -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/gen= eric-match-lto.TPo generic-match.c -flto > > > > In file included from ./tm.h:26, > > from /home/marxin/Programming/gcc/gcc/backend.h:28, > > from /home/marxin/Programming/gcc/gcc/generic-match-h= ead.c:23, > > from generic-match.c:4: > > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal erro= r: insn-attr-common.h: No such file or directory > > 2286 | #include "insn-attr-common.h" > > | ^~~~~~~~~~~~~~~~~~~~ > > compilation terminated. > > make: *** [Makefile:2678: generic-match-lto.o] Error 1 > > make: *** Waiting for unfinished jobs.... > > > > In file included from ./tm.h:26, > > from /home/marxin/Programming/gcc/gcc/backend.h:28, > > from /home/marxin/Programming/gcc/gcc/gimple-match-he= ad.c:23, > > from gimple-match.c:4: > > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal erro= r: insn-attr-common.h: No such file or directory > > 2286 | #include "insn-attr-common.h" > > | ^~~~~~~~~~~~~~~~~~~~ > > > > I explicitly added: > > gimple-match.o: gimple-match.c $(generated_files) > > generic-match.o: generic-match.c $(generated_files) > > > > But it's not obeyed. > > > > Martin > > > >> > >> Richard. > >> > >>> Thanks, > >>> Martin > > >