From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) by sourceware.org (Postfix) with ESMTPS id 079153858025; Fri, 21 May 2021 12:35:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 079153858025 Received: by mail-lf1-x12f.google.com with SMTP id j6so26773427lfr.11; Fri, 21 May 2021 05:35:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=yoaQR1Y5jXAt8+z/yQ1+QNnfRRCyVnvCkR+k7ullctQ=; b=rpba2NetPNLm+iFqgenS9UVnKKv0HjmgFUZoguFbmvMm4fkPXfZRcdQUEm7l4Blt9z wCIxoo31rruBuhRSxNYKf2afHxYhSAsVEDcUe+DQJZUt4KL+GCCaB8unr0+dXA5819fA e87HYYXLsQcDZ+8yu2VfWruSyQXbUyo0Q1XHFj0yYaat6zMDyND8jCSBHpAsgRG/yRSW eU1JV6CQO8SUJxfwexFBbqs2eoUy5anK9RFG+JOtzzr7hjTn+sgKaMmcYhteWgZW4ZcA ek5lYhOVNOyjtm2i0bYaG+WHxo97NPnQrDa6b1EMuh0Q/s5d/DcDkZr26le8kmutykYJ c08w== X-Gm-Message-State: AOAM532454is7ZBeB+3h0hg32oCAc+Ws1Fd6nS2djQoiygJAQjWcHwN1 2uEBzCQBdE818xvbBWVkfVBZX4tlbeC35qKlCbA= X-Google-Smtp-Source: ABdhPJx58o3XL6dd7zHicZ9PKMpRDDUK9yF7327awAfdK+YnEAtj5vCyhl74CfR54v99GHexB/8xGp+fum44adIGx2Q= X-Received: by 2002:a05:6512:2304:: with SMTP id o4mr2098232lfu.135.1621600521667; Fri, 21 May 2021 05:35:21 -0700 (PDT) MIME-Version: 1.0 References: <20210511145904.GM10366@gate.crashing.org> <20210512121248.GU10366@gate.crashing.org> <5f13a740-5eff-886f-2b29-52a305fdf3b1@suse.cz> <03febb0f-16fa-4048-6680-438a63b11dcf@suse.cz> In-Reply-To: <03febb0f-16fa-4048-6680-438a63b11dcf@suse.cz> From: David Edelsohn Date: Fri, 21 May 2021 08:35:09 -0400 Message-ID: Subject: Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c) To: =?UTF-8?Q?Martin_Li=C5=A1ka?= Cc: Richard Biener , GCC Mailing List , GCC Patches , Segher Boessenkool Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_SBL, URIBL_SBL_A autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 May 2021 12:35:26 -0000 On Fri, May 21, 2021 at 5:25 AM Martin Li=C5=A1ka wrote: > > On 5/20/21 2:54 PM, Richard Biener wrote: > > On Thu, May 20, 2021 at 2:34 PM Martin Li=C5=A1ka wrot= e: > >> > >> Hello. > >> > >> I've got a patch candidate that leverages partial linking for a couple= of selected object files. > >> > >> I'm sending make all-host- jX results for my machine: > >> > >> before: 3m18s (user 32m52s) > >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77d= acad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg > >> > >> after: 2m57m (user 35m) > >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77d= acad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.= svg > >> > >> One can utilize it with: > >> make -j16 all-host PARTIAL_LTO=3D1 > >> > >> @Segher, Andrew: Can you please measure time improvement for your slow= bootstrap? > >> One can also tweak --param=3Dlto-partitions=3D16 param value. > >> > >> Thoughts? > > > > You're LTO linking multiple objects here - that's almost as if you > > were doing this > > for the whole of libbackend.a ... so $(OBJS)_CLFAGS +=3D -flto and in t= he > > libbackend.a rule do a similar partial link trick. > > Yeah, apart from that one can't likely do partial linking for an archive: > > $ g++ -no-pie -flto=3Dauto --param=3Dlto-partitions=3D16 -flinker-output= =3Dnolto-rel -r libbackend.a > collect2: fatal error: ld terminated with signal 11 [Segmentation fault],= core dumped > compilation terminated. > > while ld.bfd immediately finishes. > > > > > That gets you half of a LTO bootstrap then. > > > > So why did you go from applying this per-file to multiple files? Does = $(LINKER) > > have a proper rule to pick up a jobserver? > > > > When upstreaming in any form you probably have to gate it on bootstrap-= lto > > being not active. > > Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE= ), right? > > One more thing I face is broken dependency: > $ make clean && make -j32 PARTIAL_LTO=3D1 > > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions = -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-stri= ngs -Wcast-qual -Wno-error=3Dformat-diag -Wmissing-format-attribute -Woverl= oaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength= -strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Pro= gramming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Progra= mming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/inclu= de -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programmin= g/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumbe= r/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace = -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-= match-lto.TPo gimple-match.c -flto > g++ -fcf-protection -fno-PIE -c -g -DIN_GCC -fPIC -fno-exceptions = -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-stri= ngs -Wcast-qual -Wno-error=3Dformat-diag -Wmissing-format-attribute -Woverl= oaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength= -strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Pro= gramming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Progra= mming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/inclu= de -I/home/marxin/Programming/gcc/gcc/../libcody -I/home/marxin/Programmin= g/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumbe= r/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace = -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/gener= ic-match-lto.TPo generic-match.c -flto > > In file included from ./tm.h:26, > from /home/marxin/Programming/gcc/gcc/backend.h:28, > from /home/marxin/Programming/gcc/gcc/generic-match-hea= d.c:23, > from generic-match.c:4: > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error:= insn-attr-common.h: No such file or directory > 2286 | #include "insn-attr-common.h" > | ^~~~~~~~~~~~~~~~~~~~ > compilation terminated. > make: *** [Makefile:2678: generic-match-lto.o] Error 1 > make: *** Waiting for unfinished jobs.... > > In file included from ./tm.h:26, > from /home/marxin/Programming/gcc/gcc/backend.h:28, > from /home/marxin/Programming/gcc/gcc/gimple-match-head= .c:23, > from gimple-match.c:4: > /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error:= insn-attr-common.h: No such file or directory > 2286 | #include "insn-attr-common.h" > | ^~~~~~~~~~~~~~~~~~~~ > > I explicitly added: > gimple-match.o: gimple-match.c $(generated_files) > generic-match.o: generic-match.c $(generated_files) > > But it's not obeyed. Please remember that not all targets support LTO so a fallback to a non-partial-LTO build needs to be provided and automatically invoked for those targets. Thanks, David