From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id F083F3858434; Wed, 30 Nov 2022 08:23:11 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org F083F3858434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1669796591; bh=36I0wevZmVx+kcflEFOlO1q0fCexUe5ZgEVbkBIHK0U=; h=From:To:Subject:Date:In-Reply-To:References:From; b=redvIT9jjHHTKNtrZbOx4P673ud+Vadtzn45jBj3HSURMTF4KRgTH1Vo5DZ0vvcjL 1Oo+0PexkFTxri3ZPfekKVFivqdNZgovMKVTThZdYjyrF26KdnRDW9eOCgwtF3QiUu ot6fhQG/yBcj6C1vlxE4xuN78Gb3Hz3P0ixbkJmE= From: "marxin at gcc dot gnu.org" To: gcc-bugs@gcc.gnu.org Subject: [Bug bootstrap/84402] [meta] GCC build system: parallelism bottleneck Date: Wed, 30 Nov 2022 08:23:04 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: gcc X-Bugzilla-Component: bootstrap X-Bugzilla-Version: unknown X-Bugzilla-Keywords: build, meta-bug X-Bugzilla-Severity: normal X-Bugzilla-Who: marxin at gcc dot gnu.org X-Bugzilla-Status: NEW X-Bugzilla-Resolution: X-Bugzilla-Priority: P3 X-Bugzilla-Assigned-To: unassigned at gcc dot gnu.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://gcc.gnu.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 List-Id: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D84402 --- Comment #49 from Martin Li=C5=A1ka --- One more observation I made, apparently we're trying to sort (in Makefile.i= n) OBJS with the biggest at the very beginning: 1295 # Language-independent object files. 1296 # We put the *-match.o and insn-*.o files first so that a parallel = make 1297 # will build them sooner, because they are large and otherwise tend= to be 1298 # the last objects to finish building. 1299 OBJS =3D \ 1300 gimple-match.o \ 1301 generic-match.o \ 1302 insn-attrtab.o \ 1303 insn-automata.o \ That's fine, plus we introduce dependency for all objects to depend on generated_files: 4441 # In order for parallel make to really start compiling the expensive 4442 # objects from $(OBJS) as early as possible, build all their 4443 # prerequisites strictly before all objects. 4444 $(ALL_HOST_OBJS) : | $(generated_files) Using that, we should see gimple-match.o being spawned very soon, but it's = not the case. Imagine you have already built all-host and let's see what happen= s: $ rm -f gimple-match.o ; rm -f tree*.o && make -j4 --debug=3Db libbackend.a= 2>&1 | less ... File 'gimple-match.o' does not exist. Prerequisite 'cs-bconfig.h' is newer than target 'bconfig.h'. Must remake target 'bconfig.h'. Prerequisite 'cstamp-h' is newer than target 'auto-host.h'. Must remake target 'auto-host.h'. Prerequisite 's-options' is newer than target 'optionlist'. Must remake target 'optionlist'. Prerequisite 's-gtyp-input' is newer than target 'gtyp-input.list'. Must remake target 'gtyp-input.list'. Prerequisite 's-bversion' is newer than target 'bversion.h'. Must remake target 'bversion.h'. Prerequisite 'cs-config.h' is newer than target 'config.h'. Must remake target 'config.h'. ... File 'tree-vrp.o' does not exist. File 'tree.o' does not exist. Prerequisite 's-i386-bt' is newer than target 'i386-builtin-types.inc'. Must remake target 'i386-builtin-types.inc'. File 'gimple-match.o' does not exist. Prerequisite 's-modes-h' is newer than target 'insn-modes.h'. Must remake target 'insn-modes.h'. Prerequisite 's-modes-inline-h' is newer than target 'insn-modes-inline.h'. Must remake target 'insn-modes-inline.h'. Prerequisite 's-version' is newer than target 'version= .h'. Must remake target 'version.h'. Prerequisite 's-options-h' is newer than target 'options.h= '. Must remake target 'options.h'. Prerequisite 's-genrtl-h' is newer than target 'genrtl.h'. Must remake target 'genrtl.h'. Prerequisite 's-modes-m' is newer than target 'min-insn-modes.= cc'. Must remake target 'min-insn-modes.cc'. ... File 'gimple-match.o' does not exist. Prerequisite 's-gtype' is newer than target 'gtype-desc.h'. Must remake target 'gtype-desc.h'. Prerequisite 's-constants' is newer than target 'insn-constants.h'. Must remake target 'insn-constants.h'. ... Must remake target 'tree-affine.o'. g++ -fno-PIE -c -g -DIN_GCC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common=20 -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody=20 -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o tree-affine.o -MT tree-affine.o -MMD -MP -MF ./.deps/tree-affine.TPo /home/marxin/Programming/gcc/gcc/tree-affine.cc File 'tree-call-cdce.o' does not exist. Must remake target 'tree-call-cdce.o'. g++ -fno-PIE -c -g -DIN_GCC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common=20 -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody=20 -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o tree-call-cdce.o -MT tree-call-cdce.o -MMD -MP -MF ./.deps/tree-call-cdce.TPo /home/marxin/Programming/gcc/gcc/tree-call-cdce.cc File 'tree-cfg.o' does not exist. Must remake target 'tree-cfg.o'. g++ -fno-PIE -c -g -DIN_GCC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common=20 -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody=20 -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace -o tree-cfg.o -MT tree-cfg.o -MMD -MP -MF ./.deps/tree-cfg.TPo /home/marxin/Programming/gcc/gcc/tree-cfg.cc File 'tree-cfgcleanup.o' does not exist. Must remake target 'tree-cfgcleanup.o'. So gimple-match.o has a complex dependencies that are somehow under investigation and that's why it doesn't start early :/ It's likely related to various ' $(STAMP) $name' we use, if I consid= er one: gtyp-input.list: s-gtyp-input ; @true s-gtyp-input: Makefile @: $(call write_entries_to_file,$(GTFILES),tmp-gi.list) $(SHELL) $(srcdir)/../move-if-change tmp-gi.list gtyp-input.list $(STAMP) s-gtyp-input Here we touch 's-gtyp-input' later than gtyp-input.list is created and thus gtyp-input.list always need to be remade becase it's dependency s-gtyp-input is newer. Similarly for ma= ny other rules: gimple-match.cc: s-match gimple-match-head.cc ; @true s-match: build/genmatch$(build_exeext) $(srcdir)/match.pd cfn-operators.pd $(RUN_GEN) build/genmatch$(build_exeext) --gimple $(srcdir)/match.p= d \ > tmp-gimple-match.cc $(RUN_GEN) build/genmatch$(build_exeext) --generic $(srcdir)/match.= pd \ > tmp-generic-match.cc $(SHELL) $(srcdir)/../move-if-change tmp-gimple-match.cc \ gimple-match.cc $(SHELL) $(srcdir)/../move-if-change tmp-generic-match.cc \ generic-match.cc $(STAMP) s-match Here it's even more complicated, I think s-match should be only updated if generic-match.cc is touched, otherwise, we again end up younger s-match than gimple-match.cc. Can please any GNU make expect judge here? Starting e.g. gimple-match.cc ea= rly would really help to speed up the build process.=