From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id C0B783860C37; Wed, 26 Aug 2020 10:58:26 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C0B783860C37 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1598439506; bh=ryFJVmXHVa2Agv2vqDWHMAQ1kzXYGAf6EgLTqkx+mD0=; h=From:To:Subject:Date:In-Reply-To:References:From; b=GY9acOD0ErfF6U4PrHa8c537CvKMBDrVLaD3Qyk/XiNUImE4gPhbQHioI27X9lF6t YbfDDgCeM9zgNkr42OcAp+85DZwrwWrrW0qtr6zs6/jWccisiQq3fSG2qtoQB0ioVe b6WezP0FAJiiVbd4wJLXpiAqejOQZPIxzUvyk2ZU= From: "rguenther at suse dot de" To: gcc-bugs@gcc.gnu.org Subject: [Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization Date: Wed, 26 Aug 2020 10:58:26 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: gcc X-Bugzilla-Component: bootstrap X-Bugzilla-Version: 10.0 X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: rguenther at suse dot de X-Bugzilla-Status: NEW X-Bugzilla-Resolution: X-Bugzilla-Priority: P3 X-Bugzilla-Assigned-To: unassigned at gcc dot gnu.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://gcc.gnu.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: gcc-bugs@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-bugs mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2020 10:58:26 -0000 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D96794 --- Comment #7 from rguenther at suse dot de --- On Wed, 26 Aug 2020, hubicka at ucw dot cz wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D96794 >=20 > --- Comment #3 from Jan Hubicka --- > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D96794 > >=20 > > --- Comment #2 from Martin Li=C5=A1ka --- > > (In reply to Jan Hubicka from comment #1) > > > > As seen > > > > here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b= 2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.= svg > > > >=20 > > > > each blocking linking of a GCC front-end leads to a wasted jobserve= r worker. > > > Hmm, I am not sure how to interpret the graph. I can see that there i= s a > > > funny staircase of ltranses but how that relates to jobserver workers? > >=20 > > Yes, I mean the staircase of LTRANS because at the beginning N-1 links = are > > waiting for lock: > >=20 > > [ 299s] lock-and-run.sh: (PID 7351) waiting 0 sec to acquire linkfe.lc= k from > > PID 7347 > > ... > >=20 > > For jobserver they are still running even though they sleep. > Aha, so it is extra locking mechanizm we add without jobserver > knowledge. > >=20 > >=20 > > > We limit makefile to link a binary at a time to avoid Richi's box get= ting > > > out of memory, right? > >=20 > > No. It's because we want to have a reasonable contrains which is right = now 8GB. > > Without --enable-link-mutex, we would consume ~ 10 x 1.3 GB (plus WPA p= arallel > > streaming peak), which is probably not desired. >=20 > 10x1.3GB will get consumed only if the building machine has 10 threads. > I wonder if the jobserver WPA streaming integration will happen this > year, with that snd some patches to WPA memory use we could fit in 8GB > unless very large parallelism is configured. Note even without LTO the link step will consume about 1GB for each FE, this is enough to make my box with 6GB ram swap and die miserably when bootstrapping with all languages enabled. Yes, without LTO bootstrap - ld.bfd (also gold) really are that memory hungry. > I suppose only really effective solution would to teach the jobserver > that some jobs are "big" and consume multiple tokens, that is WPA, while > other jobs like ltranses and streaming are small. The only effective solution would be to make the glue with the waiting mechanism pass on its token? Hmm, I guess since lto-wrapper invokes make again and also waits lto-wrapper itself still sits on the original token from the parent jobserver and thus this isn't really dependent on the waiting mechanism but an inherent "bug" in the way we execute the LTRANS compiles in jobserver mode?=