From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 109608 invoked by alias); 21 Sep 2019 23:09:06 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 109348 invoked by uid 89); 21 Sep 2019 23:09:06 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-5.5 required=5.0 tests=AWL,BAYES_00,KAM_SHORT,SPF_PASS autolearn=ham version=3.3.1 spammy=30min, experimenting, surprisingly, constrained X-HELO: us-smtp-1.mimecast.com Received: from us-smtp-delivery-1.mimecast.com (HELO us-smtp-1.mimecast.com) (205.139.110.120) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sat, 21 Sep 2019 23:09:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1569107342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ptJJpM5VJucoaPC09twq4MpkghaTW+mVxR498+KYvsI=; b=c1zbF2nRWOZRomwmok076vTeB3nNxON7iwlx4KWYCdNpQWmgt1tygbYrfJPAT1bCR8oUj6 6McYqgDP4LzQVWUyzICnYBi7vf971sRzqXiSsJ04TbZ5r8zW0AvVRawFebaGFBqpG3xJH6 ZTPjWAwmuJrNC9DRWf6LHdImGdUsLBs= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-190-yiURpGCDNQyxeY6Td48WFg-1; Sat, 21 Sep 2019 19:08:59 -0400 Received: by mail-ot1-f69.google.com with SMTP id h13so760242otk.20 for ; Sat, 21 Sep 2019 16:08:59 -0700 (PDT) MIME-Version: 1.0 References: <7bc19d9b-16ae-8d40-d5ea-c6b85b93aa1f@ubuntu.com> In-Reply-To: From: Jason Merrill Date: Sat, 21 Sep 2019 23:09:00 -0000 Message-ID: Subject: Re: LTO+profiled enabled builds To: Matthias Klose Cc: gcc Development X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes X-SW-Source: 2019-09/txt/msg00181.txt.bz2 Have you had a chance to try this? On Sat, Sep 14, 2019 at 2:39 PM Jason Merrill wrote: > > How does this do for you? > > On Thu, Jul 4, 2019 at 7:15 AM Matthias Klose wrote: > > > > I'm running into some issues building LTO+profiled enabled configuratio= ns in > > some constrained build environment called buildds, having four cores an= d 16GB of > > RAM. > > > > configured for all frontends (maximum number of LTO links) and configur= ed with > > > > --enable-bootstrap \ > > --with-build-config=3Dbootstrap-lto-lean \ > > --enable-link-mutex > > > > and building the make profiledbootstrap-lean target. > > > > Most builds time out after 150 minutes. > > > > A typical LTO link runs for around one minute on this hardware, however= a LTO > > link with -fprofile-use runs for up to three hours. > > > > So gcc/lock-and-run.sh runs the first lto-link, waits for all other 300= seconds, > > then removes the "stale" locks, and runs everything in parallel ... Wh= ich > > surprisingly goes well, because -flto=3Djobserver is in effect, so I do= n't see any > > memory constraints yet. > > > > The machine then starts building all front-ends, but apparently is not > > overloaded, as -flto=3Djobserver is in effect. However there is no out= put, and > > that triggers the timeout. Richi mentioned on IRC that the LTO links on= ly have > > buffered output (unless you run in debug mode), and that is only emitte= d once > > the link finishes. However even with unbuffered output, there could be= times > > when nothing is happening, no warnings? > > > > I'm currently experimenting with a modified lock-and-run.sh, which basi= cally > > sets the delay for releasing the "stale" locks to 30min instead of 5 mi= n, runs > > the LTO link in the background and checks for the status of the backgro= und job, > > emitting some "running ..." messages while not finished. Still adjusti= ng some > > parameters, but at least that succeeds on some of my configurations. > > > > The locking mechanism was introduced in 2013, > > https://gcc.gnu.org/ml/gcc-patches/2013-05/msg00001.html > > > > lock-and-run.sh should probably modified not to release the "stale" loc= ks based > > on a fixed timeout value. How? > > > > While the "no-output" problem can be fixed in the lock script as well > > (attached), this doesn't apply to third party apps. Having unbuffered = output > > and/or an option to print progress would be beneficial. > > > > Matthias > > > > > >