From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 27950 invoked by alias); 6 Oct 2011 19:21:02 -0000 Received: (qmail 27931 invoked by uid 22791); 6 Oct 2011 19:21:01 -0000 X-SWARE-Spam-Status: No, hits=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 X-Spam-Check-By: sourceware.org Received: from localhost (HELO gcc.gnu.org) (127.0.0.1) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 06 Oct 2011 19:20:47 +0000 From: "hubicka at ucw dot cz" To: gcc-bugs@gcc.gnu.org Subject: [Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map Date: Thu, 06 Oct 2011 19:21:00 -0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: gcc X-Bugzilla-Component: other X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: hubicka at ucw dot cz X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Priority: P3 X-Bugzilla-Assigned-To: unassigned at gcc dot gnu.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: X-Bugzilla-URL: http://gcc.gnu.org/bugzilla/ Auto-Submitted: auto-generated Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Mailing-List: contact gcc-bugs-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-bugs-owner@gcc.gnu.org X-SW-Source: 2011-10/txt/msg00371.txt.bz2 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #1 from Jan Hubicka 2011-10-06 19:20:45 UTC --- > When doing a very large LTO build I fail with "out of virtual memory" > > Some investigation showed the problem was not actually running out of > memory, but gcc excessively fragmenting its memory map. The Linux kernel > has a default limit of 64k mappings per process and the fragmentation > exceeded that. This lead to gc mmap allocations failing and other problems. > > A workaround is to increase /proc/sys/vm/max_map_count > > Looking at /proc/$(pidof lto1)/maps I see there are lots of 1-3 page holes > between other anonymousmemory. I think that's caused by ggc-pages free_pages() > function freeing too early and in too small chunks > (and perhaps LTO garbage collecting too much?) In gcc-2.95 times ggc-page was probably not written with 8GB of memory use in mind :) Perhaps ggc-page should simply increase the chunks as memory grows? (i.e. release to system only when 20% of memory is unused & it exceeds some minimal value) Honza