From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 9278 invoked by alias); 10 Oct 2011 10:25:32 -0000 Received: (qmail 9263 invoked by uid 22791); 10 Oct 2011 10:25:31 -0000 X-SWARE-Spam-Status: No, hits=-2.3 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW X-Spam-Check-By: sourceware.org Received: from mail-yx0-f175.google.com (HELO mail-yx0-f175.google.com) (209.85.213.175) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 10 Oct 2011 10:25:17 +0000 Received: by yxj17 with SMTP id 17so6147173yxj.20 for ; Mon, 10 Oct 2011 03:25:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.150.188.9 with SMTP id l9mr5799967ybf.15.1318242315084; Mon, 10 Oct 2011 03:25:15 -0700 (PDT) Received: by 10.151.9.9 with HTTP; Mon, 10 Oct 2011 03:25:15 -0700 (PDT) In-Reply-To: <1318190141-1220-2-git-send-email-andi@firstfloor.org> References: <1318190141-1220-1-git-send-email-andi@firstfloor.org> <1318190141-1220-2-git-send-email-andi@firstfloor.org> Date: Mon, 10 Oct 2011 10:33:00 -0000 Message-ID: Subject: Re: [PATCH 1/5] Use MADV_DONTNEED for freeing in garbage collector From: Richard Guenther To: Andi Kleen Cc: gcc-patches@gcc.gnu.org, Andi Kleen Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org X-SW-Source: 2011-10/txt/msg00727.txt.bz2 On Sun, Oct 9, 2011 at 9:55 PM, Andi Kleen wrote: > From: Andi Kleen > > Use the Linux MADV_DONTNEED call to unmap free pages in the garbage > collector.Then keep the unmapped pages in the free list. This avoid > excessive memory fragmentation on large LTO bulds, which can lead > to gcc bumping into the Linux vm_max_map limit per process. > > Based on a idea from Jakub. Shouldn't we prefer still "mapped" pages when allocating? Thus, keep the freepages list "sorted"? With the new params to call release_pages less, how does this interact with using MADV_DONTNEED? The only reason to delay MADV_DONTNEED is to avoid splitting huge-pages? Which would mean that we should rather be better at controlling where we allocate from from the free-list? Richard. > gcc/: > > 2011-10-08 =A0 Andi Kleen > > =A0 =A0 =A0 =A0PR other/50636 > =A0 =A0 =A0 =A0* config.in, configure: Regenerate. > =A0 =A0 =A0 =A0* configure.ac (madvise): Add to AC_CHECK_FUNCS. > =A0 =A0 =A0 =A0* ggc-page.c (USING_MADVISE): Add. > =A0 =A0 =A0 =A0(page_entry): Add unmapped field. > =A0 =A0 =A0 =A0(alloc_page): Check for unmapped pages. > =A0 =A0 =A0 =A0(release_pages): Add USING_MADVISE branch. > --- > =A0gcc/config.in =A0 =A0| =A0 =A06 ++++++ > =A0gcc/configure =A0 =A0| =A0 =A02 +- > =A0gcc/configure.ac | =A0 =A02 +- > =A0gcc/ggc-page.c =A0 | =A0 48 ++++++++++++++++++++++++++++++++++++++++++= +++++- > =A04 files changed, 55 insertions(+), 3 deletions(-) > > diff --git a/gcc/config.in b/gcc/config.in > index f2847d8..e8148b6 100644 > --- a/gcc/config.in > +++ b/gcc/config.in > @@ -1276,6 +1276,12 @@ > =A0#endif > > > +/* Define to 1 if you have the `madvise' function. */ > +#ifndef USED_FOR_TARGET > +#undef HAVE_MADVISE > +#endif > + > + > =A0/* Define to 1 if you have the header file. */ > =A0#ifndef USED_FOR_TARGET > =A0#undef HAVE_MALLOC_H > diff --git a/gcc/configure b/gcc/configure > index cb55dda..4a54adf 100755 > --- a/gcc/configure > +++ b/gcc/configure > @@ -9001,7 +9001,7 @@ fi > =A0for ac_func in times clock kill getrlimit setrlimit atoll atoq \ > =A0 =A0 =A0 =A0sysconf strsignal getrusage nl_langinfo \ > =A0 =A0 =A0 =A0gettimeofday mbstowcs wcswidth mmap setlocale \ > - =A0 =A0 =A0 clearerr_unlocked feof_unlocked =A0 ferror_unlocked fflush_= unlocked fgetc_unlocked fgets_unlocked =A0 fileno_unlocked fprintf_unlocked= fputc_unlocked fputs_unlocked =A0 fread_unlocked fwrite_unlocked getchar_u= nlocked getc_unlocked =A0 putchar_unlocked putc_unlocked > + =A0 =A0 =A0 clearerr_unlocked feof_unlocked =A0 ferror_unlocked fflush_= unlocked fgetc_unlocked fgets_unlocked =A0 fileno_unlocked fprintf_unlocked= fputc_unlocked fputs_unlocked =A0 fread_unlocked fwrite_unlocked getchar_u= nlocked getc_unlocked =A0 putchar_unlocked putc_unlocked madvise > =A0do : > =A0 as_ac_var=3D`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` > =A0ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" > diff --git a/gcc/configure.ac b/gcc/configure.ac > index a7b94e6..357902e 100644 > --- a/gcc/configure.ac > +++ b/gcc/configure.ac > @@ -1027,7 +1027,7 @@ define(gcc_UNLOCKED_FUNCS, clearerr_unlocked feof_u= nlocked dnl > =A0AC_CHECK_FUNCS(times clock kill getrlimit setrlimit atoll atoq \ > =A0 =A0 =A0 =A0sysconf strsignal getrusage nl_langinfo \ > =A0 =A0 =A0 =A0gettimeofday mbstowcs wcswidth mmap setlocale \ > - =A0 =A0 =A0 gcc_UNLOCKED_FUNCS) > + =A0 =A0 =A0 gcc_UNLOCKED_FUNCS madvise) > > =A0if test x$ac_cv_func_mbstowcs =3D xyes; then > =A0 AC_CACHE_CHECK(whether mbstowcs works, gcc_cv_func_mbstowcs_works, > diff --git a/gcc/ggc-page.c b/gcc/ggc-page.c > index 624f029..b0b3b3f 100644 > --- a/gcc/ggc-page.c > +++ b/gcc/ggc-page.c > @@ -50,6 +50,10 @@ along with GCC; see the file COPYING3. =A0If not see > =A0#define USING_MALLOC_PAGE_GROUPS > =A0#endif > > +#if defined(HAVE_MADVISE) && defined(MADV_DONTNEED) > +# define USING_MADVISE > +#endif > + > =A0/* Strategy: > > =A0 =A0This garbage-collecting allocator allocates objects on one of a set > @@ -277,6 +281,9 @@ typedef struct page_entry > =A0 /* The lg of size of objects allocated from this page. =A0*/ > =A0 unsigned char order; > > + =A0/* Unmapped page? */ > + =A0bool unmapped; > + > =A0 /* A bit vector indicating whether or not objects are in use. =A0The > =A0 =A0 =A0Nth bit is one if the Nth object on this page is allocated. = =A0This > =A0 =A0 =A0array is dynamically sized. =A0*/ > @@ -740,6 +747,10 @@ alloc_page (unsigned order) > > =A0 if (p !=3D NULL) > =A0 =A0 { > + =A0 =A0 =A0if (p->unmapped) > + =A0 =A0 =A0 =A0G.bytes_mapped +=3D p->bytes; > + =A0 =A0 =A0p->unmapped =3D false; > + > =A0 =A0 =A0 /* Recycle the allocated memory from this page ... =A0*/ > =A0 =A0 =A0 *pp =3D p->next; > =A0 =A0 =A0 page =3D p->page; > @@ -956,7 +967,42 @@ free_page (page_entry *entry) > =A0static void > =A0release_pages (void) > =A0{ > -#ifdef USING_MMAP > +#ifdef USING_MADVISE > + =A0page_entry *p, *start_p; > + =A0char *start; > + =A0size_t len; > + > + =A0for (p =3D G.free_pages; p; ) > + =A0 =A0{ > + =A0 =A0 =A0if (p->unmapped) > + =A0 =A0 =A0 =A0{ > + =A0 =A0 =A0 =A0 =A0p =3D p->next; > + =A0 =A0 =A0 =A0 =A0continue; > + =A0 =A0 =A0 =A0} > + =A0 =A0 =A0start =3D p->page; > + =A0 =A0 =A0len =3D p->bytes; > + =A0 =A0 =A0start_p =3D p; > + =A0 =A0 =A0p =3D p->next; > + =A0 =A0 =A0while (p && p->page =3D=3D start + len) > + =A0 =A0 =A0 =A0{ > + =A0 =A0 =A0 =A0 =A0len +=3D p->bytes; > + =A0 =A0 =A0 =A0 =A0p =3D p->next; > + =A0 =A0 =A0 =A0} > + =A0 =A0 =A0/* Give the page back to the kernel, but don't free the mapp= ing. > + =A0 =A0 =A0 =A0 This avoids fragmentation in the virtual memory map of = the > + =A0 =A0 =A0 =A0process. Next time we can reuse it by just touching it. = */ > + =A0 =A0 =A0madvise (start, len, MADV_DONTNEED); > + =A0 =A0 =A0/* Don't count those pages as mapped to not touch the garbag= e collector > + =A0 =A0 =A0 =A0 unnecessarily. */ > + =A0 =A0 =A0G.bytes_mapped -=3D len; > + =A0 =A0 =A0while (start_p !=3D p) > + =A0 =A0 =A0 =A0{ > + =A0 =A0 =A0 =A0 =A0start_p->unmapped =3D true; > + =A0 =A0 =A0 =A0 =A0start_p =3D start_p->next; > + =A0 =A0 =A0 =A0} > + =A0 =A0} > +#endif > +#if defined(USING_MMAP) && !defined(USING_MADVISE) > =A0 page_entry *p, *next; > =A0 char *start; > =A0 size_t len; > -- > 1.7.5.4 > >