From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 12435 invoked by alias); 18 Sep 2011 16:10:21 -0000 Received: (qmail 12394 invoked by uid 22791); 18 Sep 2011 16:10:09 -0000 X-SWARE-Spam-Status: No, hits=-6.2 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,SPF_HELO_PASS X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Sun, 18 Sep 2011 16:09:45 +0000 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p8IG9i8C025571 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 18 Sep 2011 12:09:44 -0400 Received: from localhost (ovpn-113-40.phx2.redhat.com [10.3.113.40]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p8IG9gcH020682 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 18 Sep 2011 12:09:44 -0400 Received: by localhost (Postfix, from userid 500) id C1F5D29C111; Sun, 18 Sep 2011 18:09:41 +0200 (CEST) From: Dodji Seketeli To: Jason Merrill Cc: gcc-patches@gcc.gnu.org, tromey@redhat.com, Laurynas Biveinis Subject: Re: [PATCH 7/7] Reduce memory waste due to non-power-of-2 allocs References: <1291979498-1604-1-git-send-email-dodji@redhat.com> <411c73e26bd46ba176c22962851cacf21b013be5.1310824121.git.dodji@redhat.com> <4E6E825F.8020408@redhat.com> <4E74EBA7.1010505@redhat.com> X-URL: http://www.redhat.com Date: Sun, 18 Sep 2011 22:30:00 -0000 In-Reply-To: <4E74EBA7.1010505@redhat.com> (Jason Merrill's message of "Sat, 17 Sep 2011 14:49:11 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org X-SW-Source: 2011-09/txt/msg01048.txt.bz2 Jason Merrill writes: > On 09/17/2011 07:08 AM, Dodji Seketeli wrote: >> OK, so the patch below extracts a public ggc_alloced_size_for_request >> function from the different implementations of the ggc allocator's >> interface, and lets new_linemap use that. > > Maybe "ggc_round_alloc_size"? OK, updated the patch below accordingly. > OK with that change if nobody else has comments this week. Thanks. Below is the updated patch. From: Dodji Seketeli Date: Tue, 17 May 2011 16:48:01 +0200 Subject: [PATCH 7/7] Reduce memory waste due to non-power-of-2 allocs This patch basically arranges for the allocation size of line_map buffers to be as close as possible to a power of two. This *significantly* decreases peak memory consumption as (macro) maps are numerous and stay live during all the compilation. The patch adds a new ggc_round_alloc_size interface to the ggc allocator. In each of the two main allocator implementations of ('page' and 'zone') the function has been extracted from the main allocation function code and returns the actual size of the allocated memory region, thus giving a chance to the caller to maximize the amount of memory it actually uses from the allocated memory region. In the 'none' allocator implementation (that uses xmalloc) the ggc_round_alloc_size just returns the requested allocation size. Tested on x86_64-unknown-linux-gnu against trunk for each allocator. libcpp/ * include/line-map.h (struct line_maps::alloced_size_for_request): New member. * line-map.c (new_linemap): Use set->alloced_size_for_request to get the actual allocated size of line maps. gcc/ * ggc.h (ggc_round_alloc_size): Declare new public entry point. * ggc-none.c (ggc_round_alloc_size): New public stub function. * ggc-page.c (ggc_alloced_size_order_for_request): New static function. Factorized from ggc_internal_alloc_stat. (ggc_round_alloc_size): New public function. Uses ggc_alloced_size_order_for_request. (ggc_internal_alloc_stat): Use ggc_alloced_size_order_for_request. * ggc-zone.c (ggc_round_alloc_size): New public function extracted from ggc_internal_alloc_zone_stat. (ggc_internal_alloc_zone_stat): Use ggc_round_alloc_size. * toplev.c (general_init): Initialize line_table->alloced_size_for_request. --- gcc/ggc-none.c | 9 +++++++ gcc/ggc-page.c | 53 +++++++++++++++++++++++++++++++++++--------- gcc/ggc-zone.c | 27 ++++++++++++++++------ gcc/ggc.h | 2 + gcc/toplev.c | 1 + libcpp/include/line-map.h | 8 ++++++ libcpp/line-map.c | 39 ++++++++++++++++++++++++++++----- 7 files changed, 114 insertions(+), 25 deletions(-) diff --git a/gcc/ggc-none.c b/gcc/ggc-none.c index 97d25b9..e57d617 100644 --- a/gcc/ggc-none.c +++ b/gcc/ggc-none.c @@ -39,6 +39,15 @@ ggc_alloc_typed_stat (enum gt_types_enum ARG_UNUSED (gte), size_t size return xmalloc (size); } +/* For a given size of memory requested for allocation, return the + actual size that is going to be allocated. */ + +size_t +ggc_round_alloc_size (size_t requested_size) +{ + return requested_size; +} + void * ggc_internal_alloc_stat (size_t size MEM_STAT_DECL) { diff --git a/gcc/ggc-page.c b/gcc/ggc-page.c index 624f029..f919a6b 100644 --- a/gcc/ggc-page.c +++ b/gcc/ggc-page.c @@ -1054,6 +1054,47 @@ static unsigned char size_lookup[NUM_SIZE_LOOKUP] = 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9 }; +/* For a given size of memory requested for allocation, return the + actual size that is going to be allocated, as well as the size + order. */ + +static void +ggc_round_alloc_size_1 (size_t requested_size, + size_t *size_order, + size_t *alloced_size) +{ + size_t order, object_size; + + if (requested_size < NUM_SIZE_LOOKUP) + { + order = size_lookup[requested_size]; + object_size = OBJECT_SIZE (order); + } + else + { + order = 10; + while (requested_size > (object_size = OBJECT_SIZE (order))) + order++; + } + + if (size_order) + *size_order = order; + if (alloced_size) + *alloced_size = object_size; +} + +/* For a given size of memory requested for allocation, return the + actual size that is going to be allocated. */ + +size_t +ggc_round_alloc_size (size_t requested_size) +{ + size_t size = 0; + + ggc_round_alloc_size_1 (requested_size, NULL, &size); + return size; +} + /* Typed allocation function. Does nothing special in this collector. */ void * @@ -1072,17 +1113,7 @@ ggc_internal_alloc_stat (size_t size MEM_STAT_DECL) struct page_entry *entry; void *result; - if (size < NUM_SIZE_LOOKUP) - { - order = size_lookup[size]; - object_size = OBJECT_SIZE (order); - } - else - { - order = 10; - while (size > (object_size = OBJECT_SIZE (order))) - order++; - } + ggc_round_alloc_size_1 (size, &order, &object_size); /* If there are non-full pages for this size allocation, they are at the head of the list. */ diff --git a/gcc/ggc-zone.c b/gcc/ggc-zone.c index d0c1d79..79c8c03 100644 --- a/gcc/ggc-zone.c +++ b/gcc/ggc-zone.c @@ -1073,6 +1073,24 @@ free_chunk (char *ptr, size_t size, struct alloc_zone *zone) fprintf (G.debug_file, "Deallocating object, chunk=%p\n", (void *)chunk); } +/* For a given size of memory requested for allocation, return the + actual size that is going to be allocated. */ + +size_t +ggc_round_alloc_size (size_t requested_size) +{ + size_t size; + + /* Make sure that zero-sized allocations get a unique and freeable + pointer. */ + if (requested_size == 0) + size = MAX_ALIGNMENT; + else + size = (requested_size + MAX_ALIGNMENT - 1) & -MAX_ALIGNMENT; + + return size; +} + /* Allocate a chunk of memory of at least ORIG_SIZE bytes, in ZONE. */ void * @@ -1084,14 +1102,7 @@ ggc_internal_alloc_zone_stat (size_t orig_size, struct alloc_zone *zone struct small_page_entry *entry; struct alloc_chunk *chunk, **pp; void *result; - size_t size = orig_size; - - /* Make sure that zero-sized allocations get a unique and freeable - pointer. */ - if (size == 0) - size = MAX_ALIGNMENT; - else - size = (size + MAX_ALIGNMENT - 1) & -MAX_ALIGNMENT; + size_t size = ggc_alloced_size_for_request (orig_size); /* Try to allocate the object from several different sources. Each of these cases is responsible for setting RESULT and SIZE to diff --git a/gcc/ggc.h b/gcc/ggc.h index 30eca66..704237c 100644 --- a/gcc/ggc.h +++ b/gcc/ggc.h @@ -145,6 +145,8 @@ extern void gt_pch_save (FILE *f); /* The internal primitive. */ extern void *ggc_internal_alloc_stat (size_t MEM_STAT_DECL); +extern size_t ggc_round_alloc_size (size_t requested_size); + #define ggc_internal_alloc(s) ggc_internal_alloc_stat (s MEM_STAT_INFO) /* Allocate an object of the specified type and size. */ diff --git a/gcc/toplev.c b/gcc/toplev.c index 2f90261..7b4bd16 100644 --- a/gcc/toplev.c +++ b/gcc/toplev.c @@ -1177,6 +1177,7 @@ general_init (const char *argv0) line_table = ggc_alloc_line_maps (); linemap_init (line_table); line_table->reallocator = realloc_for_line_map; + line_table->round_alloc_size = ggc_round_alloc_size; init_ttree (); /* Initialize register usage now so switches may override. */ diff --git a/libcpp/include/line-map.h b/libcpp/include/line-map.h index 3989c89..62b288f 100644 --- a/libcpp/include/line-map.h +++ b/libcpp/include/line-map.h @@ -53,6 +53,10 @@ typedef unsigned int source_location; /* Memory allocation function typedef. Works like xrealloc. */ typedef void *(*line_map_realloc) (void *, size_t); +/* Memory allocator function that returns the actual allocated size, + for a given requested allocation. */ +typedef size_t (*line_map_round_alloc_size_func) (size_t); + /* An ordinary line map encodes physical source locations. Those physical source locations are called "spelling locations". @@ -298,6 +302,10 @@ struct GTY(()) line_maps { /* If non-null, the allocator to use when resizing 'maps'. If null, xrealloc is used. */ line_map_realloc reallocator; + + /* The allocators' function used to know the actual size it + allocated, for a certain allocation size requested. */ + line_map_round_alloc_size_func round_alloc_size; }; /* Returns the pointer to the memory region where information about diff --git a/libcpp/line-map.c b/libcpp/line-map.c index 09dffcb..935024c 100644 --- a/libcpp/line-map.c +++ b/libcpp/line-map.c @@ -77,16 +77,43 @@ new_linemap (struct line_maps *set, if (LINEMAPS_USED (set, macro_map_p) == LINEMAPS_ALLOCATED (set, macro_map_p)) { /* We ran out of allocated line maps. Let's allocate more. */ + unsigned alloc_size; line_map_realloc reallocator = set->reallocator ? set->reallocator : xrealloc; + line_map_round_alloc_size_func round_alloc_size = + set->round_alloc_size; + + /* We are going to execute some dance to try to reduce the + overhead of the memory allocator, in case we are using the + ggc-page.c one. + + The actual size of memory we are going to get back from the + allocator is the smallest power of 2 that is greater than the + size we requested. So let's consider that size then. */ + + alloc_size = + (2 * LINEMAPS_ALLOCATED (set, macro_map_p) + 256) + * sizeof (struct line_map); + + /* Get the actual size of memory that is going to be allocated + by the allocator. */ + alloc_size = round_alloc_size (alloc_size); + + /* Now alloc_size contains the exact memory size we would get if + we have asked for the initial alloc_size amount of memory. + Let's get back to the number of macro map that amounts + to. */ LINEMAPS_ALLOCATED (set, macro_map_p) = - 2 * LINEMAPS_ALLOCATED (set, macro_map_p) + 256; - LINEMAPS_MAPS (set, macro_map_p) - = (struct line_map *) (*reallocator) (LINEMAPS_MAPS (set, macro_map_p), - LINEMAPS_ALLOCATED (set, - macro_map_p) - * sizeof (struct line_map)); + alloc_size / (sizeof (struct line_map)); + + /* And now let's really do the re-allocation. */ + LINEMAPS_MAPS (set, macro_map_p) = + (struct line_map *) (*reallocator) + (LINEMAPS_MAPS (set, macro_map_p), + (LINEMAPS_ALLOCATED (set, macro_map_p) + * sizeof (struct line_map))); + result = &LINEMAPS_MAPS (set, macro_map_p)[LINEMAPS_USED (set, macro_map_p)]; memset (result, 0, -- Dodji