* [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so @ 2016-04-02 15:34 H.J. Lu 2016-04-02 17:33 ` Mike Frysinger 0 siblings, 1 reply; 7+ messages in thread From: H.J. Lu @ 2016-04-02 15:34 UTC (permalink / raw) To: GNU C Library __libc_memalign in ld.so allocates one page at a time and tries to optimize consecutive __libc_memalign calls by hoping that the next mmap is after the current memory allocation. However, the kernel hands out mmap addresses in top-down order, so this optimization in practice never happens, with the result that we have more mmap calls and waste a bunch of space for each __libc_memalign. This change makes __libc_memalign to mmap one page extra. Worst case, the kernel never puts a backing page behind it, but best case it allows __libc_memalign to operate much much better. For elf/tst-align --direct, it reduces number of mmap calls from 12 to 9. Tested on x86-64. OK for master? H.J. --- * elf/dl-minimal.c (__libc_memalign): Mmap one extra page. --- elf/dl-minimal.c | 1 + 1 file changed, 1 insertion(+) diff --git a/elf/dl-minimal.c b/elf/dl-minimal.c index 762e65b..d6f87f1 100644 --- a/elf/dl-minimal.c +++ b/elf/dl-minimal.c @@ -75,6 +75,7 @@ __libc_memalign (size_t align, size_t n) return NULL; nup = GLRO(dl_pagesize); } + nup += GLRO(dl_pagesize); page = __mmap (0, nup, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0); if (page == MAP_FAILED) -- 2.5.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so 2016-04-02 15:34 [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so H.J. Lu @ 2016-04-02 17:33 ` Mike Frysinger 2016-04-02 17:43 ` H.J. Lu 0 siblings, 1 reply; 7+ messages in thread From: Mike Frysinger @ 2016-04-02 17:33 UTC (permalink / raw) To: H.J. Lu; +Cc: GNU C Library [-- Attachment #1: Type: text/plain, Size: 1038 bytes --] On 02 Apr 2016 08:34, H.J. Lu wrote: > __libc_memalign in ld.so allocates one page at a time and tries to > optimize consecutive __libc_memalign calls by hoping that the next > mmap is after the current memory allocation. > > However, the kernel hands out mmap addresses in top-down order, so > this optimization in practice never happens, with the result that we > have more mmap calls and waste a bunch of space for each __libc_memalign. > > This change makes __libc_memalign to mmap one page extra. Worst case, > the kernel never puts a backing page behind it, but best case it allows > __libc_memalign to operate much much better. For elf/tst-align --direct, > it reduces number of mmap calls from 12 to 9. > > --- a/elf/dl-minimal.c > +++ b/elf/dl-minimal.c > @@ -75,6 +75,7 @@ __libc_memalign (size_t align, size_t n) > return NULL; > nup = GLRO(dl_pagesize); > } > + nup += GLRO(dl_pagesize); should this be in the else case ? also the comment above this code needs updating -mike [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so 2016-04-02 17:33 ` Mike Frysinger @ 2016-04-02 17:43 ` H.J. Lu 2016-04-02 22:41 ` H.J. Lu 0 siblings, 1 reply; 7+ messages in thread From: H.J. Lu @ 2016-04-02 17:43 UTC (permalink / raw) To: GNU C Library [-- Attachment #1: Type: text/plain, Size: 1180 bytes --] On Sat, Apr 2, 2016 at 10:33 AM, Mike Frysinger <vapier@gentoo.org> wrote: > On 02 Apr 2016 08:34, H.J. Lu wrote: >> __libc_memalign in ld.so allocates one page at a time and tries to >> optimize consecutive __libc_memalign calls by hoping that the next >> mmap is after the current memory allocation. >> >> However, the kernel hands out mmap addresses in top-down order, so >> this optimization in practice never happens, with the result that we >> have more mmap calls and waste a bunch of space for each __libc_memalign. >> >> This change makes __libc_memalign to mmap one page extra. Worst case, >> the kernel never puts a backing page behind it, but best case it allows >> __libc_memalign to operate much much better. For elf/tst-align --direct, >> it reduces number of mmap calls from 12 to 9. >> >> --- a/elf/dl-minimal.c >> +++ b/elf/dl-minimal.c >> @@ -75,6 +75,7 @@ __libc_memalign (size_t align, size_t n) >> return NULL; >> nup = GLRO(dl_pagesize); >> } >> + nup += GLRO(dl_pagesize); > > should this be in the else case ? > > also the comment above this code needs updating > -mike You are right. Here is the updated patch. -- H.J. [-- Attachment #2: 0001-Reduce-number-of-mmap-calls-from-__libc_memalign-in-.patch --] [-- Type: text/x-patch, Size: 1875 bytes --] From d56ca4f3269e47cba3e8d22ba8e48cd20d470757 Mon Sep 17 00:00:00 2001 From: "H.J. Lu" <hjl.tools@gmail.com> Date: Sat, 2 Apr 2016 08:25:31 -0700 Subject: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so __libc_memalign in ld.so allocates one page at a time and tries to optimize consecutive __libc_memalign calls by hoping that the next mmap is after the current memory allocation. However, the kernel hands out mmap addresses in top-down order, so this optimization in practice never happens, with the result that we have more mmap calls and waste a bunch of space for each __libc_memalign. This change makes __libc_memalign to mmap one page extra. Worst case, the kernel never puts a backing page behind it, but best case it allows __libc_memalign to operate much much better. For elf/tst-align --direct, it reduces number of mmap calls from 12 to 9. * elf/dl-minimal.c (__libc_memalign): Mmap one extra page. --- elf/dl-minimal.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/elf/dl-minimal.c b/elf/dl-minimal.c index 762e65b..8bffdc7 100644 --- a/elf/dl-minimal.c +++ b/elf/dl-minimal.c @@ -66,7 +66,8 @@ __libc_memalign (size_t align, size_t n) if (alloc_ptr + n >= alloc_end || n >= -(uintptr_t) alloc_ptr) { - /* Insufficient space left; allocate another page. */ + /* Insufficient space left; allocate another page plus one extra + page to reduce number of mmap calls. */ caddr_t page; size_t nup = (n + GLRO(dl_pagesize) - 1) & ~(GLRO(dl_pagesize) - 1); if (__glibc_unlikely (nup == 0)) @@ -75,6 +76,8 @@ __libc_memalign (size_t align, size_t n) return NULL; nup = GLRO(dl_pagesize); } + else + nup += GLRO(dl_pagesize); page = __mmap (0, nup, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0); if (page == MAP_FAILED) -- 2.5.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so 2016-04-02 17:43 ` H.J. Lu @ 2016-04-02 22:41 ` H.J. Lu 2016-04-03 6:56 ` Andreas Schwab 0 siblings, 1 reply; 7+ messages in thread From: H.J. Lu @ 2016-04-02 22:41 UTC (permalink / raw) To: GNU C Library [-- Attachment #1: Type: text/plain, Size: 1337 bytes --] On Sat, Apr 2, 2016 at 10:43 AM, H.J. Lu <hjl.tools@gmail.com> wrote: > On Sat, Apr 2, 2016 at 10:33 AM, Mike Frysinger <vapier@gentoo.org> wrote: >> On 02 Apr 2016 08:34, H.J. Lu wrote: >>> __libc_memalign in ld.so allocates one page at a time and tries to >>> optimize consecutive __libc_memalign calls by hoping that the next >>> mmap is after the current memory allocation. >>> >>> However, the kernel hands out mmap addresses in top-down order, so >>> this optimization in practice never happens, with the result that we >>> have more mmap calls and waste a bunch of space for each __libc_memalign. >>> >>> This change makes __libc_memalign to mmap one page extra. Worst case, >>> the kernel never puts a backing page behind it, but best case it allows >>> __libc_memalign to operate much much better. For elf/tst-align --direct, >>> it reduces number of mmap calls from 12 to 9. >>> >>> --- a/elf/dl-minimal.c >>> +++ b/elf/dl-minimal.c >>> @@ -75,6 +75,7 @@ __libc_memalign (size_t align, size_t n) >>> return NULL; >>> nup = GLRO(dl_pagesize); >>> } >>> + nup += GLRO(dl_pagesize); >> >> should this be in the else case ? >> >> also the comment above this code needs updating >> -mike > > You are right. Here is the updated patch. > We can just always increment number of pages by one. -- H.J. [-- Attachment #2: 0001-Reduce-number-of-mmap-calls-from-__libc_memalign-in-.patch --] [-- Type: text/x-patch, Size: 1895 bytes --] From 4aad224c5dc8c8e8496868cc1bb00d587aa4f1ed Mon Sep 17 00:00:00 2001 From: "H.J. Lu" <hjl.tools@gmail.com> Date: Sat, 2 Apr 2016 08:25:31 -0700 Subject: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so __libc_memalign in ld.so allocates one page at a time and tries to optimize consecutive __libc_memalign calls by hoping that the next mmap is after the current memory allocation. However, the kernel hands out mmap addresses in top-down order, so this optimization in practice never happens, with the result that we have more mmap calls and waste a bunch of space for each __libc_memalign. This change makes __libc_memalign to mmap one page extra. Worst case, the kernel never puts a backing page behind it, but best case it allows __libc_memalign to operate much much better. For elf/tst-align --direct, it reduces number of mmap calls from 12 to 9. * elf/dl-minimal.c (__libc_memalign): Mmap one extra page. --- elf/dl-minimal.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/elf/dl-minimal.c b/elf/dl-minimal.c index 762e65b..0660fcf 100644 --- a/elf/dl-minimal.c +++ b/elf/dl-minimal.c @@ -66,15 +66,13 @@ __libc_memalign (size_t align, size_t n) if (alloc_ptr + n >= alloc_end || n >= -(uintptr_t) alloc_ptr) { - /* Insufficient space left; allocate another page. */ + /* Insufficient space left; allocate another page plus one extra + page to reduce number of mmap calls. */ caddr_t page; size_t nup = (n + GLRO(dl_pagesize) - 1) & ~(GLRO(dl_pagesize) - 1); - if (__glibc_unlikely (nup == 0)) - { - if (n) - return NULL; - nup = GLRO(dl_pagesize); - } + if (__glibc_unlikely (nup == 0 && n)) + return NULL; + nup += GLRO(dl_pagesize); page = __mmap (0, nup, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0); if (page == MAP_FAILED) -- 2.5.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so 2016-04-02 22:41 ` H.J. Lu @ 2016-04-03 6:56 ` Andreas Schwab 2016-04-03 13:42 ` H.J. Lu 0 siblings, 1 reply; 7+ messages in thread From: Andreas Schwab @ 2016-04-03 6:56 UTC (permalink / raw) To: H.J. Lu; +Cc: GNU C Library "H.J. Lu" <hjl.tools@gmail.com> writes: > + if (__glibc_unlikely (nup == 0 && n)) Please also fix the implicit boolean coercion. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so 2016-04-03 6:56 ` Andreas Schwab @ 2016-04-03 13:42 ` H.J. Lu 2016-04-23 12:57 ` H.J. Lu 0 siblings, 1 reply; 7+ messages in thread From: H.J. Lu @ 2016-04-03 13:42 UTC (permalink / raw) To: Andreas Schwab; +Cc: GNU C Library [-- Attachment #1: Type: text/plain, Size: 275 bytes --] On Sat, Apr 2, 2016 at 11:55 PM, Andreas Schwab <schwab@linux-m68k.org> wrote: > "H.J. Lu" <hjl.tools@gmail.com> writes: > >> + if (__glibc_unlikely (nup == 0 && n)) > > Please also fix the implicit boolean coercion. > > Andreas. > Like this? OK for master? -- H.J. [-- Attachment #2: 0001-Reduce-number-of-mmap-calls-from-__libc_memalign-in-.patch --] [-- Type: application/octet-stream, Size: 1900 bytes --] From 5d673379a907611be18ebd48a16a2df7b3ab3f9c Mon Sep 17 00:00:00 2001 From: "H.J. Lu" <hjl.tools@gmail.com> Date: Sat, 2 Apr 2016 08:25:31 -0700 Subject: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so __libc_memalign in ld.so allocates one page at a time and tries to optimize consecutive __libc_memalign calls by hoping that the next mmap is after the current memory allocation. However, the kernel hands out mmap addresses in top-down order, so this optimization in practice never happens, with the result that we have more mmap calls and waste a bunch of space for each __libc_memalign. This change makes __libc_memalign to mmap one page extra. Worst case, the kernel never puts a backing page behind it, but best case it allows __libc_memalign to operate much much better. For elf/tst-align --direct, it reduces number of mmap calls from 12 to 9. * elf/dl-minimal.c (__libc_memalign): Mmap one extra page. --- elf/dl-minimal.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/elf/dl-minimal.c b/elf/dl-minimal.c index 762e65b..c8a8f8d 100644 --- a/elf/dl-minimal.c +++ b/elf/dl-minimal.c @@ -66,15 +66,13 @@ __libc_memalign (size_t align, size_t n) if (alloc_ptr + n >= alloc_end || n >= -(uintptr_t) alloc_ptr) { - /* Insufficient space left; allocate another page. */ + /* Insufficient space left; allocate another page plus one extra + page to reduce number of mmap calls. */ caddr_t page; size_t nup = (n + GLRO(dl_pagesize) - 1) & ~(GLRO(dl_pagesize) - 1); - if (__glibc_unlikely (nup == 0)) - { - if (n) - return NULL; - nup = GLRO(dl_pagesize); - } + if (__glibc_unlikely (nup == 0 && n != 0)) + return NULL; + nup += GLRO(dl_pagesize); page = __mmap (0, nup, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0); if (page == MAP_FAILED) -- 2.5.5 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so 2016-04-03 13:42 ` H.J. Lu @ 2016-04-23 12:57 ` H.J. Lu 0 siblings, 0 replies; 7+ messages in thread From: H.J. Lu @ 2016-04-23 12:57 UTC (permalink / raw) To: Andreas Schwab; +Cc: GNU C Library On Sun, Apr 3, 2016 at 6:42 AM, H.J. Lu <hjl.tools@gmail.com> wrote: > On Sat, Apr 2, 2016 at 11:55 PM, Andreas Schwab <schwab@linux-m68k.org> wrote: >> "H.J. Lu" <hjl.tools@gmail.com> writes: >> >>> + if (__glibc_unlikely (nup == 0 && n)) >> >> Please also fix the implicit boolean coercion. >> >> Andreas. >> > > Like this? OK for master? > I am checking it now. -- H.J. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-04-23 12:57 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-04-02 15:34 [PATCH] Reduce number of mmap calls from __libc_memalign in ld.so H.J. Lu 2016-04-02 17:33 ` Mike Frysinger 2016-04-02 17:43 ` H.J. Lu 2016-04-02 22:41 ` H.J. Lu 2016-04-03 6:56 ` Andreas Schwab 2016-04-03 13:42 ` H.J. Lu 2016-04-23 12:57 ` H.J. Lu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).