* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
@ 2014-07-24 11:07 ` jtaylor.debian at googlemail dot com
2014-08-30 12:34 ` neleai at seznam dot cz
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: jtaylor.debian at googlemail dot com @ 2014-07-24 11:07 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
Julian Taylor <jtaylor.debian at googlemail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|excessive heap trimming in |excessive heap trimming in
|threaded programs |threaded programs even when
| |disabled with mallopt
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
2014-07-24 11:07 ` [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt jtaylor.debian at googlemail dot com
@ 2014-08-30 12:34 ` neleai at seznam dot cz
2014-08-30 12:40 ` jtaylor.debian at googlemail dot com
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: neleai at seznam dot cz @ 2014-08-30 12:34 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
Ondrej Bilka <neleai at seznam dot cz> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
CC| |neleai at seznam dot cz
Resolution|--- |DUPLICATE
--- Comment #1 from Ondrej Bilka <neleai at seznam dot cz> ---
Yes that is known issue
*** This bug has been marked as a duplicate of bug 1128 ***
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
2014-07-24 11:07 ` [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt jtaylor.debian at googlemail dot com
2014-08-30 12:34 ` neleai at seznam dot cz
@ 2014-08-30 12:40 ` jtaylor.debian at googlemail dot com
2015-02-12 17:46 ` jtaylor.debian at googlemail dot com
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: jtaylor.debian at googlemail dot com @ 2014-08-30 12:40 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
--- Comment #2 from Julian Taylor <jtaylor.debian at googlemail dot com> ---
my issue is not with sbrk but with madvise in threaded applications which
cannot be worked around by using mallopt like in the other bug.
the trimming can be disabled in single threaded applications but not
multithreaded ones.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
` (2 preceding siblings ...)
2014-08-30 12:40 ` jtaylor.debian at googlemail dot com
@ 2015-02-12 17:46 ` jtaylor.debian at googlemail dot com
2015-02-18 5:48 ` siddhesh at redhat dot com
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: jtaylor.debian at googlemail dot com @ 2015-02-12 17:46 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
--- Comment #3 from Julian Taylor <jtaylor.debian at googlemail dot com> ---
a patch has been posted on the list (not by me):
https://sourceware.org/ml/libc-alpha/2015-02/msg00168.html
I still think the duplicate is wrong, this is about madvise in threads, not
sbrk.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
` (3 preceding siblings ...)
2015-02-12 17:46 ` jtaylor.debian at googlemail dot com
@ 2015-02-18 5:48 ` siddhesh at redhat dot com
2015-04-02 6:46 ` cvs-commit at gcc dot gnu.org
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: siddhesh at redhat dot com @ 2015-02-18 5:48 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
Siddhesh Poyarekar <siddhesh at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
CC| |siddhesh at redhat dot com
Resolution|DUPLICATE |---
--- Comment #4 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
Right, it's not a duplicate. I'll have the patch pushed soon.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
` (4 preceding siblings ...)
2015-02-18 5:48 ` siddhesh at redhat dot com
@ 2015-04-02 6:46 ` cvs-commit at gcc dot gnu.org
2015-04-02 6:48 ` siddhesh at redhat dot com
2015-10-08 2:23 ` cvs-commit at gcc dot gnu.org
7 siblings, 0 replies; 9+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2015-04-02 6:46 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
--- Comment #5 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".
The branch, master has been updated
via c26efef9798914e208329c0e8c3c73bb1135d9e3 (commit)
from a3d9ab5070b56b49aa91be2887fa5b118012b2cd (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=c26efef9798914e208329c0e8c3c73bb1135d9e3
commit c26efef9798914e208329c0e8c3c73bb1135d9e3
Author: Mel Gorman <mgorman@suse.de>
Date: Thu Apr 2 12:14:14 2015 +0530
malloc: Consistently apply trim_threshold to all heaps [BZ #17195]
Trimming heaps is a balance between saving memory and the system overhead
required to update page tables and discard allocated pages. The malloc
option M_TRIM_THRESHOLD is a tunable that users are meant to use to decide
where this balance point is but it is only applied to the main arena.
For scalability reasons, glibc malloc has per-thread heaps but these are
shrunk with madvise() if there is one page free at the top of the heap.
In some circumstances this can lead to high system overhead if a thread
has a control flow like
while (data_to_process) {
buf = malloc(large_size);
do_stuff();
free(buf);
}
For a large size, the free() will call madvise (pagetable teardown, page
free and TLB flush) every time followed immediately by a malloc (fault,
kernel page alloc, zeroing and charge accounting). The kernel overhead
can dominate such a workload.
This patch allows the user to tune when madvise gets called by applying
the trim threshold to the per-thread heaps and using similar logic to the
main arena when deciding whether to shrink. Alternatively if the dynamic
brk/mmap threshold gets adjusted then the new values will be obeyed by
the per-thread heaps.
Bug 17195 was a test case motivated by a problem encountered in scientific
applications written in python that performance badly due to high page
fault
overhead. The basic operation of such a program was posted by Julian Taylor
https://sourceware.org/ml/libc-alpha/2015-02/msg00373.html
With this patch applied, the overhead is eliminated. All numbers in this
report are in seconds and were recorded by running Julian's program 30
times.
pyarray
glibc madvise
2.21 v2
System min 1.81 ( 0.00%) 0.00 (100.00%)
System mean 1.93 ( 0.00%) 0.02 ( 99.20%)
System stddev 0.06 ( 0.00%) 0.01 ( 88.99%)
System max 2.06 ( 0.00%) 0.03 ( 98.54%)
Elapsed min 3.26 ( 0.00%) 2.37 ( 27.30%)
Elapsed mean 3.39 ( 0.00%) 2.41 ( 28.84%)
Elapsed stddev 0.14 ( 0.00%) 0.02 ( 82.73%)
Elapsed max 4.05 ( 0.00%) 2.47 ( 39.01%)
glibc madvise
2.21 v2
User 141.86 142.28
System 57.94 0.60
Elapsed 102.02 72.66
Note that almost a minutes worth of system time is eliminted and the
program completes 28% faster on average.
To illustrate the problem without python this is a basic test-case for
the worst case scenario where every free is a madvise followed by a an
alloc
/* gcc bench-free.c -lpthread -o bench-free */
static int num = 1024;
void __attribute__((noinline,noclone)) dostuff (void *p)
{
}
void *worker (void *data)
{
int i;
for (i = num; i--;)
{
void *m = malloc (48*4096);
dostuff (m);
free (m);
}
return NULL;
}
int main()
{
int i;
pthread_t t;
void *ret;
if (pthread_create (&t, NULL, worker, NULL))
exit (2);
if (pthread_join (t, &ret))
exit (3);
return 0;
}
Before the patch, this resulted in 1024 calls to madvise. With the patch
applied,
madvise is called twice because the default trim threshold is high enough
to avoid
this.
This a more complex case where there is a mix of frees. It's simply a
different worker
function for the test case above
void *worker (void *data)
{
int i;
int j = 0;
void *free_index[num];
for (i = num; i--;)
{
void *m = malloc ((i % 58) *4096);
dostuff (m);
if (i % 2 == 0) {
free (m);
} else {
free_index[j++] = m;
}
}
for (; j >= 0; j--)
{
free(free_index[j]);
}
return NULL;
}
glibc 2.21 calls malloc 90305 times but with the patch applied, it's
called 13438. Increasing the trim threshold will decrease the number of
times it's called with the option of eliminating the overhead.
ebizzy is meant to generate a workload resembling common web application
server workloads. It is threaded with a large working set that at its core
has an allocation, do_stuff, free loop that also hits this case. The
primary
metric of the benchmark is records processed per second. This is running on
my desktop which is a single socket machine with an I7-4770 and 8 cores.
Each thread count was run for 30 seconds. It was only run once as the
performance difference is so high that the variation is insignificant.
glibc 2.21 patch
threads 1 10230 44114
threads 2 19153 84925
threads 4 34295 134569
threads 8 51007 183387
Note that the saving happens to be a concidence as the size allocated
by ebizzy was less than the default threshold. If a different number of
chunks were specified then it may also be necessary to tune the threshold
to compensate
This is roughly quadrupling the performance of this benchmark. The
difference in
system CPU usage illustrates why.
ebizzy running 1 thread with glibc 2.21
10230 records/s 306904
real 30.00 s
user 7.47 s
sys 22.49 s
22.49 seconds was spent in the kernel for a workload runinng 30 seconds.
With the
patch applied
ebizzy running 1 thread with patch applied
44126 records/s 1323792
real 30.00 s
user 29.97 s
sys 0.00 s
system CPU usage was zero with the patch applied. strace shows that glibc
running this workload calls madvise approximately 9000 times a second. With
the patch applied madvise was called twice during the workload (or 0.06
times per second).
2015-02-10 Mel Gorman <mgorman@suse.de>
[BZ #17195]
* malloc/arena.c (free): Apply trim threshold to per-thread heaps
as well as the main arena.
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 6 ++++++
NEWS | 12 ++++++------
malloc/arena.c | 13 ++++++++++---
3 files changed, 22 insertions(+), 9 deletions(-)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
` (5 preceding siblings ...)
2015-04-02 6:46 ` cvs-commit at gcc dot gnu.org
@ 2015-04-02 6:48 ` siddhesh at redhat dot com
2015-10-08 2:23 ` cvs-commit at gcc dot gnu.org
7 siblings, 0 replies; 9+ messages in thread
From: siddhesh at redhat dot com @ 2015-04-02 6:48 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
Siddhesh Poyarekar <siddhesh at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution|--- |FIXED
--- Comment #6 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
Fixed in master.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug malloc/17195] excessive heap trimming in threaded programs even when disabled with mallopt
2014-07-24 11:04 [Bug malloc/17195] New: excessive heap trimming in threaded programs jtaylor.debian at googlemail dot com
` (6 preceding siblings ...)
2015-04-02 6:48 ` siddhesh at redhat dot com
@ 2015-10-08 2:23 ` cvs-commit at gcc dot gnu.org
7 siblings, 0 replies; 9+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2015-10-08 2:23 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=17195
--- Comment #8 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".
The branch, master has been updated
via e4bc326dbbf7328775fe7dd39de1178821363e0a (commit)
from 58a3a98d8f3488b659318b1a1c6efc169a6f06bf (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=e4bc326dbbf7328775fe7dd39de1178821363e0a
commit e4bc326dbbf7328775fe7dd39de1178821363e0a
Author: Carlos O'Donell <carlos@systemhalted.org>
Date: Wed Oct 7 22:21:36 2015 -0400
malloc: Consistently apply trim_threshold to all heaps (Bug 17195)
In the per-thread arenas we apply trim_threshold-based checks
to the extra space between the pad and the top_area. This isn't
quite accurate and instead we should be harmonizing with the way
in which trim_treshold is applied everywhere else like sysrtim
and _int_free. The trimming check should be based on the size of
the top chunk and only the size of the top chunk. The following
patch harmonizes the trimming and make it consistent for the main
arena and thread arenas.
In the old code a large padding request might have meant that
trimming was not triggered. Now trimming is considered first based
on the chunk, then the pad is subtracted, and the remainder trimmed.
This is how all the other trimmings operate. I didn't measure the
performance difference of this change because it corrects what I
consider to be a behavioural anomaly. We'll need some profile driven
optimization to make this code better, and even there Ondrej and
others have better ideas on how to speedup malloc.
Tested on x86_64 with no regressions. Already reviewed by Siddhesh
Poyarekar and Mel Gorman here and discussed here:
https://sourceware.org/ml/libc-alpha/2015-05/msg00002.html
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 6 ++++++
malloc/arena.c | 10 ++++++++--
2 files changed, 14 insertions(+), 2 deletions(-)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread