* [PATCH] release all idle heaps in the non-main-arena
@ 2021-06-11 4:08 liudongyun
2021-06-11 6:41 ` Siddhesh Poyarekar
0 siblings, 1 reply; 3+ messages in thread
From: liudongyun @ 2021-06-11 4:08 UTC (permalink / raw)
To: libc-alpha
Cc: libc-maintainers, libc-locales, deng.hongjie, liu.dongyun, dai.xianjun
liu.dongyun@h3c.com report this question in follow:
The system has a total of 32G memory and 12 core cpu.
After frequent malloc and free memory by multiple threads, we found that many
physical memory that should have been released have not been released.We observe that
the maximum free memory can reach 15G.
deng.hongjie@h3c.com recommend reducing the heap size of the non-main-arena, but it
doesn't work. liu.dongyun@h3c.com found When the last heap in the user process into
the non-main-area has memory in use, the previous heap will not be released, even if the
previous heap is already free. This mechanism resulting in the memory cache reaching 15G.
liu.dongyun@h3c.com tried to solve this problem by checking to release the heap that the
current chunk belongs to and unmmaped if it is all free. Then we adjust the heap size of
the non-main-arena to 512KB. The method has achieved good results. Submit patch only to
modify one of them.
The test after modification has the following results.
With business:
--------------------------------------------------------------------------------
Memory statistics are measured in KB:
Slot 0:
Total Used Free Shared Buffers Cached FreeRatio
Mem: 32598612 7501484 25097128 0 1812 2121380 77.0%
--------------------------------------------------------------------------------
A few minutes after deleted business:
Memory statistics are measured in KB:
Slot 0:
Total Used Free Shared Buffers Cached FreeRatio
Mem: 32598612 9763528 22835084 0 32 2118208 70.0%
--------------------------------------------------------------------------------
Bugzilla: https://sourceware.org/bugzilla/show_bug.cgi?id=27976
Reported-by: liudongyun <liu.dongyun@h3c.com> and
Reported-by: denghongjie <deng.hongjie@h3c.com>
---
malloc/arena.c | 30 +++++++++++++++++++++++++++++-
malloc/malloc.c | 2 +-
2 files changed, 30 insertions(+), 2 deletions(-)
mode change 100644 => 100755 malloc/arena.c
mode change 100644 => 100755 malloc/malloc.c
diff --git a/malloc/arena.c b/malloc/arena.c
old mode 100644
new mode 100755
index 7eb110445e..7812997886
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -606,14 +606,42 @@ shrink_heap (heap_info *h, long diff)
} while (0)
static int
-heap_trim (heap_info *heap, size_t pad)
+heap_trim(heap_info *heap, heap_info *free_heap, size_t pad)
{
mstate ar_ptr = heap->ar_ptr;
unsigned long pagesz = GLRO (dl_pagesize);
mchunkptr top_chunk = top (ar_ptr), p;
heap_info *prev_heap;
long new_size, top_size, top_area, extra, prev_size, misalign;
+ heap_info *last_heap;
+ /*release part of the cache memory*/
+ last_heap = heap_for_ptr(top_chunk);
+ if ((NULL != free_heap->prev) && (last_heap != free_heap))
+ {
+ p = chunk_at_offset(free_heap, sizeof(*heap));
+ if (!inuse(p))
+ {
+ if (chunksize(p) + sizeof(*free_heap) + MINSIZE == free_heap->size)
+ {
+ while(last_heap)
+ {
+ if (last_heap->prev == free_heap)
+ {
+ last_heap->prev = free_heap->prev;
+ break;
+ }
+ last_heap = last_heap->prev;
+ }
+ ar_ptr->system_mem -= free_heap->size;
+ arena_mem -= free_heap->size;
+ unlink_chunk(ar_ptr, p);
+ delete_heap(free_heap);
+ return 1;
+ }
+ }
+ }
+
/* Can this heap go away completely? */
while (top_chunk == chunk_at_offset (heap, sizeof (*heap)))
{
diff --git a/malloc/malloc.c b/malloc/malloc.c
old mode 100644
new mode 100755
index 0e2e1747e0..40f0a8e458
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -4665,7 +4665,7 @@ _int_free (mstate av, mchunkptr p, int have_lock)
heap_info *heap = heap_for_ptr(top(av));
assert(heap->ar_ptr == av);
- heap_trim(heap, mp_.top_pad);
+ heap_trim(heap, heap_for_ptr(p), mp_.top_pad);
}
}
--
2.17.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] release all idle heaps in the non-main-arena
2021-06-11 4:08 [PATCH] release all idle heaps in the non-main-arena liudongyun
@ 2021-06-11 6:41 ` Siddhesh Poyarekar
2021-06-18 8:02 ` 答复: " Liudongyun
0 siblings, 1 reply; 3+ messages in thread
From: Siddhesh Poyarekar @ 2021-06-11 6:41 UTC (permalink / raw)
To: liudongyun, libc-alpha
Cc: libc-locales, libc-maintainers, deng.hongjie, dai.xianjun
On 6/11/21 9:38 AM, liudongyun wrote:
> liu.dongyun@h3c.com report this question in follow:
> The system has a total of 32G memory and 12 core cpu.
> After frequent malloc and free memory by multiple threads, we found that many
> physical memory that should have been released have not been released.We observe that
> the maximum free memory can reach 15G.
>
> deng.hongjie@h3c.com recommend reducing the heap size of the non-main-arena, but it
> doesn't work. liu.dongyun@h3c.com found When the last heap in the user process into
> the non-main-area has memory in use, the previous heap will not be released, even if the
> previous heap is already free. This mechanism resulting in the memory cache reaching 15G.
Does that cause any actual performance degradation or any other issues
though? heap_shrink() calls madvise (MADV_DONTNEED) on the free parts
of the heap to tell the kernel that the blocks are not needed, so a
fully freed heap ought to not have any performance impact. The RSS
footprint will depend on when the kernel reclaims those pages, which may
not be immediate since madvise is just a hint.
Alternatively, if you disable memory overcommit by setting
/proc/sys/vm/overcommit_memory to 2, heap_shrink() will drop all
permissions on the heap with mprotect(PROT_NONE) instead of doing
madvise. This has a more immediate effect on RSS usage unlike madvise.
Siddhesh
^ permalink raw reply [flat|nested] 3+ messages in thread
* 答复: [PATCH] release all idle heaps in the non-main-arena
2021-06-11 6:41 ` Siddhesh Poyarekar
@ 2021-06-18 8:02 ` Liudongyun
0 siblings, 0 replies; 3+ messages in thread
From: Liudongyun @ 2021-06-18 8:02 UTC (permalink / raw)
To: Siddhesh Poyarekar, libc-alpha
Cc: libc-locales, libc-maintainers, Denghongjie, Daixianjun
> Does that cause any actual performance degradation or any other issues though? heap_shrink() calls madvise (MADV_DONTNEED) on the free parts of the heap to tell the kernel that the blocks are not needed, so a fully freed heap ought to not have any performance impact. The RSS footprint will depend on when the kernel
> reclaims those pages, which may not be immediate since madvise is just a hint.
> Alternatively, if you disable memory overcommit by setting /proc/sys/vm/overcommit_memory to 2, heap_shrink() will drop all permissions on the heap with mprotect(PROT_NONE) instead of doing madvise. This has a more immediate effect on RSS usage unlike madvise.
We released the heap, these virtual memory will be reused in new_heap function. No potential problems have been found yet. The practice in PATCH will definitely lead to an increase in system calls. Since glibc is used in many scenarios, the patch we submitted did not modify the size of the heap to be cautious.
Compared with the current mechanism in GLIBC,the probability of a large amount of physical memory resident is very high, just holding any piece of memory in TOP TUNK will cause all the memory before TOP TUNK to reside. I think it is worth it.
These days, we coordinate with colleagues to test the performance of the patch.Our amendment is as follows.
1. Set the size of heap to 512KB (the submitted patch is 64M by default);
2. Release the unused heap (consistent with the submitted patch)
In the same test environment(Intel(R) Xeon(R) CPU @ 2.00GHz, 12 processors, 32G RAM), we try to simulate using the maximum specification of business data.
There is little performance degradation. The test results are as follows.
Package(Bytes) Not patched Patched
128 24.88Mpps 24.10Mpps
256 24.60Mpps 24.18Mpps
512 15.51Mpps 15.51Mpps
1024 7.14Mpps 7.48Mpps
1280 5.52Mpps 5.52Mpps
1400 4.89Mpps 4.89Mpps
1518 4.51Mpps 4.51Mpps
Thank you very much for your advice.
1, I have thought about this method before, but I gave up due to it's just a hint.
2, As the problem has been solved, I can't coordinate to the classmate test again for your suggestion. I'll test it myself.
-----邮件原件-----
发件人: Siddhesh Poyarekar [mailto:siddhesh@gotplt.org]
发送时间: 2021年6月11日 14:42
收件人: liudongyun (RD) <liu.dongyun@h3c.com>; libc-alpha@sourceware.org
抄送: libc-locales@sourceware.org; libc-maintainers@gnu.org; denghongjie (RD) <deng.hongjie@h3c.com>; daixianjun (RD) <dai.xianjun@h3c.com>
主题: Re: [PATCH] release all idle heaps in the non-main-arena
On 6/11/21 9:38 AM, liudongyun wrote:
> liu.dongyun@h3c.com report this question in follow:
> The system has a total of 32G memory and 12 core cpu.
> After frequent malloc and free memory by multiple threads, we found
> that many physical memory that should have been released have not been
> released.We observe that the maximum free memory can reach 15G.
>
> deng.hongjie@h3c.com recommend reducing the heap size of the
> non-main-arena, but it doesn't work. liu.dongyun@h3c.com found When
> the last heap in the user process into the non-main-area has memory in
> use, the previous heap will not be released, even if the previous heap is already free. This mechanism resulting in the memory cache reaching 15G.
Does that cause any actual performance degradation or any other issues though? heap_shrink() calls madvise (MADV_DONTNEED) on the free parts of the heap to tell the kernel that the blocks are not needed, so a fully freed heap ought to not have any performance impact. The RSS footprint will depend on when the kernel reclaims those pages, which may not be immediate since madvise is just a hint.
Alternatively, if you disable memory overcommit by setting /proc/sys/vm/overcommit_memory to 2, heap_shrink() will drop all permissions on the heap with mprotect(PROT_NONE) instead of doing madvise. This has a more immediate effect on RSS usage unlike madvise.
Siddhesh
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-06-18 8:03 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-11 4:08 [PATCH] release all idle heaps in the non-main-arena liudongyun
2021-06-11 6:41 ` Siddhesh Poyarekar
2021-06-18 8:02 ` 答复: " Liudongyun
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).