From: Florian Weimer <fweimer@redhat.com>
To: Cupertino Miranda via Libc-alpha <libc-alpha@sourceware.org>
Cc: Cupertino Miranda <cupertino.miranda@oracle.com>,
"Jose E. Marchesi" <jose.marchesi@oracle.com>,
Elena Zannoni <elena.zannoni@oracle.com>,
Cupertino Miranda <cupertinomiranda@gmail.com>
Subject: Re: [RFC] Stack allocation, hugepages and RSS implications
Date: Thu, 09 Mar 2023 11:54:42 +0100 [thread overview]
Message-ID: <87bkl2b3f1.fsf@oldenburg.str.redhat.com> (raw)
In-Reply-To: <87mt4n49ak.fsf@oracle.com> (Cupertino Miranda via Libc-alpha's message of "Wed, 08 Mar 2023 14:17:23 +0000")
* Cupertino Miranda via Libc-alpha:
> Hi everyone,
>
> For performance purposes, one of ours in-house applications requires to enable
> TRANSPARENT_HUGEPAGES_ALWAYS option in linux kernel, actually making the
> kernel to force all of the big enough and alligned memory allocations to
> reside in hugepages. I believe the reason behind this decision is to
> have more control on data location.
>
> For stack allocation, it seems that hugepages make resident set size
> (RSS) increase significantly, and without any apparent benefit, as the
> huge page will be split in small pages even before leaving glibc stack
> allocation code.
>
> As an example, this is what happens in case of a pthread_create with 2MB
> stack size:
> 1. mmap request for the 2MB allocation with PROT_NONE;
> a huge page is "registered" by the kernel
> 2. the thread descriptor is writen in the end of the stack.
> this will trigger a page exception in the kernel which will make the actual
> memory allocation of the 2MB.
> 3. an mprotect changes protection on the guard (one of the small pages of the
> allocated space):
> at this point the kernel needs to break the 2MB page into many small pages
> in order to change the protection on that memory region.
> This will eliminate any benefit of having small pages for stack allocation,
> but also makes RSS to be increaded by 2MB even though nothing was
> written to most of the small pages.
>
> As an exercise I added __madvise(..., MADV_NOHUGEPAGE) right after the
> __mmap in nptl/allocatestack.c. As expected, RSS was significantly
> reduced for the application.
Interesting. I did not expect to get hugepages right out of mmap. I
would have expected subsequent coalescing by khugepaged, taking actual
stack usage into account. But over-allocating memory might be
beneficial, see below.
(Something must be happening between step 1 & 2 to make the writes
possible.)
> In any case, I wonder if there is an actual use case where an hugepage would
> survive glibc stack allocation and will bring an actual benefit.
It can reduce TLB misses. The first-level TLB might only have 64
entries for 4K pages, for example. If the working set on the stack
(including the TCB) needs more than a couple of pages, it might
beneficial to use a 2M page and use just one TLB entry.
In your case, if your stacks are quite small, maybe you can just
allocate slightly less than 2 MiB?
The other question is whether the reported RSS is real, or if the kernel
will recover zero stack pages on memory pressure.
Thanks,
Florian
next prev parent reply other threads:[~2023-03-09 10:54 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <87pm9j4azf.fsf@oracle.com>
2023-03-08 14:17 ` Cupertino Miranda
2023-03-08 14:53 ` Cristian Rodríguez
2023-03-08 15:12 ` Cupertino Miranda
2023-03-08 17:19 ` Adhemerval Zanella Netto
2023-03-09 9:38 ` Cupertino Miranda
2023-03-09 17:11 ` Adhemerval Zanella Netto
2023-03-09 18:11 ` Cupertino Miranda
2023-03-09 18:15 ` Adhemerval Zanella Netto
2023-03-09 19:01 ` Cupertino Miranda
2023-03-09 19:11 ` Adhemerval Zanella Netto
2023-03-09 10:54 ` Florian Weimer [this message]
2023-03-09 14:29 ` Cupertino Miranda
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87bkl2b3f1.fsf@oldenburg.str.redhat.com \
--to=fweimer@redhat.com \
--cc=cupertino.miranda@oracle.com \
--cc=cupertinomiranda@gmail.com \
--cc=elena.zannoni@oracle.com \
--cc=jose.marchesi@oracle.com \
--cc=libc-alpha@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).