Re: [PATCH v5 1/1] Created tunable to force small pages on stack allocation.

[Cupertino Miranda <cupertino.miranda@oracle.com>]

> Hi Cupertino,
>
>> To contextualize the tunable was proposed to reduce stack memory usage while
>> still having THP enabled in the kernel for all other allocation types.
>> We identified a scenario which THP will be initially used for stack
>> allocation, but once protection gets set for the guard region, it will
>> split the huge page into 4k pages, when the 2M page is already marked
>> dirty, not really making use of THP benefits, but bloating RSS as all
>> the small pages are assumed as touch.
>
> So after the guard region is reserved, we lose 2MB of real memory that will
> never be used (as only the top of the stack is used), right?

Right

> The next question is whether the splitting of one huge page causes the whole
> stack mmap to use small pages too. If so then it never makes sense to use huge
> pages for the stack if the guard page is smaller than a huge page (since you lose
> 2MB of memory and never use huge pages anyway).

If one wants to get guardsize of 2MB they can specify that in
pthread_create attributes. That is already controllable through pthread
api.

My tunable is really an optimization to reduce RSS, i.e. always avoid
THP to be used for stack allocation. Even if you want to allocate big
stack spaces, those would be using small pages to reduce RSS.

Adhemerval suggestion is more of an heuristic to detect when, although
you prefer THP to be used, the configuration of the stack would not
allow to assign a huge page anyway, and instead of having the kernel bloat
the RSS by 2MB at allocation time, one would advise the kernel not to
attempt THP here.
When glibc changes the protection mode for a small guard space that
fits in the same huge page as the rest of the stack, although the huge
page is split into 4k pages, the RSS would be kept at 2MB, since the
huge page was already written on and is marked as dirty.
This is what is attempted to be avoided in Adhemerval patch by
madvise-ing in advance.

> Alternatively, if the rest of the stack can still use huge pages, we actually lose 4MB
> of RSS (since the top of the stack is a large page and is obviously used). As I
> explained, huge pages for stack can still make sense for applications that need
> a large amount of stack. For these cases you could just increase the guard size
> to avoid wasting the extra 2MB of memory.
>
>> I think you are suggesting the oposite, i.e. it would increase RSS.
>
> Where did I suggest increasing RSS?
Wouldn't increasing the guard size to 2MB not increase RSS by 2MB ?

>
>> Also Adhemerval patch intention is not to detect when the THP for stack
>> allocation makes sense, but rather the oposite. It is to detect when THP
>> will not bring any benefit.
>
> It doesn't matter which way around you do it - it is absolutely trivial either way.

Regarding condition changes I made in Adhemerval patch, it is likely
that it is not complete. In any case, the approach is more about
detecting when THP are not going to be possible, rather then changing
allocation parameters to make it possible.

Cheers,
Cupertino