From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) by sourceware.org (Postfix) with ESMTPS id 3B8853858D39 for ; Tue, 28 Mar 2023 13:57:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3B8853858D39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org Received: by mail-ot1-x330.google.com with SMTP id r40-20020a05683044a800b006a14270bc7eso3061336otv.6 for ; Tue, 28 Mar 2023 06:57:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1680011875; h=content-transfer-encoding:in-reply-to:organization:from:references :to:content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=e89KUdcZgLvrKDUEj7SL/GS6oWq5Pkt6Sw/XNXHzbK0=; b=ZprJAEpc/ChE3f+OMQkOWJ2Bi5MgOw3c39+BaKAL1YwutAJj0xR5qv5wxOnccSKt0R gpGz4GU55S+KJ9g2vXNNR1F7MMHsyrWoePTmeHOHYNl3pLoMnRLngILvbZIckQIHCDUs Fecn2U43pzMS3cFZLzcyS8+p92F96VqEuXy1i7v+ktxlWeoUxBVpl0YG2XaBAZQU0WuF Xj2xrsJH83Qd97p2vMNwopnUDVNwb0DWoJ/bSsH60ddEqowOsEg/9y0/eo4qn8VBTVFg URjd9QeD2Do17ZXLlNJTHgxOSj3AbbgPdFGAusQZvZH4biAMmmg/PibvIF2kVtIfKpBf oDhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680011875; h=content-transfer-encoding:in-reply-to:organization:from:references :to:content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e89KUdcZgLvrKDUEj7SL/GS6oWq5Pkt6Sw/XNXHzbK0=; b=ltyOreDNANiDMN/QZOwoNWzVMRbH+gpDQtORWHNtUzBlMMMgK4ami3vhwK0gVid1Ns sFxBC5HAwAtnw3YrZrq3IPVj9vrPq6vvw8cBoW0CTzDKkVuYeNZcgJv0TfQHvKqmcKvS tc5DL7nBu+jIstA7G//f+9lBcW2Ra6lkJueBEMYS05GcRxqZlM89zJFHTYym0kH02TX1 i8waHbvoWJHFaoEaAVo7hdH0liQsdFl+Y4n+OyKhL1tjMyKijpAHTeMe2zj/XHR/s0da us9y8Ckmlld24XAlSdJtsUYLMv7vCuCYJOrQF4g3muRDVQCQ5C5T9qshcOdWCG83XMci 2XMg== X-Gm-Message-State: AO0yUKWvY6T50eq16zBf4TuOVOPstr2lzgv7Bt5iaUE6pD2hjRd5liXs hfufH+KPWqMi3LU8yPWMjGi64sNC7kQHJYh8N4ErfA== X-Google-Smtp-Source: AK7set/lD001saIWsrTKSjvPFftddjJ2KTkprX/oL3a37eqtfBPwE/Wav+hqbp4ddM2hxP6nFEK1vg== X-Received: by 2002:a05:6830:630a:b0:697:eead:39d2 with SMTP id cg10-20020a056830630a00b00697eead39d2mr13048453otb.16.1680011874815; Tue, 28 Mar 2023 06:57:54 -0700 (PDT) Received: from ?IPV6:2804:1b3:a7c1:60f9:1426:1d2d:d6b:1761? ([2804:1b3:a7c1:60f9:1426:1d2d:d6b:1761]) by smtp.gmail.com with ESMTPSA id t16-20020a0568301e3000b0069f9209c84dsm7937050otr.60.2023.03.28.06.57.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Mar 2023 06:57:54 -0700 (PDT) Message-ID: <0324d4f3-6e0e-5cc3-90a3-ca042d42f000@linaro.org> Date: Tue, 28 Mar 2023 10:57:52 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: [PATCH v4 1/1] Created tunable to force small pages on stack allocation. Content-Language: en-US To: libc-alpha@sourceware.org, Cupertino Miranda References: <20230328085538.48080-1-cupertino.miranda@oracle.com> <20230328085538.48080-2-cupertino.miranda@oracle.com> From: Adhemerval Zanella Netto Organization: Linaro In-Reply-To: <20230328085538.48080-2-cupertino.miranda@oracle.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 28/03/23 05:55, Cupertino Miranda via Libc-alpha wrote: > Created tunable glibc.pthread.stack_hugetlb to control when hugepages > can be used for stack allocation. > In case THP are enabled and glibc.pthread.stack_hugetlb is set to > 0, glibc will madvise the kernel not to use allow hugepages for stack > allocations. > > Changed from v1: > - removed the __malloc_thp_mode calls to check if hugetlb is > enabled. > > Changed from v2: > - Added entry in manual/tunables.texi > - Fixed tunable default to description > - Code style corrections. > > Changes from v3: > - Improve tunables.texi. > --- > manual/tunables.texi | 14 ++++++++++++++ > nptl/allocatestack.c | 6 ++++++ > nptl/nptl-stack.c | 1 + > nptl/nptl-stack.h | 3 +++ > nptl/pthread_mutex_conf.c | 8 ++++++++ > sysdeps/nptl/dl-tunables.list | 6 ++++++ > 6 files changed, 38 insertions(+) > > diff --git a/manual/tunables.texi b/manual/tunables.texi > index 70dd2264c5..182f644851 100644 > --- a/manual/tunables.texi > +++ b/manual/tunables.texi > @@ -459,6 +459,20 @@ registration on behalf of the application. > Restartable sequences are a Linux-specific extension. > @end deftp > > +@deftp Tunable glibc.pthread.stack_hugetlb > +This tunable allows to configure pthread creation stack allocation never to use > +hugetlbs. > +This is more of an RSS optimization, for example in scenarios where many > +threads get created, keeping RSS to a minimum, but also allowing hugestlbs to > +be used for malloc allocations. > +This tunable only affects stacks allocated through thread creation process, > +i.e. it has no effect on stacks assigned with pthread_attr_setstack. > + > +The default is @samp{1} and preservs glibc default behaviour. > +When set to @samp{0}, it advises the kernel not to use hugetlbs for stack > +allocation. > +@end deftp > + We use 'Huge Pages' on glibc.malloc.hugetlb (hugestlbs sounds strange), RSS is not defined in any place (even it is a common Linux VM term), pthread_attr_setstack needs to be within a @code. What about the following: This tunable controls whether to use Huge Pages in the stacks created by @code{pthread_create}. This tunable only affects the stacks created by @theglibc{}, it has not effect on stack assigned with @code{pthread_attr_setstack}. The default is @samp{1} where the system default value is used. Setting its value to @code{0} enables the use of @code{madvise} with @code{MADV_NOHUGEPAGE} after stack creation with @code{mmap}. This is a memory utilization optimization, since internal glibc setup of either the thread descriptor and the guard page might force the kernel to move the thread stack originally backup by Huge Pages to default pages. (maybe expand a bit the problem this tunable aims to improve) > @node Hardware Capability Tunables > @section Hardware Capability Tunables > @cindex hardware capability tunables > diff --git a/nptl/allocatestack.c b/nptl/allocatestack.c > index c7adbccd6f..f9d8cdfd08 100644 > --- a/nptl/allocatestack.c > +++ b/nptl/allocatestack.c > @@ -369,6 +369,12 @@ allocate_stack (const struct pthread_attr *attr, struct pthread **pdp, > if (__glibc_unlikely (mem == MAP_FAILED)) > return errno; > > + /* Do madvise in case the tunable glibc.pthread.stack_hugetlb is > + set to 0, disabling hugetlb. */ > + if (__glibc_unlikely (__nptl_stack_hugetlb == 0) > + && __madvise (mem, size, MADV_NOHUGEPAGE) != 0) > + return errno; > + > /* SIZE is guaranteed to be greater than zero. > So we can never get a null pointer back from mmap. */ > assert (mem != NULL); > diff --git a/nptl/nptl-stack.c b/nptl/nptl-stack.c > index 5eb7773575..e829711cb5 100644 > --- a/nptl/nptl-stack.c > +++ b/nptl/nptl-stack.c > @@ -21,6 +21,7 @@ > #include > > size_t __nptl_stack_cache_maxsize = 40 * 1024 * 1024; > +int32_t __nptl_stack_hugetlb = 1; > > void > __nptl_stack_list_del (list_t *elem) > diff --git a/nptl/nptl-stack.h b/nptl/nptl-stack.h > index 34f8bbb15e..cf90b27c2b 100644 > --- a/nptl/nptl-stack.h > +++ b/nptl/nptl-stack.h > @@ -27,6 +27,9 @@ > /* Maximum size of the cache, in bytes. 40 MiB by default. */ > extern size_t __nptl_stack_cache_maxsize attribute_hidden; > > +/* Should allow stacks to use hugetlb. (1) is default. */ > +extern int32_t __nptl_stack_hugetlb; > + > /* Check whether the stack is still used or not. */ > static inline bool > __nptl_stack_in_use (struct pthread *pd) > diff --git a/nptl/pthread_mutex_conf.c b/nptl/pthread_mutex_conf.c > index 329c4cbb8f..60ef9095aa 100644 > --- a/nptl/pthread_mutex_conf.c > +++ b/nptl/pthread_mutex_conf.c > @@ -45,6 +45,12 @@ TUNABLE_CALLBACK (set_stack_cache_size) (tunable_val_t *valp) > __nptl_stack_cache_maxsize = valp->numval; > } > > +static void > +TUNABLE_CALLBACK (set_stack_hugetlb) (tunable_val_t *valp) > +{ > + __nptl_stack_hugetlb = (int32_t) valp->numval; > +} > + > void > __pthread_tunables_init (void) > { > @@ -52,5 +58,7 @@ __pthread_tunables_init (void) > TUNABLE_CALLBACK (set_mutex_spin_count)); > TUNABLE_GET (stack_cache_size, size_t, > TUNABLE_CALLBACK (set_stack_cache_size)); > + TUNABLE_GET (stack_hugetlb, int32_t, > + TUNABLE_CALLBACK (set_stack_hugetlb)); > } > #endif Different than glibc.malloc.hugetlb, we always enable it even if the system does not support THP. I think it should be ok (although not in sync with glibc.malloc.hugetlb support). > diff --git a/sysdeps/nptl/dl-tunables.list b/sysdeps/nptl/dl-tunables.list > index bd1ddb121d..4cde9500b6 100644 > --- a/sysdeps/nptl/dl-tunables.list > +++ b/sysdeps/nptl/dl-tunables.list > @@ -33,5 +33,11 @@ glibc { > maxval: 1 > default: 1 > } > + stack_hugetlb { > + type: INT_32 > + minval: 0 > + maxval: 1 > + default: 1 > + } > } > }