From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by sourceware.org (Postfix) with ESMTPS id 847A13858425 for ; Wed, 8 Mar 2023 14:17:32 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 847A13858425 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3288EBU8005499; Wed, 8 Mar 2023 14:17:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=references : from : to : cc : subject : in-reply-to : date : message-id : content-type : mime-version; s=corp-2022-7-12; bh=mSAd9Ni9MfNBYN1C8SE1DsmXu2pv1jvMKCsEiAg9T+c=; b=dvq5W/DxOSUB1tbc3kNb+xAAdwTwMpp51EcANgHZWItcxEXH2eWaSXiLMpqsKhXPDWft L3IRP3TeLwYRWMO8DOKkpMX70e7Eu7E77je/mDBrrEPD/gwDe+UtxAsl5jAdT9ovz1aX gp6XTFCooblCtxim7BHkeFmStnblh3uLyhMOTYfW6kvFpeRVhFXIBn9KQ2GFVgbnQatQ nU4u/xl89e49Otlz/mhe/eDg/lnglWWA7Etq91BeZ5dKTf7REHbenmuV9ym3+pAaicwz 9ld+cRKo3UrN6uu5STwL8+VF5Gbm2Ivz28ZM9CPUAgh2BluYf7nL6bZxRPsZG2AfQk67 9w== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3p4168r6bj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 08 Mar 2023 14:17:31 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 328DZNNf007176; Wed, 8 Mar 2023 14:17:30 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2044.outbound.protection.outlook.com [104.47.56.44]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3p6g4fs3wg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 08 Mar 2023 14:17:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kkdft1YxBo89Y25yG7qyli6Tfc02zPhQx9kK1ABYF95t9C5/XhzeZB+nmUYfKGRoeWHy6s21M4TJ/zNkIjJKG9B1SQD/i2iQIE/L9goBiqiCWvBmAIb+CpNcAFa3ypWS2kl2ipXNkCx9aTzJvZ9bIeICTUVa4ExsS3E7mvhUr923iy2rWKPVIFODsDdcsTBgITmDJVZjIxZCpr2oBzL678KE4rIEU8YSFC0Cr2qWYQAdSuphctNxSrq2ONptpUE6chJQvUhJRou5hUbSNdLa0Y7X3bEqDykIG32PyLpcNBWUeJE5h8IRC2UfgV0gQ2bbg+gdATaBby6AhsBfslxL7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mSAd9Ni9MfNBYN1C8SE1DsmXu2pv1jvMKCsEiAg9T+c=; b=QZyBUMEB067MkwgesvuRrk8W3pSeBGslIJVqPf+RepbZWxgK/1wv+W1RMehSXNhIN4+/gilemPbg5NJRK0/Iy/VDoo38Y8F+Kph+pJZQnI7iye/6XksIqDb3+v9BhBO6ocikpgJb9eB+0q6aDDK4mA9nVGoZKPjkou65y/MRcDzQ+MyGOm1RbIhvZPovY4oUqQz6vCT/yEduDCyeBQW7y+V4uiD3e3ODGqaKfRC25psBK8MG3zeBs/TBxJ6fkDBfrZJsD+QTnPfMuUO8VTbha/k59Gl8qtBvxznPMdBCooi3drUDkRcgEu9W2lAkia/vG7td0Vn/6bKaV/xvehx1Jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mSAd9Ni9MfNBYN1C8SE1DsmXu2pv1jvMKCsEiAg9T+c=; b=GDDdCYykROXXSxW2L7GAshLlhwu8Tbama75IqvHRLH4Q20ZTjBHh4MeVAcq4pb4DwbRLnilyvtrEttMo3MCY2YY3VhlKbqj/FAfqWIDVCwO+i+w7q4IT2DhHNlA4djfSgNYbrYAUSen7O6caCBxq63ir0NWx8/O9LmNeEH95Sx8= Received: from BN6PR1001MB2340.namprd10.prod.outlook.com (2603:10b6:405:30::36) by PH0PR10MB4567.namprd10.prod.outlook.com (2603:10b6:510:33::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.17; Wed, 8 Mar 2023 14:17:29 +0000 Received: from BN6PR1001MB2340.namprd10.prod.outlook.com ([fe80::a502:c948:c3f6:9728]) by BN6PR1001MB2340.namprd10.prod.outlook.com ([fe80::a502:c948:c3f6:9728%6]) with mapi id 15.20.6156.023; Wed, 8 Mar 2023 14:17:28 +0000 References: <87pm9j4azf.fsf@oracle.com> User-agent: mu4e 1.4.15; emacs 28.1 From: Cupertino Miranda To: libc-alpha@sourceware.org Cc: "Jose E. Marchesi" , Elena Zannoni , Cupertino Miranda Subject: [RFC] Stack allocation, hugepages and RSS implications In-reply-to: <87pm9j4azf.fsf@oracle.com> Date: Wed, 08 Mar 2023 14:17:23 +0000 Message-ID: <87mt4n49ak.fsf@oracle.com> Content-Type: text/plain X-ClientProxiedBy: AS4P192CA0012.EURP192.PROD.OUTLOOK.COM (2603:10a6:20b:5da::20) To BN6PR1001MB2340.namprd10.prod.outlook.com (2603:10b6:405:30::36) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR1001MB2340:EE_|PH0PR10MB4567:EE_ X-MS-Office365-Filtering-Correlation-Id: 74a54794-c44e-439d-af1b-08db1fdfd949 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cw0AwZRLWivv+OvWUGtJPDZiYhO4B3h6EjFwSBww4BETzgCHYPgh1enYXuFpKWR4mNH+Mjjk9lqDJ3rM8s/49xARAxAUZLm04tkikW1j4P2y00iJplGnnCwdITxo3ODnJoRQsRWRy4ztbSxmC7u8lzN7gcny3pBzB+sOUvjnfPdNk+dVXdm6VLS+KJaI+YN2Di2/iPK0Lcg9htZMNaHQUxlZPWS0hJ/XovpuCk2mvn/WzHBlThQAQjBx6MWV1HeU99ZHrl+ei5yO3EVrlF0PYsLu1ylt1PeNozuao+ids1wuJWHmfeJrA8JfXcDN2FZRFSFRNO/IZ8qCKSdKYGFu8zjdVVv63HBaSSKTQfQy1E8bf59AUt+ODtUAsiRiBewpsH1e/bogN0o84ez9CQi7uq8nxm9E4MZdL4IdWcnonYNLiblK4CsABS3r1HPITYqHN1WXdFoL85LZGP1SyGwjd3+U7tpkYmWcZnAEhFKgwyAYNbCl70nYsbASh8y6nnzz4OJf1gWpEIP3SOg0zzvWpu7HPOUKQd2Mh6SUNuq9v19iHeI/AuSGLkmnpiYj4J50tMHq2TIs3ZVQlSvQEmLGGTY3y+CjYvJfmmeLZ290TntE0r/7uk1HU0PJUhZ50Qsh9GpQAgLT+QXDb5tNqhCKjg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BN6PR1001MB2340.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(346002)(39860400002)(396003)(366004)(136003)(376002)(451199018)(8936002)(6666004)(5660300002)(66476007)(4326008)(6916009)(8676002)(86362001)(66946007)(66556008)(36756003)(83380400001)(54906003)(478600001)(316002)(41300700001)(6486002)(186003)(2906002)(38100700002)(44832011)(6512007)(6506007)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?S4C4d2iY3zYimnNDNeOBeOCjuICnEX5OJULp0HxITKVUXDFEsuVCCAtAnqwK?= =?us-ascii?Q?bcGQdbf6KnTl7ZhcRm38xDEsPkhznY4by6qtVgNfhhLApGFitkBcfJTXkQ/L?= =?us-ascii?Q?s72P4RrV0tGgXb+z+r0q6xdtRimhYybIUQwceOUfeIVLHvP2QpgULbkYKE3p?= =?us-ascii?Q?yo05FJgNO7s8iGiXuHQUjLxwcmfHWG7yS/p5ZaXM3CmXjJcvVChV/oa/apDN?= =?us-ascii?Q?8iIoWp2v57UvkjJ2EMOz9HYie66J1Z+qI9uVr/j3dED8oJ6o5Imi3buJgdIv?= =?us-ascii?Q?UH4+JXBFCM3R9E4W98Q1LwyakCciAEFWpwHw1l5wK2VuaVNcoaN1m6RUQx5m?= =?us-ascii?Q?v8IYfIJAVUZHpcWncu05rDfRlbGqcSQjdOjEpK6OhfBXU3cwE+e8f2O9A2qh?= =?us-ascii?Q?ceMwk/YG31n5BxFD0ff1UhvhLk4zl6PY6WGRJdhIICMeUKMEGa93FZEnWUT5?= =?us-ascii?Q?Qjmh7n9LNnVnyQnpY16XnH6c63CwiFC+r7DQBjE5yKGkhm69AuHS9p2WRlqI?= =?us-ascii?Q?2HEUhsItBGBUk1+azSF/13Dm+FqHTwxrR9oKlWJAGBGbhLIaF1Uf9Be3VohI?= =?us-ascii?Q?6SX2noQU4zvFHzeI+khp/JklTLvgLhqoqzMgqdXsP9lGF3Rgtql9clFnujwh?= =?us-ascii?Q?MKHuzi/TH67zOVQ3YQmJ6JQsAB8I7YXmfQ8GFvdUz0u5vmGiA03iZsLc5ze0?= =?us-ascii?Q?SGK5BuMKr2ixCOHQK0RahBjQbt/N4p4MSuzCwOsZ5JvZIihfqb+jqO8fkhmg?= =?us-ascii?Q?hgBWI4qifzL4+usjcLbV15mm55hCG9iz/7NYdTg0yOhO81tFKI6ZOnCJt1q6?= =?us-ascii?Q?+v9xglKK1NYf59/CL2NgzdUJZcwX7w+kZW4fEmKvR42dTA5Tvp8pJuoqeWGD?= =?us-ascii?Q?EG0wIvm4fcPfdLHP4r6e+bLdNlpG/od64pR037ZZsqR7Lrd8yNWMHiBUuHPH?= =?us-ascii?Q?rvxey5r2DOH/sJf1XIlyAwE3qVwcQW2NdbXdcioxA920hB84N1K+1gkkDKPf?= =?us-ascii?Q?5qqfKGGCp+wKSRJOptIVL/uTLZJXsjczX8VlePOZtSjSaIJS8Qwi5J5O7SML?= =?us-ascii?Q?G9Glhm3by55SX1c9WtwwYgg33/Ph/85WgnTs1PSTM8poRJnZojfx0EFJEvnI?= =?us-ascii?Q?duk56xw5QmMiOOog4Er1yJ2sPmVEv7m2dKfpHgebcmpuB/pwJo3yYhcKvaR6?= =?us-ascii?Q?c5tsTq+975tBh6Jq780/8L6uf9jAL6A6R8LmsOjz8zZ2+kS87CUYUJwG03Lq?= =?us-ascii?Q?nVxjFPBu0i8j8GEkPsaEfSkHKRwNdvDq9b4LxBnTkTky1WXHInCa8Np4LSOr?= =?us-ascii?Q?AHEMz669lG4/573IZQU6800uK8Ss+4wLc24flAuoDGLh7dx0/YWCf4t4yM/4?= =?us-ascii?Q?7Fq7dg7iTvjg+bEZ49AejYthrAaTAuDt0Kv+Hd4wkO15NygXuh4d2ASQiAtG?= =?us-ascii?Q?iXSHVSZsGoJwZabaAs3Sypmap+1UKpij2EQ1nKKk+7Vx+TNFTi7cmA7nYSEQ?= =?us-ascii?Q?eG8t20kBDLT3oAyJ+c6rLipOyqIo0omQSZNSg3CFGln7/t9O9QvLAEl+A0XL?= =?us-ascii?Q?fWcODM+1aIANxQ64XBXeI5uixkU3OiPmwukUYRdO1YivqQcUDTwvCM1Fcl9n?= =?us-ascii?Q?52SAYxzDfEmJgfWn+dWFZBEjGw3Xs12o6oKJL34NarGe?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: nQjZq6dTbhhHqS9hPYFRHJLR1lR8I2XYt/sbSy7hpxYxAqCujPphQpICpxAq9o9nnt3K/n0JKuiJchzVUTAL2XeeIGkUwgWTX2f1EM5zvctmdY7qPwD5qeM/Jbl2V5XqBOl2x7BONnDv4o4+21f/PkapaeJCSNBiuB7Ql/mz+e+Yge9LU1g75V6NJg4pDgw5lDi4lFxHuLpcRCdmp6QbzL9iftwxHT4Prjy4KiENp2u8UzFacayWeirlsP3bYzayhIoKsdjv3JT0z9fvHJU63nkpQ+eTh5CEXCQYumtzv0RYZ11sJISwKkxhVKZTyOwnjuk5H37KfPIflgoqBr7D9pijlO3VUI2ZDrq6Xsi+Z99eg5JwcbnxwLy4Hv6wVk1skNdb6t+M4HmNUA3JdCUf/7OlVwgHhMnrMdDXt3PPdxiTD5V+bDKDDoJRMZt2kIOnHw/vmbQ6ubly5+gN1HlCNkEcSBBeuXM70qy63gPyezUpTQjVRtVpaZIsruLIX/gndViFDk17AHZv3/QbY/yO3HnivRLhio6kXxQQAczjsIsacQPE9scpG7CCtovp92JluLBscd5gB4ZbOy5b1pMzMcMVfFT36Qt53pUSrPdyFD6Jwy2ONExtAkfOBt+YgjD6GoFSAaKKRqtP2vubkN/ZI4NEEWa6Z07ezBGhE3EWRFH0NGA0QUu6AND7NeyJQvdHfiluMgJCKJmghb3VgSLvvVqdKws9vM9FP5KGHbDh+MDx9Dqh+GwTdqbxLnvt5MDdC5yRnxjJpzu8c/TZfzSN09PCrMDMrRg5ORungN51o9M= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 74a54794-c44e-439d-af1b-08db1fdfd949 X-MS-Exchange-CrossTenant-AuthSource: BN6PR1001MB2340.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2023 14:17:28.8584 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kodzzDobUDBpZV2D6bMbca4eQS/eYtAJ9YIKQWcBk2U2Dut/kziYqcoZkP5gwsRMa7KAbGbuGMQPXuTzBLJluE9///dwG5SQNvYcv5klP1A= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4567 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-08_08,2023-03-08_03,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 mlxscore=0 mlxlogscore=768 adultscore=0 bulkscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2303080122 X-Proofpoint-GUID: fcV1ASlm3nhhYYkpj4b6vfPJOINd1GEl X-Proofpoint-ORIG-GUID: fcV1ASlm3nhhYYkpj4b6vfPJOINd1GEl X-Spam-Status: No, score=-6.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi everyone, For performance purposes, one of ours in-house applications requires to enable TRANSPARENT_HUGEPAGES_ALWAYS option in linux kernel, actually making the kernel to force all of the big enough and alligned memory allocations to reside in hugepages. I believe the reason behind this decision is to have more control on data location. For stack allocation, it seems that hugepages make resident set size (RSS) increase significantly, and without any apparent benefit, as the huge page will be split in small pages even before leaving glibc stack allocation code. As an example, this is what happens in case of a pthread_create with 2MB stack size: 1. mmap request for the 2MB allocation with PROT_NONE; a huge page is "registered" by the kernel 2. the thread descriptor is writen in the end of the stack. this will trigger a page exception in the kernel which will make the actual memory allocation of the 2MB. 3. an mprotect changes protection on the guard (one of the small pages of the allocated space): at this point the kernel needs to break the 2MB page into many small pages in order to change the protection on that memory region. This will eliminate any benefit of having small pages for stack allocation, but also makes RSS to be increaded by 2MB even though nothing was written to most of the small pages. As an exercise I added __madvise(..., MADV_NOHUGEPAGE) right after the __mmap in nptl/allocatestack.c. As expected, RSS was significantly reduced for the application. At this point I am very much confident that there is a real benefit in our particular use case to enforce stacks not ever to use hugepages. This RFC is to understand if I have missed some option in glibc that would allow to better control stack allocation. If not, I am tempted to propose/submit a change, in the form of a tunable, to enforce NOHUGEPAGES for stacks. In any case, I wonder if there is an actual use case where an hugepage would survive glibc stack allocation and will bring an actual benefit. Looking forward for your comments. Best regards, Cupertino