From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by sourceware.org (Postfix) with ESMTPS id 1BC19385840B for ; Mon, 29 Nov 2021 17:07:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 1BC19385840B Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1ATGrBOB024879; Mon, 29 Nov 2021 17:07:02 GMT Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3cmueeatk7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 29 Nov 2021 17:07:02 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1ATGti87187627; Mon, 29 Nov 2021 17:07:01 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2170.outbound.protection.outlook.com [104.47.59.170]) by userp3030.oracle.com with ESMTP id 3ck9swqecn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 29 Nov 2021 17:07:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kQvhKWJY2cK11y8dCryRYZ9Mq3oKPG4nqGFjqp3Oe8JTvC54oZiMOcGIg03uf+wQyo6HBzxO26uYa2LPfBlvE/AbQ3C9B3sU9x7mdNW+VRjWLep9TmaZLhrHEXXrnUG4AFvtVc637b+UVMYBwyUkHTIfsbMpLK4rgN9iJORHeNtz3hyqzOA3hvRmfWbFqxE2OlU8c5wmmACS5UVXEpkgFLcppx92FJzKIoUwA8/F/bTeJ3r5kw9u1IlK5zXLUP8w4nlXBQ+bm0DWZg528XBhziPCSgmjCp7WA4V142fErIIluRRX3r9VPUDFSL5QTEsaP+r6uC8Ipuid4mTzqM/JEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=koypU/zSqNt3mGGmijhxk1yOgBKtZwhFNju1a3j/c8g=; b=N+euFPO9Kk/qHcAtNjDlGg2lVwFjF3z1jDK85/14Ts+xGcqD7MfInQdqN64dkXHXv6sOlo5yIxTKXuAYBPQEm69LlHYQxwvsm4Y/lFHSucCIv5kjpumrqUSquhZCA1FfElQ8DS2d+foI9c8AByDUekP7FpiUNKPP/d8frhWxyXSaJyUgy/UvNr/2s7bE+7Li/Un5v4oz0A2vnNxuITtz0jM3rDqzsQ61bvUOGI38ve5of81aDvqMs0YvKJFIl0j0cpeFTVejpwldJCioVRbrDKoZz8misIXQ5dK4EiTiMJQOm4fCkrIqIijF/Fe/9NCpFwQDPexahuWg10BhKFFJiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none Received: from BYAPR10MB3208.namprd10.prod.outlook.com (2603:10b6:a03:159::10) by BYAPR10MB2904.namprd10.prod.outlook.com (2603:10b6:a03:86::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Mon, 29 Nov 2021 17:06:58 +0000 Received: from BYAPR10MB3208.namprd10.prod.outlook.com ([fe80::cdd5:fa7c:b36d:dfd2]) by BYAPR10MB3208.namprd10.prod.outlook.com ([fe80::cdd5:fa7c:b36d:dfd2%2]) with mapi id 15.20.4734.024; Mon, 29 Nov 2021 17:06:58 +0000 Message-ID: <98938444-917a-5027-4d43-061e958b21a3@oracle.com> Date: Mon, 29 Nov 2021 11:06:54 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.3.1 Subject: Re: Excessive memory consumption when using malloc() Content-Language: en-US To: Christian Hoff , Adhemerval Zanella , Carlos O'Donell , Konstantin Kharlamov , libc-help@sourceware.org References: <560ed6888a62b21362cda5385655c3a84fd354b9.camel@yandex.ru> <56522c8f847ddd27fdffedecb516f778837f9e92.camel@yandex.ru> <3f88fb7c-672a-da09-89d7-240e9991369d@gmx.net> From: Patrick McGehearty In-Reply-To: <3f88fb7c-672a-da09-89d7-240e9991369d@gmx.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: SN6PR01CA0019.prod.exchangelabs.com (2603:10b6:805:b6::32) To BYAPR10MB3208.namprd10.prod.outlook.com (2603:10b6:a03:159::10) MIME-Version: 1.0 Received: from [10.154.190.234] (138.3.200.42) by SN6PR01CA0019.prod.exchangelabs.com (2603:10b6:805:b6::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23 via Frontend Transport; Mon, 29 Nov 2021 17:06:57 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: eb294845-716f-438a-c5c2-08d9b35aa75a X-MS-TrafficTypeDiagnostic: BYAPR10MB2904: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: v6t8vrcaqztan7J40qDfQr9FeuM1rljLLVzCjYtr50dPRLQIg6Nq+4sKx2hu2xVq09e4PociMZpyYa/x08g3hh2ItqR4KmcWdURdRCiqM7fFMdFeuKqjCcis1W2RroBbXK5gH8qGOalbfta7UAgS7N8JVofz6JYY29N5xFSFcKqSuWkSr5CjvUildwBIj31DjVAPNtHqTgihy/mF9MQRVIyOkbtnJizQSfnT+NnEYzPI75Nbw9EP9kIiwPSkPl9bybYxXnTEeD+b6FrSkvy2P6OWAGd1k7jWtewcq8cG5jVcvgClNbHzWIE05EL9Sg/eItwpy890sUUlHOvsBJxn9Hk2uoqmlZrUF+eaLmPX5C85yy8PsJmtFXJuGw7cq1TSlQJD2GIsQBUNThrneNHpNlv5ShOgv2ytx/9VKPyIfP7+rdddbQJL+RRaUybuvPdjZfwRqU6ilYWquA2lnHMtVZSUqduYDT+bw6HJDfmxr4HgeKrcSomoBKwRDSiXFLDxoklURjV3705pxO6/zWm9xBJHjA8eIscmvBRj7agmg3ajGwInX+554b+0X8hPYIgTEHhMk4BUVAUjjhNQR+7FNs4CA9mihi9YbaE8QoMFgppB4Fe9pX6tNq0q/7e4t6FMQz95whw8lGlbPWSUyfW3t9kpgCcmzha+KYOmMHAfnKZKilud4ye1fFlJgHpOu+qGY7L0Vjeo5ITjY2d5lUBzVnPrDsIceXSH2MyLHyA6aAY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3208.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(508600001)(110136005)(6486002)(66476007)(2906002)(38100700002)(26005)(52116002)(38350700002)(5660300002)(8936002)(66946007)(66556008)(316002)(53546011)(8676002)(31696002)(86362001)(31686004)(6666004)(16576012)(186003)(36756003)(2616005)(956004)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NmZHUGZucWJsem5GejdlNXpyNVQzNVMrVCt1UGdVMmNvUWRiQTFrSjViL0RH?= =?utf-8?B?eis0ZU5HVW9BbTJGVHd2L05oUnZ1WE1pRGJ0dGxqcUhoNmx6U3ZZSTR1VWNR?= =?utf-8?B?RDFDeG9Na1c5WjdtZ1RXeWdFT3VUZ3dQdnhudmw1QnJ3VmtrOW9LcmVkNHBD?= =?utf-8?B?VldRcldWSmVmdHlEeUJFZGVEa0orM0w3ajZ3dUE1T1FiTU1NVHlsWjg0Smdq?= =?utf-8?B?MW9STHRlS1ZFOTJsdFRGZG0vK1ZqVnM1WGNDZzJTWmdmWXJUL3FjbjZYTUsz?= =?utf-8?B?UTBhYThSL1dQWTJkTUprNDEzd2JKWXBENWN5V2tJTVZPUzcwb2srNi9ZbjV6?= =?utf-8?B?Q09aZmNFSEdabitKVlZzVGxZeUtrQnBFNVNObHUyV2lOZHBMNDFaQjBxK2cr?= =?utf-8?B?T0RJdEEvbTZaM3o2b1lpcC9zZGwza3A3Q3dlSm9iaXF6LzJmandJNHo3c2kw?= =?utf-8?B?QmJXckJZaXZ3UTZZMjNVaGY0N2FvZFAxVkV6L1YyTHphYWtISzhTRmdxRWZh?= =?utf-8?B?NE1XWEJnK3FNbUlJVkdQYjNkV05SZ1I0Y0JoTUM4b3g3cDhId0FoZUhKLy9w?= =?utf-8?B?eEdUQTA4bW1Obmt3cGs4akhYR3R5ZmsybkdiLzRUVEt3QXdRaTJrTk9DTGhJ?= =?utf-8?B?bitMdVVJQmY5eDlhUWtlaXJnUXRqdWVodE5qV2V0NWRpRnJTaU1sK2ZlOURv?= =?utf-8?B?cC9lY1RQcG0wRVRlVWNFWEdBanpZMXpqYWpqbmpVRVZIRkh3V3p5NFpxQXl1?= =?utf-8?B?Q3U0UnJmbHU1QlFGQnBNQ0tkbTJ6R3ZCclh3SU9wY2ZFRG45a0tEeHRaa2tt?= =?utf-8?B?RDFlSkRSWTlXV0Q0bjJ3eGtvdFJtb05UUkFKc2pEWlNmYXNZZUk0WDVBaUdv?= =?utf-8?B?bDJQMG5EdFFHd0xpVi9lRFh0ME9LZEYzbDlFcm1tUnNLQ1FNdm13UDlNTHRW?= =?utf-8?B?NUl4akRHN20yY1ZyYnVkT01hVlp1NnFBRjNGZ1g1eWVPRjRqSWJ6ZFZQUEMx?= =?utf-8?B?Rm8yYUJ6ZEZoOHBQNlVCY2k4QUZnYUt4UTR0N3Q4MGdvMDRmS09uRlNpMVN5?= =?utf-8?B?RmhOZE95V0ZWanh1MmRqZi9udVk1THZFMGFJNEFGdzQ4YlVrTWloc0VtbTRU?= =?utf-8?B?UVlGUkxTK1ZHQ2xiSWpybnRCR1YzY2ZtclpKczJmNFBGMStnR2ZCNU8yQVZI?= =?utf-8?B?T0cvajZZN0lLUE14cUN4N2RqNWRNWDVvSWlTWHQ5US82WTE2VWlIU3NES2Rz?= =?utf-8?B?Umlzd254dmJQU1F6U29xWXRxZTNVY3NqcnZqdTJlM0NNUzdQNDNUWXdtbUpm?= =?utf-8?B?UzVDQ0ttTitobGdQMGdOZmQ3bm85bkQzS0tOcTl3Rk4xeU8yYS9VNnFva2VJ?= =?utf-8?B?VDNBZkpVbFNoM1YvNGVRL2tRZEZMRCt2WW83UTk2cVQ5YUEydzlpbXNQaWtn?= =?utf-8?B?eUF2RHZHU21PcWFXQm5CSmxMeFZpaCtsK0hZZUtobnE1OGxWWkY2d080ZDY4?= =?utf-8?B?OFJidzhvUjYyTHJiOXRocGJUbVNYaDR5aUY0VCt0aFpuekdyVDdvOU1HV2VU?= =?utf-8?B?OXVlVHJ4WlF6RzBualpaMk9tNThpYU5VM1RRYW5pbERqQWZIY3d0cFVkT211?= =?utf-8?B?TnlHREtKeXNiMG80U1hJclNnaW9idmVSSVVNQUxDRVNGdTArN0JrSmVLNzZX?= =?utf-8?B?ekhMalNtc21SV2xHdlVoemdaZjhkYnZWazB2b3Z1TmthUmR3SVNHMkdEMm1V?= =?utf-8?B?a1JRVnpCU3dnbEkvS3JhcW1iS2lsakNRN20zd3dQQXErMUdtY2tIYlZWTCtV?= =?utf-8?B?ajByVHpYRi9YdGU4U2o2djFxZ3Jzd2QrQitjWU9CZVJ5M0NNSlk2LzdGZFVt?= =?utf-8?B?VFU1R3lDUjg2M1YwY1hnQ0gxUWhBK2JGVmx6Zlg4K3RqMTU2OGhTbitGQTBm?= =?utf-8?B?V2IzZE50QTlNVlBmcUs3V0w2NEhhaHd1cVBEUk9Kb1pNMDc3czl4NE01TmRn?= =?utf-8?B?UnlwQ0lmcTNYYU80NHpDOFBrM1BHa3Zvbm01ZWYxdkdUb1h6c1lRc3U1Q1VY?= =?utf-8?B?L0FYQ2I1OXNRVDhBaXJHb2w5VkdYdnNEeUpqWU1VazA5aHk2Y1RIU1ZlZ1hy?= =?utf-8?B?N0h0Qk95OHhKWFlHZ1hrNFdUOHF4ZXJ3TjNxajZlYWM3bUxoREp5VlIrSVFk?= =?utf-8?B?Q0NzWFQraTdpSkNVcGd3VlJlY25FZ3lMQUhLUG1IL1NyaTQwTzNOVEVDSTcz?= =?utf-8?B?dHUwSnFEcHVJalVSY3kxS0xOSThRPT0=?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: eb294845-716f-438a-c5c2-08d9b35aa75a X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3208.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2021 17:06:58.7467 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ky2gGJigz8rHZdzQ21eOVsy9slQ/GSXKL/Epk49f0+V/rjkF4ICUG5Rt2ZsqEMJWNeVzOn1oBLN5nmF5Nr38K7mKjbVpDNL1kmLa0ankx4o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2904 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10183 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2111290081 X-Proofpoint-ORIG-GUID: _uTXl6PWr9a9pBHz2Fx6J18ad17o3mCV X-Proofpoint-GUID: _uTXl6PWr9a9pBHz2Fx6J18ad17o3mCV X-Spam-Status: No, score=-5.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-help@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-help mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Nov 2021 17:07:08 -0000 I wonder if a useful heuristic would be for under some circumstances for the malloc library to call malloc_trim() every so often, based on passage of time in addition to the current calls to malloc_trim() based on user calls to free(). If free memory is cached but unused for a 'substantial' time while the program is doing 'other' things, it gets freed for other use by the OS. Setting the default to a time is large enough to avoid excess syscall overhead while being below typical human perception would be right for most contexts. This option could be under tunable control (at least until we understand better heuristics) with two variables: 1) A variable that says yes/no we will call malloc_trim() based on time. (Initial default: NO) 2) A variable that specifies the time frequency. (Initial default: 1/10 second or reasonable approximation) This approach avoids the need to have malloc_trim() or calls to other malloc internals appear in user applications. I have not made any investigation of implementation details. If anyone wants to take this idea and run with it, they are welcome to do so. - Patrick McGehearty On 11/26/2021 12:10 PM, Christian Hoff via Libc-help wrote: > Hello Adhemerval, > > On 11/25/21 9:56 PM, Adhemerval Zanella wrote: >> What I think we might improve is to maybe add an heuristic to call >> malloc_trim once a certain level of fragmentation in the main_arena >> is found. >> The question is which metric and threshold to use.  The trimming does >> have >> a cost, however I think it worth to decrease fragmentation and memory >> utilization. > > Yes, I think this is a very good idea. It is difficult for us to tell > our customers that our application consumes so much memory even while it > is not running any computations. Of course, we can call malloc_trim() as > a workaround, but even this is difficult as this is a Java application, > so calling system functions is not so straightforward. Any glibc > improvements in this area would be highly welcome. > > > Best regards, > >    Christian >