From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from DEU01-BE0-obe.outbound.protection.outlook.com (mail-be0deu01on2084.outbound.protection.outlook.com [40.107.127.84]) by sourceware.org (Postfix) with ESMTPS id E1CE13858035 for ; Wed, 16 Aug 2023 13:51:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E1CE13858035 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=dotscene.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=dotscene.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=D3f6C/pNCmyHk1hnTDIudFg5wHKDHC84YY4fjP9+jKi25r7BZ0YaYb1iPcGDDklz+ITXsOP05JH/zFhVg6E7eocaF+58tyVDHQ1Gjn39hk7npANxNeulyRyF1KzrHATmsoYQmhJv/AfvhdD8J+R5yIRND0Huj2Yo8ql1GTAZ+9QIJRuvJLiVBBV4pTq/eN+MwUx63fmljtIW2ovwaSLcOujxh31elApkasfl0NXM1g2CIIYYAduRKcmtbG1ywaYFHppixCqWPwix88lfwxg5ZJRy1LCpx29sz4mVlYg8XKtMFdOII8go/txvxFa0Jm9hjhRz1KKJ0rANA5qi9uNYgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gjwN3M29MBkHwNgWVcnen2b5G/FgdHW3JO6Lqoc5ZJw=; b=GgxUox7SU+JNMCJpB57TT1e/jJZPBGXvaIDMJCcvjDIXSwOCWHorfaeanrOBN41/bRpDvUiW/Qxd3CMb8exk/bQh7bkty39g0iMhBEDAmNFflHl5GBeifBMIX7qHVZUApTAyebjVVyjR1dA4hYj8kYyZiRFnhTB8HLHaeMljiThZXIeePVDiw41CvvJMhFzbvkX6ZKQN6YCkWW10Lg6LHJVzKWWJJDTg7FNnpqZQ2ze+AtYlwkHkwlG4Vi4ubJZqsUcWcHXSYx5gC2voX8hwi2CbVIVS5FXrERkbN02RoOJ4eMny3hN90/By7W63LDWYxywaUgHKlZp/q8qHSPfIig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=dotscene.com; dmarc=pass action=none header.from=dotscene.com; dkim=pass header.d=dotscene.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dotscene.onmicrosoft.com; s=selector2-dotscene-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gjwN3M29MBkHwNgWVcnen2b5G/FgdHW3JO6Lqoc5ZJw=; b=cidT8N/y9DkJDrGvO419XkdKajw5cSYulbirscNICvzQu2Zp+lahyh8GDUTpjJ8Xq0NUbYs3bbsGeqmWcuN4a9z9ixdSnO4j1rxGMWdFCVjCO54gyU0HmtKnRZKasRefjncJjs7fS9aJ3fDj9N26iatc/rdo7rkQp3rSk/ZXZnc= Received: from BEZP281MB3234.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:75::12) by FR3P281MB1694.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:7f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6678.29; Wed, 16 Aug 2023 13:51:35 +0000 Received: from BEZP281MB3234.DEUP281.PROD.OUTLOOK.COM ([fe80::d092:5b48:c858:6b52]) by BEZP281MB3234.DEUP281.PROD.OUTLOOK.COM ([fe80::d092:5b48:c858:6b52%6]) with mapi id 15.20.6678.029; Wed, 16 Aug 2023 13:51:35 +0000 From: Florian Kramer To: "libc-help@sourceware.org" Subject: Questions about the heap manager in glibc Thread-Topic: Questions about the heap manager in glibc Thread-Index: AQHZ0EcNsXTSzY6UuUiji9b34C5EQw== Date: Wed, 16 Aug 2023 13:51:35 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=dotscene.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: BEZP281MB3234:EE_|FR3P281MB1694:EE_ x-ms-office365-filtering-correlation-id: f5eab64c-808f-43ca-e291-08db9e5fe7ce x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: TYtbQZr/AOf93qlx9ndR8w7L0dhPAA9hILfQOuKfTx/oJ2uu+fCjHKtqpY/6WnxsW0rq2udNsthQi76KEY+mQsJIlTZ0U5ndgZ7XCFgQSxxMsMayv6tOa1BivK1Ef2+ct8SfB+JsTDs9C78GQ+BpSP9T33CjlxRyfyTTH9+qnMGXCUz0cbrJ5t7qJoVlwzxJ4pc9C5u8XH0LVv7rUZiqwxIIvAVX8izSQCqHPKQDfkI/B/ul7zOb/X9yJptdrI9DerFdlh6idsv0TqbeL0CmVWQOSaI0afu5EPfbFPnEV+SQ3CUroD2pvHkpQVbc7sfR1CIFOQZhFdxWFEW0hoKOmYxdj49xGCQoH7ZmxelVZ6ckqNNt94qHIM0nGkm2y65i+wUdYofUR3hrgzh1dHAIdhuAfI/qtPrX5z2iFav6NsHkzvl1MdNGDWJtoj8RoH9aFNijw5Y3Qf36VCjN//c9yThXebP/xpJJKBi2mYJHkl1xNxa3j0qlJbBvQffg4WZC05A2fWxBn0gvaNq8ufYkFuVl3NmCXXuyLzLDi+XDy2QSk3sFs0ARGfGMqICym4eQ1R4BCLYFfI2uZyM6cNrPZIT6VWqqF9soLzaR4KTjDrHgknhdqm5SfJmGakIcR4hS9ZNuit5bBJO742pXEEU+GRk9XVWkLOmuGTAE9MSCk3BxPWdMjH7dvdth3AUTBXgB1kgPpoH29XwBlu9gRvKz+Q== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BEZP281MB3234.DEUP281.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(366004)(39830400003)(136003)(396003)(346002)(376002)(451199024)(1800799009)(186009)(1690799017)(316002)(6916009)(91956017)(66946007)(76116006)(64756008)(66476007)(66556008)(66446008)(122000001)(12101799020)(19627405001)(52536014)(166002)(41300700001)(5660300002)(8936002)(8676002)(38070700005)(38100700002)(66574015)(2906002)(83380400001)(26005)(55016003)(478600001)(86362001)(9686003)(33656002)(71200400001)(7696005)(6506007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?UJBDGYOOkzp1WRQLrXBPodcyKbJe1cO12ioGxPn9iDvVuDi39G+JYEhRdA?= =?iso-8859-1?Q?x417gNw1A3ll0iUjPGCAJhIqrJXOpiibWKq6v/t1jsJ5SoJFl51i3ztxMS?= =?iso-8859-1?Q?wCtiHO/XJojP8Yuy45rDekGvyahiUkY97vFqhJuyjg3UO5QAoyESaWQ3RC?= =?iso-8859-1?Q?RbMxUd5x03jwsOeu0ugEte9MKIilmi05ZeBikM05/0x1SYjH8PihRr+ipp?= =?iso-8859-1?Q?y/NUBD79koEHGrTcjs3C4HRNdUD9HGRiPhJnd4WUeLeTH+C5Kb9rMaY6dp?= =?iso-8859-1?Q?BpBlqHmOVbOFLA0V8wMCkeWbYiTItD311oUFMexHdL+3ED5xNkYuY66dEg?= =?iso-8859-1?Q?FSuPI02BUXqIjNs2c+KvN4EpnZKQt3mlM47Pavje3iHRY4TCFolNezSfX+?= =?iso-8859-1?Q?Xwyyz7bEBlBAir0U5AEW9xUlJ/e7onzfozcsvPshaliGRsuYmtFQGYq6bb?= =?iso-8859-1?Q?G+ZfULzRY6ZkaB8NhrtfwDcuq2ie+E7Ttum+uqrghqFg91EMpXh2+0ix3Q?= =?iso-8859-1?Q?/le3hgnIu/o2ouMlRI5/rhmlW9NdOK6Np6+/1bR267KpK4/NEJptTg4WHf?= =?iso-8859-1?Q?HG5klI/ZSIuMu3/v0REptukYX3wkpHzdQRRcEdudP8V9gVjNui47iX6iSY?= =?iso-8859-1?Q?aTJRfBoSj2kgCAkASaGDYjwMrLScgfbBbJ7ABWcRSEYMCVHncD4kZW8fAS?= =?iso-8859-1?Q?l/OxNWkNVVUCbYQRLx/DqSWCvRw1zotRgiRZzdFPEF0D0WJiNacuODzSr4?= =?iso-8859-1?Q?j1R6zzndLqtDsz33+bL/Z5/FGDpubXBvFbKCy49GwF3Md0FnQF2vDiBmGd?= =?iso-8859-1?Q?GBbBV+tKGRW7EUH+v13mVHURZRkijgai6EaWTzdi1TXz1GwBmdSnuvqs5E?= =?iso-8859-1?Q?bj8KfoWhOdwbCKHUeVxqzkAvEHD4a1EyGjs+BhlAh3/JXbDDH8fooVcQVl?= =?iso-8859-1?Q?gegIwJb+oqLeXK4NqhU1+lqu9j8+hqwnXxOLcHbDpIDfgbFXWpXZPQL2sp?= =?iso-8859-1?Q?En5utf65zLHK0CvnFwCXjxgT9m8+iFsffVU5grH5p02ZPy8M+qtdugSAQL?= =?iso-8859-1?Q?jaC7uaIIVEoVZtmIV/rmSK5H5EIqqAn5QYJTg9x8NoB5Kv4oB8IbFys0Fy?= =?iso-8859-1?Q?hobEwPPdyvpEpkheCpwE/kvJRp7QK6KRqow2a/lcvI7giAU1gDay1d2Mn2?= =?iso-8859-1?Q?Nn1IlieAqwQpThC0z7Zqi8DvKM3ThzHAtoo0GSYB68BMMGKABBvz+0yH5+?= =?iso-8859-1?Q?olhH8kItMT/tFm1lLl7cbIOxQaX50wkSJKehTd5A7l1GtPs5vAM33cnq4n?= =?iso-8859-1?Q?tt1+SIve37VFgUYCtXNpZCoLgfWQUdBM16RP2WlszxyX26uemZHsFAsUwb?= =?iso-8859-1?Q?RTURAkb58mkL5IqzCRaeXltgfptWdVzJTvVBwi+cUSY0QpaHhUfQZB8S/p?= =?iso-8859-1?Q?fZLFlzw6NK3JP/DK65LfKG+xmIIkkffi7o0JOM/JwxcIc+tHPMh1rKTodt?= =?iso-8859-1?Q?dGrVOzSY3IK5w8yycLOrFY9oEPzTmb3O1XtFLKu9mg+OvHzObbdaomv2ZX?= =?iso-8859-1?Q?xjGHMgOJHO9fpuUrFLceIITyFaIy2JHlW2SjOQVPpG5dlJvHCyBpso/6SU?= =?iso-8859-1?Q?GQ6+C6Pa5e2/087WROimX5o+4s0pXo4ElO?= Content-Type: multipart/alternative; boundary="_000_BEZP281MB3234D7EC6D4C9ECE2BA39712FE15ABEZP281MB3234DEUP_" MIME-Version: 1.0 X-OriginatorOrg: dotscene.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BEZP281MB3234.DEUP281.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: f5eab64c-808f-43ca-e291-08db9e5fe7ce X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Aug 2023 13:51:35.1020 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3282fbd5-d7ac-47d9-b9f6-823d45a564ff X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wIWi34vhUZ2M+/w5JZ61SnkwNFaINdjjIq7apEv/NPlM0M5Cy339k10/JLetxYkhANuWde2iomcs9GBXEc/sfg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: FR3P281MB1694 X-Spam-Status: No, score=0.9 required=5.0 tests=BAYES_50,DKIM_SIGNED,DKIM_VALID,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,TRACKER_ID,T_FILL_THIS_FORM_SHORT autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --_000_BEZP281MB3234D7EC6D4C9ECE2BA39712FE15ABEZP281MB3234DEUP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, I ran into some unexpected behaviour while writing c++ code, and wanted to = try and understand what was happening. We are using an `std::unordered_map` and inserting lots of data into it, re= sulting in about 1-2 GiB being used just by the map in our small-scale tests. The map uses the default std::allocator, = which in turn uses malloc and free from glibc (according to a quick google. We also tried a custom allocator which definitely just u= sed malloc and free). After the map is destructed and free is called on all the allocated chunks = of memory (There are a lot of them. The map seems to allocated memory in very small chunks) the memory is not returned to the= kernel, but instead remains in use by the program. We are running the software on ubuntu 22.04, so a slightly older linux kern= el. Some reading into how glibc handles allocating and freeing memory shows thi= s is an optimization, as the still held memory can be returned by a later call to malloc, which is more efficient than returning = it to the kernel and reacquiring it. The problem we ran into in our production environment is that this could lead to over 100GiB of mem= ory being reserved by a single process, without the memory being in use inside of the application (at least from a cpp point of= view). This led to semi-frequent crashes of the software, as there are several instances spawned through docker on the same machine a= ll using the systems memory. Our current fix is to just call malloc_trim to release the memory back to t= he os. I was hoping somebody on this mailing list could give more insight as to why the heap manager holds on to that much me= mory in this scenario, and also if calling malloc_trim is the intended way to handle this, or if there is a preferred method. Best Regards, Florian Florian Kramer Tel.: Mail: f.kramer@dotscene.com [https://static.dotscene.com/email-signatures/img/image002.png] dotscene GmbH Wentzingerstr. 17 79106 Freiburg Germany Tel.: +49 761 2163360 Mail: mail@dotscene.com Web: ?www.dotscene.com [https://static.dotscene.com/email-signatures/img/image004.png] [https://static.dotscene.com/email-sig= natures/img/image006.png] [https://static.dotscene.com/email-signatures/img/image008.pn= g] [https://static.dotscene.com/= email-signatures/img/image010.png] Gesch=E4ftsf=FChrer: Nicolas Trusch Maryan Wieland Handelsregister: HRB 715508 Registergericht: Amtsgericht Freiburg i.Br. Unternehmenssitz: Freiburg i.Br. Ust-IdNr.: DE309748553 This e-mail may contain confidential and/or privileged information. If you = are not the intended recipient please notify the sender immediately and des= troy this e-mail. Any unauthorized copying, disclosure or distribution of c= ontents of this e-mail is strictly forbidden. dotscene_signature_included --_000_BEZP281MB3234D7EC6D4C9ECE2BA39712FE15ABEZP281MB3234DEUP_--