From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id E591738582AD for ; Fri, 16 Sep 2022 20:14:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org E591738582AD Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1663359248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=K+stBJn8YwhYmTus/X8xX2+Crmf65U56YyAwGJYBOxk=; b=Tz7u/xdPrinuqdCQ7N7mya9kNr3sgo2AgA9X1CqekDzH3Ax1gVFANohaBzolL9Jy0jt90a lsBLEDTkJ4gEDyFn2CYcAYp7oJUAQMKcGjniM3+JISacTRZtwDJ2SsNbfOp/NUsv7174EF tZ9XbxOLrswSnO58E4BBXIa+zxyEX/k= Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-672-oP6ODpBaM5eqMzximDxE_w-1; Fri, 16 Sep 2022 16:14:06 -0400 X-MC-Unique: oP6ODpBaM5eqMzximDxE_w-1 Received: by mail-yb1-f199.google.com with SMTP id 66-20020a251145000000b006a7b4a27d04so19955934ybr.20 for ; Fri, 16 Sep 2022 13:14:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=pPLKtALhj0coadwaSwSrDzxYzuE1ViUz8a4PzjYlN94=; b=cvx0LFbm0e/hptyFwiqXnr1jt7PyWu5sslGJE0W57wJJ5pnmb9U5wxq/HY+nFzOKRO IjYcxosH9QcFHOog1nsNYuktrIvkJWs3821pXrLU+fWr6GzbEuJRRhcI9Cas6q8uDVF9 rieFe1wcTdT52fDupmlm3i97UFpCvKcnlu/uRmvELBJSDflfU1fYYTMDXyZ9LmdayRuP PFLw0YLdhgOa6zKTV6COJSy+093n3bRovFTHPCntiSp3cA/6Wn04JrxGshbpp/n8MfYi vEIpCHYjSxpRyZISgBEXeQB3vmyd3wMZiJTgvDofvII7bn3XL/0TfJPbhUWcvDNlRXcC VdGw== X-Gm-Message-State: ACrzQf1PTn94m2jHJBMVR4lModQErt30bO7cnJRMEe/5WOVBw1k81Fuf W+XrHXlasUTM1tMsrLTGesKAsScmIwbOr0tbNBXjzLb1asEtyHvTutpc9JQ4tK/8sqUkQJXzey4 xQ2NWJUAzK3uKaCmA5qP/NYHXvqvIWm0= X-Received: by 2002:a81:a093:0:b0:345:c52:945c with SMTP id x141-20020a81a093000000b003450c52945cmr5723046ywg.341.1663359246164; Fri, 16 Sep 2022 13:14:06 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6A1x4KFecE/RnV6gbDSQdmCfaEVweyGSiTgLng1/hqYCU8VoxhKhqN5lxnZelCf09uFlsoVkhAZ9WDBeIS1Uk= X-Received: by 2002:a81:a093:0:b0:345:c52:945c with SMTP id x141-20020a81a093000000b003450c52945cmr5723017ywg.341.1663359245774; Fri, 16 Sep 2022 13:14:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jeff Johnston Date: Fri, 16 Sep 2022 16:13:54 -0400 Message-ID: Subject: Re: Malloc: unusable area at the end of the heap section To: Torbjorn SVENSSON Cc: Jerome Leroux , "newlib@sourceware.org" X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/mixed; boundary="000000000000fad54a05e8d100c7" X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,HTML_MESSAGE,KAM_LOTSOFHASH,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --000000000000fad54a05e8d100c7 Content-Type: multipart/alternative; boundary="000000000000fad54905e8d100c5" --000000000000fad54905e8d100c5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ok, I am attaching the modified patch after discussing with Torbjorn about the licensing. Torbjorn, please review and let me know if there are changes you would like, otherwise, I will push. -- Jeff J. On Wed, Sep 14, 2022 at 4:52 PM Jeff Johnston wrote: > Hi Torbjorn, > > I took a look at what would be needed to make this more generic. I have a > patch almost ready, but > the one issue is that your libc/sys/arm/sysconf.c file does not have a > license. Could you please append a > newer version of the file with a license and I will complete the patch for > you to review? > > I decided to go with a simple HAVE_xxxx macro rather than a configure > option so any platform can > simply set the flag in configure.host. > > -- Jeff J. > > On Tue, Jun 7, 2022 at 12:03 PM Jeff Johnston wrote: > >> Hi Torbjorn, >> >> I think it would be useful. Do you want to modify the patch to be more >> generic? I am thinking of a var set in configure.host that sets a compi= le >> flag at the end such as _USE_SYSCONF_FOR_PAGESIZE. Then, >> the default sysconf.c can be put in the newlib/libc/unix directory. It >> is then a straight-forward exercise to add this as an enablement >> configuration option. What do you think? >> >> -- Jeff J. >> >> On Tue, Jun 7, 2022 at 2:18 AM Torbjorn SVENSSON via Newlib < >> newlib@sourceware.org> wrote: >> >>> Hello, >>> >>> A while back, I provided a patch[1] to newlib that would allow the >>> application to override the pagesize for malloc, but the patch got stal= led. >>> Maybe this would be a good time to take another look at the patch and s= ee >>> if it would actually fix the generic newlib usage or if it's still >>> something that is only applicable for small embedded targets. >>> >>> [1] https://ecos.sourceware.org/ml/newlib/current/017616.html >>> >>> Kind regards, >>> Torbj=C3=B6rn >>> >>> > -----Original Message----- >>> > From: Newlib >> > bounces+torbjorn.svensson=3Dst.com@sourceware.org> On Behalf Of Jerome >>> > Leroux >>> > Sent: den 6 juni 2022 23:18 >>> > To: newlib@sourceware.org >>> > Subject: Malloc: unusable area at the end of the heap section >>> > >>> > Hello Newlib developers, >>> > >>> > I am a user of Newlib in a project that runs on an NXP MCU. >>> > I am using MCUXpressoIDE_11.3.0_5180_prc3, which comes with GCC =E2= =80=9Carm- >>> > none-eabi-gcc.exe (GNU Arm Embedded Toolchain >>> > 9-2020-q2-update) 9.3.1 20200408 (release)=E2=80=9D and Newlib 3.3.0. >>> > >>> > I have identified an issue in malloc, and I think the problem is still >>> present in >>> > the latest version of Newlib. I could >>> > not see any changes in the incriminated code since Newlib 3.3.0. >>> > >>> > I noticed this issue only in the standard malloc implementation and >>> not in the >>> > nano-malloc version. >>> > >>> > Here is a description of the problem: >>> > The allocator splits the heap into pages. When a page is full, it >>> increases the >>> > heap size by reserving a new page in the >>> > heap section. When reserving a new page, the allocator keeps the page >>> end >>> > address aligned with malloc_getpagesize, which >>> > is set to 4096 by default. If there is not enough space to reserve the >>> full page, >>> > the allocation fails even if there is >>> > enough space in the heap to allocate the chunk of memory. >>> > Because the issue is related to the heap end address and how the link= er >>> > positions the heap, the same sequence of >>> > allocations may lead to different results (failure or success) >>> depending on the >>> > location of the heap, even if the heap >>> > size is constant. Typically, adding a new C global variable can shift >>> the start >>> > address of the heap section and cause a >>> > malloc error. >>> > >>> > For example, with a heap section of 4096 bytes (0x1000 bytes): >>> > If the heap section address is 0x20100-0x21100, during the >>> initialization, the >>> > page end address is set to 0x21000 >>> > (aligned on 4096). We will be able to allocate until the address >>> 0x21000. After >>> > that, the allocator will try to reserve >>> > a new page, but it will fail because it won=E2=80=99t be able to rese= rve a >>> 4096 bytes >>> > page from 0x21000 to 0x22000. The >>> > following allocations will fail. The usable heap size is 3840 bytes >>> (0x21000 - >>> > 0x20100) instead of 4096. >>> > If the heap section address is 0x20F00-0x21F00 (same size), with the >>> same >>> > scenario, the usable heap size is 256 bytes >>> > (0x21000 - 0x20F00). >>> > Here are two examples of heap configurations: >>> > https://gist.github.com/jerome- >>> > leroux/759159fbd3e7bb5e189dbceb04636914?permalink_comment_id=3D4191 >>> > 266#gistcomment-4191266 >>> > >>> > I did not dig into the implementation so much. From my understanding, >>> the >>> > problem comes from the usage of >>> > "malloc_getpagesize" (see >>> > https://github.com/bminor/newlib/blob/830a9b707caa5e343b6ffce7fcb2d3c >>> > a97e3259c/newlib/libc/stdlib/_mallocr.c#L198) in >>> > "malloc_extend_top" (probably here >>> > https://github.com/bminor/newlib/blob/830a9b707caa5e343b6ffce7fcb2d3c >>> > a97e3259c/newlib/libc/stdlib/_mallocr.c#L2166). >>> > I can understand it makes sense to keep the pages aligned when running >>> in a >>> > system that implements virtual memory. >>> > Still, on an MCU, the heap is just a contiguous chunk of memory >>> allocated at >>> > link time. Furthermore, the heap size is >>> > usually pretty small (a few kilobytes), so potentially wasting 4 KB of >>> memory >>> > is unacceptable. Using the default >>> > implementation of "sbrk" documented at >>> > https://sourceware.org/newlib/libc.html#index-sbrk will lead to the >>> > problem. >>> > >>> > I have written a simple example that demonstrates the issue (see >>> > https://gist.github.com/jerome- >>> > leroux/759159fbd3e7bb5e189dbceb04636914 ). To reproduce the problem, >>> > define the macros >>> > HEAP_SECTION_START_SYMBOL and HEAP_SECTION_END_SYMBOL, which >>> > are specific to your environment. Then call the function >>> > "test_malloc()". >>> > >>> > I tried to find someone with the same issue, but I couldn=E2=80=99t. = The >>> related >>> > commits/discussions I found are: >>> > - >>> > https://github.com/bminor/newlib/commit/4a3d0a5a5d829c05868a34658eb >>> > 45731dbb5112b >>> > - >>> https://stackoverflow.com/questions/39088598/malloc-in-newlib-does-it- >>> > waste-memory-after-one-big-failure-allocation >>> > >>> > Can anyone confirm what I have noticed? >>> > >>> > Thanks. >>> > >>> > -- >>> > Jerome Leroux >>> >>> --000000000000fad54905e8d100c5-- --000000000000fad54a05e8d100c7 Content-Type: text/x-patch; charset="US-ASCII"; name="0001-Implement-sysconf-for-Arm.patch" Content-Disposition: attachment; filename="0001-Implement-sysconf-for-Arm.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_l84x3qwu0 RnJvbSBhYWUzODMxZTYzMzRlZWQ1NzFlOGMyZWIwNDI5NWU1M2NmMjRmMDI0 IE1vbiBTZXAgMTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBKZWZmIEpvaG5zdG9u IDxqam9obnN0bkByZWRoYXQuY29tPgpEYXRlOiBGcmksIDE2IFNlcCAyMDIy IDE2OjA0OjIxIC0wNDAwClN1YmplY3Q6IFtQQVRDSF0gSW1wbGVtZW50IHN5 c2NvbmYgZm9yIEFybQoKLSBhZGQgc3VwcG9ydCBmb3IgdXNpbmcgc3lzY29u ZiB0byBnZXQgcGFnZSBzaXplIGluIF9tYWxsb2NyLmMgdmlhCiAgSEFWRV9T WVNDT05GX1BBR0VTSVpFIGZsYWcgc2V0IGluIGNvbmZpZ3VyZS5ob3N0Ci0g c2V0IGZsYWcgaW4gY29uZmlndXJlLmhvc3QgZm9yIGFybSBhbmQgYWRkIGEg ZGVmYXVsdCBzeXNjb25mIGltcGxlbWVudGF0aW9uCiAgaW4gbGliYy9zeXMv YXJtIHRoYXQgcmV0dXJucyB0aGUgcGFnZSBzaXplCi0gdGhlIGRlZmF1bHQg aW1wbGVtZW50YXRpb24gY2FuIGJlIG92ZXJyaWRkZW4gb3V0c2lkZSBuZXds aWIgdG8gYWxsb3cgYQogIGRpZmZlcmVudCBwYWdlIHNpemUgdG8gaW1wcm92 ZSBtYWxsb2Mgb24gZGV2aWNlcyB3aXRoIGEgc21hbGwgZm9vdHByaW50CiAg d2l0aG91dCBuZWVkaW5nIHRvIHJlYnVpbGQgbmV3bGliCi0gdGhpcyBwYXRj aCBpcyBiYXNlZCBvbiBhIGNvbnRyaWJ1dGlvbiBmcm9tIFRvcmJqb3JuIFN2 ZW5zc29uIGFuZAogIE5pa2xhcyBEYWhscXVpc3QgKGh0dHBzOi8vZWNvcy5z b3VyY2V3YXJlLm9yZy9tbC9uZXdsaWIvY3VycmVudC8wMTc2MTYuaHRtbCkK LS0tCiBuZXdsaWIvY29uZmlndXJlLmhvc3QgICAgICAgICAgICB8ICAyICsr CiBuZXdsaWIvbGliYy9zdGRsaWIvX21hbGxvY3IuYyAgICB8ICAyICsrCiBu ZXdsaWIvbGliYy9zeXMvYXJtL01ha2VmaWxlLmluYyB8ICAyICstCiBuZXds aWIvbGliYy9zeXMvYXJtL3N5c2NvbmYuYyAgICB8IDMwICsrKysrKysrKysr KysrKysrKysrKysrKysrKysrKwogNCBmaWxlcyBjaGFuZ2VkLCAzNSBpbnNl cnRpb25zKCspLCAxIGRlbGV0aW9uKC0pCiBjcmVhdGUgbW9kZSAxMDA2NDQg bmV3bGliL2xpYmMvc3lzL2FybS9zeXNjb25mLmMKCmRpZmYgLS1naXQgYS9u ZXdsaWIvY29uZmlndXJlLmhvc3QgYi9uZXdsaWIvY29uZmlndXJlLmhvc3QK aW5kZXggOThjZTA3ZC4uMzJkMTQzNiAxMDA2NDQKLS0tIGEvbmV3bGliL2Nv bmZpZ3VyZS5ob3N0CisrKyBiL25ld2xpYi9jb25maWd1cmUuaG9zdApAQCAt NjI4LDYgKzYyOCw3IEBAIG5ld2xpYl9jZmxhZ3M9IiR7bmV3bGliX2NmbGFn c30gLURDTE9DS19QUk9WSURFRCAtRE1BTExPQ19QUk9WSURFRCAtREVYSVRf UFJPVklECiAJOzsKICAgYXJtKi0qLXBlKQogCXN5c2NhbGxfZGlyPXN5c2Nh bGxzCisJbmV3bGliX2NmbGFncz0iJHtuZXdsaWJfY2ZsYWdzfSAtREhBVkVf U1lTQ09ORl9QQUdFU0laRSIKIAk7OwogICBhcm0qLSotKikKIAlzeXNjYWxs X2Rpcj1zeXNjYWxscwpAQCAtNjQyLDYgKzY0Myw3IEBAIG5ld2xpYl9jZmxh Z3M9IiR7bmV3bGliX2NmbGFnc30gLURDTE9DS19QUk9WSURFRCAtRE1BTExP Q19QUk9WSURFRCAtREVYSVRfUFJPVklECiAjICAgICAgICAgbmV3bGliX2Nm bGFncz0iJHtuZXdsaWJfY2ZsYWdzfSAtREFSTV9SRFBfTU9OSVRPUiIKIAkg IG5ld2xpYl9jZmxhZ3M9IiR7bmV3bGliX2NmbGFnc30gLURBUk1fUkRJX01P TklUT1IiCiAJZmkKKwluZXdsaWJfY2ZsYWdzPSIke25ld2xpYl9jZmxhZ3N9 IC1ESEFWRV9TWVNDT05GX1BBR0VTSVpFIgogCTs7CiAgIGF2ciopCiAJbmV3 bGliX2NmbGFncz0iJHtuZXdsaWJfY2ZsYWdzfSAtRE5PX0VYRUMgLURTTUFM TF9NRU1PUlkgLURNSVNTSU5HX1NZU0NBTExfTkFNRVMiCmRpZmYgLS1naXQg YS9uZXdsaWIvbGliYy9zdGRsaWIvX21hbGxvY3IuYyBiL25ld2xpYi9saWJj L3N0ZGxpYi9fbWFsbG9jci5jCmluZGV4IDRiNTM5OTcuLjE5OTdiNmQgMTAw NjQ0Ci0tLSBhL25ld2xpYi9saWJjL3N0ZGxpYi9fbWFsbG9jci5jCisrKyBi L25ld2xpYi9saWJjL3N0ZGxpYi9fbWFsbG9jci5jCkBAIC0zMjAsMTIgKzMy MCwxNCBAQCBleHRlcm4gIkMiIHsKICNlbmRpZgogCiAjaWZuZGVmIF9XSU4z MgorI2lmbmRlZiBIQVZFX1NZU0NPTkZfUEFHRVNJWkUKICNpZmRlZiBTTUFM TF9NRU1PUlkKICNkZWZpbmUgbWFsbG9jX2dldHBhZ2VzaXplICgxMjgpCiAj ZWxzZQogI2RlZmluZSBtYWxsb2NfZ2V0cGFnZXNpemUgKDQwOTYpCiAjZW5k aWYKICNlbmRpZgorI2VuZGlmCiAKICNpZiBfX1NURF9DCiBleHRlcm4gdm9p ZCBfX21hbGxvY19sb2NrKHN0cnVjdCBfcmVlbnQgKik7CmRpZmYgLS1naXQg YS9uZXdsaWIvbGliYy9zeXMvYXJtL01ha2VmaWxlLmluYyBiL25ld2xpYi9s aWJjL3N5cy9hcm0vTWFrZWZpbGUuaW5jCmluZGV4IDQ5MGE5NjMuLjAxMjI5 NTYgMTAwNjQ0Ci0tLSBhL25ld2xpYi9saWJjL3N5cy9hcm0vTWFrZWZpbGUu aW5jCisrKyBiL25ld2xpYi9saWJjL3N5cy9hcm0vTWFrZWZpbGUuaW5jCkBA IC0xLDYgKzEsNiBAQAogQU1fQ1BQRkxBR1NfJUMlID0gLUkkKHNyY2Rpcikv bGliYy9tYWNoaW5lL2FybQogCi1saWJjX2FfU09VUkNFUyArPSAlRCUvYWNj ZXNzLmMgJUQlL2FlYWJpX2F0ZXhpdC5jCitsaWJjX2FfU09VUkNFUyArPSAl RCUvYWNjZXNzLmMgJUQlL2FlYWJpX2F0ZXhpdC5jICVEJS9zeXNjb25mLmMK IGlmIE1BWV9TVVBQTFlfU1lTQ0FMTFMKIGxpYmNfYV9TT1VSQ0VTICs9ICVE JS9saWJjZnVuYy5jICVEJS90cmFwLlMgJUQlL3N5c2NhbGxzLmMKIGVuZGlm CmRpZmYgLS1naXQgYS9uZXdsaWIvbGliYy9zeXMvYXJtL3N5c2NvbmYuYyBi L25ld2xpYi9saWJjL3N5cy9hcm0vc3lzY29uZi5jCm5ldyBmaWxlIG1vZGUg MTAwNjQ0CmluZGV4IDAwMDAwMDAuLmRiZWQ3ZDcKLS0tIC9kZXYvbnVsbAor KysgYi9uZXdsaWIvbGliYy9zeXMvYXJtL3N5c2NvbmYuYwpAQCAtMCwwICsx LDMwIEBACisvKiBsaWJjL3N5cy9hcm0vc3lzY29uZi5jIC0gVGhlIHN5c2Nv bmYgZnVuY3Rpb24gKi8KKworLyogQ29weXJpZ2h0IDIwMjAsIFNUTWljcm9l bGVjdHJvbmljcworICoKKyAqIEFsbCByaWdodHMgcmVzZXJ2ZWQuCisgKgor ICogUmVkaXN0cmlidXRpb24sIG1vZGlmaWNhdGlvbiwgYW5kIHVzZSBpbiBz b3VyY2UgYW5kIGJpbmFyeSBmb3JtcyBpcyBwZXJtaXR0ZWQKKyAqIHByb3Zp ZGVkIHRoYXQgdGhlIGFib3ZlIGNvcHlyaWdodCBub3RpY2UgYW5kIGZvbGxv d2luZyBwYXJhZ3JhcGggYXJlCisgKiBkdXBsaWNhdGVkIGluIGFsbCBzdWNo IGZvcm1zLgorICoKKyAqIFRoaXMgZmlsZSBpcyBkaXN0cmlidXRlZCBXSVRI T1VUIEFOWSBXQVJSQU5UWTsgd2l0aG91dCBldmVuIHRoZSBpbXBsaWVkCisg KiB3YXJyYW50eSBvZiBNRVJDSEFOVEFCSUxJVFkgb3IgRklUTkVTUyBGT1Ig QSBQQVJUSUNVTEFSIFBVUlBPU0UuCisgKi8KKworI2luY2x1ZGUgPHVuaXN0 ZC5oPgorI2luY2x1ZGUgPGVycm5vLmg+CisKK2xvbmcgc3lzY29uZihpbnQg bmFtZSkKK3sKKyAgc3dpdGNoIChuYW1lKQorICB7CisgIGNhc2UgX1NDX1BB R0VTSVpFOgorICAgIHJldHVybiA0MDk2OworCisgIGRlZmF1bHQ6CisgICAg ZXJybm8gPSBFSU5WQUw7CisgICAgcmV0dXJuIC0xOworICB9CisgIHJldHVy biAtMTsgLyogQ2FuJ3QgZ2V0IGhlcmUgKi8KK30KLS0gCjEuOC4zLjEKCg== --000000000000fad54a05e8d100c7--