From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 327AC385116E for ; Wed, 14 Sep 2022 20:52:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 327AC385116E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1663188738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=V8AcbnpQhBzW5bnaiufNEPqJrGUP1VnxB5WZPzZSct4=; b=CKoDYakA4HLGl2cnHFjSWtT53AmhtSOdXsoKRsZDL19hmoLeIaVa62/KwJqk9eJUD4IOjo mq6epHpMVc1bZKF5GbKTm3Ct2n2OiYZqOMeWiQgaMjNeWiLBrPnFBlQ05Anq0oyNot62eF rBH5qNzwLFVjVCUbKx5cezxTSM1oqr4= Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-517-saS2m6qTMQGDiamohwiAZA-1; Wed, 14 Sep 2022 16:52:17 -0400 X-MC-Unique: saS2m6qTMQGDiamohwiAZA-1 Received: by mail-yb1-f200.google.com with SMTP id u12-20020a25094c000000b006a9ad6b2cebso14003231ybm.15 for ; Wed, 14 Sep 2022 13:52:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=V8AcbnpQhBzW5bnaiufNEPqJrGUP1VnxB5WZPzZSct4=; b=uNsd1OhT78BMoj9M7kTLOZX3iISsfbfE+zkA1gUDHdhwDoDS118/qaQjo7KrTylwEz nn/7FEkrp34I/JqYKJ6bdrPqBXDyUcFNrW2qMJQ+143lUJqpWY4KO0SfIJ7lSjHJyFvN ch3w+X8/ChAbtNEL/63b8uP61bX3XnMvkHwIO/MUCPRM4gquwyzUuptSUHSmj9n/g4+j samNVJyN2J+2ws91k5/R03mBuXEnX0D4iG7yp8Z74voaNCRb6m73yUBeIaDJ2TvkSK9Z GRnbEH8iKV14FL40sv2qIl9xhPlivwvpUG1nEdNID0PAe4atFXulChJEZdKo4SdfgcMp RNUg== X-Gm-Message-State: ACgBeo2IGIoq6P6nR63rQ4+rzPV+BYE9zPpl52tEE8hJnS7CVYxolvgJ B+7ReEZvz9V4wl/apm/ZRo4aduM6hkZIUV/XXgxeIlWViPIrR312aoLhJd7qIPQ8+BFDGBt2nMo b4tLvdDeUEo1vHHE+YYsE5kaKiAaGxuc= X-Received: by 2002:a25:4e0b:0:b0:6a8:ec00:c1c5 with SMTP id c11-20020a254e0b000000b006a8ec00c1c5mr30377959ybb.525.1663188736967; Wed, 14 Sep 2022 13:52:16 -0700 (PDT) X-Google-Smtp-Source: AA6agR4IHzn587kIR/blBIhPtNtjBhc7+yMEkh32gPlmdXxRL4qytutzTnpliAoiQIrdIeCh+r+F6N415J4wc5adkMo= X-Received: by 2002:a25:4e0b:0:b0:6a8:ec00:c1c5 with SMTP id c11-20020a254e0b000000b006a8ec00c1c5mr30377941ybb.525.1663188736662; Wed, 14 Sep 2022 13:52:16 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jeff Johnston Date: Wed, 14 Sep 2022 16:52:05 -0400 Message-ID: Subject: Re: Malloc: unusable area at the end of the heap section To: Torbjorn SVENSSON Cc: Jerome Leroux , "newlib@sourceware.org" X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/alternative; boundary="000000000000d7f02e05e8a94d55" X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,HTML_MESSAGE,KAM_LOTSOFHASH,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --000000000000d7f02e05e8a94d55 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Torbjorn, I took a look at what would be needed to make this more generic. I have a patch almost ready, but the one issue is that your libc/sys/arm/sysconf.c file does not have a license. Could you please append a newer version of the file with a license and I will complete the patch for you to review? I decided to go with a simple HAVE_xxxx macro rather than a configure option so any platform can simply set the flag in configure.host. -- Jeff J. On Tue, Jun 7, 2022 at 12:03 PM Jeff Johnston wrote: > Hi Torbjorn, > > I think it would be useful. Do you want to modify the patch to be more > generic? I am thinking of a var set in configure.host that sets a compile > flag at the end such as _USE_SYSCONF_FOR_PAGESIZE. Then, > the default sysconf.c can be put in the newlib/libc/unix directory. It is > then a straight-forward exercise to add this as an enablement configurati= on > option. What do you think? > > -- Jeff J. > > On Tue, Jun 7, 2022 at 2:18 AM Torbjorn SVENSSON via Newlib < > newlib@sourceware.org> wrote: > >> Hello, >> >> A while back, I provided a patch[1] to newlib that would allow the >> application to override the pagesize for malloc, but the patch got stall= ed. >> Maybe this would be a good time to take another look at the patch and see >> if it would actually fix the generic newlib usage or if it's still >> something that is only applicable for small embedded targets. >> >> [1] https://ecos.sourceware.org/ml/newlib/current/017616.html >> >> Kind regards, >> Torbj=C3=B6rn >> >> > -----Original Message----- >> > From: Newlib > > bounces+torbjorn.svensson=3Dst.com@sourceware.org> On Behalf Of Jerome >> > Leroux >> > Sent: den 6 juni 2022 23:18 >> > To: newlib@sourceware.org >> > Subject: Malloc: unusable area at the end of the heap section >> > >> > Hello Newlib developers, >> > >> > I am a user of Newlib in a project that runs on an NXP MCU. >> > I am using MCUXpressoIDE_11.3.0_5180_prc3, which comes with GCC =E2=80= =9Carm- >> > none-eabi-gcc.exe (GNU Arm Embedded Toolchain >> > 9-2020-q2-update) 9.3.1 20200408 (release)=E2=80=9D and Newlib 3.3.0. >> > >> > I have identified an issue in malloc, and I think the problem is still >> present in >> > the latest version of Newlib. I could >> > not see any changes in the incriminated code since Newlib 3.3.0. >> > >> > I noticed this issue only in the standard malloc implementation and not >> in the >> > nano-malloc version. >> > >> > Here is a description of the problem: >> > The allocator splits the heap into pages. When a page is full, it >> increases the >> > heap size by reserving a new page in the >> > heap section. When reserving a new page, the allocator keeps the page >> end >> > address aligned with malloc_getpagesize, which >> > is set to 4096 by default. If there is not enough space to reserve the >> full page, >> > the allocation fails even if there is >> > enough space in the heap to allocate the chunk of memory. >> > Because the issue is related to the heap end address and how the linker >> > positions the heap, the same sequence of >> > allocations may lead to different results (failure or success) >> depending on the >> > location of the heap, even if the heap >> > size is constant. Typically, adding a new C global variable can shift >> the start >> > address of the heap section and cause a >> > malloc error. >> > >> > For example, with a heap section of 4096 bytes (0x1000 bytes): >> > If the heap section address is 0x20100-0x21100, during the >> initialization, the >> > page end address is set to 0x21000 >> > (aligned on 4096). We will be able to allocate until the address >> 0x21000. After >> > that, the allocator will try to reserve >> > a new page, but it will fail because it won=E2=80=99t be able to reser= ve a 4096 >> bytes >> > page from 0x21000 to 0x22000. The >> > following allocations will fail. The usable heap size is 3840 bytes >> (0x21000 - >> > 0x20100) instead of 4096. >> > If the heap section address is 0x20F00-0x21F00 (same size), with the >> same >> > scenario, the usable heap size is 256 bytes >> > (0x21000 - 0x20F00). >> > Here are two examples of heap configurations: >> > https://gist.github.com/jerome- >> > leroux/759159fbd3e7bb5e189dbceb04636914?permalink_comment_id=3D4191 >> > 266#gistcomment-4191266 >> > >> > I did not dig into the implementation so much. From my understanding, >> the >> > problem comes from the usage of >> > "malloc_getpagesize" (see >> > https://github.com/bminor/newlib/blob/830a9b707caa5e343b6ffce7fcb2d3c >> > a97e3259c/newlib/libc/stdlib/_mallocr.c#L198) in >> > "malloc_extend_top" (probably here >> > https://github.com/bminor/newlib/blob/830a9b707caa5e343b6ffce7fcb2d3c >> > a97e3259c/newlib/libc/stdlib/_mallocr.c#L2166). >> > I can understand it makes sense to keep the pages aligned when running >> in a >> > system that implements virtual memory. >> > Still, on an MCU, the heap is just a contiguous chunk of memory >> allocated at >> > link time. Furthermore, the heap size is >> > usually pretty small (a few kilobytes), so potentially wasting 4 KB of >> memory >> > is unacceptable. Using the default >> > implementation of "sbrk" documented at >> > https://sourceware.org/newlib/libc.html#index-sbrk will lead to the >> > problem. >> > >> > I have written a simple example that demonstrates the issue (see >> > https://gist.github.com/jerome- >> > leroux/759159fbd3e7bb5e189dbceb04636914 ). To reproduce the problem, >> > define the macros >> > HEAP_SECTION_START_SYMBOL and HEAP_SECTION_END_SYMBOL, which >> > are specific to your environment. Then call the function >> > "test_malloc()". >> > >> > I tried to find someone with the same issue, but I couldn=E2=80=99t. T= he related >> > commits/discussions I found are: >> > - >> > https://github.com/bminor/newlib/commit/4a3d0a5a5d829c05868a34658eb >> > 45731dbb5112b >> > - >> https://stackoverflow.com/questions/39088598/malloc-in-newlib-does-it- >> > waste-memory-after-one-big-failure-allocation >> > >> > Can anyone confirm what I have noticed? >> > >> > Thanks. >> > >> > -- >> > Jerome Leroux >> >> --000000000000d7f02e05e8a94d55--