From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 7469C38303C6 for ; Tue, 7 Jun 2022 16:04:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7469C38303C6 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654617841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=CGBJ5uuz+lBQe5Np5stF4e+ZxXjR2MQy/2/AAQ638/8=; b=SXd8mNfvNUKypHZX3zLLTeHO1Zd8OgvbpFkAOrrl468m4croZb/3FIRaDUqB9pxg0eKY7y rYt2HOJTZLKEDIPezqxGQUs0I4SmSwHGffxPZGz7FsWSGIAf2swBwGnEtm/A8uAx7BYb5Q RYa67rwflBRyc787A6rN0uoIsZ2UD9c= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-663-FYi3TRStOt2gGYD7vbJr2Q-1; Tue, 07 Jun 2022 12:03:59 -0400 X-MC-Unique: FYi3TRStOt2gGYD7vbJr2Q-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-30c14765d55so152479267b3.13 for ; Tue, 07 Jun 2022 09:03:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CGBJ5uuz+lBQe5Np5stF4e+ZxXjR2MQy/2/AAQ638/8=; b=vRglumQkOvdvtz8ZTm0+ryTyoa1DJOkcpMU6793HIRUggNK5wktJQzSMB2nXPlLqWa YikONtjPx/aqOVz9hoxuBsag7kWrwzAl2qBUv0r3LVltcfviSuoXEnf753a/QiZuJldt bP9kc86vRsrfemmBubeE3MT4OyIyfepvJ+CEKpOOr1VQWdAnXoozfJqbVxjTjurG6mTB a6oIy1jRDPFxqiwBNEKu19FTEGzSp16H0tdvMHBVP+rNnQ4PDrJ1DaYq+3dZtYUqs7G6 yKvIC1vO7sbizcPlUDYMkI9ILhuFyuMuFsbyqjzbjCAD/Vdjb/XKfX+a1RWV24nN/iKE vhpA== X-Gm-Message-State: AOAM531wPPxKOZrlUmVRi2OHQCWDRNI8TeY7rl8E3wFaiVuGNvHRaZ2v h+mRC6C3mGPCRax+kPb88ufwLNpBLQE9df3JutUJBiolg7BszTr0jD9+ZqeQlZTQo76sJVehp+K MCoeFNuR9jneC2HILPupCqiDBQTRZSA8= X-Received: by 2002:a05:6902:709:b0:663:6108:777c with SMTP id k9-20020a056902070900b006636108777cmr15666556ybt.279.1654617838612; Tue, 07 Jun 2022 09:03:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwGM8IPbkvFC8Z9jnk8uWei0u+1XItpb1+NKN9rot6pcp/Scpi5Jsl36E41zfsunLXpQfrdvFioffvXxR5bZWo= X-Received: by 2002:a05:6902:709:b0:663:6108:777c with SMTP id k9-20020a056902070900b006636108777cmr15666515ybt.279.1654617838142; Tue, 07 Jun 2022 09:03:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jeff Johnston Date: Tue, 7 Jun 2022 12:03:47 -0400 Message-ID: Subject: Re: Malloc: unusable area at the end of the heap section To: Torbjorn SVENSSON Cc: Jerome Leroux , "newlib@sourceware.org" Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jjohnstn@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-5.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, HTML_MESSAGE, KAM_LOTSOFHASH, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: newlib@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Newlib mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2022 16:04:11 -0000 Hi Torbjorn, I think it would be useful. Do you want to modify the patch to be more generic? I am thinking of a var set in configure.host that sets a compile flag at the end such as _USE_SYSCONF_FOR_PAGESIZE. Then, the default sysconf.c can be put in the newlib/libc/unix directory. It is then a straight-forward exercise to add this as an enablement configuration option. What do you think? -- Jeff J. On Tue, Jun 7, 2022 at 2:18 AM Torbjorn SVENSSON via Newlib < newlib@sourceware.org> wrote: > Hello, > > A while back, I provided a patch[1] to newlib that would allow the > application to override the pagesize for malloc, but the patch got stalle= d. > Maybe this would be a good time to take another look at the patch and see > if it would actually fix the generic newlib usage or if it's still > something that is only applicable for small embedded targets. > > [1] https://ecos.sourceware.org/ml/newlib/current/017616.html > > Kind regards, > Torbj=C3=B6rn > > > -----Original Message----- > > From: Newlib > bounces+torbjorn.svensson=3Dst.com@sourceware.org> On Behalf Of Jerome > > Leroux > > Sent: den 6 juni 2022 23:18 > > To: newlib@sourceware.org > > Subject: Malloc: unusable area at the end of the heap section > > > > Hello Newlib developers, > > > > I am a user of Newlib in a project that runs on an NXP MCU. > > I am using MCUXpressoIDE_11.3.0_5180_prc3, which comes with GCC =E2=80= =9Carm- > > none-eabi-gcc.exe (GNU Arm Embedded Toolchain > > 9-2020-q2-update) 9.3.1 20200408 (release)=E2=80=9D and Newlib 3.3.0. > > > > I have identified an issue in malloc, and I think the problem is still > present in > > the latest version of Newlib. I could > > not see any changes in the incriminated code since Newlib 3.3.0. > > > > I noticed this issue only in the standard malloc implementation and not > in the > > nano-malloc version. > > > > Here is a description of the problem: > > The allocator splits the heap into pages. When a page is full, it > increases the > > heap size by reserving a new page in the > > heap section. When reserving a new page, the allocator keeps the page e= nd > > address aligned with malloc_getpagesize, which > > is set to 4096 by default. If there is not enough space to reserve the > full page, > > the allocation fails even if there is > > enough space in the heap to allocate the chunk of memory. > > Because the issue is related to the heap end address and how the linker > > positions the heap, the same sequence of > > allocations may lead to different results (failure or success) dependin= g > on the > > location of the heap, even if the heap > > size is constant. Typically, adding a new C global variable can shift > the start > > address of the heap section and cause a > > malloc error. > > > > For example, with a heap section of 4096 bytes (0x1000 bytes): > > If the heap section address is 0x20100-0x21100, during the > initialization, the > > page end address is set to 0x21000 > > (aligned on 4096). We will be able to allocate until the address > 0x21000. After > > that, the allocator will try to reserve > > a new page, but it will fail because it won=E2=80=99t be able to reserv= e a 4096 > bytes > > page from 0x21000 to 0x22000. The > > following allocations will fail. The usable heap size is 3840 bytes > (0x21000 - > > 0x20100) instead of 4096. > > If the heap section address is 0x20F00-0x21F00 (same size), with the sa= me > > scenario, the usable heap size is 256 bytes > > (0x21000 - 0x20F00). > > Here are two examples of heap configurations: > > https://gist.github.com/jerome- > > leroux/759159fbd3e7bb5e189dbceb04636914?permalink_comment_id=3D4191 > > 266#gistcomment-4191266 > > > > I did not dig into the implementation so much. From my understanding, t= he > > problem comes from the usage of > > "malloc_getpagesize" (see > > https://github.com/bminor/newlib/blob/830a9b707caa5e343b6ffce7fcb2d3c > > a97e3259c/newlib/libc/stdlib/_mallocr.c#L198) in > > "malloc_extend_top" (probably here > > https://github.com/bminor/newlib/blob/830a9b707caa5e343b6ffce7fcb2d3c > > a97e3259c/newlib/libc/stdlib/_mallocr.c#L2166). > > I can understand it makes sense to keep the pages aligned when running > in a > > system that implements virtual memory. > > Still, on an MCU, the heap is just a contiguous chunk of memory > allocated at > > link time. Furthermore, the heap size is > > usually pretty small (a few kilobytes), so potentially wasting 4 KB of > memory > > is unacceptable. Using the default > > implementation of "sbrk" documented at > > https://sourceware.org/newlib/libc.html#index-sbrk will lead to the > > problem. > > > > I have written a simple example that demonstrates the issue (see > > https://gist.github.com/jerome- > > leroux/759159fbd3e7bb5e189dbceb04636914 ). To reproduce the problem, > > define the macros > > HEAP_SECTION_START_SYMBOL and HEAP_SECTION_END_SYMBOL, which > > are specific to your environment. Then call the function > > "test_malloc()". > > > > I tried to find someone with the same issue, but I couldn=E2=80=99t. Th= e related > > commits/discussions I found are: > > - > > https://github.com/bminor/newlib/commit/4a3d0a5a5d829c05868a34658eb > > 45731dbb5112b > > - https://stackoverflow.com/questions/39088598/malloc-in-newlib-does-it= - > > waste-memory-after-one-big-failure-allocation > > > > Can anyone confirm what I have noticed? > > > > Thanks. > > > > -- > > Jerome Leroux > >