From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa2.mentor.iphmx.com (esa2.mentor.iphmx.com [68.232.141.98]) by sourceware.org (Postfix) with ESMTPS id 858F43858D28 for ; Fri, 10 Feb 2023 15:11:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 858F43858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com X-IronPort-AV: E=Sophos;i="5.97,287,1669104000"; d="scan'208";a="97675573" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa2.mentor.iphmx.com with ESMTP; 10 Feb 2023 07:11:32 -0800 IronPort-SDR: NsfkaoDFQe4pR6BwsDB9SlnVWGIRiT6WggoAzOFrYJimCZBXOYbFyWzDZLoyEOsX1pXkC3bl64 M25VC3n9hsRhAO22VZVhCddzuC/Q0iMG39bBnTqTUXdZoIOdH4xwVL15MqSFTVvOe/KkBtlbTj uegPGwFuuREZbvLLObuE+KsgooN+LzMuq840LbNX+u6FCAXo0oN7hDHjVOuz60BJF85sF6gsMA fXDoRmEEjClRGaRPm/SkOHaEZXDCKo69gddiTl36ehcZ3NmHkWZBZomTauUAmuZyxXdIb1E9Rm iYM= From: Thomas Schwinge To: Andrew Stubbs , Jakub Jelinek , Tobias Burnus CC: Subject: Re: [PATCH] libgomp, openmp: pinned memory In-Reply-To: References: <20220104155558.GG2646553@tucnak> <48ee767a-0d90-53b4-ea54-9deba9edd805@codesourcery.com> <20220104182829.GK2646553@tucnak> <20220104184740.GL2646553@tucnak> User-Agent: Notmuch/0.29.3+94~g74c3f1b (https://notmuchmail.org) Emacs/28.2 (x86_64-pc-linux-gnu) Date: Fri, 10 Feb 2023 16:11:23 +0100 Message-ID: <87wn4pzhb8.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-12.mgc.mentorg.com (139.181.222.12) To svr-ies-mbx-10.mgc.mentorg.com (139.181.222.10) X-Spam-Status: No, score=-5.9 required=5.0 tests=BAYES_00,HEADER_FROM_DIFFERENT_DOMAINS,KAM_DMARC_STATUS,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi! Re OpenMP 'pinned' memory allocator trait semantics vs. 'omp_realloc': On 2022-01-13T13:53:03+0000, Andrew Stubbs wrote: > On 05/01/2022 17:07, Andrew Stubbs wrote: >> [...], I'm working on an implementation using mmap instead of malloc >> for pinned allocations. [...] > This means that large allocations will now be page aligned and therefore > pin the smallest number of pages for the size requested, and that that > memory will be unpinned automatically when freed via munmap, or moved > via mremap. > --- /dev/null > +++ b/libgomp/config/linux/allocator.c > +static void * > +linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, > + size_t oldsize, size_t size, int oldpin, int pin) > +{ > + if (oldpin && pin) > + { > + void *newaddr =3D mremap (addr, oldsize, size, MREMAP_MAYMOVE); > + if (newaddr =3D=3D MAP_FAILED) > + return NULL; > + > + return newaddr; > + } > + else if (oldpin || pin) > + { > + void *newaddr =3D linux_memspace_alloc (memspace, size, pin); > + if (newaddr) > + { > + memcpy (newaddr, addr, oldsize < size ? oldsize : size); > + linux_memspace_free (memspace, addr, oldsize, oldpin); > + } > + > + return newaddr; > + } > + else > + return realloc (addr, size); > +} I did wonder if 'mremap' with 'MREMAP_MAYMOVE' is really acceptable here, given OpenMP 5.2, 6.2 "Memory Allocators": "Allocators with the 'pinned' trait defined to be 'true' ensure that their allocations remain in the same storage resource at the same location for their entire lifetime." I'd have read into this that 'realloc' may shrink or enlarge the region (unless even that considered faulty), but the region must not be moved ("same location"), thus no 'MREMAP_MAYMOVE'; see 'man 2 mremap' (2019-03-06): 'MREMAP_MAYMOVE' By default, if there is not sufficient space to expand a mapping a= t its current location, then 'mremap()' fails. If this flag is specified, = then the kernel is permitted to relocate the mapping to a new virtual addre= ss, if necessary. If the mapping is relocated, then absolute pointers into= the old mapping location become invalid (offsets relative to the starting = address of the mapping should be employed). ..., but then I saw that OpenMP 5.2, 18.13.9 'omp_realloc' is specified such that it isn't expected to 'realloc' in-place, but rather it "deallocates previously allocated memory and requests a memory allocation", which I understand that it does end a "lifetime" and then establish a new "lifetime", which means that 'MREMAP_MAYMOVE' in fact is fine (as implemented)? Further I read in 'man 2 mremap' (2019-03-06): If the memory segment specified by *old_address* and *old_size* is lo= cked (using 'mlock(2)' or similar), then this lock is maintained when the s= egment is resized and/or relocated. As a consequence, the amount of memory= locked by the process may change. (The current proposed code evidently does make use of that; OK.) But then in 'NOTES' I read: If 'mremap()' is used to move or expand an area locked with 'mlock(2)' = or equivalent, the 'mremap()' call will make a best effort to populate the = new area but will not fail with 'ENOMEM' if the area cannot be populated. What exactly is that supposed to tell us: "will make a best effort [...] but will not fail"? Isn't that in conflict with the earlier statement? So can we rely on 'mremap' together with 'mlock' or not? (This topic remains valid even if we follow through the idea of using CUDA to register page-locked memory, because that's not available in all configurations, and we then still want to do the 'mmap'/'mlock' thing, I suppose.) Gr=C3=BC=C3=9Fe Thomas ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstra=C3=9Fe 201= , 80634 M=C3=BCnchen; Gesellschaft mit beschr=C3=A4nkter Haftung; Gesch=C3= =A4ftsf=C3=BChrer: Thomas Heurung, Frank Th=C3=BCrauf; Sitz der Gesellschaf= t: M=C3=BCnchen; Registergericht M=C3=BCnchen, HRB 106955