From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com [68.232.129.153]) by sourceware.org (Postfix) with ESMTPS id 87802384F00E for ; Mon, 13 Feb 2023 14:38:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 87802384F00E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="100576220" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa1.mentor.iphmx.com with ESMTP; 13 Feb 2023 06:38:37 -0800 IronPort-SDR: VXT76MQBOMXZ4RkW/NMKnPyb+fUBOocbrOkxrod0LkFxJNo4SjuvllLV3uc8yqMmZEzTiiB95C uSKlv+mLk+DS2RkIQAj51Ury2gDxP/mrI+C5VDqHwg/xJz80e4hAlic/TiezkTxR+xkawsTbkp SvesYNABoxW1jtBZqoObNzbNJqXq4/dKmggJ/6ukhTOb3uhAb3C+Sd8g6L/nGCT+cz6ZlT92bW A2ST1CM8MJvszL/fLU3g+dPOadcI5JR0ysZ1/HarLnvl7KysaYpDBJeGfNyHcS1AvCQOcyaOy2 lkk= From: Thomas Schwinge To: Andrew Stubbs , Hafiz Abid Qadeer , Jakub Jelinek , Tobias Burnus CC: Subject: -foffload-memory=pinned (was: [PATCH 1/5] openmp: Add -foffload-memory) In-Reply-To: <20220308113059.688551-2-abidh@codesourcery.com> References: <20220308113059.688551-1-abidh@codesourcery.com> <20220308113059.688551-2-abidh@codesourcery.com> User-Agent: Notmuch/0.29.3+94~g74c3f1b (https://notmuchmail.org) Emacs/28.2 (x86_64-pc-linux-gnu) Date: Mon, 13 Feb 2023 15:38:29 +0100 Message-ID: <87sff9zl3u.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-12.mgc.mentorg.com (139.181.222.12) To svr-ies-mbx-10.mgc.mentorg.com (139.181.222.10) X-Spam-Status: No, score=-5.9 required=5.0 tests=BAYES_00,HEADER_FROM_DIFFERENT_DOMAINS,KAM_DMARC_STATUS,SPF_HELO_PASS,SPF_PASS,TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi! On 2022-03-08T11:30:55+0000, Hafiz Abid Qadeer wro= te: > From: Andrew Stubbs > > Add a new option. It will be used in follow-up patches. > --- a/gcc/doc/invoke.texi > +++ b/gcc/doc/invoke.texi > +@option{-foffload-memory=3Dpinned} forces all host memory to be pinned (= this > +mode may require the user to increase the ulimit setting for locked memo= ry). So, this is currently implemented via 'mlockall', which, as discussed, (a) has issues ('ulimit -l'), and (b) doesn't actually achieve what it meant to achieve (because it doesn't register the page-locked memory with the GPU driver). So one idea was to re-purpose the unified shared memory 'gcc/omp-low.cc:pass_usm_transform' (compiler pass that "changes calls to malloc/free/calloc/realloc and operator new to memory allocation functions in libgomp with allocator=3Dompx_unified_shared_mem_alloc"), . (I have not yet looked into that in detail.) Here's now a different idea. As '-foffload-memory=3Dpinned', per the name of the option, concerns itself with memory used in offloading but not host execution generally, why are we actually attempting to "[force] all host memory to be pinned" -- why not just the memory that's being used with offloading? That is, if '-foffload-memory=3Dpinned' is set, register as page-locked with the GPU driver all memory that appears in OMP offloading data regions, such as OpenMP 'target' 'map' clauses etc. That way, this is directed at the offloading data transfers, as itended, but at the same time we don't "waste" page-locked memory for generic host memory allocations. What do you think -- you, who've spent a lot more time on this topic than I have, so it's likely possible that I fail to realize some "details"? Gr=C3=BC=C3=9Fe Thomas ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstra=C3=9Fe 201= , 80634 M=C3=BCnchen; Gesellschaft mit beschr=C3=A4nkter Haftung; Gesch=C3= =A4ftsf=C3=BChrer: Thomas Heurung, Frank Th=C3=BCrauf; Sitz der Gesellschaf= t: M=C3=BCnchen; Registergericht M=C3=BCnchen, HRB 106955