From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa3.mentor.iphmx.com (esa3.mentor.iphmx.com [68.232.137.180]) by sourceware.org (Postfix) with ESMTPS id 76004383C7C3 for ; Thu, 8 Dec 2022 12:52:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 76004383C7C3 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com X-IronPort-AV: E=Sophos;i="5.96,227,1665475200"; d="scan'208";a="88826575" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa3.mentor.iphmx.com with ESMTP; 08 Dec 2022 04:52:21 -0800 IronPort-SDR: 2w57cJizHIhAoX/w4UEKlLOOi39dEl+bNIjeM/1DHj8hDCs44R4bzW9w2NxLblPJgS8lUZlqsJ eUpJj9JuatcAk8TKghOtPjZG1sLIUHd/6tk391Ho3UIwIFXbndeqdBC1pwDolSq2jbvBNOXtPD zOYY735F0yjF/Ph52hVz/rfqapWC2Zvf6SU2kxne2pHHw/k+0w49WqODch+4P4SKderXbQyxvX IVplUyQ5YaZSxV4UrEnw+MQz8uYhKcGMlzVhV34FLYJc1cbgFj2u3lFK65GKD7yukO850rCUKK Dko= Message-ID: Date: Thu, 8 Dec 2022 12:51:34 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.5.1 Subject: Re: [PATCH 02/17] libgomp: pinned memory To: Jakub Jelinek , Thomas Schwinge CC: References: Content-Language: en-GB From: Andrew Stubbs In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-10.mgc.mentorg.com (139.181.222.10) To svr-ies-mbx-11.mgc.mentorg.com (139.181.222.11) X-Spam-Status: No, score=-6.6 required=5.0 tests=BAYES_00,HEADER_FROM_DIFFERENT_DOMAINS,KAM_DMARC_STATUS,NICE_REPLY_A,SPF_HELO_PASS,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 08/12/2022 12:11, Jakub Jelinek wrote: > On Thu, Jul 07, 2022 at 11:34:33AM +0100, Andrew Stubbs wrote: >> >> Implement the OpenMP pinned memory trait on Linux hosts using the mlock >> syscall. Pinned allocations are performed using mmap, not malloc, to ensure >> that they can be unpinned safely when freed. > > As I said before, I think the pinned memory is too precious to waste it this > way, we should handle the -> pinned case through memkind_create_fixed on > mmap + mlock area, that way we can create even quite small pinned > allocations. This has been delayed due to other priorities, but our current plan is to switch to using cudaHostAlloc, when available, but we can certainly use memkind_create_fixed for the fallback case (including amdgcn). Using Cuda might be trickier to implement because there's a layering violation inherent in routing target independent allocations through the nvptx plugin, but benchmarking shows that that's the only way to get the faster path through the Cuda black box; being pinned is good because it avoids page faults, but apparently if Cuda *knows* it is pinned then you get a speed boost even when there would be *no* faults (i.e. on a quiet machine). Additionally, Cuda somehow ignores the OS-defining limits. Thomas Schwinge has been assigned this task and will be getting to it soonish. Andrew