From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 766323851174 for ; Wed, 29 Jun 2022 10:45:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 766323851174 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-484-lI9E9w2kNI2aqU5ZzY6a1Q-1; Wed, 29 Jun 2022 06:45:40 -0400 X-MC-Unique: lI9E9w2kNI2aqU5ZzY6a1Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5126185A7A4; Wed, 29 Jun 2022 10:45:39 +0000 (UTC) Received: from tucnak.zalov.cz (unknown [10.39.192.30]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8555B1410DD8; Wed, 29 Jun 2022 10:45:39 +0000 (UTC) Received: from tucnak.zalov.cz (localhost [127.0.0.1]) by tucnak.zalov.cz (8.17.1/8.17.1) with ESMTPS id 25TAjaXR3997460 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 29 Jun 2022 12:45:37 +0200 Received: (from jakub@localhost) by tucnak.zalov.cz (8.17.1/8.17.1/Submit) id 25TAjZSl3997459; Wed, 29 Jun 2022 12:45:35 +0200 Date: Wed, 29 Jun 2022 12:45:35 +0200 From: Jakub Jelinek To: Andrew Stubbs Cc: gcc-patches@gcc.gnu.org Subject: Re: [committed] openmp: Add support for HBW or large capacity or interleaved memory through the libmemkind.so library Message-ID: Reply-To: Jakub Jelinek References: MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jun 2022 10:45:45 -0000 On Tue, Jun 28, 2022 at 10:29:53PM +0100, Andrew Stubbs wrote: > On 09/06/2022 09:19, Jakub Jelinek via Gcc-patches wrote: > > + switch (memspace) > > + { > > + case omp_high_bw_mem_space: > > +#ifdef LIBGOMP_USE_MEMKIND > > + struct gomp_memkind_data *memkind_data; > > + memkind_data = gomp_get_memkind (); > > + if (data.partition == omp_atv_interleaved > > + && memkind_data->kinds[GOMP_MEMKIND_HBW_INTERLEAVE]) > > + { > > + data.memkind = GOMP_MEMKIND_HBW_INTERLEAVE; > > + break; > > + } > > + else if (memkind_data->kinds[GOMP_MEMKIND_HBW_PREFERRED]) > > + { > > + data.memkind = GOMP_MEMKIND_HBW_PREFERRED; > > + break; > > + } > > +#endif > > + return omp_null_allocator; > > + case omp_large_cap_mem_space: > > +#ifdef LIBGOMP_USE_MEMKIND > > + memkind_data = gomp_get_memkind (); > > + if (memkind_data->kinds[GOMP_MEMKIND_DAX_KMEM_ALL]) > > + data.memkind = GOMP_MEMKIND_DAX_KMEM_ALL; > > + else if (memkind_data->kinds[GOMP_MEMKIND_DAX_KMEM]) > > + data.memkind = GOMP_MEMKIND_DAX_KMEM; > > +#endif > > + break; > > + default: > > +#ifdef LIBGOMP_USE_MEMKIND > > + if (data.partition == omp_atv_interleaved) > > + { > > + memkind_data = gomp_get_memkind (); > > + if (memkind_data->kinds[GOMP_MEMKIND_INTERLEAVE]) > > + data.memkind = GOMP_MEMKIND_INTERLEAVE; > > + } > > +#endif > > + break; > > + } > > + > > This conflicts with mine and Abid's patches to implement the device > allocators and host unified shared memory via the overridable > "MEMSPACE_ALLOC" hooks. I can still use those for the "else" case, but > they'll be inactive on any configuration where libmemkind exists. That's > fine for the device code, and may be OK for USM (given that libmemkind won't > have an option for that). There's a problem for the NVidia-specific > host-memory pinning I have planned though. > > How do you propose we resolve this conflict, please? > > Ideally it will be in such a way that the conditional is resolved at compile > time and the routine can be inlined (so no fake-OO function pointers in > structs, I think). memkind isn't used just because it exists, but because it supports some requested trait that isn't otherwise supported. Using callbacks instead of selecting the allocator to call using ifs is possible if all those callbacks have the same arguments or if we add enough dummy arguments to the callbacks such that some wrappers can handle it all. Right now, memkind is used if ->memkind in the allocator data isn't GOMP_MEMKIND_NONE, so when it is, you can certainly call some callback (performance question might be whether to call it in the else case always using callback or whether to call malloc etc. directly if callback is NULL). And omp_init_allocator needs to decide what to do if one asks for features that need memkind as well as for features that need whatever you/Abid have been working on. A possible resolution is punt (return omp_null_allocator), or prefer one feature over the other one or vice versa. >From the features currently handled by memkind, even before my changes for consistency with libomp from llvm omp_high_bw_mem_space was considered a hard request, ditto omp_atk_pinned omp_atv_true, but the rest was just taken as optimization hint. Jakub