From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 2AFEB383DBAA for ; Thu, 9 Jun 2022 11:58:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 2AFEB383DBAA Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-360-xlJ4NcsgP0ODALLUf3H6JA-1; Thu, 09 Jun 2022 07:57:59 -0400 X-MC-Unique: xlJ4NcsgP0ODALLUf3H6JA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 510943C01D97; Thu, 9 Jun 2022 11:57:59 +0000 (UTC) Received: from tucnak.zalov.cz (unknown [10.39.192.11]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 07BA140CFD0A; Thu, 9 Jun 2022 11:57:58 +0000 (UTC) Received: from tucnak.zalov.cz (localhost [127.0.0.1]) by tucnak.zalov.cz (8.17.1/8.17.1) with ESMTPS id 259BvqTf1627240 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Thu, 9 Jun 2022 13:57:53 +0200 Received: (from jakub@localhost) by tucnak.zalov.cz (8.17.1/8.17.1/Submit) id 259BvqW11627239; Thu, 9 Jun 2022 13:57:52 +0200 Date: Thu, 9 Jun 2022 13:57:52 +0200 From: Jakub Jelinek To: Thomas Schwinge Cc: gcc-patches@gcc.gnu.org Subject: Re: [committed] openmp: Add support for HBW or large capacity or interleaved memory through the libmemkind.so library Message-ID: Reply-To: Jakub Jelinek References: <87a6am5epb.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 In-Reply-To: <87a6am5epb.fsf@euler.schwinge.homeip.net> X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jun 2022 11:58:06 -0000 On Thu, Jun 09, 2022 at 12:11:28PM +0200, Thomas Schwinge wrote: > On 2022-06-09T10:19:03+0200, Jakub Jelinek via Gcc-patches wrote: > > This patch adds support for dlopening libmemkind.so > > Instead of 'dlopen'ing literally 'libmemkind.so': > > > --- libgomp/allocator.c.jj 2022-06-08 08:21:03.099446883 +0200 > > +++ libgomp/allocator.c 2022-06-08 13:41:45.647133610 +0200 > > > + void *handle = dlopen ("libmemkind.so", RTLD_LAZY); > > ..., shouldn't this instead 'dlopen' 'libmemkind.so.0'? At least for > Debian/Ubuntu, the latter ('libmemkind.so.0') is shipped in the "library" > package: I agree and I've actually noticed it too right before committing, but I thought I'll investigate and tweak incrementally because "libmemkind.so" is what I've actually tested (it is what llvm libomp uses). > > $ apt-file list libmemkind0 | grep -F libmemkind.so > libmemkind0: /usr/lib/x86_64-linux-gnu/libmemkind.so.0 > libmemkind0: /usr/lib/x86_64-linux-gnu/libmemkind.so.0.0.1 > > ..., but the former ('libmemkind.so') only in the "development" package: > > $ apt-file list libmemkind-dev | grep -F libmemkind.so > libmemkind-dev: /usr/lib/x86_64-linux-gnu/libmemkind.so > > ..., which users of GCC/libgomp shouldn't have to care about. Similarly in Fedora memkind package provides just /usr/lib64/libautohbw.so.0 /usr/lib64/libautohbw.so.0.0.0 /usr/lib64/libmemkind.so.0 /usr/lib64/libmemkind.so.0.0.1 /usr/lib64/libmemtier.so.0 /usr/lib64/libmemtier.so.0.0.0 and /usr/lib64/libautohbw.so /usr/lib64/libmemkind.so /usr/lib64/libmemtier.so comes from memkind-devel. > Any plans about test cases for this? (Not trivial, I suppose?) That is the hard part. All the testing I've done so far were for atv_interleaved: #include int main () { omp_alloctrait_t traits[3] = { { omp_atk_alignment, 64 }, { omp_atk_fallback, omp_atv_null_fb }, { omp_atk_partition, omp_atv_interleaved } }; omp_allocator_handle_t a; a = omp_init_allocator (omp_default_mem_space, 3, traits); if (a == omp_null_allocator) return 1; void *p = omp_alloc (128, a); if (!p) return 2; void *q = omp_realloc (p, 256, a, a); if (!q) return 3; void *r = omp_calloc (1, 512, a); if (!r) return 4; omp_free (q, a); omp_free (r, a); return 0; } because that is something that works even on my devel WS, though in the testcase one doesn't figure out if memkind was actually available and whether the memory was indeed interleaved or not, just that it works (I could certainly also store some data and read them back after realloc, and also test one without omp_atk_alignment which effectively prevents memkind_realloc from being called and uses allocation + deallocation), but that is it. I've actually stepped through in the debugger to verify memkind_* is called... Now for HBW memory, some googling around and brief look at the memkind source shows that it probably supports just Intel Xeon Phi HBW memory, I'm waiting for access to such a box right now but it might take a few days. For the DAX stuff, I admit I don't know what it exactly is (what kind of hw it needs). > > --- libgomp/config/linux/allocator.c.jj 2022-06-08 08:58:23.197078191 +0200 > > +++ libgomp/config/linux/allocator.c 2022-06-08 09:39:15.108410730 +0200 > > @@ -0,0 +1,36 @@ > > > +#define _GNU_SOURCE > > +#include "libgomp.h" > > +#if defined(PLUGIN_SUPPORT) && defined(LIBGOMP_USE_PTHREADS) > > +#define LIBGOMP_USE_MEMKIND > > +#endif > > + > > +#include "../../../allocator.c" > > Given this use of 'PLUGIN_SUPPORT' (and thus 'dlopen' etc.) for something > different than libgomp plugins (offloading), might move 'DL_LIBS', > 'PLUGIN_SUPPORT' from 'libgomp/plugin/configfrag.ac' into > 'libgomp/configure.ac', and 'libgomp_la_LIBADD += $(DL_LIBS)' from > 'libgomp/plugin/Makefrag.am' into 'libgomp/Makefile.am'. Maybe, but libgomp/plugin/configfrag.ac is included unconditionally and the memkind support is some kind of plugin too, just not offloading plugin, but allocator plugin... Didn't want to spend too much time on it and PLUGIN_SUPPORT is right now solely about dlsym exists and -ldl works and has been added. Jakub