From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id A4170385AE5E for ; Tue, 11 Oct 2022 10:49:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A4170385AE5E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665485349; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references; bh=duAvuZc0GI7siSr4P/e9weMkJghBpya1wBJ30Ar0Cmo=; b=TONqobGd4FGtVA2xgIpZEe5zkYsSWT5K2R1lWJAq8OlUuSUt8OzQDsExH7FHPdql4/039q NWtGGWwqWhrmllsdqIOCDijW5h3fcmgHhz0lUqe0TbUoicaz0QiYobWQRIrrWb1QMoCYYX 7y7K62pihKLMF6sphW0TcuAQl6xJp4s= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-609-EifKmg3GPKak4R0n_IutCw-1; Tue, 11 Oct 2022 06:49:08 -0400 X-MC-Unique: EifKmg3GPKak4R0n_IutCw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D27CE1C05AAD; Tue, 11 Oct 2022 10:49:07 +0000 (UTC) Received: from tucnak.zalov.cz (unknown [10.39.192.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 70272207AC34; Tue, 11 Oct 2022 10:49:07 +0000 (UTC) Received: from tucnak.zalov.cz (localhost [127.0.0.1]) by tucnak.zalov.cz (8.17.1/8.17.1) with ESMTPS id 29BAn2Sk1581474 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Tue, 11 Oct 2022 12:49:03 +0200 Received: (from jakub@localhost) by tucnak.zalov.cz (8.17.1/8.17.1/Submit) id 29BAn0QB1581313; Tue, 11 Oct 2022 12:49:01 +0200 Date: Tue, 11 Oct 2022 12:49:00 +0200 From: Jakub Jelinek To: Tobias Burnus Cc: gcc-patches , Alexander Monakov , Tom de Vries Subject: Re: [Patch][v5] libgomp/nvptx: Prepare for reverse-offload callback handling Message-ID: Reply-To: Jakub Jelinek References: <57b3ae5e-8f15-8bea-fa09-39bccbaa2414@codesourcery.com> <3f0fc49f-b07f-bee2-51a8-a5d03f1c33ed@codesourcery.com> <30e3ed49-0d14-8015-57ef-3d70b1dea69a@codesourcery.com> <3ebce406-46e4-8f98-8c53-83b61423644e@codesourcery.com> <798d7ee1-2ffa-a591-38cb-a9ad421265d0@codesourcery.com> MIME-Version: 1.0 In-Reply-To: <798d7ee1-2ffa-a591-38cb-a9ad421265d0@codesourcery.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,KAM_SHORT,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Fri, Oct 07, 2022 at 04:26:58PM +0200, Tobias Burnus wrote: > libgomp/nvptx: Prepare for reverse-offload callback handling > > This patch adds a stub 'gomp_target_rev' in the host's target.c, which will > later handle the reverse offload. > For nvptx, it adds support for forwarding the offload gomp_target_ext call > to the host by setting values in a struct on the device and querying it on > the host - invoking gomp_target_rev on the result. > > For host-device consistency guarantee reasons, reverse offload is currently > limited -march=sm_70 (for libgomp). > > gcc/ChangeLog: > > * config/nvptx/mkoffload.cc (process): Warn if the linked-in libgomp.a > has not been compiled with sm_70 or higher and disable code gen then. > > include/ChangeLog: > > * cuda/cuda.h (enum CUdevice_attribute): Add > CU_DEVICE_ATTRIBUTE_UNIFIED_ADDRESSING. > (CU_MEMHOSTALLOC_DEVICEMAP): Define. > (cuMemHostAlloc): Add prototype. > > libgomp/ChangeLog: > > * config/nvptx/icv-device.c (GOMP_DEVICE_NUM_VAR): Remove > 'static' for this variable. > * config/nvptx/libgomp-nvptx.h: New file. > * config/nvptx/target.c: Include it. > (GOMP_ADDITIONAL_ICVS): Declare extern var. > (GOMP_REV_OFFLOAD_VAR): Declare var. > (GOMP_target_ext): Handle reverse offload. > * libgomp-plugin.h (GOMP_PLUGIN_target_rev): New prototype. > * libgomp-plugin.c (GOMP_PLUGIN_target_rev): New, call ... > * target.c (gomp_target_rev): ... this new stub function. > * libgomp.h (gomp_target_rev): Declare. > * libgomp.map (GOMP_PLUGIN_1.4): New; add GOMP_PLUGIN_target_rev. > * plugin/cuda-lib.def (cuMemHostAlloc): Add. > * plugin/plugin-nvptx.c: Include libgomp-nvptx.h. > (struct ptx_device): Add rev_data member. > (nvptx_open_device): #if 0 unused check; add > unified address assert check. > (GOMP_OFFLOAD_get_num_devices): Claim unified address > support. > (GOMP_OFFLOAD_load_image): Free rev_fn_table if no > offload functions exist. Make offload var available > on host and device. > (rev_off_dev_to_host_cpy, rev_off_host_to_dev_cpy): New. > (GOMP_OFFLOAD_run): Handle reverse offload. So, does this mean one has to have gcc configured --with-arch=sm_70 or later to make reverse offloading work (and then on the other side no support for older PTX arches at all)? If yes, I was kind of hoping we could arrange for it to be more user-friendly, build libgomp.a normally (sm_35 or what is the default), build the single TU in libgomp that needs the sm_70 stuff with -march=sm_70 and arrange for mkoffload to link in the sm_70 stuff only if the user wants reverse offload (or has requires reverse_offload?). In that case ignore sm_60 and older devices, if reverse offload isn't wanted, don't link in the part that needs sm_70 and make stuff working on sm_35 and later. Or perhaps have 2 versions of target.o, one sm_35 and one sm_70 and let mkoffload choose among them. > + /* The code for nvptx for GOMP_target_ext in libgomp/config/nvptx/target.c > + for < sm_70 exists but is disabled here as it is unclear whether there > + is the required consistency between host and device. > + See https://gcc.gnu.org/pipermail/gcc-patches/2022-October/602715.html > + for details. */ > + warning_at (input_location, 0, > + "Disabling offload-code generation for this device type: " > + "% can only be fulfilled " > + "for % or higher"); > + inform (UNKNOWN_LOCATION, > + "Reverse offload requires that GCC is configured with " > + "%<--with-arch=sm_70%> or higher and not overridden by a lower " > + "value for %<-foffload-options=nvptx-none=-march=%>"); Diagnostics (sure, Fortran FE is an exception) shouldn't start with capital letters). > @@ -519,10 +523,20 @@ nvptx_open_device (int n) > CU_DEVICE_ATTRIBUTE_MAX_THREADS_PER_MULTIPROCESSOR, dev); > ptx_dev->max_threads_per_multiprocessor = pi; > > +#if 0 > + int async_engines; > r = CUDA_CALL_NOCHECK (cuDeviceGetAttribute, &async_engines, > CU_DEVICE_ATTRIBUTE_ASYNC_ENGINE_COUNT, dev); > if (r != CUDA_SUCCESS) > async_engines = 1; > +#endif Please avoid #if 0 code. > + > + /* Required below for reverse offload as implemented, but with compute > + capability >= 2.0 and 64bit device processes, this should be universally be > + the case; hence, an assert. */ > + r = CUDA_CALL_NOCHECK (cuDeviceGetAttribute, &pi, > + CU_DEVICE_ATTRIBUTE_UNIFIED_ADDRESSING, dev); > + assert (r == CUDA_SUCCESS && pi); > > for (int i = 0; i != GOMP_DIM_MAX; i++) > ptx_dev->default_dims[i] = 0; > @@ -1179,8 +1193,10 @@ GOMP_OFFLOAD_get_num_devices (unsigned int omp_requires_mask) > { > int num_devices = nvptx_get_num_devices (); > /* Return -1 if no omp_requires_mask cannot be fulfilled but > - devices were present. */ > - if (num_devices > 0 && omp_requires_mask != 0) > + devices were present. Unified-shared address: see comment in 2 spaces after . rather than 1. > --- a/libgomp/target.c > +++ b/libgomp/target.c > @@ -2925,6 +2925,25 @@ GOMP_target_ext (int device, void (*fn) (void *), size_t mapnum, > htab_free (refcount_set); > } > > +/* Handle reverse offload. This is called by the device plugins for a > + reverse offload; it is not called if the outer target runs on the host. */ Likewise. Jakub