From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2056.outbound.protection.outlook.com [40.107.215.56]) by sourceware.org (Postfix) with ESMTPS id 101563858C39 for ; Fri, 13 Jan 2023 13:17:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 101563858C39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=siemens.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BWswtfPtcAeNPNSWl8NcLqNj0R6PcN89rD+9/JLPB4X4y1eQhj6EcH9Ym9PT9Rt7+/iJUHUu+E+5VlxAmi6iyb42UEOmmMs6v8M82T5DyMxClr4b4AvULCx0cbRpGwKoX+N+prriniG2wfuMpJ5rBLtnG6I5tafHn+GGO2W83krHeVLu/eI7VOpsC10sg+oIIl62Ydj5lACTHZNGs9+bZnKyUqiZU0FhQWNjhQDlB0TVOvdETcwSSqh6//KRFC4cqMyc68lTLj7vxZmaqCIawFXTiLYV93fQ8sf341ki1YCy5a+qAaUJw3Zvh4NG8YX1k+FXtvF1F3CgGrYHs8iQUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=N2M4T1icpExc5wz7Vv3Ei3ub4ZtCXJzHMBtB8Kb76Kg=; b=h6L8HXR8QXf+8xPr3Efnm1DHtNW6R/91F1pEGSR0kgQgbP3q8kQlXmDg3AOMsXQms1vLG3trTxeV1sPxBJRNkz3mL9zW4WHTUWdOg/5hsCG4i1WFKqs2/j5ovlSHNNVi49qvq7l8+NnfHvtQk9qv69XWcqb8QHB//Bn2mAyVU7ZciEXStE1MG/T7Xty5sadWLcnqmiA2Rk095mDvytR0T0kdaOKskkGRQyB3Ghk5f3V2EM3mQ8JlaSXMDX53kcYQ7TY7JfjKGwgNmpcAx0zFpKk7VM87nm5zSHfCH4INjkN///7w1QJtSNE060MRiWLG43qbAe1xYCogBZBcwM21RQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=N2M4T1icpExc5wz7Vv3Ei3ub4ZtCXJzHMBtB8Kb76Kg=; b=JMEB0CUQkIi9MmULZ/8xr4bieEiD/gtInXot0PEgS/VCYNzPU8jFQ3C+uHrmyp+fZpF2xsu1h5dO3FBpJfun76iOOVdcOOv5PAYZBdYu8CLwXoV+phHaBBtgNRs9ydjNikowDqWanKsaSAT7MNGXG2CXbeKZQAgRYcCwbW+yOB3m/1CZyehDyf74DAWssZ7Kg++a6CoChVwDcgRF3/UNI+btAmjS8XdMWqz4PgcfBOmABKOW9Ld6t50WPWeIMQ9qfOOkPu/5Sm62dfzRDDjAavxePuo46nGI0yoL9ieY2cp6waUI6hXOusQxSMdOzp2bVe3M9xYJVRVr2JqSC20qtQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from SG2PR06MB5430.apcprd06.prod.outlook.com (2603:1096:4:1ba::14) by TYZPR06MB5121.apcprd06.prod.outlook.com (2603:1096:400:1c3::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.11; Fri, 13 Jan 2023 13:17:48 +0000 Received: from SG2PR06MB5430.apcprd06.prod.outlook.com ([fe80::3f58:2ff3:cd56:3b84]) by SG2PR06MB5430.apcprd06.prod.outlook.com ([fe80::3f58:2ff3:cd56:3b84%7]) with mapi id 15.20.6002.013; Fri, 13 Jan 2023 13:17:47 +0000 Message-ID: Date: Fri, 13 Jan 2023 21:17:43 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: nvptx: Avoid deadlock in 'cuStreamAddCallback' callback, error case (was: [PATCH 6/6, OpenACC, libgomp] Async re-work, nvptx changes) To: Thomas Schwinge , gcc-patches@gcc.gnu.org, Chung-Lin Tang , Tom de Vries References: <9523b49a-0454-e0a9-826d-5eeec2a8c973@mentor.com> <87zgan6eug.fsf@euler.schwinge.homeip.net> Content-Language: en-US From: Chung-Lin Tang In-Reply-To: <87zgan6eug.fsf@euler.schwinge.homeip.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: TYCPR01CA0049.jpnprd01.prod.outlook.com (2603:1096:405:2::13) To SG2PR06MB5430.apcprd06.prod.outlook.com (2603:1096:4:1ba::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SG2PR06MB5430:EE_|TYZPR06MB5121:EE_ X-MS-Office365-Filtering-Correlation-Id: 147bf741-0b95-4c93-c659-08daf5689087 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JUCCIEWLljdU5T9DJaHJUyAn2YqoGx0DpmP15tW0Rpjdqc/WkLwc+LXC9Ctnn+td0aE4QdyywMmK2ZrZ3L/VrcRgbrRG+Kmei0PsdbWz6JC8N8jtGM9oS10OLwYyYUVERZBnYKJQvtvVUXJyo1SY/YDDzUHNneJAJELE2DUOp+R/3imahDV2RTCSA8e6yJvNkCLl5O2pnK+T2x3V43ca//npIoCitcQanrSJLu6dwGbjFZd9bBfahbP9vH743EIlaYOcOUC8Y9opgcQ1GD+9jQSqvwP2YzeeB3o0ZeU4MRan2AEHgAS9pTnLP9+HdtzBZ3J0CsUmADQtpAJk8KS+U/C0xtUTBFIxdBF7JFtUcvGw2TQf+MJCO2JTGMzsyb9xd4hzuoVo2piQx3l89Dp4hijZ30U4RK5ydX+OTbrkUigLUKOvLeqQpx+d8IsCvDB3n6bCQLdxx2eE/aHDN4HoaNgjmdN67JpImokz64PatT0ckVgEKssGeqvKaB8vMoFcBwYrRfUcTrLEBvqVSoLEoBN6wiOK8JLxuAaYEjZQNxTGYLU2Q94CluJ5WNFpnPB/4ia7OXlJ2SwT/6EZ5LL5BKLIrDmtW8zmlqVfy9awRaJGlO06jd9L6XjROrG5+pLs2q6Wrmy/mLTjxYz9z3d5vab0P8cSnKtzMccu5Xcok6nTtM74iKoj2/Ps149H6/kIwdd2zwP4yEylk2PVEnMoTikV+niCVaZq1C6zdywhIc7G5+1fP//pCpilbds+bXiKL9Q7e8QtQ3G9BHNCv4Hh9Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SG2PR06MB5430.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(376002)(346002)(366004)(396003)(39860400002)(136003)(451199015)(186003)(31696002)(36756003)(26005)(5660300002)(8936002)(41300700001)(6512007)(53546011)(6486002)(966005)(6506007)(83380400001)(316002)(110136005)(478600001)(6666004)(66556008)(66946007)(8676002)(66476007)(38100700002)(2616005)(82960400001)(86362001)(31686004)(2906002)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RStra3AvRVpiM1VuNEc0YWNqaW1jN281MmJjaEVmMTVmTnFxWS9aZlU5K1lE?= =?utf-8?B?M05OVXQ1K2tGOUpZblB5UFVhcSszd1JMUWp4ZzU3ek1xdTZ0RUxNOGRlbmRx?= =?utf-8?B?N2JXL1lscERDT1lkYllZNDRxQVY5ZkVkSzlJTHo5aVFOTUREWXg1dUVta250?= =?utf-8?B?WFRSNElNSWI3bzg1WGlaQkllK0FaNFhpUjFFUVZ0RnFMMWsweCtjMm9aZmJj?= =?utf-8?B?UCtJNlhZNTdGc1dheGdXbXpNUDlIRzhpRUJZVy9MeldjN0I4cGxBU3lPT2hS?= =?utf-8?B?TzM5SEkydGZmbmN3ZW5PRTFDVXc4WHZBaHdKT29TQTV1MW1vQkp2T0N0MFF4?= =?utf-8?B?TVNQOVBBNklyelpnais4bEtRR2xOUkQ1VkU1T2txMWhOYzRkNy9GOU5EdHVh?= =?utf-8?B?eTlSVE96eXdEaDhtQjdTWlhUekxNY3VtdWNiT1FuSUNPVlJwckVIcFl4Y2Vt?= =?utf-8?B?RS8yYm54NkRnbGtUS1l3Y25Fcm5Qa0ZxalVWeDUrVE9TU1A3V0ttY2x2THc0?= =?utf-8?B?VFhocGIyOHpGRGY3ek5xMWhCT1pONHozaVNJOTJyNXVaSHd5RFRPU1B2bkgz?= =?utf-8?B?dGNuMlNlZjRyelNVYnk4R0ZSN3M4d1hHUGg4UTlJMVNmREpabXpBQmRGSith?= =?utf-8?B?VmhsU0I3N3JMNTNKbTdMT1VDU1pjMlNEbnl4STJoZVowQ2hlQXBaNVkzUGxI?= =?utf-8?B?cFdoVVBkTnZuWnc5WHlZQUVDdWtuOXJWYkZSNmdzQTRRb3BpUWZiK1M2OVY0?= =?utf-8?B?UXpoWDd2eUdpRkx6Vm4rZXR4WFpPYy9NZWpBcG55VndGSjZKd3JvKzVNa2Y4?= =?utf-8?B?QmpXbzlOUWcvNlpabDNQYlJ5dzNITWdFaTZucUhXdFZ3c2RjOXFXejJCcFlk?= =?utf-8?B?VVZqZ1VJazExbS91T25DNWd1UUdGWW5NL3dQVjEwQmxyMXdXc1oyeSt1MkhF?= =?utf-8?B?aXBmQk5aNjY2Z1JSNTV3WTB6c29Fb3VxTDBQNkNwaHI1NXRMY0lXZGI3RmFO?= =?utf-8?B?ZFZ1NzBGNEFLZWtLTEgrbWV4Sm80cmcvRjYrMmFaVVdXazY3SDc0VGdJZjRB?= =?utf-8?B?MllEK1dkMWlqL1hDRDBHZ0NDV1E0YTl2d3dOUGlrSk1qZ1VKdkkrMVdGcmlw?= =?utf-8?B?WkVMK01ITmFCSzhSNXZ2bVBJL096bCtOekpiZVZVVnZzbUROTk5qYmdjY0dU?= =?utf-8?B?d015L0FSUFFRcFJic2lZUk5kYlBUeTAvMHNVSTBiekhKU0pnLytSTFE5MGRx?= =?utf-8?B?QitrZUMvblhIMnpXQ2tSUzl1cTB6NkI4RTlYWEIwY1orV3ZiaklvUFU1MjhG?= =?utf-8?B?djlKMmViazJKQjNGc093QklLVGJ3SFVxNStBZVhEbDdjQ2lwbzhMMDN5QmEw?= =?utf-8?B?eUxWTHF0Yit3UU44a3pXcTMyUDhVUnRqWC8xbUl1TEczTGpvMEwyVDh4alZt?= =?utf-8?B?RTU0NytUK3FGbFFybjU1MER6c0duais4djB3SXI2YXFSd2pzV3BFKytuTXZN?= =?utf-8?B?QTdWTlBWRkJ2ZGpxSmx6NEM0WkNDT1FvenlSaHpHTXg4dkMvVnU1QWJ3VHo2?= =?utf-8?B?VlBzaGtSQmJkOVNHNG5lcUE5blI1UXBNMDBmNkxvWU0zV2VGNHV4VlJuM2pR?= =?utf-8?B?NUREeWlYRTV0QitjMWd6VzBhRytNZDVNQ0FsdE95SUs4dnJoVEVzVEFxNFdv?= =?utf-8?B?cHl1eFJlUXFQNnpTdHlTZTJxTk9zeHZZMG5OZ1NaZEd6Y00vTUg4aGlEc0hQ?= =?utf-8?B?alRKS3d0c0U0c3NoRTMrTVpnZzNXeEJrYmUvb2FLbk9Pd3I0cDdVc1crT1F4?= =?utf-8?B?TFFNb0NJRXVaRXJxSHNBSG1UcnovNGViVCt2elVGR3EydE93VnR0TDkyWm9X?= =?utf-8?B?VE9BaURrL1ZoQWhrT3FVUFJpc2hHZGJsRlVtcUhEK1FNQ05CL0NnK3B3czhU?= =?utf-8?B?MmdsTGViakVKOVg2dWFsc0dSaGNBSStHcU9lZk10NS9GZER0NjkwTitQN2NC?= =?utf-8?B?bUZ5RVdVNEIrTGFseXBsSVlydElXS1ZmYzA1YS9Td3R3MmUvVExlMzFvekd2?= =?utf-8?B?K1c1d0dsQS9Nc2QvM0ppRXUzeVQ2TWJZWFQxZ3NkNEM2TkdUaDd5dVI0Zlg4?= =?utf-8?B?Y0E4Vmo3aFpyUFl2TGtOc2VnRk11SXltSnhDY1VtRittbEt5MXBxaDA5NCs3?= =?utf-8?B?eEE9PQ==?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 147bf741-0b95-4c93-c659-08daf5689087 X-MS-Exchange-CrossTenant-AuthSource: SG2PR06MB5430.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jan 2023 13:17:47.8754 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VGQItoJKB528uWck70S2nu0bUfXxSpfi4PLojDb5hcYFYIMnHVoadOsLIt4G443dBJDxcrlgv2998cAxMQNOZ39LjWC6P9B1Ar+gF4OrsoY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZPR06MB5121 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,DKIMWL_WL_MED,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi Thomas, On 2023/1/12 9:51 PM, Thomas Schwinge wrote: > In my case, 'cuda_callback_wrapper' (expectedly) gets invoked with > 'res != CUDA_SUCCESS' ("an illegal memory access was encountered"). > When we invoke 'GOMP_PLUGIN_fatal', this attempts to shut down the device > (..., which deadlocks); that's generally problematic: per > https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__STREAM.html#group__CUDA__STREAM_1g613d97a277d7640f4cb1c03bd51c2483 > "'cuStreamAddCallback' [...] Callbacks must not make any CUDA API calls". I remember running into this myself when first creating this async support (IIRC in my case it was cuFree()-ing something) yet you've found another mistake here! :) > Given that eventually we must reach a host/device synchronization point > (latest when the device is shut down at program termination), and the > non-'CUDA_SUCCESS' will be upheld until then, it does seem safe to > replace this 'GOMP_PLUGIN_fatal' with 'GOMP_PLUGIN_error' as per the > "nvptx: Avoid deadlock in 'cuStreamAddCallback' callback, error case" > attached. OK to push? I think this patch is fine. Actual approval powers are your's or Tom's :) > > (Might we even skip 'GOMP_PLUGIN_error' here, understanding that the > error will be caught and reported at the next host/device synchronization > point? But I've not verified that.) Actually, the CUDA driver API docs are a bit vague on what exactly this CUresult arg to the callback actually means. The 'res != CUDA_SUCCESS' handling here was basically just generic handling. I am not really sure what is the true right thing to do here (is the error still retained by CUDA after the callback completes?) Chung-Lin