From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by sourceware.org (Postfix) with ESMTPS id 9B03338CCBC5 for ; Wed, 5 Jun 2024 06:23:06 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9B03338CCBC5 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 9B03338CCBC5 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1717568588; cv=none; b=XdD3bnMpzcxvUwKIq2b0zAwn1+4jnAaEeXR8EjfSByIk843HAUG/mOMYHfhK5/SF9A/vmlQ9mHOlsuhvH709OxIWtjWjXzyj2J8D/OvkVt4Nb0PihzXYz9rayFAroDpm7jQt6vJNHdqZLiD0bCjs4s8n7I7oVs6y9StOXcdkJHM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1717568588; c=relaxed/simple; bh=3uTafRSbDF3L9Rspfiiwdtp9mXCijcxngLFEgcnv+z8=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=rarjm1hYg8HfQf7wy3s4gLFrHR2Ua/Lgis+bw+ZboZ02tOIpJmNIBUB/rijWa/aNbThAo6egmgZ9pIUcoK8Rb8Q2Y+7TgLWPFR3d3b5qJG53xyXYKhSVlhADDhs0hYQqlWi6qP8yuiPv4bdceCrcu7XTya71EMbmwsa27oJHn0U= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from murzim.nue2.suse.org (unknown [10.168.4.243]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 961C021A31 for ; Wed, 5 Jun 2024 06:23:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1717568585; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=9J30zRPu40kzpV+2TfdVoZ7WkRoFm3qIkWHJaZ5jCJQ=; b=PGuTnVaqyqssO+1Or6zB7yacpQQA0NRMZmh0xdOYDbNDp+QKFiolMpHlvTQCKd+nM5L0GC 5cVoII8UqKOz6XgiVtvORvBNAk5/AxVcgftZvEIfvyWYGDbnjXJa2pfU6CP/8HuvsLjsPw yPz4KnFf8lMApQKdbc+UUfUS+4cBUzs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1717568585; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=9J30zRPu40kzpV+2TfdVoZ7WkRoFm3qIkWHJaZ5jCJQ=; b=7RZ19TfCxo/fO7OnxmikSp8ADs9BuAD66oijyvH1ULm2QByonFTQgIe+2+Gue9GQKpDz0A 3DMkrF2DvugN3/DQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1717568585; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=9J30zRPu40kzpV+2TfdVoZ7WkRoFm3qIkWHJaZ5jCJQ=; b=PGuTnVaqyqssO+1Or6zB7yacpQQA0NRMZmh0xdOYDbNDp+QKFiolMpHlvTQCKd+nM5L0GC 5cVoII8UqKOz6XgiVtvORvBNAk5/AxVcgftZvEIfvyWYGDbnjXJa2pfU6CP/8HuvsLjsPw yPz4KnFf8lMApQKdbc+UUfUS+4cBUzs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1717568585; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=9J30zRPu40kzpV+2TfdVoZ7WkRoFm3qIkWHJaZ5jCJQ=; b=7RZ19TfCxo/fO7OnxmikSp8ADs9BuAD66oijyvH1ULm2QByonFTQgIe+2+Gue9GQKpDz0A 3DMkrF2DvugN3/DQ== Date: Wed, 5 Jun 2024 08:23:05 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH 3/4] Add double reduction support for SLP vectorization MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Level: X-Spamd-Result: default: False [0.57 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MISSING_MID(2.50)[]; NEURAL_SPAM_SHORT(2.06)[0.685]; NEURAL_HAM_LONG(-0.89)[-0.886]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_HAS_DN(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; MIME_TRACE(0.00)[0:+] X-Spam-Score: 0.57 X-Spam-Status: No, score=-10.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,MISSING_MID,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Message-ID: <20240605062305._oIFrgxUsOcWHz3jcff-5G5dVEl1s8dDNHamGU6PnU8@z> The following makes double reduction vectorization work when using (single-lane) SLP vectorization. * tree-vect-loop.cc (vect_analyze_scalar_cycles_1): Queue double reductions in LOOP_VINFO_REDUCTIONS. (vect_create_epilog_for_reduction): Remove asserts disabling SLP for double reductions. (vectorizable_reduction): Analyze SLP double reductions only once and start off the correct places. * tree-vect-slp.cc (vect_get_and_check_slp_defs): Allow vect_double_reduction_def. (vect_build_slp_tree_2): Fix condition for the ignored reduction initial values. * tree-vect-stmts.cc (vect_analyze_stmt): Allow vect_double_reduction_def. --- gcc/tree-vect-loop.cc | 35 +++++++++++++++++++++++++---------- gcc/tree-vect-slp.cc | 3 ++- gcc/tree-vect-stmts.cc | 4 ++++ 3 files changed, 31 insertions(+), 11 deletions(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index ccd6acef5c5..b9e8e9b5559 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -685,6 +685,8 @@ vect_analyze_scalar_cycles_1 (loop_vec_info loop_vinfo, class loop *loop, STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_double_reduction_def; STMT_VINFO_DEF_TYPE (reduc_stmt_info) = vect_double_reduction_def; + /* Make it accessible for SLP vectorization. */ + LOOP_VINFO_REDUCTIONS (loop_vinfo).safe_push (reduc_stmt_info); } else { @@ -5975,7 +5977,6 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, stmt_vec_info rdef_info = stmt_info; if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_double_reduction_def) { - gcc_assert (!slp_node); double_reduc = true; stmt_info = loop_vinfo->lookup_def (gimple_phi_arg_def (stmt_info->stmt, 0)); @@ -6020,7 +6021,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, { outer_loop = loop; loop = loop->inner; - gcc_assert (!slp_node && double_reduc); + gcc_assert (double_reduc); } vectype = STMT_VINFO_REDUC_VECTYPE (reduc_info); @@ -6035,7 +6036,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, for induc_val, use initial_def. */ if (STMT_VINFO_REDUC_TYPE (reduc_info) == INTEGER_INDUC_COND_REDUCTION) induc_val = STMT_VINFO_VEC_INDUC_COND_INITIAL_VAL (reduc_info); - /* ??? Coverage for double_reduc and 'else' isn't clear. */ + /* ??? Coverage for 'else' isn't clear. */ } else { @@ -7605,15 +7606,16 @@ vectorizable_reduction (loop_vec_info loop_vinfo, STMT_VINFO_TYPE (stmt_info) = reduc_vec_info_type; return true; } - if (slp_node) - { - slp_node_instance->reduc_phis = slp_node; - /* ??? We're leaving slp_node to point to the PHIs, we only - need it to get at the number of vector stmts which wasn't - yet initialized for the instance root. */ - } if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_double_reduction_def) { + if (gimple_bb (stmt_info->stmt) != loop->header) + { + /* For SLP we arrive here for both the inner loop LC PHI and + the outer loop PHI. The latter is what we want to analyze + the reduction with. */ + gcc_assert (slp_node); + return true; + } use_operand_p use_p; gimple *use_stmt; bool res = single_imm_use (gimple_phi_result (stmt_info->stmt), @@ -7622,6 +7624,14 @@ vectorizable_reduction (loop_vec_info loop_vinfo, phi_info = loop_vinfo->lookup_stmt (use_stmt); } + if (slp_node) + { + slp_node_instance->reduc_phis = slp_node; + /* ??? We're leaving slp_node to point to the PHIs, we only + need it to get at the number of vector stmts which wasn't + yet initialized for the instance root. */ + } + /* PHIs should not participate in patterns. */ gcc_assert (!STMT_VINFO_RELATED_STMT (phi_info)); gphi *reduc_def_phi = as_a (phi_info->stmt); @@ -7637,6 +7647,11 @@ vectorizable_reduction (loop_vec_info loop_vinfo, bool only_slp_reduc_chain = true; stmt_info = NULL; slp_tree slp_for_stmt_info = slp_node ? slp_node_instance->root : NULL; + /* For double-reductions we start SLP analysis at the inner loop LC PHI + which is the def of the outer loop live stmt. */ + if (STMT_VINFO_DEF_TYPE (reduc_info) == vect_double_reduction_def + && slp_node) + slp_for_stmt_info = SLP_TREE_CHILDREN (slp_for_stmt_info)[0]; while (reduc_def != PHI_RESULT (reduc_def_phi)) { stmt_vec_info def = loop_vinfo->lookup_def (reduc_def); diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index ba1190c7155..7e3d0107b4e 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -778,6 +778,7 @@ vect_get_and_check_slp_defs (vec_info *vinfo, unsigned char swap, case vect_constant_def: case vect_internal_def: case vect_reduction_def: + case vect_double_reduction_def: case vect_induction_def: case vect_nested_cycle: case vect_first_order_recurrence: @@ -1906,7 +1907,7 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node, class loop *loop = LOOP_VINFO_LOOP (loop_vinfo); /* Reduction initial values are not explicitely represented. */ if (def_type != vect_first_order_recurrence - && !nested_in_vect_loop_p (loop, stmt_info)) + && gimple_bb (stmt_info->stmt) == loop->header) skip_args[loop_preheader_edge (loop)->dest_idx] = true; /* Reduction chain backedge defs are filled manually. ??? Need a better way to identify a SLP reduction chain PHI. diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index c82381e799e..5098b7fab6a 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -13260,6 +13260,10 @@ vect_analyze_stmt (vec_info *vinfo, || relevance == vect_used_only_live)); break; + case vect_double_reduction_def: + gcc_assert (!bb_vinfo && node); + break; + case vect_induction_def: case vect_first_order_recurrence: gcc_assert (!bb_vinfo); -- 2.35.3