From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 4539 invoked by alias); 27 Mar 2017 20:27:44 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Received: (qmail 4520 invoked by uid 89); 27 Mar 2017 20:27:44 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-23.8 required=5.0 tests=AWL,BAYES_00,GIT_PATCH_0,GIT_PATCH_1,GIT_PATCH_2,GIT_PATCH_3,KAM_LAZY_DOMAIN_SECURITY,KAM_STOCKGEN,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.2 spammy=ciao, 1075, qs, immediately! X-HELO: mx0a-001b2d01.pphosted.com Received: from mx0a-001b2d01.pphosted.com (HELO mx0a-001b2d01.pphosted.com) (148.163.156.1) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 27 Mar 2017 20:27:39 +0000 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v2RKNUmp123961 for ; Mon, 27 Mar 2017 16:27:39 -0400 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0a-001b2d01.pphosted.com with ESMTP id 29f9busb9u-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Mon, 27 Mar 2017 16:27:38 -0400 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Mar 2017 21:27:35 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Mon, 27 Mar 2017 21:27:35 +0100 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v2RKRYKh40173784; Mon, 27 Mar 2017 20:27:34 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8CAFFA404D; Mon, 27 Mar 2017 21:27:05 +0100 (BST) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70174A4040; Mon, 27 Mar 2017 21:27:04 +0100 (BST) Received: from oc5510024614.ibm.com (unknown [9.145.166.8]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Mar 2017 21:27:04 +0100 (BST) Received: by oc5510024614.ibm.com (Postfix, from userid 500) id B2BE21AAF5; Mon, 27 Mar 2017 22:27:35 +0200 (CEST) Date: Mon, 27 Mar 2017 20:50:00 -0000 From: Dominik Vogt To: gcc-patches@gcc.gnu.org Cc: Andreas Krebbel , Ulrich Weigand Subject: [PATCH] S/390: Optimize atomic_compare_exchange and atomic_compare builtins. Reply-To: vogt@linux.vnet.ibm.com Mail-Followup-To: vogt@linux.vnet.ibm.com, gcc-patches@gcc.gnu.org, Andreas Krebbel , Ulrich Weigand MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="W/nzBZO5zC0uMSeA" Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-12-10) X-TM-AS-GCONF: 00 x-cbid: 17032720-0020-0000-0000-00000330C09F X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17032720-0021-0000-0000-000040FD0B95 Message-Id: <20170327202735.GA21766@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-03-27_17:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1703270168 X-SW-Source: 2017-03/txt/msg01417.txt.bz2 --W/nzBZO5zC0uMSeA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-length: 404 The attached patch optimizes the atomic_exchange and atomic_compare patterns on s390 and s390x (mostly limited to SImode and DImode). Among general optimizaation, the changes fix most of the problems reported in PR 80080: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80080 Bootstrapped and regression tested on a zEC12 with s390 and s390x biarch. Ciao Dominik ^_^ ^_^ -- Dominik Vogt IBM Germany --W/nzBZO5zC0uMSeA Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=0001-ChangeLog Content-length: 1537 gcc/ChangeLog-dv-atomic-gcc7 * (s390_expand_cs_hqi): Removed. (s390_expand_cs, s390_expand_atomic_exchange_tdsi): New prototypes. (s390_cc_modes_compatible): Export. * config/s390/predicates.md ("memory_nosymref_operand"): New predicate for compare-and-swap. * config/s390/s390.c(s390_emit_compare_and_swap): Handle all integer modes. (s390_cc_modes_compatible): Remove static. (s390_expand_cs_hqi): Make static. (s390_expand_cs_tdsi): Generate an explicit compare before trying compare-and-swap, in some cases. (s390_expand_cs): Wrapper function. (s390_expand_atomic_exchange_tdsi): New backend specific expander for atomic_exchange. * config/s390/s390.md (define_peephole2): New peephole to help combining the load-and-test pattern with volatile memory. ("cstorecc4"): Deal with CCZmode too. ("sne", "sneccz1_ne", "sneccz1_eq"): Renamed and duplicated pattern. ("sneccz_ne", "sneccz_eq"): New. ("atomic_compare_and_swap"): Merge the patterns for small and large integers. Forbid symref memory operands. Move expander to s390.c. ("atomic_compare_and_swap_internal") ("*atomic_compare_and_swap_1") ("*atomic_compare_and_swap_2") ("*atomic_compare_and_swap_3"): Forbid symref memory operands. ("atomic_exchange"): Allow and implement all integer modes. gcc/testsuite/ChangeLog-dv-atomic-gcc7 * gcc.target/s390/md/atomic_compare_exchange-1.c: New test. * gcc.target/s390/md/atomic_compare_exchange-1.inc: New test. * gcc.target/s390/md/atomic_exchange-1.inc: New test. --W/nzBZO5zC0uMSeA Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-S-390-Optimize-atomic_compare_exchange-and-atomic_co.patch" Content-length: 36585 >From 17822384e33b4b98c299ab25969907eb2b9184ee Mon Sep 17 00:00:00 2001 From: Dominik Vogt Date: Thu, 23 Feb 2017 17:23:11 +0100 Subject: [PATCH] S/390: Optimize atomic_compare_exchange and atomic_compare builtins. 1) Use the load-and-test instructions for atomic_exchange if the value is 0. 2) If IS_WEAK is true, compare the memory contents before a compare-and-swap and skip the CS instructions if the value is not the expected one. --- gcc/config/s390/predicates.md | 13 + gcc/config/s390/s390-protos.h | 5 +- gcc/config/s390/s390.c | 178 ++++++++++- gcc/config/s390/s390.md | 217 +++++++++---- .../gcc.target/s390/md/atomic_compare_exchange-1.c | 84 ++++++ .../s390/md/atomic_compare_exchange-1.inc | 336 +++++++++++++++++++++ .../gcc.target/s390/md/atomic_exchange-1.c | 309 +++++++++++++++++++ 7 files changed, 1075 insertions(+), 67 deletions(-) create mode 100644 gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.c create mode 100644 gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.inc create mode 100644 gcc/testsuite/gcc.target/s390/md/atomic_exchange-1.c diff --git a/gcc/config/s390/predicates.md b/gcc/config/s390/predicates.md index 0c82efc..aadb454 100644 --- a/gcc/config/s390/predicates.md +++ b/gcc/config/s390/predicates.md @@ -67,6 +67,19 @@ return true; }) +;; Like memory_operand, but rejects symbol references. +(define_predicate "memory_nosymref_operand" + (match_operand 0 "memory_operand") +{ + if (SYMBOL_REF_P (XEXP (op, 0))) + return false; + if (GET_CODE (op) == SUBREG + && GET_CODE (SUBREG_REG (op)) == MEM + && SYMBOL_REF_P (XEXP (XEXP (op, 0), 0))) + return false; + return true; +}) + ;; Return true if OP is a valid operand for the BRAS instruction. ;; Allow SYMBOL_REFs and @PLT stubs. diff --git a/gcc/config/s390/s390-protos.h b/gcc/config/s390/s390-protos.h index 7f06a20..81644b9 100644 --- a/gcc/config/s390/s390-protos.h +++ b/gcc/config/s390/s390-protos.h @@ -81,6 +81,7 @@ extern bool s390_overlap_p (rtx, rtx, HOST_WIDE_INT); extern bool s390_offset_p (rtx, rtx, rtx); extern int tls_symbolic_operand (rtx); +extern machine_mode s390_cc_modes_compatible (machine_mode, machine_mode); extern bool s390_match_ccmode (rtx_insn *, machine_mode); extern machine_mode s390_tm_ccmode (rtx, rtx, bool); extern machine_mode s390_select_ccmode (enum rtx_code, rtx, rtx); @@ -112,8 +113,8 @@ extern void s390_expand_vec_strlen (rtx, rtx, rtx); extern void s390_expand_vec_movstr (rtx, rtx, rtx); extern bool s390_expand_addcc (enum rtx_code, rtx, rtx, rtx, rtx, rtx); extern bool s390_expand_insv (rtx, rtx, rtx, rtx); -extern void s390_expand_cs_hqi (machine_mode, rtx, rtx, rtx, - rtx, rtx, bool); +extern bool s390_expand_cs (machine_mode, rtx, rtx, rtx, rtx, rtx, bool); +extern void s390_expand_atomic_exchange_tdsi (rtx, rtx, rtx); extern void s390_expand_atomic (machine_mode, enum rtx_code, rtx, rtx, rtx, bool); extern void s390_expand_tbegin (rtx, rtx, rtx, bool); diff --git a/gcc/config/s390/s390.c b/gcc/config/s390/s390.c index e800323..14770a2 100644 --- a/gcc/config/s390/s390.c +++ b/gcc/config/s390/s390.c @@ -1240,7 +1240,7 @@ s390_set_has_landing_pad_p (bool value) mode which is compatible with both. Otherwise, return VOIDmode. */ -static machine_mode +machine_mode s390_cc_modes_compatible (machine_mode m1, machine_mode m2) { if (m1 == m2) @@ -1751,7 +1751,25 @@ static rtx s390_emit_compare_and_swap (enum rtx_code code, rtx old, rtx mem, rtx cmp, rtx new_rtx) { - emit_insn (gen_atomic_compare_and_swapsi_internal (old, mem, cmp, new_rtx)); + switch (GET_MODE (mem)) + { + case QImode: + case HImode: + case SImode: + emit_insn (gen_atomic_compare_and_swapsi_internal (old, mem, cmp, + new_rtx)); + break; + case DImode: + emit_insn (gen_atomic_compare_and_swapdi_internal (old, mem, cmp, + new_rtx)); + break; + case TImode: + emit_insn (gen_atomic_compare_and_swapti_internal (old, mem, cmp, + new_rtx)); + break; + default: + gcc_unreachable (); + } return s390_emit_compare (code, gen_rtx_REG (CCZ1mode, CC_REGNUM), const0_rtx); } @@ -6709,7 +6727,7 @@ s390_two_part_insv (struct alignment_context *ac, rtx *seq1, rtx *seq2, the memory location, CMP the old value to compare MEM with and NEW_RTX the value to set if CMP == MEM. */ -void +static void s390_expand_cs_hqi (machine_mode mode, rtx btarget, rtx vtarget, rtx mem, rtx cmp, rtx new_rtx, bool is_weak) { @@ -6785,6 +6803,160 @@ s390_expand_cs_hqi (machine_mode mode, rtx btarget, rtx vtarget, rtx mem, NULL_RTX, 1, OPTAB_DIRECT), 1); } +/* Variant of s390_expand_cs for SI, DI and TI modes. */ +static void +s390_expand_cs_tdsi (machine_mode mode, rtx btarget, rtx vtarget, rtx mem, + rtx cmp, rtx new_rtx, bool is_weak) +{ + rtx output = vtarget; + rtx_code_label *skip_cs_label = NULL; + bool do_const_opt = false; + + if (!register_operand (output, mode)) + output = gen_reg_rtx (mode); + + /* If IS_WEAK is true and the INPUT value is a constant, compare the memory + with the constant first and skip the compare_and_swap because its very + expensive and likely to fail anyway. + Note 1: This is done only for IS_WEAK because C11 suggest that spurious + fails are possible in that case. + Note 2: It may be useful to do this also for non-constant INPUT. + Note 3: Currently only targets with "load on condition" re supported + (z196 and newer). */ + + if (TARGET_CPU_Z196 + && (mode == SImode || (mode == DImode && TARGET_ZARCH))) + do_const_opt = (is_weak && CONST_INT_P (cmp)); + + if (do_const_opt) + { + const int very_unlikely = REG_BR_PROB_BASE / 100 - 1; + rtx cc = gen_rtx_REG (CCZmode, CC_REGNUM); + + skip_cs_label = gen_label_rtx (); + emit_move_insn (output, mem); + emit_move_insn (btarget, const0_rtx); + emit_insn (gen_rtx_SET (cc, gen_rtx_COMPARE (CCZmode, output, cmp))); + s390_emit_jump (skip_cs_label, gen_rtx_NE (VOIDmode, cc, const0_rtx)); + add_int_reg_note (get_last_insn (), REG_BR_PROB, very_unlikely); + /* If the jump is not taken, OUTPUT is the expected value. */ + cmp = output; + /* Reload newval to a register manually, *after* the compare and jump + above. Otherwise Reload might place it before the jump. */ + } + else + cmp = force_reg (mode, cmp); + new_rtx = force_reg (mode, new_rtx); + s390_emit_compare_and_swap (EQ, output, mem, cmp, new_rtx); + + /* We deliberately accept non-register operands in the predicate + to ensure the write back to the output operand happens *before* + the store-flags code below. This makes it easier for combine + to merge the store-flags code with a potential test-and-branch + pattern following (immediately!) afterwards. */ + if (output != vtarget) + emit_move_insn (vtarget, output); + + if (skip_cs_label != NULL) + emit_label (skip_cs_label); + if (TARGET_Z196) + { + rtx cc, cond, ite; + + cc = gen_rtx_REG ((do_const_opt) ? CCZmode : CCZ1mode, CC_REGNUM); + cond = gen_rtx_EQ (VOIDmode, cc, const0_rtx); + ite = gen_rtx_IF_THEN_ELSE (SImode, cond, const1_rtx, + (do_const_opt) ? btarget : const0_rtx); + emit_insn (gen_rtx_SET (btarget, ite)); + } + else + { + rtx cc, cond; + + cc = gen_rtx_REG ((do_const_opt) ? CCZmode : CCZ1mode, CC_REGNUM); + cond = gen_rtx_EQ (SImode, cc, const0_rtx); + emit_insn (gen_cstorecc4 (btarget, cond, cc, const0_rtx)); + } +} + +/* Expand an atomic compare and swap operation. MEM is the memory location, + CMP the old value to compare MEM with and NEW_RTX the value to set if + CMP == MEM. */ + +bool +s390_expand_cs (machine_mode mode, rtx btarget, rtx vtarget, rtx mem, + rtx cmp, rtx new_rtx, bool is_weak) +{ + if (GET_MODE_BITSIZE (mode) >= 4 + && GET_MODE_BITSIZE (mode) > MEM_ALIGN (mem)) + return false; + + /* If the memory address isn't in a register already, reload it now to allow + for better optimization in the Rtl passes. Otherwise Reload does it much + later and it might end up inside a loop. */ + if (!REG_P (XEXP (mem, 0))) + { + rtx ref; + + ref = force_reg (Pmode, XEXP (mem, 0)); + mem = gen_rtx_MEM (mode, ref); + } + + switch (mode) + { + case TImode: + case DImode: + case SImode: + s390_expand_cs_tdsi (mode, btarget, vtarget, mem, cmp, new_rtx, is_weak); + break; + case HImode: + case QImode: + s390_expand_cs_hqi (mode, btarget, vtarget, mem, cmp, new_rtx, is_weak); + break; + default: + gcc_unreachable (); + } + + return true; +} + +/* Expand an atomic_exchange operation simulated with a compare-and-swap loop. + The memory location MEM is set to INPUT. OUTPUT is set to the previous value + of MEM. */ + +void +s390_expand_atomic_exchange_tdsi (rtx output, rtx mem, rtx input) +{ + machine_mode mode = GET_MODE (mem); + rtx_code_label *csloop; + + if (TARGET_Z196 + && (mode == DImode || mode == SImode) + && CONST_INT_P (input) && INTVAL (input) == 0) + { + emit_move_insn (output, const0_rtx); + if (mode == DImode) + emit_insn (gen_atomic_fetch_anddi (output, mem, const0_rtx, input)); + else + emit_insn (gen_atomic_fetch_andsi (output, mem, const0_rtx, input)); + return; + } + + if (!REG_P (input)) + { + rtx tmp; + + tmp = gen_reg_rtx (mode); + emit_move_insn (tmp, input); + input = tmp; + } + emit_move_insn (output, mem); + csloop = gen_label_rtx (); + emit_label (csloop); + s390_emit_jump (csloop, s390_emit_compare_and_swap (NE, output, mem, output, + input)); +} + /* Expand an atomic operation CODE of mode MODE. MEM is the memory location and VAL the value to play with. If AFTER is true then store the value MEM holds after the operation, if AFTER is false then store the value MEM diff --git a/gcc/config/s390/s390.md b/gcc/config/s390/s390.md index 93a0bc6..34d76b2 100644 --- a/gcc/config/s390/s390.md +++ b/gcc/config/s390/s390.md @@ -893,6 +893,21 @@ [(set_attr "op_type" "RR,RXY") (set_attr "z10prop" "z10_fr_E1,z10_fwd_A3") ]) +; Peephole to combine a load-and-test from volatile memory which combine does +; not do. +(define_peephole2 + [(set (match_operand:GPR 0 "register_operand") + (match_operand:GPR 2 "memory_operand")) + (set (reg CC_REGNUM) + (compare (match_dup 0) (match_operand:GPR 1 "const0_operand")))] + "s390_match_ccmode(insn, CCSmode) && TARGET_EXTIMM + && GENERAL_REG_P (operands[0]) + && satisfies_constraint_T (operands[2])" + [(parallel + [(set (reg:CCS CC_REGNUM) + (compare:CCS (match_dup 2) (match_dup 1))) + (set (match_dup 0) (match_dup 2))])]) + ; ltr, lt, ltgr, ltg (define_insn "*tst_cconly_extimm" [(set (reg CC_REGNUM) @@ -2060,6 +2075,16 @@ (set_attr "cpu_facility" "*,*,*,*,vx,*,vx,*,*,*,*,*,*") ]) +(define_split + [(parallel + [(set (match_operand:GPR 0 "register_operand") + (mem:GPR (match_operand:P 1 "larl_operand"))) + (set (match_operand:P 2 "register_operand") + (match_dup 1))])] + "" + [(set (match_dup 2) (match_dup 1)) + (set (match_dup 0) (mem:GPR (match_dup 2)))]) + (define_peephole2 [(set (match_operand:SI 0 "register_operand" "") (mem:SI (match_operand 1 "address_operand" "")))] @@ -6499,26 +6524,112 @@ [(parallel [(set (match_operand:SI 0 "register_operand" "") (match_operator:SI 1 "s390_eqne_operator" - [(match_operand:CCZ1 2 "register_operand") + [(match_operand 2 "register_operand") (match_operand 3 "const0_operand")])) (clobber (reg:CC CC_REGNUM))])] "" - "emit_insn (gen_sne (operands[0], operands[2])); - if (GET_CODE (operands[1]) == EQ) - emit_insn (gen_xorsi3 (operands[0], operands[0], const1_rtx)); + "machine_mode mode = GET_MODE (operands[2]); + if (GET_CODE (operands[1]) != EQ && GET_CODE (operands[1]) != NE) + FAIL; + if (!s390_cc_modes_compatible (mode, CCZmode)) + FAIL; + if (TARGET_Z196) + { + rtx cc, cond, ite; + + cc = gen_rtx_REG (mode, CC_REGNUM); + cond = gen_rtx_EQ (VOIDmode, cc, const0_rtx); + ite = gen_rtx_IF_THEN_ELSE (SImode, cond, const1_rtx, const0_rtx); + emit_insn (gen_rtx_SET (operands[0], ite)); + } + else + { + if (mode == CCZ1mode) + { + if (GET_CODE (operands[1]) == EQ && TARGET_EXTIMM) + emit_insn (gen_sneccz1_eq (operands[0], operands[2])); + else + emit_insn (gen_sneccz1_ne (operands[0], operands[2])); + } + else + { + if (GET_CODE (operands[1]) == EQ && TARGET_EXTIMM) + emit_insn (gen_sneccz_eq (operands[0], operands[2])); + else + emit_insn (gen_sneccz_ne (operands[0], operands[2])); + } + if (GET_CODE (operands[1]) == EQ && !TARGET_EXTIMM) + emit_insn (gen_xorsi3 (operands[0], operands[0], const1_rtx)); + } DONE;") -(define_insn_and_split "sne" +(define_insn_and_split "sneccz1_ne" [(set (match_operand:SI 0 "register_operand" "=d") (ne:SI (match_operand:CCZ1 1 "register_operand" "0") - (const_int 0))) - (clobber (reg:CC CC_REGNUM))] + (const_int 0)))] "" "#" "reload_completed" - [(parallel - [(set (match_dup 0) (ashiftrt:SI (match_dup 0) (const_int 28))) - (clobber (reg:CC CC_REGNUM))])]) + [(set (match_dup 0) (lshiftrt:SI (match_dup 0) (const_int 28)))]) + +(define_insn_and_split "sneccz1_eq" + [(set (reg:CCZ1 CC_REGNUM) + (ne:CCZ1 (match_operand:CCZ1 1 "register_operand" "0") + (const_int 0))) + (set (match_operand:SI 0 "register_operand" "=d") + (eq:SI (match_dup 1) (const_int 0)))] + "TARGET_EXTIMM" + "#" + "&& reload_completed" + [(set (match_dup 0) (lshiftrt:SI (match_dup 0) (const_int 28))) + (parallel + [(set (reg:CCZ CC_REGNUM) + (compare:CCZ (xor:SI (match_dup 0) (const_int 1)) + (const_int 0))) + (set (match_dup 0) + (xor:SI (match_dup 0) (const_int 1)))])]) + +(define_insn_and_split "sneccz_ne" + [(set (reg:CCZ1 CC_REGNUM) + (eq:CCZ1 (match_operand:CCZ 1 "register_operand" "0") + (const_int 0))) + (set (match_operand:SI 0 "register_operand" "=d") + (ne:SI (match_dup 1) (const_int 0)))] + "!TARGET_Z196" + "#" + "&& reload_completed" + [(set (match_dup 0) (lshiftrt:SI (match_dup 0) (const_int 28))) + (parallel + [(set (reg:CCAN CC_REGNUM) + (compare:CCAN (neg:SI (abs:SI (match_dup 0))) + (const_int 0))) + (set (match_dup 0) + (neg:SI (abs:SI (match_dup 0))))]) + (set (match_dup 0) (lshiftrt:SI (match_dup 0) (const_int 31)))]) + +(define_insn_and_split "sneccz_eq" + [(set (reg:CCZ1 CC_REGNUM) + (ne:CCZ1 (match_operand:CCZ 1 "register_operand" "0") + (const_int 0))) + (set (match_operand:SI 0 "register_operand" "=d") + (eq:SI (match_dup 1) (const_int 0)))] + "!TARGET_Z196 && TARGET_EXTIMM" + "#" + "&& reload_completed" + [(set (match_dup 0) (lshiftrt:SI (match_dup 0) (const_int 28))) + (parallel + [(set (reg:CCAN CC_REGNUM) + (compare:CCAN (neg:SI (abs:SI (match_dup 0))) + (const_int 0))) + (set (match_dup 0) + (neg:SI (abs:SI (match_dup 0))))]) + (set (match_dup 0) (lshiftrt:SI (match_dup 0) (const_int 31))) + (parallel + [(set (reg:CCZ CC_REGNUM) + (compare:CCZ (xor:SI (match_dup 0) (const_int 1)) + (const_int 0))) + (set (match_dup 0) + (xor:SI (match_dup 0) (const_int 1)))])]) ;; @@ -10174,60 +10285,28 @@ (define_expand "atomic_compare_and_swap" [(match_operand:SI 0 "register_operand") ;; bool success output - (match_operand:DGPR 1 "nonimmediate_operand");; oldval output - (match_operand:DGPR 2 "memory_operand") ;; memory - (match_operand:DGPR 3 "register_operand") ;; expected intput - (match_operand:DGPR 4 "register_operand") ;; newval intput + (match_operand:DINT 1 "nonimmediate_operand");; oldval output + (match_operand:DINT 2 "memory_nosymref_operand") ;; memory + (match_operand:DINT 3 "general_operand") ;; expected intput + (match_operand:DINT 4 "general_operand") ;; newval intput (match_operand:SI 5 "const_int_operand") ;; is_weak (match_operand:SI 6 "const_int_operand") ;; success model (match_operand:SI 7 "const_int_operand")] ;; failure model "" { - rtx cc, cmp, output = operands[1]; - - if (!register_operand (output, mode)) - output = gen_reg_rtx (mode); - - if (MEM_ALIGN (operands[2]) < GET_MODE_BITSIZE (GET_MODE (operands[2]))) + bool rc; + rc = s390_expand_cs (mode, operands[0], operands[1], operands[2], + operands[3], operands[4], INTVAL (operands[5])); + if (rc) + DONE; + else FAIL; - - emit_insn (gen_atomic_compare_and_swap_internal - (output, operands[2], operands[3], operands[4])); - - /* We deliberately accept non-register operands in the predicate - to ensure the write back to the output operand happens *before* - the store-flags code below. This makes it easier for combine - to merge the store-flags code with a potential test-and-branch - pattern following (immediately!) afterwards. */ - if (output != operands[1]) - emit_move_insn (operands[1], output); - - cc = gen_rtx_REG (CCZ1mode, CC_REGNUM); - cmp = gen_rtx_EQ (SImode, cc, const0_rtx); - emit_insn (gen_cstorecc4 (operands[0], cmp, cc, const0_rtx)); - DONE; -}) - -(define_expand "atomic_compare_and_swap" - [(match_operand:SI 0 "register_operand") ;; bool success output - (match_operand:HQI 1 "nonimmediate_operand") ;; oldval output - (match_operand:HQI 2 "memory_operand") ;; memory - (match_operand:HQI 3 "general_operand") ;; expected intput - (match_operand:HQI 4 "general_operand") ;; newval intput - (match_operand:SI 5 "const_int_operand") ;; is_weak - (match_operand:SI 6 "const_int_operand") ;; success model - (match_operand:SI 7 "const_int_operand")] ;; failure model - "" -{ - s390_expand_cs_hqi (mode, operands[0], operands[1], operands[2], - operands[3], operands[4], INTVAL (operands[5])); - DONE; }) (define_expand "atomic_compare_and_swap_internal" [(parallel [(set (match_operand:DGPR 0 "register_operand") - (match_operand:DGPR 1 "memory_operand")) + (match_operand:DGPR 1 "memory_nosymref_operand")) (set (match_dup 1) (unspec_volatile:DGPR [(match_dup 1) @@ -10241,7 +10320,7 @@ ; cdsg, csg (define_insn "*atomic_compare_and_swap_1" [(set (match_operand:TDI 0 "register_operand" "=r") - (match_operand:TDI 1 "memory_operand" "+S")) + (match_operand:TDI 1 "memory_nosymref_operand" "+S")) (set (match_dup 1) (unspec_volatile:TDI [(match_dup 1) @@ -10258,7 +10337,7 @@ ; cds, cdsy (define_insn "*atomic_compare_and_swapdi_2" [(set (match_operand:DI 0 "register_operand" "=r,r") - (match_operand:DI 1 "memory_operand" "+Q,S")) + (match_operand:DI 1 "memory_nosymref_operand" "+Q,S")) (set (match_dup 1) (unspec_volatile:DI [(match_dup 1) @@ -10278,7 +10357,7 @@ ; cs, csy (define_insn "*atomic_compare_and_swapsi_3" [(set (match_operand:SI 0 "register_operand" "=r,r") - (match_operand:SI 1 "memory_operand" "+Q,S")) + (match_operand:SI 1 "memory_nosymref_operand" "+Q,S")) (set (match_dup 1) (unspec_volatile:SI [(match_dup 1) @@ -10374,16 +10453,30 @@ DONE; }) +;; Pattern to implement atomic_exchange with a compare-and-swap loop. The code +;; generated by the middleend is not good. (define_expand "atomic_exchange" - [(match_operand:HQI 0 "register_operand") ;; val out - (match_operand:HQI 1 "memory_operand") ;; memory - (match_operand:HQI 2 "general_operand") ;; val in + [(match_operand:DINT 0 "register_operand") ;; val out + (match_operand:DINT 1 "memory_nosymref_operand") ;; memory + (match_operand:DINT 2 "general_operand") ;; val in (match_operand:SI 3 "const_int_operand")] ;; model "" { - s390_expand_atomic (mode, SET, operands[0], operands[1], - operands[2], false); - DONE; + if (mode == HImode || mode == QImode) + { + s390_expand_atomic (mode, SET, operands[0], operands[1], + operands[2], false); + DONE; + } + else if (mode == SImode || TARGET_ZARCH) + { + if (MEM_ALIGN (operands[1]) < GET_MODE_BITSIZE (mode)) + FAIL; + s390_expand_atomic_exchange_tdsi (operands[0], operands[1], operands[2]); + DONE; + } + else + FAIL; }) ;; diff --git a/gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.c b/gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.c new file mode 100644 index 0000000..5cc026d --- /dev/null +++ b/gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.c @@ -0,0 +1,84 @@ +/* Machine description pattern tests. */ + +/* { dg-do compile } */ +/* { dg-options "" } */ +/* { dg-do run { target { s390_useable_hw } } } */ + +#include + +struct +{ +#ifdef __s390xx__ + __int128 dummy128; + __int128 mem128; +#endif + long long dummy64; + long long mem64; + int dummy32; + int mem32; + short mem16l; + short mem16h; + char mem8ll; + char mem8lh; + char mem8hl; + char mem8hh; +} mem_s; + +#define TYPE char +#define FN(SUFFIX) f8 ## SUFFIX +#define FNS(SUFFIX) "f8" #SUFFIX +#include "atomic_compare_exchange-1.inc" + +#define TYPE short +#define FN(SUFFIX) f16 ##SUFFIX +#define FNS(SUFFIX) "f16" #SUFFIX +#include "atomic_compare_exchange-1.inc" + +#define TYPE int +#define FN(SUFFIX) f32 ## SUFFIX +#define FNS(SUFFIX) "f32" #SUFFIX +#include "atomic_compare_exchange-1.inc" + +#define TYPE long long +#define FN(SUFFIX) f64 ## SUFFIX +#define FNS(SUFFIX) "f64" #SUFFIX +#include "atomic_compare_exchange-1.inc" + +#ifdef __s390xx__ +#define TYPE __int128 +#define FN(SUFFIX) f128 ## SUFFIX +#define FNS(SUFFIX) "f128" #SUFFIX +#include "atomic_compare_exchange-1.inc" +#endif + +int main(void) +{ + int err_count = 0; + int i; + + for (i = -1; i <= 2; i++) + { + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f8_validate(&mem_s.mem8ll, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f8_validate(&mem_s.mem8lh, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f8_validate(&mem_s.mem8hl, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f8_validate(&mem_s.mem8hh, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f16_validate(&mem_s.mem16l, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f16_validate(&mem_s.mem16h, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f32_validate(&mem_s.mem32, i, 1); + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f64_validate(&mem_s.mem64, i, 1); +#ifdef __s390xx__ + __builtin_memset(&mem_s, 0x99, sizeof(mem_s)); + err_count += f128_validate(&mem_s.mem128, i, 1); +#endif + } + + return err_count; +} diff --git a/gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.inc b/gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.inc new file mode 100644 index 0000000..199aaa3 --- /dev/null +++ b/gcc/testsuite/gcc.target/s390/md/atomic_compare_exchange-1.inc @@ -0,0 +1,336 @@ +/* -*-c-*- */ + +#undef NEW +#define NEW 3 + +__attribute__ ((noinline)) +int FN(_bo)(TYPE *mem, TYPE *old_ret, TYPE old) +{ + *old_ret = old; + return __atomic_compare_exchange_n (mem, (void *)old_ret, NEW, 1, 2, 0); +} + +__attribute__ ((noinline)) +void FN(_o)(TYPE *mem, TYPE *old_ret, TYPE old) +{ + *old_ret = old; + __atomic_compare_exchange_n (mem, (void *)old_ret, NEW, 1, 2, 0); + return; +} + +__attribute__ ((noinline)) +int FN(_b)(TYPE *mem, TYPE old) +{ + return __atomic_compare_exchange_n (mem, (void *)&old, NEW, 1, 2, 0); +} + +__attribute__ ((noinline)) +void FN()(TYPE *mem, TYPE old) +{ + __atomic_compare_exchange_n (mem, (void *)&old, NEW, 1, 2, 0); + return; +} + +/* Const != 0 old value. */ +__attribute__ ((noinline)) +int FN(_c1_bo)(TYPE *mem, TYPE *old_ret) +{ + *old_ret = 1; + return __atomic_compare_exchange_n (mem, (void *)old_ret, NEW, 1, 2, 0); +} + +__attribute__ ((noinline)) +void FN(_c1_o)(TYPE *mem, TYPE *old_ret) +{ + *old_ret = 1; + __atomic_compare_exchange_n (mem, (void *)old_ret, NEW, 1, 2, 0); + return; +} + +__attribute__ ((noinline)) +int FN(_c1_b)(TYPE *mem) +{ + TYPE old = 1; + return __atomic_compare_exchange_n (mem, (void *)&old, NEW, 1, 2, 0); +} + +__attribute__ ((noinline)) +void FN(_c1)(TYPE *mem) +{ + TYPE old = 1; + __atomic_compare_exchange_n (mem, (void *)&old, NEW, 1, 2, 0); + return; +} + +/* Const == 0 old value. */ +__attribute__ ((noinline)) +int FN(_c0_bo)(TYPE *mem, TYPE *old_ret) +{ + *old_ret = 0; + return __atomic_compare_exchange_n (mem, (void *)old_ret, NEW, 1, 2, 0); +} + +__attribute__ ((noinline)) +void FN(_c0_o)(TYPE *mem, TYPE *old_ret) +{ + *old_ret = 0; + __atomic_compare_exchange_n (mem, (void *)old_ret, NEW, 1, 2, 0); + return; +} + +__attribute__ ((noinline)) +int FN(_c0_b)(TYPE *mem) +{ + TYPE old = 0; + return __atomic_compare_exchange_n (mem, (void *)&old, NEW, 1, 2, 0); +} + +__attribute__ ((noinline)) +void FN(_c0)(TYPE *mem) +{ + TYPE old = 0; + __atomic_compare_exchange_n (mem, (void *)&old, NEW, 1, 2, 0); + return; +} + +int FN(_validate_mem)(TYPE *mem, TYPE expected_mem) +{ + if (*mem != expected_mem) + { + fprintf(stderr, " BAD: mem %d != expected mem %d\n", + *mem, expected_mem); + return 1; + } + + return 0; +} + +int FN(_validate_rc)(int rc, int expected_rc) +{ + if (rc != expected_rc) + { + fprintf(stderr, " BAD: rc %d != expected rc %d\n", + rc, expected_rc); + return 1; + } + + return 0; +} + +int FN(_validate_old_ret)(int old_ret, int expected_old_ret) +{ + if (old_ret != expected_old_ret) + { + fprintf(stderr, " BAD: old_ret %d != expected old_ret %d\n", + old_ret, expected_old_ret); + return 1; + } + + return 0; +} + +int FN(_validate)(TYPE *mem, TYPE init_mem, TYPE old) +{ + int err_count = 0; + int rc; + TYPE expected_mem; + int expected_rc; + TYPE old_ret; + int failed; + const char *fname; + + fprintf(stderr, "%s: init_mem %d @ %p\n", __FUNCTION__, init_mem, mem); + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + expected_rc = (init_mem == old); + fname = FNS(_bo); + rc = FN(_bo)(mem, &old_ret, old); + failed |= FN(_validate_rc)(rc, expected_rc); + failed |= FN(_validate_old_ret)(old_ret, init_mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + fname = FNS(_o); + FN(_o)(mem, &old_ret, old); + failed |= FN(_validate_old_ret)(old_ret, init_mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + expected_rc = (init_mem == old); + fname = FNS(_b); + rc = FN(_b)(mem, old); + failed |= FN(_validate_rc)(rc, expected_rc); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + fname = FNS(); + FN()(mem, old); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + expected_rc = (init_mem == old); + fname = FNS(_c1_bo); + rc = FN(_c1_bo)(mem, &old_ret); + failed |= FN(_validate_rc)(rc, expected_rc); + failed |= FN(_validate_old_ret)(old_ret, init_mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + fname = FNS(_c1_o); + FN(_c1_o)(mem, &old_ret); + failed |= FN(_validate_old_ret)(old_ret, init_mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + expected_rc = (init_mem == old); + fname = FNS(_c1_b); + rc = FN(_c1_b)(mem); + failed |= FN(_validate_rc)(rc, expected_rc); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 1; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + fname = FNS(_c1); + FN(_c1)(mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 0; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + expected_rc = (init_mem == old); + fname = FNS(_c0_bo); + rc = FN(_c0_bo)(mem, &old_ret); + failed |= FN(_validate_rc)(rc, expected_rc); + failed |= FN(_validate_old_ret)(old_ret, init_mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 0; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + fname = FNS(_c0_o); + FN(_c0_o)(mem, &old_ret); + failed |= FN(_validate_old_ret)(old_ret, init_mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 0; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + expected_rc = (init_mem == old); + fname = FNS(_c0_b); + rc = FN(_c0_b)(mem); + failed |= FN(_validate_rc)(rc, expected_rc); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + { + failed = 0; + old = 0; + *mem = init_mem; + expected_mem = (init_mem == old) ? NEW : *mem; + fname = FNS(_c0); + FN(_c0)(mem); + failed |= FN(_validate_mem)(mem, expected_mem); + if (failed) + { + fprintf(stderr, " FAIL: %s: near line %d\n", fname, __LINE__ - 3); + err_count++; + } + } + + return err_count; +} + +#undef TYPE +#undef MEM +#undef FN +#undef FNS diff --git a/gcc/testsuite/gcc.target/s390/md/atomic_exchange-1.c b/gcc/testsuite/gcc.target/s390/md/atomic_exchange-1.c new file mode 100644 index 0000000..f82b213 --- /dev/null +++ b/gcc/testsuite/gcc.target/s390/md/atomic_exchange-1.c @@ -0,0 +1,309 @@ +/* Machine description pattern tests. */ + +/* { dg-do compile } */ +/* { dg-options "-lpthread -latomic" } */ +/* { dg-do run { target { s390_useable_hw } } } */ + +/**/ + +char +ae_8_0 (char *lock) +{ + return __atomic_exchange_n (lock, 0, 2); +} + +char +ae_8_1 (char *lock) +{ + return __atomic_exchange_n (lock, 1, 2); +} + +char g8; + +char +ae_8_g_0 (void) +{ + return __atomic_exchange_n (&g8, 0, 2); +} + +char +ae_8_g_1 (void) +{ + return __atomic_exchange_n (&g8, 1, 2); +} + +/**/ + +short +ae_16_0 (short *lock) +{ + return __atomic_exchange_n (lock, 0, 2); +} + +short +ae_16_1 (short *lock) +{ + return __atomic_exchange_n (lock, 1, 2); +} + +short g16; + +short +ae_16_g_0 (void) +{ + return __atomic_exchange_n (&g16, 0, 2); +} + +short +ae_16_g_1 (void) +{ + return __atomic_exchange_n (&g16, 1, 2); +} + +/**/ + +int +ae_32_0 (int *lock) +{ + return __atomic_exchange_n (lock, 0, 2); +} + +int +ae_32_1 (int *lock) +{ + return __atomic_exchange_n (lock, 1, 2); +} + +int g32; + +int +ae_32_g_0 (void) +{ + return __atomic_exchange_n (&g32, 0, 2); +} + +int +ae_32_g_1 (void) +{ + return __atomic_exchange_n (&g32, 1, 2); +} + +/**/ + +long long +ae_64_0 (long long *lock) +{ + return __atomic_exchange_n (lock, 0, 2); +} + +long long +ae_64_1 (long long *lock) +{ + return __atomic_exchange_n (lock, 1, 2); +} + +long long g64; + +long long + ae_64_g_0 (void) +{ + return __atomic_exchange_n (&g64, 0, 2); +} + +long long +ae_64_g_1 (void) +{ + return __atomic_exchange_n (&g64, 1, 2); +} + +/**/ + +#ifdef __s390x__ +__int128 +ae_128_0 (__int128 *lock) +{ + return __atomic_exchange_n (lock, 0, 2); +} + +__int128 +ae_128_1 (__int128 *lock) +{ + return __atomic_exchange_n (lock, 1, 2); +} + +__int128 g128; + +__int128 +ae_128_g_0 (void) +{ + return __atomic_exchange_n (&g128, 0, 2); +} + +__int128 +ae_128_g_1 (void) +{ + return __atomic_exchange_n (&g128, 1, 2); +} + +#endif + +int main(void) +{ + int i; + + for (i = 0; i <= 2; i++) + { + int oval = i; + + { + char lock; + char rval; + + lock = oval; + rval = ae_8_0 (&lock); + if (lock != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + lock = oval; + rval = ae_8_1 (&lock); + if (lock != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g8 = oval; + rval = ae_8_g_0 (); + if (g8 != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g8 = oval; + rval = ae_8_g_1 (); + if (g8 != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + } + { + short lock; + short rval; + + lock = oval; + rval = ae_16_0 (&lock); + if (lock != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + lock = oval; + rval = ae_16_1 (&lock); + if (lock != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g16 = oval; + rval = ae_16_g_0 (); + if (g16 != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g16 = oval; + rval = ae_16_g_1 (); + if (g16 != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + } + { + int lock; + int rval; + + lock = oval; + rval = ae_32_0 (&lock); + if (lock != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + lock = oval; + rval = ae_32_1 (&lock); + if (lock != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g32 = oval; + rval = ae_32_g_0 (); + if (g32 != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g32 = oval; + rval = ae_32_g_1 (); + if (g32 != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + } + { + long long lock; + long long rval; + + lock = oval; + rval = ae_64_0 (&lock); + if (lock != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + lock = oval; + rval = ae_64_1 (&lock); + if (lock != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g64 = oval; + rval = ae_64_g_0 (); + if (g64 != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g64 = oval; + rval = ae_64_g_1 (); + if (g64 != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + } + +#ifdef __s390x__ + { + __int128 lock; + __int128 rval; + + lock = oval; + rval = ae_128_0 (&lock); + if (lock != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + lock = oval; + rval = ae_128_1 (&lock); + if (lock != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g128 = oval; + rval = ae_128_g_0 (); + if (g128 != 0) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + g128 = oval; + rval = ae_128_g_1 (); + if (g128 != 1) + __builtin_abort (); + if (rval != oval) + __builtin_abort (); + } +#endif + } + + return 0; +} -- 2.3.0 --W/nzBZO5zC0uMSeA--