From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id 00FC83858D37 for ; Mon, 28 Oct 2024 13:31:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 00FC83858D37 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 00FC83858D37 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1730122294; cv=none; b=UmHaGKQH14ri+9OY4R7eBGiBmXZ6Uqvc++dWf0W2213PFTaFTMMyHL4U4tBzGyUEgGNj6fyatAlJ1/edccvFPNnqqZFUk631hw/h8W7A17jIP928OSbpeGN+85lL28kQ+cE9NtX0ctm0BVvo12O+w1qxqRNbTCl/JmmtolgcWeo= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1730122294; c=relaxed/simple; bh=HXUCo54FAr8ZsEXDqW66zl1M1+g2keuoTh1n7TKxndM=; h=DKIM-Signature:Message-ID:Date:MIME-Version:Subject:To:From; b=d/Dvq683qVrZCBNQ/FRdiW8xLsEd6nEJAL8dm/5szloiOCCYks/DXpOgTHp067gT/FuATYtjbgINriV1C+s/AXigVmHB1VKKXYPlp7mAQh5rb+AGs7eweBkgh3f2BWtX+aqI/iAz2w3ODJIbHvSYpGb2Dl8ypGchWAM76guNtGk= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1730122283; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=IphA9sbSCc38VpVHwY8MwebUenBdFdI3VUTKWI//TAE=; b=jK+fQPWPfHuf/kVkZ/aScXnTSWo1NWNGLZ83luQK5v/R4614AVS7kjrF4Wgkp8tOVgCjA7 1n/zQgL3GodRkkAbPdhV9Q7Wo2Jl61XE8Gwi/rvee3sSHzZvNBBvG33gSTdOMMzEFD6rZi eB6B/mBjJ/Erlu6NvtEX+knbST0srOc= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-pATs3_QnMC6SHbbfkADEEQ-1; Mon, 28 Oct 2024 09:31:22 -0400 X-MC-Unique: pATs3_QnMC6SHbbfkADEEQ-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7b161d4c422so813902685a.0 for ; Mon, 28 Oct 2024 06:31:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730122282; x=1730727082; h=content-transfer-encoding:in-reply-to:organization:autocrypt:from :content-language:references:to:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IphA9sbSCc38VpVHwY8MwebUenBdFdI3VUTKWI//TAE=; b=TgFJFkWDKXUW5xUXt05Df3gQSyujJLsDm3b4aYbGGzHoLn8E1hmEu4lixOoET28pJO OAtNnk0dK3ABd6LONqN/SCcDUHPNZvo8DS7w1FClWAqWFIxXnfa9UKg+mPklmj7bsUhN 81x7mLnCy2YA3h8R89aokCitoa1t+W8TACidl3wF8xtXFTtGSjtxGq0jqCIqgPdp6wPI EYsu5hqNBXDL1zVLJ/0C0XuxRe1fctGNSsnOkueps3o4l75SiinWx+eLIsuZABNc+bYX w/b7aHpnUyjXFldeo5fBoeDQ1r9r7ZoQ+XCSNqGq8q+Bv7JadaPyd2WI8xJHGpKAz9hU 8aFA== X-Forwarded-Encrypted: i=1; AJvYcCVDuHNJUSpirKCmHrmF8XI1yxDAOyXtQCqKtvb+ShSWKIKDsot6vXSXQNClOn9L5SxvXS6UTJ9ko07j@sourceware.org X-Gm-Message-State: AOJu0Yx+S3a0DkNDztUvgUz4AzjvQc+KNTwKQ6LaqI3VDxHrbWyzYl+S NODXDjFuTPy88HavtSCpFAT8FZxOEgw5je0MlrhlGsdBQ9VBSNuyr14FAcIazKrttJSGqaepTBo JO7fNnfirpd27AD/Fp0V118U40N+4lBTrU53AylAN4tLvf0iOdSGHKwYjk+dPvRK3yw== X-Received: by 2002:a05:620a:450e:b0:7a9:b904:d608 with SMTP id af79cd13be357-7b193ed48b6mr1092209785a.6.1730122282021; Mon, 28 Oct 2024 06:31:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE9Bighj++4doBd7XNt6RCbf35BLQnV4fqdGxJq+iFfEzF+jMYIjtgiVvA5WrRgWui70Nyu/Q== X-Received: by 2002:a05:620a:450e:b0:7a9:b904:d608 with SMTP id af79cd13be357-7b193ed48b6mr1092207485a.6.1730122281587; Mon, 28 Oct 2024 06:31:21 -0700 (PDT) Received: from [192.168.0.241] ([198.48.244.52]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7b18d345cc7sm317960385a.118.2024.10.28.06.31.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 28 Oct 2024 06:31:21 -0700 (PDT) Message-ID: <1c62e9f5-f8a8-4c8e-86e8-11c1d2b52f2b@redhat.com> Date: Mon, 28 Oct 2024 09:31:17 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] aarch64: Small optimisation in AdvSIMD erf and erfc To: Joe Ramsay , libc-alpha@sourceware.org References: <20241024161916.1013404-1-Joe.Ramsay@arm.com> From: Carlos O'Donell Autocrypt: addr=carlos@redhat.com; keydata= xsFNBFef5BoBEACvJ15QMMZh4stKHbz0rs78XsOdxuug37dumTx6ngrDCwZ61k7nHQ+uxLuo QvLSc6YJGBEfiNFbs1hvhRFNR7xJbzRYmin7kJZZ/06fH2cgTkQhN0mRBP8KsKKT+7SvvBL7 85ZfAhArWf5m5Tl0CktZ8yoG8g9dM4SgdvdSdzZUaWBVHc6TjdAb9YEQ1/jpyfHsQp+PWLuQ ZI8nZUm+I3IBDLkbbuJVQklKzpT1b8yxVSsHCyIPFRqDDUjPL5G4WnUVy529OzfrciBvHdxG sYYDV8FX7fv6V/S3eL6qmZbObivIbLD2NbeDqw6vNpr+aehEwgwNbMVuVfH1PVHJV8Qkgxg4 PqPgQC7GbIhxxYroGbLJCQ41j25M+oqCO/XW/FUu/9x0vY5w0RsZFhlmSP5lBDcaiy3SUgp3 MSTePGuxpPlLVMePxKvabSS7EErLKlrAEmDgnUYYdPqGCefA+5N9Rn2JPfP7SoQEp2pHhEyM 6Xg9x7TJ+JNuDowQCgwussmeDt2ZUeMl3s1f6/XePfTd3l8c8Yn5Fc8reRa28dFANU6oXiZf 7/h3iQXPg81BsLMJK3aA/nyajRrNxL8dHIx7BjKX0/gxpOozlUHZHl73KhAvrBRaqLrr2tIP LkKrf3d7wdz4llg4NAGIU4ERdTTne1QAwS6x2tNa9GO9tXGPawARAQABzSpDYXJsb3MgTydE b25lbGwgKFdvcmspIDxjYXJsb3NAcmVkaGF0LmNvbT7CwZUEEwEIAD8CGwMGCwkIBwMCBhUI AgkKCwQWAgMBAh4BAheAFiEEcnNUKzmWLfeymZMUFnkrTqJTQPgFAmagDwgFCRDhXm4ACgkQ FnkrTqJTQPgLlw/+JD7l4tj8l8hAMUlszrlIT6IhKSODzjrGO+6d9Y6T9vyE2kk4Xbn+kdJf uBl+wj2+U15MsQe9Z4RwowIB3YHHXgj53M2OjqOAY/sRWXZVDfmVj03hqW8D7zFxjc0SZ9cI TI0MwrDWc+Fr3naXeo7HhgjUmULfPndxb8NHVV4Ds2DTkZoUMwB8l3dboD+nKi5GbfVBf3Q5 cBw0CPkxPl0hxD9sr5IMgWIKVLtvztMIXv2xWAavqk8pQjk0zCYd46GcA8d9pZuac24e9NbM ZzTxu6cP0sKhub1JFIadyBHtJnEV/8Auc8nXJ63QY3h0QVCJYV35gQeejEdMD94in2XTkxk0 A/xCp32bmSZv5flsmdAIv5LK4jTKLvzd6BSy/v7qlpgQ7sNaxQ/JRd+8YuBIiUVIp/kgGezD qtGZSpvPCFuG3LxsdvAu7JAzBY3sfBd2lSGOeHX/JK0nQ6s97j4HlSuXIabSOdsCI5UGSOq5 thbIqfK3ewUSUB0yGvWf7EyuZugtCZOaFGpvcT3ix9/sP1fTRlJl+bNjMcO8GwedDoy85oeg yLCEV9gejCr+NijLfPYtb1s8o0hYu13uBojFyBv+bkUI5hTQaVLacq7VglA/QLOy/3mtM2v5 4OEotiNXbKypHFKnoks/MFpP4xdwxGX5jU4MgFg80aPFGr0oZVXOwU0EV5/kGgEQAKvTJke+ QSjATmz11ALKle/SSEpUwL5QOpt3xomEATcYAamww0HADfGTKdUR+aWgOK3vqu6Sicr1zbuZ jHCs2GaIgRoqh1HKVgCmaJYjizvidHluqrox6qqc9PG0bWb0f5xGQw+X2z+bEinzv4qaep1G 1OuYgvG49OpHTgZMiJq9ncHCxkD2VEJKgMywGJ4Agdl+NWVn0T7w6J+/5QmBIE8hh4NzpYfr xzWCJ9iZ3skG4zBGB4YEacc3+oeEoybc10h6tqhQNrtIiSRJH+SUJvOiNH8oMXPLAjfFVy3d 4BOgyxJhE0UhmQIQHMJxCBw81fQD10d0dcru0rAIEldEpt2UXqOr0rOALDievMF/2BKQiOA7 PbMC3/dwuNHDlClQzdjil8O7UsIgf3IMFaIbQoUEvjlgf5cm9a94gWABcfI1xadAq9vcIB5v +9fM71xDgdELnZThTd8LByrG99ExVMcG2PZYXJllVDQDZqYA1PjD9e0yHq5whJi3BrZgwDaL 5vYZEb1EMyH+BQLO3Zw/Caj8W6mooGHgNveRQ1g9FYn3NUp7UvS22Zt/KW4pCpbgkQZefxup KO6QVNwwggV44cTQ37z5onGbNPD8+2k2mmC0OEtGBkj+VH39tRk+uLOcuXlGNSVk3xOyxni0 Nk9M0GvTvPKoah9gkvL/+AofN/31ABEBAAHCwXwEGAEIACYCGwwWIQRyc1QrOZYt97KZkxQW eStOolNA+AUCZqAPEAUJEOFedgAKCRAWeStOolNA+D38D/9WnZY9fUmPhZVwpDnhIXvlXgqX cspZJEBWNS5ArFn8CLcje7z9hzX3+86lqkEeohTmlgtTg4ctZzM+XKyWSiqHCRCR+FX5SKaa 1VveBtwvjTSVmtV1m0rNHEvUZ5x47A8NadWqYi6uOQ22FhEqUOiwJ7EHzk4w9W3gT1913XT1 vmkCn6FtQcrQvJT7pP+oA0YIVs8ADayJcqWHM+Ez7L2fpfAzBDhIS7dq2MYU8LQOQAsx1y7H 6njp5dN/OI/aN/RL6XeX1Kxl4Xe+hc+tq457fLAUnmaevUldvKThuj+5/Cd4DW25MxaqinfY m/U6pBQ4ZwQPGWA0f+GKiJcLosSRXxIuEdZAl82ht+KgT3zhV/BvQRmrD6wX3ywPkJap8h4K ibwz3r6NbHKdCX22ok58oE8NAWtmTRTKXDhh8oWOKdIYjX6jJzdb/F8rPNoEY3UiYbaNTxt5 TE9VD+yWilYO796HMXjXenCOlghy3HFmZbsQ4N+FlG6LQD7cnwm56kcrJk1IlnQXOSOd2BA2 qNbM1Ohry3B+1F4Oaee+ZKH2C5y7Kx0y3m1b5X7Wpx76H5BeUAp6dQi6nNYeqM9PglZIMvSe O4uRThl5mMDx8MXQz6M9qQ5anYwre+/TudTfCzcTpgXod1wEqi2ErJ5jNgh18DRlSQ3tbDvG O0FatDMfJw== Organization: Red Hat In-Reply-To: <20241024161916.1013404-1-Joe.Ramsay@arm.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-12.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 10/24/24 12:19 PM, Joe Ramsay wrote: > In both routines, reduce register pressure such that GCC 14 emits no > spills for erf and fewer spills for erfc. Also use more efficient > comparison for the special-case in erf. Are there any microbenchmarks that show the performance benefit? > --- > Thanks, > Joe > sysdeps/aarch64/fpu/erf_advsimd.c | 25 ++++++++++++++++--------- > sysdeps/aarch64/fpu/erfc_advsimd.c | 13 +++++++------ > 2 files changed, 23 insertions(+), 15 deletions(-) > > diff --git a/sysdeps/aarch64/fpu/erf_advsimd.c b/sysdeps/aarch64/fpu/erf_advsimd.c > index 19cbb7d0f4..c0116735e4 100644 > --- a/sysdeps/aarch64/fpu/erf_advsimd.c > +++ b/sysdeps/aarch64/fpu/erf_advsimd.c > @@ -22,19 +22,21 @@ > static const struct data > { > float64x2_t third; > - float64x2_t tenth, two_over_five, two_over_fifteen; > - float64x2_t two_over_nine, two_over_fortyfive; > + float64x2_t tenth, two_over_five, two_over_nine; > + double two_over_fifteen, two_over_fortyfive; > float64x2_t max, shift; > + uint64x2_t max_idx; > #if WANT_SIMD_EXCEPT > float64x2_t tiny_bound, huge_bound, scale_minus_one; > #endif > } data = { > + .max_idx = V2 (768), > .third = V2 (0x1.5555555555556p-2), /* used to compute 2/3 and 1/6 too. */ > - .two_over_fifteen = V2 (0x1.1111111111111p-3), > + .two_over_fifteen = 0x1.1111111111111p-3, > .tenth = V2 (-0x1.999999999999ap-4), > .two_over_five = V2 (-0x1.999999999999ap-2), > .two_over_nine = V2 (-0x1.c71c71c71c71cp-3), > - .two_over_fortyfive = V2 (0x1.6c16c16c16c17p-5), > + .two_over_fortyfive = 0x1.6c16c16c16c17p-5, > .max = V2 (5.9921875), /* 6 - 1/128. */ > .shift = V2 (0x1p45), > #if WANT_SIMD_EXCEPT > @@ -87,8 +89,8 @@ float64x2_t VPCS_ATTR V_NAME_D1 (erf) (float64x2_t x) > float64x2_t a = vabsq_f64 (x); > /* Reciprocal conditions that do not catch NaNs so they can be used in BSLs > to return expected results. */ > - uint64x2_t a_le_max = vcleq_f64 (a, dat->max); > - uint64x2_t a_gt_max = vcgtq_f64 (a, dat->max); > + uint64x2_t a_le_max = vcaleq_f64 (x, dat->max); > + uint64x2_t a_gt_max = vcagtq_f64 (x, dat->max); > > #if WANT_SIMD_EXCEPT > /* |x| huge or tiny. */ > @@ -115,7 +117,7 @@ float64x2_t VPCS_ATTR V_NAME_D1 (erf) (float64x2_t x) > segfault. */ > uint64x2_t i > = vsubq_u64 (vreinterpretq_u64_f64 (z), vreinterpretq_u64_f64 (shift)); > - i = vbslq_u64 (a_le_max, i, v_u64 (768)); > + i = vbslq_u64 (a_le_max, i, dat->max_idx); > struct entry e = lookup (i); > > float64x2_t r = vsubq_f64 (z, shift); > @@ -125,14 +127,19 @@ float64x2_t VPCS_ATTR V_NAME_D1 (erf) (float64x2_t x) > float64x2_t d2 = vmulq_f64 (d, d); > float64x2_t r2 = vmulq_f64 (r, r); > > + float64x2_t two_over_fifteen_and_fortyfive > + = vld1q_f64 (&dat->two_over_fifteen); > + > /* poly (d, r) = 1 + p1(r) * d + p2(r) * d^2 + ... + p5(r) * d^5. */ > float64x2_t p1 = r; > float64x2_t p2 > = vfmsq_f64 (dat->third, r2, vaddq_f64 (dat->third, dat->third)); > float64x2_t p3 = vmulq_f64 (r, vfmaq_f64 (v_f64 (-0.5), r2, dat->third)); > - float64x2_t p4 = vfmaq_f64 (dat->two_over_five, r2, dat->two_over_fifteen); > + float64x2_t p4 = vfmaq_laneq_f64 (dat->two_over_five, r2, > + two_over_fifteen_and_fortyfive, 0); > p4 = vfmsq_f64 (dat->tenth, r2, p4); > - float64x2_t p5 = vfmaq_f64 (dat->two_over_nine, r2, dat->two_over_fortyfive); > + float64x2_t p5 = vfmaq_laneq_f64 (dat->two_over_nine, r2, > + two_over_fifteen_and_fortyfive, 1); > p5 = vmulq_f64 (r, vfmaq_f64 (vmulq_f64 (v_f64 (0.5), dat->third), r2, p5)); > > float64x2_t p34 = vfmaq_f64 (p3, d, p4); > diff --git a/sysdeps/aarch64/fpu/erfc_advsimd.c b/sysdeps/aarch64/fpu/erfc_advsimd.c > index f1b3bfe830..2f2f755c46 100644 > --- a/sysdeps/aarch64/fpu/erfc_advsimd.c > +++ b/sysdeps/aarch64/fpu/erfc_advsimd.c > @@ -24,8 +24,8 @@ static const struct data > { > uint64x2_t offset, table_scale; > float64x2_t max, shift; > - float64x2_t p20, p40, p41, p42; > - float64x2_t p51, p52; > + float64x2_t p20, p40, p41, p51; > + double p42, p52; > double qr5[2], qr6[2], qr7[2], qr8[2], qr9[2]; > #if WANT_SIMD_EXCEPT > float64x2_t uflow_bound; > @@ -41,9 +41,9 @@ static const struct data > .p20 = V2 (0x1.5555555555555p-2), /* 1/3, used to compute 2/3 and 1/6. */ > .p40 = V2 (-0x1.999999999999ap-4), /* 1/10. */ > .p41 = V2 (-0x1.999999999999ap-2), /* 2/5. */ > - .p42 = V2 (0x1.1111111111111p-3), /* 2/15. */ > + .p42 = 0x1.1111111111111p-3, /* 2/15. */ > .p51 = V2 (-0x1.c71c71c71c71cp-3), /* 2/9. */ > - .p52 = V2 (0x1.6c16c16c16c17p-5), /* 2/45. */ > + .p52 = 0x1.6c16c16c16c17p-5, /* 2/45. */ > /* Qi = (i+1) / i, Ri = -2 * i / ((i+1)*(i+2)), for i = 5, ..., 9. */ > .qr5 = { 0x1.3333333333333p0, -0x1.e79e79e79e79ep-3 }, > .qr6 = { 0x1.2aaaaaaaaaaabp0, -0x1.b6db6db6db6dbp-3 }, > @@ -157,9 +157,10 @@ float64x2_t V_NAME_D1 (erfc) (float64x2_t x) > float64x2_t p1 = r; > float64x2_t p2 = vfmsq_f64 (dat->p20, r2, vaddq_f64 (dat->p20, dat->p20)); > float64x2_t p3 = vmulq_f64 (r, vfmaq_f64 (v_f64 (-0.5), r2, dat->p20)); > - float64x2_t p4 = vfmaq_f64 (dat->p41, r2, dat->p42); > + float64x2_t p42_p52 = vld1q_f64 (&dat->p42); > + float64x2_t p4 = vfmaq_laneq_f64 (dat->p41, r2, p42_p52, 0); > p4 = vfmsq_f64 (dat->p40, r2, p4); > - float64x2_t p5 = vfmaq_f64 (dat->p51, r2, dat->p52); > + float64x2_t p5 = vfmaq_laneq_f64 (dat->p51, r2, p42_p52, 1); > p5 = vmulq_f64 (r, vfmaq_f64 (vmulq_f64 (v_f64 (0.5), dat->p20), r2, p5)); > /* Compute p_i using recurrence relation: > p_{i+2} = (p_i + r * Q_{i+1} * p_{i+1}) * R_{i+1}. */ -- Cheers, Carlos.