From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by sourceware.org (Postfix) with ESMTPS id 033AB3987C1C for ; Thu, 20 May 2021 20:03:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 033AB3987C1C Received: by mail-pj1-x1029.google.com with SMTP id b13-20020a17090a8c8db029015cd97baea9so5861637pjo.0 for ; Thu, 20 May 2021 13:03:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=naiI+sFvMadtJoHAzpuJIqG/+/+AKS7BsqIfTs9xCeA=; b=g/uzRvgUKU0PiOHzFNL4dTxeouYqOYb8l6A7eq+2BtBAHoqUHSTAlA1Uqexa5WyF5v +8SckaWPNfKpF+U+ql+q8zLHDLiLNVRtKsYVB3uaDulXGYtdImsueNZbPhCem0kYe/zc wWKS5RguwwqGNlHSRgEGmfKHn98LYc0LiaBlN1Ghnl6gxjUXL5XfhcLxfJbxY5KvrfNW vECJJ2AtFzbCITbFN3r+Nv6HeAp3s4QfT7r6m37XmW9x2NZAEvij0fzVz/LFG6DU8BLm ntAR1NudfW7KSnTtgPOsOWw/8YHPusDEMP05PvRbHk02GQ8QSm+htiMF/Ym1qEN7p9Y+ x++A== X-Gm-Message-State: AOAM530pdhoTMbRSPyPvqfmYtrOdzbC2OjF7I1VyhxsDBohxLbaBqNCE ZMLxwtPOeg1aWS1ps5i7zCJFg91J/1CVWzRjvQDLubrt628= X-Google-Smtp-Source: ABdhPJwgKlpYz1TISKUjX++ezHaT5B7cU8t4VrnQcPw/Lew1aApUxu0vtiksFwKN9IFl9qjh/osD6ewiAbzINFp+vV8= X-Received: by 2002:a17:902:6b8c:b029:ea:f54f:c330 with SMTP id p12-20020a1709026b8cb02900eaf54fc330mr7926779plk.10.1621541001035; Thu, 20 May 2021 13:03:21 -0700 (PDT) MIME-Version: 1.0 References: <20210520184404.2901975-1-goldstein.w.n@gmail.com> In-Reply-To: From: Noah Goldstein Date: Thu, 20 May 2021 16:03:10 -0400 Message-ID: Subject: Re: [PATCH v1] x86: Improve memset-vec-unaligned-erms.S To: "H.J. Lu" Cc: GNU C Library , "Carlos O'Donell" Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-10.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 May 2021 20:03:24 -0000 On Thu, May 20, 2021 at 3:40 PM H.J. Lu wrote: > > On Thu, May 20, 2021 at 11:45 AM Noah Goldstein wrote: > > > > No bug. This commit makes a few small improvements to > > memset-vec-unaligned-erms.S. The changes are 1) only aligning to 64 > > instead of 128. Either alignment will perform equally well in a loop > > and 128 just increases the odds of having to do an extra iteration > > which can be significant overhead for small values. 2) Align some > > targets and the loop. 3) Remove an ALU from the alignment process. 4) > > Reorder the last 4x VEC so that they are stored after the loop. 5) > > Move the condition for leq 8x VEC to before the alignment > > process. test-memset and test-wmemset are both passing. > > > > Signed-off-by: Noah Goldstein > > --- > > Tests where run on the following CPUs: > > > > Skylake: https://ark.intel.com/content/www/us/en/ark/products/149091/intel-core-i7-8565u-processor-8m-cache-up-to-4-60-ghz.html > > > > Icelake: https://ark.intel.com/content/www/us/en/ark/products/196597/intel-core-i7-1065g7-processor-8m-cache-up-to-3-90-ghz.html > > > > Tigerlake: https://ark.intel.com/content/www/us/en/ark/products/208921/intel-core-i7-1165g7-processor-12m-cache-up-to-4-70-ghz-with-ipu.html > > > > All times are the geometric mean of N=50. The unit of time is > > seconds. > > > > "Cur" refers to the current implementation > > "New" refers to this patches implementation > > > > Performance data attached in memset-data.pdf > > > > Some notes on the numbers: > > > > I only included numbers for the proper VEC_SIZE for the corresponding > > cpu. > > > > skl -> avx2 > > icl -> evex > > tgl -> evex > > > > The changes only affect sizes > 2 * VEC_SIZE. The performance > > differences in the size <= 2 * VEC_SIZE come from changes in alignment > > after linking (i.e ENTRY aligns to 16, but performance can be affected > > by alignment % 64 or alignment % 4096) and generally affects > > throughput only, not latency (i.e with an lfence to the benchmark loop > > the deviations go away). Generally I think they can be ignored (both > > positive and negative affects). > > > > The interesting part of the data is in the medium size range [128, > > 1024] where the new implementation has a reasonable speedup. This is > > especially pronounced when the more conservative alignment saves a > > full loop iteration. The only significant exception is > > skylake-avx2-erms case for size = 416, alignment = 416 where the > > current implementation is meaningfully faster. I am unsure of the root > > cause for this. The skylake-avx2 case only performs a bit worse in > > this case which makes me think part of it is code alignment related, > > though comparative to the speedup in other size/alignment > > configurations it is still a trough. Despite this, I still think the > > numbers are overall an improvement. > > > > As well due to aligning the loop (and possibly slightly more efficient > > DSB behavior with the replacement of addq 4 * VEC_SIZE in the loop > > with subq -4 * VEC_SIZE) in the non-erms cases there is often a slight > > improvement to the main loop for large sizes. > > > > .../multiarch/memset-vec-unaligned-erms.S | 50 +++++++++++-------- > > 1 file changed, 28 insertions(+), 22 deletions(-) > > > > diff --git a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S > > index 08cfa49bd1..ff196844a0 100644 > > --- a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S > > +++ b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S > > @@ -173,17 +173,22 @@ ENTRY (MEMSET_SYMBOL (__memset, unaligned_erms)) > > VMOVU %VEC(0), (%rdi) > > VZEROUPPER_RETURN > > > > + .p2align 4 > > L(stosb_more_2x_vec): > > cmp __x86_rep_stosb_threshold(%rip), %RDX_LP > > ja L(stosb) > > +#else > > + .p2align 4 > > #endif > > L(more_2x_vec): > > - cmpq $(VEC_SIZE * 4), %rdx > > - ja L(loop_start) > > + /* Stores to first 2x VEC before cmp as any path forward will > > + require it. */ > > VMOVU %VEC(0), (%rdi) > > VMOVU %VEC(0), VEC_SIZE(%rdi) > > - VMOVU %VEC(0), -VEC_SIZE(%rdi,%rdx) > > + cmpq $(VEC_SIZE * 4), %rdx > > + ja L(loop_start) > > VMOVU %VEC(0), -(VEC_SIZE * 2)(%rdi,%rdx) > > + VMOVU %VEC(0), -VEC_SIZE(%rdi,%rdx) > > L(return): > > #if VEC_SIZE > 16 > > ZERO_UPPER_VEC_REGISTERS_RETURN > > @@ -192,28 +197,29 @@ L(return): > > #endif > > > > L(loop_start): > > - leaq (VEC_SIZE * 4)(%rdi), %rcx > > - VMOVU %VEC(0), (%rdi) > > - andq $-(VEC_SIZE * 4), %rcx > > - VMOVU %VEC(0), -VEC_SIZE(%rdi,%rdx) > > - VMOVU %VEC(0), VEC_SIZE(%rdi) > > - VMOVU %VEC(0), -(VEC_SIZE * 2)(%rdi,%rdx) > > VMOVU %VEC(0), (VEC_SIZE * 2)(%rdi) > > - VMOVU %VEC(0), -(VEC_SIZE * 3)(%rdi,%rdx) > > VMOVU %VEC(0), (VEC_SIZE * 3)(%rdi) > > - VMOVU %VEC(0), -(VEC_SIZE * 4)(%rdi,%rdx) > > - addq %rdi, %rdx > > - andq $-(VEC_SIZE * 4), %rdx > > - cmpq %rdx, %rcx > > - je L(return) > > + cmpq $(VEC_SIZE * 8), %rdx > > + jbe L(loop_end) > > + andq $-(VEC_SIZE * 2), %rdi > > + subq $-(VEC_SIZE * 4), %rdi > > + leaq -(VEC_SIZE * 4)(%rax, %rdx), %rcx > > + .p2align 4 > > L(loop): > > - VMOVA %VEC(0), (%rcx) > > - VMOVA %VEC(0), VEC_SIZE(%rcx) > > - VMOVA %VEC(0), (VEC_SIZE * 2)(%rcx) > > - VMOVA %VEC(0), (VEC_SIZE * 3)(%rcx) > > - addq $(VEC_SIZE * 4), %rcx > > - cmpq %rcx, %rdx > > - jne L(loop) > > + VMOVA %VEC(0), (%rdi) > > + VMOVA %VEC(0), VEC_SIZE(%rdi) > > + VMOVA %VEC(0), (VEC_SIZE * 2)(%rdi) > > + VMOVA %VEC(0), (VEC_SIZE * 3)(%rdi) > > + subq $-(VEC_SIZE * 4), %rdi > > + cmpq %rcx, %rdi > > + jb L(loop) > > +L(loop_end): > > + /* NB: rax is set as ptr in MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. > > + rdx as length is also unchanged. */ > > + VMOVU %VEC(0), -(VEC_SIZE * 4)(%rax, %rdx) > > + VMOVU %VEC(0), -(VEC_SIZE * 3)(%rax, %rdx) > > + VMOVU %VEC(0), -(VEC_SIZE * 2)(%rax, %rdx) > > + VMOVU %VEC(0), -VEC_SIZE(%rax, %rdx) > > VZEROUPPER_SHORT_RETURN > > > > .p2align 4 > > -- > > 2.25.1 > > > > LGTM. Awesome! For future patches do you prefer performance numbers like this or raw text? Or some other alternative? > > Reviewed-by: H.J. Lu > > Thanks. > > -- > H.J.