On Thu, May 20, 2021 at 2:45 PM Noah Goldstein wrote: > > No bug. This commit makes a few small improvements to > memset-vec-unaligned-erms.S. The changes are 1) only aligning to 64 > instead of 128. Either alignment will perform equally well in a loop > and 128 just increases the odds of having to do an extra iteration > which can be significant overhead for small values. 2) Align some > targets and the loop. 3) Remove an ALU from the alignment process. 4) > Reorder the last 4x VEC so that they are stored after the loop. 5) > Move the condition for leq 8x VEC to before the alignment > process. test-memset and test-wmemset are both passing. > > Signed-off-by: Noah Goldstein > --- > Tests where run on the following CPUs: > > Skylake: https://ark.intel.com/content/www/us/en/ark/products/149091/intel-core-i7-8565u-processor-8m-cache-up-to-4-60-ghz.html > > Icelake: https://ark.intel.com/content/www/us/en/ark/products/196597/intel-core-i7-1065g7-processor-8m-cache-up-to-3-90-ghz.html > > Tigerlake: https://ark.intel.com/content/www/us/en/ark/products/208921/intel-core-i7-1165g7-processor-12m-cache-up-to-4-70-ghz-with-ipu.html > > All times are the geometric mean of N=50. The unit of time is > seconds. > > "Cur" refers to the current implementation > "New" refers to this patches implementation > > Performance data attached in memset-data.pdf > > Some notes on the numbers: > > I only included numbers for the proper VEC_SIZE for the corresponding > cpu. > > skl -> avx2 > icl -> evex > tgl -> evex > > The changes only affect sizes > 2 * VEC_SIZE. The performance > differences in the size <= 2 * VEC_SIZE come from changes in alignment > after linking (i.e ENTRY aligns to 16, but performance can be affected > by alignment % 64 or alignment % 4096) and generally affects > throughput only, not latency (i.e with an lfence to the benchmark loop > the deviations go away). Generally I think they can be ignored (both > positive and negative affects). > > The interesting part of the data is in the medium size range [128, > 1024] where the new implementation has a reasonable speedup. This is > especially pronounced when the more conservative alignment saves a > full loop iteration. The only significant exception is > skylake-avx2-erms case for size = 416, alignment = 416 where the > current implementation is meaningfully faster. I am unsure of the root > cause for this. The skylake-avx2 case only performs a bit worse in > this case which makes me think part of it is code alignment related, > though comparative to the speedup in other size/alignment > configurations it is still a trough. Despite this, I still think the > numbers are overall an improvement. > > As well due to aligning the loop (and possibly slightly more efficient > DSB behavior with the replacement of addq 4 * VEC_SIZE in the loop > with subq -4 * VEC_SIZE) in the non-erms cases there is often a slight > improvement to the main loop for large sizes. > > .../multiarch/memset-vec-unaligned-erms.S | 50 +++++++++++-------- > 1 file changed, 28 insertions(+), 22 deletions(-) > > diff --git a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S > index 08cfa49bd1..ff196844a0 100644 > --- a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S > +++ b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S > @@ -173,17 +173,22 @@ ENTRY (MEMSET_SYMBOL (__memset, unaligned_erms)) > VMOVU %VEC(0), (%rdi) > VZEROUPPER_RETURN > > + .p2align 4 > L(stosb_more_2x_vec): > cmp __x86_rep_stosb_threshold(%rip), %RDX_LP > ja L(stosb) > +#else > + .p2align 4 > #endif > L(more_2x_vec): > - cmpq $(VEC_SIZE * 4), %rdx > - ja L(loop_start) > + /* Stores to first 2x VEC before cmp as any path forward will > + require it. */ > VMOVU %VEC(0), (%rdi) > VMOVU %VEC(0), VEC_SIZE(%rdi) > - VMOVU %VEC(0), -VEC_SIZE(%rdi,%rdx) > + cmpq $(VEC_SIZE * 4), %rdx > + ja L(loop_start) > VMOVU %VEC(0), -(VEC_SIZE * 2)(%rdi,%rdx) > + VMOVU %VEC(0), -VEC_SIZE(%rdi,%rdx) > L(return): > #if VEC_SIZE > 16 > ZERO_UPPER_VEC_REGISTERS_RETURN > @@ -192,28 +197,29 @@ L(return): > #endif > > L(loop_start): > - leaq (VEC_SIZE * 4)(%rdi), %rcx > - VMOVU %VEC(0), (%rdi) > - andq $-(VEC_SIZE * 4), %rcx > - VMOVU %VEC(0), -VEC_SIZE(%rdi,%rdx) > - VMOVU %VEC(0), VEC_SIZE(%rdi) > - VMOVU %VEC(0), -(VEC_SIZE * 2)(%rdi,%rdx) > VMOVU %VEC(0), (VEC_SIZE * 2)(%rdi) > - VMOVU %VEC(0), -(VEC_SIZE * 3)(%rdi,%rdx) > VMOVU %VEC(0), (VEC_SIZE * 3)(%rdi) > - VMOVU %VEC(0), -(VEC_SIZE * 4)(%rdi,%rdx) > - addq %rdi, %rdx > - andq $-(VEC_SIZE * 4), %rdx > - cmpq %rdx, %rcx > - je L(return) > + cmpq $(VEC_SIZE * 8), %rdx > + jbe L(loop_end) > + andq $-(VEC_SIZE * 2), %rdi > + subq $-(VEC_SIZE * 4), %rdi > + leaq -(VEC_SIZE * 4)(%rax, %rdx), %rcx > + .p2align 4 > L(loop): > - VMOVA %VEC(0), (%rcx) > - VMOVA %VEC(0), VEC_SIZE(%rcx) > - VMOVA %VEC(0), (VEC_SIZE * 2)(%rcx) > - VMOVA %VEC(0), (VEC_SIZE * 3)(%rcx) > - addq $(VEC_SIZE * 4), %rcx > - cmpq %rcx, %rdx > - jne L(loop) > + VMOVA %VEC(0), (%rdi) > + VMOVA %VEC(0), VEC_SIZE(%rdi) > + VMOVA %VEC(0), (VEC_SIZE * 2)(%rdi) > + VMOVA %VEC(0), (VEC_SIZE * 3)(%rdi) > + subq $-(VEC_SIZE * 4), %rdi > + cmpq %rcx, %rdi > + jb L(loop) > +L(loop_end): > + /* NB: rax is set as ptr in MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. > + rdx as length is also unchanged. */ > + VMOVU %VEC(0), -(VEC_SIZE * 4)(%rax, %rdx) > + VMOVU %VEC(0), -(VEC_SIZE * 3)(%rax, %rdx) > + VMOVU %VEC(0), -(VEC_SIZE * 2)(%rax, %rdx) > + VMOVU %VEC(0), -VEC_SIZE(%rax, %rdx) > VZEROUPPER_SHORT_RETURN > > .p2align 4 > -- > 2.25.1 >