* V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130]
@ 2021-01-04 15:17 H.J. Lu
2021-01-04 15:22 ` Florian Weimer
0 siblings, 1 reply; 5+ messages in thread
From: H.J. Lu @ 2021-01-04 15:17 UTC (permalink / raw)
To: libc-alpha
When copying with "rep movsb", if the distance between source and
destination is N*4GB + [1..63] with N >= 0, performance may be very
slow. This patch updates memmove-vec-unaligned-erms.S for AVX and
AVX512 versions with the distance in RCX:
cmpl $63, %ecx
// Don't use "rep movsb" if ECX <= 63
jbe L(Don't use rep movsb")
Use "rep movsb"
Benchtests data with bench-memcpy, bench-memcpy-large, bench-memcpy-random
and bench-memcpy-walk on Skylake, Ice Lake and Tiger Lake show that its
performance impact is within noise range as "rep movsb" is only used for
data size >= 4KB.
Changes from V1:
1. Check distance of N*4GB + [1..63] with N >= 0 instead of [1..63].
---
.../multiarch/memmove-vec-unaligned-erms.S | 21 +++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S
index 7d54095f04..0980c95378 100644
--- a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S
+++ b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S
@@ -56,6 +56,13 @@
# endif
#endif
+/* Avoid short distance rep movsb only with non-SSE vector. */
+#ifndef AVOID_SHORT_DISTANCE_REP_MOVSB
+# define AVOID_SHORT_DISTANCE_REP_MOVSB (VEC_SIZE > 16)
+#else
+# define AVOID_SHORT_DISTANCE_REP_MOVSB 0
+#endif
+
#ifndef PREFETCH
# define PREFETCH(addr) prefetcht0 addr
#endif
@@ -243,7 +250,21 @@ L(movsb):
cmpq %r9, %rdi
/* Avoid slow backward REP MOVSB. */
jb L(more_8x_vec_backward)
+# if AVOID_SHORT_DISTANCE_REP_MOVSB
+ movq %rdi, %rcx
+ subq %rsi, %rcx
+ jmp 2f
+# endif
1:
+# if AVOID_SHORT_DISTANCE_REP_MOVSB
+ movq %rsi, %rcx
+ subq %rdi, %rcx
+2:
+/* Avoid "rep movsb" if RCX, the distance between source and destination,
+ is N*4GB + [1..63] with N >= 0. */
+ cmpl $63, %ecx
+ jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */
+# endif
mov %RDX_LP, %RCX_LP
rep movsb
L(nop):
--
2.29.2
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130]
2021-01-04 15:17 V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130] H.J. Lu
@ 2021-01-04 15:22 ` Florian Weimer
2021-01-04 15:27 ` H.J. Lu
0 siblings, 1 reply; 5+ messages in thread
From: Florian Weimer @ 2021-01-04 15:22 UTC (permalink / raw)
To: H.J. Lu via Libc-alpha
* H. J. Lu via Libc-alpha:
> 1:
> +# if AVOID_SHORT_DISTANCE_REP_MOVSB
> + movq %rsi, %rcx
> + subq %rdi, %rcx
> +2:
> +/* Avoid "rep movsb" if RCX, the distance between source and destination,
> + is N*4GB + [1..63] with N >= 0. */
> + cmpl $63, %ecx
> + jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */
> +# endif
> mov %RDX_LP, %RCX_LP
> rep movsb
> L(nop):
Why not use _LP names here? I think the %ecx comparison at least can
give false results on x86-64 (64-bit).
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130]
2021-01-04 15:22 ` Florian Weimer
@ 2021-01-04 15:27 ` H.J. Lu
2021-01-04 15:47 ` Florian Weimer
0 siblings, 1 reply; 5+ messages in thread
From: H.J. Lu @ 2021-01-04 15:27 UTC (permalink / raw)
To: Florian Weimer; +Cc: H.J. Lu via Libc-alpha
On Mon, Jan 4, 2021 at 7:22 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * H. J. Lu via Libc-alpha:
>
> > 1:
> > +# if AVOID_SHORT_DISTANCE_REP_MOVSB
> > + movq %rsi, %rcx
> > + subq %rdi, %rcx
> > +2:
> > +/* Avoid "rep movsb" if RCX, the distance between source and destination,
> > + is N*4GB + [1..63] with N >= 0. */
> > + cmpl $63, %ecx
> > + jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */
> > +# endif
> > mov %RDX_LP, %RCX_LP
> > rep movsb
> > L(nop):
>
> Why not use _LP names here? I think the %ecx comparison at least can
> give false results on x86-64 (64-bit).
>
This is done on purpose since we want to avoid "rep movsb" for distances of
N*4GB + [1..63] with N >= 0 which include 0x100000003.
--
H.J.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130]
2021-01-04 15:27 ` H.J. Lu
@ 2021-01-04 15:47 ` Florian Weimer
2021-01-04 15:54 ` H.J. Lu
0 siblings, 1 reply; 5+ messages in thread
From: Florian Weimer @ 2021-01-04 15:47 UTC (permalink / raw)
To: H.J. Lu; +Cc: H.J. Lu via Libc-alpha
* H. J. Lu:
> On Mon, Jan 4, 2021 at 7:22 AM Florian Weimer <fweimer@redhat.com> wrote:
>>
>> * H. J. Lu via Libc-alpha:
>>
>> > 1:
>> > +# if AVOID_SHORT_DISTANCE_REP_MOVSB
>> > + movq %rsi, %rcx
>> > + subq %rdi, %rcx
>> > +2:
>> > +/* Avoid "rep movsb" if RCX, the distance between source and destination,
>> > + is N*4GB + [1..63] with N >= 0. */
>> > + cmpl $63, %ecx
>> > + jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */
>> > +# endif
>> > mov %RDX_LP, %RCX_LP
>> > rep movsb
>> > L(nop):
>>
>> Why not use _LP names here? I think the %ecx comparison at least can
>> give false results on x86-64 (64-bit).
>>
>
> This is done on purpose since we want to avoid "rep movsb" for distances of
> N*4GB + [1..63] with N >= 0 which include 0x100000003.
Ah, and the comment is quite clear (the commit subject less so).
I tried to make sense of the assembler code, and I think the change is
okay because L(movsb) is only reached when there is more to copy than
twice the vector size.
Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130]
2021-01-04 15:47 ` Florian Weimer
@ 2021-01-04 15:54 ` H.J. Lu
0 siblings, 0 replies; 5+ messages in thread
From: H.J. Lu @ 2021-01-04 15:54 UTC (permalink / raw)
To: Florian Weimer, Libc-stable Mailing List; +Cc: H.J. Lu via Libc-alpha
On Mon, Jan 4, 2021 at 7:47 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * H. J. Lu:
>
> > On Mon, Jan 4, 2021 at 7:22 AM Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >> * H. J. Lu via Libc-alpha:
> >>
> >> > 1:
> >> > +# if AVOID_SHORT_DISTANCE_REP_MOVSB
> >> > + movq %rsi, %rcx
> >> > + subq %rdi, %rcx
> >> > +2:
> >> > +/* Avoid "rep movsb" if RCX, the distance between source and destination,
> >> > + is N*4GB + [1..63] with N >= 0. */
> >> > + cmpl $63, %ecx
> >> > + jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */
> >> > +# endif
> >> > mov %RDX_LP, %RCX_LP
> >> > rep movsb
> >> > L(nop):
> >>
> >> Why not use _LP names here? I think the %ecx comparison at least can
> >> give false results on x86-64 (64-bit).
> >>
> >
> > This is done on purpose since we want to avoid "rep movsb" for distances of
> > N*4GB + [1..63] with N >= 0 which include 0x100000003.
>
> Ah, and the comment is quite clear (the commit subject less so).
It isn't easy to describe it with so few letters.
> I tried to make sense of the assembler code, and I think the change is
> okay because L(movsb) is only reached when there is more to copy than
> twice the vector size.
>
That is correct. I am checking it in. I will backport it to release branches
next week.
Thanks.
--
H.J.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-01-04 15:55 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-04 15:17 V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130] H.J. Lu
2021-01-04 15:22 ` Florian Weimer
2021-01-04 15:27 ` H.J. Lu
2021-01-04 15:47 ` Florian Weimer
2021-01-04 15:54 ` H.J. Lu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).