Re: memcpy performance on skylake server

["Ji, Cheng" <jicheng1017@gmail.com>]

Thanks for the information. We did some quick experiments. Indeed, using
normal temporal stores is ~20% faster than using non-temporal stores in
this case.

Cheng

On Wed, Jul 14, 2021 at 9:27 PM H.J. Lu <hjl.tools@gmail.com> wrote:

> On Wed, Jul 14, 2021 at 5:58 AM Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
> >
> >
> >
> > On 06/07/2021 05:17, Ji, Cheng via Libc-help wrote:
> > > Hello,
> > >
> > > I found that memcpy is slower on skylake server CPUs during our
> > > optimization work, and I can't really explain what we got and need some
> > > guidance here.
> > >
> > > The problem is that memcpy is noticeably slower than a simple for loop
> when
> > > copying large chunks of data. This genuinely sounds like an amateur
> mistake
> > > in our testing code but here's what we have tried:
> > >
> > > * The test data is large enough: 1GB.
> > > * We noticed a change quite a while ago regarding skylake and AVX512:
> > >
> https://patchwork.ozlabs.org/project/glibc/patch/20170418183712.GA22211@intel.com/
> > > * We updated glibc from 2.17 to the latest 2.33, we did see memcpy is
> 5%
> > > faster but still slower than a simple loop.
> > > * We tested on multiple bare metal machines with different cpus: Xeon
> Gold
> > > 6132, Gold 6252, Silver 4114, as well as a virtual machine on google
> cloud,
> > > the result is reproducible.
> > > * On an older generation Xeon E5-2630 v3, memcpy is about 50% faster
> than
> > > the simple loop. On my desktop (i7-7700k) memcpy is also significantly
> > > faster.
> > > * numactl is used to ensure everything is running on a single core.
> > > * The code is compiled by gcc 10.3
> > >
> > > The numbers on a Xeon Gold 6132, with glibc 2.33:
> > > simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> > > simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> > > simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> > > simple_copy 3.68 seconds, 5.44 GiB/s 5.71 GB/s
> > >
> > > The result is worse with system provided glibc 2.17:
> > > simple_memcpy 4.38 seconds, 4.57 GiB/s 4.79 GB/s
> > > simple_copy 3.68 seconds, 5.43 GiB/s 5.70 GB/s
> > > simple_memcpy 4.38 seconds, 4.56 GiB/s 4.78 GB/s
> > > simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> > >
> > >
> > > The code to generate this result (compiled with g++ -O2 -g, run with:
> numactl
> > > --membind 0 --physcpubind 0 -- ./a.out)
> > > =====
> > >
> > > #include <chrono>
> > > #include <cstring>
> > > #include <functional>
> > > #include <string>
> > > #include <vector>
> > >
> > > class TestCase {
> > >     using clock_t = std::chrono::high_resolution_clock;
> > >     using sec_t = std::chrono::duration<double, std::ratio<1>>;
> > >
> > > public:
> > >     static constexpr size_t NUM_VALUES = 128 * (1 << 20); // 128
> million *
> > > 8 bytes = 1GiB
> > >
> > >     void init() {
> > >         vals_.resize(NUM_VALUES);
> > >         for (size_t i = 0; i < NUM_VALUES; ++i) {
> > >             vals_[i] = i;
> > >         }
> > >         dest_.resize(NUM_VALUES);
> > >     }
> > >
> > >     void run(std::string name, std::function<void(const int64_t *,
> int64_t
> > > *, size_t)> &&func) {
> > >         // ignore the result from first run
> > >         func(vals_.data(), dest_.data(), vals_.size());
> > >         constexpr size_t count = 20;
> > >         auto start = clock_t::now();
> > >         for (size_t i = 0; i < count; ++i) {
> > >             func(vals_.data(), dest_.data(), vals_.size());
> > >         }
> > >         auto end = clock_t::now();
> > >         double duration =
> > > std::chrono::duration_cast<sec_t>(end-start).count();
> > >         printf("%s %.2f seconds, %.2f GiB/s, %.2f GB/s\n", name.data(),
> > > duration,
> > >                sizeof(int64_t) * NUM_VALUES / double(1 << 30) * count /
> > > duration,
> > >                sizeof(int64_t) * NUM_VALUES / double(1e9) * count /
> > > duration);
> > >     }
> > >
> > > private:
> > >     std::vector<int64_t> vals_;
> > >     std::vector<int64_t> dest_;
> > > };
> > >
> > > void simple_memcpy(const int64_t *src, int64_t *dest, size_t n) {
> > >     memcpy(dest, src, n * sizeof(int64_t));
> > > }
> > >
> > > void simple_copy(const int64_t *src, int64_t *dest, size_t n) {
> > >     for (size_t i = 0; i < n; ++i) {
> > >         dest[i] = src[i];
> > >     }
> > > }
> > >
> > > int main(int, char **) {
> > >     TestCase c;
> > >     c.init();
> > >
> > >     c.run("simple_memcpy", simple_memcpy);
> > >     c.run("simple_copy", simple_copy);
> > >     c.run("simple_memcpy", simple_memcpy);
> > >     c.run("simple_copy", simple_copy);
> > > }
> > >
> > > =====
> > >
> > > The assembly of simple_copy generated by gcc is very simple:
> > > Dump of assembler code for function _Z11simple_copyPKlPlm:
> > >    0x0000000000401440 <+0>:     mov    %rdx,%rcx
> > >    0x0000000000401443 <+3>:     test   %rdx,%rdx
> > >    0x0000000000401446 <+6>:     je     0x401460
> <_Z11simple_copyPKlPlm+32>
> > >    0x0000000000401448 <+8>:     xor    %eax,%eax
> > >    0x000000000040144a <+10>:    nopw   0x0(%rax,%rax,1)
> > >    0x0000000000401450 <+16>:    mov    (%rdi,%rax,8),%rdx
> > >    0x0000000000401454 <+20>:    mov    %rdx,(%rsi,%rax,8)
> > >    0x0000000000401458 <+24>:    inc    %rax
> > >    0x000000000040145b <+27>:    cmp    %rax,%rcx
> > >    0x000000000040145e <+30>:    jne    0x401450
> <_Z11simple_copyPKlPlm+16>
> > >    0x0000000000401460 <+32>:    retq
> > >
> > > When compiling with -O3, gcc vectorized the loop using xmm0, the
> > > simple_loop is around 1% faster.
> >
> > Usually differences of that magnitude falls either in noise or may be
> something
> > related to OS jitter.
> >
> > >
> > > I took a brief look at the glibc source code. Though I don't have
> enough
> > > knowledge to understand it yet, I'm curious about the underlying
> mechanism.
> > > Thanks.
> >
> > H.J, do you have any idea what might be happening here?
>
> From Intel optimization guide:
>
> 2.2.2 Non-Temporal Stores on Skylake Server Microarchitecture
> Because of the change in the size of each bank of last level cache on
> Skylake Server microarchitecture, if
> an application, library, or driver only considers the last level cache
> to determine the size of on-chip cacheper-core, it may see a reduction
> with Skylake Server microarchitecture and may use non-temporal store
> with smaller blocks of memory writes. Since non-temporal stores evict
> cache lines back to memory, this
> may result in an increase in the number of subsequent cache misses and
> memory bandwidth demands
> on Skylake Server microarchitecture, compared to the previous Intel
> Xeon processor family.
> Also, because of a change in the handling of accesses resulting from
> non-temporal stores by Skylake
> Server microarchitecture, the resources within each core remain busy
> for a longer duration compared to
> similar accesses on the previous Intel Xeon processor family. As a
> result, if a series of such instructions
> are executed, there is a potential that the processor may run out of
> resources and stall, thus limiting the
> memory write bandwidth from each core.
> The increase in cache misses due to overuse of non-temporal stores and
> the limit on the memory write
> bandwidth per core for non-temporal stores may result in reduced
> performance for some applications.
>
> --
> H.J.
>