memcpy performance on skylake server

memcpy performance on skylake server

[Ji, Cheng]

Hello,

I found that memcpy is slower on skylake server CPUs during our
optimization work, and I can't really explain what we got and need some
guidance here.

The problem is that memcpy is noticeably slower than a simple for loop when
copying large chunks of data. This genuinely sounds like an amateur mistake
in our testing code but here's what we have tried:

* The test data is large enough: 1GB.
* We noticed a change quite a while ago regarding skylake and AVX512:
https://patchwork.ozlabs.org/project/glibc/patch/20170418183712.GA22211@intel.com/
* We updated glibc from 2.17 to the latest 2.33, we did see memcpy is 5%
faster but still slower than a simple loop.
* We tested on multiple bare metal machines with different cpus: Xeon Gold
6132, Gold 6252, Silver 4114, as well as a virtual machine on google cloud,
the result is reproducible.
* On an older generation Xeon E5-2630 v3, memcpy is about 50% faster than
the simple loop. On my desktop (i7-7700k) memcpy is also significantly
faster.
* numactl is used to ensure everything is running on a single core.
* The code is compiled by gcc 10.3

The numbers on a Xeon Gold 6132, with glibc 2.33:
simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
simple_copy 3.68 seconds, 5.44 GiB/s 5.71 GB/s

The result is worse with system provided glibc 2.17:
simple_memcpy 4.38 seconds, 4.57 GiB/s 4.79 GB/s
simple_copy 3.68 seconds, 5.43 GiB/s 5.70 GB/s
simple_memcpy 4.38 seconds, 4.56 GiB/s 4.78 GB/s
simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s


The code to generate this result (compiled with g++ -O2 -g, run with: numactl
--membind 0 --physcpubind 0 -- ./a.out)
=====

#include <chrono>
#include <cstring>
#include <functional>
#include <string>
#include <vector>

class TestCase {
    using clock_t = std::chrono::high_resolution_clock;
    using sec_t = std::chrono::duration<double, std::ratio<1>>;

public:
    static constexpr size_t NUM_VALUES = 128 * (1 << 20); // 128 million *
8 bytes = 1GiB

    void init() {
        vals_.resize(NUM_VALUES);
        for (size_t i = 0; i < NUM_VALUES; ++i) {
            vals_[i] = i;
        }
        dest_.resize(NUM_VALUES);
    }

    void run(std::string name, std::function<void(const int64_t *, int64_t
*, size_t)> &&func) {
        // ignore the result from first run
        func(vals_.data(), dest_.data(), vals_.size());
        constexpr size_t count = 20;
        auto start = clock_t::now();
        for (size_t i = 0; i < count; ++i) {
            func(vals_.data(), dest_.data(), vals_.size());
        }
        auto end = clock_t::now();
        double duration =
std::chrono::duration_cast<sec_t>(end-start).count();
        printf("%s %.2f seconds, %.2f GiB/s, %.2f GB/s\n", name.data(),
duration,
               sizeof(int64_t) * NUM_VALUES / double(1 << 30) * count /
duration,
               sizeof(int64_t) * NUM_VALUES / double(1e9) * count /
duration);
    }

private:
    std::vector<int64_t> vals_;
    std::vector<int64_t> dest_;
};

void simple_memcpy(const int64_t *src, int64_t *dest, size_t n) {
    memcpy(dest, src, n * sizeof(int64_t));
}

void simple_copy(const int64_t *src, int64_t *dest, size_t n) {
    for (size_t i = 0; i < n; ++i) {
        dest[i] = src[i];
    }
}

int main(int, char **) {
    TestCase c;
    c.init();

    c.run("simple_memcpy", simple_memcpy);
    c.run("simple_copy", simple_copy);
    c.run("simple_memcpy", simple_memcpy);
    c.run("simple_copy", simple_copy);
}

=====

The assembly of simple_copy generated by gcc is very simple:
Dump of assembler code for function _Z11simple_copyPKlPlm:
   0x0000000000401440 <+0>:     mov    %rdx,%rcx
   0x0000000000401443 <+3>:     test   %rdx,%rdx
   0x0000000000401446 <+6>:     je     0x401460 <_Z11simple_copyPKlPlm+32>
   0x0000000000401448 <+8>:     xor    %eax,%eax
   0x000000000040144a <+10>:    nopw   0x0(%rax,%rax,1)
   0x0000000000401450 <+16>:    mov    (%rdi,%rax,8),%rdx
   0x0000000000401454 <+20>:    mov    %rdx,(%rsi,%rax,8)
   0x0000000000401458 <+24>:    inc    %rax
   0x000000000040145b <+27>:    cmp    %rax,%rcx
   0x000000000040145e <+30>:    jne    0x401450 <_Z11simple_copyPKlPlm+16>
   0x0000000000401460 <+32>:    retq

When compiling with -O3, gcc vectorized the loop using xmm0, the
simple_loop is around 1% faster.

I took a brief look at the glibc source code. Though I don't have enough
knowledge to understand it yet, I'm curious about the underlying mechanism.
Thanks.

Cheng

Re: memcpy performance on skylake server

[Adhemerval Zanella]

On 06/07/2021 05:17, Ji, Cheng via Libc-help wrote:
> Hello,
> 
> I found that memcpy is slower on skylake server CPUs during our
> optimization work, and I can't really explain what we got and need some
> guidance here.
> 
> The problem is that memcpy is noticeably slower than a simple for loop when
> copying large chunks of data. This genuinely sounds like an amateur mistake
> in our testing code but here's what we have tried:
> 
> * The test data is large enough: 1GB.
> * We noticed a change quite a while ago regarding skylake and AVX512:
> https://patchwork.ozlabs.org/project/glibc/patch/20170418183712.GA22211@intel.com/
> * We updated glibc from 2.17 to the latest 2.33, we did see memcpy is 5%
> faster but still slower than a simple loop.
> * We tested on multiple bare metal machines with different cpus: Xeon Gold
> 6132, Gold 6252, Silver 4114, as well as a virtual machine on google cloud,
> the result is reproducible.
> * On an older generation Xeon E5-2630 v3, memcpy is about 50% faster than
> the simple loop. On my desktop (i7-7700k) memcpy is also significantly
> faster.
> * numactl is used to ensure everything is running on a single core.
> * The code is compiled by gcc 10.3
> 
> The numbers on a Xeon Gold 6132, with glibc 2.33:
> simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> simple_copy 3.68 seconds, 5.44 GiB/s 5.71 GB/s
> 
> The result is worse with system provided glibc 2.17:
> simple_memcpy 4.38 seconds, 4.57 GiB/s 4.79 GB/s
> simple_copy 3.68 seconds, 5.43 GiB/s 5.70 GB/s
> simple_memcpy 4.38 seconds, 4.56 GiB/s 4.78 GB/s
> simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> 
> 
> The code to generate this result (compiled with g++ -O2 -g, run with: numactl
> --membind 0 --physcpubind 0 -- ./a.out)
> =====
> 
> #include <chrono>
> #include <cstring>
> #include <functional>
> #include <string>
> #include <vector>
> 
> class TestCase {
>     using clock_t = std::chrono::high_resolution_clock;
>     using sec_t = std::chrono::duration<double, std::ratio<1>>;
> 
> public:
>     static constexpr size_t NUM_VALUES = 128 * (1 << 20); // 128 million *
> 8 bytes = 1GiB
> 
>     void init() {
>         vals_.resize(NUM_VALUES);
>         for (size_t i = 0; i < NUM_VALUES; ++i) {
>             vals_[i] = i;
>         }
>         dest_.resize(NUM_VALUES);
>     }
> 
>     void run(std::string name, std::function<void(const int64_t *, int64_t
> *, size_t)> &&func) {
>         // ignore the result from first run
>         func(vals_.data(), dest_.data(), vals_.size());
>         constexpr size_t count = 20;
>         auto start = clock_t::now();
>         for (size_t i = 0; i < count; ++i) {
>             func(vals_.data(), dest_.data(), vals_.size());
>         }
>         auto end = clock_t::now();
>         double duration =
> std::chrono::duration_cast<sec_t>(end-start).count();
>         printf("%s %.2f seconds, %.2f GiB/s, %.2f GB/s\n", name.data(),
> duration,
>                sizeof(int64_t) * NUM_VALUES / double(1 << 30) * count /
> duration,
>                sizeof(int64_t) * NUM_VALUES / double(1e9) * count /
> duration);
>     }
> 
> private:
>     std::vector<int64_t> vals_;
>     std::vector<int64_t> dest_;
> };
> 
> void simple_memcpy(const int64_t *src, int64_t *dest, size_t n) {
>     memcpy(dest, src, n * sizeof(int64_t));
> }
> 
> void simple_copy(const int64_t *src, int64_t *dest, size_t n) {
>     for (size_t i = 0; i < n; ++i) {
>         dest[i] = src[i];
>     }
> }
> 
> int main(int, char **) {
>     TestCase c;
>     c.init();
> 
>     c.run("simple_memcpy", simple_memcpy);
>     c.run("simple_copy", simple_copy);
>     c.run("simple_memcpy", simple_memcpy);
>     c.run("simple_copy", simple_copy);
> }
> 
> =====
> 
> The assembly of simple_copy generated by gcc is very simple:
> Dump of assembler code for function _Z11simple_copyPKlPlm:
>    0x0000000000401440 <+0>:     mov    %rdx,%rcx
>    0x0000000000401443 <+3>:     test   %rdx,%rdx
>    0x0000000000401446 <+6>:     je     0x401460 <_Z11simple_copyPKlPlm+32>
>    0x0000000000401448 <+8>:     xor    %eax,%eax
>    0x000000000040144a <+10>:    nopw   0x0(%rax,%rax,1)
>    0x0000000000401450 <+16>:    mov    (%rdi,%rax,8),%rdx
>    0x0000000000401454 <+20>:    mov    %rdx,(%rsi,%rax,8)
>    0x0000000000401458 <+24>:    inc    %rax
>    0x000000000040145b <+27>:    cmp    %rax,%rcx
>    0x000000000040145e <+30>:    jne    0x401450 <_Z11simple_copyPKlPlm+16>
>    0x0000000000401460 <+32>:    retq
> 
> When compiling with -O3, gcc vectorized the loop using xmm0, the
> simple_loop is around 1% faster.

Usually differences of that magnitude falls either in noise or may be something
related to OS jitter. 

> 
> I took a brief look at the glibc source code. Though I don't have enough
> knowledge to understand it yet, I'm curious about the underlying mechanism.
> Thanks.

H.J, do you have any idea what might be happening here? 

Re: memcpy performance on skylake server

[H.J. Lu]

On Wed, Jul 14, 2021 at 5:58 AM Adhemerval Zanella
<adhemerval.zanella@linaro.org> wrote:
>
>
>
> On 06/07/2021 05:17, Ji, Cheng via Libc-help wrote:
> > Hello,
> >
> > I found that memcpy is slower on skylake server CPUs during our
> > optimization work, and I can't really explain what we got and need some
> > guidance here.
> >
> > The problem is that memcpy is noticeably slower than a simple for loop when
> > copying large chunks of data. This genuinely sounds like an amateur mistake
> > in our testing code but here's what we have tried:
> >
> > * The test data is large enough: 1GB.
> > * We noticed a change quite a while ago regarding skylake and AVX512:
> > https://patchwork.ozlabs.org/project/glibc/patch/20170418183712.GA22211@intel.com/
> > * We updated glibc from 2.17 to the latest 2.33, we did see memcpy is 5%
> > faster but still slower than a simple loop.
> > * We tested on multiple bare metal machines with different cpus: Xeon Gold
> > 6132, Gold 6252, Silver 4114, as well as a virtual machine on google cloud,
> > the result is reproducible.
> > * On an older generation Xeon E5-2630 v3, memcpy is about 50% faster than
> > the simple loop. On my desktop (i7-7700k) memcpy is also significantly
> > faster.
> > * numactl is used to ensure everything is running on a single core.
> > * The code is compiled by gcc 10.3
> >
> > The numbers on a Xeon Gold 6132, with glibc 2.33:
> > simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> > simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> > simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> > simple_copy 3.68 seconds, 5.44 GiB/s 5.71 GB/s
> >
> > The result is worse with system provided glibc 2.17:
> > simple_memcpy 4.38 seconds, 4.57 GiB/s 4.79 GB/s
> > simple_copy 3.68 seconds, 5.43 GiB/s 5.70 GB/s
> > simple_memcpy 4.38 seconds, 4.56 GiB/s 4.78 GB/s
> > simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> >
> >
> > The code to generate this result (compiled with g++ -O2 -g, run with: numactl
> > --membind 0 --physcpubind 0 -- ./a.out)
> > =====
> >
> > #include <chrono>
> > #include <cstring>
> > #include <functional>
> > #include <string>
> > #include <vector>
> >
> > class TestCase {
> >     using clock_t = std::chrono::high_resolution_clock;
> >     using sec_t = std::chrono::duration<double, std::ratio<1>>;
> >
> > public:
> >     static constexpr size_t NUM_VALUES = 128 * (1 << 20); // 128 million *
> > 8 bytes = 1GiB
> >
> >     void init() {
> >         vals_.resize(NUM_VALUES);
> >         for (size_t i = 0; i < NUM_VALUES; ++i) {
> >             vals_[i] = i;
> >         }
> >         dest_.resize(NUM_VALUES);
> >     }
> >
> >     void run(std::string name, std::function<void(const int64_t *, int64_t
> > *, size_t)> &&func) {
> >         // ignore the result from first run
> >         func(vals_.data(), dest_.data(), vals_.size());
> >         constexpr size_t count = 20;
> >         auto start = clock_t::now();
> >         for (size_t i = 0; i < count; ++i) {
> >             func(vals_.data(), dest_.data(), vals_.size());
> >         }
> >         auto end = clock_t::now();
> >         double duration =
> > std::chrono::duration_cast<sec_t>(end-start).count();
> >         printf("%s %.2f seconds, %.2f GiB/s, %.2f GB/s\n", name.data(),
> > duration,
> >                sizeof(int64_t) * NUM_VALUES / double(1 << 30) * count /
> > duration,
> >                sizeof(int64_t) * NUM_VALUES / double(1e9) * count /
> > duration);
> >     }
> >
> > private:
> >     std::vector<int64_t> vals_;
> >     std::vector<int64_t> dest_;
> > };
> >
> > void simple_memcpy(const int64_t *src, int64_t *dest, size_t n) {
> >     memcpy(dest, src, n * sizeof(int64_t));
> > }
> >
> > void simple_copy(const int64_t *src, int64_t *dest, size_t n) {
> >     for (size_t i = 0; i < n; ++i) {
> >         dest[i] = src[i];
> >     }
> > }
> >
> > int main(int, char **) {
> >     TestCase c;
> >     c.init();
> >
> >     c.run("simple_memcpy", simple_memcpy);
> >     c.run("simple_copy", simple_copy);
> >     c.run("simple_memcpy", simple_memcpy);
> >     c.run("simple_copy", simple_copy);
> > }
> >
> > =====
> >
> > The assembly of simple_copy generated by gcc is very simple:
> > Dump of assembler code for function _Z11simple_copyPKlPlm:
> >    0x0000000000401440 <+0>:     mov    %rdx,%rcx
> >    0x0000000000401443 <+3>:     test   %rdx,%rdx
> >    0x0000000000401446 <+6>:     je     0x401460 <_Z11simple_copyPKlPlm+32>
> >    0x0000000000401448 <+8>:     xor    %eax,%eax
> >    0x000000000040144a <+10>:    nopw   0x0(%rax,%rax,1)
> >    0x0000000000401450 <+16>:    mov    (%rdi,%rax,8),%rdx
> >    0x0000000000401454 <+20>:    mov    %rdx,(%rsi,%rax,8)
> >    0x0000000000401458 <+24>:    inc    %rax
> >    0x000000000040145b <+27>:    cmp    %rax,%rcx
> >    0x000000000040145e <+30>:    jne    0x401450 <_Z11simple_copyPKlPlm+16>
> >    0x0000000000401460 <+32>:    retq
> >
> > When compiling with -O3, gcc vectorized the loop using xmm0, the
> > simple_loop is around 1% faster.
>
> Usually differences of that magnitude falls either in noise or may be something
> related to OS jitter.
>
> >
> > I took a brief look at the glibc source code. Though I don't have enough
> > knowledge to understand it yet, I'm curious about the underlying mechanism.
> > Thanks.
>
> H.J, do you have any idea what might be happening here?

From Intel optimization guide:

2.2.2 Non-Temporal Stores on Skylake Server Microarchitecture
Because of the change in the size of each bank of last level cache on
Skylake Server microarchitecture, if
an application, library, or driver only considers the last level cache
to determine the size of on-chip cacheper-core, it may see a reduction
with Skylake Server microarchitecture and may use non-temporal store
with smaller blocks of memory writes. Since non-temporal stores evict
cache lines back to memory, this
may result in an increase in the number of subsequent cache misses and
memory bandwidth demands
on Skylake Server microarchitecture, compared to the previous Intel
Xeon processor family.
Also, because of a change in the handling of accesses resulting from
non-temporal stores by Skylake
Server microarchitecture, the resources within each core remain busy
for a longer duration compared to
similar accesses on the previous Intel Xeon processor family. As a
result, if a series of such instructions
are executed, there is a potential that the processor may run out of
resources and stall, thus limiting the
memory write bandwidth from each core.
The increase in cache misses due to overuse of non-temporal stores and
the limit on the memory write
bandwidth per core for non-temporal stores may result in reduced
performance for some applications.

-- 
H.J.

Re: memcpy performance on skylake server

[Ji, Cheng]

Thanks for the information. We did some quick experiments. Indeed, using
normal temporal stores is ~20% faster than using non-temporal stores in
this case.

Cheng

On Wed, Jul 14, 2021 at 9:27 PM H.J. Lu <hjl.tools@gmail.com> wrote:

> On Wed, Jul 14, 2021 at 5:58 AM Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
> >
> >
> >
> > On 06/07/2021 05:17, Ji, Cheng via Libc-help wrote:
> > > Hello,
> > >
> > > I found that memcpy is slower on skylake server CPUs during our
> > > optimization work, and I can't really explain what we got and need some
> > > guidance here.
> > >
> > > The problem is that memcpy is noticeably slower than a simple for loop
> when
> > > copying large chunks of data. This genuinely sounds like an amateur
> mistake
> > > in our testing code but here's what we have tried:
> > >
> > > * The test data is large enough: 1GB.
> > > * We noticed a change quite a while ago regarding skylake and AVX512:
> > >
> https://patchwork.ozlabs.org/project/glibc/patch/20170418183712.GA22211@intel.com/
> > > * We updated glibc from 2.17 to the latest 2.33, we did see memcpy is
> 5%
> > > faster but still slower than a simple loop.
> > > * We tested on multiple bare metal machines with different cpus: Xeon
> Gold
> > > 6132, Gold 6252, Silver 4114, as well as a virtual machine on google
> cloud,
> > > the result is reproducible.
> > > * On an older generation Xeon E5-2630 v3, memcpy is about 50% faster
> than
> > > the simple loop. On my desktop (i7-7700k) memcpy is also significantly
> > > faster.
> > > * numactl is used to ensure everything is running on a single core.
> > > * The code is compiled by gcc 10.3
> > >
> > > The numbers on a Xeon Gold 6132, with glibc 2.33:
> > > simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> > > simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> > > simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
> > > simple_copy 3.68 seconds, 5.44 GiB/s 5.71 GB/s
> > >
> > > The result is worse with system provided glibc 2.17:
> > > simple_memcpy 4.38 seconds, 4.57 GiB/s 4.79 GB/s
> > > simple_copy 3.68 seconds, 5.43 GiB/s 5.70 GB/s
> > > simple_memcpy 4.38 seconds, 4.56 GiB/s 4.78 GB/s
> > > simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
> > >
> > >
> > > The code to generate this result (compiled with g++ -O2 -g, run with:
> numactl
> > > --membind 0 --physcpubind 0 -- ./a.out)
> > > =====
> > >
> > > #include <chrono>
> > > #include <cstring>
> > > #include <functional>
> > > #include <string>
> > > #include <vector>
> > >
> > > class TestCase {
> > >     using clock_t = std::chrono::high_resolution_clock;
> > >     using sec_t = std::chrono::duration<double, std::ratio<1>>;
> > >
> > > public:
> > >     static constexpr size_t NUM_VALUES = 128 * (1 << 20); // 128
> million *
> > > 8 bytes = 1GiB
> > >
> > >     void init() {
> > >         vals_.resize(NUM_VALUES);
> > >         for (size_t i = 0; i < NUM_VALUES; ++i) {
> > >             vals_[i] = i;
> > >         }
> > >         dest_.resize(NUM_VALUES);
> > >     }
> > >
> > >     void run(std::string name, std::function<void(const int64_t *,
> int64_t
> > > *, size_t)> &&func) {
> > >         // ignore the result from first run
> > >         func(vals_.data(), dest_.data(), vals_.size());
> > >         constexpr size_t count = 20;
> > >         auto start = clock_t::now();
> > >         for (size_t i = 0; i < count; ++i) {
> > >             func(vals_.data(), dest_.data(), vals_.size());
> > >         }
> > >         auto end = clock_t::now();
> > >         double duration =
> > > std::chrono::duration_cast<sec_t>(end-start).count();
> > >         printf("%s %.2f seconds, %.2f GiB/s, %.2f GB/s\n", name.data(),
> > > duration,
> > >                sizeof(int64_t) * NUM_VALUES / double(1 << 30) * count /
> > > duration,
> > >                sizeof(int64_t) * NUM_VALUES / double(1e9) * count /
> > > duration);
> > >     }
> > >
> > > private:
> > >     std::vector<int64_t> vals_;
> > >     std::vector<int64_t> dest_;
> > > };
> > >
> > > void simple_memcpy(const int64_t *src, int64_t *dest, size_t n) {
> > >     memcpy(dest, src, n * sizeof(int64_t));
> > > }
> > >
> > > void simple_copy(const int64_t *src, int64_t *dest, size_t n) {
> > >     for (size_t i = 0; i < n; ++i) {
> > >         dest[i] = src[i];
> > >     }
> > > }
> > >
> > > int main(int, char **) {
> > >     TestCase c;
> > >     c.init();
> > >
> > >     c.run("simple_memcpy", simple_memcpy);
> > >     c.run("simple_copy", simple_copy);
> > >     c.run("simple_memcpy", simple_memcpy);
> > >     c.run("simple_copy", simple_copy);
> > > }
> > >
> > > =====
> > >
> > > The assembly of simple_copy generated by gcc is very simple:
> > > Dump of assembler code for function _Z11simple_copyPKlPlm:
> > >    0x0000000000401440 <+0>:     mov    %rdx,%rcx
> > >    0x0000000000401443 <+3>:     test   %rdx,%rdx
> > >    0x0000000000401446 <+6>:     je     0x401460
> <_Z11simple_copyPKlPlm+32>
> > >    0x0000000000401448 <+8>:     xor    %eax,%eax
> > >    0x000000000040144a <+10>:    nopw   0x0(%rax,%rax,1)
> > >    0x0000000000401450 <+16>:    mov    (%rdi,%rax,8),%rdx
> > >    0x0000000000401454 <+20>:    mov    %rdx,(%rsi,%rax,8)
> > >    0x0000000000401458 <+24>:    inc    %rax
> > >    0x000000000040145b <+27>:    cmp    %rax,%rcx
> > >    0x000000000040145e <+30>:    jne    0x401450
> <_Z11simple_copyPKlPlm+16>
> > >    0x0000000000401460 <+32>:    retq
> > >
> > > When compiling with -O3, gcc vectorized the loop using xmm0, the
> > > simple_loop is around 1% faster.
> >
> > Usually differences of that magnitude falls either in noise or may be
> something
> > related to OS jitter.
> >
> > >
> > > I took a brief look at the glibc source code. Though I don't have
> enough
> > > knowledge to understand it yet, I'm curious about the underlying
> mechanism.
> > > Thanks.
> >
> > H.J, do you have any idea what might be happening here?
>
> From Intel optimization guide:
>
> 2.2.2 Non-Temporal Stores on Skylake Server Microarchitecture
> Because of the change in the size of each bank of last level cache on
> Skylake Server microarchitecture, if
> an application, library, or driver only considers the last level cache
> to determine the size of on-chip cacheper-core, it may see a reduction
> with Skylake Server microarchitecture and may use non-temporal store
> with smaller blocks of memory writes. Since non-temporal stores evict
> cache lines back to memory, this
> may result in an increase in the number of subsequent cache misses and
> memory bandwidth demands
> on Skylake Server microarchitecture, compared to the previous Intel
> Xeon processor family.
> Also, because of a change in the handling of accesses resulting from
> non-temporal stores by Skylake
> Server microarchitecture, the resources within each core remain busy
> for a longer duration compared to
> similar accesses on the previous Intel Xeon processor family. As a
> result, if a series of such instructions
> are executed, there is a potential that the processor may run out of
> resources and stall, thus limiting the
> memory write bandwidth from each core.
> The increase in cache misses due to overuse of non-temporal stores and
> the limit on the memory write
> bandwidth per core for non-temporal stores may result in reduced
> performance for some applications.
>
> --
> H.J.
>

Re: memcpy performance on skylake server

[Patrick McGehearty]

More in-depth discussion of tuning non-temporal stores for x86
can be found at:
http://patches-tcwg.linaro.org/patch/41797/

- Patrick McGehearty


On 7/15/2021 2:32 AM, Ji, Cheng via Libc-help wrote:
> Thanks for the information. We did some quick experiments. Indeed, using
> normal temporal stores is ~20% faster than using non-temporal stores in
> this case.
>
> Cheng
>
> On Wed, Jul 14, 2021 at 9:27 PM H.J. Lu <hjl.tools@gmail.com> wrote:
>
>> On Wed, Jul 14, 2021 at 5:58 AM Adhemerval Zanella
>> <adhemerval.zanella@linaro.org> wrote:
>>>
>>>
>>> On 06/07/2021 05:17, Ji, Cheng via Libc-help wrote:
>>>> Hello,
>>>>
>>>> I found that memcpy is slower on skylake server CPUs during our
>>>> optimization work, and I can't really explain what we got and need some
>>>> guidance here.
>>>>
>>>> The problem is that memcpy is noticeably slower than a simple for loop
>> when
>>>> copying large chunks of data. This genuinely sounds like an amateur
>> mistake
>>>> in our testing code but here's what we have tried:
>>>>
>>>> * The test data is large enough: 1GB.
>>>> * We noticed a change quite a while ago regarding skylake and AVX512:
>>>>
>> https://patchwork.ozlabs.org/project/glibc/patch/20170418183712.GA22211@intel.com/
>>>> * We updated glibc from 2.17 to the latest 2.33, we did see memcpy is
>> 5%
>>>> faster but still slower than a simple loop.
>>>> * We tested on multiple bare metal machines with different cpus: Xeon
>> Gold
>>>> 6132, Gold 6252, Silver 4114, as well as a virtual machine on google
>> cloud,
>>>> the result is reproducible.
>>>> * On an older generation Xeon E5-2630 v3, memcpy is about 50% faster
>> than
>>>> the simple loop. On my desktop (i7-7700k) memcpy is also significantly
>>>> faster.
>>>> * numactl is used to ensure everything is running on a single core.
>>>> * The code is compiled by gcc 10.3
>>>>
>>>> The numbers on a Xeon Gold 6132, with glibc 2.33:
>>>> simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
>>>> simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
>>>> simple_memcpy 4.18 seconds, 4.79 GiB/s 5.02 GB/s
>>>> simple_copy 3.68 seconds, 5.44 GiB/s 5.71 GB/s
>>>>
>>>> The result is worse with system provided glibc 2.17:
>>>> simple_memcpy 4.38 seconds, 4.57 GiB/s 4.79 GB/s
>>>> simple_copy 3.68 seconds, 5.43 GiB/s 5.70 GB/s
>>>> simple_memcpy 4.38 seconds, 4.56 GiB/s 4.78 GB/s
>>>> simple_copy 3.68 seconds, 5.44 GiB/s 5.70 GB/s
>>>>
>>>>
>>>> The code to generate this result (compiled with g++ -O2 -g, run with:
>> numactl
>>>> --membind 0 --physcpubind 0 -- ./a.out)
>>>> =====
>>>>
>>>> #include <chrono>
>>>> #include <cstring>
>>>> #include <functional>
>>>> #include <string>
>>>> #include <vector>
>>>>
>>>> class TestCase {
>>>>      using clock_t = std::chrono::high_resolution_clock;
>>>>      using sec_t = std::chrono::duration<double, std::ratio<1>>;
>>>>
>>>> public:
>>>>      static constexpr size_t NUM_VALUES = 128 * (1 << 20); // 128
>> million *
>>>> 8 bytes = 1GiB
>>>>
>>>>      void init() {
>>>>          vals_.resize(NUM_VALUES);
>>>>          for (size_t i = 0; i < NUM_VALUES; ++i) {
>>>>              vals_[i] = i;
>>>>          }
>>>>          dest_.resize(NUM_VALUES);
>>>>      }
>>>>
>>>>      void run(std::string name, std::function<void(const int64_t *,
>> int64_t
>>>> *, size_t)> &&func) {
>>>>          // ignore the result from first run
>>>>          func(vals_.data(), dest_.data(), vals_.size());
>>>>          constexpr size_t count = 20;
>>>>          auto start = clock_t::now();
>>>>          for (size_t i = 0; i < count; ++i) {
>>>>              func(vals_.data(), dest_.data(), vals_.size());
>>>>          }
>>>>          auto end = clock_t::now();
>>>>          double duration =
>>>> std::chrono::duration_cast<sec_t>(end-start).count();
>>>>          printf("%s %.2f seconds, %.2f GiB/s, %.2f GB/s\n", name.data(),
>>>> duration,
>>>>                 sizeof(int64_t) * NUM_VALUES / double(1 << 30) * count /
>>>> duration,
>>>>                 sizeof(int64_t) * NUM_VALUES / double(1e9) * count /
>>>> duration);
>>>>      }
>>>>
>>>> private:
>>>>      std::vector<int64_t> vals_;
>>>>      std::vector<int64_t> dest_;
>>>> };
>>>>
>>>> void simple_memcpy(const int64_t *src, int64_t *dest, size_t n) {
>>>>      memcpy(dest, src, n * sizeof(int64_t));
>>>> }
>>>>
>>>> void simple_copy(const int64_t *src, int64_t *dest, size_t n) {
>>>>      for (size_t i = 0; i < n; ++i) {
>>>>          dest[i] = src[i];
>>>>      }
>>>> }
>>>>
>>>> int main(int, char **) {
>>>>      TestCase c;
>>>>      c.init();
>>>>
>>>>      c.run("simple_memcpy", simple_memcpy);
>>>>      c.run("simple_copy", simple_copy);
>>>>      c.run("simple_memcpy", simple_memcpy);
>>>>      c.run("simple_copy", simple_copy);
>>>> }
>>>>
>>>> =====
>>>>
>>>> The assembly of simple_copy generated by gcc is very simple:
>>>> Dump of assembler code for function _Z11simple_copyPKlPlm:
>>>>     0x0000000000401440 <+0>:     mov    %rdx,%rcx
>>>>     0x0000000000401443 <+3>:     test   %rdx,%rdx
>>>>     0x0000000000401446 <+6>:     je     0x401460
>> <_Z11simple_copyPKlPlm+32>
>>>>     0x0000000000401448 <+8>:     xor    %eax,%eax
>>>>     0x000000000040144a <+10>:    nopw   0x0(%rax,%rax,1)
>>>>     0x0000000000401450 <+16>:    mov    (%rdi,%rax,8),%rdx
>>>>     0x0000000000401454 <+20>:    mov    %rdx,(%rsi,%rax,8)
>>>>     0x0000000000401458 <+24>:    inc    %rax
>>>>     0x000000000040145b <+27>:    cmp    %rax,%rcx
>>>>     0x000000000040145e <+30>:    jne    0x401450
>> <_Z11simple_copyPKlPlm+16>
>>>>     0x0000000000401460 <+32>:    retq
>>>>
>>>> When compiling with -O3, gcc vectorized the loop using xmm0, the
>>>> simple_loop is around 1% faster.
>>> Usually differences of that magnitude falls either in noise or may be
>> something
>>> related to OS jitter.
>>>
>>>> I took a brief look at the glibc source code. Though I don't have
>> enough
>>>> knowledge to understand it yet, I'm curious about the underlying
>> mechanism.
>>>> Thanks.
>>> H.J, do you have any idea what might be happening here?
>>  From Intel optimization guide:
>>
>> 2.2.2 Non-Temporal Stores on Skylake Server Microarchitecture
>> Because of the change in the size of each bank of last level cache on
>> Skylake Server microarchitecture, if
>> an application, library, or driver only considers the last level cache
>> to determine the size of on-chip cacheper-core, it may see a reduction
>> with Skylake Server microarchitecture and may use non-temporal store
>> with smaller blocks of memory writes. Since non-temporal stores evict
>> cache lines back to memory, this
>> may result in an increase in the number of subsequent cache misses and
>> memory bandwidth demands
>> on Skylake Server microarchitecture, compared to the previous Intel
>> Xeon processor family.
>> Also, because of a change in the handling of accesses resulting from
>> non-temporal stores by Skylake
>> Server microarchitecture, the resources within each core remain busy
>> for a longer duration compared to
>> similar accesses on the previous Intel Xeon processor family. As a
>> result, if a series of such instructions
>> are executed, there is a potential that the processor may run out of
>> resources and stall, thus limiting the
>> memory write bandwidth from each core.
>> The increase in cache misses due to overuse of non-temporal stores and
>> the limit on the memory write
>> bandwidth per core for non-temporal stores may result in reduced
>> performance for some applications.
>>
>> --
>> H.J.
>>