std::chrono is much slower than native requests...?

std::chrono is much slower than native requests...?

[Paul Smith]

I'm using GCC 8.1 (built myself) on GNU/Linux for this testing.  The
code is compiled with -O2 -g.

I replaced the arch-specific implementation of a function we use a LOT
which returns a value in microseconds that we compare to get elapsed
time (but never convert into a timestamp) with std::chrono like this:

  using std::chrono;
  return time_point_cast<microseconds>(steady_clock::now())
             .time_since_epoch().count();

Unfortunately after I did this I got pinged by the perf folks that I
had introduced a 24% slowdown in some benchmarks.  These benchmarks are
testing general application operations, and they compute elapsed time
using a Python framework so that computation is not impacted by the
above change.  The slowdown is widespread across many tests.  I can
reproduce this slowdown on my local system.

I've tried it with both steady_clock and system_clock and both show the
same slowdown.

I looked at the code in libstdc++-v3 and as far as I can tell I'm using
CLOCK_MONOTONIC for my steady_clock; this is in libstdc++-v3/config.h:

  #define _GLIBCXX_USE_CLOCK_GETTIME_SYSCALL 1
  #define _GLIBCXX_USE_CLOCK_MONOTONIC 1
  #define _GLIBCXX_USE_CLOCK_REALTIME 1

But if I replace the above code with a direct call to clock_gettime():

  struct timespec ts;
  clock_gettime(CLOCK_MONOTONIC, &ts);
  return ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000;

then I get back all my performance!

Any idea why the std::chrono implementation is so much slower?  I
looked at the implementation and I didn't see anything obvious.

Here's a gdb disassembly of the "fast" clock_gettime() version:

   <+0>:   push   %rbp
   <+1>:   mov    $0x1,%edi
   <+6>:   mov    %rsp,%rbp
   <+9>:   lea    -0x10(%rsp),%rsp
   <+14>:  lea    -0x10(%rbp),%rsi
   <+18>:  callq  0x40c730 <clock_gettime@plt>
   <+23>:  mov    -0x8(%rbp),%rcx
   <+27>:  movabs $0x20c49ba5e353f7cf,%rdx
   <+37>:  imul   $0xf4240,-0x10(%rbp),%rsi
   <+45>:  mov    %rcx,%rax
   <+48>:  imul   %rdx
   <+51>:  sar    $0x3f,%rcx
   <+55>:  leaveq
   <+56>:  sar    $0x7,%rdx
   <+60>:  sub    %rcx,%rdx
   <+63>:  lea    (%rsi,%rdx,1),%rax
   <+67>:  retq
     xchg   %ax,%ax
     nopw   %cs:0x0(%rax,%rax,1)

And here's a disassembly of the slow chrono version:

   <+0>:   push   %rbp
   <+1>:   mov    %rsp,%rbp
   <+4>:   callq  0xe62720 <_ZNSt6chrono3_V212steady_clock3nowEv>
   <+9>:   movabs $0x20c49ba5e353f7cf,%rdx
   <+19>:  mov    %rax,%rcx
   <+22>:  imul   %rdx
   <+25>:  sar    $0x3f,%rcx
   <+29>:  pop    %rbp
   <+30>:  sar    $0x7,%rdx
   <+34>:  mov    %rdx,%rax
   <+37>:  sub    %rcx,%rax
   <+40>:  retq
     nop
     nopw   0x0(%rax,%rax,1)

And of the std::chrono::steady_clock::now() function:

Dump of assembler code for function _ZNSt6chrono3_V212steady_clock3nowEv:
   <+0>:     sub    $0x18,%rsp
   <+4>:     mov    $0x1,%esi
   <+9>:     mov    $0xe4,%edi
   <+14>:    xor    %eax,%eax
   <+16>:    mov    %rsp,%rdx
   <+19>:    callq  0x40b950 <syscall@plt>
   <+24>:    imul   $0x3b9aca00,(%rsp),%rax
   <+32>:    add    0x8(%rsp),%rax
   <+37>:    add    $0x18,%rsp
   <+41>:    retq

My assembly is pretty lame, so maybe I'm missing something, but it
seems pretty close to the same thing to me, and does appear to be using
CLOCK_MONOTONIC (which is code 1) in the syscall.

Re: std::chrono is much slower than native requests...?

[Alexander Monakov]

On Sat, 6 Jul 2019, Paul Smith wrote:

> I looked at the code in libstdc++-v3 and as far as I can tell I'm using
> CLOCK_MONOTONIC for my steady_clock; this is in libstdc++-v3/config.h:
> 
>   #define _GLIBCXX_USE_CLOCK_GETTIME_SYSCALL 1

^ this is a problem, your libstdc++ library is configured to use the syscall
directly instead of letting the libc go via the VDSO fastpath. Most likely
you're building GCC against old glibc where clock_gettime requires linking with
-lrt; have a look at the documentation for --enable-libstdcxx-time= in the
libstdc++ manual,
https://gcc.gnu.org/onlinedocs/gcc-9.1.0/libstdc++/manual/manual/configure.html

> Here's a gdb disassembly of the "fast" clock_gettime() version:
> 
[..]
>    <+18>:  callq  0x40c730 <clock_gettime@plt>

this typically goes via VDSO without a context switch

> And of the std::chrono::steady_clock::now() function:
> 
[..]
>    <+19>:    callq  0x40b950 <syscall@plt>

this requires a context switch to the kernel and back.

Alexander

Re: std::chrono is much slower than native requests...?

[Paul Smith]

On Sat, 2019-07-06 at 18:29 +0300, Alexander Monakov wrote:
> ^ this is a problem, your libstdc++ library is configured to use the syscall
> directly instead of letting the libc go via the VDSO fastpath.

Oh snap!

I'm fully aware of the VDSO situation but I didn't grok the fact that
the syscall meant I wasn't using it.

Hm, this code is built with a sysroot designed to work on Red Hat EL
6.5 and newer systems; I suppose the RHEL 6.5 glibc must not have used
the clock_gettime VDSO (or if it did my GCC configure doesn't find it).

OK, that gives me a clear direction.

Thanks Alexander!

Re: std::chrono is much slower than native requests...?

[Paul Smith]

On Sat, 2019-07-06 at 12:10 -0400, Paul Smith wrote:
> On Sat, 2019-07-06 at 18:29 +0300, Alexander Monakov wrote:
> > ^ this is a problem, your libstdc++ library is configured to use the syscall
> > directly instead of letting the libc go via the VDSO fastpath.
> 
> Oh snap!

OK I've delved into this.  I guess it's intended behavior, although it
seems a little magic.

Basically if your sysroot does not have at least glibc 2.17 then the
libstdc++-v3 configure script will always choose the slower, syscall
versions of clock_gettime() even if a faster version is available (in
the rt library).

You can work around this issue by forcing the libstdc++-v3 configure to
use the rt library, by adding this configure option:

  --enable-libstdcxx-time=rt

There is some discussion of this in the libstdc++ manual section on
configuration:

  https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html

but it's not likely something you'll stumble upon on your own, and it
definitely isn't made clear the performance penalties you might pay if
you don't use it.  It might be useful to at least discuss this in the
docs, although I suppose systems using glibc <2.17 are getting more
rare every day.

Thanks for the heads-up!

Re: std::chrono is much slower than native requests...?

[Jonathan Wakely]

On Sat, 6 Jul 2019 at 22:28, Paul Smith <psmith@gnu.org> wrote:
>
> On Sat, 2019-07-06 at 12:10 -0400, Paul Smith wrote:
> > On Sat, 2019-07-06 at 18:29 +0300, Alexander Monakov wrote:
> > > ^ this is a problem, your libstdc++ library is configured to use the syscall
> > > directly instead of letting the libc go via the VDSO fastpath.
> >
> > Oh snap!
>
> OK I've delved into this.  I guess it's intended behavior, although it
> seems a little magic.
>
> Basically if your sysroot does not have at least glibc 2.17 then the
> libstdc++-v3 configure script will always choose the slower, syscall
> versions of clock_gettime() even if a faster version is available (in
> the rt library).
>
> You can work around this issue by forcing the libstdc++-v3 configure to
> use the rt library, by adding this configure option:
>
>   --enable-libstdcxx-time=rt
>
> There is some discussion of this in the libstdc++ manual section on
> configuration:
>
>   https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html
>
> but it's not likely something you'll stumble upon on your own, and it
> definitely isn't made clear the performance penalties you might pay if
> you don't use it.  It might be useful to at least discuss this in the
> docs, although I suppose systems using glibc <2.17 are getting more
> rare every day.

There are performance penalties to using it too, so it's not just a
case of saying "hey, you should use this!"

If you link to librt on GNU/Linux then you get a dependency on
libpthread which causes libstdc++ to always assume your program is
multithreaded, and use atomic ops for reference counting even in
single-threaded programs.

Re: std::chrono is much slower than native requests...?

[Paul Smith]

On Sun, 2019-07-07 at 20:31 +0100, Jonathan Wakely wrote:
> > It might be useful to at least discuss this in the
> > docs, although I suppose systems using glibc <2.17 are getting more
> > rare every day.
> 
> There are performance penalties to using it too, so it's not just a
> case of saying "hey, you should use this!"
> 
> If you link to librt on GNU/Linux then you get a dependency on
> libpthread which causes libstdc++ to always assume your program is
> multithreaded, and use atomic ops for reference counting even in
> single-threaded programs.

Yes, that information is presented in the docs, which is good.  But I
think the other side of this (that selecting "rt" on older glibc
implementations will give a 22-24% performance increase when calling
steady_clock() / system_clock()) should also be mentioned.

Unless you're building for a very specific target/need, you likely will
want your compiler to be able to be able to create multi-threaded
programs.

Cheers!

Re: std::chrono is much slower than native requests...?

[Jonathan Wakely]

On Sun, 7 Jul 2019 at 21:18, Paul Smith <psmith@gnu.org> wrote:
>
> On Sun, 2019-07-07 at 20:31 +0100, Jonathan Wakely wrote:
> > > It might be useful to at least discuss this in the
> > > docs, although I suppose systems using glibc <2.17 are getting more
> > > rare every day.
> >
> > There are performance penalties to using it too, so it's not just a
> > case of saying "hey, you should use this!"
> >
> > If you link to librt on GNU/Linux then you get a dependency on
> > libpthread which causes libstdc++ to always assume your program is
> > multithreaded, and use atomic ops for reference counting even in
> > single-threaded programs.
>
> Yes, that information is presented in the docs, which is good.  But I
> think the other side of this (that selecting "rt" on older glibc
> implementations will give a 22-24% performance increase when calling
> steady_clock() / system_clock()) should also be mentioned.
>
> Unless you're building for a very specific target/need, you likely will
> want your compiler to be able to be able to create multi-threaded
> programs.

I'm not sure what you mean here. The compiler is able to create
multi-threaded programs either way. The issue is whether
single-threaded programs pay a cost that's only needed by
multi-threaded programs or not. With the "rt" option you get a
compiler that is not able to use a libstdc++ optimisation normally
enabled for single-threaded programs.

Re: std::chrono is much slower than native requests...?

[Jonathan Wakely]

On Sun, 7 Jul 2019 at 22:00, Jonathan Wakely wrote:
>
> On Sun, 7 Jul 2019 at 21:18, Paul Smith <psmith@gnu.org> wrote:
> >
> > On Sun, 2019-07-07 at 20:31 +0100, Jonathan Wakely wrote:
> > > > It might be useful to at least discuss this in the
> > > > docs, although I suppose systems using glibc <2.17 are getting more
> > > > rare every day.
> > >
> > > There are performance penalties to using it too, so it's not just a
> > > case of saying "hey, you should use this!"
> > >
> > > If you link to librt on GNU/Linux then you get a dependency on
> > > libpthread which causes libstdc++ to always assume your program is
> > > multithreaded, and use atomic ops for reference counting even in
> > > single-threaded programs.
> >
> > Yes, that information is presented in the docs, which is good.  But I
> > think the other side of this (that selecting "rt" on older glibc
> > implementations will give a 22-24% performance increase when calling
> > steady_clock() / system_clock()) should also be mentioned.

How's this?

--- a/libstdc++-v3/doc/xml/manual/configure.xml
+++ b/libstdc++-v3/doc/xml/manual/configure.xml
@@ -166,18 +166,24 @@

  <varlistentry><term><code>--enable-libstdcxx-time=OPTION</code></term>
  <listitem><para>Enables link-type checks for the availability of the
-       clock_gettime clocks, used in the implementation of [time.clock],
-       and of the nanosleep and sched_yield functions, used in the
+       <function>clock_gettime</function> clocks, used in the implementation
+       of [time.clock], and of the <function>nanosleep</function> and
+       <function>sched_yield</function> functions, used in the
        implementation of [thread.thread.this] of the 2011 ISO C++ standard.
        The choice OPTION=yes checks for the availability of the facilities
        in libc and libposix4.  In case it's needed the latter is also linked
-       to libstdc++ as part of the build process.  OPTION=rt also searches
-       (and, if needed, links) librt.   Note that the latter is not always
-       desirable because, in glibc, for example, in turn it triggers the
-       linking of libpthread too, which activates locking, a large overhead
-       for single-thread programs.  OPTION=no skips the tests completely.
+       to libstdc++ as part of the build process.  OPTION=rt also checks in
+       librt (and, if it's needed, links to it).  Note that linking to librt
+       is not always desirable because for glibc it requires linking to
+       libpthread too, which causes all reference counting to use atomic
+       operations, resulting in a potentially large overhead for
+       single-threaded programs.  OPTION=no skips the tests completely.
        The default is OPTION=auto, which skips the checks and enables the
        features only for targets known to support them.
+       For Linux targets, if <function>clock_gettime</function> is not used
+       then the [time.clock] implementation will use a system call to access
+       the realtime and monotonic clocks, which is significantly slower than
+       the C library's <function>clock_gettime</function> function.
     </para>
  </listitem></varlistentry>





> > Unless you're building for a very specific target/need, you likely will
> > want your compiler to be able to be able to create multi-threaded
> > programs.
>
> I'm not sure what you mean here. The compiler is able to create
> multi-threaded programs either way. The issue is whether
> single-threaded programs pay a cost that's only needed by
> multi-threaded programs or not. With the "rt" option you get a
> compiler that is not able to use a libstdc++ optimisation normally
> enabled for single-threaded programs.

Re: std::chrono is much slower than native requests...?

[Oleg Endo]

On Sun, 2019-07-07 at 22:00 +0100, Jonathan Wakely wrote:
> On Sun, 7 Jul 2019 at 21:18, Paul Smith <psmith@gnu.org> wrote:
> > 
> > On Sun, 2019-07-07 at 20:31 +0100, Jonathan Wakely wrote:
> > > > It might be useful to at least discuss this in the
> > > > docs, although I suppose systems using glibc <2.17 are getting
> > > > more
> > > > rare every day.
> > > 
> > > There are performance penalties to using it too, so it's not just
> > > a
> > > case of saying "hey, you should use this!"
> > > 
> > > If you link to librt on GNU/Linux then you get a dependency on
> > > libpthread which causes libstdc++ to always assume your program
> > > is
> > > multithreaded, and use atomic ops for reference counting even in
> > > single-threaded programs.
> > 
> > Yes, that information is presented in the docs, which is good.  But
> > I
> > think the other side of this (that selecting "rt" on older glibc
> > implementations will give a 22-24% performance increase when
> > calling
> > steady_clock() / system_clock()) should also be mentioned.
> > 
> > Unless you're building for a very specific target/need, you likely
> > will
> > want your compiler to be able to be able to create multi-threaded
> > programs.
> 
> I'm not sure what you mean here. The compiler is able to create
> multi-threaded programs either way. The issue is whether
> single-threaded programs pay a cost that's only needed by
> multi-threaded programs or not. With the "rt" option you get a
> compiler that is not able to use a libstdc++ optimisation normally
> enabled for single-threaded programs.

Even with single-threaded programs there could be async signal
handlers.... and on MCU/baremetal targets we've got interrupt handlers,
which are conceptually the same.  I'd expect shared_ptr to just work
there, too.  But if all it takes is linking in pthread, that's no
problem.

Cheers,
Oleg

Re: std::chrono is much slower than native requests...?

[Paul Smith]

On Sun, 2019-07-07 at 22:17 +0100, Jonathan Wakely wrote:
> How's this?

That looks good to me.

If I'm being honest with myself I doubt I would have noticed this
either way; I'm not sure I even read the separate configuration docs
for libstdc++.

However, all in all it's better (IMO) to have it noted than not.

Re: std::chrono is much slower than native requests...?

[Jonathan Wakely]

On Mon, 8 Jul 2019 at 13:19, Oleg Endo <oleg.endo@t-online.de> wrote:
>
> On Sun, 2019-07-07 at 22:00 +0100, Jonathan Wakely wrote:
> > On Sun, 7 Jul 2019 at 21:18, Paul Smith <psmith@gnu.org> wrote:
> > >
> > > On Sun, 2019-07-07 at 20:31 +0100, Jonathan Wakely wrote:
> > > > > It might be useful to at least discuss this in the
> > > > > docs, although I suppose systems using glibc <2.17 are getting
> > > > > more
> > > > > rare every day.
> > > >
> > > > There are performance penalties to using it too, so it's not just
> > > > a
> > > > case of saying "hey, you should use this!"
> > > >
> > > > If you link to librt on GNU/Linux then you get a dependency on
> > > > libpthread which causes libstdc++ to always assume your program
> > > > is
> > > > multithreaded, and use atomic ops for reference counting even in
> > > > single-threaded programs.
> > >
> > > Yes, that information is presented in the docs, which is good.  But
> > > I
> > > think the other side of this (that selecting "rt" on older glibc
> > > implementations will give a 22-24% performance increase when
> > > calling
> > > steady_clock() / system_clock()) should also be mentioned.
> > >
> > > Unless you're building for a very specific target/need, you likely
> > > will
> > > want your compiler to be able to be able to create multi-threaded
> > > programs.
> >
> > I'm not sure what you mean here. The compiler is able to create
> > multi-threaded programs either way. The issue is whether
> > single-threaded programs pay a cost that's only needed by
> > multi-threaded programs or not. With the "rt" option you get a
> > compiler that is not able to use a libstdc++ optimisation normally
> > enabled for single-threaded programs.
>
> Even with single-threaded programs there could be async signal
> handlers.... and on MCU/baremetal targets we've got interrupt handlers,
> which are conceptually the same.  I'd expect shared_ptr to just work
> there, too.

What do you base that expectation on? Only accesses to
std::sig_atomic_t variables are safe in signal handlers.

>  But if all it takes is linking in pthread, that's no
> problem.

It's orthogonal to this discussion anyway. The
--enable-libstdcxx-time=rt option can cause all ref counting to be
atomic, which isn't going to make the signal handler case any worse.

Re: std::chrono is much slower than native requests...?

[Oleg Endo]

On Mon, 2019-07-08 at 14:41 +0100, Jonathan Wakely wrote:
> On Mon, 8 Jul 2019 at 13:19, Oleg Endo <oleg.endo@t-online.de> wrote:
> > 
> > Even with single-threaded programs there could be async signal
> > handlers.... and on MCU/baremetal targets we've got interrupt
> > handlers,
> > which are conceptually the same.  I'd expect shared_ptr to just
> > work
> > there, too.
> 
> What do you base that expectation on? Only accesses to
> std::sig_atomic_t variables are safe in signal handlers.

Oops, missed that one. Thanks!

Cheers,
Oleg