[Bug nptl/4209] New: Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[Bug nptl/4209] New: Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

Some applications use semaphores for ensuring mutual exclusion, the purpose
mutexes are best suited for. By comparing the performance of such an application
with linuxthreads versus NPTL I discovered that under NPTL semaphores work
slower than under linuxthreads. This behavior occurs at least on ppc32 and i386.

The measurements I performed show that mutexes are about 30% faster under NPTL
than under linuxthreads, and that semaphores are about 70% slower:

$ uname -a
Linux pc-100 2.6.18.6 #8 Sun Feb 4 11:17:43 CET 2007 i686 athlon i386 GNU/Linux
$ /lib/libc.so.6
GNU C Library stable release version 2.5 (20061011), by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 4.1.2 20061115 (prerelease) (SUSE Linux).
Compiled on a Linux 2.6.18 system on 2006-11-26.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        NoVersion patch for broken glibc 2.0 binaries
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
$ /home/bart/glibc236/lib/libc.so.6
GNU C Library stable release version 2.3.6, by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.1.2 20061115 (prerelease) (SUSE Linux).
Compiled on a Linux >>2.6.18.6<< system on 2007-03-17.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        linuxthreads-0.10 by Xavier Leroy
        BIND-8.2.3-T5B
        libthread_db work sponsored by Alpha Processor Inc
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
$ ./perf
NPTL
mutex     Elapsed: 408611 us; per iteration: 40 ns.
semaphore Elapsed: 1901436 us; per iteration: 190 ns.
$ LD_LIBRARY_PATH=/home/bart/glibc236/lib: /home/bart/glibc236/lib/ld-linux.so.2
./glibc236-perf
linuxthreads
mutex     Elapsed: 538978 us; per iteration: 53 ns.
semaphore Elapsed: 1097891 us; per iteration: 109 ns.

-- 
           Summary: Performance issue: NPTL semaphores work slower than
                    linuxthreads semaphores
           Product: glibc
           Version: 2.4
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nptl
        AssignedTo: drepper at redhat dot com
        ReportedBy: bart dot vanassche at gmail dot com
                CC: glibc-bugs at sources dot redhat dot com
 GCC build triplet: i686-suse-linux-gnu
  GCC host triplet: i686-suse-linux-gnu
GCC target triplet: i686-suse-linux-gnu


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-03-17 11:38 -------
Created an attachment (id=1624)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=1624&action=view)
Source code of test program


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-03-17 11:39 -------
Created an attachment (id=1625)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=1625&action=view)
Shell script that I used for compiling glibc 2.3.6, the most recent glibc with
linuxthreads support


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2007-03-17 16:46 -------
And where is a bug?  If you have proposals make them.  Otherwise go away.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-03-18 09:16 -------
(In reply to comment #3)
> And where is a bug?  If you have proposals make them.  Otherwise go away.

My proposal is to modify sem_post() such that it only calls lll_futex_wake() if
a thread is waiting in sem_wait(). This can be done by adding an atomic counter
to sem_t that represents the number of threads currently waiting inside sem_t.
This makes single-threaded usage of sem_post() and sem_wait() four times faster
[NPTL], and can speed up multithreaded usage of sem_post(). Of course this
optimization is only possible for semaphores used within the same process, and
not for semaphores shared over processes.

$ ./perf2
NPTL
mutex                      elapsed:   386132 us; per iteration:   38 ns.
semaphore                  elapsed:  2608915 us; per iteration:  260 ns.
custom semaphore           elapsed:   163885 us; per iteration:   16 ns.
semaphore ping-pong        elapsed: 19507114 us; per iteration: 1950 ns.
custom semaphore ping-pong elapsed: 10037632 us; per iteration: 1003 ns.
$ LD_LIBRARY_PATH=/home/bart/glibc236/lib: /home/bart/glibc236/lib/ld-linux.so.2
./glibc236-perf2
linuxthreads
mutex                      elapsed:   537161 us; per iteration:   53 ns.
semaphore                  elapsed:  1147894 us; per iteration:  114 ns.
custom semaphore           elapsed:   156364 us; per iteration:   15 ns.
semaphore ping-pong        elapsed: 28256860 us; per iteration: 2825 ns.
custom semaphore ping-pong elapsed: 18066323 us; per iteration: 1806 ns.

Note: because of the nature of this optimization the execution time of the
"custom semaphore ping-pong" test is much more variable than the execution time
of the other tests. It varies between 1000 ns per iteration (context switch time
?) and 2000 ns per iteration (context switch time + futex system call time ?).

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-03-18 09:18 -------
Created an attachment (id=1628)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=1628&action=view)
Source code of second version of test program.

Includes a sample implementation of faster sem_post() and sem_wait() functions
and a performance test of these functions.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-03-18 19:44 -------
(In reply to comment #4)
Sorry, but one of the above comments is wrong: on second thought, implementing a
semaphore waiter count will also work for process shared semaphores.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-03-28 10:23 -------
Above I added a suggestion for making the semaphore implementation faster.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2007-08-23 18:29 -------
For some time now the version in cvs does things differently.  It of course is
always thread safe, unlike your code, so the benchmark as-is is not usable.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #1625|application/octet-stream    |text/plain
          mime type|                            |


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug nptl/4209] Performance issue: NPTL semaphores work slower than linuxthreads semaphores

[bart dot vanassche at gmail dot com]

------- Additional Comments From bart dot vanassche at gmail dot com  2007-09-05 12:44 -------
(In reply to comment #8)
> For some time now the version in cvs does things differently.  It of course is
> always thread safe, unlike your code, so the benchmark as-is is not usable.

Apparently these optimizations are not yet included in the most recent glibc
release (2.6.1) ? These are the changes I found in CVS:
http://sourceware.org/cgi-bin/cvsweb.cgi/libc/nptl/sysdeps/unix/sysv/linux/sem_post.c.diff?r1=1.6&r2=1.7&cvsroot=glibc&f=h
http://sourceware.org/cgi-bin/cvsweb.cgi/libc/nptl/sysdeps/unix/sysv/linux/powerpc/sem_post.c.diff?r1=1.5&r2=1.6&cvsroot=glibc&f=h
...

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=4209

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.