[Bug nptl/12674] New: sem_post/sem_wait race causing sem_post to return EINVAL

[Bug nptl/12674] New: sem_post/sem_wait race causing sem_post to return EINVAL

[dhatch at ilm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

           Summary: sem_post/sem_wait race causing sem_post to return
                    EINVAL
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: nptl
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: dhatch@ilm.com


Created attachment 5671
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5671
the test program, to be run in gdb as described

There appears to be a race in the implementation of sem_post/sem_wait on AMD64
(nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S in the source code)
which sometimes causes sem_post to access freed memory
and to fail with EINVAL.

In a nutshell, if sem_post happens to go to sleep
right after it increments sem->value
but before it looks at sem->nwaiters,
another thread can sail through a sem_wait without blocking
and destroy the semaphore,
so that when the sem_post thread wakes up and looks at sem->nwaiters,
it is looking at already-freed (and possibly unmapped) memory.

The bug was originally filed as gentoo bug 93366
( http://bugs.gentoo.org/show_bug.cgi?id=93366 ).

It's extremely hard to reproduce,
and I don't have a simple program that can demonstrate the problem reliably
by just running it (for less than a million years).
But it can be reproduced consistently 
either by hacking up the sem_post source code
and adding a sleep() at a crucial point,
or by carefully stopping and resuming the threads
in a debugger with thread-specific breakpoints.
I'll include instructions for doing the latter using gdb >=7.1.

We're observing the problem on an AMD64 machine
running RHEL5.3 Linux,
with glibc-2.5-34.el5_3.1
and gcc-4.1.2-44.el5,
which I know is ancient 
but I also downloaded the most current glibc source code today
and compiled the sem_post.S and sem_wait.S from it,
and I can still reproduce the problem using those.


Here are the instructions for reproducing the problem
using gdb 7.1 or 7.2 on the attached program
(gdb 7.0.1 and earlier fail with a supposed syntax error
on the "b *(sem_post+18) thread 3").


% gcc -Wall -g semtest.c -lpthread -o semtest
% gdb ./semtest

    # per http://sourceware.org/gdb/onlinedocs/gdb/Non_002dStop-Mode.html ...
    # Enable the async interface.
    set target-async 1
    # If using the CLI, pagination breaks non-stop.
    set pagination off
    # Finally, turn it on!
    set non-stop on 

    b waiter
    b poster
    r
        # thread 2 stops in waiter
        # thread 3 stops in poster

    t 2
    b sem_wait thread 2 
    c
        # thread 2 (waiter) stops at the beginning of sem_wait(varsem)

    disas sem_post 
        # look for the "cmpq $0x0,0x8(%rdi)" and put a breakpoint there.
        # in older versions it's sem_post+4;
        # in newer versions it's sem_post+18.
    t 3
    b *(sem_post+18) thread 3    <-- or sem_post+4 or whatever
    c
        # thread 3 (poster) stops at the breakpoint inside sem_post,
        # after incrementing varsem->value (4-byte value 0 bytes into the
object)
        # but before looking at varsem->nwaiters (8-byte value 8 bytes into the
object)

    t 2
    b free thread 2
    c
        # thread 2 (waiter) sails through the sem_wait without blocking,
        # calls sem_destroy(varsem),
        # trashes the memory,
        # and stops at the beginning of free

    t 3
    c
        # thread 3 (poster) resumes in the middle of sem_post,
        # looks at varsem->nwaiters and sees it's nonzero (trash)
        # so it makes the FUTEX_WAKE syscall which returns EINVAL,
        # the program exits with error message
        # "sem_post() in poster: Invalid argument"



I hope I am not overinflating this bug's severity by calling
it "critical" ("major" would feel more appropriate to me,
but there seems to be no "major" option, only "normal" and "critical").
Although failure is rare,
we are about to be forced to implement our own semaphores
rather than using the posix semaphores because of this bug,
so it does seem rather severe.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[dhatch at ilm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Don Hatch <dhatch at ilm dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dhatch at ilm dot com

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[albertito at blitiri dot com.ar]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Alberto Bertogli <albertito at blitiri dot com.ar> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |albertito at blitiri dot
                   |                            |com.ar

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[vapier at gentoo dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Mike Frysinger <vapier at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |toolchain at gentoo dot org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[ppluzhnikov at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Paul Pluzhnikov <ppluzhnikov at google dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ppluzhnikov at google dot
                   |                            |com

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID

--- Comment #1 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-04-17 03:49:43 UTC ---
Why would this at all be a bug?  The fact that the sem_wait succeeds doesn't
indicate at all that the semaphore is unused and destroying an unused semaphore
is of course completely illegal.  Your code is wrong in assuming what it does. 
You have to wait for the sem_post call to also return before destroying the
semaphore.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[dhatch at ilm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #2 from Don Hatch <dhatch at ilm dot com> 2011-04-18 19:25:40 UTC ---
(In reply to comment #1)
> Why would this at all be a bug?  The fact that the sem_wait succeeds doesn't
> indicate at all that the semaphore is unused and destroying an unused semaphore
> is of course completely illegal.  Your code is wrong in assuming what it does. 
> You have to wait for the sem_post call to also return before destroying the
> semaphore.

Hi Ulrich,
Thanks for looking at this.

We're not completely confident that this usage is legal...
but we're not convinced yet that it's illegal either.

In our program, the sem_post itself is intended to indicate to the waiting
thread that it's safe to destroy the semaphore
(and, in a real program, to destroy some associated resource as well).
If the waiter thread has to wait for the sem_post call to return, as you say,
what would be a mechanism for doing that?  Another semaphore?
Would you agree that then either the semaphore,
or the semaphore-that-protects-the-semaphore, etc.
would need to be an object that persists significantly longer
than the resources being protected?
Maybe this is a reasonable or necessary
restriction, but it's a significant one,
and if it's intentional, it would be very helpful to have it documented.

Various manual pages I've seen which come close to mentioning it,
and which seem to me to (weakly) to imply my usage is legal, are:

sem_destroy man page from my RHEL5.3 distribution (man-pages-2.39-12.el5):
"Destroying  a  semaphore  that other processes or threads are currently
blocked on (in sem_wait(3)) produces undefined behaviour."
(doesn't mention sem_post, but it seems like this would be
the appropriate place to mention it if it's illegal,
and the fact that it doesn't mention it seems to imply it's legal).

Various other sem_destroy man pages, such as the one from
Open Group Base Specifications
(http://pubs.opengroup.org/onlinepubs/009695399/functions/sem_destroy.html)
say:
"It is safe to destroy an initialized semaphore upon which no threads are
currently blocked. The effect of destroying a semaphore upon which other
threads are currently blocked is undefined."
(the most literal reading of this implies that in my case,
it's safe to destroy the semaphore, since it's certainly
the case that no threads are currently blocked on it).

The pthread_mutex_destroy man page (from man-pages-2.39-12.el5):
"It shall be safe to destroy an  initialized  mutex  that  is  unlocked.
Attempting to destroy a locked mutex results in undefined behavior."
(again, a literal reading of this implies my usage is safe.
of course this is talking about mutexes, not semaphores,
but I imagine all the same limitations and considerations apply).

The pthread_cond_destroy man page (from man-pages-2.39-12.el5):
"It shall be safe to destroy  an  initialized  condition  variable  upon which 
no threads are currently blocked. Attempting to destroy a condition variable
upon which other threads are currently blocked results in undefined behavior."
(my comment on this would be the same as for pthread_mutex above)

Unfortunately I don't have access to the pthreads standard...
does it take a definite position on this?  If it does,
it would be great to have that clarification added to all these man pages
so that future programmers will have no doubts about it.

Thanks,
Don Hatch

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[dhatch at ilm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #3 from Don Hatch <dhatch at ilm dot com> 2011-04-19 00:04:29 UTC ---
Upon further reading,
I see that the pthread_mutex_destroy man page
and the pthread_cond_destroy man page
both explicitly say that doing the analogous thing
to a mutex or condition variable is legal.

>From the same pthread_mutex_destroy man page
that I quoted earlier (from man-pages-2.39-12.el5):
"A mutex can be destroyed immediately after it is unlocked. For example,
consider the following code: [...]  In this case obj is reference counted and
obj_done() is called whenever a reference to the object is dropped. 
Implementations are required  to allow an object to be destroyed and freed and
potentially unmapped (for example, lines A and B) immediately after the object
is unlocked  (line C)."

>From the same pthread_cond_destroy man page
that I quoted earlier (from man-pages-2.39-12.el5):
"A condition variable can be destroyed immediately after all the threads that
are blocked on it are awakened. For example, consider the  following code:
[...] In  this  example,  the  condition variable and its list element may be
freed (line B) immediately after all threads waiting for it  are  awakened
(line A), since the mutex and the code ensure that no other thread can touch
the element to be deleted.

So if it's really the case that
posix semaphores don't provide the same guarantee
(or even if the spec says they do but the current implementation doesn't),
I think we can get that guarantee by implementing our own semaphores
in terms of mutexes and/or condition variables
(that is assuming the implementation of mutexes and condition variables
really does conform to the above quoted passages).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #4 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-04-19 11:25:39 UTC ---
There cannot be any question that it is illegal.  You pass a pointer to the
semaphore to sem_post and just because it is half-finished and a sem_wait
succeeds this doesn't mean the call must be done.  As I said, only when the
sem_post call also returns is the semaphore unused.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[dhatch at ilm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #5 from Don Hatch <dhatch at ilm dot com> 2011-04-19 22:06:36 UTC ---
(In reply to comment #4)
> There cannot be any question that it is illegal.  You pass a pointer to the
> semaphore to sem_post and just because it is half-finished and a sem_wait
> succeeds this doesn't mean the call must be done.  As I said, only when the
> sem_post call also returns is the semaphore unused.

Hi Ulrich,

Sorry if it seems I am belaboring this.
I understand your assertion; it's the same thing you said
in your first reply (Comment 1), right?
But it's not clear to me on what basis you are making this assertion.
Is it based on the spec,
or are you stating what you believe to be common sense and obvious?

If it is from the spec, please say so (quoting the relevant passage if
possible)
and that will end the discussion.
(And I will open a bug report against the man page,
asking for it to be amended to include the clarifying passage from the spec.)

But if you are arguing from common sense, then I think you are certainly wrong
about it being obvious or the only reasonable interpretation.
One could equally well say from common sense
"you pass to mutex_unlock a pointer to the mutex and just because it is
half-finished and a subsequent mutex lock-and-unlock in another thread succeeds
this doesn't mean the call must be done... only when the first mutex_unlock
call also returns is the mutex unused"...
and yet the spec very explicitly disagrees
(as I quoted from the man page, which I assume is taken from the spec).
Similarly for condition variables.
All this leads me to believe that it was likely the intent of the spec authors
to say that, in general,
it is legal and legitimate usage to destroy any lock-like object
as soon as it is released for the last time by another thread
(which is always *before* the releasing function literally returns in that
other thread).
So it is on that basis that I say I believe it may be the intent of the spec
that my usage is legal, contrary to your assertion.

If you are still sure this is not the case,
would you please elaborate on your reasoning?

Thanks.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[lopresti at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Pat <lopresti at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lopresti at gmail dot com

--- Comment #6 from Pat <lopresti at gmail dot com> 2011-08-04 04:48:18 UTC ---
Ulrich does not understand your question because he is assuming you are an
idiot.

In particular, he is not reading your bug report carefully enough to recognize
that the behavior you have identified is a fundamental race rendering
semaphores useless.

It is a pretty simple question, really.  Given a semaphore initialized to zero,
one thread that makes one call to sem_wait, and another that makes one call to
sem_post.  Which thread can safely destroy the semaphore?

Obviously, the thread calling sem_post cannot destroy the semaphore because it
cannot know that sem_wait has returned.

You have shown that the one calling sem_wait cannot destroy it either because
of this broken implementation.

So neither thread can destroy the semaphore without adding some additional
synchronization mechanism.  Ulrich says the behavior is not broken, which as
you rightly point out is ludicrous.  But that's Ulrich for you.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
                 CC|                            |bugdal at aerifal dot cx
         Resolution|INVALID                     |

--- Comment #7 from Rich Felker <bugdal at aerifal dot cx> 2011-08-07 18:08:55 UTC ---
After reading this bug report and followup comments, I have identified and
fixed the corresponding bug in musl's implementation of POSIX semaphores. The
fix is very easy and the same approach could easily be used to fix glibc/NPTL.
There is not yet a release with the fix, but you can see the commit/diff in our
git repository at:

http://git.etalabs.net/cgi-bin/gitweb.cgi?p=musl;a=commitdiff;h=88c4e720317845a8e01aee03f142ba82674cd23d;hp=88798393cab009ce78fe498051072db71ba9d035

The basic idea is that a waiter stores a flag that it's waiting in the atomic
semaphore value field, in addition to incrementing the waiter count. This way
sem_post can see "old" waiters by examining the waiters count *before*
atomically upping the semaphore value, and can see a "last minute" waiter in
the old semaphore value when it atomically replaces it with compare-and-swap.

The same approach works for fixing the corresponding bug in mutexes and
rwlocks. Note that a similar bug also exists for barriers, and I have a clean
solution for non-process-shared barriers, but no solution for process-shared
barriers that's not subject to failure cases.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[kevin.dempsey at aculab dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Kevin Dempsey <kevin.dempsey at aculab dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kevin.dempsey at aculab dot
                   |                            |com

--- Comment #8 from Kevin Dempsey <kevin.dempsey at aculab dot com> 2012-02-10 16:26:32 UTC ---
We have been getting the same problem on an Amazon EC2 instance running a
Fedora 8 (2.6.21.7-5.fc8 kernel-xen) based image with glibc.i686 2.7-2, using
the nosegneg variant. The program aborts when sem_post() returns an error and
has been averaging one failure every three months.

Having seen this bug report, I have been testing with a program based on the
original reporters source. On an EC2 instance I have not had it run for more
than 4 hours before failing (I have not seen a failure on bare metal). When a
failure does occur the strace output shows the futex() syscall has been made
with an invalid operation:
12072 futex(0x9152098, 0x1010101 /* FUTEX_??? */, 1) = -1 ENOSYS (Function not
implemented)
presumably because the PRIVATE field has been overwritten.

>From the glibc source repository it appears that this race was introduced when
the change was made to make sem_post() only call FUTEX_WAKE when there are
threads waiting. In fact, with the test program forced to use the old
implementation (using .symver) I haven't had it fail.

If the value and nwaiters were next to each other then they could both be
accessed atomically using cmpxchg8b (on i586 and later). Perhaps then somebody
skilled in the art could eliminate the race condition?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[kevin.dempsey at aculab dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #9 from Kevin Dempsey <kevin.dempsey at aculab dot com> 2012-02-10 16:28:04 UTC ---
Created attachment 6206
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6206
EC2 test program

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[carlos at systemhalted dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Carlos O'Donell <carlos at systemhalted dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at systemhalted dot
                   |                            |org
         AssignedTo|drepper.fsp at gmail dot    |carlos at systemhalted dot
                   |com                         |org

--- Comment #10 from Carlos O'Donell <carlos at systemhalted dot org> 2012-02-16 15:25:45 UTC ---
Reviewing...

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[raj at mischievous dot us]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

raj at mischievous dot us changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |raj at mischievous dot us

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[piotr.stanczyk at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

Piotr <piotr.stanczyk at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |piotr.stanczyk at gmail dot
                   |                            |com

--- Comment #11 from Piotr <piotr.stanczyk at gmail dot com> 2013-01-12 00:40:20 UTC ---
Hi, 

Just checking in to see if anyone has had a chance to look into this one at
all. Many thanks in advance.

Piotr

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[raj at mischievous dot us]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

raj at mischievous dot us changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|raj at mischievous dot us   |

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[piotr.stanczyk at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #12 from Piotr <piotr.stanczyk at gmail dot com> 2013-01-24 16:51:15 UTC ---
(In reply to comment #11)
> Hi, 
> 
> Just checking in to see if anyone has had a chance to look into this one at
> all. Many thanks in advance.
> 
> Piotr

Can anyone comment on this one, please?

Thanks

Piotr

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[michael.ballantyne at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #14 from Michael Ballantyne <michael.ballantyne at gmail dot com> ---
I've also run into this bug. Appears to be the cause of random failures in our
scientific computing framework's parallel back end that have kept us from
declaring it production ready for quite some time.

I reproduced in debian jessie with eglibc 2.17.

I've attached an even simpler test-case for purposes of reproducing in gdb
(simple.c). Gdb instructions below.

set target-async 1
set pagination off
set non-stop on
b poster
b sem_wait
r

disas sem_post

# half-finish the sem-post
b *(sem_post+18) thread 2
t 2
c

# run the sem_wait, destroy, and trash memory
b free thread 1
t 1
c

# finish the sem_post and get an error
t 2
c

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|carlos at redhat dot com           |unassigned at sourceware dot org

--- Comment #15 from Carlos O'Donell <carlos at redhat dot com> ---
I'm not looking at this right now, but someone should review this if we want a
change in 2.19 which is open now.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[bugdal at aerifal dot cx]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #16 from Rich Felker <bugdal at aerifal dot cx> ---
The cause of the EINVAL is that

    orl     PRIVATE(%rdi), %esi

is being performed after the semaphore value is changed. To be correct, nothing
can be read from the semaphore value after the atomic instruction which changes
the semaphore value. Moving the check for number of waiters to before the
atomic operation, however, introduces a race condition which is even worse.
There are ways around this, such as the approach we use in musl (having both a
waiters counter and a new-waiter flag on the atomic so that the waiters count
can be read first), but such approaches would be fairly invasive and would
require careful review.

I think we could fix the most common manifestation of this bug simply by moving
the load of the PRIVATE field to take place before the atomic instruction. With
that change, the only observably incorrect behavior possible would be invalid
memory access (SIGSEGV or SIGBUS) if the storage for the semaphore was actually
unmapped (munmap or negative sbrk). This is still a possibility, and thus still
a bug which should be fixed, but it's much less likely/common than the EINVAL
issue that was actually reported.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[ismail at donmez dot ws]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

ismail at donmez dot ws changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ismail at donmez dot ws

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[ljanyst at cern dot ch]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

ljanyst at cern dot ch changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ljanyst at cern dot ch

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[triegel at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Torvald Riegel <triegel at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |triegel at redhat dot com

--- Comment #17 from Torvald Riegel <triegel at redhat dot com> ---
This is conceptually related to Bug 13690, whose resolution depends on the
outcome of a POSIX request for clarification.  The same kind of wording that
needs to be clarified for that bug is not present in the semaphore
specification, but it's essentially the same question of when POSIX
synchronization objects can be safely destroyed.  Therefore, I think it's good
to wait for a result of the clarification request.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[jackie.rosen at hushmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Jackie Rosen <jackie.rosen at hushmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jackie.rosen at hushmail dot com

--- Comment #18 from Jackie Rosen <jackie.rosen at hushmail dot com> ---
*** Bug 260998 has been marked as a duplicate of this bug. ***
Seen from the domain http://volichat.com
Page where seen: http://volichat.com/adult-chat-rooms
Marked for reference. Resolved as fixed @bugzilla.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[schwab at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Andreas Schwab <schwab at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|jackie.rosen at hushmail dot com   |

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[kevin.dempsey at aculab dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #19 from Kevin Dempsey <kevin.dempsey at aculab dot com> ---
Now that the austin group have clarified the expected behaviour of mutexes
(http://austingroupbugs.net/view.php?id=811) can progress be made on fixing
this?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[triegel at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #20 from Torvald Riegel <triegel at redhat dot com> ---
That's already work in progress.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[mtk.manpages at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Michael Kerrisk <mtk.manpages at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mtk.manpages at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

--- Comment #21 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, master has been updated
       via  042e1521c794a945edc43b5bfa7e69ad70420524 (commit)
      from  a8db092ec0c6742a9d41e1715946e90d4edfeec1 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=042e1521c794a945edc43b5bfa7e69ad70420524

commit 042e1521c794a945edc43b5bfa7e69ad70420524
Author: Carlos O'Donell <carlos@systemhalted.org>
Date:   Wed Jan 21 00:46:16 2015 -0500

    Fix semaphore destruction (bug 12674).

    This commit fixes semaphore destruction by either using 64b atomic
    operations (where available), or by using two separate fields when only
    32b atomic operations are available.  In the latter case, we keep a
    conservative estimate of whether there are any waiting threads in one
    bit of the field that counts the number of available tokens, thus
    allowing sem_post to atomically both add a token and determine whether
    it needs to call futex_wake.

    See:
    https://sourceware.org/ml/libc-alpha/2014-12/msg00155.html

-----------------------------------------------------------------------

Summary of changes:
 ChangeLog                                         |   52 +++
 NEWS                                              |   25 +-
 nptl/DESIGN-sem.txt                               |   46 --
 nptl/Makefile                                     |    5 +-
 nptl/sem_getvalue.c                               |   26 +-
 nptl/sem_init.c                                   |   35 +-
 nptl/sem_open.c                                   |    9 +-
 nptl/sem_post.c                                   |   67 +++-
 nptl/sem_timedwait.c                              |   96 +----
 nptl/sem_trywait.c                                |   50 ---
 nptl/sem_wait.c                                   |  101 ++---
 nptl/sem_waitcommon.c                             |  467 +++++++++++++++++++++
 nptl/structsem.sym                                |   12 -
 nptl/tst-sem11.c                                  |    9 +-
 nptl/tst-sem13.c                                  |   18 +-
 sysdeps/nptl/internaltypes.h                      |   24 +-
 sysdeps/unix/sysv/linux/alpha/sem_post.c          |    5 -
 sysdeps/unix/sysv/linux/i386/i486/sem_post.S      |  150 -------
 sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S |  327 --------------
 sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S   |   67 ---
 sysdeps/unix/sysv/linux/i386/i486/sem_wait.S      |  343 ---------------
 sysdeps/unix/sysv/linux/i386/i586/sem_post.S      |   19 -
 sysdeps/unix/sysv/linux/i386/i586/sem_timedwait.S |   19 -
 sysdeps/unix/sysv/linux/i386/i586/sem_trywait.S   |   19 -
 sysdeps/unix/sysv/linux/i386/i586/sem_wait.S      |   19 -
 sysdeps/unix/sysv/linux/i386/i686/sem_post.S      |   19 -
 sysdeps/unix/sysv/linux/i386/i686/sem_timedwait.S |   19 -
 sysdeps/unix/sysv/linux/i386/i686/sem_trywait.S   |   19 -
 sysdeps/unix/sysv/linux/i386/i686/sem_wait.S      |   19 -
 sysdeps/unix/sysv/linux/powerpc/sem_post.c        |   71 ----
 sysdeps/unix/sysv/linux/x86_64/sem_post.S         |   75 ----
 sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S    |  380 -----------------
 sysdeps/unix/sysv/linux/x86_64/sem_trywait.S      |   47 --
 sysdeps/unix/sysv/linux/x86_64/sem_wait.S         |  176 --------
 34 files changed, 732 insertions(+), 2103 deletions(-)
 delete mode 100644 nptl/DESIGN-sem.txt
 delete mode 100644 nptl/sem_trywait.c
 create mode 100644 nptl/sem_waitcommon.c
 delete mode 100644 nptl/structsem.sym
 delete mode 100644 sysdeps/unix/sysv/linux/alpha/sem_post.c
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i486/sem_post.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i486/sem_wait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i586/sem_post.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i586/sem_timedwait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i586/sem_trywait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i586/sem_wait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i686/sem_post.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i686/sem_timedwait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i686/sem_trywait.S
 delete mode 100644 sysdeps/unix/sysv/linux/i386/i686/sem_wait.S
 delete mode 100644 sysdeps/unix/sysv/linux/powerpc/sem_post.c
 delete mode 100644 sysdeps/unix/sysv/linux/x86_64/sem_post.S
 delete mode 100644 sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S
 delete mode 100644 sysdeps/unix/sysv/linux/x86_64/sem_trywait.S
 delete mode 100644 sysdeps/unix/sysv/linux/x86_64/sem_wait.S

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nptl/12674] sem_post/sem_wait race causing sem_post to return EINVAL

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12674

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #22 from Carlos O'Donell <carlos at redhat dot com> ---
Fixed by commit 042e1521c794a945edc43b5bfa7e69ad70420524

-- 
You are receiving this mail because:
You are on the CC list for the bug.