[Bug nptl/14652] New: Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[Bug nptl/14652] New: Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14652

             Bug #: 14652
           Summary: Cancelling a pthread_cond_wait with PRIO_INHERIT mutex
                    causes a deadlock
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nptl
        AssignedTo: unassigned@sourceware.org
        ReportedBy: siddhesh@redhat.com
                CC: drepper.fsp@gmail.com
    Classification: Unclassified


This is different from bug 14477, which is a deadlock due to an incorrect
exception table.  This problem is sometimes seen when a program with multiple
waiters and signallers are cancelled.  One of the waiters could get cancelled
at a point after it has returned successfully from a futex_wait and before it
disables async cancellation.  Since a wait with PRIO_INHERIT mutex uses the
FUTEX_WAIT_REQUEUE_PI futex operation, the futex syscall returns with the mutex
locked.  The cleanup handler subsequently tries to lock this already locked
mutex, resulting in a deadlock.

The following program demonstrates this on i386 as well as x86_64:

#include <pthread.h>
#include <stdio.h>

#define NUM 5
#define ITERS 100000

pthread_mutex_t mutex;
pthread_cond_t cond;

void cleanup (void *u)
{
  pthread_mutex_unlock (&mutex);
}

void *
signaller (void *u)
{
  int i;

  for (i = 0; i < ITERS / NUM; i++)
    {
      pthread_mutex_lock (&mutex);
      pthread_cond_signal (&cond);
      puts ("signalled");
      fflush (stdout);
      pthread_mutex_unlock (&mutex);
    }
}

void *
waiter (void *u)
{
  int i;

  for (i = 0; i < ITERS / NUM; i++)
    {
      struct timespec ts;
      clock_gettime(CLOCK_REALTIME, &ts);
      ts.tv_sec += 20;

      pthread_mutex_lock (&mutex);
      pthread_cleanup_push (cleanup, NULL);
      puts ("waiting...");
      pthread_cond_timedwait (&cond, &mutex, &ts);
      puts ("waited");
      fflush (stdout);
      pthread_mutex_unlock (&mutex);
      pthread_cleanup_pop (0);
    }
}

int
main (void)
{
  pthread_t w[NUM];
  pthread_t s;
  pthread_mutexattr_t attr;
  int i;

  for (i = 0; i < ITERS; i++)
    {
      pthread_mutexattr_init (&attr);
      pthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_INHERIT);
      pthread_cond_init (&cond, NULL);
      pthread_mutex_init (&mutex, &attr);

      for (i = 0; i < NUM; i++)
        pthread_create (&w[i], NULL, waiter, NULL);

      pthread_create (&s, NULL, signaller, NULL);

      for (i = 0; i < NUM; i++)
        {
          pthread_cancel (w[i]);
          pthread_join (w[i], NULL);
        }
      pthread_cancel (s);
      pthread_join (s, NULL);
    }

  return 0;
}

The program hangs after some iterations.  I have a patch in the works that
should fix this.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14652] Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14652

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at sourceware    |siddhesh at redhat dot com
                   |dot org                     |

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14652] Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=14652

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #1 from Rich Felker <bugdal at aerifal dot cx> 2012-10-01 19:54:43 UTC ---
This is a special case of bug # 12683. EVERY single cancellation point with
side effects has the same issue: cancellation can wrongly be acted upon after
the side effect took place. I suspect it's impossible to fix this bug without
addressing the general issue. I have a detailed article about it on my blog at
http://ewontfix.com/2/

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14652] Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14652

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #2 from Siddhesh Poyarekar <siddhesh at redhat dot com> 2012-10-03 05:27:08 UTC ---
Thanks for documenting the problem.  It is only related to this bug at a very
high level though, since it applies specifically to the pthread_cond_*wait
implementation on x86_64 and i386, which is fundamentally different from the
cancellation points in syscall wrappers.

The syscall wrappers don't have any cancellation handling on their own. 
pthread_cond_*wait code on the other hand already does cancellation handling
with its exception tables and the condvar_cleanup function, so the fix for this
specific problem is not very complicated.

There is the other problem of a cancel being called on a thread that was in an
async-cancel-unsafe signal handler that got called when async cancellation was
enabled, but that again will have to be treated differently.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14652] Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=14652

--- Comment #3 from Rich Felker <bugdal at aerifal dot cx> 2012-10-03 13:14:47 UTC ---
OK, then I think this suggests an approach for fixing the bug elsewhere:
replacing the "enable ascync cancel ; make syscall ; disable async cancel"
approach currently in use with an asm function that makes the syscall using the
proper exception tables.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14652] Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14652

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #4 from Siddhesh Poyarekar <siddhesh at redhat dot com> 2012-10-10 07:28:24 UTC ---
Committed fix:
http://sourceware.org/git/?p=glibc.git;a=commitdiff;h=0e3b5d6a6859d74a18033d3a55e0ee92340437b3

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14652] Cancelling a pthread_cond_wait with PRIO_INHERIT mutex causes a deadlock

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14652

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.