[Bug nptl/14417] New: EAGAIN handling in pthread_cond_wait make pulseaudio hang

[Bug nptl/14417] New: EAGAIN handling in pthread_cond_wait make pulseaudio hang

[dev at blino dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

             Bug #: 14417
           Summary: EAGAIN handling in pthread_cond_wait make pulseaudio
                    hang
           Product: glibc
           Version: 2.16
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nptl
        AssignedTo: unassigned@sourceware.org
        ReportedBy: dev@blino.org
                CC: drepper.fsp@gmail.com
    Classification: Unclassified


Since commit c5a0802a682dba23f92d47f0f99775aebfbe2539 (Handle EAGAIN from
FUTEX_WAIT_REQUEUE_PI), pulseaudio apps often hang.

Example backtrace from gnome-shell and libcanberra:

Thread 1 (Thread 0x7f975cfe8900 (LWP 2367)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:144
#1  0x00007f97558a4aa0 in pa_threaded_mainloop_wait (m=0x21bdf00) at
pulse/thread-mainloop.c:206
#2 0x00007f97460aaa0b in pulse_driver_play (c=0x21b5280, id=<optimized
out>, proplist=0x704c9c0, cb=<optimized out>, userdata=<optimized
out>) at pulse.c:1085
#3 0x00007f975955624e in ca_context_play_full (c=c@entry=0x21b5280,
id=id@entry=1, p=0x704c9c0, cb=cb@entry=0x0,
userdata=userdata@entry=0x0) at common.c:522
#4  0x00007f97595565cf in ca_context_play (c=0x21b5280, id=1) at common.c:462
...

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[dev at blino dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Olivier Blin <dev at blino dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |law at redhat dot com

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[dev at blino dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Olivier Blin <dev at blino dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |schwab@linux-m68k.org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[dev at blino dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

--- Comment #1 from Olivier Blin <dev at blino dot org> 2012-07-29 14:40:13 UTC ---
Most distributions (Fedora, Debian, openSUSE) revert this patch.

Quoting an email from Jeff below
http://sourceware.org/ml/libc-alpha/2012-01/msg00002.html

An FYI, this patch:


commit c5a0802a682dba23f92d47f0f99775aebfbe2539
Author: Andreas Schwab <schwab@redhat.com>
Date:   Mon Nov 28 13:38:19 2011 +0100

    Handle EAGAIN from FUTEX_WAIT_REQUEUE_PI


Has been reported as causing numerous problems in Fedora & Debian.  I
don't think anyone has done any serious analysis of the issue, but the
patch has been pulled from both distributions because of the
instability it's introduced.

https://bugzilla.redhat.com/show_bug.cgi?id=769421
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=651899

It might be worth your time to dig further into the change or pull it
from 2.15 pending a deeper investigation.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[carlos_odonell at mentor dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Carlos O'Donell <carlos_odonell at mentor dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos_odonell at mentor
                   |                            |dot com
   Target Milestone|---                         |2.17

--- Comment #2 from Carlos O'Donell <carlos_odonell at mentor dot com> 2012-08-03 03:43:48 UTC ---
This is going to be a tricky issue to resolve.

We don't have a smaller test case than "stuff stops working well", but clearly
the range of failures means that the changes in libc are broken.

Have you found a smaller test case that uses just pthreads?

In the meantime I do think the right solution is to revert the patch. We'll see
if we can do that for 2.17.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[law at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

--- Comment #3 from law at redhat dot com 2012-08-03 17:52:01 UTC ---
Olivier,
Thanks for opening the report.  It's been on my TODO list for a while.

Everyone else,
If someone has a good understanding of the core pthread_condattr_t structure,
particularly the relationships between the _seq fields, documenting them might
be a significant help.

One of the things that was incredibly frustrating trying to debug this issue
was the inadequate documentation of key data structures.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[dev at blino dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

--- Comment #4 from Olivier Blin <dev at blino dot org> 2012-08-03 21:55:44 UTC ---
Carlos: sorry, I don't have a more specific test case. I just noticed the
pulsaudio issue after we upgraded from glibc 2.14 to 2.16 in Mageia, and that
all other mainstream distributions revert his commit

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[law at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

--- Comment #5 from law at redhat dot com 2012-08-03 22:00:22 UTC ---
pulseaudio is the current best known way to reproduce the problem. 


AUDIODRIVER=pulseaudio play -n -c1  synth whitenoise band -n 100 20 \
        band -n 50 20 gain +25 fade h 1 864000 1


Fails once or twice every ten attempts within the first few seconds.  I was
able to make it fail regularly with a strategic breakpoint in the low level
pthread code after releasing on the locks (details fade, but clearly it depends
on arranging the the kernel to return EAGAIN to the futex call).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #6 from Rich Felker <bugdal at aerifal dot cx> 2012-08-04 20:49:17 UTC ---
I would hesitate to try fixing this without also looking at and possibly fixing
bug #12875. The whole sequence number wakeup thing for condition variables is
fundamentally broken in NPTL and needs to be fixed. Basically the issue is that
the current code is over-engineered to avoid spurious wakeups, but in the
process it suppresses some non-spurious wakeups...

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[triegel at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Torvald Riegel <triegel at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |triegel at redhat dot com

--- Comment #7 from Torvald Riegel <triegel at redhat dot com> 2012-09-18 14:37:24 UTC ---
(In reply to comment #6)
> I would hesitate to try fixing this without also looking at and possibly fixing
> bug #12875. The whole sequence number wakeup thing for condition variables is
> fundamentally broken in NPTL and needs to be fixed.

How so?  You seem to see issues that go beyond bug #12875 (which I don't see as
being a bug currently).  If so, please link to them here.

> Basically the issue is that
> the current code is over-engineered to avoid spurious wakeups, but in the
> process it suppresses some non-spurious wakeups...

I'm not aware of any lost wake-ups for non-PI cond vars.  If you have provided
an alternative implementation proposal in the past, could you link to it when
making such comments please?  If you haven't but have a proposal now, please
link to it here or post to glibc-alpha.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |siddhesh at redhat dot com
         AssignedTo|unassigned at sourceware    |siddhesh at redhat dot com
                   |dot org                     |

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[siddhesh at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14417

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #8 from Siddhesh Poyarekar <siddhesh at redhat dot com> 2012-10-05 13:28:55 UTC ---
Fixed in master:
http://sourceware.org/git/?p=glibc.git;a=commit;h=c30e8edf7c56e55a81173da39f3e721ab17b9db6

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nptl/14417] EAGAIN handling in pthread_cond_wait make pulseaudio hang

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14417

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.