FAIL nptl/tst-robustpi4

FAIL nptl/tst-robustpi4

[Stefan Liebler]

Hi,

On s390, I've recognized a FAIL in nptl/tst-robustpi4 in about 16 of 
10000 iterations of this testcase.
Does anyone else see failures here, too?

If the test fails, I get:
tst-robustpi4: ../nptl/pthread_mutex_lock.c:424: 
__pthread_mutex_lock_full: Assertion `INTERNAL_SYSCALL_ERRNO (e, __err) 
!= ESRCH || !robust' failed.
Didn't expect signal from child: got `Aborted'


The mutex is a "robust pi" one, thus the futex-syscall returned ESRCH here.
However, the comment before the assertion claims:
/* ESRCH can happen only for non-robust PI mutexes where
    the owner of the lock died.  */

The coredumps show that the tf-thread has already finished
and the do_test-thread tried pthread_mutex_lock(&m1) and shall return 
EOWNERDEAD (see nptl/tst-robust1.c:202 e = LOCK (&m1);).
But instead the futex syscall returns ESRCH.
and (gdb) p/x m1->__data.__lock
= 0xc0000000
= FUTEX_OWNER_DIED | FUTEX_WAITERS

Furthermore, the coredumps always show an even value in round-variable.
Thus tf is not joined (tst-robust1.c:190) before calling 
pthread_mutex_lock(tst-robust1.c:202).

If the do_test-thread waits a bit (e.g. doing something in a loop) 
before locking the mutex, the tf-thread has already called 
__exit_thread(pthread_create.c:478).
Then m1->__data.__lock is already marked with FUTEX_OWNER_DIED | 0 
before calling the futex syscall in pthread_mutex_lock 
(pthread_mutex_lock.c:411).
Then the futex syscall takes over the mutex by setting __lock to 
FUTEX_OWNER_DIED | do_test-TID and is returning 0.
If I run the test with such a "wait-loop" for several times, I see no fails.

If the do_test-thread locks the mutex before __exit_thread() is called 
in tf-thread, the futex syscall sets FUTEX_WAITERS bit and blocks until 
tf-thread has exited.
Afterwards 0 is returned and m1->__data.__lock is 0xc0000000
= FUTEX_OWNER_DIED | FUTEX_WAITERS.
If I run the test with a "wait-loop" before pthread_testcancel in 
tf-thread for several times as described, I see no fails, too.

It seems as a race between futex- and exit-syscall causes ESRCH result 
from futex-syscall.

I see those fails with Linux 4.8 / 4.9 running in a z/VM guest
as well as with 4.6 on a LPAR (but less often).

Bye
Stefan

Re: FAIL nptl/tst-robustpi4

[Carlos O'Donell]

On 01/26/2017 10:29 AM, Stefan Liebler wrote:
> It seems as a race between futex- and exit-syscall causes ESRCH
> result from futex-syscall.
> 
> I see those fails with Linux 4.8 / 4.9 running in a z/VM guest as
> well as with 4.6 on a LPAR (but less often).

I've seen tst-robustpi7 and tst-robustpi8 failures on all hardware
across a wide number of kernels, but never tst-robustpi4.

https://sourceware.org/bugzilla/show_bug.cgi?id=19004

The robustpi support is certainly not very robust as Torvald's
recent fixes show, and there still remains at least one design
flaw that can't be fixed.

e.g.
https://sourceware.org/bugzilla/show_bug.cgi?id=14485

and...
https://sourceware.org/bugzilla/show_bug.cgi?id=19089

-- 
Cheers,
Carlos.

Re: FAIL nptl/tst-robustpi4

[Torvald Riegel]

On Thu, 2017-01-26 at 11:12 -0500, Carlos O'Donell wrote:
> On 01/26/2017 10:29 AM, Stefan Liebler wrote:
> > It seems as a race between futex- and exit-syscall causes ESRCH
> > result from futex-syscall.

I'll have a closer look at this.

> > I see those fails with Linux 4.8 / 4.9 running in a z/VM guest as
> > well as with 4.6 on a LPAR (but less often).
> 
> I've seen tst-robustpi7 and tst-robustpi8 failures on all hardware
> across a wide number of kernels, but never tst-robustpi4.
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=19004
> 
> The robustpi support is certainly not very robust as Torvald's
> recent fixes show, and there still remains at least one design
> flaw that can't be fixed.
> 
> e.g.
> https://sourceware.org/bugzilla/show_bug.cgi?id=14485

The underlying problem for that bug does not affect PI+robust, just
robust, I think.  Unless I forgot about something, PI+robust should
always use the kernel to unlock.

Re: FAIL nptl/tst-robustpi4 [BZ 23183]

[Stefan Liebler]

On 01/26/2017 05:22 PM, Torvald Riegel wrote:
> On Thu, 2017-01-26 at 11:12 -0500, Carlos O'Donell wrote:
>> On 01/26/2017 10:29 AM, Stefan Liebler wrote:
>>> It seems as a race between futex- and exit-syscall causes ESRCH
>>> result from futex-syscall.
> 
> I'll have a closer look at this.
> 
>>> I see those fails with Linux 4.8 / 4.9 running in a z/VM guest as
>>> well as with 4.6 on a LPAR (but less often).
>>
>> I've seen tst-robustpi7 and tst-robustpi8 failures on all hardware
>> across a wide number of kernels, but never tst-robustpi4.
>>
>> https://sourceware.org/bugzilla/show_bug.cgi?id=19004
>>
>> The robustpi support is certainly not very robust as Torvald's
>> recent fixes show, and there still remains at least one design
>> flaw that can't be fixed.
>>
>> e.g.
>> https://sourceware.org/bugzilla/show_bug.cgi?id=14485
> 
> The underlying problem for that bug does not affect PI+robust, just
> robust, I think.  Unless I forgot about something, PI+robust should
> always use the kernel to unlock.
> 
> 
> 


Hi,

in the meantime, Florian Weimer could also reproduce this issue and 
opened the bugzilla Bug 23183 - tst-robustpi4 test failure 
(https://sourceware.org/bugzilla/show_bug.cgi?id=23183).

I've also dig a bit deeper - see details in bugzilla - and was also able 
to reproduce it on intel.

If the thread with locked mutex is executing the exit-syscall
while the main-thread is executing the futex-syscall,
then it could lead to this ESRCH return value of the futex-syscall which 
triggers the assertion.

In this situation, the futex-syscall has already added the FUTEX_WAITERS 
bit to the lock-value and is then calling attach_to_pi_owner().

The exit-syscall is now setting the lock-value to FUTEX_WAITERS | 
FUTEX_OWNER_DIED and is proceeding.

attach_to_pi_owner() is now e.g. trying to get the owner-task and/or is 
testing if the owner is currently exiting. In those cases, ESRCH is 
returned!

Back in glibc, this assertion is triggered:
/* ESRCH can happen only for non-robust PI mutexes where
    the owner of the lock died.  */
assert (INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust);

The assertion/comment does not agree with the current behaviour of the 
kernel. Any ideas?

Bye
Stefan

Re: FAIL nptl/tst-robustpi4 [BZ 23183]

[Florian Weimer]

On 06/29/2018 08:54 AM, Stefan Liebler wrote:
> On 01/26/2017 05:22 PM, Torvald Riegel wrote:
>> On Thu, 2017-01-26 at 11:12 -0500, Carlos O'Donell wrote:
>>> On 01/26/2017 10:29 AM, Stefan Liebler wrote:
>>>> It seems as a race between futex- and exit-syscall causes ESRCH
>>>> result from futex-syscall.
>>
>> I'll have a closer look at this.
>>
>>>> I see those fails with Linux 4.8 / 4.9 running in a z/VM guest as
>>>> well as with 4.6 on a LPAR (but less often).
>>>
>>> I've seen tst-robustpi7 and tst-robustpi8 failures on all hardware
>>> across a wide number of kernels, but never tst-robustpi4.
>>>
>>> https://sourceware.org/bugzilla/show_bug.cgi?id=19004
>>>
>>> The robustpi support is certainly not very robust as Torvald's
>>> recent fixes show, and there still remains at least one design
>>> flaw that can't be fixed.
>>>
>>> e.g.
>>> https://sourceware.org/bugzilla/show_bug.cgi?id=14485
>>
>> The underlying problem for that bug does not affect PI+robust, just
>> robust, I think.  Unless I forgot about something, PI+robust should
>> always use the kernel to unlock.

> in the meantime, Florian Weimer could also reproduce this issue and 
> opened the bugzilla Bug 23183 - tst-robustpi4 test failure 
> (https://sourceware.org/bugzilla/show_bug.cgi?id=23183).
> 
> I've also dig a bit deeper - see details in bugzilla - and was also able 
> to reproduce it on intel.
> 
> If the thread with locked mutex is executing the exit-syscall
> while the main-thread is executing the futex-syscall,
> then it could lead to this ESRCH return value of the futex-syscall which 
> triggers the assertion.
> 
> In this situation, the futex-syscall has already added the FUTEX_WAITERS 
> bit to the lock-value and is then calling attach_to_pi_owner().
> 
> The exit-syscall is now setting the lock-value to FUTEX_WAITERS | 
> FUTEX_OWNER_DIED and is proceeding.
> 
> attach_to_pi_owner() is now e.g. trying to get the owner-task and/or is 
> testing if the owner is currently exiting. In those cases, ESRCH is 
> returned!

Does the kernel look at the TID and determine that it no longer exists, 
or does it use the FUTEX_OWNER_DIED bit to detect this situation?

I'm worried that using the TID introduces a TID race here.

Thanks,
Florian

Re: FAIL nptl/tst-robustpi4 [BZ 23183]

[Stefan Liebler]

On 06/29/2018 09:39 AM, Florian Weimer wrote:
> On 06/29/2018 08:54 AM, Stefan Liebler wrote:
>> On 01/26/2017 05:22 PM, Torvald Riegel wrote:
>>> On Thu, 2017-01-26 at 11:12 -0500, Carlos O'Donell wrote:
>>>> On 01/26/2017 10:29 AM, Stefan Liebler wrote:
>>>>> It seems as a race between futex- and exit-syscall causes ESRCH
>>>>> result from futex-syscall.
>>>
>>> I'll have a closer look at this.
>>>
>>>>> I see those fails with Linux 4.8 / 4.9 running in a z/VM guest as
>>>>> well as with 4.6 on a LPAR (but less often).
>>>>
>>>> I've seen tst-robustpi7 and tst-robustpi8 failures on all hardware
>>>> across a wide number of kernels, but never tst-robustpi4.
>>>>
>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=19004
>>>>
>>>> The robustpi support is certainly not very robust as Torvald's
>>>> recent fixes show, and there still remains at least one design
>>>> flaw that can't be fixed.
>>>>
>>>> e.g.
>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=14485
>>>
>>> The underlying problem for that bug does not affect PI+robust, just
>>> robust, I think.  Unless I forgot about something, PI+robust should
>>> always use the kernel to unlock.
> 
>> in the meantime, Florian Weimer could also reproduce this issue and 
>> opened the bugzilla Bug 23183 - tst-robustpi4 test failure 
>> (https://sourceware.org/bugzilla/show_bug.cgi?id=23183).
>>
>> I've also dig a bit deeper - see details in bugzilla - and was also 
>> able to reproduce it on intel.
>>
>> If the thread with locked mutex is executing the exit-syscall
>> while the main-thread is executing the futex-syscall,
>> then it could lead to this ESRCH return value of the futex-syscall 
>> which triggers the assertion.
>>
>> In this situation, the futex-syscall has already added the 
>> FUTEX_WAITERS bit to the lock-value and is then calling 
>> attach_to_pi_owner().
>>
>> The exit-syscall is now setting the lock-value to FUTEX_WAITERS | 
>> FUTEX_OWNER_DIED and is proceeding.
>>
>> attach_to_pi_owner() is now e.g. trying to get the owner-task and/or 
>> is testing if the owner is currently exiting. In those cases, ESRCH is 
>> returned!
> 
> Does the kernel look at the TID and determine that it no longer exists, 
> or does it use the FUTEX_OWNER_DIED bit to detect this situation?
> 
> I'm worried that using the TID introduces a TID race here.
> 
> Thanks,
> Florian
> 

There can be different cases.
If the exit-syscall has already called handle_futex_death(). Then the 
TID is zeroed and the lock value is FUTEX_WAITERS | FUTEX_OWNER_DIED. 
Then attach_to_pi_onwer() fails with ESRCH due to:
pid_t pid = uval & FUTEX_TID_MASK;
struct task_struct *p;
if (!pid) return -ESRCH;
...
But even if the TID is available, it can fail due to:
	p = find_get_task_by_vpid(pid);
	if (!p) return -ESRCH;

If we've got the task_struct and if (p->flags & PF_EXITING) is true, it 
depends on (p->flags & PF_EXITPIDONE). Then either ESRCH is returned 
immediately. Or EAGAIN is returned which leads to calling 
attach_to_pi_owner() again. After one/some rounds, it finally returns ESRCH.