Several tst-robust* tests time out with recent Linux kernel

Several tst-robust* tests time out with recent Linux kernel

[Xi Ruoyao]

Hi,

With Linux 6.7.0-rc1, several tst-robust* tests time out on x86_64:

FAIL: nptl/tst-robust1
FAIL: nptl/tst-robust3
FAIL: nptl/tst-robust4
FAIL: nptl/tst-robust6
FAIL: nptl/tst-robust7
FAIL: nptl/tst-robust9

This does not happen with Linux 6.6.0.  Do you have some clue about it?

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: Several tst-robust* tests time out with recent Linux kernel

[Xi Ruoyao]

On Tue, 2023-11-14 at 02:33 +0800, Xi Ruoyao wrote:
> Hi,
> 
> With Linux 6.7.0-rc1, several tst-robust* tests time out on x86_64:
> 
> FAIL: nptl/tst-robust1
> FAIL: nptl/tst-robust3
> FAIL: nptl/tst-robust4
> FAIL: nptl/tst-robust6
> FAIL: nptl/tst-robust7
> FAIL: nptl/tst-robust9
> 
> This does not happen with Linux 6.6.0.  Do you have some clue about
> it?

Bisected to the kernel commit:

commit 5694289ce183bc3336407a78c8c722a0b9208f9b (HEAD)
Author: peterz@infradead.org <peterz@infradead.org>
Date:   Thu Sep 21 12:45:08 2023 +0200

    futex: Flag conversion
    
    Futex has 3 sets of flags:
    
     - legacy futex op bits
     - futex2 flags
     - internal flags
    
    Add a few helpers to convert from the API flags into the internal
    flags.
    
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
    Reviewed-by: Andr<C3><A9> Almeida <andrealmeid@igalia.com>
    Link: https://lore.kernel.org/r/20230921105247.722140574@noisy.programming.kicks-ass.net

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: Several tst-robust* tests time out with recent Linux kernel

[Peter Zijlstra]

On Tue, Nov 14, 2023 at 05:46:43PM +0800, Xi Ruoyao wrote:
> On Tue, 2023-11-14 at 02:33 +0800, Xi Ruoyao wrote:
> > Hi,
> > 
> > With Linux 6.7.0-rc1, several tst-robust* tests time out on x86_64:
> > 
> > FAIL: nptl/tst-robust1
> > FAIL: nptl/tst-robust3
> > FAIL: nptl/tst-robust4
> > FAIL: nptl/tst-robust6
> > FAIL: nptl/tst-robust7
> > FAIL: nptl/tst-robust9
> > 
> > This does not happen with Linux 6.6.0.  Do you have some clue about
> > it?
> 
> Bisected to the kernel commit:
> 
> commit 5694289ce183bc3336407a78c8c722a0b9208f9b (HEAD)
> Author: peterz@infradead.org <peterz@infradead.org>
> Date:   Thu Sep 21 12:45:08 2023 +0200
> 
>     futex: Flag conversion
>     
>     Futex has 3 sets of flags:
>     
>      - legacy futex op bits
>      - futex2 flags
>      - internal flags
>     
>     Add a few helpers to convert from the API flags into the internal
>     flags.
>     
>     Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>     Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
>     Reviewed-by: Andr<C3><A9> Almeida <andrealmeid@igalia.com>
>     Link: https://lore.kernel.org/r/20230921105247.722140574@noisy.programming.kicks-ass.net

I can confirm. I'm also going crazy trying to figure out how this
happens.

The below is sufficient to make it unhappy...

/me most puzzled

---
diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h
index b5379c0e6d6d..1a1f9301251f 100644
--- a/kernel/futex/futex.h
+++ b/kernel/futex/futex.h
@@ -17,7 +17,7 @@
  * restarts.
  */
 #ifdef CONFIG_MMU
-# define FLAGS_SHARED		0x01
+# define FLAGS_SHARED		0x10
 #else
 /*
  * NOMMU does not have per process address space. Let the compiler optimize
@@ -25,8 +25,8 @@
  */
 # define FLAGS_SHARED		0x00
 #endif
-#define FLAGS_CLOCKRT		0x02
-#define FLAGS_HAS_TIMEOUT	0x04
+#define FLAGS_CLOCKRT		0x20
+#define FLAGS_HAS_TIMEOUT	0x40
 
 #ifdef CONFIG_FAIL_FUTEX
 extern bool should_fail_futex(bool fshared);

Re: Several tst-robust* tests time out with recent Linux kernel

[Peter Zijlstra]

On Tue, Nov 14, 2023 at 04:31:00PM +0100, Peter Zijlstra wrote:
> On Tue, Nov 14, 2023 at 05:46:43PM +0800, Xi Ruoyao wrote:
> > On Tue, 2023-11-14 at 02:33 +0800, Xi Ruoyao wrote:
> > > Hi,
> > > 
> > > With Linux 6.7.0-rc1, several tst-robust* tests time out on x86_64:
> > > 
> > > FAIL: nptl/tst-robust1
> > > FAIL: nptl/tst-robust3
> > > FAIL: nptl/tst-robust4
> > > FAIL: nptl/tst-robust6
> > > FAIL: nptl/tst-robust7
> > > FAIL: nptl/tst-robust9
> > > 
> > > This does not happen with Linux 6.6.0.  Do you have some clue about
> > > it?
> > 
> > Bisected to the kernel commit:
> > 
> > commit 5694289ce183bc3336407a78c8c722a0b9208f9b (HEAD)
> > Author: peterz@infradead.org <peterz@infradead.org>
> > Date:   Thu Sep 21 12:45:08 2023 +0200
> > 
> >     futex: Flag conversion
> >     
> >     Futex has 3 sets of flags:
> >     
> >      - legacy futex op bits
> >      - futex2 flags
> >      - internal flags
> >     
> >     Add a few helpers to convert from the API flags into the internal
> >     flags.
> >     
> >     Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> >     Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
> >     Reviewed-by: Andr<C3><A9> Almeida <andrealmeid@igalia.com>
> >     Link: https://lore.kernel.org/r/20230921105247.722140574@noisy.programming.kicks-ass.net
> 
> I can confirm. I'm also going crazy trying to figure out how this
> happens.
> 
> The below is sufficient to make it unhappy...
> 
> /me most puzzled
> 
> ---
> diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h
> index b5379c0e6d6d..1a1f9301251f 100644
> --- a/kernel/futex/futex.h
> +++ b/kernel/futex/futex.h
> @@ -17,7 +17,7 @@
>   * restarts.
>   */
>  #ifdef CONFIG_MMU
> -# define FLAGS_SHARED		0x01
> +# define FLAGS_SHARED		0x10
>  #else
>  /*
>   * NOMMU does not have per process address space. Let the compiler optimize

Just the above seems sufficient.

Re: Several tst-robust* tests time out with recent Linux kernel

[Florian Weimer]

* Peter Zijlstra:

>> diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h
>> index b5379c0e6d6d..1a1f9301251f 100644
>> --- a/kernel/futex/futex.h
>> +++ b/kernel/futex/futex.h
>> @@ -17,7 +17,7 @@
>>   * restarts.
>>   */
>>  #ifdef CONFIG_MMU
>> -# define FLAGS_SHARED		0x01
>> +# define FLAGS_SHARED		0x10
>>  #else
>>  /*
>>   * NOMMU does not have per process address space. Let the compiler optimize
>
> Just the above seems sufficient.

There are a few futex_wake calls which hard-code the flags argument as
1:

kernel/futex/core.c=637=static int handle_futex_death(u32 __user *uaddr, struct task_struct *curr,
--
kernel/futex/core.c-686-         * this.
kernel/futex/core.c-687-         */
kernel/futex/core.c-688-        owner = uval & FUTEX_TID_MASK;
kernel/futex/core.c-689-
kernel/futex/core.c-690-        if (pending_op && !pi && !owner) {
kernel/futex/core.c:691:                futex_wake(uaddr, 1, 1, FUTEX_BITSET_MATCH_ANY);
kernel/futex/core.c-692-                return 0;
kernel/futex/core.c-693-        }
kernel/futex/core.c-694-
kernel/futex/core.c-695-        if (owner != task_pid_vnr(curr))
kernel/futex/core.c-696-                return 0;
--
kernel/futex/core.c-739-        /*
kernel/futex/core.c-740-         * Wake robust non-PI futexes here. The wakeup of
kernel/futex/core.c-741-         * PI futexes happens in exit_pi_state():
kernel/futex/core.c-742-         */
kernel/futex/core.c-743-        if (!pi && (uval & FUTEX_WAITERS))
kernel/futex/core.c:744:                futex_wake(uaddr, 1, 1, FUTEX_BITSET_MATCH_ANY);
kernel/futex/core.c-745-
kernel/futex/core.c-746-        return 0;
kernel/futex/core.c-747-}
kernel/futex/core.c-748-
kernel/futex/core.c-749-/*


Thanks,
Florian

Re: Several tst-robust* tests time out with recent Linux kernel

[Peter Zijlstra]

On Tue, Nov 14, 2023 at 05:43:20PM +0100, Florian Weimer wrote:
> * Peter Zijlstra:
> 
> >> diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h
> >> index b5379c0e6d6d..1a1f9301251f 100644
> >> --- a/kernel/futex/futex.h
> >> +++ b/kernel/futex/futex.h
> >> @@ -17,7 +17,7 @@
> >>   * restarts.
> >>   */
> >>  #ifdef CONFIG_MMU
> >> -# define FLAGS_SHARED		0x01
> >> +# define FLAGS_SHARED		0x10
> >>  #else
> >>  /*
> >>   * NOMMU does not have per process address space. Let the compiler optimize
> >
> > Just the above seems sufficient.
> 
> There are a few futex_wake calls which hard-code the flags argument as
> 1:
> 
> kernel/futex/core.c=637=static int handle_futex_death(u32 __user *uaddr, struct task_struct *curr,
> --
> kernel/futex/core.c-686-         * this.
> kernel/futex/core.c-687-         */
> kernel/futex/core.c-688-        owner = uval & FUTEX_TID_MASK;
> kernel/futex/core.c-689-
> kernel/futex/core.c-690-        if (pending_op && !pi && !owner) {
> kernel/futex/core.c:691:                futex_wake(uaddr, 1, 1, FUTEX_BITSET_MATCH_ANY);
> kernel/futex/core.c-692-                return 0;
> kernel/futex/core.c-693-        }
> kernel/futex/core.c-694-
> kernel/futex/core.c-695-        if (owner != task_pid_vnr(curr))
> kernel/futex/core.c-696-                return 0;
> --
> kernel/futex/core.c-739-        /*
> kernel/futex/core.c-740-         * Wake robust non-PI futexes here. The wakeup of
> kernel/futex/core.c-741-         * PI futexes happens in exit_pi_state():
> kernel/futex/core.c-742-         */
> kernel/futex/core.c-743-        if (!pi && (uval & FUTEX_WAITERS))
> kernel/futex/core.c:744:                futex_wake(uaddr, 1, 1, FUTEX_BITSET_MATCH_ANY);
> kernel/futex/core.c-745-
> kernel/futex/core.c-746-        return 0;
> kernel/futex/core.c-747-}
> kernel/futex/core.c-748-
> kernel/futex/core.c-749-/*

Urgh, thanks!

Confirmed, the below cures things. Although I should probably make that
FLAGS_SIZE_32 | FLAGS_SHARED against Linus' tree.

Let me go do a proper patch.

---
 kernel/futex/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index d1d7b3c175a4..e7793f0d5757 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -687,7 +687,7 @@ static int handle_futex_death(u32 __user *uaddr, struct task_struct *curr,
 	owner = uval & FUTEX_TID_MASK;
 
 	if (pending_op && !pi && !owner) {
-		futex_wake(uaddr, 1, 1, FUTEX_BITSET_MATCH_ANY);
+		futex_wake(uaddr, FLAGS_SHARED, 1, FUTEX_BITSET_MATCH_ANY);
 		return 0;
 	}
 
@@ -740,7 +740,7 @@ static int handle_futex_death(u32 __user *uaddr, struct task_struct *curr,
 	 * PI futexes happens in exit_pi_state():
 	 */
 	if (!pi && (uval & FUTEX_WAITERS))
-		futex_wake(uaddr, 1, 1, FUTEX_BITSET_MATCH_ANY);
+		futex_wake(uaddr, FLAGS_SHARED, 1, FUTEX_BITSET_MATCH_ANY);
 
 	return 0;
 }

Re: Several tst-robust* tests time out with recent Linux kernel

[Edgecombe, Rick P]

On Tue, 2023-11-14 at 21:14 +0100, Peter Zijlstra wrote:
> Urgh, thanks!
> 
> Confirmed, the below cures things. Although I should probably make
> that
> FLAGS_SIZE_32 | FLAGS_SHARED against Linus' tree.
> 
> Let me go do a proper patch.

I saw these fail on the glibc shadow stack branch today, and I also saw
this one failing:
FAIL: nptl/tst-robustpi8

It spit out:
mutex_timedlock of 41 in thread 1 failed with 22
child did not die of a signal in round 1

After the fix here I saw the others pass, but still not tst-robustpi8.
Not sure if it is some shadow stack complication. I can try to dig in
tomorrow if the problem doesn't jump out.

Re: Several tst-robust* tests time out with recent Linux kernel

[Peter Zijlstra]

On Wed, Nov 15, 2023 at 01:11:20AM +0000, Edgecombe, Rick P wrote:
> On Tue, 2023-11-14 at 21:14 +0100, Peter Zijlstra wrote:
> > Urgh, thanks!
> > 
> > Confirmed, the below cures things. Although I should probably make
> > that
> > FLAGS_SIZE_32 | FLAGS_SHARED against Linus' tree.
> > 
> > Let me go do a proper patch.
> 
> I saw these fail on the glibc shadow stack branch today, and I also saw
> this one failing:
> FAIL: nptl/tst-robustpi8

tip/locking/urgent (branch with the fix on) gets me:

root@ivb-ep:/usr/local/src/glibc# ./build/nptl/tst-robustpi8 
running child
verifying locks
running child
verifying locks
running child
verifying locks
running child
verifying locks
running child
verifying locks
root@ivb-ep:/usr/local/src/glibc#

Which, to my untrained eye, looks like a pass to me.

Re: Several tst-robust* tests time out with recent Linux kernel

[Edgecombe, Rick P]

On Wed, 2023-11-15 at 09:51 +0100, Peter Zijlstra wrote:
> On Wed, Nov 15, 2023 at 01:11:20AM +0000, Edgecombe, Rick P wrote:
> > On Tue, 2023-11-14 at 21:14 +0100, Peter Zijlstra wrote:
> > > Urgh, thanks!
> > > 
> > > Confirmed, the below cures things. Although I should probably
> > > make
> > > that
> > > FLAGS_SIZE_32 | FLAGS_SHARED against Linus' tree.
> > > 
> > > Let me go do a proper patch.
> > 
> > I saw these fail on the glibc shadow stack branch today, and I also
> > saw
> > this one failing:
> > FAIL: nptl/tst-robustpi8
> 
> tip/locking/urgent (branch with the fix on) gets me:
> 
> root@ivb-ep:/usr/local/src/glibc# ./build/nptl/tst-robustpi8 
> running child
> verifying locks
> running child
> verifying locks
> running child
> verifying locks
> running child
> verifying locks
> running child
> verifying locks
> root@ivb-ep:/usr/local/src/glibc#
> 
> Which, to my untrained eye, looks like a pass to me.

It bisects to this for me:
fbeb558b0dd0 ("futex/pi: Fix recursive rt_mutex waiter state")

Reading the patch, I'm not immediately clear what is going on but a few
comments stood out: "There be dragons here" "What could possibly go
wrong..." "This is a somewhat dangerous proposition".

Seems a likelihood of some race, but it reproduces reliably on my
machine. Haven't dug into debugging it yet. Any pointers?

Re: Several tst-robust* tests time out with recent Linux kernel

[Edgecombe, Rick P]

A bit more info...

The error returned to userspace is originating from:
https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L295

'uval' is often zero in that error case, but sometimes just a
mismatching value like: uval=0x567, task_pid_vnr()=0x564


Depending on the number of CPUs the VM is running on it reproduces or
not. When it does reproduce, the newly added path here is taken:
https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L1185
The path is taken a lot during the test, sometimes >400 times before
the above linked error is generated during the syscall. When it doesn't
reproduce, I never saw that new path taken.

More print statements make the reproduction less reliable, so it does
seem to have a race in the mix at least somewhat. Otherwise, I haven't
tried to understand what is going on here with all this highwire
locking.

Hope it helps.

Re: Several tst-robust* tests time out with recent Linux kernel

[Stefan Liebler]

[-- Attachment #1: Type: text/plain, Size: 2794 bytes --]

On 17.11.23 02:22, Edgecombe, Rick P wrote:
> A bit more info...
> 
> The error returned to userspace is originating from:
> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L295
> 
> 'uval' is often zero in that error case, but sometimes just a
> mismatching value like: uval=0x567, task_pid_vnr()=0x564
> 
> 
> Depending on the number of CPUs the VM is running on it reproduces or
> not. When it does reproduce, the newly added path here is taken:
> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L1185
> The path is taken a lot during the test, sometimes >400 times before
> the above linked error is generated during the syscall. When it doesn't
> reproduce, I never saw that new path taken.
> 
> More print statements make the reproduction less reliable, so it does
> seem to have a race in the mix at least somewhat. Otherwise, I haven't
> tried to understand what is going on here with all this highwire
> locking.
> 
> Hope it helps.
Hi,

I've also observed fails in glibc testcase nptl/tst-robust8pi with:
mutex_timedlock of 66 in thread 7 failed with 22
=> pthread_mutex_timedlock returns 22=EINVAL

I've saw it on s390x. There I've used kernel with
commit 120d99901eb288f1d21db3976df4ba347b28f9c7
s390/vfio-ap: do not reset queue removed from host config

But I also saw it on a x86_64 kvm-guest with Fedora 39 and
copr-repository with vanilla kernel:
Linux fedora 6.7.0-0.rc8.20240107gt52b1853b.366.vanilla.fc39.x86_64 #1
SMP PREEMPT_DYNAMIC Sun Jan  7 06:17:30 UTC 2024 x86_64 GNU/Linux

And reported it to libc-alpha ("FAILING nptl/tst-robust8pi"
https://sourceware.org/pipermail/libc-alpha/2024-January/154150.html)
where Florian Weimer pointed me to this thread.

I've reduced the test (see attachement) and now have only one process
with three threads. I only use one mutex with attributes like the
original testcase: PTHREAD_MUTEX_ROBUST_NP, PTHREAD_PROCESS_SHARED,
PTHREAD_PRIO_INHERIT.
Every thread is doing a loop with pthread_mutex_timedlock(abstime={0,0})
and if locked, pthread_mutex_unlock.

I've added some uprobes before and after the futex-syscall in
__futex_lock_pi64(in pthread_mutex_timedlock) and futex_unlock_pi(in
pthread_mutex_unlock). For me __ASSUME_FUTEX_LOCK_PI2 is not available,
but __ASSUME_TIME64_SYSCALLS is defined.

For me it looks like this (simplified ubprobes-trace):
<thread> <timestamp>: <probe>
t1 4309589.419744: before syscall in __futex_lock_pi64

t3 4309589.419745: before syscall in futex_unlock_pi
t2 4309589.419745: before syscall in __futex_lock_pi64

t3 4309589.419747: after syscall in futex_unlock_pi
t2 4309589.419747: after syscall in __futex_lock_pi64 ret=-22=EINVAL

t1 4309589.419748: after syscall in __futex_lock_pi64 ret=-110=ETIMEDOUT

Can you please have a look again?

Bye,
Stefan Liebler

[-- Attachment #2: tst-robust8pi-20240118.c --]
[-- Type: text/x-csrc, Size: 4161 bytes --]

//CFLAGS=-pthread
//LDFLAGS=-lpthread
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <assert.h>
#include <errno.h>
#include <unistd.h>

#define NUM_THREADS 3
#define THREAD_FUNC thr_func
#define USE_BARRIER 1
#ifndef ROUNDS
# define ROUNDS 100000000
#endif

typedef struct thr_info
{
  int nr;
  pthread_t thread;
} __attribute__ ((aligned (256))) thr_info_t;

#define THR_INIT()				\
  thr_info_t *thr = (thr_info_t *) arg;

#define THR_PRINTF(fmt, ...)			\
  printf ("#%d: " fmt, thr->nr, __VA_ARGS__)

#define THR_PUTS(msg)				\
  printf ("#%d: " msg "\n", thr->nr)

#if USE_BARRIER != 0
static pthread_barrier_t thrs_barrier;
#endif

static pthread_mutex_t mtx;
static const struct timespec before = { 0, 0 };

/* ###################################################################
   thread func
   ###############################################################  */

static void *
thr_func (void *arg)
{
  THR_INIT ();
  int state = 0;
  int fct;

#if 0
  /* 3 threads, 1xfct=0=pthread_mutex_lock, 2xfct=1=pthread_mutex_timedlock: EINVAL.  */
  fct = (thr->nr + 1) % 2;
#elif 0
  /* 3 threads, 2xfct=0=pthread_mutex_lock, 1xfct=1=pthread_mutex_timedlock: no fails.  */
  fct = (thr->nr) % 2;
#elif 1
  /* >3 threads, fct=1=only pthread_mutex_timedlock: EINVAL.  */
  fct = 1;
#endif

  int round = 0;
  THR_PRINTF ("started: fct=%d\n", fct);
#if USE_BARRIER != 0
  pthread_barrier_wait (&thrs_barrier);
#endif
  while (1)
    {

      if (state == 0)
	{
	  round ++;
	  int e;

	  switch (fct)
	    {
	    case 0:
	      e = pthread_mutex_lock (&mtx);
	      if (e != 0)
		{
		  THR_PRINTF ("mutex_lock failed with %d (round=%d)\n", e, round);
		  exit (1);
		}
	      state = 1;
	      break;
	    case 1:
	      e = pthread_mutex_timedlock (&mtx, &before);
	      if (e != 0 && e != ETIMEDOUT)
		{
		  THR_PRINTF ("mutex_timedlock failed with %d (round=%d)\n", e, round);
		  exit (1);
		}
	      break;
	    default:
	      e = pthread_mutex_trylock (&mtx);
	      if (e != 0 && e != EBUSY)
		{
		  THR_PRINTF ("mutex_trylock failed with %d (round=%d)\n", e, round);
		  exit (1);
		}
	      break;
	    }

	  if (e == EOWNERDEAD)
	    pthread_mutex_consistent (&mtx);

	  if (e == 0 || e == EOWNERDEAD)
	    state = 1;
	}
      else
	{
	  int e = pthread_mutex_unlock (&mtx);
	  if (e != 0)
	    {
	      THR_PRINTF ("mutex_unlock of failed with %d (round=%d)\n", e, round);
	      exit (1);
	    }
	  state = 0;
	}

      if (round >= ROUNDS)
	{
	  THR_PRINTF ("REACHED round %d. => exit\n", ROUNDS);
	  if (state != 0)
	    {
	      int e = pthread_mutex_unlock (&mtx);
	      if (e != 0)
		{
		  THR_PRINTF ("mutex_unlock@exit of failed with %d (round=%d)\n", e, round);
		  exit (1);
		}
	    }
	  break;
	}
    }

  return NULL;
}

int
main (void)
{
  int i;
  printf ("main: start %d threads.\n", NUM_THREADS);

#if USE_BARRIER != 0
  pthread_barrier_init (&thrs_barrier, NULL, NUM_THREADS + 1);
#endif

  pthread_mutexattr_t ma;
  if (pthread_mutexattr_init (&ma) != 0)
    {
      puts ("mutexattr_init failed");
      return 0;
    }
  if (pthread_mutexattr_setrobust (&ma, PTHREAD_MUTEX_ROBUST_NP) != 0)
    {
      puts ("mutexattr_setrobust failed");
      return 1;
    }
  if (pthread_mutexattr_setpshared (&ma, PTHREAD_PROCESS_SHARED) != 0)
    {
      puts ("mutexattr_setpshared failed");
      return 1;
    }
  if (pthread_mutexattr_setprotocol (&ma, PTHREAD_PRIO_INHERIT) != 0)
    {
      puts ("pthread_mutexattr_setprotocol failed");
      return 1;
    }

  if (pthread_mutex_init (&mtx, &ma) != 0)
    {
      puts ("pthread_mutex_init failed");
      return 1;
    }

  thr_info_t thrs[NUM_THREADS];
  for (i = 0; i < NUM_THREADS; i++)
    {
      thrs[i].nr = i;
      assert (pthread_create (&(thrs[i].thread), NULL, THREAD_FUNC, &(thrs[i]))
	      == 0);;
    }

#if USE_BARRIER != 0
  /* All threads start work after this barrier.  */
  pthread_barrier_wait (&thrs_barrier);
#endif

  for (i = 0; i < NUM_THREADS; i++)
    {
      pthread_join (thrs[i].thread, NULL);
    }

#if USE_BARRIER != 0
  pthread_barrier_destroy (&thrs_barrier);
#endif

  printf ("main: end.\n");
  return EXIT_SUCCESS;
}

Re: Several tst-robust* tests time out with recent Linux kernel

[Stefan Liebler]

On 19.01.24 14:56, Stefan Liebler wrote:
> On 17.11.23 02:22, Edgecombe, Rick P wrote:
>> A bit more info...
>>
>> The error returned to userspace is originating from:
>> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L295
>>
>> 'uval' is often zero in that error case, but sometimes just a
>> mismatching value like: uval=0x567, task_pid_vnr()=0x564
>>
>>
>> Depending on the number of CPUs the VM is running on it reproduces or
>> not. When it does reproduce, the newly added path here is taken:
>> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L1185
>> The path is taken a lot during the test, sometimes >400 times before
>> the above linked error is generated during the syscall. When it doesn't
>> reproduce, I never saw that new path taken.
>>
>> More print statements make the reproduction less reliable, so it does
>> seem to have a race in the mix at least somewhat. Otherwise, I haven't
>> tried to understand what is going on here with all this highwire
>> locking.
>>
>> Hope it helps.
> Hi,
> 
> I've also observed fails in glibc testcase nptl/tst-robust8pi with:
> mutex_timedlock of 66 in thread 7 failed with 22
> => pthread_mutex_timedlock returns 22=EINVAL
> 
> I've saw it on s390x. There I've used kernel with
> commit 120d99901eb288f1d21db3976df4ba347b28f9c7
> s390/vfio-ap: do not reset queue removed from host config
> 
> But I also saw it on a x86_64 kvm-guest with Fedora 39 and
> copr-repository with vanilla kernel:
> Linux fedora 6.7.0-0.rc8.20240107gt52b1853b.366.vanilla.fc39.x86_64 #1
> SMP PREEMPT_DYNAMIC Sun Jan  7 06:17:30 UTC 2024 x86_64 GNU/Linux
> 
> And reported it to libc-alpha ("FAILING nptl/tst-robust8pi"
> https://sourceware.org/pipermail/libc-alpha/2024-January/154150.html)
> where Florian Weimer pointed me to this thread.
> 
> I've reduced the test (see attachement) and now have only one process
> with three threads. I only use one mutex with attributes like the
> original testcase: PTHREAD_MUTEX_ROBUST_NP, PTHREAD_PROCESS_SHARED,
> PTHREAD_PRIO_INHERIT.
> Every thread is doing a loop with pthread_mutex_timedlock(abstime={0,0})
> and if locked, pthread_mutex_unlock.
> 
> I've added some uprobes before and after the futex-syscall in
> __futex_lock_pi64(in pthread_mutex_timedlock) and futex_unlock_pi(in
> pthread_mutex_unlock). For me __ASSUME_FUTEX_LOCK_PI2 is not available,
> but __ASSUME_TIME64_SYSCALLS is defined.
> 
> For me it looks like this (simplified ubprobes-trace):
> <thread> <timestamp>: <probe>
> t1 4309589.419744: before syscall in __futex_lock_pi64
> 
> t3 4309589.419745: before syscall in futex_unlock_pi
> t2 4309589.419745: before syscall in __futex_lock_pi64
> 
> t3 4309589.419747: after syscall in futex_unlock_pi
> t2 4309589.419747: after syscall in __futex_lock_pi64 ret=-22=EINVAL
> 
> t1 4309589.419748: after syscall in __futex_lock_pi64 ret=-110=ETIMEDOUT
> 
> Can you please have a look again?
> 
> Bye,
> Stefan Liebler

FYI
kernel commit "futex: Prevent the reuse of stale pi_state"
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/patch/?id=e626cb02ee8399fd42c415e542d031d185783903

fixes my issue on s390x.

Re: Several tst-robust* tests time out with recent Linux kernel

[Edgecombe, Rick P]

On Fri, 2024-01-19 at 14:56 +0100, Stefan Liebler wrote:
> I've reduced the test (see attachement) and now have only one process
> with three threads.

This tests fails on my setup as well:
main: start 3 threads.
#0: started: fct=1
#1: started: fct=1
#2: started: fct=1
#2: mutex_timedlock failed with 22 (round=28772)

But, after this patch:
https://lore.kernel.org/all/20240116130810.ji1YCxpg@linutronix.de/

...the attached test hangs.

However, the glibc test that was failing for me "nptl/tst-robustpi8"
passes with the linked patch applied. So I think that patch fixes the
issue I hit.

What is passing supposed to look like on the attached test?

Re: Several tst-robust* tests time out with recent Linux kernel

[Stefan Liebler]

On 29.01.24 23:23, Edgecombe, Rick P wrote:
> On Fri, 2024-01-19 at 14:56 +0100, Stefan Liebler wrote:
>> I've reduced the test (see attachement) and now have only one process
>> with three threads.
> 
> This tests fails on my setup as well:
> main: start 3 threads.
> #0: started: fct=1
> #1: started: fct=1
> #2: started: fct=1
> #2: mutex_timedlock failed with 22 (round=28772)
> 
> But, after this patch:
> https://lore.kernel.org/all/20240116130810.ji1YCxpg@linutronix.de/
> 
> ...the attached test hangs.
> 
> However, the glibc test that was failing for me "nptl/tst-robustpi8"
> passes with the linked patch applied. So I think that patch fixes the
> issue I hit.
> 
> What is passing supposed to look like on the attached test?

kernel commit "futex: Prevent the reuse of stale pi_state"
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/patch/?id=e626cb02ee8399fd42c415e542d031d185783903

fixes the issue on s390x.

With this commit, the test runs to the end:
main: start 3 threads.
#0: started: fct=1
#1: started: fct=1
#2: started: fct=1
#2: REACHED round 100000000. => exit
#0: REACHED round 100000000. => exit
#1: REACHED round 100000000. => exit
main: end.

If you want you can reduce the number of rounds by compiling with
-DROUNDS=XYZ or manually adjusting the ROUNDS macro define.