From: Jakub Jelinek <jakub@redhat.com>
To: Ulrich Drepper <drepper@redhat.com>
Cc: Glibc hackers <libc-hacker@sources.redhat.com>
Subject: [PATCH] private futex fixes
Date: Tue, 24 Jul 2007 15:59:00 -0000 [thread overview]
Message-ID: <20070724160250.GG4603@sunsite.mff.cuni.cz> (raw)
Hi!
This patch removes lll_private_futex_* macros, fixes x86_64
__lll_private_flag macro and updates many ports to sync with the
x86_64 and powerpc changes. Tested on i386 and x86_64.
I have checked all lll_futex_* and lll_*lock uses inside of
ld.so, libc.so and librt.so and found one place where
libc.so uses private futex, while nptl can use shared futex
on the same memory spot - flockfile/funlockfile. This bug
is present before and after this patch, I'll address that separately.
2007-07-24 Jakub Jelinek <jakub@redhat.com>
* allocatestack.c (__nptl_setxid, __wait_lookup_done): Replace
lll_private_futex_* (*) with lll_futex_* (*, LLL_PRIVATE).
* pthread_create.c (start_thread): Likewise.
* init.c (sighandler_setxid): Likewise.
* sysdeps/alpha/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/ia64/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/i386/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/s390/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/powerpc/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/x86_64/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/sparc/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/sh/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/pthread/aio_misc.h (AIO_MISC_NOTIFY, AIO_MISC_WAIT):
Likewise.
* sysdeps/pthread/gai_misc.h (GAI_MISC_NOTIFY, GAI_MISC_WAIT):
Likewise.
* sysdeps/unix/sysv/linux/unregister-atfork.c (__unregister_atfork):
Likewise.
* sysdeps/unix/sysv/linux/rtld-lowlevel.h (__rtld_waitzero,
__rtld_notify): Likewise.
* sysdeps/unix/sysv/linux/fork.c (__libc_fork): Likewise.
* sysdeps/unix/sysv/linux/powerpc/pthread_once.c (clear_once_control,
__pthread_once): Likewise.
* sysdeps/unix/sysv/linux/alpha/pthread_once.c (clear_once_control,
__pthread_once): Add LLL_PRIVATE as last argument to lll_futex_*.
* sysdeps/unix/sysv/linux/alpha/lowlevellock.h (FUTEX_PRIVATE_FLAG,
LLL_PRIVATE, LLL_SHARED, __lll_private_flag): Define.
(lll_futex_wait): Add private argument, define as wrapper around
lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake): Add private argument,
use __lll_private_flag macro.
(lll_robust_mutex_dead, __lll_mutex_unlock, __lll_robust_mutex_unlock,
__lll_mutex_unlock_force): Pass LLL_SHARED as last arg to lll_futex_*.
* sysdeps/unix/sysv/linux/ia64/pthread_once.c (clear_once_control,
__pthread_once): Add LLL_PRIVATE as last argument to lll_futex_*.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h (FUTEX_PRIVATE_FLAG,
LLL_PRIVATE, LLL_SHARED, __lll_private_flag): Define.
(lll_futex_wait): Add private argument, define as wrapper around
lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake): Add private argument,
use __lll_private_flag macro.
(__lll_mutex_unlock, __lll_robust_mutex_unlock, lll_wait_tid,
__lll_mutex_unlock_force): Pass LLL_SHARED as last arg to lll_futex_*.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h (__lll_private_flag):
Define.
(lll_futex_timed_wait, lll_futex_wake): Use it.
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
* sysdeps/unix/sysv/linux/s390/pthread_once.c (clear_once_control,
__pthread_once): Add LLL_PRIVATE as last argument to lll_futex_*.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h (FUTEX_PRIVATE_FLAG,
LLL_PRIVATE, LLL_SHARED, __lll_private_flag): Define.
(lll_futex_wait): Add private argument, define as wrapper around
lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake): Add private argument,
use __lll_private_flag macro.
(lll_robust_mutex_dead, __lll_mutex_unlock, __lll_robust_mutex_unlock,
lll_wait_tid, __lll_mutex_unlock_force): Pass LLL_SHARED as last arg
to lll_futex_*.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (__lll_private_flag):
Fix !__ASSUME_PRIVATE_FUTEX non-constant private case.
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
* sysdeps/unix/sysv/linux/sparc/pthread_once.c (clear_once_control,
__pthread_once): Add LLL_PRIVATE as last argument to lll_futex_*.
* sysdeps/unix/sysv/linux/sparc/lowlevellock.h (FUTEX_PRIVATE_FLAG,
LLL_PRIVATE, LLL_SHARED, __lll_private_flag): Define.
(lll_futex_wait): Add private argument, define as wrapper around
lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake): Add private argument,
use __lll_private_flag macro.
(lll_robust_mutex_dead, __lll_mutex_unlock, __lll_robust_mutex_unlock,
lll_wait_tid, __lll_mutex_unlock_force): Pass LLL_SHARED as last arg
to lll_futex_*.
* sysdeps/unix/sysv/linux/sh/lowlevellock.h (__lll_private_flag):
Define.
(lll_futex_timed_wait, lll_futex_wake): Use it.
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
--- libc/nptl/allocatestack.c.jj 2007-06-18 13:29:25.000000000 +0200
+++ libc/nptl/allocatestack.c 2007-07-24 11:43:58.000000000 +0200
@@ -951,7 +951,7 @@ __nptl_setxid (struct xid_command *cmdp)
int cur = cmdp->cntr;
while (cur != 0)
{
- lll_private_futex_wait (&cmdp->cntr, cur);
+ lll_futex_wait (&cmdp->cntr, cur, LLL_PRIVATE);
cur = cmdp->cntr;
}
@@ -1037,7 +1037,7 @@ __wait_lookup_done (void)
continue;
do
- lll_private_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT);
+ lll_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT, LLL_PRIVATE);
while (*gscope_flagp == THREAD_GSCOPE_FLAG_WAIT);
}
@@ -1059,7 +1059,7 @@ __wait_lookup_done (void)
continue;
do
- lll_private_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT);
+ lll_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT, LLL_PRIVATE);
while (*gscope_flagp == THREAD_GSCOPE_FLAG_WAIT);
}
--- libc/nptl/pthread_create.c.jj 2007-06-08 09:13:50.000000000 +0200
+++ libc/nptl/pthread_create.c 2007-07-24 11:43:19.000000000 +0200
@@ -385,7 +385,7 @@ start_thread (void *arg)
/* Some other thread might call any of the setXid functions and expect
us to reply. In this case wait until we did that. */
do
- lll_private_futex_wait (&pd->setxid_futex, 0);
+ lll_futex_wait (&pd->setxid_futex, 0, LLL_PRIVATE);
while (pd->cancelhandling & SETXID_BITMASK);
/* Reset the value so that the stack can be reused. */
--- libc/nptl/init.c.jj 2007-06-29 10:19:55.000000000 +0200
+++ libc/nptl/init.c 2007-07-24 11:43:34.000000000 +0200
@@ -216,7 +216,7 @@ sighandler_setxid (int sig, siginfo_t *s
__xidcmd->id[1], __xidcmd->id[2]);
if (atomic_decrement_val (&__xidcmd->cntr) == 0)
- lll_private_futex_wake (&__xidcmd->cntr, 1);
+ lll_futex_wake (&__xidcmd->cntr, 1, LLL_PRIVATE);
/* Reset the SETXID flag. */
struct pthread *self = THREAD_SELF;
@@ -225,7 +225,7 @@ sighandler_setxid (int sig, siginfo_t *s
/* And release the futex. */
self->setxid_futex = 1;
- lll_private_futex_wake (&self->setxid_futex, 1);
+ lll_futex_wake (&self->setxid_futex, 1, LLL_PRIVATE);
}
--- libc/nptl/sysdeps/alpha/tls.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/alpha/tls.h 2007-07-24 11:44:15.000000000 +0200
@@ -131,7 +131,7 @@ typedef struct
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/ia64/tls.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/ia64/tls.h 2007-07-24 11:44:30.000000000 +0200
@@ -173,7 +173,7 @@ register struct pthread *__thread_self _
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/i386/tls.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/i386/tls.h 2007-07-24 11:44:44.000000000 +0200
@@ -449,7 +449,7 @@ union user_desc_init
: "i" (offsetof (struct pthread, header.gscope_flag)), \
"0" (THREAD_GSCOPE_FLAG_UNUSED)); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/pthread/aio_misc.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/pthread/aio_misc.h 2007-07-24 11:45:46.000000000 +0200
@@ -30,7 +30,7 @@
#define AIO_MISC_NOTIFY(waitlist) \
do { \
if (*waitlist->counterp > 0 && --*waitlist->counterp == 0) \
- lll_private_futex_wake (waitlist->counterp, 1); \
+ lll_futex_wake (waitlist->counterp, 1, LLL_PRIVATE); \
} while (0)
#define AIO_MISC_WAIT(result, futex, timeout, cancel) \
@@ -49,8 +49,8 @@
int status; \
do \
{ \
- status = lll_private_futex_timed_wait (futexaddr, oldval, \
- timeout); \
+ status = lll_futex_timed_wait (futexaddr, oldval, timeout, \
+ LLL_PRIVATE); \
if (status != -EWOULDBLOCK) \
break; \
\
--- libc/nptl/sysdeps/pthread/gai_misc.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/pthread/gai_misc.h 2007-07-24 11:45:16.000000000 +0200
@@ -31,7 +31,7 @@
#define GAI_MISC_NOTIFY(waitlist) \
do { \
if (*waitlist->counterp > 0 && --*waitlist->counterp == 0) \
- lll_private_futex_wake (waitlist->counterp, 1); \
+ lll_futex_wake (waitlist->counterp, 1, LLL_PRIVATE); \
} while (0)
#define GAI_MISC_WAIT(result, futex, timeout, cancel) \
@@ -50,8 +50,8 @@
int status; \
do \
{ \
- status = lll_private_futex_timed_wait (futexaddr, oldval, \
- timeout); \
+ status = lll_futex_timed_wait (futexaddr, oldval, timeout, \
+ LLL_PRIVATE); \
if (status != -EWOULDBLOCK) \
break; \
\
--- libc/nptl/sysdeps/s390/tls.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/s390/tls.h 2007-07-24 11:46:01.000000000 +0200
@@ -183,7 +183,7 @@ typedef struct
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/powerpc/tls.h.jj 2007-06-08 09:13:51.000000000 +0200
+++ libc/nptl/sysdeps/powerpc/tls.h 2007-07-24 11:46:14.000000000 +0200
@@ -190,7 +190,7 @@ register void *__thread_register __asm__
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/x86_64/tls.h.jj 2007-06-08 09:13:53.000000000 +0200
+++ libc/nptl/sysdeps/x86_64/tls.h 2007-07-24 11:48:03.000000000 +0200
@@ -355,7 +355,7 @@ typedef struct
: "i" (offsetof (struct pthread, header.gscope_flag)), \
"0" (THREAD_GSCOPE_FLAG_UNUSED)); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/sparc/tls.h.jj 2007-06-08 09:13:52.000000000 +0200
+++ libc/nptl/sysdeps/sparc/tls.h 2007-07-24 11:48:17.000000000 +0200
@@ -151,7 +151,7 @@ register struct pthread *__thread_self _
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/sh/tls.h.jj 2007-07-07 22:09:51.000000000 +0200
+++ libc/nptl/sysdeps/sh/tls.h 2007-07-24 11:48:29.000000000 +0200
@@ -164,7 +164,7 @@ typedef struct
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_private_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/unix/sysv/linux/alpha/pthread_once.c.jj 2007-01-25 10:58:42.000000000 +0100
+++ libc/nptl/sysdeps/unix/sysv/linux/alpha/pthread_once.c 2007-07-24 12:40:15.000000000 +0200
@@ -1,4 +1,4 @@
-/* Copyright (C) 2003, 2004 Free Software Foundation, Inc.
+/* Copyright (C) 2003, 2004, 2007 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
@@ -28,7 +28,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
int
@@ -72,7 +72,7 @@ __pthread_once (pthread_once_t *once_con
break;
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, oldval);
+ lll_futex_wait (once_control, oldval, LLL_PRIVATE);
}
/* This thread is the first here. Do the initialization.
@@ -88,7 +88,7 @@ __pthread_once (pthread_once_t *once_con
atomic_increment (once_control);
/* Wake up all other threads. */
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
return 0;
}
--- libc/nptl/sysdeps/unix/sysv/linux/alpha/lowlevellock.h.jj 2007-06-04 08:42:06.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/alpha/lowlevellock.h 2007-07-24 12:51:21.000000000 +0200
@@ -36,34 +36,63 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
+
/* Initializer for compatibility lock. */
#define LLL_MUTEX_LOCK_INITIALIZER (0)
-#define lll_futex_wait(futexp, val) \
- ({ \
- INTERNAL_SYSCALL_DECL (__err); \
- long int __ret; \
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAIT, (val), 0); \
- INTERNAL_SYSCALL_ERROR_P (__ret, __err)? -__ret : __ret; \
- })
+#define lll_futex_wait(futexp, val, private) \
+ lll_futex_timed_wait (futexp, val, NULL, private)
-#define lll_futex_timed_wait(futexp, val, timespec) \
+#define lll_futex_timed_wait(futexp, val, timespec, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAIT, (val), (timespec)); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \
+ __lll_private_flag (FUTEX_WAIT, private), \
+ (val), (timespec)); \
INTERNAL_SYSCALL_ERROR_P (__ret, __err)? -__ret : __ret; \
})
-#define lll_futex_wake(futexp, nr) \
+#define lll_futex_wake(futexp, nr, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAKE, (nr), 0); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \
+ __lll_private_flag (FUTEX_WAKE, private), \
+ (nr), 0); \
INTERNAL_SYSCALL_ERROR_P (__ret, __err)? -__ret : __ret; \
})
@@ -72,7 +101,7 @@
{ \
int *__futexp = &(futexv); \
atomic_or (__futexp, FUTEX_OWNER_DIED); \
- lll_futex_wake (__futexp, 1); \
+ lll_futex_wake (__futexp, 1, LLL_SHARED); \
} \
while (0)
@@ -198,7 +227,7 @@ __lll_mutex_unlock (int *futex)
{
int val = atomic_exchange_rel (futex, 0);
if (__builtin_expect (val > 1, 0))
- lll_futex_wake (futex, 1);
+ lll_futex_wake (futex, 1, LLL_SHARED);
}
#define lll_mutex_unlock(futex) __lll_mutex_unlock(&(futex))
@@ -208,7 +237,7 @@ __lll_robust_mutex_unlock (int *futex, i
{
int val = atomic_exchange_rel (futex, 0);
if (__builtin_expect (val & mask, 0))
- lll_futex_wake (futex, 1);
+ lll_futex_wake (futex, 1, LLL_SHARED);
}
#define lll_robust_mutex_unlock(futex) \
__lll_robust_mutex_unlock(&(futex), FUTEX_WAITERS)
@@ -218,7 +247,7 @@ static inline void __attribute__ ((alway
__lll_mutex_unlock_force (int *futex)
{
(void) atomic_exchange_rel (futex, 0);
- lll_futex_wake (futex, 1);
+ lll_futex_wake (futex, 1, LLL_SHARED);
}
#define lll_mutex_unlock_force(futex) __lll_mutex_unlock_force(&(futex))
@@ -252,10 +281,10 @@ typedef int lll_lock_t;
thread ID while the clone is running and is reset to zero
afterwards. */
#define lll_wait_tid(tid) \
- do { \
- __typeof (tid) __tid; \
- while ((__tid = (tid)) != 0) \
- lll_futex_wait (&(tid), __tid); \
+ do { \
+ __typeof (tid) __tid; \
+ while ((__tid = (tid)) != 0) \
+ lll_futex_wait (&(tid), __tid, LLL_SHARED); \
} while (0)
extern int __lll_timedwait_tid (int *, const struct timespec *)
--- libc/nptl/sysdeps/unix/sysv/linux/ia64/pthread_once.c.jj 2007-01-25 10:58:42.000000000 +0100
+++ libc/nptl/sysdeps/unix/sysv/linux/ia64/pthread_once.c 2007-07-24 12:29:44.000000000 +0200
@@ -1,4 +1,4 @@
-/* Copyright (C) 2003, 2004 Free Software Foundation, Inc.
+/* Copyright (C) 2003, 2004, 2007 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Jakub Jelinek <jakub@redhat.com>, 2003.
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -65,7 +65,7 @@ __pthread_once (once_control, init_routi
if (((oldval ^ newval) & -4) == 0)
{
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, newval);
+ lll_futex_wait (once_control, newval, LLL_PRIVATE);
continue;
}
}
@@ -84,7 +84,7 @@ __pthread_once (once_control, init_routi
atomic_increment (once_control);
/* Wake up all other threads. */
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
break;
}
--- libc/nptl/sysdeps/unix/sysv/linux/ia64/lowlevellock.h.jj 2007-06-04 08:42:06.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/ia64/lowlevellock.h 2007-07-24 12:29:00.000000000 +0200
@@ -36,6 +36,39 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
+
/* Delay in spinlock loop. */
#define BUSY_WAIT_NOP asm ("hint @pause")
@@ -43,18 +76,22 @@
/* Initializer for compatibility lock. */
#define LLL_MUTEX_LOCK_INITIALIZER (0)
-#define lll_futex_wait(futex, val) lll_futex_timed_wait (futex, val, 0)
+#define lll_futex_wait(futex, val, private) \
+ lll_futex_timed_wait (futex, val, NULL, private)
-#define lll_futex_timed_wait(ftx, val, timespec) \
+#define lll_futex_timed_wait(ftx, val, timespec, private) \
({ \
- DO_INLINE_SYSCALL(futex, 4, (long) (ftx), FUTEX_WAIT, (int) (val), \
- (long) (timespec)); \
+ DO_INLINE_SYSCALL(futex, 4, (long) (ftx), \
+ __lll_private_flag (FUTEX_WAIT, private), \
+ (int) (val), (long) (timespec)); \
_r10 == -1 ? -_retval : _retval; \
})
-#define lll_futex_wake(ftx, nr) \
+#define lll_futex_wake(ftx, nr, private) \
({ \
- DO_INLINE_SYSCALL(futex, 3, (long) (ftx), FUTEX_WAKE, (int) (nr)); \
+ DO_INLINE_SYSCALL(futex, 3, (long) (ftx), \
+ __lll_private_flag (FUTEX_WAKE, private), \
+ (int) (nr)); \
_r10 == -1 ? -_retval : _retval; \
})
@@ -188,7 +225,7 @@ extern int __lll_robust_timedlock_wait (
int __val = atomic_exchange_rel (__futex, 0); \
\
if (__builtin_expect (__val > 1, 0)) \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, LLL_SHARED); \
}))
#define lll_mutex_unlock(futex) \
__lll_mutex_unlock(&(futex))
@@ -200,7 +237,7 @@ extern int __lll_robust_timedlock_wait (
int __val = atomic_exchange_rel (__futex, 0); \
\
if (__builtin_expect (__val & FUTEX_WAITERS, 0)) \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, LLL_SHARED); \
}))
#define lll_robust_mutex_unlock(futex) \
__lll_robust_mutex_unlock(&(futex))
@@ -210,7 +247,7 @@ extern int __lll_robust_timedlock_wait (
((void) ({ \
int *__futex = (futex); \
(void) atomic_exchange_rel (__futex, 0); \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, LLL_SHARED); \
}))
#define lll_mutex_unlock_force(futex) \
__lll_mutex_unlock_force(&(futex))
@@ -241,12 +278,12 @@ typedef int lll_lock_t;
thread ID while the clone is running and is reset to zero
afterwards. */
#define lll_wait_tid(tid) \
- do \
- { \
- __typeof (tid) __tid; \
- while ((__tid = (tid)) != 0) \
- lll_futex_wait (&(tid), __tid); \
- } \
+ do \
+ { \
+ __typeof (tid) __tid; \
+ while ((__tid = (tid)) != 0) \
+ lll_futex_wait (&(tid), __tid, LLL_SHARED); \
+ } \
while (0)
extern int __lll_timedwait_tid (int *, const struct timespec *)
--- libc/nptl/sysdeps/unix/sysv/linux/unregister-atfork.c.jj 2007-06-08 09:13:52.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/unregister-atfork.c 2007-07-24 11:57:36.000000000 +0200
@@ -104,7 +104,7 @@ __unregister_atfork (dso_handle)
atomic_decrement (&deleted->handler->refcntr);
unsigned int val;
while ((val = deleted->handler->refcntr) != 0)
- lll_private_futex_wait (&deleted->handler->refcntr, val);
+ lll_futex_wait (&deleted->handler->refcntr, val, LLL_PRIVATE);
deleted = deleted->next;
}
--- libc/nptl/sysdeps/unix/sysv/linux/rtld-lowlevel.h.jj 2007-06-29 10:19:56.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/rtld-lowlevel.h 2007-07-24 11:57:56.000000000 +0200
@@ -33,12 +33,12 @@
int val = word; \
if (val == 0) \
break; \
- lll_private_futex_wait (&(word), val); \
+ lll_futex_wait (&(word), val, LLL_PRIVATE); \
} \
} while (0)
#define __rtld_notify(word) \
- lll_private_futex_wake (&(word), 1)
+ lll_futex_wake (&(word), 1, LLL_PRIVATE)
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/fork.c.jj 2007-06-08 09:13:52.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/fork.c 2007-07-24 12:00:37.000000000 +0200
@@ -203,7 +203,7 @@ __libc_fork (void)
if (atomic_decrement_and_test (&allp->handler->refcntr)
&& allp->handler->need_signal)
- lll_private_futex_wake (allp->handler->refcntr, 1);
+ lll_futex_wake (allp->handler->refcntr, 1, LLL_PRIVATE);
allp = allp->next;
}
--- libc/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h.jj 2007-06-08 09:13:52.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h 2007-07-24 13:54:58.000000000 +0200
@@ -50,6 +50,33 @@
#define LLL_SHARED FUTEX_PRIVATE_FLAG
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ({ unsigned int __fl = ((private) ^ FUTEX_PRIVATE_FLAG); \
+ asm ("andl %%fs:%P1, %0" : "+r" (__fl) \
+ : "i" (offsetof (struct pthread, header.private_futex))); \
+ __fl | (fl); }))
+# endif
+#endif
+
+
/* Initializer for compatibility lock. */
#define LLL_MUTEX_LOCK_INITIALIZER (0)
#define LLL_MUTEX_LOCK_INITIALIZER_LOCKED (1)
@@ -167,8 +194,8 @@ LLL_STUB_UNWIND_INFO_END
LLL_EBX_LOAD \
: "=a" (__status) \
: "0" (SYS_futex), LLL_EBX_REG (futex), "S" (timeout), \
- "c" (FUTEX_WAIT), "d" (_val), \
- "i" (offsetof (tcbhead_t, sysinfo)) \
+ "c" (__lll_private_flag (FUTEX_WAIT, private)), \
+ "d" (_val), "i" (offsetof (tcbhead_t, sysinfo)) \
: "memory"); \
__status; \
})
@@ -183,81 +210,11 @@ LLL_STUB_UNWIND_INFO_END
LLL_EBX_LOAD \
: "=a" (__ignore) \
: "0" (SYS_futex), LLL_EBX_REG (futex), \
- "c" (FUTEX_WAKE), "d" (_nr), \
- "i" (0) /* phony, to align next arg's number */, \
- "i" (offsetof (tcbhead_t, sysinfo))); \
- } while (0)
-
-
-#define lll_private_futex_wait(futex, val) \
- lll_private_futex_timed_wait (futex, val, NULL)
-
-
-#ifdef __ASSUME_PRIVATE_FUTEX
-# define lll_private_futex_timed_wait(futex, val, timeout) \
- ({ \
- int __status; \
- register __typeof (val) _val asm ("edx") = (val); \
- __asm __volatile (LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__status) \
- : "0" (SYS_futex), LLL_EBX_REG (futex), "S" (timeout), \
- "c" (FUTEX_WAIT | FUTEX_PRIVATE_FLAG)), "d" (_val), \
- "i" (offsetof (tcbhead_t, sysinfo)) \
- : "memory"); \
- __status; \
- })
-
-
-# define lll_private_futex_wake(futex, nr) \
- do { \
- int __ignore; \
- register __typeof (nr) _nr asm ("edx") = (nr); \
- __asm __volatile (LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__ignore) \
- : "0" (SYS_futex), LLL_EBX_REG (futex), \
- "c" (FUTEX_WAKE | FUTEX_PRIVATE_FLAG), "d" (_nr), \
+ "c" (__lll_private_flag (FUTEX_WAKE, private)), \
+ "d" (_nr), \
"i" (0) /* phony, to align next arg's number */, \
"i" (offsetof (tcbhead_t, sysinfo))); \
} while (0)
-#else
-# define lll_private_futex_timed_wait(futex, val, timeout) \
- ({ \
- int __status; \
- int __ignore; \
- register __typeof (val) _val asm ("edx") = (val); \
- __asm __volatile ("movl %%gs:%P7, %%ecx\n\t" \
- LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__status), "=c" (__ignore) \
- : LLL_EBX_REG (futex), "0" (SYS_futex), "S" (timeout), \
- "d" (_val), "i" (offsetof (tcbhead_t, sysinfo)), \
- "i" (PRIVATE_FUTEX) \
- : "memory"); \
- __status; \
- })
-
-
-# define lll_private_futex_wake(futex, nr) \
- do { \
- int __ignore; \
- int __ignore2; \
- register __typeof (nr) _nr asm ("edx") = (nr); \
- __asm __volatile ("orl %%gs:%P7, %%ecx\n\t" \
- LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__ignore), "=c" (__ignore2) \
- : LLL_EBX_REG (futex), "0" (SYS_futex), \
- "1" (FUTEX_WAKE), "d" (_nr), \
- "i" (offsetof (tcbhead_t, sysinfo)), \
- "i" (PRIVATE_FUTEX)); \
- } while (0)
-#endif
/* Does not preserve %eax and %ecx. */
--- libc/nptl/sysdeps/unix/sysv/linux/s390/pthread_once.c.jj 2007-01-25 10:58:42.000000000 +0100
+++ libc/nptl/sysdeps/unix/sysv/linux/s390/pthread_once.c 2007-07-24 13:01:28.000000000 +0200
@@ -1,4 +1,4 @@
-/* Copyright (C) 2003 Free Software Foundation, Inc.
+/* Copyright (C) 2003, 2007 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Martin Schwidefsky <schwidefsky@de.ibm.com>, 2003.
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -76,7 +76,7 @@ __pthread_once (once_control, init_routi
if (((oldval ^ newval) & -4) == 0)
{
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, newval);
+ lll_futex_wait (once_control, newval, LLL_PRIVATE);
continue;
}
}
@@ -101,7 +101,7 @@ __pthread_once (once_control, init_routi
: "m" (*once_control) : "cc" );
/* Wake up all other threads. */
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
break;
}
--- libc/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h.jj 2007-06-04 08:42:06.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h 2007-07-24 13:01:02.000000000 +0200
@@ -35,31 +35,50 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
/* Initializer for compatibility lock. */
#define LLL_MUTEX_LOCK_INITIALIZER (0)
-#define lll_futex_wait(futex, val) \
- ({ \
- register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAIT; \
- register unsigned long int __r4 asm ("4") = (unsigned long int) (val); \
- register unsigned long int __r5 asm ("5") = 0ul; \
- register unsigned long __result asm ("2"); \
- \
- __asm __volatile ("svc %b1" \
- : "=d" (__result) \
- : "i" (SYS_futex), "0" (__r2), "d" (__r3), \
- "d" (__r4), "d" (__r5) \
- : "cc", "memory" ); \
- __result; \
- })
-
+#define lll_futex_wait(futex, val, private) \
+ lll_futex_timed_wait (futex, val, NULL, private)
-#define lll_futex_timed_wait(futex, val, timespec) \
+#define lll_futex_timed_wait(futex, val, timespec, private) \
({ \
register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAIT; \
+ register unsigned long int __r3 asm ("3") \
+ = __lll_private_flag (FUTEX_WAIT, private); \
register unsigned long int __r4 asm ("4") = (unsigned long int) (val); \
register unsigned long int __r5 asm ("5") = (unsigned long int)(timespec);\
register unsigned long int __result asm ("2"); \
@@ -73,10 +92,11 @@
})
-#define lll_futex_wake(futex, nr) \
+#define lll_futex_wake(futex, nr, private) \
({ \
register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAKE; \
+ register unsigned long int __r3 asm ("3") \
+ __lll_private_flag (FUTEX_WAKE, private); \
register unsigned long int __r4 asm ("4") = (unsigned long int) (nr); \
register unsigned long int __result asm ("2"); \
\
@@ -94,7 +114,7 @@
int *__futexp = &(futexv); \
\
atomic_or (__futexp, FUTEX_OWNER_DIED); \
- lll_futex_wake (__futexp, 1); \
+ lll_futex_wake (__futexp, 1, LLL_SHARED); \
} \
while (0)
@@ -271,7 +291,7 @@ __lll_mutex_unlock (int *futex)
lll_compare_and_swap (futex, oldval, newval, "slr %2,%2");
if (oldval > 1)
- lll_futex_wake (futex, 1);
+ lll_futex_wake (futex, 1, LLL_SHARED);
}
#define lll_mutex_unlock(futex) \
__lll_mutex_unlock(&(futex))
@@ -286,7 +306,7 @@ __lll_robust_mutex_unlock (int *futex, i
lll_compare_and_swap (futex, oldval, newval, "slr %2,%2");
if (oldval & mask)
- lll_futex_wake (futex, 1);
+ lll_futex_wake (futex, 1, LLL_SHARED);
}
#define lll_robust_mutex_unlock(futex) \
__lll_robust_mutex_unlock(&(futex), FUTEX_WAITERS)
@@ -297,7 +317,7 @@ __attribute__ ((always_inline))
__lll_mutex_unlock_force (int *futex)
{
*futex = 0;
- lll_futex_wake (futex, 1);
+ lll_futex_wake (futex, 1, LLL_SHARED);
}
#define lll_mutex_unlock_force(futex) \
__lll_mutex_unlock_force(&(futex))
@@ -338,7 +358,7 @@ __lll_wait_tid (int *ptid)
int tid;
while ((tid = *ptid) != 0)
- lll_futex_wait (ptid, tid);
+ lll_futex_wait (ptid, tid, LLL_SHARED);
}
#define lll_wait_tid(tid) __lll_wait_tid(&(tid))
--- libc/nptl/sysdeps/unix/sysv/linux/powerpc/pthread_once.c.jj 2007-07-24 10:50:54.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/powerpc/pthread_once.c 2007-07-24 11:58:20.000000000 +0200
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_private_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -74,7 +74,7 @@ __pthread_once (pthread_once_t *once_con
break;
/* Same generation, some other thread was faster. Wait. */
- lll_private_futex_wait (once_control, oldval);
+ lll_futex_wait (once_control, oldval, LLL_PRIVATE);
}
@@ -92,7 +92,7 @@ __pthread_once (pthread_once_t *once_con
atomic_increment (once_control);
/* Wake up all other threads. */
- lll_private_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
return 0;
}
--- libc/nptl/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h.jj 2007-07-24 10:50:54.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h 2007-07-24 11:59:00.000000000 +0200
@@ -134,15 +134,6 @@
})
-#define lll_private_futex_wait(futexp, val) \
- lll_futex_timed_wait (futexp, val, NULL, LLL_PRIVATE)
-
-#define lll_private_futex_timed_wait(futexp, val, timeout) \
- lll_futex_timed_wait (futexp, val, timeout, LLL_PRIVATE)
-
-#define lll_private_futex_wake(futexp, val) \
- lll_futex_wake (futexp, val, LLL_PRIVATE)
-
#ifdef UP
# define __lll_acq_instr ""
# define __lll_rel_instr ""
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h.jj 2007-07-24 10:50:55.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h 2007-07-24 13:54:41.000000000 +0200
@@ -71,8 +71,8 @@
: (fl)) \
: ({ unsigned int __fl = ((private) ^ FUTEX_PRIVATE_FLAG); \
asm ("andl %%fs:%P1, %0" : "+r" (__fl) \
- : "i" offsetof (struct pthread, header.private_futex)); \
- __fl | (fl); })
+ : "i" (offsetof (struct pthread, header.private_futex))); \
+ __fl | (fl); }))
# endif
#endif
@@ -215,15 +215,6 @@ LLL_STUB_UNWIND_INFO_END
} while (0)
-#define lll_private_futex_wait(futex, val) \
- lll_futex_timed_wait (futex, val, NULL, LLL_PRIVATE)
-
-#define lll_private_futex_timed_wait(futex, val, timeout) \
- lll_futex_timed_wait (futex, val, timeout, LLL_PRIVATE)
-
-#define lll_private_futex_wake(futex, nr) \
- lll_futex_wake (futex, nr, LLL_PRIVATE)
-
/* Does not preserve %eax and %ecx. */
extern int __lll_mutex_lock_wait (int *__futex, int __val) attribute_hidden;
--- libc/nptl/sysdeps/unix/sysv/linux/sparc/pthread_once.c.jj 2007-06-08 09:13:53.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/sparc/pthread_once.c 2007-07-24 13:15:55.000000000 +0200
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_private_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -65,7 +65,7 @@ __pthread_once (once_control, init_routi
if (((oldval ^ newval) & -4) == 0)
{
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, newval);
+ lll_futex_wait (once_control, newval, LLL_PRIVATE);
continue;
}
}
@@ -84,7 +84,7 @@ __pthread_once (once_control, init_routi
atomic_increment (once_control);
/* Wake up all other threads. */
- lll_private_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
break;
}
--- libc/nptl/sysdeps/unix/sysv/linux/sparc/lowlevellock.h.jj 2007-06-04 08:42:06.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/sparc/lowlevellock.h 2007-07-24 13:15:28.000000000 +0200
@@ -35,37 +35,66 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
+
/* Initializer for compatibility lock. */
#define LLL_MUTEX_LOCK_INITIALIZER (0)
-#define lll_futex_wait(futexp, val) \
- ({ \
- INTERNAL_SYSCALL_DECL (__err); \
- long int __ret; \
- \
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAIT, (val), 0); \
- __ret; \
- })
+#define lll_futex_wait(futexp, val, private) \
+ lll_futex_timed_wait (futexp, val, NULL, private)
-#define lll_futex_timed_wait(futexp, val, timespec) \
+#define lll_futex_timed_wait(futexp, val, timespec, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
\
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAIT, (val), (timespec)); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \
+ __lll_private_flag (FUTEX_WAIT, private), \
+ (val), (timespec)); \
__ret; \
})
-#define lll_futex_wake(futexp, nr) \
+#define lll_futex_wake(futexp, nr, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
\
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAKE, (nr), 0); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \
+ __lll_private_flag (FUTEX_WAKE, private), \
+ (nr), 0); \
__ret; \
})
@@ -86,7 +115,7 @@
{ \
int *__futexp = &(futexv); \
atomic_or (__futexp, FUTEX_OWNER_DIED); \
- lll_futex_wake (__futexp, 1); \
+ lll_futex_wake (__futexp, 1, LLL_SHARED); \
} \
while (0)
@@ -212,7 +241,7 @@ __lll_robust_mutex_timedlock (int *futex
int *__futex = &(lock); \
int __val = atomic_exchange_24_rel (__futex, 0); \
if (__builtin_expect (__val > 1, 0)) \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, LLL_SHARED); \
}))
#define lll_robust_mutex_unlock(lock) \
@@ -220,14 +249,14 @@ __lll_robust_mutex_timedlock (int *futex
int *__futex = &(lock); \
int __val = atomic_exchange_rel (__futex, 0); \
if (__builtin_expect (__val & FUTEX_WAITERS, 0)) \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, LLL_SHARED); \
}))
#define lll_mutex_unlock_force(lock) \
((void) ({ \
int *__futex = &(lock); \
(void) atomic_exchange_24_rel (__futex, 0); \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, LLL_SHARED); \
}))
#define lll_mutex_islocked(futex) \
@@ -255,12 +284,12 @@ typedef int lll_lock_t;
thread ID while the clone is running and is reset to zero
afterwards. */
#define lll_wait_tid(tid) \
- do \
- { \
- __typeof (tid) __tid; \
- while ((__tid = (tid)) != 0) \
- lll_futex_wait (&(tid), __tid); \
- } \
+ do \
+ { \
+ __typeof (tid) __tid; \
+ while ((__tid = (tid)) != 0) \
+ lll_futex_wait (&(tid), __tid, LLL_SHARED); \
+ } \
while (0)
extern int __lll_timedwait_tid (int *, const struct timespec *)
--- libc/nptl/sysdeps/unix/sysv/linux/sh/lowlevellock.h.jj 2007-06-18 13:29:27.000000000 +0200
+++ libc/nptl/sysdeps/unix/sysv/linux/sh/lowlevellock.h 2007-07-24 12:35:09.000000000 +0200
@@ -40,6 +40,31 @@
#define LLL_SHARED FUTEX_PRIVATE_FLAG
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
+
+
/* Initializer for compatibility lock. */
#define LLL_MUTEX_LOCK_INITIALIZER (0)
#define LLL_MUTEX_LOCK_INITIALIZER_LOCKED (1)
@@ -278,7 +303,7 @@ extern int __lll_mutex_unlock_wake (int
1: mov r1,r15"\
: "=&r" (__ignore) : "r" (__futex), "r" (FUTEX_OWNER_DIED) \
: "r0", "r1", "memory"); \
- lll_futex_wake (__futex, 1, 0); })
+ lll_futex_wake (__futex, 1, LLL_SHARED); })
#define lll_mutex_islocked(futex) \
(futex != 0)
@@ -312,7 +337,8 @@ typedef int lll_lock_t;
int __status; \
register unsigned long __r3 asm ("r3") = SYS_futex; \
register unsigned long __r4 asm ("r4") = (unsigned long) (futex); \
- register unsigned long __r5 asm ("r5") = FUTEX_WAIT; \
+ register unsigned long __r5 asm ("r5") \
+ = __lll_private_flag (FUTEX_WAIT, private); \
register unsigned long __r6 asm ("r6") = (unsigned long) (val); \
register unsigned long __r7 asm ("r7") = (timeout); \
__asm __volatile (SYSCALL_WITH_INST_PAD \
@@ -329,7 +355,8 @@ typedef int lll_lock_t;
int __ignore; \
register unsigned long __r3 asm ("r3") = SYS_futex; \
register unsigned long __r4 asm ("r4") = (unsigned long) (futex); \
- register unsigned long __r5 asm ("r5") = FUTEX_WAKE; \
+ register unsigned long __r5 asm ("r5") \
+ = __lll_private_flag (FUTEX_WAKE, private); \
register unsigned long __r6 asm ("r6") = (unsigned long) (nr); \
register unsigned long __r7 asm ("r7") = 0; \
__asm __volatile (SYSCALL_WITH_INST_PAD \
@@ -340,81 +367,6 @@ typedef int lll_lock_t;
} while (0)
-#define lll_private_futex_wait(futex, val) \
- lll_private_futex_timed_wait (futex, val, NULL)
-
-
-#ifdef __ASSUME_PRIVATE_FUTEX
-# define lll_private_futex_timed_wait(futex, val, timeout) \
- ({ \
- int __status; \
- register unsigned long __r3 asm ("r3") = SYS_futex; \
- register unsigned long __r4 asm ("r4") = (unsigned long) (futex); \
- register unsigned long __r5 asm ("r5") = FUTEX_WAIT | FUTEX_PRIVATE_FLAG; \
- register unsigned long __r6 asm ("r6") = (unsigned long) (val); \
- register unsigned long __r7 asm ("r7") = (timeout); \
- __asm __volatile (SYSCALL_WITH_INST_PAD \
- : "=z" (__status) \
- : "r" (__r3), "r" (__r4), "r" (__r5), \
- "r" (__r6), "r" (__r7) \
- : "memory", "t"); \
- __status; \
- })
-
-
-# define lll_private_futex_wake(futex, nr) \
- do { \
- int __ignore; \
- register unsigned long __r3 asm ("r3") = SYS_futex; \
- register unsigned long __r4 asm ("r4") = (unsigned long) (futex); \
- register unsigned long __r5 asm ("r5") = FUTEX_WAKE | FUTEX_PRIVATE_FLAG; \
- register unsigned long __r6 asm ("r6") = (unsigned long) (nr); \
- register unsigned long __r7 asm ("r7") = 0; \
- __asm __volatile (SYSCALL_WITH_INST_PAD \
- : "=z" (__ignore) \
- : "r" (__r3), "r" (__r4), "r" (__r5), \
- "r" (__r6), "r" (__r7) \
- : "memory", "t"); \
- } while (0)
-
-
-#else
-# define lll_private_futex_timed_wait(futex, val, timeout) \
- ({ \
- int __status; \
- register unsigned long __r3 asm ("r3") = SYS_futex; \
- register unsigned long __r4 asm ("r4") = (unsigned long) (futex); \
- register unsigned long __r5 asm ("r5"); \
- register unsigned long __r6 asm ("r6") = (unsigned long) (val); \
- register unsigned long __r7 asm ("r7") = (timeout); \
- __r5 = THREAD_GETMEM (THREAD_SELF, header.private_futex); \
- __asm __volatile (SYSCALL_WITH_INST_PAD \
- : "=z" (__status) \
- : "r" (__r3), "r" (__r4), "r" (__r5), \
- "r" (__r6), "r" (__r7) \
- : "memory", "t"); \
- __status; \
- })
-
-
-# define lll_private_futex_wake(futex, nr) \
- do { \
- int __ignore; \
- register unsigned long __r3 asm ("r3") = SYS_futex; \
- register unsigned long __r4 asm ("r4") = (unsigned long) (futex); \
- register unsigned long __r5 asm ("r5") = FUTEX_WAKE; \
- register unsigned long __r6 asm ("r6") = (unsigned long) (nr); \
- register unsigned long __r7 asm ("r7") = 0; \
- __r5 |= THREAD_GETMEM (THREAD_SELF, header.private_futex); \
- __asm __volatile (SYSCALL_WITH_INST_PAD \
- : "=z" (__ignore) \
- : "r" (__r3), "r" (__r4), "r" (__r5), \
- "r" (__r6), "r" (__r7) \
- : "memory", "t"); \
- } while (0)
-#endif
-
-
/* The states of a lock are:
0 - untaken
1 - taken by one user
@@ -438,7 +390,7 @@ extern int __lll_wait_tid (int *tid) att
do { \
__typeof (tid) *__tid = &(tid); \
while (*__tid != 0) \
- lll_futex_wait (__tid, *__tid, 0); \
+ lll_futex_wait (__tid, *__tid, LLL_SHARED); \
} while (0)
extern int __lll_timedwait_tid (int *tid, const struct timespec *abstime)
Jakub
reply other threads:[~2007-07-24 15:59 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070724160250.GG4603@sunsite.mff.cuni.cz \
--to=jakub@redhat.com \
--cc=drepper@redhat.com \
--cc=libc-hacker@sources.redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).