* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
@ 2020-05-03 10:56 ` slyfox at inbox dot ru
2020-05-04 13:14 ` nsz at gcc dot gnu.org
` (38 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: slyfox at inbox dot ru @ 2020-05-03 10:56 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Sergei Trofimovich <slyfox at inbox dot ru> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |slyfox at inbox dot ru,
| |toolchain at gentoo dot org
--- Comment #18 from Sergei Trofimovich <slyfox at inbox dot ru> ---
In https://bugs.gentoo.org/719674#c12 gentoo sees nptl/tst-stack4 crashes
somewhat reliably on arm64:
# while :; do date; env
GCONV_PATH=/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/iconvdata
LOCPATH=/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/localedata
LC_ALL=C
/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/elf/ld-linux-aarch64.so.1
--library-path
/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/math:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/elf:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/dlfcn:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/nss:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/nis:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/rt:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/resolv:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/mathvec:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/support:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/crypt:/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/nptl::/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl//dlfcn
/var/tmp/portage/sys-libs/glibc-2.30-r8/work/build-arm64-aarch64-unknown-linux-gnu-nptl/nptl/tst-stack4;
done
Sun 03 May 2020 10:42:08 AM UTC
Sun 03 May 2020 10:42:21 AM UTC
Sun 03 May 2020 10:42:34 AM UTC
Didn't expect signal from child: got `Segmentation fault'
...
Sun 03 May 2020 10:42:56 AM UTC
malloc(): invalid size (unsorted)
Didn't expect signal from child: got `Aborted'
..
Sun 03 May 2020 10:46:21 AM UTC
free(): corrupted unsorted chunks
Didn't expect signal from child: got `Aborted'
...
Sun 03 May 2020 10:46:55 AM UTC
Didn't expect signal from child: got `Segmentation fault'
Sun 03 May 2020 10:47:04 AM UTC
double free or corruption (!prev)
Didn't expect signal from child: got `Aborted'
...
Sun 03 May 2020 10:50:54 AM UTC
free(): invalid pointer
Didn't expect signal from child: got `Aborted'
...
Sun 03 May 2020 10:52:12 AM UTC
tst-stack4: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av)
&& old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse
(old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
Didn't expect signal from child: got `Aborted'
Does it look like the same issue described here?
# lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 96
On-line CPU(s) list: 0-95
Thread(s) per core: 1
Core(s) per socket: 48
Socket(s): 2
Vendor ID: Cavium
Model: 1
Model name: ThunderX 88XX
Stepping: 0x1
BogoMIPS: 200.00
L1d cache: 32K
L1i cache: 78K
L2 cache: 16384K
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32
# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-unknown-linux-gnu/9.3.0/lto-wrapper
Target: aarch64-unknown-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-9.3.0/work/gcc-9.3.0/configure
--host=aarch64-unknown-linux-gnu --build=aarch64-unknown-linux-gnu
--prefix=/usr --bindir=/usr/aarch64-unknown-linux-gnu/gcc-bin/9.3.0
--includedir=/usr/lib/gcc/aarch64-unknown-linux-gnu/9.3.0/include
--datadir=/usr/share/gcc-data/aarch64-unknown-linux-gnu/9.3.0
--mandir=/usr/share/gcc-data/aarch64-unknown-linux-gnu/9.3.0/man
--infodir=/usr/share/gcc-data/aarch64-unknown-linux-gnu/9.3.0/info
--with-gxx-include-dir=/usr/lib/gcc/aarch64-unknown-linux-gnu/9.3.0/include/g++-v9
--with-python-dir=/share/gcc-data/aarch64-unknown-linux-gnu/9.3.0/python
--enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt
--disable-werror --with-system-zlib --enable-nls --without-included-gettext
--enable-checking=release --with-bugurl=https://bugs.gentoo.org/
--with-pkgversion='Gentoo 9.3.0 p2' --disable-esp --enable-libstdcxx-time
--enable-shared --enable-threads=posix --enable-__cxa_atexit
--enable-clocale=gnu --disable-multilib --disable-altivec --disable-fixed-point
--enable-libgomp --disable-libmudflap --disable-libssp --disable-libada
--disable-systemtap --enable-vtable-verify --enable-lto --without-isl
--enable-default-pie --enable-default-ssp
Thread model: posix
gcc version 9.3.0 (Gentoo 9.3.0 p2)
# uname -r
4.9.0-4-arm64
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
2020-05-03 10:56 ` [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen slyfox at inbox dot ru
@ 2020-05-04 13:14 ` nsz at gcc dot gnu.org
2020-05-04 18:25 ` slyfox at inbox dot ru
` (37 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2020-05-04 13:14 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Szabolcs Nagy <nsz at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |nsz at gcc dot gnu.org
--- Comment #19 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
(In reply to Sergei Trofimovich from comment #18)
> ...
> Sun 03 May 2020 10:46:55 AM UTC
> Didn't expect signal from child: got `Segmentation fault'
> Sun 03 May 2020 10:47:04 AM UTC
> double free or corruption (!prev)
> Didn't expect signal from child: got `Aborted'
> ...
> Sun 03 May 2020 10:50:54 AM UTC
> free(): invalid pointer
> Didn't expect signal from child: got `Aborted'
> ...
> Sun 03 May 2020 10:52:12 AM UTC
> tst-stack4: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top
> (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE &&
> prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)'
> failed.
> Didn't expect signal from child: got `Aborted'
>
> Does it look like the same issue described here?
it can be related, hard to tell.
(your failures are consistently heap corruptions
detected in malloc/free, instead of dynamic tls
related state corruption)
if you can rebuild glibc try the patches from
comment 7 if they don't help then your issue
is different. (if the issue disappears we don't
know if the new barriers just masked your issue
or fixed them).
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
2020-05-03 10:56 ` [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen slyfox at inbox dot ru
2020-05-04 13:14 ` nsz at gcc dot gnu.org
@ 2020-05-04 18:25 ` slyfox at inbox dot ru
2020-05-27 17:17 ` sourceware.org@the-compiler.org
` (36 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: slyfox at inbox dot ru @ 2020-05-04 18:25 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #20 from Sergei Trofimovich <slyfox at inbox dot ru> ---
(In reply to Szabolcs Nagy from comment #19)
> (In reply to Sergei Trofimovich from comment #18)
> > ...
> > Sun 03 May 2020 10:46:55 AM UTC
> > Didn't expect signal from child: got `Segmentation fault'
> > Sun 03 May 2020 10:47:04 AM UTC
> > double free or corruption (!prev)
> > Didn't expect signal from child: got `Aborted'
> > ...
> > Sun 03 May 2020 10:50:54 AM UTC
> > free(): invalid pointer
> > Didn't expect signal from child: got `Aborted'
> > ...
> > Sun 03 May 2020 10:52:12 AM UTC
> > tst-stack4: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top
> > (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE &&
> > prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)'
> > failed.
> > Didn't expect signal from child: got `Aborted'
> >
> > Does it look like the same issue described here?
>
> it can be related, hard to tell.
> (your failures are consistently heap corruptions
> detected in malloc/free, instead of dynamic tls
> related state corruption)
>
> if you can rebuild glibc try the patches from
> comment 7 if they don't help then your issue
> is different. (if the issue disappears we don't
> know if the new barriers just masked your issue
> or fixed them).
Tried patches from #comment7 on glibc-2.30. No failures after 100 test runs.
Usually fails after 3-4 runs.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (2 preceding siblings ...)
2020-05-04 18:25 ` slyfox at inbox dot ru
@ 2020-05-27 17:17 ` sourceware.org@the-compiler.org
2020-05-28 13:15 ` sourceware.org at devrx dot org
` (35 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: sourceware.org@the-compiler.org @ 2020-05-27 17:17 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Florian Bruhin <sourceware.org@the-compiler.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sourceware.org@the-compiler
| |.org
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (3 preceding siblings ...)
2020-05-27 17:17 ` sourceware.org@the-compiler.org
@ 2020-05-28 13:15 ` sourceware.org at devrx dot org
2020-07-08 20:15 ` sjkingo88 at gmail dot com
` (34 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: sourceware.org at devrx dot org @ 2020-05-28 13:15 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
sourceware.org at devrx dot org <sourceware.org at devrx dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sourceware.org at devrx dot org
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (4 preceding siblings ...)
2020-05-28 13:15 ` sourceware.org at devrx dot org
@ 2020-07-08 20:15 ` sjkingo88 at gmail dot com
2020-09-09 22:12 ` jg at jguk dot org
` (33 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: sjkingo88 at gmail dot com @ 2020-07-08 20:15 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Sam Kingston <sjkingo88 at gmail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sjkingo88 at gmail dot com
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (5 preceding siblings ...)
2020-07-08 20:15 ` sjkingo88 at gmail dot com
@ 2020-09-09 22:12 ` jg at jguk dot org
2020-09-28 15:27 ` jg at jguk dot org
` (32 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: jg at jguk dot org @ 2020-09-09 22:12 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Jonny Grant <jg at jguk dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jg at jguk dot org
--- Comment #21 from Jonny Grant <jg at jguk dot org> ---
running latest Ubuntu 20.04.1 LTS
$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9) 2.31
Just got this when launching google-chrome from the command line
Inconsistency detected by ld.so: ../elf/dl-tls.c: 481: _dl_allocate_tls_init:
Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!
Command exited with non-zero status 127
Could that assert be updated to give more information if you have another
assert_str() macro that could be used.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (6 preceding siblings ...)
2020-09-09 22:12 ` jg at jguk dot org
@ 2020-09-28 15:27 ` jg at jguk dot org
2020-10-09 10:51 ` jg at jguk dot org
` (31 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: jg at jguk dot org @ 2020-09-28 15:27 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #22 from Jonny Grant <jg at jguk dot org> ---
It's a bit surprising there is an assert() in a release build in Ubuntu.
Usually assert() would be for a debug build.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (7 preceding siblings ...)
2020-09-28 15:27 ` jg at jguk dot org
@ 2020-10-09 10:51 ` jg at jguk dot org
2020-10-09 20:21 ` carlos at redhat dot com
` (30 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: jg at jguk dot org @ 2020-10-09 10:51 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #23 from Jonny Grant <jg at jguk dot org> ---
On my computer any C program compiled with assert(0) dumps a core file, but
this glibc issue assert does not dump a core file. Is there an issue with this
assert macro in glibc? The message output on the terminal is different from the
standard macro
$ ./a
a: a.c:6: main: Assertion `0' failed.
Aborted (core dumped)
$ cat a.c
// gcc -Wall -o a a.c
#include <assert.h>
int main()
{
assert(0);
}
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (8 preceding siblings ...)
2020-10-09 10:51 ` jg at jguk dot org
@ 2020-10-09 20:21 ` carlos at redhat dot com
2020-10-12 1:29 ` i.palachev at samsung dot com
` (29 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: carlos at redhat dot com @ 2020-10-09 20:21 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #24 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Jonny Grant from comment #23)
> On my computer any C program compiled with assert(0) dumps a core file, but
> this glibc issue assert does not dump a core file. Is there an issue with
> this assert macro in glibc? The message output on the terminal is different
> from the standard macro
Please ask these questions on libc-help@sourceware.org where developers can
help you with any issues you have trying to put together a reproducer.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (9 preceding siblings ...)
2020-10-09 20:21 ` carlos at redhat dot com
@ 2020-10-12 1:29 ` i.palachev at samsung dot com
2020-10-23 11:47 ` sebastien at debian dot org
` (28 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: i.palachev at samsung dot com @ 2020-10-12 1:29 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Ilya Palachev <i.palachev at samsung dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC|i.palachev at samsung dot com |
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (10 preceding siblings ...)
2020-10-12 1:29 ` i.palachev at samsung dot com
@ 2020-10-23 11:47 ` sebastien at debian dot org
2020-10-29 3:12 ` lvying.system.thoughts at gmail dot com
` (27 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: sebastien at debian dot org @ 2020-10-23 11:47 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Sébastien Villemot <sebastien at debian dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sebastien at debian dot org
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (11 preceding siblings ...)
2020-10-23 11:47 ` sebastien at debian dot org
@ 2020-10-29 3:12 ` lvying.system.thoughts at gmail dot com
2020-11-11 8:31 ` stli at linux dot ibm.com
` (26 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: lvying.system.thoughts at gmail dot com @ 2020-10-29 3:12 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
lvying <lvying.system.thoughts at gmail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |lvying.system.thoughts@gmai
| |l.com
--- Comment #25 from lvying <lvying.system.thoughts at gmail dot com> ---
Hi, when I use Szabolcs Nagy's comment 2 comment 3, I got Assertion:
Inconsistency detected by ld.so: dl-tls.c: 517: _dl_allocate_tls_init:
Assertion `listp != NULL' failed!
Also, I try this testcase pacth:
https://patchwork.ozlabs.org/project/glibc/patch/5836CC80.9070101@arm.com/
After I put patch the patch into the glibc code, and run nptl testcase, I got:
../scripts/evaluate-test.sh nptl/tst-tls7 $? false false >
/root/build/build/glibc/nptl/tst-tls7.test-result
Inconsistency detected by ld.so: dl-tls.c: 517: _dl_allocate_tls_init:
Assertion `listp != NULL' failed!
Both of the testcases got different assertion, not the assertion:
Inconsistency detected by ld.so: dl-tls.c: 493: _dl_allocate_tls_init:
Assertion `listp->slotinfo[cnt].gen <= _rtld_local._dl_tls_generation' failed!
So, how to reproduce this problem to get the same failure.
And is there a plan to solve this problem? This problem has been going on for a
long time.
Thanks!
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (12 preceding siblings ...)
2020-10-29 3:12 ` lvying.system.thoughts at gmail dot com
@ 2020-11-11 8:31 ` stli at linux dot ibm.com
2020-12-01 2:15 ` lvying.system.thoughts at gmail dot com
` (25 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: stli at linux dot ibm.com @ 2020-11-11 8:31 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Stefan Liebler <stli at linux dot ibm.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |stli at linux dot ibm.com
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (13 preceding siblings ...)
2020-11-11 8:31 ` stli at linux dot ibm.com
@ 2020-12-01 2:15 ` lvying.system.thoughts at gmail dot com
2020-12-24 16:59 ` nsz at gcc dot gnu.org
` (24 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: lvying.system.thoughts at gmail dot com @ 2020-12-01 2:15 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #26 from lvying <lvying.system.thoughts at gmail dot com> ---
Update test result:
I use glibc2.28 source code to reproduce this problem:
1. testcase: Szabolcs Nagy's comment 2 comment 3
result: Inconsistency detected by ld.so: dl-tls.c: 517:
_dl_allocate_tls_init: Assertion `listp != NULL' failed!
Not the same assertion
2. testcase: add Szabolcs Nagy's testcase v3 into nptl testcase:
https://patchwork.ozlabs.org/project/glibc/patch/5836CC80.9070101@arm.com/
tst-tls7 result: same as No.1
3. testcase same as NO.2, However I add usleep(1000) between
_dl_add_to_slotinfo and ++GL(dl_tls_generation) at file elf/dl-open.c.
tst-tls7 result: same as No.1
tst-stack4 can reliably reproduce this problem
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (14 preceding siblings ...)
2020-12-01 2:15 ` lvying.system.thoughts at gmail dot com
@ 2020-12-24 16:59 ` nsz at gcc dot gnu.org
2021-01-30 15:44 ` azhelev+sourceware at mailbox dot org
` (23 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2020-12-24 16:59 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #27 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
i wrote down some more background before i resubmit my patches:
https://sourceware.org/pipermail/libc-alpha/2020-December/121090.html
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (15 preceding siblings ...)
2020-12-24 16:59 ` nsz at gcc dot gnu.org
@ 2021-01-30 15:44 ` azhelev+sourceware at mailbox dot org
2021-02-07 23:04 ` carlos at redhat dot com
` (22 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: azhelev+sourceware at mailbox dot org @ 2021-01-30 15:44 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
azhelev <azhelev+sourceware at mailbox dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |azhelev+sourceware@mailbox.
| |org
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (16 preceding siblings ...)
2021-01-30 15:44 ` azhelev+sourceware at mailbox dot org
@ 2021-02-07 23:04 ` carlos at redhat dot com
2021-02-17 16:00 ` nsz at gcc dot gnu.org
` (21 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: carlos at redhat dot com @ 2021-02-07 23:04 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #28 from Carlos O'Donell <carlos at redhat dot com> ---
I still see this issue in 2.33 testing. I saw it recently for ppc64 BE testing.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (17 preceding siblings ...)
2021-02-07 23:04 ` carlos at redhat dot com
@ 2021-02-17 16:00 ` nsz at gcc dot gnu.org
2021-02-23 2:48 ` carlos at redhat dot com
` (20 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2021-02-17 16:00 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #29 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
i have a new patch set that includes a different fix for this bug:
https://sourceware.org/pipermail/libc-alpha/2021-February/122626.html
the new fix takes the dlopen lock at thread creation time instead
of just using atomics (which cannot work for fixing the same race
with dlclose: bug 27111).
using atomics is still necessary for tls access.
it will likely take a few review iterations to get this in glibc.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (18 preceding siblings ...)
2021-02-17 16:00 ` nsz at gcc dot gnu.org
@ 2021-02-23 2:48 ` carlos at redhat dot com
2021-03-15 11:45 ` shanzhikun at gmail dot com
` (19 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: carlos at redhat dot com @ 2021-02-23 2:48 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Carlos O'Donell <carlos at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://sourceware.org/bugz
| |illa/show_bug.cgi?id=19924
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (19 preceding siblings ...)
2021-02-23 2:48 ` carlos at redhat dot com
@ 2021-03-15 11:45 ` shanzhikun at gmail dot com
2021-04-20 12:43 ` lilydjwg at gmail dot com
` (18 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: shanzhikun at gmail dot com @ 2021-03-15 11:45 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Sdrkun <shanzhikun at gmail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |shanzhikun at gmail dot com
--- Comment #30 from Sdrkun <shanzhikun at gmail dot com> ---
(In reply to Szabolcs Nagy from comment #29)
> i have a new patch set that includes a different fix for this bug:
> https://sourceware.org/pipermail/libc-alpha/2021-February/122626.html
>
> the new fix takes the dlopen lock at thread creation time instead
> of just using atomics (which cannot work for fixing the same race
> with dlclose: bug 27111).
>
> using atomics is still necessary for tls access.
>
> it will likely take a few review iterations to get this in glibc.
Hi,Szabolcs,
Do you know when will these patches be reviewed?
Their Delegate is still Nobody,
https://patchwork.sourceware.org/project/glibc/list/?series=1673.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (20 preceding siblings ...)
2021-03-15 11:45 ` shanzhikun at gmail dot com
@ 2021-04-20 12:43 ` lilydjwg at gmail dot com
2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
` (17 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: lilydjwg at gmail dot com @ 2021-04-20 12:43 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
lilydjwg at gmail dot com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |lilydjwg at gmail dot com
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (21 preceding siblings ...)
2021-04-20 12:43 ` lilydjwg at gmail dot com
@ 2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
` (16 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2021-05-11 16:17 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #31 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=1387ad6225c2222f027790e3f460e31aa5dd2c54
commit 1387ad6225c2222f027790e3f460e31aa5dd2c54
Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
Date: Wed Dec 30 19:19:37 2020 +0000
elf: Fix data races in pthread_create and TLS access [BZ #19329]
DTV setup at thread creation (_dl_allocate_tls_init) is changed
to take the dlopen lock, GL(dl_load_lock). Avoiding data races
here without locks would require design changes: the map that is
accessed for static TLS initialization here may be concurrently
freed by dlclose. That use after free may be solved by only
locking around static TLS setup or by ensuring dlclose does not
free modules with static TLS, however currently every link map
with TLS has to be accessed at least to see if it needs static
TLS. And even if that's solved, still a lot of atomics would be
needed to synchronize DTV related globals without a lock. So fix
both bug 19329 and bug 27111 with a lock that prevents DTV setup
running concurrently with dlopen or dlclose.
_dl_update_slotinfo at TLS access still does not use any locks
so CONCURRENCY NOTES are added to explain the synchronization.
The early exit from the slotinfo walk when max_modid is reached
is not strictly necessary, but does not hurt either.
An incorrect acquire load was removed from _dl_resize_dtv: it
did not synchronize with any release store or fence and
synchronization is now handled separately at thread creation
and TLS access time.
There are still a number of racy read accesses to globals that
will be changed to relaxed MO atomics in a followup patch. This
should not introduce regressions compared to existing behaviour
and avoid cluttering the main part of the fix.
Not all TLS access related data races got fixed here: there are
additional races at lazy tlsdesc relocations see bug 27137.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (22 preceding siblings ...)
2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
@ 2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
` (15 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2021-05-11 16:17 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #32 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=f4f8f4d4e0f92488431b268c8cd9555730b9afe9
commit f4f8f4d4e0f92488431b268c8cd9555730b9afe9
Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
Date: Wed Dec 30 19:19:37 2020 +0000
elf: Use relaxed atomics for racy accesses [BZ #19329]
This is a follow up patch to the fix for bug 19329. This adds relaxed
MO atomics to accesses that were previously data races but are now
race conditions, and where relaxed MO is sufficient.
The race conditions all follow the pattern that the write is behind the
dlopen lock, but a read can happen concurrently (e.g. during tls access)
without holding the lock. For slotinfo entries the read value only
matters if it reads from a synchronized write in dlopen or dlclose,
otherwise the related dtv entry is not valid to access so it is fine
to leave it in an inconsistent state. The same applies for
GL(dl_tls_max_dtv_idx) and GL(dl_tls_generation), but there the
algorithm relies on the fact that the read of the last synchronized
write is an increasing value.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (23 preceding siblings ...)
2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
@ 2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
2021-05-11 16:24 ` nsz at gcc dot gnu.org
` (14 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2021-05-11 16:17 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #33 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=572bd547d57a39b6cf0ea072545dc4048921f4c3
commit 572bd547d57a39b6cf0ea072545dc4048921f4c3
Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
Date: Thu Dec 31 13:59:38 2020 +0000
elf: Fix DTV gap reuse logic [BZ #27135]
For some reason only dlopen failure caused dtv gaps to be reused.
It is possible that the intent was to never reuse modids for a
different module, but after dlopen failure all gaps are reused
not just the ones caused by the unfinished dlopened.
So the code has to handle reused modids already which seems to
work, however the data races at thread creation and tls access
(see bug 19329 and bug 27111) may be more severe if slots are
reused so this is scheduled after those fixes. I think fixing
the races are not simpler if reuse is disallowed and reuse has
other benefits, so set GL(dl_tls_dtv_gaps) whenever entries are
removed from the middle of the slotinfo list. The value does
not have to be correct: incorrect true value causes the next
modid query to do a slotinfo walk, incorrect false will leave
gaps and new entries are added at the end.
Fixes bug 27135.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (24 preceding siblings ...)
2021-05-11 16:17 ` cvs-commit at gcc dot gnu.org
@ 2021-05-11 16:24 ` nsz at gcc dot gnu.org
2021-09-11 4:54 ` xujing99 at huawei dot com
` (13 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2021-05-11 16:24 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Szabolcs Nagy <nsz at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|--- |2.34
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #34 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
fixed for 2.34
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (25 preceding siblings ...)
2021-05-11 16:24 ` nsz at gcc dot gnu.org
@ 2021-09-11 4:54 ` xujing99 at huawei dot com
2021-09-11 5:06 ` xujing99 at huawei dot com
` (12 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: xujing99 at huawei dot com @ 2021-09-11 4:54 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
xujing <xujing99 at huawei dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |xujing99 at huawei dot com
--- Comment #35 from xujing <xujing99 at huawei dot com> ---
(In reply to cvs-commit@gcc.gnu.org from comment #31)
> The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
>
> https://sourceware.org/git/gitweb.cgi?p=glibc.git;
> h=1387ad6225c2222f027790e3f460e31aa5dd2c54
>
> commit 1387ad6225c2222f027790e3f460e31aa5dd2c54
> Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
> Date: Wed Dec 30 19:19:37 2020 +0000
>
> elf: Fix data races in pthread_create and TLS access [BZ #19329]
>
> DTV setup at thread creation (_dl_allocate_tls_init) is changed
> to take the dlopen lock, GL(dl_load_lock). Avoiding data races
> here without locks would require design changes: the map that is
> accessed for static TLS initialization here may be concurrently
> freed by dlclose. That use after free may be solved by only
> locking around static TLS setup or by ensuring dlclose does not
> free modules with static TLS, however currently every link map
> with TLS has to be accessed at least to see if it needs static
> TLS. And even if that's solved, still a lot of atomics would be
> needed to synchronize DTV related globals without a lock. So fix
> both bug 19329 and bug 27111 with a lock that prevents DTV setup
> running concurrently with dlopen or dlclose.
>
> _dl_update_slotinfo at TLS access still does not use any locks
> so CONCURRENCY NOTES are added to explain the synchronization.
> The early exit from the slotinfo walk when max_modid is reached
> is not strictly necessary, but does not hurt either.
>
> An incorrect acquire load was removed from _dl_resize_dtv: it
> did not synchronize with any release store or fence and
> synchronization is now handled separately at thread creation
> and TLS access time.
>
> There are still a number of racy read accesses to globals that
> will be changed to relaxed MO atomics in a followup patch. This
> should not introduce regressions compared to existing behaviour
> and avoid cluttering the main part of the fix.
>
> Not all TLS access related data races got fixed here: there are
> additional races at lazy tlsdesc relocations see bug 27137.
>
> Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
this patch use dl_load_lock in _dl_allocate_tls_init, is there a problem when
dlopen a dynamic library which will call pthread_create? I think it will cause
dl_load_lock and dl_load_lock dead lock.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (26 preceding siblings ...)
2021-09-11 4:54 ` xujing99 at huawei dot com
@ 2021-09-11 5:06 ` xujing99 at huawei dot com
2021-09-11 10:44 ` xujing99 at huawei dot com
` (11 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: xujing99 at huawei dot com @ 2021-09-11 5:06 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #36 from xujing <xujing99 at huawei dot com> ---
(In reply to xujing from comment #35)
> (In reply to cvs-commit@gcc.gnu.org from comment #31)
> > The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
> >
> > https://sourceware.org/git/gitweb.cgi?p=glibc.git;
> > h=1387ad6225c2222f027790e3f460e31aa5dd2c54
> >
> > commit 1387ad6225c2222f027790e3f460e31aa5dd2c54
> > Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
> > Date: Wed Dec 30 19:19:37 2020 +0000
> >
> > elf: Fix data races in pthread_create and TLS access [BZ #19329]
> >
> > DTV setup at thread creation (_dl_allocate_tls_init) is changed
> > to take the dlopen lock, GL(dl_load_lock). Avoiding data races
> > here without locks would require design changes: the map that is
> > accessed for static TLS initialization here may be concurrently
> > freed by dlclose. That use after free may be solved by only
> > locking around static TLS setup or by ensuring dlclose does not
> > free modules with static TLS, however currently every link map
> > with TLS has to be accessed at least to see if it needs static
> > TLS. And even if that's solved, still a lot of atomics would be
> > needed to synchronize DTV related globals without a lock. So fix
> > both bug 19329 and bug 27111 with a lock that prevents DTV setup
> > running concurrently with dlopen or dlclose.
> >
> > _dl_update_slotinfo at TLS access still does not use any locks
> > so CONCURRENCY NOTES are added to explain the synchronization.
> > The early exit from the slotinfo walk when max_modid is reached
> > is not strictly necessary, but does not hurt either.
> >
> > An incorrect acquire load was removed from _dl_resize_dtv: it
> > did not synchronize with any release store or fence and
> > synchronization is now handled separately at thread creation
> > and TLS access time.
> >
> > There are still a number of racy read accesses to globals that
> > will be changed to relaxed MO atomics in a followup patch. This
> > should not introduce regressions compared to existing behaviour
> > and avoid cluttering the main part of the fix.
> >
> > Not all TLS access related data races got fixed here: there are
> > additional races at lazy tlsdesc relocations see bug 27137.
> >
> > Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
>
> this patch use dl_load_lock in _dl_allocate_tls_init, is there a problem
> when dlopen a dynamic library which will call pthread_create? I think it
> will cause dl_load_lock and dl_load_lock dead lock.
dlopen will hold on dl_load_lock, and pthread_create will call
_dl_allocate_tls_init and then will hold on dl_load_lock. Finally, it will
cause dead lock.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (27 preceding siblings ...)
2021-09-11 5:06 ` xujing99 at huawei dot com
@ 2021-09-11 10:44 ` xujing99 at huawei dot com
2021-09-12 1:23 ` nsz at gcc dot gnu.org
` (10 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: xujing99 at huawei dot com @ 2021-09-11 10:44 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #37 from xujing <xujing99 at huawei dot com> ---
(In reply to cvs-commit@gcc.gnu.org from comment #31)
> The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
>
> https://sourceware.org/git/gitweb.cgi?p=glibc.git;
> h=1387ad6225c2222f027790e3f460e31aa5dd2c54
>
> commit 1387ad6225c2222f027790e3f460e31aa5dd2c54
> Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
> Date: Wed Dec 30 19:19:37 2020 +0000
>
> elf: Fix data races in pthread_create and TLS access [BZ #19329]
>
> DTV setup at thread creation (_dl_allocate_tls_init) is changed
> to take the dlopen lock, GL(dl_load_lock). Avoiding data races
> here without locks would require design changes: the map that is
> accessed for static TLS initialization here may be concurrently
> freed by dlclose. That use after free may be solved by only
> locking around static TLS setup or by ensuring dlclose does not
> free modules with static TLS, however currently every link map
> with TLS has to be accessed at least to see if it needs static
> TLS. And even if that's solved, still a lot of atomics would be
> needed to synchronize DTV related globals without a lock. So fix
> both bug 19329 and bug 27111 with a lock that prevents DTV setup
> running concurrently with dlopen or dlclose.
>
> _dl_update_slotinfo at TLS access still does not use any locks
> so CONCURRENCY NOTES are added to explain the synchronization.
> The early exit from the slotinfo walk when max_modid is reached
> is not strictly necessary, but does not hurt either.
>
> An incorrect acquire load was removed from _dl_resize_dtv: it
> did not synchronize with any release store or fence and
> synchronization is now handled separately at thread creation
> and TLS access time.
>
> There are still a number of racy read accesses to globals that
> will be changed to relaxed MO atomics in a followup patch. This
> should not introduce regressions compared to existing behaviour
> and avoid cluttering the main part of the fix.
>
> Not all TLS access related data races got fixed here: there are
> additional races at lazy tlsdesc relocations see bug 27137.
>
> Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Hi! I think this commit may cause an ABBA deadlock problem which i mentioned in
https://sourceware.org/bugzilla/show_bug.cgi?id=28331.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (28 preceding siblings ...)
2021-09-11 10:44 ` xujing99 at huawei dot com
@ 2021-09-12 1:23 ` nsz at gcc dot gnu.org
2021-09-13 2:50 ` xujing99 at huawei dot com
` (9 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2021-09-12 1:23 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #38 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
(In reply to xujing from comment #35)
> (In reply to cvs-commit@gcc.gnu.org from comment #31)
> > commit 1387ad6225c2222f027790e3f460e31aa5dd2c54
> > Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
> > Date: Wed Dec 30 19:19:37 2020 +0000
> >
> > elf: Fix data races in pthread_create and TLS access [BZ #19329]
> >
> this patch use dl_load_lock in _dl_allocate_tls_init, is there a problem
> when dlopen a dynamic library which will call pthread_create? I think it
> will cause dl_load_lock and dl_load_lock dead lock.
the real bug is that ctors are run with the dlopen lock held.
that can causes deadlocks anyway (a ctor can create threads
and that thread can call dlopen). this is bug 15686 which is not
easy to fix, but that's the right solution. (in general, running
user callbacks while libc internal locks are held is wrong.)
that bug is now more exposed because the lock is also taken
at _dl_allocate_tls_init during thread creation. however i
expect that to be called in the parent thread only, so there
should be no deadlock when ctor calls pthread_create, only
when the child thread calls it again (which i considered rare).
if you have example code that you think should work but now
deadlocks, then please report it.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (29 preceding siblings ...)
2021-09-12 1:23 ` nsz at gcc dot gnu.org
@ 2021-09-13 2:50 ` xujing99 at huawei dot com
2021-09-13 8:25 ` nsz at gcc dot gnu.org
` (8 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: xujing99 at huawei dot com @ 2021-09-13 2:50 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #39 from xujing <xujing99 at huawei dot com> ---
(In reply to Szabolcs Nagy from comment #38)
> (In reply to xujing from comment #35)
> > (In reply to cvs-commit@gcc.gnu.org from comment #31)
> > > commit 1387ad6225c2222f027790e3f460e31aa5dd2c54
> > > Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
> > > Date: Wed Dec 30 19:19:37 2020 +0000
> > >
> > > elf: Fix data races in pthread_create and TLS access [BZ #19329]
> > >
> > this patch use dl_load_lock in _dl_allocate_tls_init, is there a problem
> > when dlopen a dynamic library which will call pthread_create? I think it
> > will cause dl_load_lock and dl_load_lock dead lock.
>
> the real bug is that ctors are run with the dlopen lock held.
> that can causes deadlocks anyway (a ctor can create threads
> and that thread can call dlopen). this is bug 15686 which is not
> easy to fix, but that's the right solution. (in general, running
> user callbacks while libc internal locks are held is wrong.)
>
> that bug is now more exposed because the lock is also taken
> at _dl_allocate_tls_init during thread creation. however i
> expect that to be called in the parent thread only, so there
> should be no deadlock when ctor calls pthread_create, only
> when the child thread calls it again (which i considered rare).
>
> if you have example code that you think should work but now
> deadlocks, then please report it.
I'm sorry, I misled you. I think there is an ABBA deadlock issue in some
scenarios.
If I have a c++ dynamic library(named libA.so) that contains a global object,
the global object will call the post-constructor at initialization and hold
it's own lock(named A_lock) when dlopen loads libA.so. Assume that two threads
execute the following process：
Thread1:dlopen(libA.so) => hold dl_load_lock => load libA.so => init global
object from libA.so => wait for hold A_lock
Thread2:my own code hold A_lock => pthread_create => _dl_allocate_tls_init
=> wait for hold dl_load_lock
In this case, an ABBA deadlock occurs. Is this a bug?
My stack looks like this:
Thread 1 (LWP 136013):
#0 0x00007f57a108510d in ?? () from /usr/lib64/libpthread.so.0
#1 0x00007f57a107e4d1 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#1 stack waiting for holding A_lock
...
#6 0x00007f5781c1bb8b in LogProcess::Init (strProcName=...,
nProcHandle=nProcHandle@entry=0) at
./service/biz_frame/code/server/src/logging/logprocess.cpp:107
...
#20 0x00007f57a0fef21f in _dl_catch_exception () from /usr/lib64/libc.so.6
#21 0x00007f57a786442b in ?? () from /lib64/ld-linux-x86-64.so.2
#22 0x00007f57a3de2296 in ?? () from /usr/lib64/libdl.so.2
#23 0x00007f57a0fef21f in _dl_catch_exception () from /usr/lib64/libc.so.6
#24 0x00007f57a0fef2af in _dl_catch_error () from /usr/lib64/libc.so.6
#25 0x00007f57a3de2985 in ?? () from /usr/lib64/libdl.so.2
#26 0x00007f57a3de2351 in dlopen () from /usr/lib64/libdl.so.2
...
...
#38 0x00007f57a0fb3520 in clone () from /usr/lib64/libc.so.6
Thread 2 (LWP 134627):
#0 0x00007f57a108510d in ?? () from /usr/lib64/libpthread.so.0
#1 0x00007f57a107e580 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#2 0x00007f57a7863835 in _dl_allocate_tls_init () from
/lib64/ld-linux-x86-64.so.2
#3 0x00007f57a107cb7c in pthread_create () from /usr/lib64/libpthread.so.0
...
#10 Stack holding A_lock
...
#14 0x0000561689e0d579 in main ()
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (30 preceding siblings ...)
2021-09-13 2:50 ` xujing99 at huawei dot com
@ 2021-09-13 8:25 ` nsz at gcc dot gnu.org
2021-09-16 14:42 ` nsz at gcc dot gnu.org
` (7 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2021-09-13 8:25 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #40 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
(In reply to xujing from comment #39)
> I'm sorry, I misled you. I think there is an ABBA deadlock issue in some
> scenarios.
>
> If I have a c++ dynamic library(named libA.so) that contains a global
> object, the global object will call the post-constructor at initialization
> and hold it's own lock(named A_lock) when dlopen loads libA.so. Assume that
> two threads execute the following process：
> Thread1:dlopen(libA.so) => hold dl_load_lock => load libA.so => init
> global
> object from libA.so => wait for hold A_lock
> Thread2:my own code hold A_lock => pthread_create =>
> _dl_allocate_tls_init
> => wait for hold dl_load_lock
> In this case, an ABBA deadlock occurs. Is this a bug?
yes i think this should work (it is a lock ordering
issue between a user and libc internal lock, which
is only possible if user code is run while a libc
lock is held)
note that if you replace pthread_create with dlopen
that deadlocks too. so it's still bug 15686. but it
may be more common than i expected. i think we need
to look at fixing that bug. (fixing the dynamic tls
race of this bug without locks in pthread_create is
very hard, so i don't think we can revert the quoted
patch)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (31 preceding siblings ...)
2021-09-13 8:25 ` nsz at gcc dot gnu.org
@ 2021-09-16 14:42 ` nsz at gcc dot gnu.org
2021-09-21 13:20 ` nsz at gcc dot gnu.org
` (6 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2021-09-16 14:42 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #41 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
i'm trying to fix the ABBA deadlock by introducing a new
lock into dlopen that protects tls state and gets released
before init functions are run. using that lock at thread
creation would fix the issue.
this only requires a small amount of changes, but it seems
to be difficult to ensure that the new lock is released on
all failure paths within _dl_open_worker (which sometimes
uses longjmp for error handling).
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (32 preceding siblings ...)
2021-09-16 14:42 ` nsz at gcc dot gnu.org
@ 2021-09-21 13:20 ` nsz at gcc dot gnu.org
2021-10-04 14:12 ` cvs-commit at gcc dot gnu.org
` (5 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: nsz at gcc dot gnu.org @ 2021-09-21 13:20 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #42 from Szabolcs Nagy <nsz at gcc dot gnu.org> ---
i opened bug 28357 for the ABBA deadlock in pthread_create.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (33 preceding siblings ...)
2021-09-21 13:20 ` nsz at gcc dot gnu.org
@ 2021-10-04 14:12 ` cvs-commit at gcc dot gnu.org
2021-10-19 12:23 ` cvs-commit at gcc dot gnu.org
` (4 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2021-10-04 14:12 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #43 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Szabolcs Nagy <nsz@sourceware.org>:
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=83b5323261bb72313bffcf37476c1b8f0847c736
commit 83b5323261bb72313bffcf37476c1b8f0847c736
Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
Date: Wed Sep 15 15:16:19 2021 +0100
elf: Avoid deadlock between pthread_create and ctors [BZ #28357]
The fix for bug 19329 caused a regression such that pthread_create can
deadlock when concurrent ctors from dlopen are waiting for it to finish.
Use a new GL(dl_load_tls_lock) in pthread_create that is not taken
around ctors in dlopen.
The new lock is also used in __tls_get_addr instead of GL(dl_load_lock).
The new lock is held in _dl_open_worker and _dl_close_worker around
most of the logic before/after the init/fini routines. When init/fini
routines are running then TLS is in a consistent, usable state.
In _dl_open_worker the new lock requires catching and reraising dlopen
failures that happen in the critical section.
The new lock is reinitialized in a fork child, to keep the existing
behaviour and it is kept recursive in case malloc interposition or TLS
access from signal handlers can retake it. It is not obvious if this
is necessary or helps, but avoids changing the preexisting behaviour.
The new lock may be more appropriate for dl_iterate_phdr too than
GL(dl_load_write_lock), since TLS state of an incompletely loaded
module may be accessed. If the new lock can replace the old one,
that can be a separate change.
Fixes bug 28357.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (34 preceding siblings ...)
2021-10-04 14:12 ` cvs-commit at gcc dot gnu.org
@ 2021-10-19 12:23 ` cvs-commit at gcc dot gnu.org
2023-02-09 12:56 ` louisbenaze at gmail dot com
` (3 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2021-10-19 12:23 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #44 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The release/2.34/master branch has been updated by Florian Weimer
<fw@sourceware.org>:
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=024a7640ab9ecea80e527f4e4d7f7a1868e952c5
commit 024a7640ab9ecea80e527f4e4d7f7a1868e952c5
Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
Date: Wed Sep 15 15:16:19 2021 +0100
elf: Avoid deadlock between pthread_create and ctors [BZ #28357]
The fix for bug 19329 caused a regression such that pthread_create can
deadlock when concurrent ctors from dlopen are waiting for it to finish.
Use a new GL(dl_load_tls_lock) in pthread_create that is not taken
around ctors in dlopen.
The new lock is also used in __tls_get_addr instead of GL(dl_load_lock).
The new lock is held in _dl_open_worker and _dl_close_worker around
most of the logic before/after the init/fini routines. When init/fini
routines are running then TLS is in a consistent, usable state.
In _dl_open_worker the new lock requires catching and reraising dlopen
failures that happen in the critical section.
The new lock is reinitialized in a fork child, to keep the existing
behaviour and it is kept recursive in case malloc interposition or TLS
access from signal handlers can retake it. It is not obvious if this
is necessary or helps, but avoids changing the preexisting behaviour.
The new lock may be more appropriate for dl_iterate_phdr too than
GL(dl_load_write_lock), since TLS state of an incompletely loaded
module may be accessed. If the new lock can replace the old one,
that can be a separate change.
Fixes bug 28357.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
(cherry picked from commit 83b5323261bb72313bffcf37476c1b8f0847c736)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (35 preceding siblings ...)
2021-10-19 12:23 ` cvs-commit at gcc dot gnu.org
@ 2023-02-09 12:56 ` louisbenaze at gmail dot com
2023-02-09 13:11 ` jg at jguk dot org
` (2 subsequent siblings)
39 siblings, 0 replies; 40+ messages in thread
From: louisbenaze at gmail dot com @ 2023-02-09 12:56 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Louis Benazet <louisbenaze at gmail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |louisbenaze at gmail dot com
--- Comment #45 from Louis Benazet <louisbenaze at gmail dot com> ---
Hello, it seems this bugfix has reached Ubuntu 22.04, which is nice. However
some of my users use earlier Ubuntu versions (18.04 and 20.04) where it is not
available. How can I apply the fix on those versions? I tried checking out the
glibc version available in Ubuntu 18.04 (2.27) and cherry-picking the fixes.
But I get a conflict I don't know how to solve because the base files are very
different.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (36 preceding siblings ...)
2023-02-09 12:56 ` louisbenaze at gmail dot com
@ 2023-02-09 13:11 ` jg at jguk dot org
2024-05-16 7:27 ` fweimer at redhat dot com
2024-05-16 7:31 ` fweimer at redhat dot com
39 siblings, 0 replies; 40+ messages in thread
From: jg at jguk dot org @ 2023-02-09 13:11 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
--- Comment #46 from Jonny Grant <jg at jguk dot org> ---
(In reply to Louis Benazet from comment #45)
> Hello, it seems this bugfix has reached Ubuntu 22.04, which is nice. However
> some of my users use earlier Ubuntu versions (18.04 and 20.04) where it is
> not available. How can I apply the fix on those versions? I tried checking
> out the glibc version available in Ubuntu 18.04 (2.27) and cherry-picking
> the fixes. But I get a conflict I don't know how to solve because the base
> files are very different.
Perhaps you could ask Ubuntu to backport the fix. They are quite old releases,
but for LTS they might consider your request. If you need something today, it's
simpler just to upgrade.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (37 preceding siblings ...)
2023-02-09 13:11 ` jg at jguk dot org
@ 2024-05-16 7:27 ` fweimer at redhat dot com
2024-05-16 7:31 ` fweimer at redhat dot com
39 siblings, 0 replies; 40+ messages in thread
From: fweimer at redhat dot com @ 2024-05-16 7:27 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Florian Weimer <fweimer at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://sourceware.org/bugz
| |illa/show_bug.cgi?id=28357
CC| |fweimer at redhat dot com
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread* [Bug dynamic-link/19329] dl-tls.c assert failure at concurrent pthread_create and dlopen
[not found] <bug-19329-131@http.sourceware.org/bugzilla/>
` (38 preceding siblings ...)
2024-05-16 7:27 ` fweimer at redhat dot com
@ 2024-05-16 7:31 ` fweimer at redhat dot com
39 siblings, 0 replies; 40+ messages in thread
From: fweimer at redhat dot com @ 2024-05-16 7:31 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=19329
Florian Weimer <fweimer at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://sourceware.org/bugz
| |illa/show_bug.cgi?id=27135
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 40+ messages in thread