[Bug sanitizer/59410] New: Some tsan tests fail with 4GB RAM

[Bug sanitizer/59410] New: Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

            Bug ID: 59410
           Summary: Some tsan tests fail with 4GB RAM
           Product: gcc
           Version: 4.9.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: hjl.tools at gmail dot com
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org

On a Linux/x86-64 machine with 4GB RAM, I got failures like:

FAIL: c-c++-common/tsan/atomic_stack.c  -O0  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -O1  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -O2  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -O3 -fomit-frame-pointer  output
pattern test, is FATAL: ThreadSanitizer can not mmap the shadow memory
(something is mapped at 0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -O3 -g  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -Os  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -O2 -flto -fno-use-linker-plugin
-flto-partition=none  output pattern test, is FATAL: ThreadSanitizer can not
mmap the shadow memory (something is mapped at 0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/atomic_stack.c  -O2 -flto -fuse-linker-plugin
-fno-fat-lto-objects  output pattern test, is FATAL: ThreadSanitizer can not
mmap the shadow memory (something is mapped at 0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/fd_pipe_race.c  -O0  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/fd_pipe_race.c  -O1  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FAIL: c-c++-common/tsan/fd_pipe_race.c  -O2  output pattern test, is FATAL:
ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)

I didn't get those on machines with >= 6GB RAM.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[dvyukov at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #2 from Dmitry Vyukov <dvyukov at google dot com> ---
It seems that this kernel has ASLR disabled, so kernel maps libraries at 0x55.
Tsan does not support this ATM.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
BTW, the tsan.exp tests don't seem to be as cheap as was claimed during the
patch submission, I'd prefer to at least throttle the torture options down to
say -O0 and -O2 rather than so many different variants when the tests are
really small and optimizations don't really affect them that much if at all.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #3 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to H.J. Lu from comment #0)
> On a Linux/x86-64 machine with 4GB RAM, I got failures like:
> 
> FAIL: c-c++-common/tsan/atomic_stack.c  -O0  output pattern test, is FATAL:
> ThreadSanitizer can not mmap the shadow memory (something is mapped at
> 0x555555554000 < 0x7cf000000000)

This warning is not about physical RAM, but about virtual RAM. 
This systems is not compatible with the tsan's shadow mapping.
Can you show the /proc/self/maps of the process before it dies 
(just put a breakpoint on __tsan_init)? 

I assume that asan tests pass on this machine and you don't have strict rlimit 
on virtual memory.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #4 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to Dmitry Vyukov from comment #2)
> It seems that this kernel has ASLR disabled, so kernel maps libraries at
> 0x55. Tsan does not support this ATM.
BTW, the situation with tsan's shadow became worse after 
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a3defbe5c337dbc6da911f8cc49ae3cc3b49b453
where I see "Based on original patch by H.J. Lu and Josh Boyer."

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Have any attempt for saner tsan shadow memory mapping be done in the last year?
I mean, there were some PRs or mailing list threads about it being worth to
support also non-PIE executables etc., understandably it didn't happen for GCC
4.8, because there wasn't enough time for it, but do we have to live with that
limitation forever?

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2013-12-06
     Ever confirmed|0                           |1

--- Comment #6 from H.J. Lu <hjl.tools at gmail dot com> ---
I got those failures on this machine:

[hjl@gnu-ivb-1 ~]$ cat /proc/sys/kernel/randomize_va_space
0
[hjl@gnu-ivb-1 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       3926916    2649084    1277832          0      25948    2142616
-/+ buffers/cache:     480520    3446396
Swap:     25165820      35648   25130172
[hjl@gnu-32 ~]$ uname -a
Linux gnu-32.sc.intel.com 3.11.10-200.fc19.x86_64 #1 SMP Sun Dec 1 07:47:10 PST
2013 x86_64 x86_64 x86_64 GNU/Linux
[hjl@gnu-ivb-1 ~]$ 

They are OK on

[hjl@gnu-32 ~]$ cat /proc/sys/kernel/randomize_va_space
0
[hjl@gnu-32 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       6102624    4243468    1859156          0     480728    2897048
-/+ buffers/cache:     865692    5236932
Swap:     14335996      35768   14300228
[hjl@gnu-32 ~]$ uname -a
Linux gnu-32.sc.intel.com 3.11.10-200.fc19.x86_64 #1 SMP Sun Dec 1 07:47:10 PST
2013 x86_64 x86_64 x86_64 GNU/Linux
[hjl@gnu-32 ~]$ 

The only difference is 4GB RAM vs 6GB RAM.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #7 from H.J. Lu <hjl.tools at gmail dot com> ---
On failed machine:

[hjl@gnu-ivb-1 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30583
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30583
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[hjl@gnu-ivb-1 ~]$ 

On working machine:

[hjl@gnu-32 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 47571
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 47571
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[hjl@gnu-32 ~]$

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

Kostya Serebryany <kcc at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |UNCONFIRMED
   Last reconfirmed|2013-12-06 00:00:00         |
     Ever confirmed|1                           |0

--- Comment #8 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to Jakub Jelinek from comment #5)
> Have any attempt for saner tsan shadow memory mapping be done in the last
> year?
Nope. We don't want to add extra complexity w/o a strong reason; 
so far we did not see such reason. (If you want to continue discussing it, 
please let's not hijack this bug report)

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #9 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to H.J. Lu from comment #6)
> I got those failures on this machine:

Admittedly, I never ran tsan tests on a 4Gb machine.
Does clang's tsan also fail there? 
Can you show /proc/self/maps of the failing process?

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #10 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #3)
> (In reply to H.J. Lu from comment #0)
> > On a Linux/x86-64 machine with 4GB RAM, I got failures like:
> > 
> > FAIL: c-c++-common/tsan/atomic_stack.c  -O0  output pattern test, is FATAL:
> > ThreadSanitizer can not mmap the shadow memory (something is mapped at
> > 0x555555554000 < 0x7cf000000000)
> 
> This warning is not about physical RAM, but about virtual RAM. 
> This systems is not compatible with the tsan's shadow mapping.
> Can you show the /proc/self/maps of the process before it dies 

555555554000-555555555000 r-xp 00000000 08:11 34221424                  
/export/build/gnu/gcc-x32/build-x86_64-linux/gcc/testsuite/atomic_stack.exe
555555755000-555555756000 rw-p 00001000 08:11 34221424                  
/export/build/gnu/gcc-x32/build-x86_64-linux/gcc/testsuite/atomic_stack.exe
7d0000000000-7d0800000000 ---p 00000000 00:00 0 
7d0800000000-7d0800010000 rw-p 00000000 00:00 0 
7d0800010000-7d0bffff0000 ---p 00000000 00:00 0 
7d0bffff0000-7d0c00000000 rw-p 00000000 00:00 0 
7d0c00000000-7d6400000000 ---p 00000000 00:00 0 
7d6400000000-7d6400020000 rw-p 00000000 00:00 0 
7d6400020000-7d67ffff0000 ---p 00000000 00:00 0 
7d67ffff0000-7d6800000000 rw-p 00000000 00:00 0 
7d6800000000-7e0000000000 ---p 00000000 00:00 0 
7e0000000000-7e0000003000 rw-p 00000000 00:00 0 
7ffff4000000-7ffff5000000 rw-p 00000000 00:00 0 
7ffff5f1c000-7ffff5f31000 r-xp 00000000 08:05 1445037                   
/usr/lib64/libgcc_s-4.8.2-20131111.so.1
7ffff5f31000-7ffff6131000 ---p 00015000 08:05 1445037                   
/usr/lib64/libgcc_s-4.8.2-20131111.so.1
7ffff6131000-7ffff6132000 rw-p 00015000 08:05 1445037                   
/usr/lib64/libgcc_s-4.8.2-20131111.so.1
7ffff6132000-7ffff621e000 r-xp 00000000 08:11 34215160                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6.0.20
7ffff621e000-7ffff641d000 ---p 000ec000 08:11 34215160                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6.0.20
7ffff641d000-7ffff6425000 r--p 000eb000 08:11 34215160                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6.0.20
7ffff6425000-7ffff6427000 rw-p 000f3000 08:11 34215160                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6.0.20
7ffff6427000-7ffff643c000 rw-p 00000000 00:00 0 
7ffff643c000-7ffff643f000 r-xp 00000000 08:05 1443711                   
/usr/lib64/libdl-2.17.so
7ffff643f000-7ffff663e000 ---p 00003000 08:05 1443711                   
/usr/lib64/libdl-2.17.so
7ffff663e000-7ffff663f000 r--p 00002000 08:05 1443711                   
/usr/lib64/libdl-2.17.so
7ffff663f000-7ffff6640000 rw-p 00003000 08:05 1443711                   
/usr/lib64/libdl-2.17.so
7ffff6640000-7ffff6656000 r-xp 00000000 08:05 1443800                   
/usr/lib64/libpthread-2.17.so
7ffff6656000-7ffff6855000 ---p 00016000 08:05 1443800                   
/usr/lib64/libpthread-2.17.so
7ffff6855000-7ffff6856000 r--p 00015000 08:05 1443800                   
/usr/lib64/libpthread-2.17.so
7ffff6856000-7ffff6857000 rw-p 00016000 08:05 1443800                   
/usr/lib64/libpthread-2.17.so
7ffff6857000-7ffff685b000 rw-p 00000000 00:00 0 
7ffff685b000-7ffff6a10000 r-xp 00000000 08:05 1443557                   
/usr/lib64/libc-2.17.so
7ffff6a10000-7ffff6c0f000 ---p 001b5000 08:05 1443557                   
/usr/lib64/libc-2.17.so
7ffff6c0f000-7ffff6c13000 r--p 001b4000 08:05 1443557                   
/usr/lib64/libc-2.17.so
7ffff6c13000-7ffff6c15000 rw-p 001b8000 08:05 1443557                   
/usr/lib64/libc-2.17.so
7ffff6c15000-7ffff6c1a000 rw-p 00000000 00:00 0 
7ffff6c1a000-7ffff6d1b000 r-xp 00000000 08:05 1443818                   
/usr/lib64/libm-2.17.so
7ffff6d1b000-7ffff6f1a000 ---p 00101000 08:05 1443818                   
/usr/lib64/libm-2.17.so
7ffff6f1a000-7ffff6f1b000 r--p 00100000 08:05 1443818                   
/usr/lib64/libm-2.17.so
7ffff6f1b000-7ffff6f1c000 rw-p 00101000 08:05 1443818                   
/usr/lib64/libm-2.17.so
7ffff6f1c000-7ffff6fac000 r-xp 00000000 08:11 34216164                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libsanitizer/tsan/.libs/libtsan.so.0.0.0
7ffff6fac000-7ffff71ab000 ---p 00090000 08:11 34216164                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libsanitizer/tsan/.libs/libtsan.so.0.0.0
7ffff71ab000-7ffff71ae000 rw-p 0008f000 08:11 34216164                  
/export/build/gnu/gcc-x32/build-x86_64-linux/x86_64-unknown-linux-gnu/libsanitizer/tsan/.libs/libtsan.so.0.0.0
7ffff71ae000-7ffff7ddc000 rw-p 00000000 00:00 0 
7ffff7ddc000-7ffff7dfd000 r-xp 00000000 08:05 1443142                   
/usr/lib64/ld-2.17.so
7ffff7f64000-7ffff7fd5000 rw-p 00000000 00:00 0 
7ffff7ff8000-7ffff7ffa000 rw-p 00000000 00:00 0 
7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 00020000 08:05 1443142                   
/usr/lib64/ld-2.17.so
7ffff7ffd000-7ffff7ffe000 rw-p 00021000 08:05 1443142                   
/usr/lib64/ld-2.17.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                         
[stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                 
[vsyscall]

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #12 from H.J. Lu <hjl.tools at gmail dot com> ---
For some reason, tsan tests aren't run on 6GB machine.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #11 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> 555555554000-555555555000 r-xp 00000000 08:11 34221424                  
> /export/build/gnu/gcc-x32/build-x86_64-linux/gcc/testsuite/atomic_stack.exe

So, the executable is loaded into 555555554000, which intersects with 
tsan's shadow. 
See tsan/rtl/tsan_platform.h, around "C++ linux memory layout".
In our experience this happens when ASLR is off. 
And this is caused by the kernel patch I mentioned above. 
https://code.google.com/p/thread-sanitizer/wiki/CppManual?ts=1386348951&updated=CppManual#FAQ

We have not seen other reason for such mapping, maybe you know one :)

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #14 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #11)
> > 555555554000-555555555000 r-xp 00000000 08:11 34221424                  
> > /export/build/gnu/gcc-x32/build-x86_64-linux/gcc/testsuite/atomic_stack.exe
> 
> So, the executable is loaded into 555555554000, which intersects with 
> tsan's shadow. 
> See tsan/rtl/tsan_platform.h, around "C++ linux memory layout".
> In our experience this happens when ASLR is off. 
> And this is caused by the kernel patch I mentioned above. 
> https://code.google.com/p/thread-sanitizer/wiki/
> CppManual?ts=1386348951&updated=CppManual#FAQ
> 
> We have not seen other reason for such mapping, maybe you know one :)

Kernel is free to load PIE at ANY address it wants.  But
you can specify where to load PIE via a linker switch

-Ttext-segment 0x855555000000

to tell kernel to load PIE to a specific address.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #15 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
What I mean, unlike asan where the shadow memory shift and base is part of the
ABI, in tsan, while performance sensitive, the MemToShadow is still library
implementation detail.  So, I think it shouldn't be that difficult to support
it.
Perhaps have a compilation mode, one most performant where you'd hardcode what
you do right now and just give up if it can't be made to work, and another
slightly slower one where some of the components (I guess the shift can remain
hardcoded, but ored in constant can be variable, and perhaps some other
operation can be added to the function too).  Then during libtsan
initialization it can just see if the default memory layout is viable and
otherwise try some other one.  I thought we've discussed it to some extent
already, but don't remember where exactly.

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #16 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> Kernel is free to load PIE at ANY address it wants.  But
> you can specify where to load PIE via a linker switch
> 
> -Ttext-segment 0x855555000000
> 
> to tell kernel to load PIE to a specific address.

Hm. Interesting. What do I do wrong? 
% clang++ simple_race.cc -fsanitize=thread -Wl,-Ttext-segment=0x7d0000000000 ; 
(setarch x86_64 -R ./a.out )
FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FATAL: Make sure to compile with -fPIE and to link with -pie.

% clang++ simple_race.cc -fsanitize=thread ;  (setarch x86_64 -R ./a.out )
FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped at
0x555555554000 < 0x7cf000000000)
FATAL: Make sure to compile with -fPIE and to link with -pie.

% clang++ simple_race.cc -fsanitize=thread ;  ( ./a.out )
==================
WARNING: ThreadSanitizer: data race (pid=6559)

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #17 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> already, but don't remember where exactly.
Please let's move the discussion about non-PIE here:
https://code.google.com/p/thread-sanitizer/issues/detail?id=5

[Bug sanitizer/59410] Some tsan tests fail with 4GB RAM

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #18 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #16)
> > Kernel is free to load PIE at ANY address it wants.  But
> > you can specify where to load PIE via a linker switch
> > 
> > -Ttext-segment 0x855555000000
> > 
> > to tell kernel to load PIE to a specific address.
> 
> Hm. Interesting. What do I do wrong? 
> % clang++ simple_race.cc -fsanitize=thread -Wl,-Ttext-segment=0x7d0000000000
> ;  (setarch x86_64 -R ./a.out )
> FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped
> at 0x555555554000 < 0x7cf000000000)
> FATAL: Make sure to compile with -fPIE and to link with -pie.

Please show the output of

# readelf -lW a.out

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2013-12-06
            Summary|Some tsan tests fail with   |tsan tests fail with
                   |address randomization       |address randomization
                   |disabled                    |disabled
     Ever confirmed|0                           |1

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #19 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to H.J. Lu from comment #18)
> (In reply to Kostya Serebryany from comment #16)
> > > Kernel is free to load PIE at ANY address it wants.  But
> > > you can specify where to load PIE via a linker switch
> > > 
> > > -Ttext-segment 0x855555000000
> > > 
> > > to tell kernel to load PIE to a specific address.
> > 
> > Hm. Interesting. What do I do wrong? 
> > % clang++ simple_race.cc -fsanitize=thread -Wl,-Ttext-segment=0x7d0000000000
> > ;  (setarch x86_64 -R ./a.out )
> > FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped
> > at 0x555555554000 < 0x7cf000000000)
> > FATAL: Make sure to compile with -fPIE and to link with -pie.
> 
> Please show the output of
> 
> # readelf -lW a.out

Your address must be sensible.  Otherwise kernel will ignore it.
Please try "-Ttext-segment 0x855555000000".

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #21 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #20)
> > > 
> > > # readelf -lW a.out
> > 
> > Your address must be sensible.  Otherwise kernel will ignore it.
> > Please try "-Ttext-segment 0x855555000000".
> 
> How is 0x855555000000 censible if it's beyond the address space? 
> (Or I miss something?)
> 
> Anyway, here is an experiment that proves that on my box 
> -Ttext-segment is ignored if ASLR is off. 
> 
>

That is true.  The kernel change was made to fix:

https://bugzilla.kernel.org/show_bug.cgi?id=36372

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #22 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> That is true.  The kernel change was made to fix:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=36372

Could you please explain the situation? 
What was fixed and in which kernel?

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #23 from H.J. Lu <hjl.tools at gmail dot com> ---
I opened:

https://bugzilla.kernel.org/show_bug.cgi?id=66721

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #24 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #22)
> > That is true.  The kernel change was made to fix:
> > 
> > https://bugzilla.kernel.org/show_bug.cgi?id=36372
> 
> Could you please explain the situation? 
> What was fixed and in which kernel?

The kernel bug is explained at

https://bugzilla.redhat.com/show_bug.cgi?id=708563
https://bugzilla.kernel.org/show_bug.cgi?id=36372

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #25 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> https://bugzilla.kernel.org/show_bug.cgi?id=66721

Good! Looking forward for a fix and hoping to have it in the next Ubuntu LTS.
Any chance? 
Is there anything else we could do with this tsan issue?
(given that this is our FAQ entry #1)

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #26 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #25)
> > https://bugzilla.kernel.org/show_bug.cgi?id=66721
> 
> Good! Looking forward for a fix and hoping to have it in the next Ubuntu LTS.
> Any chance? 
> Is there anything else we could do with this tsan issue?
> (given that this is our FAQ entry #1)

TSAN tests shouldn't depend on kernel to load PIE at a suitable
address at random since kernel may load PIE at the "wrong"
address.  TSAN tests should specify a load address which
works with TSAN.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

Yury Gribov <y.gribov at samsung dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |y.gribov at samsung dot com

--- Comment #27 from Yury Gribov <y.gribov at samsung dot com> ---
(In reply to Jakub Jelinek from comment #1)
> BTW, the tsan.exp tests don't seem to be as cheap as was claimed during the
> patch submission

I guess this may depend on the machine, Max's is rather powerful.

> I'd prefer to at least throttle the torture options down
> to say -O0 and -O2 rather than so many different variants

Agreed, we'll send a fix tomorrow.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #29 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
It is not that easy.  gold before February 2013 doesn't grok -Ttext-segment,
you need -Ttext there instead.  See
http://gcc.gnu.org/ml/gcc-patches/2013-02/msg00777.html
And, given that gcc can be configured to dynamically decide which of the
linkers to use, it is a portability nightmare.  So IMNSHO it is much better to
improve libtsan to handle it correctly.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #30 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
We could add -Ttext-segment=0x7d0000000000 (not 0x855555000000!),
to either the tests or the driver, but
a) see Jakub's comment about Gold
b) it will not fix anything, and
c) it will not help when ASLR is off.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #31 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Jakub Jelinek from comment #29)
> It is not that easy.  gold before February 2013 doesn't grok -Ttext-segment,
> you need -Ttext there instead.  See
> http://gcc.gnu.org/ml/gcc-patches/2013-02/msg00777.html
> And, given that gcc can be configured to dynamically decide which of the
> linkers to use, it is a portability nightmare.  So IMNSHO it is much better
> to improve libtsan to handle it correctly.

We can pass -fuse-ld=bfd to gcc driver if it is supported and
-fuse-ld=gold doesn't take -Ttext-segment.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #32 from Yury Gribov <y.gribov at samsung dot com> ---
Jakub has posted patch which may fix this:
http://gcc.gnu.org/ml/gcc-patches/2014-01/msg02044.html

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #33 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Author: jakub
Date: Fri Jan 31 21:25:23 2014
New Revision: 207371

URL: http://gcc.gnu.org/viewcvs?rev=207371&root=gcc&view=rev
Log:
    PR sanitizer/59410
    * lib/tsan-dg.exp (tsan_init): Instead of not running any
    tsan tests if trivial testcase doesn't run, set dg-do-what-default
    to compile.
    (tsan_finish): Restore dg-do-what-default.
    * g++.dg/tsan/atomic_free.C: Remove dg-do line.
    * g++.dg/tsan/fd_close_norace2.C: Likewise.
    * g++.dg/tsan/default_options.C: Likewise.
    * g++.dg/tsan/aligned_vs_unaligned_race.C: Likewise.
    * g++.dg/tsan/atomic_free2.C: Likewise.
    * g++.dg/tsan/cond_race.C: Likewise.
    * g++.dg/tsan/fd_close_norace.C: Likewise.
    * g++.dg/tsan/benign_race.C: Likewise.
    * c-c++-common/tsan/fd_pipe_race.c: Likewise.
    * c-c++-common/tsan/simple_race.c: Likewise.
    * c-c++-common/tsan/mutexset1.c: Likewise.
    * c-c++-common/tsan/thread_leak2.c: Likewise.
    * c-c++-common/tsan/tls_race.c: Likewise.
    * c-c++-common/tsan/write_in_reader_lock.c: Likewise.
    * c-c++-common/tsan/race_on_barrier2.c: Likewise.
    * c-c++-common/tsan/free_race2.c: Likewise.
    * c-c++-common/tsan/thread_leak.c: Likewise.
    * c-c++-common/tsan/thread_leak1.c: Likewise.
    * c-c++-common/tsan/race_on_barrier.c: Likewise.
    * c-c++-common/tsan/free_race.c: Likewise.
    * c-c++-common/tsan/sleep_sync.c: Likewise.
    * c-c++-common/tsan/tiny_race.c: Likewise.
    * c-c++-common/tsan/race_on_mutex2.c: Likewise.
    * c-c++-common/tsan/atomic_stack.c: Likewise.
    * c-c++-common/tsan/race_on_mutex.c: Likewise.  Adjust line numbers
    in dg-output regexps.
    * c-c++-common/tsan/simple_stack.c: Likewise.

Modified:
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/c-c++-common/tsan/atomic_stack.c
    trunk/gcc/testsuite/c-c++-common/tsan/fd_pipe_race.c
    trunk/gcc/testsuite/c-c++-common/tsan/free_race.c
    trunk/gcc/testsuite/c-c++-common/tsan/free_race2.c
    trunk/gcc/testsuite/c-c++-common/tsan/mutexset1.c
    trunk/gcc/testsuite/c-c++-common/tsan/race_on_barrier.c
    trunk/gcc/testsuite/c-c++-common/tsan/race_on_barrier2.c
    trunk/gcc/testsuite/c-c++-common/tsan/race_on_mutex.c
    trunk/gcc/testsuite/c-c++-common/tsan/race_on_mutex2.c
    trunk/gcc/testsuite/c-c++-common/tsan/simple_race.c
    trunk/gcc/testsuite/c-c++-common/tsan/simple_stack.c
    trunk/gcc/testsuite/c-c++-common/tsan/sleep_sync.c
    trunk/gcc/testsuite/c-c++-common/tsan/thread_leak.c
    trunk/gcc/testsuite/c-c++-common/tsan/thread_leak1.c
    trunk/gcc/testsuite/c-c++-common/tsan/thread_leak2.c
    trunk/gcc/testsuite/c-c++-common/tsan/tiny_race.c
    trunk/gcc/testsuite/c-c++-common/tsan/tls_race.c
    trunk/gcc/testsuite/c-c++-common/tsan/write_in_reader_lock.c
    trunk/gcc/testsuite/g++.dg/tsan/aligned_vs_unaligned_race.C
    trunk/gcc/testsuite/g++.dg/tsan/atomic_free.C
    trunk/gcc/testsuite/g++.dg/tsan/atomic_free2.C
    trunk/gcc/testsuite/g++.dg/tsan/benign_race.C
    trunk/gcc/testsuite/g++.dg/tsan/cond_race.C
    trunk/gcc/testsuite/g++.dg/tsan/default_options.C
    trunk/gcc/testsuite/g++.dg/tsan/fd_close_norace.C
    trunk/gcc/testsuite/g++.dg/tsan/fd_close_norace2.C
    trunk/gcc/testsuite/lib/tsan-dg.exp

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[dvyukov at google dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

Dmitry Vyukov <dvyukov at google dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dvyukov at google dot com

--- Comment #34 from Dmitry Vyukov <dvyukov at google dot com> ---
I propose to close this as obsolete. Or does somebody see any actionable items
here?
Non-pie binaries are supported by tsan now. Non-aslr (0x55555) are not.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WONTFIX

--- Comment #35 from H.J. Lu <hjl.tools at gmail dot com> ---
Binutils 2.25 will mark PIE with
non-zero load address as executable
so that any kernel will load it at
specified address.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[kcc at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #36 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
While we are at it, H.J., is there any hope with 
https://bugzilla.kernel.org/show_bug.cgi?id=66721 ?

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #37 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #36)
> While we are at it, H.J., is there any hope with 
> https://bugzilla.kernel.org/show_bug.cgi?id=66721 ?

I will see what I can do.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[dvyukov at google dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #39 from Dmitry Vyukov <dvyukov at google dot com> ---
>Isomalloc works best when we can assure it as large a range of common, unoccupied virtual address space as possible. Thus, it's much happier when ASLR is disabled.

I do not follow you here. How does ASLR give you larger continuous region? ASLR
maps libraries in the middle of virtual address space (0x5555), so if anything
it gives smaller continuous region.
On top of that tsan itself uses lots of memory and restricts user mappings to
0x7e00-0x7ffff range.
I would like to understand problem better before we do anything.

[Bug sanitizer/59410] tsan tests fail with address randomization disabled

[dvyukov at google dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59410

--- Comment #40 from Dmitry Vyukov <dvyukov at google dot com> ---
Phil, please move the discussion to:
https://groups.google.com/forum/#!forum/thread-sanitizer
This is not gcc specific, and more people will be able to see it in the tsan
group.