[Bug sanitizer/59733] New: [4.9 Regression] bootstrap-asan failed to bootstrap

[Bug sanitizer/59733] New: [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

            Bug ID: 59733
           Summary: [4.9 Regression] bootstrap-asan failed to bootstrap
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: hjl.tools at gmail dot com
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org

On Linux/x86-64 with 24GB RAM, r206441 configured with

.../configure --enable-clocale=gnu --with-system-zlib --with-demangler-in-ld
--enable-languages=c,c++,fortran,java,lto,objc,obj-c++,go  --prefix=/usr/local
--enable-gnu-indirect-function --disable-werror --with-arch=corei7
--with-cpu=slm --with-build-config="bootstrap-asan" --with-fpmath=sse

gave me

/export/build/gnu/gcc-asan/build-x86_64-linux/./prev-gcc/xg++
-B/export/build/gnu/gcc-asan/build-x86_64-linux/./prev-gcc/
-B/usr/local/x86_64-unknown-linux-gnu/bin/ -nostdinc++
-B/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs
-B/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/libsupc++/.libs
-I/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu
-I/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/include
-I/export/gnu/import/git/gcc/libstdc++-v3/libsupc++
-L/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs
-L/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/libsupc++/.libs
-c   -O2 -g  -fsanitize=address -DIN_GCC    -fno-exceptions -fno-rtti
-fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings
-Wcast-qual -Wmissing-format-attribute -pedantic -Wno-long-long
-Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -I.
-I. -I/export/gnu/import/git/gcc/gcc -I/export/gnu/import/git/gcc/gcc/.
-I/export/gnu/import/git/gcc/gcc/../include
-I/export/gnu/import/git/gcc/gcc/../libcpp/include 
-I/export/gnu/import/git/gcc/gcc/../libdecnumber
-I/export/gnu/import/git/gcc/gcc/../libdecnumber/bid -I../libdecnumber
-I/export/gnu/import/git/gcc/gcc/../libbacktrace    -o insn-recog.o -MT
insn-recog.o -MMD -MP -MF ./.deps/insn-recog.TPo insn-recog.c
==1395==ERROR: AddressSanitizer failed to allocate 0x25000 (151552) bytes of
LargeMmapAllocator: 12
ERROR: Failed to mmap

20131224 bootstrapped OK.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.9.0

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2014-01-09
                 CC|                            |kirill.yukhin at intel dot com
     Ever confirmed|0                           |1

--- Comment #1 from H.J. Lu <hjl.tools at gmail dot com> ---
It is caused by r206221.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #2 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
It doesn't matter which change triggered it, this looks like libsanitizer
internal error, it assumes no internal allocations will need to be large, but
apparently some are.  So, the question is what is the backtrace for this error.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #3 from H.J. Lu <hjl.tools at gmail dot com> ---
Breakpoint 6, __sanitizer::Report (
    format=format@entry=0x2821768 "ERROR: %s failed to allocate 0x%zx (%zd)
bytes of %s: %d\n")
---Type <return> to continue, or q <return> to quit---
    at
/export/gnu/import/git/gcc/libsanitizer/sanitizer_common/sanitizer_printf.cc:276
276    void Report(const char *format, ...) {
(gdb) bt
#0  __sanitizer::Report (
    format=format@entry=0x2821768 "ERROR: %s failed to allocate 0x%zx (%zd)
bytes of %s: %d\n")
    at
/export/gnu/import/git/gcc/libsanitizer/sanitizer_common/sanitizer_printf.cc:276
#1  0x000000000062a660 in __sanitizer::MmapOrDie (size=<optimized out>, 
    size@entry=151552, mem_type=mem_type@entry=0x2817d54 "LargeMmapAllocator")
    at
/export/gnu/import/git/gcc/libsanitizer/sanitizer_common/sanitizer_posix.cc:64
#2  0x00000000005e1cf4 in Allocate (alignment=8, size=146992, 
    stat=0x36d0d58 <__asan::allocator+2097752>, 
    this=0x34d0b08 <__asan::allocator+8>)
    at
/export/gnu/import/git/gcc/libsanitizer/sanitizer_common/sanitizer_allocator.h:1009
#3  Allocate (cleared=false, alignment=8, size=146992, cache=0x7ffff7ab00c8, 
    this=0x34d0b00 <__asan::allocator>)
    at
/export/gnu/import/git/gcc/libsanitizer/sanitizer_common/sanitizer_allocator.h:1251
#4  __asan::Allocate (size=<optimized out>, size@entry=142896, 
    alignment=<optimized out>, alignment@entry=8, 
    stack=stack@entry=0x7fffffffc640, 
    alloc_type=alloc_type@entry=__asan::FROM_MALLOC, 
    can_fill=can_fill@entry=true)
---Type <return> to continue, or q <return> to quit---
    at /export/gnu/import/git/gcc/libsanitizer/asan/asan_allocator2.cc:352
#5  0x00000000005e2578 in __asan::asan_malloc (size=size@entry=142896, 
    stack=stack@entry=0x7fffffffc640)
    at /export/gnu/import/git/gcc/libsanitizer/asan/asan_allocator2.cc:619
#6  0x000000000061ad92 in __interceptor_malloc (size=size@entry=142896)
    at /export/gnu/import/git/gcc/libsanitizer/asan/asan_malloc_linux.cc:73
#7  0x0000000002808a88 in xmalloc (size=142896)
    at /export/gnu/import/git/gcc/libiberty/xmalloc.c:147
#8  0x0000000000ea4e1d in calc_dfs_tree_nonrec (di=0x7fffffffd090, 
    bb=<basic_block 0x7fffde747410 (3598)>, reverse=<optimized out>)
    at /export/gnu/import/git/gcc/gcc/dominance.c:237
#9  0x0000000000ea5c0c in calc_dfs_tree (di=di@entry=0x7fffffffd090, 
    reverse=reverse@entry=true)
    at /export/gnu/import/git/gcc/gcc/dominance.c:374
#10 0x0000000000ea98b1 in calculate_dominance_info (
    dir=dir@entry=CDI_POST_DOMINATORS)
    at /export/gnu/import/git/gcc/gcc/dominance.c:662
#11 0x0000000001bc8a3f in execute_late_warn_uninitialized ()
    at /export/gnu/import/git/gcc/gcc/tree-ssa-uninit.c:2297
#12 (anonymous namespace)::pass_late_warn_uninitialized::execute (
    this=<optimized out>)
    at /export/gnu/import/git/gcc/gcc/tree-ssa-uninit.c:2388
#13 0x00000000014ff145 in execute_one_pass (
---Type <return> to continue, or q <return> to quit---
    pass=pass@entry=<opt_pass* 0x608000007720 "uninit"(145)>)
    at /export/gnu/import/git/gcc/gcc/passes.c:2226
#14 0x00000000014ffb51 in execute_pass_list (
    pass=<opt_pass* 0x608000007720 "uninit"(145)>)
    at /export/gnu/import/git/gcc/gcc/passes.c:2279
#15 0x00000000014ffb77 in execute_pass_list (
    pass=<opt_pass* 0x60800000a420 "*all_optimizations"(-1)>)
    at /export/gnu/import/git/gcc/gcc/passes.c:2280
#16 0x0000000000dff83b in expand_function (
    node=node@entry=<cgraph_node* 0x7fffe0b567b0 "peephole2_insns">)
    at /export/gnu/import/git/gcc/gcc/cgraphunit.c:1764
#17 0x0000000000e04c11 in expand_all_functions ()
    at /export/gnu/import/git/gcc/gcc/cgraphunit.c:1898
#18 compile () at /export/gnu/import/git/gcc/gcc/cgraphunit.c:2242
#19 0x0000000000e065eb in finalize_compilation_unit ()
    at /export/gnu/import/git/gcc/gcc/cgraphunit.c:2319
#20 0x000000000089cb53 in cp_write_global_declarations ()
    at /export/gnu/import/git/gcc/gcc/cp/decl2.c:4431
#21 0x00000000016f0135 in compile_file ()
    at /export/gnu/import/git/gcc/gcc/toplev.c:562
#22 0x00000000016f505c in do_compile ()
    at /export/gnu/import/git/gcc/gcc/toplev.c:1914
#23 toplev_main (argc=76, argv=0x7fffffffd9d8)
---Type <return> to continue, or q <return> to quit---
    at /export/gnu/import/git/gcc/gcc/toplev.c:1990
#24 0x0000003cdda21b45 in __libc_start_main () from /lib64/libc.so.6
#25 0x00000000005ded71 in _start ()

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #4 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
BTW, are you sure you don't have some ulimit imposed limits (say on virtual
address space)?
Can you strace it (perhaps just strace -o log -e mmap,munmap )?

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #5 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Jakub Jelinek from comment #4)
> BTW, are you sure you don't have some ulimit imposed limits (say on virtual
> address space)?

[hjl@gnu-mic-2 gcc]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192280
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 192280
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[hjl@gnu-mic-2 gcc]$ 

> Can you strace it (perhaps just strace -o log -e mmap,munmap )?

I will do it.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #6 from H.J. Lu <hjl.tools at gmail dot com> ---
Created attachment 31788
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=31788&action=edit
strace -o log -e mmap,munmap

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Can't reproduce this, at least not if I use sufficiently lower make -jN factor
(used -j2).  With that
../configure --enable-languages=c,c++,fortran,java,lto,objc,obj-c++,go
--with-build-config=bootstrap-asan --disable-werror
built all the way through compare and died only on fixincl not being asan
bootstrap compatible.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #8 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Jakub Jelinek from comment #7)
> Can't reproduce this, at least not if I use sufficiently lower make -jN
> factor (used -j2).  With that
> ../configure --enable-languages=c,c++,fortran,java,lto,objc,obj-c++,go
> --with-build-config=bootstrap-asan --disable-werror
> built all the way through compare and died only on fixincl not being asan
> bootstrap compatible.

It still fails for me.  It has nothing to do with "make -jN". When I
ran it by hand, I got

/export/build/gnu/gcc-asan/build-x86_64-linux/./prev-gcc/xg++
-B/export/build/gnu/gcc-asan/build-x86_64-linux/./prev-gcc/
-B/usr/local/x86_64-unknown-linux-gnu/bin/ -nostdinc++
-B/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs
-B/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/libsupc++/.libs
-I/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu
-I/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/include
-I/export/gnu/import/git/gcc-misc/libstdc++-v3/libsupc++
-L/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs
-L/export/build/gnu/gcc-asan/build-x86_64-linux/prev-x86_64-unknown-linux-gnu/libstdc++-v3/libsupc++/.libs
-c   -O2 -g  -fsanitize=address -DIN_GCC    -fno-exceptions -fno-rtti
-fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings
-Wcast-qual -Wmissing-format-attribute -pedantic -Wno-long-long
-Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -I.
-I. -I/export/gnu/import/git/gcc-misc/gcc
-I/export/gnu/import/git/gcc-misc/gcc/.
-I/export/gnu/import/git/gcc-misc/gcc/../include
-I/export/gnu/import/git/gcc-misc/gcc/../libcpp/include 
-I/export/gnu/import/git/gcc-misc/gcc/../libdecnumber
-I/export/gnu/import/git/gcc-misc/gcc/../libdecnumber/bid -I../libdecnumber
-I/export/gnu/import/git/gcc-misc/gcc/../libbacktrace    -o insn-recog.o -MT
insn-recog.o -MMD -MP -MF ./.deps/insn-recog.TPo insn-recog.c
==19786==ERROR: AddressSanitizer failed to allocate 0x24000 (147456) bytes of
LargeMmapAllocator: 12
ERROR: Failed to mmap

I did it on a Fedora 19 machine with 24GB RAM and 48GB swap.  I configured GCC
with

--enable-clocale=gnu --with-system-zlib --with-demangler-in-ld
--enable-languages=c,c++ --prefix=/usr/local --enable-gnu-indirect-function
--disable-werror --with-arch=corei7 --with-cpu=slm
--with-build-config=bootstrap-asan --with-fpmath=sse

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #9 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> It still fails for me.  It has nothing to do with "make -jN". When I

I tried running this on the fresh gcc trunk.

During the 3-rd stage I get this:
checking how to run the C preprocessor... ../libiberty/libiberty.a(regex.o): In
function `byte_compile_range':
... gcc-boot/libiberty/../../gcc/libiberty/regex.c:4495: undefined reference to
`__asan_report_load1'

Before that the builds works fine, but compilation of insn-recog.c
with asan-ified gcc is indeed expensive: takes ~8 minutes and 4+Gb RAM
This excessive memory usage may be the reason of your OOM in case you 
have some sort of limit (Are you by any chance building in x32 mode?)

non-asanified build of insn-recog.c takes ~3.5 minutes and 700M RAM. 

The asan slowdown here is reasonable, but the memory overhead is surprisingly
large...

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #10 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
H.J., can you send the contents of /proc/PID/{maps,smaps,status}
of the failing process before it dies?
(you can use ASAN_OPTIONS=sleep_before_dying=1000)

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #11 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #9)
> > It still fails for me.  It has nothing to do with "make -jN". When I
> 
> I tried running this on the fresh gcc trunk.
> 
> During the 3-rd stage I get this:
> checking how to run the C preprocessor... ../libiberty/libiberty.a(regex.o):
> In function `byte_compile_range':
> ... gcc-boot/libiberty/../../gcc/libiberty/regex.c:4495: undefined reference
> to `__asan_report_load1'

Please try hjl/asan branch at

http://gcc.gnu.org/git/?p=gcc.git;a=summary

It fixes this problem among others.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #12 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #10)
> H.J., can you send the contents of /proc/PID/{maps,smaps,status}
> of the failing process before it dies?
> (you can use ASAN_OPTIONS=sleep_before_dying=1000)

If you run cc1plus under GDB, you will get

Breakpoint 6, __sanitizer::Report (
    format=format@entry=0x284f6f0 "ERROR: %s failed to allocate 0x%zx (%zd)
bytes of %s: %d\n")

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #13 from Kostya Serebryany <kcc at gcc dot gnu.org> ---

> If you run cc1plus under GDB, you will get
> 
> Breakpoint 6, __sanitizer::Report (
>     format=format@entry=0x284f6f0 "ERROR: %s failed to allocate 0x%zx (%zd)
> bytes of %s: %d\n")
This happens due to OOM, we need to understand why OOM happens.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #14 from H.J. Lu <hjl.tools at gmail dot com> ---
Created attachment 31946
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=31946&action=edit
/proc/cc1plus/{maps,smaps,status}

This is the output of "cat /proc/cc1plus/{maps,smaps,status}".

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #15 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
This looks very weird: 
602001200000-602001210000 rw-p 00000000 00:00 0 
602001210000-602001220000 rw-p 00000000 00:00 0 

We have two adjacent mappings with the same perms
 which are not merged by the kernel into one.
This is not what I typically see on my linux. 
Is this some kind of modified kernel config or something? 

If your kernel behaves this way no winder that mmap eventually fails.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #16 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Kostya Serebryany from comment #15)
> This looks very weird: 
> 602001200000-602001210000 rw-p 00000000 00:00 0 
> 602001210000-602001220000 rw-p 00000000 00:00 0 
> 
> We have two adjacent mappings with the same perms
>  which are not merged by the kernel into one.
> This is not what I typically see on my linux. 
> Is this some kind of modified kernel config or something? 
> 
> If your kernel behaves this way no winder that mmap eventually fails.

I am running 3.12.8-200.fc19.x86_64 from Fedora 19.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #17 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
When I look at my /proc/PID/maps of cc1plus, I see this: 

...
610000000000-610000030000 rw-p 00000000 00:00 0 
610000030000-611000000000 ---p 00000000 00:00 0 
611000000000-611000980000 rw-p 00000000 00:00 0 
611000980000-612000000000 ---p 00000000 00:00 0 
612000000000-6120001b0000 rw-p 00000000 00:00 0 
6120001b0000-613000000000 ---p 00000000 00:00 0 
613000000000-613000370000 rw-p 00000000 00:00 0 
613000370000-614000000000 ---p 00000000 00:00 0 
614000000000-6140001a0000 rw-p 00000000 00:00 0 
6140001a0000-615000000000 ---p 00000000 00:00 0 
615000000000-615000460000 rw-p 00000000 00:00 0 
...
As you can see, there are no adjacent mappings with same perms. 

Here is a tiny test:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
int main() {
  static volatile void *x;
  for (int i = 0; i < 1000000; i++)
    x = malloc(1000);
  printf("PID %d\n", getpid());
  sleep(100);
}

% clang -fsanitize=address nmalloc.c ; ./a.out 

The mapping at the end looks like this:
...
2008fff7000-10007fff8000 rw-p 00000000 00:00 0 
600000000000-619000000000 ---p 00000000 00:00 0 
619000000000-61904c4c0000 rw-p 00000000 00:00 0 
61904c4c0000-640000000000 ---p 00000000 00:00 0 
640000000000-640000003000 rw-p 00000000 00:00 0 
7f434fff7000-7f434fffb000 rw-p 00000000 00:00 0 
...
Note: only 4 mappings that start with 6.

I bet that on your OS this test will fail under asan; please check.

I don't know what makes Linux behave this way, we haven't seen it before.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

Kostya Serebryany <kcc at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |eugeni.stepanov at gmail dot com

--- Comment #18 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
eugenis@ says he sees something similar on Ubuntu 13.10.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #19 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to Kostya Serebryany from comment #18)
> eugenis@ says he sees something similar on Ubuntu 13.10.

Actually, only on Ubuntu 14.04 candidate. 
Looks like a fresh regression in Kernel.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #20 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> ---
I can confirm that the issue can be reproduced on 3.13.0-5-generic (ubuntu
14.04 candidate), and can not be reproduced on 3.11.0-15-generic (ubuntu
13.10).

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #21 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> ---
A reproducer without ASan:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
int main() {
  char * p = (char*)0x61904c1e0000;
  for (int i = 0; i < 100; ++i)
    mmap(p + i * 4096, 4096, PROT_WRITE | PROT_READ, MAP_PRIVATE |
MAP_ANONYMOUS, 0, 0);
  printf("PID %d\n", getpid());
  sleep(100);
}

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #22 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
A quick workaround would be to change 
  static const uptr kUserMapSize = 1 << 16;
in sanitizer_common/sanitizer_allocator.h
to something like 17 or 18.
We can also try using mremap. 

However I'd like to hear from kernel experts whether this is considered 
a kernel bug or a feature.

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

--- Comment #23 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
Relevant Kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=67651

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

Kostya Serebryany <kcc at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |kcc at gcc dot gnu.org
   Target Milestone|4.9.0                       |---

--- Comment #24 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
I increased the mmap granularity in sanitizer's allocator to 
allow our bot on ubuntu pre-14.04 to pass.
http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/sanitizer_common/sanitizer_allocator.h?r1=200197&r2=200196&pathrev=200197
Feel free to apply the same change to GCC trunk. 

Once the kernel bug is fixed and the fix reaches ubuntu 14.04 tree 
we will likely revert this change. 
There is nothing else we can do on sanitizer side -- the kernel bug 
is significant (e.g. it makes gimp crash) and so such system is unusable
for many other purposes. 

Let's keep this bug open so that we don't forget to revert the workaround.

FTR, here is a program that detects the presence of a broken kernel:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <assert.h>
int main() {
  char *p = (char*)0x600000000000;
  size_t i;
  for (i = 0; i < 100000; i++) {
    void *addr = p + i * 4096;
    void *ret = mmap(addr, 4096, PROT_WRITE | PROT_READ,
                     MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
    if (addr != ret) {
      fprintf(stderr, "failed at iteration %zd\n", i);
      char command[100];
      snprintf(command, sizeof(command), "cat /proc/%d/maps | head -30",
getpid());
      system(command);
      return 1;
    }
  }
}


On a broken kernel it will print something like this:

failed at iteration 65514
00400000-00401000 r-xp 00000000 fc:00 20991316  /tmp/a.out
00600000-00601000 r--p 00000000 fc:00 20991316  /tmp/a.out
00601000-00602000 rw-p 00001000 fc:00 20991316  /tmp/a.out
600000000000-600000001000 rw-p 00000000 00:00 0 
600000001000-600000002000 rw-p 00000000 00:00 0 
600000002000-600000003000 rw-p 00000000 00:00 0 
600000003000-600000004000 rw-p 00000000 00:00 0 
600000004000-600000005000 rw-p 00000000 00:00 0 
...

[Bug sanitizer/59733] [4.9 Regression] bootstrap-asan failed to bootstrap

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

Kostya Serebryany <kcc at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |WONTFIX

--- Comment #25 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
Even with this workaround asan is almost unusable with the buggy kernel.
I suspect that when a process has too many mappings it gets too slow.
The machine where we have our clang-asan bootstrap gets very slow and
unreliable.

We have reverted the kernel on that bot 
(http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/)
and reverted the above workaround in llvm trunk.
http://llvm.org/viewvc/llvm-project?rev=200217&view=rev

The kernel fix went into Linus's tree for 3.14 last week,
and was tagged for stable backport. Nothing else we could do here.