[Bug sanitizer/113430] New: Trivial program segfaults intermittently with ASAN since Linux 6.7

[Bug sanitizer/113430] New: Trivial program segfaults intermittently with ASAN since Linux 6.7

[tavianator at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

            Bug ID: 113430
           Summary: Trivial program segfaults intermittently with ASAN
                    since Linux 6.7
           Product: gcc
           Version: 13.2.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: tavianator at gmail dot com
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org
  Target Milestone: ---

Since updating to Linux 6.7, I'm getting intermittent segfaults with ASAN and
ASLR enabled.

$ cat foo.c
int main(void) {
        return 0;
}
$ gcc -fsanitize=address foo.c -o foo
$ while ./foo; do :; done
AddressSanitizer:DEADLYSIGNAL
=================================================================
==337494==ERROR: AddressSanitizer: SEGV on unknown address 0x636c68879e78 (pc
0x7dde493b538f bp 0x000000000000 sp 0x7ffc78949970 T0)
==337494==The signal is caused by a READ memory access.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.
tavianator@graphene $ gcc --version
gcc (GCC) 13.2.1 20230801
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ uname -a
Linux graphene 6.7.0-arch3-1 #1 SMP PREEMPT_DYNAMIC Sat, 13 Jan 2024 14:37:14
+0000 x86_64 GNU/Linux

Here's the backtrace:

(gdb) set disable-randomization off
(gdb) run
Starting program: /home/tavianator/code/bfs/foo 
[Thread debugging using libthread_db enabled]                                   
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
do_lookup_x (undef_name=undef_name@entry=0x761941b3e6d8
"_thread_db_sizeof_pthread", new_hash=new_hash@entry=3872132951,
old_hash=old_hash@entry=0x7ffff16f0cc8, ref=0x0, 
    result=result@entry=0x7ffff16f0cd0, scope=<optimized out>, i=0,
version=0x0, flags=3, skip=<optimized out>, type_class=0, undef_map=<optimized
out>) at dl-lookup.c:405
405           const ElfW(Sym) *symtab = (const void *) D_PTR (map,
l_info[DT_SYMTAB]);                                                             
(gdb) bt
#0  do_lookup_x (undef_name=undef_name@entry=0x761941b3e6d8
"_thread_db_sizeof_pthread", new_hash=new_hash@entry=3872132951,
old_hash=old_hash@entry=0x7ffff16f0cc8, ref=0x0, 
    result=result@entry=0x7ffff16f0cd0, scope=<optimized out>, i=0,
version=0x0, flags=3, skip=<optimized out>, type_class=0, undef_map=<optimized
out>) at dl-lookup.c:405
#1  0x00007619421e20b8 in _dl_lookup_symbol_x (undef_name=0x761941b3e6d8
"_thread_db_sizeof_pthread", undef_map=<optimized out>, ref=0x7ffff16f0d58,
symbol_scope=<optimized out>, 
    version=0x0, type_class=0, flags=3, skip_map=0x0) at dl-lookup.c:793
#2  0x000076194197300e in do_sym (handle=<optimized out>, name=0x761941b3e6d8
"_thread_db_sizeof_pthread", who=0x761941afffb3
<__sanitizer::ThreadDescriptorSize()+35>, 
    vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:146
#3  0x0000761941973331 in _dl_sym (handle=<optimized out>, name=<optimized
out>, who=<optimized out>) at dl-sym.c:195
#4  0x00007619418a6ae8 in dlsym_doit (a=a@entry=0x7ffff16f0fc0) at dlsym.c:40
#5  0x00007619421d94e1 in __GI__dl_catch_exception
(exception=exception@entry=0x7ffff16f0f20, operate=0x7619418a6ad0 <dlsym_doit>,
args=0x7ffff16f0fc0) at dl-catch.c:237
#6  0x00007619421d9603 in _dl_catch_error (objname=0x7ffff16f0f78,
errstring=0x7ffff16f0f80, mallocedp=0x7ffff16f0f77, operate=<optimized out>,
args=<optimized out>) at dl-catch.c:256
#7  0x00007619418a64f7 in _dlerror_run (operate=operate@entry=0x7619418a6ad0
<dlsym_doit>, args=args@entry=0x7ffff16f0fc0) at dlerror.c:138
#8  0x00007619418a6b75 in dlsym_implementation (dl_caller=<optimized out>,
name=<optimized out>, handle=<optimized out>) at dlsym.c:54
#9  ___dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:68
#10 0x0000761941afffb3 in __sanitizer::ThreadDescriptorSize () at
/usr/src/debug/gcc/gcc/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:298
#11 0x0000761941b017ae in __sanitizer::ThreadDescriptorSize () at
/usr/src/debug/gcc/gcc/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:294
#12 __sanitizer::GetTls (size=0x7ffff16f1098, addr=0x7619421b0040) at
/usr/src/debug/gcc/gcc/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:498
#13 __sanitizer::GetThreadStackAndTls (main=true,
stk_addr=stk_addr@entry=0x7619421b0020, stk_size=stk_size@entry=0x7ffff16f10a0,
tls_addr=tls_addr@entry=0x7619421b0040, 
    tls_size=tls_size@entry=0x7ffff16f1098) at
/usr/src/debug/gcc/gcc/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:595
#14 0x0000761941af0ff4 in __asan::AsanThread::SetThreadStackAndTls
(this=this@entry=0x7619421b0000, options=<optimized out>) at
/usr/src/debug/gcc/gcc/libsanitizer/asan/asan_thread.h:77
#15 0x0000761941af14ee in __asan::AsanThread::Init
(this=this@entry=0x7619421b0000, options=options@entry=0x0) at
/usr/src/debug/gcc/gcc/libsanitizer/asan/asan_thread.cpp:234
#16 0x0000761941af19e5 in __asan::AsanThread::ThreadStart
(this=this@entry=0x7619421b0000, os_id=338380) at
/usr/src/debug/gcc/gcc/libsanitizer/asan/asan_thread.cpp:264
#17 0x0000761941af2604 in __asan::CreateMainThread () at
/usr/src/debug/gcc/gcc/libsanitizer/asan/asan_thread.cpp:295
#18 0x0000761941aee9df in __asan::AsanInitInternal () at
/usr/src/debug/gcc/gcc/libsanitizer/asan/asan_rtl.cpp:480
#19 0x00007619421dd02a in _dl_init (main_map=0x76194220c2d0, argc=1,
argv=0x7ffff16f11a8, env=0x7ffff16f11b8) at dl-init.c:122
#20 0x00007619421f32d0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#21 0x0000000000000001 in ?? ()
#22 0x00007ffff16f1e1a in ?? ()
#23 0x0000000000000000 in ?? ()

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN since Linux 6.7

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #1 from Sam James <sjames at gcc dot gnu.org> ---
If you could find the time to bisect the kernel (perhaps in a VM), that may
well be helpful.

Would also be interesting to know if Clang suffers from the same thing (given
we import libsanitizer from them).

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN since Linux 6.7

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #2 from Sam James <sjames at gcc dot gnu.org> ---
Ah, I see you mentioned the recent ASLR kerfuffle in
https://github.com/llvm/llvm-project/issues/78354#issuecomment-1894606165.

That config in some distros' kernel configs change was made for
https://lore.kernel.org/linux-mm/87il3ur1ik.fsf@gentoo.org/.

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[xry111 at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

Xi Ruoyao <xry111 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Trivial program segfaults   |Trivial program segfaults
                   |intermittently with ASAN    |intermittently with ASAN
                   |since Linux 6.7             |with large
                   |                            |CONFIG_ARCH_MMAP_RND_BITS
                   |                            |in kernel configuration
                 CC|                            |xry111 at gcc dot gnu.org

--- Comment #3 from Xi Ruoyao <xry111 at gcc dot gnu.org> ---
Updated the title to make it more precise.

Note that even with Linux 6.7 the default value of CONFIG_ARCH_MMAP_RND_BITS is
still 28 (32 is set by some distro maintainer who apparently does not know this
will hit the sanitizer runtime), so "since Linux 6.7" is just misleading.

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #4 from Sam James <sjames at gcc dot gnu.org> ---
(In reply to Xi Ruoyao from comment #3)

I didn't update it because I wasn't certain if it was the same thing, although
it seems very likely. But fair enough.

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[tavianator at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #5 from Tavian Barnes <tavianator at gmail dot com> ---
(In reply to Xi Ruoyao from comment #3)
> Updated the title to make it more precise.
> 
> Note that even with Linux 6.7 the default value of CONFIG_ARCH_MMAP_RND_BITS
> is still 28 (32 is set by some distro maintainer who apparently does not
> know this will hit the sanitizer runtime), so "since Linux 6.7" is just
> misleading.

Yep agreed.  I didn't expect such a patch from Arch, so I assumed it was a
change in the default kernel config.  For completeness, here's the Arch bug:
https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/20

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

Sam James <sjames at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://reviews.llvm.org/D1
                   |                            |48280,
                   |                            |https://reviews.llvm.org/D1
                   |                            |48193

--- Comment #6 from Sam James <sjames at gcc dot gnu.org> ---
https://github.com/google/sanitizers/issues/1614#issuecomment-1885369007 ->
https://reviews.llvm.org/D148280 and https://reviews.llvm.org/D148193.

So this is likely fixed for 14 already in the most recent sync from Jakub, and
might even be fixed in the sync before that?

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
r14-263-gd53b3d94aaf211ffb2159614f5aaaf03ceb861cc in particular

[Bug sanitizer/113430] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[dmjpp at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

Dimitrij Mijoski <dmjpp at hotmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmjpp at hotmail dot com

--- Comment #8 from Dimitrij Mijoski <dmjpp at hotmail dot com> ---
This bug manifested at large on Github Actions CI/CI system in the last few
days most likely because Ubuntu's kernel also got updated to use 32 random
bits. Here is the bug report
https://github.com/actions/runner-images/issues/9491 . It would be a good idea
to backport the fix.

[Bug sanitizer/113430] [12/13 only] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Trivial program segfaults   |[12/13 only] Trivial
                   |intermittently with ASAN    |program segfaults
                   |with large                  |intermittently with ASAN
                   |CONFIG_ARCH_MMAP_RND_BITS   |with large
                   |in kernel configuration     |CONFIG_ARCH_MMAP_RND_BITS
                   |                            |in kernel configuration
   Target Milestone|---                         |12.4

[Bug sanitizer/113430] [11/12/13 only] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #9 from Sam James <sjames at gcc dot gnu.org> ---
Created attachment 57708
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57708&action=edit
0001-libsanitizer-fix-ASAN-with-aggressive-CONFIG_ARCH_MM.patch

Untested patch for 13.

[Bug sanitizer/113430] [11/12/13 only] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #10 from Sam James <sjames at gcc dot gnu.org> ---
I don't plan on pursuing it myself, leaving it to someone else, as I can't
reproduce on my main workstation and I don't want to faff w/ kernel config.

[Bug sanitizer/113430] [11/12/13 only] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[dmjpp at hotmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #11 from Dimitrij Mijoski <dmjpp at hotmail dot com> ---
(In reply to Sam James from comment #10)
> I don't plan on pursuing it myself, leaving it to someone else, as I can't
> reproduce on my main workstation and I don't want to faff w/ kernel config.

You should be able to modify the kernel parameter at runtime by running:

sudo sysctl vm.mmap_rnd_bits=32

That should be enough to reproduce the issue. The fix is to cherry-pick the
changes to asan_allocator.h but also to lsan_allocator.h from this patch
r14-263-gd53b3d94aaf211ffb2159614f5aaaf03ceb861cc. You missed lsan_allocator.h
in your patch.

[Bug sanitizer/113430] [11/12/13 only] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[xry111 at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

--- Comment #12 from Xi Ruoyao <xry111 at gcc dot gnu.org> ---
(In reply to Dimitrij Mijoski from comment #8)
> This bug manifested at large on Github Actions CI/CI system in the last few
> days most likely because Ubuntu's kernel also got updated to use 32 random
> bits. Here is the bug report
> https://github.com/actions/runner-images/issues/9491 . It would be a good
> idea to backport the fix.

But then backporting the fix here won't really help because Ubuntu is not
building GCC from the upstream release branch.  Ubuntu maintainers could just
apply the patch downstream when they decide to increase random bits anyway, not
sure why they didn't.

[Bug sanitizer/113430] [11/12/13 only] Trivial program segfaults intermittently with ASAN with large CONFIG_ARCH_MMAP_RND_BITS in kernel configuration

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113430

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|12.4                        |12.5

--- Comment #13 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 12.4 is being released, retargeting bugs to GCC 12.5.