[Bug sanitizer/56393] New: SIGSEGV when -fsanitize=address and dynamic lib with global objects

[Bug sanitizer/56393] New: SIGSEGV when -fsanitize=address and dynamic lib with global objects

[t-gcc-bugzilla at snowelm dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

             Bug #: 56393
           Summary: SIGSEGV when -fsanitize=address and dynamic lib with
                    global objects
    Classification: Unclassified
           Product: gcc
           Version: 4.8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: t-gcc-bugzilla@snowelm.com
                CC: dodji@gcc.gnu.org, dvyukov@gcc.gnu.org,
                    jakub@gcc.gnu.org, kcc@gcc.gnu.org


Created attachment 29493
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29493
gdb session log

When linked with a dynamic library with a globally initialized class object, a
compiled code with address sanitizer dies with SIGSEGV.  I think this is a 
typical usecase...

Test case: 
(boost 1.53 was installed with --build-type=complete.
 tested by libs compiled with g++ 4.7.2 as well as libs compiled with head gcc)
---------------------------------
#include <boost/thread/shared_mutex.hpp>
#include <cstdio>
int main() {
    printf("hello\n");
    boost::upgrade_mutex m;
}
----------------------------------

$ g++ -g -fsanitize=address -I/usr/local/include hoge2.cpp -o a.out
-lboost_thread -lboost_system
$ ./a.out
Segmentation fault (core dumped)

The segfault is before the printf.
Without -fsanitize=address it runs normally.  If the program is linked with
static libs (-lboost_thread-mt-s -lboost_system-mt-s), it runs normally.
Note that the libraries are compiled without -fsanitize=address.

I investigated the situation by gdb (with -lboost_thread-mt-d
-lboost_system-mt-d). Please see the attached log.
For me it seems that some initializer in the shared lib kicks sanitized code of
some object constructor before the sanitizer gets ready.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #1 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-19 13:49:34 UTC ---
what happens if you link asan statically (-static-libasan)?

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[t-gcc-bugzilla at snowelm dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #2 from Takaki Makino <t-gcc-bugzilla at snowelm dot com> 2013-02-19 14:02:13 UTC ---
Thanks Kostya.

Unfortunately -static-libasan didn't help:

$ g++ -g -fsanitize=address -I/usr/local/include hoge2.cpp -o a.out
-lboost_thread-mt-d -lboost_system-mt-d -static-libasan
$ ./a.out
Segmentation fault (core dumped

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #3 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-19 14:11:32 UTC ---
Looks like __asan_init is not called early enough. 
To confirm, run 
  ASAN_OPTIONS=verbosity=1 ./a.out
if __asan_init is called it should print something like this: 

==9255== Parsed ASAN_OPTIONS: verbosity=1
==9255== AddressSanitizer: libc interceptors initialized
|| `[0x200000000000, 0x7fffffffffff]` || HighMem    ||
...

If you don't see this, the crash happens before __asan_init. 

Does this happen with clang? 

(Or, can you give the exact instructions on how to install boost 1.53?)

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[t-gcc-bugzilla at snowelm dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #4 from Takaki Makino <t-gcc-bugzilla at snowelm dot com> 2013-02-19 15:22:38 UTC ---
Created attachment 29499
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29499
reduced testcase

I tried to made a fairly reduced testcase from boost.
Attached please find the tarball.

make CXX=/your/head/gcc

two programs are built and run.
The statically linked one, a.out.static, runs happily, 
while a.out.dynamic, the dynamically linked one, fails with SIGSEGV.
Tested on linux x86_64.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[t-gcc-bugzilla at snowelm dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #5 from Takaki Makino <t-gcc-bugzilla at snowelm dot com> 2013-02-19 15:26:41 UTC ---
To reply Kostya's comment,
As shown in the reduced testcase, the program crashes before 
ASAN_OPTIONS=verbosity=1 takes effect.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

Kostya Serebryany <kcc at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2013-02-19
     Ever Confirmed|0                           |1

--- Comment #6 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-19 18:33:32 UTC ---
The problem also shows up with clang. 
Funny. C++ global init has greater priority than C global init. 

This is the usual place from where __asan_init is called: 

gcc: 
#0  __asan_init_v1 () at ../../../../gcc/libsanitizer/asan/asan_rtl.cc:315
#1  0x0000000000417819 in _GLOBAL__sub_I_00099_1_g ()
#2  0x000000000041788d in __libc_csu_init ()
#3  0x00007ffff761b700 in __libc_start_main 
#4  0x0000000000401ec9 in _start ()

clang: 
#0  __asan_init_v1 () 
#1  0x0000000000419dcd in asan.module_ctor ()
#2  0x0000000000419e8d in __libc_csu_init ()
#3  0x00007ffff761b700 in __libc_start_main 
#4  0x0000000000419b15 in _start ()

This is where your test case crashes: 
#0  0x000000000041a864 in boost::exception_ptr
boost::get_static_exception_object<boost::bad_alloc_>() ()
#1  0x00007ffff7ff62c9 in __cxx_global_var_init () from ./libhoge.so
#2  0x00007ffff7ff62f9 in global constructors keyed to a () from ./libhoge.so
#3  0x00007ffff7de9306 in call_init 
#4  0x00007ffff7de93df in call_init 
#5  _dl_init 
#6  0x00007ffff7ddb6ea in _dl_start_user

Apparently, _dl_start_user starts before _start. 

In fact, I think that we saw something like this before and this is why 
in one of our settings we run __asan_init from the preinit array. I.e. from
here: 

#0  __asan_init_v1 () at 
#1  0x00007ffff7de943e in _dl_init
#2  0x00007ffff7ddb6ea in _dl_start_user () from /lib64/ld-linux-x86-64.so.2

Looks like it's time to enable the preinit hack by default on linux, it's
pretty stable now.
Thoughts? 

You may temporary change "defined(ASAN_USE_PREINIT_ARRAY)" to '1' in
asan_rtl.cc,
or add something like this into your code: 
  __attribute__((section(".preinit_array")))
    typeof(__asan_init) *__asan_preinit =__asan_init;

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

Evgeniy Stepanov <eugeni.stepanov at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |eugeni.stepanov at gmail
                   |                            |dot com

--- Comment #7 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> 2013-02-19 20:35:12 UTC ---
You've got a constructor of an uninstrumented shared library calling back into
the application code. Of course, it happens before constructors of the main
executable run. This breaks ASan initialization order.

boost::get_static_exception_object<boost::bad_alloc_> is compiled in both
hoge1.o and hoge2.o, both as weak symbols, one of them is ASan-instrumented.
The call from a shared library constructor is resolved to the instrumented
version from the main executable.

This shows that bad things can happen implicitly, in very harmlessly looking
code. We probably should not rely on normal constructors and switch
.preinit_array. The only thing that is stronger than .preinit_array is
DF_1_INITFIRST, we can't use it with static libasan, unfortunately.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #8 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-20 14:30:16 UTC ---
With http://llvm.org/viewvc/llvm-project?rev=175623&view=rev
__asan_init is called from preinit_array, thus fixing this problem.

Will try to merge to gcc in a few days.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> 2013-02-20 14:43:02 UTC ---
.preinit_array is only processed in executables, so this only affects
-static-libasan, not linking it as shared library, and only when the program is
linked with -fsanitize=address.  And, -Wl,-z,initfirst can't be really used on
Linux, because glibc dynamic linker only supports exactly one initfirst shared
library, which it assumes is libpthread.so.0.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #10 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-20 14:49:20 UTC ---
(In reply to comment #9)
> .preinit_array is only processed in executables, so this only affects
> -static-libasan, 

Right. This is the only mode officially supported by clang on linux anyway.

> not linking it as shared library, and only when the program is
> linked with -fsanitize=address.  And, -Wl,-z,initfirst can't be really used on
> Linux, because glibc dynamic linker only supports exactly one initfirst shared
> library, which it assumes is libpthread.so.0.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #11 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> 2013-02-20 14:57:02 UTC ---
Yes, dynamic libasan is still busted.
We could link a small .o into executables built with dynamic libasan; that .o
would contain a .preinit record pointing to the dynamically-linked __asan_init.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #12 from Jakub Jelinek <jakub at gcc dot gnu.org> 2013-02-20 15:02:20 UTC ---
Yeah, we can do that.  But it will still only cover the case where executable
is linked with -fsanitize=address.  If you only sanitize some shared library
and use it by non-sanitized executable, it could still break, though perhaps
not as often as in this case.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #13 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> 2013-02-20 18:29:57 UTC ---
I don't see what can be done in that case. We should just declare it
unsupported.

We've got this problem on Android, where an instrumented JNI library is loaded
into Dalvik VM, which is outside of user control. We "solve" it by requiring
that the runtime library is LD_PRELOAD-ed into the DVM (Android has a mechanism
to do this on an individual app basis on rooted devices).

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[amonakov at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

Alexander Monakov <amonakov at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |amonakov at gcc dot gnu.org

--- Comment #14 from Alexander Monakov <amonakov at gcc dot gnu.org> 2013-02-21 10:54:13 UTC ---
(In reply to comment #13)
> We've got this problem on Android, where an instrumented JNI library is loaded
> into Dalvik VM, which is outside of user control. We "solve" it by requiring
> that the runtime library is LD_PRELOAD-ed into the DVM (Android has a mechanism
> to do this on an individual app basis on rooted devices).

OT, but what is this mechanism you speak of?  Currently this bug is the top
google hit for "Dalvik sanitizer LD_PRELOAD", and I don't see how it might work
if the VM only forks, not execs.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #15 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-21 11:35:51 UTC ---
r196201 landed the fresh asan run-time into gcc.
-static-libasan should work well now, please try.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #16 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> 2013-02-21 11:51:17 UTC ---
(In reply to comment #14)
> (In reply to comment #13)
> > We've got this problem on Android, where an instrumented JNI library is loaded
> > into Dalvik VM, which is outside of user control. We "solve" it by requiring
> > that the runtime library is LD_PRELOAD-ed into the DVM (Android has a mechanism
> > to do this on an individual app basis on rooted devices).
> 
> OT, but what is this mechanism you speak of?  Currently this bug is the top
> google hit for "Dalvik sanitizer LD_PRELOAD", and I don't see how it might work
> if the VM only forks, not execs.

https://android.googlesource.com/platform/frameworks/base/+/master/core/java/com/android/internal/os/ZygoteConnection.java

See the code for applyInvokeWithSystemProperty().

Also, https://code.google.com/p/address-sanitizer/wiki/Android.
Sorry, this page was outdated until just now.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[t-gcc-bugzilla at snowelm dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #17 from Takaki Makino <t-gcc-bugzilla at snowelm dot com> 2013-02-22 06:38:33 UTC ---
(In reply to comment #15)
> r196201 landed the fresh asan run-time into gcc.
> -static-libasan should work well now, please try.

It works for me, thank you very much for your efforts.
Why don't you set -static-libasan as default?
Because I still don't understand in which case dynamic libasan is useful...

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #18 from Andrew Pinski <pinskia at gcc dot gnu.org> 2013-02-22 06:55:59 UTC ---
(In reply to comment #17)
> It works for me, thank you very much for your efforts.
> Why don't you set -static-libasan as default?
> Because I still don't understand in which case dynamic libasan is useful...

It is more useful than having a static only version of libasan really.  I think
the asan guys are wrong in requiring static only stuff.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #19 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-02-22 07:01:51 UTC ---
In clang, static asan-rt is the default and the only option on Linux. 
(On Mac, the only option is dynamic, but that's a quite unfortunate limitation 
of the platform, it costs us quite a bit of pain during deployment).
I would also prefer to have static as the default on gcc-linux (for both asan
and tsan). 

In regular cases, when you build your own program static is what you need.

There is one case where we do need dynamic asan-rt and we build it separately:
C++ code called via SWIG from a python program. 
We already have the pre-built python binary and can not link anything to it
statically. 
But the deployment is rather painful here too, it would have been much simpler
if we could link asan-rt to python statically.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[t-gcc-bugzilla at snowelm dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #20 from Takaki Makino <t-gcc-bugzilla at snowelm dot com> 2013-02-22 09:01:30 UTC ---
I understand why dynamic libasan is important.
Still it seems for me -static-libasan can be default, except when -shared is
given.
(just because I have no idea how the shared case could be solved...)

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #21 from Jakub Jelinek <jakub at gcc dot gnu.org> 2013-02-22 16:07:46 UTC ---
Author: jakub
Date: Fri Feb 22 16:07:36 2013
New Revision: 196222

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=196222
Log:
    PR sanitizer/56393
    * config/gnu-user.h (LIBASAN_EARLY_SPEC): Link in libasan_preinit.o
    if not linking a shared library.

    * lib/asan-dg.exp (asan_link_flags): Add
    -B${gccpath}/libsanitizer/asan/ to flags.

    * asan/Makefile.am (nodist_toolexeclib_HEADERS): Set to
    libasan_preinit.o.
    (libasan_preinit.o): Depend on asan_preinit.o.
    * asan/Makefile.in: Regenerated.
    * asan/asan_preinit.cc: New file, synced from upstream.
    * asan/asan_rtl.cc: Remove preinit stuff, synced from upstream.

Added:
    trunk/libsanitizer/asan/asan_preinit.cc
Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/config/gnu-user.h
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/lib/asan-dg.exp
    trunk/libsanitizer/ChangeLog
    trunk/libsanitizer/asan/Makefile.am
    trunk/libsanitizer/asan/Makefile.in
    trunk/libsanitizer/asan/asan_rtl.cc

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[david.abdurachmanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

David Abdurachmanov <david.abdurachmanov at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |david.abdurachmanov at
                   |                            |gmail dot com

--- Comment #22 from David Abdurachmanov <david.abdurachmanov at gmail dot com> 2013-04-07 08:44:21 UTC ---
Has this been resolved in the final 4.8.0 (r196952)? I checked some changes and
they seems to be in.

I have a number (<100) C++/C packages (incl. boost 1.51.00) compiled w/o
address sanitizer and I am only enabling it for the main software using all
these packages.  Yet compilation fails of the main software segflaut from
boost. Reminder, that boost and ROOT is not compiled w/ address sanitizer.

I tried -static-libasan, yet in that case linker cannot resolve asan symbols
while shared library is being created. libasan_preinit.o, libasan.so, and
libasan.a is in my GCC package under ./lib64.

I added the following options to default CXXFLAGS, which also ended up on
LDFLAGS: -static-libasan -fsanitize=address -fno-omit-frame-pointer -g -O0

### SEGFAULT ###

The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00002b33b9930d8a in
boost::exception_detail::get_static_exception_object<boost::exception_detail::bad_alloc_>
() at /build/davidlt/test-asan/a/slc6_amd64_gcc480/external/boost
/1.51.0-cms2/include/boost/exception/detail/exception_ptr.hpp:117
#6  0x00002b33ba70685c in _GLOBAL__sub_I_future.cpp () from
/build/davidlt/test-asan/a/tmp/BUILDROOT/2c73b4475e8345752c405e046bb5182f/opt/cmssw/slc6_amd64_gcc480/cms/cmssw/CMSSW_6_2
_X_2013-04-06-0200/external/slc6_amd64_gcc480/lib/libboost_thread.so.1.51.0
#7  0x0000003326c0e57f in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#8  0x0000003326c12c25 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#9  0x0000003326c0e196 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x0000003326c1246a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#11 0x0000003327400f66 in dlopen_doit () from /lib64/libdl.so.2
#12 0x0000003326c0e196 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#13 0x000000332740129c in _dlerror_run () from /lib64/libdl.so.2
#14 0x0000003327400ee1 in dlopen

### UNDEFINED REFERENCES ###

/build/davidlt/test-asan/a/tmp/BUILDROOT/eea1b2b6d889c55f315fbba1ee425ca6/opt/cmssw/slc6_amd64_gcc480/cms/cmssw/CMSSW_6_2_X_2013-04-06-0200-cms/src/FWCore/Version/src/GetFileFormatVersion.cc:5:
error: undefined reference to
'__asan_init_v1'/build/davidlt/test-asan/a/tmp/BUILDROOT/eea1b2b6d889c55f315fbba1ee425ca6/opt/cmssw/slc6_amd64_gcc480/cms/cmssw/CMSSW_6_2_X_2013-04-06-0200-cms/src/FWCore/Version/src/GetReleaseVersion.cc:8:
error: undefined reference to
'__asan_report_load1'/build/davidlt/test-asan/a/tmp/BUILDROOT/eea1b2b6d889c55f315fbba1ee425ca6/opt/cmssw/slc6_amd64_gcc480/cms/cmssw/CMSSW_6_2_X_2013-04-06-0200-cms/src/FWCore/Version/src/GetReleaseVersion.cc:11:
error: undefined reference to
'__asan_unregister_globals'/build/davidlt/test-asan/a/tmp/BUILDROOT/eea1b2b6d889c55f315fbba1ee425ca6/opt/cmssw/slc6_amd64_gcc480/cms/cmssw/CMSSW_6_2_X_2013-04-06-0200-cms/src/FWCore/Version/src/GetReleaseVersion.cc:11:
error: undefined reference to
'__asan_init_v1'/build/davidlt/test-asan/a/tmp/BUILDROOT/eea1b2b6d889c55f315fbba1ee425ca6/opt/cmssw/slc6_amd64_gcc480/cms/cmssw/CMSSW_6_2_X_2013-04-06-0200-cms/src/FWCore/Version/src/GetReleaseVersion.cc:11:
error: undefined reference to '__asan_register_globals'

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #23 from Kostya Serebryany <kcc at gcc dot gnu.org> 2013-04-08 06:32:45 UTC ---
(In reply to comment #22)
> Has this been resolved in the final 4.8.0 (r196952)? I checked some changes and
> they seems to be in.
> 
> I have a number (<100) C++/C packages (incl. boost 1.51.00) compiled w/o
> address sanitizer and I am only enabling it for the main software using all
> these packages.  Yet compilation fails of the main software segflaut from

David, please provide more info on the segfault. 
E.g. run with env. var. ASAN_OPTIONS=verbosity=1 and also 
disassemble the failing instruction in gdb. 
I suspect that the non-instrumented code runs before __asan_init_v1 and then
calls instrumented code. This will never work. 

You need to make sure that __asan_init_v1 is called before any instrumented
code.
One way: put this function into the preinint array; this is automatically done
if you use -static-libasan; you can do it manually in your main module. 
Another way: instrument everything.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #24 from Jakub Jelinek <jakub at gcc dot gnu.org> 2013-04-08 06:56:30 UTC ---
"One way: put this function into the preinint array; this is automatically done
if you use -static-libasan;"

That is actually done whenever the binary is linked with -fsanitize=address, no
matter whether -static-libasan is used also or not.

Just
readelf -d binary | grep PREINIT
to verify.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[david.abdurachmanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #25 from David Abdurachmanov <david.abdurachmanov at gmail dot com> ---
I downloaded GCC 4.8.1 (final) and binutils 2.23.1 (default ld.gold). The
problem is that boost and a lot other C/C++ packages are not compiled w/ ASan.
One way would be to guarantee every single C/C++ RPM package correctly
compiling w/ ASan (quite some effort). I prefer to instrument the main software
(+ some critical C/C++ packages in future) for now.

I assume I cannot use -static-libasan for shared objects (-shared) as
PREINIT_ARRAY section would be a duplicate. Then -static-libasan should be only
used for the final executable binary? 

I tried something like that:

c++ -fsanitize=address -shared -Wl,-E -Wl,-z,defs -Wl,-v ./test.o -o libtest.so
-static-libasan

I see it doesn't pass to linker:

<long_path>/lib64/libasan_preinit.o -Bstatic --whole-archive -lasan
--no-whole-archive -Bdynamic

And linker dies w/ undefined references of ASan.

Do I need to make sure that **only** executable binaries gets ASan linked
statically (-static-libasan)? (Thus they always have PREINIT_ARRAY section
initializing ASan).

david

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #26 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to David Abdurachmanov from comment #25)
> I downloaded GCC 4.8.1 (final) and binutils 2.23.1 (default ld.gold). The
> problem is that boost and a lot other C/C++ packages are not compiled w/
> ASan. One way would be to guarantee every single C/C++ RPM package correctly
> compiling w/ ASan (quite some effort). 

This will be indeed quite some effort. But I still want this to happen! 

> I prefer to instrument the main
> software (+ some critical C/C++ packages in future) for now.
> 
> I assume I cannot use -static-libasan for shared objects (-shared) as
> PREINIT_ARRAY section would be a duplicate. Then -static-libasan should be
> only used for the final executable binary? 
> 
> I tried something like that:
> 
> c++ -fsanitize=address -shared -Wl,-E -Wl,-z,defs -Wl,-v ./test.o -o
> libtest.so -static-libasan
> 
> I see it doesn't pass to linker:
> 
> <long_path>/lib64/libasan_preinit.o -Bstatic --whole-archive -lasan
> --no-whole-archive -Bdynamic
> 
> And linker dies w/ undefined references of ASan.

The linker complains because you have -Wl,-z,defs
Just remove it. 

> 
> Do I need to make sure that **only** executable binaries gets ASan linked
> statically (-static-libasan)? (Thus they always have PREINIT_ARRAY section
> initializing ASan).
> 
> david

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

Yury Gribov <y.gribov at samsung dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |y.gribov at samsung dot com

--- Comment #27 from Yury Gribov <y.gribov at samsung dot com> ---
(In reply to Evgeniy Stepanov from comment #13)
> I don't see what can be done in that case. We should just declare it
> unsupported.

Evgeniy,

Do you think we can somehow detect this invalid usage at link/runtime and
inform user? Such invalid linkage tends to slip in in large SW projects with
have precompiled parts and causes complete mess at runtime.

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #28 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> ---
This will not just detect such situations, but fix them:

We could link a small .o into executables built with dynamic libasan; that .o
would contain a .preinit record pointing to the dynamically-linked __asan_init.

It would still break if main executable is linked without -fsanitize=address,
through. No idea how to detect that.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[pluto at agmk dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

Pawel Sikora <pluto at agmk dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pluto at agmk dot net

--- Comment #29 from Pawel Sikora <pluto at agmk dot net> ---
(In reply to Evgeniy Stepanov from comment #28)

> It would still break if main executable is linked without
> -fsanitize=address, through. No idea how to detect that.

you can check a weak symbol address from exec point of view.

$ cat something_small.cpp 
int something_small = 42;

$ cat main.cpp 
#include <cstdio>
int __attribute__((weak)) something_small;
int main()
{
        if ( something_small != 0 )
                std::printf( "something_small linked : %d\n", something_small
);
        else
                std::printf( "something_small missed\n" );
}

$ g++ something_small.cpp -Wall -c

$ g++ main.cpp -o main
./main
something_small missed

$ g++ main.cpp -o main something_small.o
./main
something_small linked : 42

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #30 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> ---
Yes, but that must run before library constructors => .preinit_array.
I'm not sure we want to add code like this to .preinit_array of all binaries
that are NOT linked with AddressSanitizer ;)

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[tetra2005 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

Yuri Gribov <tetra2005 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tetra2005 at gmail dot com

--- Comment #31 from Yuri Gribov <tetra2005 at gmail dot com> ---
(In reply to Evgeniy Stepanov from comment #28)
> It would still break if main executable is linked without
> -fsanitize=address, through. No idea how to detect that.

Yup, that's what I was talking about.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[tetra2005 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #32 from Yuri Gribov <tetra2005 at gmail dot com> ---
(In reply to Evgeniy Stepanov from comment #30)
> Yes, but that must run before library constructors => .preinit_array.
> I'm not sure we want to add code like this to .preinit_array of all binaries
> that are NOT linked with AddressSanitizer ;)

Speculating on Pawel's suggestion: what if we add a marker symbol to
asan_preinit.o and then check it's presence (with dlsym) at the beginning of
__asan_init?

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #33 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> ---
We could link a small .o to ASan executables that contains a .preinit_array
chunk and also defines a symbol that all ASan-ized libraries depend on.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[tetra2005 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #34 from Yuri Gribov <tetra2005 at gmail dot com> ---
(In reply to Evgeniy Stepanov from comment #33)
> We could link a small .o to ASan executables that contains a .preinit_array
> chunk and also defines a symbol that all ASan-ized libraries depend on.

Gcc already links asan_preinit.o (both for normal and -static-libasan builds)
so we could just reuse it.

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #35 from Yury Gribov <y.gribov at samsung dot com> ---
Created attachment 31018
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=31018&action=edit
Draft patch to verify that executable is sanitized

Here's a draft patch which does more or less what Evgeniy proposed. It will
cause libasan to fail to load if main executable is not sanitized.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #37 from Yury Gribov <y.gribov at samsung dot com> ---
I absolutely agree with Jakub that this patch may result in reduced
functionality i.e. some projects may start failing. What if we replace dynamic
linker error with warning in __asan_init? We could simply check that symbol is
present via dlsym().

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[eugeni.stepanov at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #38 from Evgeniy Stepanov <eugeni.stepanov at gmail dot com> ---
Aren't we discussing the crash that happens when shadow is accessed _before_
__asan_init?

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #39 from Yury Gribov <y.gribov at samsung dot com> ---
True. My problems manifest themselves later in initialization process so I
forgot about OPs problem. It seems that my last suggestion would not be able to
catch all the instances of this error indeed.

The original proposal (which results in link error) should be able to rule out
all problems at the cost of reduced functionality. Personally I think that
safety and defined behavior are more important...

Thoughts?

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #41 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
(In reply to Yury Gribov from comment #40)
> >> Jakub Jelinek wrote:
> >> Previously ASAN would often work
> >> even when binary wasn't linked with -fsanitize=address,
> >> though sometimes it wouldn't
> >
> > Yury Gribov wrote:
> > this patch may result in reduced functionality
> > i.e. some projects may start failing.
> 
> Actually do we have an example of unsanitized binary working with sanitized
> dynamic lib? Currently I see all REAL() macro resolving to NULL in this case
> (and then causing segfaults in __asan_init).
> 
> -Y

It will work with LD_PRELOAD=libasan.so.0.  In any case, the Linux
__interception::GetRealFunctionAddress should be probably improved, it blindly
assumes that the libc/libpthread version of the functions must come after the
libasan ones in the search scope, while it can come before them.  IMHO if dlsym
(RTLD_NEXT, ...) fails, then it should retry with dlopen ("libc.so.6",
RTLD_NOLOAD); and dlsym on the result of that if it didn't fail (resp.
libpthread.so.0 for pthread_* of course).

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #42 from Yury Gribov <y.gribov at samsung dot com> ---
(In reply to Jakub Jelinek from comment #41)
> It will work with LD_PRELOAD=libasan.so.0.

I was under the same impression. Unfortunately this seems to fail if executable
takes address of libc API (that's not an infrequent situation). I can't outline
all the details right now but this results with asan's dlsym returning address
of .plt entry in executable rather than libc symbol and .plt itself resolving
to asan wrapper.

I agree that there are some situations where this works but semantics becomes
too tricky (IMHO of course).

> IMHO if dlsym (RTLD_NEXT, ...) fails, then it should retry with dlopen
> ("libc.so.6", RTLD_NOLOAD); and dlsym on the result of that if it didn't
> fail (resp. libpthread.so.0 for pthread_* of course).

Hm, this looks like a dangerous path. Should we assume that malloc should
necessarily come from libc? What if app itself overrides malloc? That's not to
mention that kcc is unlikely to make this change in upstream libsanitizer.

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #43 from Yury Gribov <y.gribov at samsung dot com> ---
(In reply to Yury Gribov from comment #42)
> asan's dlsym
> returning address of .plt entry in executable rather than libc symbol and
> .plt itself resolving to asan wrapper.

Causing segfault later.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #44 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
(In reply to Yury Gribov from comment #42)
> (In reply to Jakub Jelinek from comment #41)
> > It will work with LD_PRELOAD=libasan.so.0.
> 
> I was under the same impression. Unfortunately this seems to fail if
> executable takes address of libc API (that's not an infrequent situation). I
> can't outline all the details right now but this results with asan's dlsym
> returning address of .plt entry in executable rather than libc symbol and
> .plt itself resolving to asan wrapper.

How could dlsym (RTLD_NEXT, ...) from a shared library return address of .plt
in the executable?  The executable is always first in the search scope, and
RTLD_NEXT means start looking from the current shared library (not including
it) onwards.

> > IMHO if dlsym (RTLD_NEXT, ...) fails, then it should retry with dlopen
> > ("libc.so.6", RTLD_NOLOAD); and dlsym on the result of that if it didn't
> > fail (resp. libpthread.so.0 for pthread_* of course).
> 
> Hm, this looks like a dangerous path. Should we assume that malloc should
> necessarily come from libc? What if app itself overrides malloc? That's not
> to mention that kcc is unlikely to make this change in upstream libsanitizer.

Why?  If dlsym returns NULL, then libasan is last in the search scope for those
symbols, so even if malloc etc. is overridden, the overriding comes earlier in
the search scope.  Of course, without binary being linked with -lasan and
without LD_PRELOAD=libasan.so.0, the checking features of libasan can be very
limited, the main point is if you have some library you built with asan for
some reason and need to run some other binary against that library, ideally it
would just work, even if it doesn't find all the issues in the shared library.

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #45 from Yury Gribov <y.gribov at samsung dot com> ---
(In reply to Jakub Jelinek from comment #44)
> How could dlsym (RTLD_NEXT, ...) from a shared library return address of
> .plt in the executable?

Ok, I'm an idiot. The problem I faced was caused by global `export
LD_PRELOAD=libasan.so.0' in my /etc/profile. It turns out that if we preload
asan for executable which does not load any sanitized lib, __asan_init_v1 will
not get called and thus wrappers will not be properly initialized:

 $ gdb a.out
 Reading symbols from /mnt/scratch/ygribov/vdtools/tests/signal/a.out...done.
 (gdb) set env LD_PRELOAD /home/ygribov/install/gcc-master/lib64/libasan.so.0
 (gdb) b __asan_init_v1
 Function "__asan_init_v1" not defined.
 Make breakpoint pending on future shared library load? (y or [n]) y
 Breakpoint 1 (__asan_init_v1) pending.
 (gdb) run
 ...
 Program received signal SIGSEGV, Segmentation fault.
 0x0000000000000000 in ?? ()
 (gdb) bt
 #0  0x0000000000000000 in ?? ()
 #1  0x00007ffff4e4fa62 in signal () at
/home/ygribov/gcc/gcc-master/libsanitizer/asan/asan_interceptors.cc:133
 #2  0x0000000000400523 in main ()

Should probably be easy to fix by adding __asan_init_v1 to a list of libasan's
initializers. Persuading kcc to do this will be tricky though.

> > > IMHO if dlsym (RTLD_NEXT, ...) fails, then it should retry with dlopen
> > > ("libc.so.6", RTLD_NOLOAD); and dlsym on the result of that if it didn't
> > > fail (resp. libpthread.so.0 for pthread_* of course).
> > 
> > Hm, this looks like a dangerous path. Should we assume that malloc should
> > necessarily come from libc? What if app itself overrides malloc? That's not
> > to mention that kcc is unlikely to make this change in upstream libsanitizer.
> 
> Why?  If dlsym returns NULL, then libasan is last in the search scope for
> those symbols, so even if malloc etc. is overridden, the overriding comes
> earlier in the search scope.

Agreed but this overriding does not necessarily come from libc. What if
application or some intermediate shared lib overloads malloc? Mixing overloaded
and libc's allocators may easily cause havoc if allocated pointers are passed
across shared lib boundaries.

> Of course, without binary being linked with
> -lasan and without LD_PRELOAD=libasan.so.0,

Small correction: IMHO there is very little chance that unsanitized executable
is going to work without LD_PRELOAD (because dlsym calls will return NULLs).

> the checking features of libasan
> can be very limited, the main point is if you have some library you built
> with asan for some reason and need to run some other binary against that
> library, ideally it would just work, even if it doesn't find all the issues
> in the shared library.

Ok, I see your point. I wish we had some way to provide useful diagnostic
messages though. Current situation (mysterious segfaults in initialization
code) is too inconvenient for ordinary user.

-Y

[Bug sanitizer/56393] SIGSEGV when -fsanitize=address and dynamic lib with global objects

[y.gribov at samsung dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56393

--- Comment #46 from Yury Gribov <y.gribov at samsung dot com> ---
Can we close this one? Does not seem to repro in trunk.