public inbox for gcc-bugs@sourceware.org
help / color / mirror / Atom feed
* [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
@ 2012-11-28 21:44 howarth at nitro dot med.uc.edu
  2012-11-28 21:48 ` [Bug sanitizer/55521] " howarth at nitro dot med.uc.edu
                   ` (18 more replies)
  0 siblings, 19 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-28 21:44 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

             Bug #: 55521
           Summary: many instances of ASAN:SIGSEGV failures in g++
                    testsuite with -fsanitize=address
    Classification: Unclassified
           Product: gcc
           Version: 4.8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: howarth@nitro.med.uc.edu
                CC: dodji@gcc.gnu.org, dvyukov@gcc.gnu.org,
                    jakub@gcc.gnu.org, kcc@gcc.gnu.org


Using...

make -k check-g++ RUNTESTFLAGS="--target_board=unix'{-fsanitize=address}'"

on x86_64-apple-darwin12 produces execution failures in g++ test suite of the
form....

Executing on host:
/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc/testsuite/g++/../../g++
-B/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc/testsuite/g++/../../
/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/gcc/testsuite/g++.dg/eh/cond1.C
 -fno-diagnostics-show-caret  -nostdinc++
-I/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/include/x86_64-apple-darwin12.2.0
-I/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/include
-I/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/libsupc++
-I/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/include/backward
-I/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/testsuite/util
-fmessage-length=0 -std=c++98  -pedantic-errors -Wno-long-long   
-L/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs

-B/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs

-L/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs
 -multiply_defined suppress -lm   -fsanitize=address -o ./cond1.exe    (timeout
= 300)
PASS: g++.dg/eh/cond1.C -std=c++98 (test for excess errors)
Setting LD_LIBRARY_PATH to
.:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc:.:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc
ASAN:SIGSEGV
=================================================================
==80716== ERROR: AddressSanitizer: SEGV on unknown address 0x0000ffd27000 (pc
0x0000ffd27000 sp 0x7fff4ffcd448 bp 0x7fff4ffcd510 T0)
AddressSanitizer can not provide additional info.
    #0 0xffd26fff
(/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc/testsuite/g++/./cond1.exe+0xf00f7fff)
    #1 0x7fff8bd827e0 (/usr/lib/system/libdyld.dylib+0x27e0)
    #2 0x0
Stats: 0M malloced (0M for red zones) by 3 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 0 calls
Stats: 0M really freed by 0 calls
Stats: 1M (384 full pages) mmaped in 3 calls
  mmaps   by size class: 7:4095; 8:2047; 9:1023;
  mallocs by size class: 7:1; 8:1; 9:1;
  frees   by size class:
  rfrees  by size class:
Stats: malloc large: 0 small slow: 3
==80716== ABORTING
Executing on host: addr2line -f -e /usr/lib/system/libdyld.dylib 0x27e0  
(timeout = 300)
Executing on host: addr2line -f -e
/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc/testsuite/g++/./cond1.exe
0xf00f7fff   (timeout = 300)
FAIL: g++.dg/eh/cond1.C -std=c++98 execution test

There are 522 instances out of the 841 failures seen in the g++ testsuite with
-fsanitize=address...

http://gcc.gnu.org/ml/gcc-testresults/2012-11/msg02388.html


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
@ 2012-11-28 21:48 ` howarth at nitro dot med.uc.edu
  2012-11-28 21:55 ` howarth at nitro dot med.uc.edu
                   ` (17 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-28 21:48 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #1 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-28 21:47:41 UTC ---
Created attachment 28823
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=28823
assembly file for failing g++.dg/eh/cond1.C -std=c++98 with -fsanitize=address


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
  2012-11-28 21:48 ` [Bug sanitizer/55521] " howarth at nitro dot med.uc.edu
@ 2012-11-28 21:55 ` howarth at nitro dot med.uc.edu
  2012-11-28 22:16 ` howarth at nitro dot med.uc.edu
                   ` (16 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-28 21:55 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #2 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-28 21:55:14 UTC ---
The majority of the libstdc+++-v3 testsuite failures seen with
-fsanitize=address appear to be of the same form. It is difficult to get an
accurate count because many instances show multiple ASAN:SIGSEGV in a single
failure such as...

Executing on host: /sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/./gcc/g++
-shared-libgcc -B/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/./gcc
-nostdinc++
-L/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/src
-L/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/src/.libs
-B/sw/lib/gcc4.8/x86_64-apple-darwin12.2.0/bin/
-B/sw/lib/gcc4.8/x86_64-apple-darwin12.2.0/lib/ -isystem
/sw/lib/gcc4.8/x86_64-apple-darwin12.2.0/include -isystem
/sw/lib/gcc4.8/x86_64-apple-darwin12.2.0/sys-include
-B/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs
-D_GLIBCXX_ASSERT -fmessage-length=0 -ffunction-sections -fdata-sections -g -O2
-g -O2 -DLOCALEDIR="." -nostdinc++
-I/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/include/x86_64-apple-darwin12.2.0
-I/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/include
-I/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/libsupc++
-I/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/include/backward
-I/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/testsuite/util
/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/libstdc++-v3/testsuite/20_util/shared_ptr/thread/default_weaktoshared.cc
   -std=gnu++0x  ./libtestc++.a -L/sw/lib -liconv  -lm   -fsanitize=address -o
./default_weaktoshared.exe    (timeout = 600)
PASS: 20_util/shared_ptr/thread/default_weaktoshared.cc (test for excess
errors)
Setting LD_LIBRARY_PATH to
:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/../libgomp/.libs:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs::/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/../libgomp/.libs:/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/./libstdc++-v3/src/.libs
ASAN:SIGSEGV
ASAN:SIGSEGV
ASAN:SIGSEGV
ASAN:SIGSEGV
ASAN:SIGSEGV
ASAN:SIGSEGV
=================================================================
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
ASAN:SIGSEGV
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
==85451== ERROR: AddressSanitizer: SEGV on unknown address 0x0000ffd27000 (pc
0x0000ffd27000 sp 0x000110528e28 bp 0x100000000000 T2)
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
AddressSanitizer can not provide additional info.
    #0 0xffd26fffASAN:SIGSEGV
ASAN:SIGSEGV
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
ASAN:SIGSEGV
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.
==85451== AddressSanitizer: while reporting a bug found another one.Ignoring.

(/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/testsuite/./default_weaktoshared.exe+0xf27e3fff)
Thread T2 created by T0 here:
    #0 0x10d815de6 (/sw/lib/gcc4.8/lib/libasan.0.dylib+0x5de6)
    #1 0x10d5452b6
(/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/x86_64-apple-darwin12.2.0/libstdc++-v3/testsuite/./default_weaktoshared.exe+0x1000022b6)
Stats: 0M malloced (0M for red zones) by 2054 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 2 calls
Stats: 0M really freed by 0 calls
Stats: 3M (769 full pages) mmaped in 6 calls
  mmaps   by size class: 7:8190; 8:2047; 9:1023; 11:255; 14:32;
  mallocs by size class: 7:2011; 8:10; 9:11; 11:10; 14:12;
  frees   by size class: 7:1; 14:1;
  rfrees  by size class:
Stats: malloc large: 0 small slow: 55
==85451== ABORTING
FAIL: 20_util/shared_ptr/thread/default_weaktoshared.cc execution test
extra_tool_flags are:
  -std=gnu++0x


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
  2012-11-28 21:48 ` [Bug sanitizer/55521] " howarth at nitro dot med.uc.edu
  2012-11-28 21:55 ` howarth at nitro dot med.uc.edu
@ 2012-11-28 22:16 ` howarth at nitro dot med.uc.edu
  2012-11-28 22:29 ` howarth at nitro dot med.uc.edu
                   ` (15 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-28 22:16 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #3 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-28 22:16:25 UTC ---
I don't see the g++.dg/eh/cond1.C execution failures on x86_64 Fedora 15 so
this may well be darwin specific. In gdb, on x86_64 darwin12 this traces as...

# gdb ./cond1.exe                                                              
              GNU gdb 6.3.50-20050815 (Apple version gdb-1822) (Sun Aug  5
03:00:42 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared
libraries ...... done

(gdb) break main
Breakpoint 1 at 0x10000113b: file
/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/gcc/testsuite/g++.dg/eh/cond1.C,
line 21.
(gdb) r
Starting program:
/sw/src/fink.build/gcc48-4.8.0-1000/darwin_objdir/gcc/testsuite/g++/cond1.exe 
Reading symbols for shared libraries +++++.................................
done

Breakpoint 1, main (argc=1, argv=0x7fff5fbff848) at
/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/gcc/testsuite/g++.dg/eh/cond1.C:21
21        (argc+1 ? has_destructor() : throw 0);
(gdb) s
has_destructor::~has_destructor (this=0x7fff5fbff77f) at
/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/gcc/testsuite/g++.dg/eh/cond1.C:10
10        ~has_destructor() { }
(gdb) 
main (argc=1, argv=0x7fff5fbff848) at
/sw/src/fink.build/gcc48-4.8.0-1000/gcc-4.8-20121127/gcc/testsuite/g++.dg/eh/cond1.C:22
22        CI((argc+1 ? throw 0 : has_destructor()));
(gdb) 
__cxa_allocate_exception (thrown_size=132) at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:102
102      ret = malloc (thrown_size);
(gdb) 
0x00000001000790da in dyld_stub_malloc () at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:204
204    __cxxabiv1::__cxa_free_dependent_exception
(gdb) 
0x00007fff8bd80878 in dyld_stub_binder ()
(gdb) 
Single stepping until exit from function dyld_stub_binder, 
which has no line number information.
0x00007fff8bd808a5 in misaligned_stack_error_entering_dyld_stub_binder ()
(gdb) 
Single stepping until exit from function
misaligned_stack_error_entering_dyld_stub_binder, 
which has no line number information.
0x00007fff8bd808e1 in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
0x00007fff8bd81f61 in _dyld_fast_stub_entry ()
(gdb) 
Single stepping until exit from function _Z21_dyld_fast_stub_entryPvl, 
which has no line number information.
0x00007fff5fc03fbd in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
(gdb) 
Single stepping until exit from function
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm, 
which has no line number information.
0x00007fff8bd808ee in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
0x00007fff94c3bb7e in malloc ()
(gdb) 
Single stepping until exit from function malloc, 
which has no line number information.
__cxa_allocate_exception (thrown_size=132) at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:104
104      if (! ret)
(gdb) 
102      ret = malloc (thrown_size);
(gdb) 
104      if (! ret)
(gdb) 
132      __cxa_eh_globals *globals = __cxa_get_globals ();
(gdb) 
__cxa_get_globals () at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_globals.cc:63
63    { return get_global(); }
(gdb) 
0x0000000100079254 in dyld_stub___emutls_get_address () at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:204
204    __cxxabiv1::__cxa_free_dependent_exception
(gdb) 
0x00007fff8bd80878 in dyld_stub_binder ()
(gdb) 
Single stepping until exit from function dyld_stub_binder, 
which has no line number information.
0x00007fff8bd808a5 in misaligned_stack_error_entering_dyld_stub_binder ()
(gdb) 
Single stepping until exit from function
misaligned_stack_error_entering_dyld_stub_binder, 
which has no line number information.
0x00007fff8bd808e1 in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
0x00007fff8bd81f61 in _dyld_fast_stub_entry ()
(gdb) 
Single stepping until exit from function _Z21_dyld_fast_stub_entryPvl, 
which has no line number information.
0x00007fff5fc03fbd in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
(gdb) 
Single stepping until exit from function
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm, 
which has no line number information.
0x00007fff8bd808ee in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
__emutls_get_address (obj=0x1000bc520) at
../../../gcc-4.8-20121127/libgcc/emutls.c:128
128    {
(gdb) 
Current language:  auto; currently c
139      pointer offset = obj->loc.offset;
(gdb) 
141      if (__builtin_expect (offset == 0, 0))
(gdb) 
700        return __gthrw_(pthread_once) (__once, __func);
(gdb) 
0x0000000102bfb42a in dyld_stub_pthread_once ()
(gdb) 
Single stepping until exit from function dyld_stub_pthread_once, 
which has no line number information.

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x00000000ffd27000
0x00000000ffd27000 in ?? ()
(gdb) 
Cannot access memory at address 0xffd27000
Cannot find bounds of current function
(gdb)


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (2 preceding siblings ...)
  2012-11-28 22:16 ` howarth at nitro dot med.uc.edu
@ 2012-11-28 22:29 ` howarth at nitro dot med.uc.edu
  2012-11-29  0:47 ` howarth at nitro dot med.uc.edu
                   ` (14 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-28 22:29 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #4 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-28 22:28:55 UTC ---
Debugging a similar build of cond1.exe on darwin12 without -fsanitize=address
show exactly the same trace except...

Single stepping until exit from function dyld_stub_pthread_once, 
which has no line number information.

Program exited normally.

Will need to retest x86_64 Fedora 15 with a emutls build of gcc trunk to make
sure this isn't a problem with emutls vs -fsanitize=address.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (3 preceding siblings ...)
  2012-11-28 22:29 ` howarth at nitro dot med.uc.edu
@ 2012-11-29  0:47 ` howarth at nitro dot med.uc.edu
  2012-11-29 21:25 ` howarth at nitro dot med.uc.edu
                   ` (13 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-29  0:47 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #5 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-29 00:47:14 UTC ---
This problem doesn't occur with x86_64 Fedora 15 built with --disable-tls.
Also, using llvm 3.2's clang++ on x86_64-apple-darwin12, I can compile...

/sw/opt/llvm-3.2/bin/clang++ -fsanitize=address cond1.C

and the test case runs fine whereas...

g++-fsf-4.8 -fsanitize=address cond1.C

produces...

 ./a.out
ASAN:SIGSEGV
=================================================================
==473== ERROR: AddressSanitizer: SEGV on unknown address 0x0000ffd27000 (pc
0x0000ffd27000 sp 0x7fff5928e848 bp 0x7fff5928e910 T0)
AddressSanitizer can not provide additional info.
    #0 0xffd26fff (/Users/howarth/./a.out+0xf93b5fff)
    #1 0x7fff8bd827e0 (/usr/lib/system/libdyld.dylib+0x27e0)
    #2 0x0
Stats: 0M malloced (0M for red zones) by 3 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 0 calls
Stats: 0M really freed by 0 calls
Stats: 1M (384 full pages) mmaped in 3 calls
  mmaps   by size class: 7:4095; 8:2047; 9:1023; 
  mallocs by size class: 7:1; 8:1; 9:1; 
  frees   by size class: 
  rfrees  by size class: 
Stats: malloc large: 0 small slow: 3
==473== ABORTING


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (4 preceding siblings ...)
  2012-11-29  0:47 ` howarth at nitro dot med.uc.edu
@ 2012-11-29 21:25 ` howarth at nitro dot med.uc.edu
  2012-11-30 18:01 ` howarth at nitro dot med.uc.edu
                   ` (12 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-29 21:25 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #6 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-29 21:25:07 UTC ---
Opened radr://12777299 so that the darwin linker maintainer could look at this
issue. His analysis of the failing test case so far is...

----------------------------------------------------------------------------------

I debugged this a bit and it seems the mach_override patching of __cxa_throw is
bogus.  The start of that function is patched to jump to garbage.

Breakpoint 1, 0x0000000100001c19 in main ()
(gdb) display/i $pc
2: x/i $pc  0x100001c19 <main+318>:     callq  0x100016386
<dyld_stub___cxa_throw>
(gdb) si
0x0000000100016386 in dyld_stub___cxa_throw ()
2: x/i $pc  0x100016386 <dyld_stub___cxa_throw>:        jmpq   *0xae1c(%rip)   
    # 0x1000211a8
(gdb)
0x0000000102244870 in __cxa_throw ()
2: x/i $pc  0x102244870 <__cxa_throw>:  jmpq   0xffd27000
(gdb)  # the above its __cxa_throw in gcc's libstdc++.6.dylib.  The first
instruction has been patch to jump to a garbage address.

(gdb) x/8i 0x102244870-8
0x102244868
<_ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception+56>: std
0x102244869
<_ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception+57>:
(bad)
0x10224486a
<_ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception+58>:
decl   (%rdi)
0x10224486c
<_ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception+60>:
(bad)
0x10224486d
<_ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception+61>: add
   %r8b,(%rax)
0x102244870 <__cxa_throw>:    jmpq   0xffd27000
0x102244875 <__cxa_throw+5>:    or     (%rax),%eax
0x102244877 <__cxa_throw+7>:    push   %rbx
(gdb)
(gdb) watch *0x102244870
Hardware watchpoint 2: *4330899568
(gdb) r

Old value = -788165304
New value = -1373139991
0x0000000100016203 in __asan_mach_override_ptr_custom ()
(gdb) bt
#0  0x0000000100016203 in __asan_mach_override_ptr_custom ()
#1  0x0000000100015a9e in __interception::OverrideFunction ()
#2  0x00007fff5fc13378 in ImageLoaderMachO::doModInitFunctions ()
#3  0x00007fff5fc13762 in ImageLoaderMachO::doInitialization ()
#4  0x00007fff5fc1006e in ImageLoader::recursiveInitialization ()
#5  0x00007fff5fc0feba in ImageLoader::runInitializers ()
#6  0x00007fff5fc01fc0 in dyld::initializeMainExecutable ()
#7  0x00007fff5fc05b04 in dyld::_main ()
#8  0x00007fff5fc01397 in dyldbootstrap::start ()
#9  0x00007fff5fc0105e in _dyld_start ()
(gdb) x/8i 0x102244870
0x102244870 <__cxa_throw>:      jmpq   0xffd27000
0x102244875 <__cxa_throw+5>:    or     (%rax),%eax
0x102244877 <__cxa_throw+7>:    push   %rbx
0x102244878 <__cxa_throw+8>:    lea    -0x20(%rdi),%rbx
0x10224487c <__cxa_throw+12>:   mov    %rsi,-0x70(%rdi)
# Here is where the patching is being done


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (5 preceding siblings ...)
  2012-11-29 21:25 ` howarth at nitro dot med.uc.edu
@ 2012-11-30 18:01 ` howarth at nitro dot med.uc.edu
  2012-12-01  1:51 ` howarth at nitro dot med.uc.edu
                   ` (11 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-11-30 18:01 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #7 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-11-30 18:01:28 UTC ---
This issue is further analyzed and a potential fix to mach_override proposed in
http://lists.cs.uiuc.edu/pipermail/llvmdev/2012-November/056478.html.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (6 preceding siblings ...)
  2012-11-30 18:01 ` howarth at nitro dot med.uc.edu
@ 2012-12-01  1:51 ` howarth at nitro dot med.uc.edu
  2012-12-01 13:55 ` jakub at gcc dot gnu.org
                   ` (10 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-01  1:51 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #8 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-01 01:50:37 UTC ---
Note that fixing this by replacing mach_override with interpose will require
FSF gcc to be enhanced to provide the missing blocks support (which was
developed after the GPLv3 transition and will thus have to be coded from
scratch).


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (7 preceding siblings ...)
  2012-12-01  1:51 ` howarth at nitro dot med.uc.edu
@ 2012-12-01 13:55 ` jakub at gcc dot gnu.org
  2012-12-01 21:37 ` howarth at nitro dot med.uc.edu
                   ` (9 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: jakub at gcc dot gnu.org @ 2012-12-01 13:55 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-12-01 13:54:49 UTC ---
Or just say asan isn't supported on darwin.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (8 preceding siblings ...)
  2012-12-01 13:55 ` jakub at gcc dot gnu.org
@ 2012-12-01 21:37 ` howarth at nitro dot med.uc.edu
  2012-12-01 21:43 ` howarth at nitro dot med.uc.edu
                   ` (8 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-01 21:37 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #10 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-01 21:37:10 UTC ---
Created attachment 28847
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=28847
mockup patch for switching darwin to mac function interpose


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (9 preceding siblings ...)
  2012-12-01 21:37 ` howarth at nitro dot med.uc.edu
@ 2012-12-01 21:43 ` howarth at nitro dot med.uc.edu
  2012-12-01 23:27 ` howarth at nitro dot med.uc.edu
                   ` (7 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-01 21:43 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #11 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-01 21:42:41 UTC ---
(In reply to comment #10)
> Created attachment 28847 [details]
> mockup patch for switching darwin to mac function interpose

This patch bootstraps gcc trunk to build libasan with functional mac function
interpose support that passes the asan.exp testsuite without regressions. It
also eliminates PR55521 by avoiding the use of mach_override. It still needs
some cleanup. In particular, I have used 'if 0' to avoid dispatch calls on FSF
gcc which required the missing blocks support. This needs to be replaced by a
test on an appropriate definition specific to FSF gcc (any ideas?). As I
understand it, the use of blocks is for objc which isn't a huge deal at the
moment.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (10 preceding siblings ...)
  2012-12-01 21:43 ` howarth at nitro dot med.uc.edu
@ 2012-12-01 23:27 ` howarth at nitro dot med.uc.edu
  2012-12-01 23:50 ` howarth at nitro dot med.uc.edu
                   ` (6 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-01 23:27 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

Jack Howarth <howarth at nitro dot med.uc.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #28847|0                           |1
        is obsolete|                            |

--- Comment #12 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-01 23:27:17 UTC ---
Created attachment 28849
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=28849
preliminary mac function interpose patch


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (11 preceding siblings ...)
  2012-12-01 23:27 ` howarth at nitro dot med.uc.edu
@ 2012-12-01 23:50 ` howarth at nitro dot med.uc.edu
  2012-12-01 23:51 ` howarth at nitro dot med.uc.edu
                   ` (5 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-01 23:50 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #13 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-01 23:50:15 UTC ---
Created attachment 28850
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=28850
revised preliminary mac function interpose patch

Add missing removal of usage of USING_MACH_OVERRIDE from
libsanitizer/interception/Makefile.am.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (12 preceding siblings ...)
  2012-12-01 23:50 ` howarth at nitro dot med.uc.edu
@ 2012-12-01 23:51 ` howarth at nitro dot med.uc.edu
  2012-12-02  2:58 ` howarth at nitro dot med.uc.edu
                   ` (4 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-01 23:51 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

Jack Howarth <howarth at nitro dot med.uc.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #28849|0                           |1
        is obsolete|                            |
  Attachment #28850|0                           |1
        is obsolete|                            |

--- Comment #14 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-01 23:51:05 UTC ---
Created attachment 28851
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=28851
revised preliminary mac function interpose patch


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (13 preceding siblings ...)
  2012-12-01 23:51 ` howarth at nitro dot med.uc.edu
@ 2012-12-02  2:58 ` howarth at nitro dot med.uc.edu
  2012-12-02  5:38 ` howarth at nitro dot med.uc.edu
                   ` (3 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-02  2:58 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #15 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-02 02:57:46 UTC ---
Patch to switch libsanitizer on darwin from mach_override to mac interpose
functions submitted to gcc-patches at
http://gcc.gnu.org/ml/gcc-patches/2012-12/msg00055.html.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (14 preceding siblings ...)
  2012-12-02  2:58 ` howarth at nitro dot med.uc.edu
@ 2012-12-02  5:38 ` howarth at nitro dot med.uc.edu
  2012-12-02 21:25 ` howarth at nitro dot med.uc.edu
                   ` (2 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-02  5:38 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #16 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-02 05:38:09 UTC ---
Using mac function interposition on x86_64-apple-darwin12 produces only 8
remaining g++ testsuite failures emitting ASAN:SIGSEGV messages compared to 10
seen in the g++ testsuite under x86_64 Fedora 15. Also on
x86_64-apple-darwin12, the switch from mach_override to mac function
interposition reduces the unexpected failures in the g++ testsuite from 841 to
323. This is still far more than seen under x86_64 Fedora 15 (which shows 107)
but still is a major improvement over mach_override.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (15 preceding siblings ...)
  2012-12-02  5:38 ` howarth at nitro dot med.uc.edu
@ 2012-12-02 21:25 ` howarth at nitro dot med.uc.edu
  2012-12-03  4:10 ` dvyukov at google dot com
  2012-12-04 14:53 ` howarth at nitro dot med.uc.edu
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-02 21:25 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

--- Comment #17 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-02 21:25:32 UTC ---
Revised patch posted at
http://gcc.gnu.org/ml/gcc-patches/2012-12/msg00084.html. Also tested with a
build of xplor-nih (a complex mix of c, c++ and fortran code) which has always
had optimization issues with FSF gcc (as xplor-nih has its own heap memory
manager routines in fortran). A build of xplor-nih with -fsanitize=address
produced 41 failures out of 154 tests but all of the failures emitted
AddressSanitizer output (often unknown-crash on address) rather than
segfaulting.


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (16 preceding siblings ...)
  2012-12-02 21:25 ` howarth at nitro dot med.uc.edu
@ 2012-12-03  4:10 ` dvyukov at google dot com
  2012-12-04 14:53 ` howarth at nitro dot med.uc.edu
  18 siblings, 0 replies; 20+ messages in thread
From: dvyukov at google dot com @ 2012-12-03  4:10 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

Dmitry Vyukov <dvyukov at google dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dvyukov at google dot com

--- Comment #18 from Dmitry Vyukov <dvyukov at google dot com> 2012-12-03 04:08:52 UTC ---
(In reply to comment #17)
> Revised patch posted at
> http://gcc.gnu.org/ml/gcc-patches/2012-12/msg00084.html. Also tested with a
> build of xplor-nih (a complex mix of c, c++ and fortran code) which has always
> had optimization issues with FSF gcc (as xplor-nih has its own heap memory
> manager routines in fortran). A build of xplor-nih with -fsanitize=address
> produced 41 failures out of 154 tests but all of the failures emitted
> AddressSanitizer output (often unknown-crash on address) rather than
> segfaulting.

I guess it's asan signal handler transforms paging faults into "unknown-crash
on address".

But the real asan reports, do they make sense? I mean is it real bugs in user
code, or some nonsense?


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [Bug sanitizer/55521] many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address
  2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
                   ` (17 preceding siblings ...)
  2012-12-03  4:10 ` dvyukov at google dot com
@ 2012-12-04 14:53 ` howarth at nitro dot med.uc.edu
  18 siblings, 0 replies; 20+ messages in thread
From: howarth at nitro dot med.uc.edu @ 2012-12-04 14:53 UTC (permalink / raw)
  To: gcc-bugs


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55521

Jack Howarth <howarth at nitro dot med.uc.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |FIXED

--- Comment #19 from Jack Howarth <howarth at nitro dot med.uc.edu> 2012-12-04 14:51:53 UTC ---
Fixed at...

r194120 | kcc | 2012-12-04 02:00:35 -0500 (Tue, 04 Dec 2012) | 1 line

[libsanitizer] Fix PR55521 by switching libsanitizer from mach_override to mac
interpose functions on darwin


^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2012-12-04 14:53 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-11-28 21:44 [Bug sanitizer/55521] New: many instances of ASAN:SIGSEGV failures in g++ testsuite with -fsanitize=address howarth at nitro dot med.uc.edu
2012-11-28 21:48 ` [Bug sanitizer/55521] " howarth at nitro dot med.uc.edu
2012-11-28 21:55 ` howarth at nitro dot med.uc.edu
2012-11-28 22:16 ` howarth at nitro dot med.uc.edu
2012-11-28 22:29 ` howarth at nitro dot med.uc.edu
2012-11-29  0:47 ` howarth at nitro dot med.uc.edu
2012-11-29 21:25 ` howarth at nitro dot med.uc.edu
2012-11-30 18:01 ` howarth at nitro dot med.uc.edu
2012-12-01  1:51 ` howarth at nitro dot med.uc.edu
2012-12-01 13:55 ` jakub at gcc dot gnu.org
2012-12-01 21:37 ` howarth at nitro dot med.uc.edu
2012-12-01 21:43 ` howarth at nitro dot med.uc.edu
2012-12-01 23:27 ` howarth at nitro dot med.uc.edu
2012-12-01 23:50 ` howarth at nitro dot med.uc.edu
2012-12-01 23:51 ` howarth at nitro dot med.uc.edu
2012-12-02  2:58 ` howarth at nitro dot med.uc.edu
2012-12-02  5:38 ` howarth at nitro dot med.uc.edu
2012-12-02 21:25 ` howarth at nitro dot med.uc.edu
2012-12-03  4:10 ` dvyukov at google dot com
2012-12-04 14:53 ` howarth at nitro dot med.uc.edu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).