[Bug sanitizer/59286] New: segfault in __sanitizer::StackDepotGet

[Bug sanitizer/59286] New: segfault in __sanitizer::StackDepotGet

[Joost.VandeVondele at mat dot ethz.ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

            Bug ID: 59286
           Summary: segfault in  __sanitizer::StackDepotGet
           Product: gcc
           Version: 4.9.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: Joost.VandeVondele at mat dot ethz.ch
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org

Running our tsan instrumented code, I'm seeing a segfault in tsan. I have no
suitable testcase for this yet (short of building CP2K), so I'm posting the
backtrace here in case this rings a bell / triggers some suggestions on what
might be going on. I'll try to do some further testing.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff4a27428 in __sanitizer::StackDepotGet (id=8388952,
size=0x7ffffffcb8f8) at
../../../../gcc/libsanitizer/sanitizer_common/sanitizer_stackdepot.cc:192
192          if (s->id == id) {
(gdb) bt
#0  0x00007ffff4a27428 in __sanitizer::StackDepotGet (id=8388952,
size=0x7ffffffcb8f8) at
../../../../gcc/libsanitizer/sanitizer_common/sanitizer_stackdepot.cc:192
#1  0x00007ffff4a1d9de in __tsan::ScopedReport::AddLocation (this=0x800158,
this@entry=0x7ffffffcb9f0, addr=140737488140536, addr@entry=137748196274048,
size=140737321271672, 
    size@entry=8) at ../../../../gcc/libsanitizer/tsan/tsan_rtl_report.cc:339
#2  0x00007ffff4a1ed30 in __tsan::ReportRace (thr=<optimized out>) at
../../../../gcc/libsanitizer/tsan/tsan_rtl_report.cc:697
#3  0x00007ffff4a21e02 in __tsan_report_race_thunk () at
../../../../gcc/libsanitizer/tsan/tsan_rtl_amd64.S:122
#4  0x00007ffff49ef9c8 in HandleRace (old=..., cur=..., shadow_mem=<optimized
out>, thr=<optimized out>) at ../../../../gcc/libsanitizer/tsan/tsan_rtl.cc:376
#5  MemoryAccessImpl (cur=..., shadow_mem=<optimized out>, kIsAtomic=<optimized
out>, kAccessIsWrite=<optimized out>, kAccessSizeLog=<optimized out>,
addr=<optimized out>, 
    thr=<optimized out>) at ../../../../gcc/libsanitizer/tsan/tsan_rtl.cc:460
#6  __tsan::MemoryAccess (thr=0x7ffff61f2780, pc=496049752, addr=82100428376,
kAccessSizeLog=8, kAccessIsWrite=true, kIsAtomic=true)
    at ../../../../gcc/libsanitizer/tsan/tsan_rtl.cc:531
#7  0x00007ffff67e62c0 in timings::timeset (routinen=<error reading variable:
Cannot access memory at address 0x3fe13824d8597625>, 
    handle=<error reading variable: Cannot access memory at address
0x3fe13824d8597005>, _routinen=<optimized out>) at
/data/vjoost/clean/cp2k/cp2k/src/../src/timings.F:254
(gdb) print s
$1 = (__sanitizer::StackDesc *) 0x4d634810890c558b
(gdb) print s->id
Cannot access memory at address 0x4d634810890c5593
(gdb) print id
$2 = 8388952
(gdb) list
187        CHECK_LT(idx, kTabSize);
188        atomic_uintptr_t *p = &depot.tab[idx];
189        uptr v = atomic_load(p, memory_order_consume);
190        StackDesc *s = (StackDesc*)(v & ~1);
191        for (; s; s = s->link) {
192          if (s->id == id) {
193            *size = s->size;
194            return s->stack;
195          }
196        }
(gdb) print idx
$3 = 4476
(gdb) print kTabSize
$5 = 1048576
(gdb) print depot.tab[idx]
$6 = {val_dont_use = 140737321271672}
(gdb) print depot

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[Joost.VandeVondele at mat dot ethz.ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

Joost VandeVondele <Joost.VandeVondele at mat dot ethz.ch> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Joost.VandeVondele at mat dot ethz
                   |                            |.ch

--- Comment #2 from Joost VandeVondele <Joost.VandeVondele at mat dot ethz.ch> ---
(In reply to Kostya Serebryany from comment #1)
> does this happen with clang's tsan?

This is a Fortran package, so trying clang is not really an option.

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #3 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
Can you post the exact command line to reproduce the failure? 
(We should have CP2K sources somewhere)

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[Joost.VandeVondele at mat dot ethz.ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #4 from Joost VandeVondele <Joost.VandeVondele at mat dot ethz.ch> ---
(In reply to Kostya Serebryany from comment #3)
> Can you post the exact command line to reproduce the failure? 
> (We should have CP2K sources somewhere)

The exact command line is rather easy:

cd cp2k/tests/QS/regtest-hybrid-1
export OMP_NUM_THREADS=4 ; ../../../exe/test_tsan/cp2k.ssmp H2O-hybrid-pbe0.inp

However, building a tsan instrumented CP2K is non-trivial, as it requires
libgomp to be built with tsan (see
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55561#c15 for a howto), and some of
the dependent libraries must be built with tsan as well. I'm happy to help, but
could take some time. Should I post detailed instructions?


Meanwhile another CP2K input fails with:

Unexpected mmap in InternalAllocator!

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #5 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> However, building a tsan instrumented CP2K is non-trivial, as it requires

Maybe let's do some remote debugging then :) 
The crash looks like someone corrupted the internal tsan's memory. 
Can you insert some Printf statements in sanitizer_stackdepot.cc
to see how we get this crazy pointer value: 0x4d634810890c5593

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[Joost.VandeVondele at mat dot ethz.ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #6 from Joost VandeVondele <Joost.VandeVondele at mat dot ethz.ch> ---
(In reply to Kostya Serebryany from comment #5)
> > However, building a tsan instrumented CP2K is non-trivial, as it requires
> 
> Maybe let's do some remote debugging then :) 

We can give this a try, but I might need rather detailed instructions. I.e.
starting from a sample printf statement and a suggested line number. With my
current setup, I get the segfault in a new location:

Program received signal SIGSEGV, Segmentation fault.
__sanitizer::find (s=0xffffffffffffffff, s@entry=0x7ffff4c2e8b8,
stack=stack@entry=0x7ffff4d797b8, size=size@entry=19,
hash=hash@entry=1116003544)
    at
../../../../gcc/libsanitizer/sanitizer_common/sanitizer_stackdepot.cc:109
109        if (s->hash == hash && s->size == size) {
(gdb) bt 7
#0  __sanitizer::find (s=0xffffffffffffffff, s@entry=0x7ffff4c2e8b8,
stack=stack@entry=0x7ffff4d797b8, size=size@entry=19,
hash=hash@entry=1116003544)
    at
../../../../gcc/libsanitizer/sanitizer_common/sanitizer_stackdepot.cc:109
#1  0x00007ffff35ae1a1 in __sanitizer::StackDepotPut (stack=0x7ffff4d797b8,
size=19)
    at
../../../../gcc/libsanitizer/sanitizer_common/sanitizer_stackdepot.cc:150
#2  0x00007ffff3570e4d in __tsan::CurrentStackId (thr=0xffffffffffffffff,
pc=140737301157816) at ../../../../gcc/libsanitizer/tsan/tsan_rtl.cc:305
#3  0x00007ffff35a1404 in __tsan::user_alloc (thr=0x7ffff4d79780,
pc=140737276073318, sz=24, align=<optimized out>)
    at ../../../../gcc/libsanitizer/tsan/tsan_mman.cc:110
#4  0x00007ffff358d58c in __interceptor_malloc (size=24) at
../../../../gcc/libsanitizer/tsan/tsan_interceptors.cc:447
#5  0x00007ffff57ba78c in timings::timestop (handle=339) at
/data/vjoost/clean/cp2k/cp2k/src/../src/timings.F:328
#6  0x00007ffff76accf6 in dbcsr_error_handling::dbcsr_error_stop (handler=1,
error=...)
    at
/data/vjoost/clean/cp2k/cp2k/src/../src/dbcsr_lib/dbcsr_error_handling.F:180
(More stack frames follow...)
(gdb) print s
$3 = (__sanitizer::StackDesc *) 0xffffffffffffffff

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #8 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
Just insert more printfs everywhere you can :) 
E.g. print everything around "s->link = s2" in StackDepotPut

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[Joost.VandeVondele at mat dot ethz.ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #9 from Joost VandeVondele <Joost.VandeVondele at mat dot ethz.ch> ---
(In reply to Kostya Serebryany from comment #8)
> Just insert more printfs everywhere you can :) 
> E.g. print everything around "s->link = s2" in StackDepotPut

hmm I can write a lot of printfs, but it is not very targetted..

However, I think I got a little further. For this kind of crash:
Getting 0x7fffed22e328
Following 0x7ffff04b8a80
Following 0x40027bd6cd50653b

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed2f5100 (LWP 9760)]
0x00007fffebbae530 in __sanitizer::StackDepotGet (id=3978818064, size=0x0)
    at
../../../../gcc/libsanitizer/sanitizer_common/sanitizer_stackdepot.cc:197
197           if (s->id == id) {
(gdb) print s
$3 = (__sanitizer::StackDesc *) 0x40027bd6cd50653b

I have put a hardware breakpoint on this field
break __sanitizer::StackDepotGet
awatch ((StackDesc*)0x7ffff04b8a80)->link
(which is the link that gets corrupted).

This breakpoint gets activated from CP2K at:

[Switching to Thread 0x7fffed3ec100 (LWP 9804)]
Hardware access (read/write) watchpoint 13: ((StackDesc*)0x7ffff04b8a80)->link

Value = (PTR TO -> ( __sanitizer::StackDesc )) 0x40027bd6cd50653b
0x00007fffee8811fe in hfx_load_balance_methods::estimate_basic (p=...)
    at /data/vjoost/clean/cp2k/cp2k/src/../src/hfx_load_balance_methods.F:1829
1829           p1=p(1) ; p2=p(2) ; p3=p(3) ; p4=p(4)
(gdb) bt
#0  0x00007fffee8811fe in hfx_load_balance_methods::estimate_basic (p=...)
    at /data/vjoost/clean/cp2k/cp2k/src/../src/hfx_load_balance_methods.F:1829
#1  0x00007fffee881020 in hfx_load_balance_methods::cost_model (nsa=1, nsb=1,
nsc=1, nsd=1, npgfa=6, npgfb=6, 
    npgfc=6, npgfd=6, ratio=-0.3026277383289448, p1=..., p2=..., p3=...)
    at /data/vjoost/clean/cp2k/cp2k/src/../src/hfx_load_balance_methods.F:1817
#2  0x00007fffee87ee8c in hfx_load_balance_methods::estimate_block_cost
(natom=3, nkind=2, list_ij=..., 
    list_kl=..., set_list_ij=..., set_list_kl=..., iatom_start=1, iatom_end=1,
jatom_start=1, jatom_end=1, 
    katom_start=1, katom_end=1, latom_start=1, latom_end=1, particle_set=...,
coeffs_set=..., coeffs_kind=..., 
    is_assoc_atomic_block_global=..., do_periodic=.FALSE., kind_of=...,
basis_parameter=..., pmax_set=..., 
    pmax_atom=..., pmax_blocks=0, cell=0x7d3000012d80, do_p_screening=.FALSE.,
map_atom_to_kind_atom=..., 
    eval_type=1, log10_eps_schwarz=-10, log_2=0.3010299956639812,
coeffs_kind_max0=1.1049525569372649, 
    use_virial=.FALSE., atomic_pair_list=...)
    at /data/vjoost/clean/cp2k/cp2k/src/../src/hfx_load_balance_methods.F:2212

This is the 'correct' place for corruption, as this routine is only called for
those runs that segfault.

Potentially interesting is that this is also a routine that is somewhat special
in Fortran, i.e. a contained subroutine, which presumably is treated somewhat
special by the compiler (not sure about the C-like equivalent, maybe nested
functions or so ?)

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[Joost.VandeVondele at mat dot ethz.ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

--- Comment #10 from Joost VandeVondele <Joost.VandeVondele at mat dot ethz.ch> ---
well, maybe a more simple reason. If I export
 export OMP_STACKSIZE=32M
(i.e. stack size for the threads), the segfault disappears... does that sound
like a good reason (i.e. tsan instrumented binary might require more stack), or
does this seem just lucky coincidence ?

[Bug sanitizer/59286] segfault in __sanitizer::StackDepotGet

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59286

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #12 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Invalid as mentioned, stack size increases slightly with tsan enabled which
causes the needed increase in stack size overall.