Re: [PATCH] asan unit tests from llvm lit-test incremental changes

[Jakub Jelinek <jakub@redhat.com>]

On Thu, Dec 13, 2012 at 11:44:12AM +0400, Konstantin Serebryany wrote:
> We are discussing it from time to time.
> Sometimes, if e.g. an error happens inside a qsort callback,
> the fp-based unwinder fails to unwind through libc, while _Unwind would work.
> 
> I was opposed to this sometime ago because _Unwind often produced
> buggy stack traces on Ubuntu Lucid (the version we cared about).

Weird, must be some distro modifications, we've been using _Unwind based
backtraces everywhere for many years successfully, glibc backtrace uses it
too, pthread_cancel as well.

> >>  and perhaps for
> >> > these malloc wrappers like ::operator new, ::operator new[] and their
> >> > const std::nothrow_t& variants libasan could intercept them, call
> >> > malloc and if that returns NULL, call the original corresponding function
> >> > so that it deals with exceptions, new handler etc.
> 
> Hmm.. Why's that?
> Calling libc's malloc or libstdc++'s operator new in asan run-time is
> really a bad idea.

I didn't mean calling libc malloc, I meant calling libstdc++'s operator new,
which then calls malloc (== any, thus asan version), but does some
additional bookkeeping for failures.

The thing is that libstdc++'s operator new:
_GLIBCXX_WEAK_DEFINITION void *
operator new (std::size_t sz) _GLIBCXX_THROW (std::bad_alloc)
{
  void *p;

  /* malloc (0) is unpredictable; avoid it.  */
  if (sz == 0)
    sz = 1;
  p = (void *) malloc (sz);
  while (p == 0)
    {
      new_handler handler = __new_handler;
      if (! handler)
        _GLIBCXX_THROW_OR_ABORT(bad_alloc());
      handler ();
      p = (void *) malloc (sz);
    }

  return p;
}

_GLIBCXX_WEAK_DEFINITION void*
operator new[] (std::size_t sz) _GLIBCXX_THROW (std::bad_alloc)
{
  return ::operator new(sz);
}
etc. aren't built with frame pointers, therefore ebp/rbp may be used for
anything, therefore non-unwind based backtrace will stop on that or get
confused.  What I meant was have
void *
operator new (std::size_t sz) throw (std::bad_alloc)
{
  void *p = malloc (sz);
  if (__builtin_expect (p == NULL, 0))
    call_original_operator_new (sz);
  return p;
}
and similarly for operator new[] etc. in libasan, forcefully built with
-fno-omit-frame-pointer, so that in the likely case that malloc doesn't
return NULL the non-_Unwind based backtrace in malloc would unwind well
through operator new as well as operator new[].  Or if libasan malloc really
never returns NULL and you don't plan to ever change that (why?), you could
just make operator new/operator new[] etc. in libasan aliases to malloc.

> asan's allocator should never return 0 anyway, it should simply crash.
> I don't think we want to support new handler at all.

Does it?  Then it will abort perfectly valid programs.

	Jakub