glibc 2.26 mtrace broken, missing allocations

glibc 2.26 mtrace broken, missing allocations

[Stefani Seibold]

Hi,

when using mtrace i get a report of a reallocation which has an address
which was not reported.

For example:

@ /usr/lib64/libgobject-2.0.so.0:(g_signal_newv+0x23d)[0x7ffff7eb409d] - 0x5555559344c0
@ /usr/lib64/libglib-2.0.so.0:(g_malloc+0x19)[0x7ffff7244039] + 0x555555922d40 0x60
@ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] < 0x5555558f0ea0
@ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] > 0x5555558f0ea0 0x10
@ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] < 0x5555558f0e70

The ingoing address 0x5555558f0ea0 for the realloc was not reported by
an other alloc.

The process (gvim -f) is single threaded and it is always the same
address without address layout randomization.

How is this possible? Are there allocation functions which are not
traced by mtrace?

My glibc version:

GNU C Library (Gentoo 2.26-r6 p7) stable release version 2.26, by Roland McGrath et al.
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 7.3.0.
Available extensions:
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B

Re: glibc 2.26 mtrace broken, missing allocations

[Carlos O'Donell]

On 04/05/2018 03:01 PM, Stefani Seibold wrote:
> Hi,
> 
> when using mtrace i get a report of a reallocation which has an address
> which was not reported.
> 
> For example:
> 
> @ /usr/lib64/libgobject-2.0.so.0:(g_signal_newv+0x23d)[0x7ffff7eb409d] - 0x5555559344c0
> @ /usr/lib64/libglib-2.0.so.0:(g_malloc+0x19)[0x7ffff7244039] + 0x555555922d40 0x60
> @ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] < 0x5555558f0ea0
> @ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] > 0x5555558f0ea0 0x10
> @ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] < 0x5555558f0e70
> 
> The ingoing address 0x5555558f0ea0 for the realloc was not reported by
> an other alloc.

That's odd.

> The process (gvim -f) is single threaded and it is always the same
> address without address layout randomization.

OK, so as a single-threaded process it should be perfectly safe to use mtrace
(which is not MT-safe).

> How is this possible? Are there allocation functions which are not
> traced by mtrace?

I don't have any good answer for you.

The allocation functions all have hooks into the hook functions which are used
by mtrace. They are embedded into the libc.so.6 malloc API functions directly

e.g.

3026 void *
3027 __libc_malloc (size_t bytes)
3028 {
3029   mstate ar_ptr;
3030   void *victim;
3031 
3032   void *(*hook) (size_t, const void *)
3033     = atomic_forced_read (__malloc_hook);
3034   if (__builtin_expect (hook != NULL, 0))
3035     return (*hook)(bytes, RETURN_ADDRESS (0));

...

3136   void *(*hook) (void *, size_t, const void *) =
3137     atomic_forced_read (__realloc_hook);
3138   if (__builtin_expect (hook != NULL, 0))
3139     return (*hook)(oldmem, bytes, RETURN_ADDRESS (0));

And happen right away and record the result.

You should see *almost* all the results.

In theory the early dynamic loader bootstrap uses dl-minimal.c which has micro
allocator there for early bootstrap before libc.so.6's malloc can be called,
but none of those addresses should ever leak into the post-bootstrap for a realloc.
If such a thing did happen it would be a bug.

You can try running mcheck() and associated functions to enable additional checking.
Likewise mprobe().

DJ, You've been poking at this area, any thoughts?

-- 
Cheers,
Carlos.

Re: glibc 2.26 mtrace broken, missing allocations

[Stefani Seibold]

On Thu, 2018-04-05 at 15:21 -0500, Carlos O'Donell wrote:
> On 04/05/2018 03:01 PM, Stefani Seibold wrote:
> > Hi,
> > 
> > when using mtrace i get a report of a reallocation which has an
> > address
> > which was not reported.
> > 
> > For example:
> > 
> > @ /usr/lib64/libgobject-
> > 2.0.so.0:(g_signal_newv+0x23d)[0x7ffff7eb409d] - 0x5555559344c0
> > @ /usr/lib64/libglib-2.0.so.0:(g_malloc+0x19)[0x7ffff7244039] +
> > 0x555555922d40 0x60
> > @ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] <
> > 0x5555558f0ea0
> > @ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] >
> > 0x5555558f0ea0 0x10
> > @ /usr/lib64/libglib-2.0.so.0:(g_realloc+0x20)[0x7ffff72440f0] <
> > 0x5555558f0e70
> > 
> > The ingoing address 0x5555558f0ea0 for the realloc was not reported
> > by
> > an other alloc.
> 
> That's odd.
> 
> > The process (gvim -f) is single threaded and it is always the same
> > address without address layout randomization.
> 
> OK, so as a single-threaded process it should be perfectly safe to
> use mtrace
> (which is not MT-safe).
> 
> > How is this possible? Are there allocation functions which are not
> > traced by mtrace?
> 
> I don't have any good answer for you.
> 
> The allocation functions all have hooks into the hook functions which
> are used
> by mtrace. They are embedded into the libc.so.6 malloc API functions
> directly
> 
> e.g.
> 
> 3026 void *
> 3027 __libc_malloc (size_t bytes)
> 3028 {
> 3029   mstate ar_ptr;
> 3030   void *victim;
> 3031 
> 3032   void *(*hook) (size_t, const void *)
> 3033     = atomic_forced_read (__malloc_hook);
> 3034   if (__builtin_expect (hook != NULL, 0))
> 3035     return (*hook)(bytes, RETURN_ADDRESS (0));
> 
> ...
> 
> 3136   void *(*hook) (void *, size_t, const void *) =
> 3137     atomic_forced_read (__realloc_hook);
> 3138   if (__builtin_expect (hook != NULL, 0))
> 3139     return (*hook)(oldmem, bytes, RETURN_ADDRESS (0));
> 
> And happen right away and record the result.
> 
> You should see *almost* all the results.
> 
> In theory the early dynamic loader bootstrap uses dl-minimal.c which
> has micro
> allocator there for early bootstrap before libc.so.6's malloc can be
> called,
> but none of those addresses should ever leak into the post-bootstrap
> for a realloc.
> If such a thing did happen it would be a bug.
> 
> You can try running mcheck() and associated functions to enable
> additional checking.
> Likewise mprobe().
> 

I just tried mcheck() and mcheck_pedantic(). No difference at all.

> DJ, You've been poking at this area, any thoughts?
> 

Re: glibc 2.26 mtrace broken, missing allocations

[Carlos O'Donell]

On 04/05/2018 03:33 PM, Stefani Seibold wrote:
> I just tried mcheck() and mcheck_pedantic(). No difference at all.

I assume also then that valgrind with full checking doesn't show any
problems either?

If that's the case then you would have to debug the application under
trace and just count the malloc's to see where it came from.

I'd be happy to help at that point if you did some of the heavy lifting
to find the place the allocation comes from.

-- 
Cheers,
Carlos.

Re: glibc 2.26 mtrace broken, missing allocations

[DJ Delorie]

"Carlos O'Donell" <carlos@redhat.com> writes:
> DJ, You've been poking at this area, any thoughts?

The only thing I can think of off-hand is, that it's a malloc that
happened before mtrace() was called.  This can happen if, for example,
there's a ctor/init function that's called before main().

Using ltrace to find any pre-mtrace allocs might be fruitful.

Or my malloc trace branch, which hooks sooner than main().