[Bug libstdc++/53169] New: Memory leak in std::vector<std::vector*>

[Bug libstdc++/53169] New: Memory leak in std::vector<std::vector*>

[antoinep92 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

             Bug #: 53169
           Summary: Memory leak in std::vector<std::vector*>
    Classification: Unclassified
           Product: gcc
           Version: 4.7.0
            Status: UNCONFIRMED
          Severity: critical
          Priority: P3
         Component: libstdc++
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: antoinep92@gmail.com


Created attachment 27268
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=27268
Test case: 500Mb used after memory neutral function returns!

The attached source is a minimal test case, implementing a sparse array of
std::vectors in class Collection, and test() demonstrates its use in a memory
neutral way (all allocated objects are freed).

When compiled on x86-64 linux with gcc 4.6.1, gcc 4.7.1 and clang 3.0 (using
GNU libstdc++), tools such as top show that memory increases when running
test(), but does not not decrease after the function exits: 500Mb are lost in
this test case. Just increase to loop count and make that 4Gb if you wish: the
amount of  leaked memory don't seem to be bounded.
`valgrind --leak-check=full ./a.out` reports there is not a single byte leaked,
which I double checked with the heap profiler from google perf tools.

The memory is reserved by libstdc++ and unavailable to other processes or
subsequent malloc/frees within the same program. Subsequent C++ STL allocations
(e.g. resizing a big vector) on the other hand don't register on process memory
and seem to ruse the reserved buffers; sometimes they even trigger deallocation
of the "leaked" memory.
For example when running test() multiple times, the bug only occurs on the
first call. Subsequent calls free memory when done. I guess the reserved memory
is accounted for within libstdc++ internals, and deallocated on finalization
which is why valgrind can't see it.

My application is a sequence of memory intensive operations, and this bug
prevents memory from being freed between steps, and the system quickly runs out
of memory.

Thanks a lot!

*** Note: As this is quite time-critical for me, if someone points me in the
right direction (files, classes involved) I can try to investigate this and
send back a patch.

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

--- Comment #1 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-04-30 14:28:11 UTC ---
(In reply to comment #0)
> The attached source is a minimal test case, implementing a sparse array of
> std::vectors in class Collection, and test() demonstrates its use in a memory
> neutral way (all allocated objects are freed).

Unless you call Collection::at(n) twice for the same n, in which case you leak
raw[n]

> When compiled on x86-64 linux with gcc 4.6.1, gcc 4.7.1 and clang 3.0 (using
> GNU libstdc++), tools such as top show that memory increases when running
> test(), but does not not decrease after the function exits:

That's how GNU/Linux works.  A process' memory will not decrease, but that
doesn't mean it's leaked. 


> 500Mb are lost in
> this test case. Just increase to loop count and make that 4Gb if you wish: the
> amount of  leaked memory don't seem to be bounded.
> `valgrind --leak-check=full ./a.out` reports there is not a single byte leaked,
> which I double checked with the heap profiler from google perf tools.

Indeed. No memory is leaked. The memory is returned to the heap and will be
available for further allocations.

> The memory is reserved by libstdc++ and unavailable to other processes or

It's virtual memory, if you delete it then it's available to other processes,
even if your process' memory usage doesn't appear to have decreased according
to top.

> subsequent malloc/frees within the same program. Subsequent C++ STL allocations
> (e.g. resizing a big vector) on the other hand don't register on process memory
> and seem to ruse the reserved buffers; sometimes they even trigger deallocation
> of the "leaked" memory.

In the default configuration libstdc++ just uses new/delete which uses malloc,
i.e. there's no memory caching in libstdc++ allocators
You can see this for yourself in include/ext/new_allocator.h

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

--- Comment #2 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-04-30 14:59:05 UTC ---
By changing your main to:

int main() {
    test();
    sleep(10);
        char* p = (char*)malloc(1024 * 127);
        for (int i=0; i < 100; ++i)
            p[1024 * i] = 'a' + (i%26);
        sleep(10);
        free(p);
        sleep(10);
    return 0;
}

I see that freeing the small malloc'd memory region (which is below glibc's
MMAP_THRESHOLD value) does actually trigger the earlier new'd memory to be
returned to the system too.

So it's possible to get the memory libstdc+ allocates to be returned to the
system, but it's under the control of glibc, nothing to do with std::vector or
libstdc++

If your memory usage patterns don't result in memory being returned then there
are several posibilities, including not freeing memory when you think you are,
or fragmenting the heap so that later allocations cannot re-use memory returned
to freelists and must allocate new memory using mmap.

Not a GCC bug though.

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

--- Comment #3 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-04-30 15:13:07 UTC ---
Similarly, calling malloc_trim(0) after test() causes glibc to immediately
return the memory to the system (requiring a sbrk system call next time memory
is needed)

You should probably use sbrk and/or experiment with
http://www.gnu.org/software/libc/manual/html_node/Statistics-of-Malloc.html to
see what's actually happening in your program, and maybe using mallopt() to
adjust things.

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-04-30 15:16:14 UTC ---
(In reply to comment #3)
> You should probably use sbrk and/or experiment with

Sorry, I meant use strace, *not* sbrk!

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[antoinep92 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

--- Comment #5 from Antoine Poliakov <antoinep92 at gmail dot com> 2012-04-30 15:27:16 UTC ---
(In reply to comment #2)
> So it's possible to get the memory libstdc+ allocates to be returned to the
> system, but it's under the control of glibc, nothing to do with std::vector or
> libstdc++
> 
> If your memory usage patterns don't result in memory being returned then there
> are several posibilities, including not freeing memory when you think you are,
> or fragmenting the heap so that later allocations cannot re-use memory returned
> to freelists and must allocate new memory using mmap.
> 
> Not a GCC bug though.

Thanks a lot for explaining this, and sorry for the invalid bug report.
There is probably a lot of memory fragmentation going on, but what's bothering
me is that if you launch several processes with { test(); sleep(30); }, waiting
each time for test() to return you will eventually begin to swap and go out of
memory. So the freed memory is not exactly available to other processes: that's
not just a counter problem.
Thanks for your feedback, and the <MMAP_THRESHOLD trick. I'm probably too harsh
on the system allocator and I will probably use a custom allocator using a
memory pool, which will allow me to minimize fragmentation and free the memory
in a predictable way.

I only see unconfirmed and resolved status, not invalid. Can you mark the bug
as invalid, please.

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[antoinep92 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

--- Comment #6 from Antoine Poliakov <antoinep92 at gmail dot com> 2012-04-30 15:31:17 UTC ---
(In reply to comment #4)
> (In reply to comment #3)
> > You should probably use sbrk and/or experiment with
> 
> Sorry, I meant use strace, *not* sbrk!

malloc_trim(0) saved the day!
Thanks again

[Bug libstdc++/53169] Memory leak in std::vector<std::vector*>

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53169

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID

--- Comment #7 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-04-30 15:51:57 UTC ---
(In reply to comment #5)
> I only see unconfirmed and resolved status, not invalid. Can you mark the bug
> as invalid, please.

As the reporter you could have done so yourself.  Done now though.