`free` fails to reclaim memory

`free` fails to reclaim memory

[Konstantin Kharlamov]

Hello, while researching Emacs "leaking memory"¹, we may have actually stumbled upon glibc problem: glibc may not return freed memory to the OS, and by "memory" I mean here not a few kilobytes or megabytes for the sake of cache, but megabytes or even gigabytes. Having researched for a bit, I got the following testcase:

    // credits to https://stackoverflow.com/a/48652734/2388257#glibc-application-holding-onto-unused-memory-until-just-before-exit
    #include <cstdlib>
    #include <cstdio>
    #include <unistd.h>

    int main() {
        static const int N = 50000;
        static void *arr[N];

        for (int i = 0; i < N; i++)
            arr[i] = std::malloc(1024);

        // reverse to simplify allocators job
        for (int i = N - 1; i >= 0; i--)
            std::free(arr[i]);
        puts("Measure");
        sleep(1e5);
    }

This code allocates 50M of memory, and then frees it back. When it goes to sleep, measuring memory with `smem -kc "name pid pss" | grep -P "\ba\b"` will give 50M, despite that memory was freed. Original snippet was allocating 5G, none of which would return to the system as well. This is on glibc 2.32

The stackoverflow answer recommends calling `malloc_trim` for memory to return, but it is an odd recommendation because you can't expect everyone to do workarounds for glibc-specific problems.

Is there any chance this is not an intended behavior, but an actual glibc bug?

1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45200

On Sun, 2020-12-13 at 00:53 -0500, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <hi-angel@yandex.ru>
> > Cc: 45200@debbugs.gnu.org
> > Date: Sun, 13 Dec 2020 01:44:13 +0300
> >
> > Alright, fair enough. I crafted up another testcase, it may be better. The
> > following code first temporarily disables GC, then it prints "hello" 1000000
> > times, and finally it calls GC manually.
> >
> > I call `emacs -Q`, then measure PSS, then evaluate the code below, then
> > again measure PSS.
> >
> >     (let ((i 1000000))
> >       (setq gc-cons-threshold most-positive-fixnum)
> >       (while (> i 0)
> >         (print "hello")
> >         (setq i (- i 1)))
> >       (garbage-collect))
> >
> > The loop takes 20-30 seconds for me, I think. PSS before is ≈41M, and PSS
> > after is 266.3M. That is ≈200M of memory just vanished.
>
> That memory hasn't vanished, it is in your libc's malloc arena,
> available for future allocations.  When and if it will be given back
> to the OS is up to the specifics of the malloc implementation.  E.g.,
> when I do the above on MS-Windows, where malloc is more eager to return
> memory to the OS, I end up with just 40 MB footprint, and if I then
> invoke GC manually, the memory goes down almost to the original value:
> 14 MB vs 12 MB after startup.
>
> There are many places on the Internet which explain why the memory
> footprint of a program doesn't go back to the original value even
> though the program frees all the heap memory it allocated.  I suggest
> to read some of those explanations.

Re: `free` fails to reclaim memory

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

On Sun, Dec 13, 2020 at 02:41:06PM +0300, Konstantin Kharlamov wrote:
> Hello, while researching Emacs "leaking memory"¹, we may have actually stumbled upon glibc problem: glibc may not return freed memory to the OS, and by "memory" I mean here not a few kilobytes or megabytes for the sake of cache, but megabytes or even gigabytes. Having researched for a bit, I got the following testcase:

See also [1], especially glibc.malloc.mmap_threshold (default: start
with 128 KB and then adjust dynamically), which decides when to use
mmap (which can be returned to the OS on free) for a chunk, and
glibc.malloc.trim_threshold, which decides the minimum size of frees
(from sbrk) to return to the os (which is set by default at 128 KB)

So libc's malloc seems to try to do things, perhaps the chunks you
are using in your test program are too small to trigger what you
are expecting.

Note that if you mix malloc and free, the freed memory is reused,
so your experiment doesn't necessarily explain the "leaking
memory" thing.

Cheers

[1] https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: `free` fails to reclaim memory

[Konstantin Kharlamov]

On Sun, 2020-12-13 at 12:57 +0100, tomas@tuxteam.de wrote:
> On Sun, Dec 13, 2020 at 02:41:06PM +0300, Konstantin Kharlamov wrote:
> > Hello, while researching Emacs "leaking memory"¹, we may have actually
> > stumbled upon glibc problem: glibc may not return freed memory to the OS,
> > and by "memory" I mean here not a few kilobytes or megabytes for the sake of
> > cache, but megabytes or even gigabytes. Having researched for a bit, I got
> > the following testcase:
> 
> See also [1], especially glibc.malloc.mmap_threshold (default: start
> with 128 KB and then adjust dynamically), which decides when to use
> mmap (which can be returned to the OS on free) for a chunk, and
> glibc.malloc.trim_threshold, which decides the minimum size of frees
> (from sbrk) to return to the os (which is set by default at 128 KB)
> 
> So libc's malloc seems to try to do things, perhaps the chunks you
> are using in your test program are too small to trigger what you
> are expecting.

Thanks, I might look at tunables, but I'd also like to point out that this
doesn't make it less of a glibc problem. Because you can't expect everyone to
tune app up specifically for glibc. There're various libc libs on different
platforms, and it would take a lot of effort from developers to tune an app for
every existing libc out there.

> Note that if you mix malloc and free, the freed memory is reused,
> so your experiment doesn't necessarily explain the "leaking
> memory" thing.

Thanks, right, I just checked with Emacs, and I do see that memory is reused on
later runs of its testcase, which I guess confirms that this memory is held by
glibc.

> [1] 
> https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
>  - t

Re: `free` fails to reclaim memory

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

On Sun, Dec 13, 2020 at 03:14:39PM +0300, Konstantin Kharlamov wrote:

[...]

> Thanks, I might look at tunables, but I'd also like to point out that this
> doesn't make it less of a glibc problem. Because you can't expect everyone to
> tune app up specifically for glibc. There're various libc libs on different
> platforms, and it would take a lot of effort from developers to tune an app for
> every existing libc out there.

That's right. On the other hand, what we are seeing as the "libc defaults"
is most probably the result of a compromise. The library has to behave
sanely for a wide range of malloc/free behaviours. Returning space to the
OS for every 16 byte free will quickly become prohibitive (it involves
a syscall, after all).

I don't think the libc authors will see that as a bug.

If the generic behaviour doesn't fit your application's usage pattern,
that's what the tunables are for. Or the explicit call to malloc_trim.

In most applications, you'll be calling malloc and free in some interleaved
pattern, so the virtual size will stabilise at some point corresponding
to the maximum used space. In those cases, it seems a "good thing" that
this space isn't returned to the OS "too early" -- that would slow down
memory management.

That's what one should see in the Emacs case, I think.

> > Note that if you mix malloc and free, the freed memory is reused,
> > so your experiment doesn't necessarily explain the "leaking
> > memory" thing.
> 
> Thanks, right, I just checked with Emacs, and I do see that memory is reused on
> later runs of its testcase, which I guess confirms that this memory is held by
> glibc.

Thanks for investigating!

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: `free` fails to reclaim memory

[Konstantin Kharlamov]

On Sun, 2020-12-13 at 13:32 +0100, tomas@tuxteam.de wrote:
> On Sun, Dec 13, 2020 at 03:14:39PM +0300, Konstantin Kharlamov wrote:
>
> [...]
>
> > Thanks, I might look at tunables, but I'd also like to point out that this
> > doesn't make it less of a glibc problem. Because you can't expect everyone
> > to
> > tune app up specifically for glibc. There're various libc libs on different
> > platforms, and it would take a lot of effort from developers to tune an app
> > for
> > every existing libc out there.
>
> That's right. On the other hand, what we are seeing as the "libc defaults"
> is most probably the result of a compromise. The library has to behave
> sanely for a wide range of malloc/free behaviours. Returning space to the
> OS for every 16 byte free will quickly become prohibitive (it involves
> a syscall, after all).
>
> I don't think the libc authors will see that as a bug.

For these usecases Glibc already has a per-thread malloc cache, to which, if I correctly understand, the discussed behavior is irrelevant. It's just failure of libc to figure out it conserved too much memory, and that it needs to free some up.

So, I see no reason why it would not be considered a bug. Either way, since this discussion doesn't seem to be too popular, I just went ahead and reported it: https://sourceware.org/bugzilla/show_bug.cgi?id=27103

> If the generic behaviour doesn't fit your application's usage pattern,
> that's what the tunables are for. Or the explicit call to malloc_trim.
>
> In most applications, you'll be calling malloc and free in some interleaved
> pattern, so the virtual size will stabilise at some point corresponding
> to the maximum used space. In those cases, it seems a "good thing" that
> this space isn't returned to the OS "too early" -- that would slow down
> memory management.
>
> That's what one should see in the Emacs case, I think.

Well, in my case it won't stabilize. The usecase is that I disabled memory-threshold for Emacs garbage-collector (for performance reasons), and instead I hooked-up running GC to idle time.

Now, there can easily appear short-burst workloads which would allocate lots of memory before GC gets a chance to run. And then what? The memory Emacs just took will hardly be used again, but glibc isn't gonna return memory either.

Re: `free` fails to reclaim memory

[Konstantin Kharlamov]

On Sun, 2020-12-13 at 12:57 +0100, tomas@tuxteam.de wrote:
> On Sun, Dec 13, 2020 at 02:41:06PM +0300, Konstantin Kharlamov wrote:
> > Hello, while researching Emacs "leaking memory"¹, we may have actually
> > stumbled upon glibc problem: glibc may not return freed memory to the OS,
> > and by "memory" I mean here not a few kilobytes or megabytes for the sake of
> > cache, but megabytes or even gigabytes. Having researched for a bit, I got
> > the following testcase:
> 
> See also [1], especially glibc.malloc.mmap_threshold (default: start
> with 128 KB and then adjust dynamically), which decides when to use
> mmap (which can be returned to the OS on free) for a chunk, and
> glibc.malloc.trim_threshold, which decides the minimum size of frees
> (from sbrk) to return to the os (which is set by default at 128 KB)
> 
> So libc's malloc seems to try to do things, perhaps the chunks you
> are using in your test program are too small to trigger what you
> are expecting.
> 
> Note that if you mix malloc and free, the freed memory is reused,
> so your experiment doesn't necessarily explain the "leaking
> memory" thing.
> 
> Cheers
> 
> [1]
> https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
>  - t

Btw, now that I think of that, I'm wondering: is there any way to figure out how much memory in a running process is used by that glibc-owned memory?

I'm interested because I just remember this another bug¹ which also affects me, and nobody knows what exactly takes up that memory out there. Could be the same situation as with Emacs.

1: https://github.com/qutebrowser/qutebrowser/issues/1476

Re: `free` fails to reclaim memory

[tomas]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

On Sun, Dec 13, 2020 at 03:33:14PM +0300, Konstantin Kharlamov wrote:

[...]

> Btw, now that I think of that, I'm wondering: is there any way to figure out how much memory in a running process is used by that glibc-owned memory?

Hm. Valgrind [1] comes to mind. For libc's malloc there is some
debugging support [2]. This, and hours and hours of fun :)

Cheers

[1] https://www.valgrind.org/
[2] https://www.gnu.org/software/libc/manual/html_node/Allocation-Debugging.html

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: `free` fails to reclaim memory

[Konstantin Kharlamov]

On Sun, 2020-12-13 at 13:42 +0100, tomas@tuxteam.de wrote:
> On Sun, Dec 13, 2020 at 03:33:14PM +0300, Konstantin Kharlamov wrote:
> 
> [...]
> 
> > Btw, now that I think of that, I'm wondering: is there any way to figure out
> > how much memory in a running process is used by that glibc-owned memory?
> 
> Hm. Valgrind [1] comes to mind. For libc's malloc there is some
> debugging support [2]. This, and hours and hours of fun :)
> 
> Cheers
> 
> [1] https://www.valgrind.org/
> [2]
> https://www.gnu.org/software/libc/manual/html_node/Allocation-Debugging.html
> 
>  - t

As I understand, both 1 and 2 are tools to find memory-leaks. Memory leaks in turn are detected upon app exit. If I exit the testcase, there will be no memory leaks, because memory glibc got a hold of will be freed by glibc.

So, I don't see any possible way to use 1 or 2 to figure it out.