[Bug malloc/30852] New: Big deallocation takes 8 minutes

[Bug malloc/30852] New: Big deallocation takes 8 minutes

[safinaskar at mail dot ru]

https://sourceware.org/bugzilla/show_bug.cgi?id=30852

            Bug ID: 30852
           Summary: Big deallocation takes 8 minutes
           Product: glibc
           Version: 2.37
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: malloc
          Assignee: unassigned at sourceware dot org
          Reporter: safinaskar at mail dot ru
  Target Milestone: ---

Consider this code:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>

int
main ()
{
  printf ("Started\n");
  system ("date");
  system ("free -h");
  size_t n = 1000 * 1000 * 1000;
  uint8_t **a = malloc (n * sizeof (uint8_t *));
  for(size_t i = 0; i != n; ++i)
    {
      a[i] = malloc (sizeof (uint8_t));
    }
  printf ("Allocated\n");
  system ("date");
  system ("free -h");
  for(size_t i = 0; i != n; ++i)
    {
      free (a[i]);
    }
  system ("date");
  free (a);
  printf ("Deallocated\n");
  system ("date");
  system ("free -h");
  return 0;
}

I did run this program (with -O3) and got this result:

Started
Thu Sep 14 14:20:48 UTC 2023
               total        used        free      shared  buff/cache  
available
Mem:            31Gi        15Gi        14Gi       358Mi       2.0Gi       
15Gi
Swap:          163Gi        37Gi       126Gi
Allocated
Thu Sep 14 14:22:15 UTC 2023
               total        used        free      shared  buff/cache  
available
Mem:            31Gi        30Gi       261Mi       314Mi       1.0Gi      
528Mi
Swap:          163Gi        43Gi       120Gi
Thu Sep 14 14:30:58 UTC 2023
Deallocated
Thu Sep 14 14:30:59 UTC 2023
               total        used        free      shared  buff/cache  
available
Mem:            31Gi        27Gi       4.0Gi       223Mi       863Mi      
4.1Gi
Swap:          163Gi        53Gi       110Gi

As you can see, allocation took less than 2 minutes, but deallocation took more
than 8 minutes.

In this point you may say: "Okay, your program eats a lot of memory. Of course,
your program will be slow. Why you think this is a bug?" I think this is a bug,
because deallocating takes a lot more time than allocating (2 minutes vs 8
minutes).

I report this not for fun. The code above is reduced test case from my actual
production code base. In my actual code my program reads a lot of data from
files and then processes them. I noticed that actual execution takes several
minutes, but deallocation takes a lot more.

Debian Linux sid x86_64
gcc version 13.2.0
glibc 2.37

Originally found in Rust: https://github.com/rust-lang/rust/issues/115814

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30852] Big deallocation takes 8 minutes

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30852

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |adhemerval.zanella at linaro dot o
                   |                            |rg

--- Comment #1 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
This testcase roughly allocates about ~38GB and from the swap used value it
*does* seem that you are stressing it instead of the allocator.

With a testcase adjusted to measure the time directly (instead of calling
system) using clock_gettime on a machine with enough memory (so swap is not
used):

glibc 2.37:
allocation: 15.267103395 seconds
free:       5.583120996  seconds

glibc 2.38:
allocation: 15.316940207 seconds
free:       5.149215347  seconds

master:
allocation: 15.840980439 seconds
free:       5.157396663  seconds

Could you check with perf if the swap utilization is indeed the case? If it
were the case, there is not much glibc or any allocator can do. I would depend
heavily of how swap was configured (does it use compression or encryption?),
the IO bandwidth, the media throughput, the system load, etc.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30852] Big deallocation takes 8 minutes

[safinaskar at mail dot ru]

https://sourceware.org/bugzilla/show_bug.cgi?id=30852

--- Comment #2 from Askar Safin <safinaskar at mail dot ru> ---
Thanks for investigation!

> from the swap used value it *does* seem that you are stressing it instead of the allocator

Yes, exactly! Program's occupied memory doesn't fit in physical memory. And I
don't like that in such situation deallocation takes very long time.

My actual production program is written in Rust. In Rust deallocation happens
automatically. This means that I have to insert ugly "_Exit(0)" call before the
end of program to prevent deallocation to make the program faster. The need for
such ugly hacks means that something gone wrong

> does it use compression or encryption?
No

> the IO bandwidth, the media throughput
Swap is located on NVMe SSD device (in my laptop)

> the system load
I had a lot of apps opened in that time

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30852] Big deallocation takes 8 minutes

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30852

--- Comment #3 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
(In reply to Askar Safin from comment #2)
> Thanks for investigation!
> 
> > from the swap used value it *does* seem that you are stressing it instead of the allocator
> 
> Yes, exactly! Program's occupied memory doesn't fit in physical memory. And
> I don't like that in such situation deallocation takes very long time.
> 
> My actual production program is written in Rust. In Rust deallocation
> happens automatically. This means that I have to insert ugly "_Exit(0)" call
> before the end of program to prevent deallocation to make the program
> faster. The need for such ugly hacks means that something gone wrong

Again, once you start to use swap you are stressing multiple different
subsystems that are out of the scope of glibc allocator.  Just run a perf
record/report and you will see multiple kernel symbols involved in the block
layer, I/O subsystem, etc.

You might get better results with a different allocator, that moves the
required metadata out of returned allocated memory.  It *might* add less memory
pressure, since in theory a free should not touch the allocated memory.  It
would not help if the system decide to swap out the metadata pages, it would
need the allocator to MAP_LOCKED or mlock the pages, which came with its own
caveats.

But I think this is really out of scope of glibc allocator.

> 
> > does it use compression or encryption?
> No
> 
> > the IO bandwidth, the media throughput
> Swap is located on NVMe SSD device (in my laptop)
> 
> > the system load
> I had a lot of apps opened in that time

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30852] Big deallocation takes 8 minutes

[safinaskar at mail dot ru]

https://sourceware.org/bugzilla/show_bug.cgi?id=30852

Askar Safin <safinaskar at mail dot ru> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |INVALID

--- Comment #4 from Askar Safin <safinaskar at mail dot ru> ---
Okay, I'm closing

-- 
You are receiving this mail because:
You are on the CC list for the bug.