[Bug debuginfod/31103] New: periodically malloc_trim()

[Bug debuginfod/31103] New: periodically malloc_trim()

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

            Bug ID: 31103
           Summary: periodically malloc_trim()
           Product: elfutils
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: debuginfod
          Assignee: unassigned at sourceware dot org
          Reporter: fche at redhat dot com
                CC: elfutils-devel at sourceware dot org
  Target Milestone: ---

debuginfod's memory usage can fluctuate widely during the scan & groom phases. 
In the same places where we call sqlite3_db_release_memory() - and maybe more
of the same - we should subsequently call malloc_trim().

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

Di Chen <dichen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at sourceware dot org   |dichen at redhat dot com
                 CC|                            |dichen at redhat dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #1 from Di Chen <dichen at redhat dot com> ---
Hey Frank,

Are you talking about a change like this?

```
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 524be948..fb9c1661 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -69,6 +69,7 @@ extern "C" {
 #include <unistd.h>
 #include <fcntl.h>
 #include <netdb.h>
+#include <malloc.h>


 /* If fts.h is included before config.h, its indirect inclusions may not
@@ -4197,8 +4198,7 @@ void groom()

   (void) statfs_free_enough_p(db_path, "database"); // report sqlite
filesystem size

-  sqlite3_db_release_memory(db); // shrink the process if possible
-  sqlite3_db_release_memory(dbq); // ... for both connections
+  malloc_trim(0);
   debuginfod_pool_groom(); // and release any debuginfod_client objects we've
been holding onto

   fdcache.limit(0,0,0,0); // release the fdcache contents

```

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
Right, but the sqlite3* calls are still needed.  I'm especially curious whether
any impact from the malloc_trim() is measurable.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

--- Comment #3 from Mark Wielaard <mark at klomp dot org> ---
Also note mallopt
https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html

M_TRIM_THRESHOLD

    This is the minimum size (in bytes) of the top-most, releasable chunk that
will trigger a system call in order to return memory to the system.

    If this parameter is not set, the default value is set as 128 KiB and the
threshold is adjusted dynamically to suit the allocation patterns of the
program. If the parameter is set, the dynamic adjustment is disabled and the
value is set statically to the provided input.

    This parameter can also be set for the process at startup by setting the
environment variable MALLOC_TRIM_THRESHOLD_ to the desired value.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #4 from Di Chen <dichen at redhat dot com> ---
@fche, @mjw, @amerey

I don't have many clue about how to debug/measure this memory fluctuation
problem.

What I did:
  1) use GDB to check the call graph.
      thread_main_groom -> groom
        -> sqlite3_db_release_memory(db)
        -> sqlite3_db_release_memory(dbq)

  2) use valgrind to run a leak-check, but found nothing.
       $ valgrind --tool=memcheck --leak-check=full ./debuginfod/debuginfod

Could you give me some suggestion?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> ---
Yeah, not surprised that there are no leaks.  What this malloc_trim() and the
preceding sqlite3_db_release_memory() calls are supposed to do is to release
memory back to the operating system.  So perhaps a getrusage() call in between
could be informative.  i.e.:

   struct rusage pre,during,post;
   getrusage(RUSAGE_SELF, &pre);
   sqlite3_db_release_memory( ...);
   getrusage(RUSAGE_SELF, &during);
   malloc_trim();
   getrusage(RUSAGE_SELF, &post);

and then the ru_idrss value should show some decrease ... except on
linux, ru_idrss is not supplied by the kernel.  So instead something like
the VmSize or VmData fields in /proc/self/status, ugh?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #6 from Di Chen <dichen at redhat dot com> ---
Thanks Frank, I have debugged with rusage, but the difference between "during"
and "post" is not that obvious. I believe it's due to my small
.debuginfod.sqlite.

```
(gdb) p during
$2 = {ru_utime = {tv_sec = 0, tv_usec = 21790}, ru_stime = {tv_sec = 0, 
    tv_usec = 29003}, {ru_maxrss = 103836, __ru_maxrss_word = 103836}, {
    ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {
    ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 967, 
    __ru_minflt_word = 967}, {ru_majflt = 0, __ru_majflt_word = 0}, {
    ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, 
    __ru_inblock_word = 0}, {ru_oublock = 128, __ru_oublock_word = 128}, {
    ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, 
    __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {
    ru_nvcsw = 1229, __ru_nvcsw_word = 1229}, {ru_nivcsw = 1, 
    __ru_nivcsw_word = 1}}
(gdb) p post
$3 = {ru_utime = {tv_sec = 0, tv_usec = 21790}, ru_stime = {tv_sec = 0, 
    tv_usec = 30393}, {ru_maxrss = 103836, __ru_maxrss_word = 103836}, {
    ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {
    ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 967, 
    __ru_minflt_word = 967}, {ru_majflt = 0, __ru_majflt_word = 0}, {
    ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, 
    __ru_inblock_word = 0}, {ru_oublock = 128, __ru_oublock_word = 128}, {
    ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, 
    __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {
    ru_nvcsw = 1273, __ru_nvcsw_word = 1273}, {ru_nivcsw = 1, 
    __ru_nivcsw_word = 1}}


[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): grooming database
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): database record counts:
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): file d/e 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): file s 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): archive d/e 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): archive sref 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): archive sdef 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): buildids 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): filenames 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): fileparts 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): files scanned (#) 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): files scanned (mb) 0
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): index db size (mb) 21
[Thu 14 Mar 2024 03:22:47 AM GMT] (4738/4749): groomed database in 0.0129799s
```

Do you know where I can get a big .debuginfod.sqlite, so that I can measure the
performance on it?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #7 from Frank Ch. Eigler <fche at redhat dot com> ---
Right, the linux kernel does not share enough memory consumption data in
rusage, so you need to read /proc/self/statm or such and compare its snapshots.

For memory release, it's not so much the size of the database that matters, but
an amount of -activity- on the database before the release attempt.  It's the
memory allocation for temporary purposes (answering queries, adding data to the
database during scans) that we'd be hoping to release.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #8 from Di Chen <dichen at redhat dot com> ---
Got it. I have used GDB to make debuginfod stop at
  1) before malloc_trim(0)
    $ cat /proc/$$/statm 
    228487 3456 2976 48 0 19049 0
  2) after malloc_trim(0)
    $ cat /proc/$$/statm 
    228462 3456 2976 48 0 19024 0

debuginfod's VmSize and VmData did shrink some.

$ cat /proc/$$/status
Name:   debuginfod
...

VmSize:   913948 kB -> 913848 kB
VmData:    76060 kB -> 75960 kB

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #9 from Frank Ch. Eigler <fche at redhat dot com> ---
Yeah, OK, for a not-too-busy one, that sounds okay.  Can you show a proposed
final diff?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #10 from Di Chen <dichen at redhat dot com> ---
Yes, patch submitted:
https://sourceware.org/pipermail/elfutils-devel/2024q1/006916.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #11 from Frank Ch. Eigler <fche at redhat dot com> ---
further malloc_trim()s may be investigated in other spots.

    PR31103: debuginfod: periodically call malloc_trim(0)

    Add malloc_trim() for releasing memory which is allocated for
    temporary purposes, e.g. answering queries, adding data to the
    database during scans.  This patch just adds one call after the groom
    cycle, but others could be added around webapi query handling or
    scanning ops too.

    Signed-off-by: Di Chen <dichen@redhat.com>


(Sorry about committing with the wrong git-author!)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/31103] periodically malloc_trim()

[dichen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31103

--- Comment #12 from Di Chen <dichen at redhat dot com> ---
@fche
no problem, thanks for merging it :D

-- 
You are receiving this mail because:
You are on the CC list for the bug.