[Bug malloc/30723] New: posix_memalign performance regression

[Bug malloc/30723] New: posix_memalign performance regression

[toolybird at tuta dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

            Bug ID: 30723
           Summary: posix_memalign performance regression
           Product: glibc
           Version: 2.38
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: malloc
          Assignee: unassigned at sourceware dot org
          Reporter: toolybird at tuta dot io
  Target Milestone: ---

Already reported on the glibc list [1] but creating a ticket here for tracking
purposes. This is currently breaking ffmpeg based apps [2]

[1] https://sourceware.org/pipermail/libc-alpha/2023-August/150653.html
[2] https://bugs.archlinux.org/task/79300

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] posix_memalign performance regression in glibc-2.38 (since 24cdd6c71debfd10a9f7cb217fe2a2c4c486ed6f?)

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dj at redhat dot com,
                   |                            |sam at gentoo dot org,
                   |                            |xry111 at xry111 dot site
            Summary|posix_memalign performance  |posix_memalign performance
                   |regression                  |regression in glibc-2.38
                   |                            |(since
                   |                            |24cdd6c71debfd10a9f7cb217fe
                   |                            |2a2c4c486ed6f?)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] posix_memalign performance regression in glibc-2.38 (since 24cdd6c71debfd10a9f7cb217fe2a2c4c486ed6f?)

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://github.com/mpv-play
                   |                            |er/mpv/issues/12076

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] posix_memalign performance regression in glibc-2.38 (since 24cdd6c71debfd10a9f7cb217fe2a2c4c486ed6f?)

[freswa at archlinux dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Frederik Schwan <freswa at archlinux dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |freswa at archlinux dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] posix_memalign performance regression in glibc-2.38 (since 24cdd6c71debfd10a9f7cb217fe2a2c4c486ed6f?)

[dilfridge at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Andreas K. Huettel <dilfridge at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |https://bugs.archlinux.org/
                   |                            |task/79300
                 CC|                            |dilfridge at gentoo dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] posix_memalign performance regression in glibc-2.38 (since 24cdd6c71debfd10a9f7cb217fe2a2c4c486ed6f?)

[kocelfc at tutanota dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Kostadin Shishmanov <kocelfc at tutanota dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kocelfc at tutanota dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Repeated calls posix_memalign calls produce long free lists, high fragmentation

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
            Summary|posix_memalign performance  |Repeated calls
                   |regression in glibc-2.38    |posix_memalign calls
                   |(since                      |produce long free lists,
                   |24cdd6c71debfd10a9f7cb217fe |high fragmentation
                   |2a2c4c486ed6f?)             |
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2023-08-09
                 CC|                            |fweimer at redhat dot com
           Assignee|unassigned at sourceware dot org   |fweimer at redhat dot com

--- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
Patch posted:

[PATCH] malloc: Enable merging of remainders in memalign (bug 30723)
<https://inbox.sourceware.org/libc-alpha/877cq4yrpg.fsf@oldenburg.str.redhat.com/T/>

We may need additional heuristics to bound the free list scanning.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Repeated posix_memalign calls produce long free lists, high fragmentation

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Repeated calls              |Repeated posix_memalign
                   |posix_memalign calls        |calls produce long free
                   |produce long free lists,    |lists, high fragmentation
                   |high fragmentation          |

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Repeated posix_memalign calls produce long free lists, high fragmentation

[toolybird at tuta dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

--- Comment #2 from Toolybird <toolybird at tuta dot io> ---
Hey thanks for the patch! Unfortunately multiple testers in the downstream Arch
ticket are still getting 100% CPU when using the "mpv" app. Please see this
comment [1] for a supposed reproducer (I haven't tested it yet...my role is
downstream bug wrangler, so I'm just the message relay here currently..)

[1] https://bugs.archlinux.org/task/79300#comment220881

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Repeated posix_memalign calls produce long free lists, high fragmentation

[toolybird at tuta dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

--- Comment #3 from Toolybird <toolybird at tuta dot io> ---
Latest patch [1] on top of the previous one appears to have done the trick.
Thanks so much for all your efforts. We will get this out to users once it hits
the tree (and hopefully the 2.38 branch).

[1]
https://inbox.sourceware.org/libc-alpha/87pm3uajev.fsf@oldenburg.str.redhat.com/

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Repeated posix_memalign calls produce long free lists, high fragmentation

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

--- Comment #4 from Florian Weimer <fweimer at redhat dot com> ---
First part committed:

commit 542b1105852568c3ebc712225ae78b8c8ba31a78
Author: Florian Weimer <fweimer@redhat.com>
Date:   Fri Aug 11 11:18:17 2023 +0200

    malloc: Enable merging of remainders in memalign (bug 30723)

    Previously, calling _int_free from _int_memalign could put remainders
    into the tcache or into fastbins, where they are invisible to the
    low-level allocator.  This results in missed merge opportunities
    because once these freed chunks become available to the low-level
    allocator, further memalign allocations (even of the same size are)
    likely obstructing merges.

    Furthermore, during forwards merging in _int_memalign, do not
    completely give up when the remainder is too small to serve as a
    chunk on its own.  We can still give it back if it can be merged
    with the following unused chunk.  This makes it more likely that
    memalign calls in a loop achieve a compact memory layout,
    independently of initial heap layout.

    Drop some useless (unsigned long) casts along the way, and tweak
    the style to more closely match GNU on changed lines.

    Reviewed-by: DJ Delorie <dj@redhat.com>

Second part is still under review, I need to send a v2.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Repeated posix_memalign calls produce long free lists, high fragmentation

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

--- Comment #5 from Florian Weimer <fweimer at redhat dot com> ---
Second part committed:

commit 0dc7fc1cf094406a138e4d1bcf9553e59edcf89d
Author: Florian Weimer <fweimer@redhat.com>
Date:   Thu Aug 10 19:36:56 2023 +0200

    malloc: Remove bin scanning from memalign (bug 30723)

    On the test workload (mpv --cache=yes with VP9 video decoding), the
    bin scanning has a very poor success rate (less than 2%).  The tcache
    scanning has about 50% success rate, so keep that.

    Update comments in malloc/tst-memalign-2 to indicate the purpose
    of the tests.  Even with the scanning removed, the additional
    merging opportunities since commit 542b1105852568c3ebc712225ae78b
    ("malloc: Enable merging of remainders in memalign (bug 30723)")
    are sufficient to pass the existing large bins test.

    Remove leftover variables from _int_free from refactoring in the
    same commit.

    Reviewed-by: DJ Delorie <dj@redhat.com>

I will handle the 2.38 backport after a brief delay.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Poor posix_memalign performance with long free lists

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Repeated posix_memalign     |Poor posix_memalign
                   |calls produce long free     |performance with long free
                   |lists, high fragmentation   |lists

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/30723] Poor posix_memalign performance with long free lists

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30723

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #6 from Florian Weimer <fweimer at redhat dot com> ---
The glibc 2.38 release branch has been fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.