[Bug libc/27651] New: Performance regression after updating to 2.33

[Bug libc/27651] New: Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

            Bug ID: 27651
           Summary: Performance regression after updating to 2.33
           Product: glibc
           Version: 2.33
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: rcd99 at zipmail dot com.br
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

I tested this many times to make sure: after upgrading to 2.33 I noticed a 15%
performance drop on 'Shadow of Tomb Raider' benchmark. Switching back to 2.32
(and nothing else!) the performance goes back to normal.

Although I'm on Slackware current, I also tried glibc 2.33 package from Arch
distro, and the behavior is precisely the same.

I want to be clear about this: I didn't change anything else apart from glibc.
Same drivers, same kernel, same everything! I'm 100% sure this is glibc's
fault.

Not sure if this helps, but I tried to track down which files are causing the
performance issue, and it turns out that it's something inside /lib64 folder. I
believe it's one of the following files or the combination of them:
ld-2.33.so
libc-2.33.so
libdl-2.33.so
libpthread-2.33.so
libpthread-2.33.so
libthread_db-1.0.so
libutil-2.33

Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |adhemerval.zanella at linaro dot o
                   |                            |rg

--- Comment #1 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Could you provide a perf profile before and after update to glibc 2.33 to show
where it might be the possible performance regression?

I guess that "'Shadow of Tomb Raider' benchmark" is a closed source binary not
easily distributable, so unless we have more information through either how to
reproduce (without actually buying the game) or with profile dumps this issue
will be most likely ignored.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #2 from Fulalas <rcd99 at zipmail dot com.br> ---
I'll be glad to help! But I've never done a perf profile before. Would you mind
telling me how I should do it?

BTW, there's free demo of 'Shadow Of The Tomb Raider', and it has a built-in
benchmark, just like the full version.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #3 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
The performance profile can be obtained by the 'perf' tool [1], which is
supported and distributed by mostly of the distributions (it is a kernel
project hosted within the linux kernel).

The most simple usage is the record/report, where you issue:

  $ perf record <program>

  $ perf report --stdio

The first command will generate a large file containing the performance data
obtained by the kernel while the second will parse and dump to console a perf
per sybmol cpu cycles spent (the default counter).

Doing it with both glibc 2.32 and glibc 2.33 we can have an idea of what is
happening.

My wild guess is maybe something related to a memory routine selection, I need
to check if something has changed on x86_64 side regarding it. 

[1] https://perf.wiki.kernel.org/index.php/Main_Page

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #4 from Fulalas <rcd99 at zipmail dot com.br> ---
Thanks! Since this is the first time I'm doing this, please let me know if I
did something wrong:

2.32:
https://www.mediafire.com/file/wkgrv36b5nsl2ac/2.32.txt

2.33:
https://www.mediafire.com/file/wkgrv36b5nsl2ac/2.32.txt

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #5 from Fulalas <rcd99 at zipmail dot com.br> ---
Oops! I pasted the wrong link for 2.33, sorry. Here it goes:
https://www.mediafire.com/file/9lgaiskstinquxw/2.33.txt

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #6 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Thanks, this was helpful.  It seems that on glibc 2.33 the binary is issuing a
lof of more select calls:

     1.83%  ShadowOfTheTomb  libc-2.33.so                   [.] __select
     1.79%  ShadowOfTheTomb  libc-2.33.so                   [.]
__libc_disable_asynccancel
     1.75%  ShadowOfTheTomb  libc-2.33.so                   [.]
__libc_enable_asynccancel

Although I can't really tell from the date what is actually generating them. 
Could you reran the 2.33 profile with the perf record '-g' options? It
generated call-graph records that can show from where the __select is being
called.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #7 from Fulalas <rcd99 at zipmail dot com.br> ---
Cool! :)

Here it goes: https://www.mediafire.com/file/oii46d32zyx9qoq/2.33-g.txt

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #8 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
I think I know what is happening here: the 2433d39b69743 changed select to use
pselect6 for all architectures.  However, on architecture with __NR_select
(x86_64 for instance) the kernel normalizes the passed timeout instead of
returning an invalid one. For instance, the passed timeval { 0, 5000000 } is
interpreted as { 5, 0 }.

Now that we use pselect we return EINVAL for such inputs.  The kernel
normalization is not really portable (POSIX does not specify that the timeout
should be normalized) and the code is subject to integer overflow (the tv_sec
can overflow). Another issue is the normalization was done only for
architecture that originally use __NR_select or __NR__newselect (which is the
wrapper for __NR_select on architecture with compat mode, such as i686).

I think we need to keep doing the normalization, at least for architecture that
used to define __NR_select/__NR__newselect.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #9 from Fulalas <rcd99 at zipmail dot com.br> ---
I can only understand some fragments of what you said. I hope there's a
solution for the performance regression. And if I can help somehow, please let
me know :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #10 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
If you could check whether the patch does help the performance regression it
would be helpful.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #11 from Fulalas <rcd99 at zipmail dot com.br> ---
I'll be happy to test it! If I git clone https://sourceware.org/git/glibc.git
am I going to have your patch?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #12 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
It is not upstream yet, but I created a personal branch based on master with
the fix applied [1].  If you could check this branch it would be helpful.

[1]
https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/azanella/bz27651-select

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #13 from Fulalas <rcd99 at zipmail dot com.br> ---
I'm happy to inform that your patch fixed the issue! Thank you very much! :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[rcd99 at zipmail dot com.br]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

Fulalas <rcd99 at zipmail dot com.br> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #14 from Fulalas <rcd99 at zipmail dot com.br> ---
Fixed

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #15 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Adhemerval Zanella
<azanella@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9d7c5cc38e58fb0923e88901f87174a511b61552

commit 9d7c5cc38e58fb0923e88901f87174a511b61552
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Wed Mar 31 13:53:34 2021 -0300

    linux: Normalize and return timeout on select (BZ #27651)

    The commit 2433d39b697, which added time64 support to select, changed
    the function to use __NR_pselect6 (or __NR_pelect6_time64) on all
    architectures.  However, on architectures where the symbol was
    implemented with __NR_select the kernel normalizes the passed timeout
    instead of return EINVAL.  For instance, the input timeval
    { 0, 5000000 } is interpreted as { 5, 0 }.

    And as indicated by BZ #27651, this semantic seems to be expected
    and changing it results in some performance issues (most likely
    the program does not check the return code and keeps issuing
    select with unormalized tv_usec argument).

    To avoid a different semantic depending whether which syscall the
    architecture used to issue, select now always normalize the timeout
    input.  This is a slight change for some ABIs (for instance aarch64).

    Checked on x86_64-linux-gnu and i686-linux-gnu.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|2.33                        |2.32

--- Comment #16 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
Fixed on 2.33 and I will backport to 2.32 as well.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[adhemerval.zanella at linaro dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

Adhemerval Zanella <adhemerval.zanella at linaro dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|2.32                        |2.33

--- Comment #17 from Adhemerval Zanella <adhemerval.zanella at linaro dot org> ---
My confusion, I meant fixed on 2.34 and I will backport to 2.33.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |2.34

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

--- Comment #18 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The release/2.33/master branch has been updated by Adhemerval Zanella
<azanella@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=8380ca5833ef2a11bf0162f2290f4f8c85ce3b90

commit 8380ca5833ef2a11bf0162f2290f4f8c85ce3b90
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Wed Mar 31 13:53:34 2021 -0300

    linux: Normalize and return timeout on select (BZ #27651)

    The commit 2433d39b697, which added time64 support to select, changed
    the function to use __NR_pselect6 (or __NR_pelect6_time64) on all
    architectures.  However, on architectures where the symbol was
    implemented with __NR_select the kernel normalizes the passed timeout
    instead of return EINVAL.  For instance, the input timeval
    { 0, 5000000 } is interpreted as { 5, 0 }.

    And as indicated by BZ #27651, this semantic seems to be expected
    and changing it results in some performance issues (most likely
    the program does not check the return code and keeps issuing
    select with unormalized tv_usec argument).

    To avoid a different semantic depending whether which syscall the
    architecture used to issue, select now always normalize the timeout
    input.  This is a slight change for some ABIs (for instance aarch64).

    Checked on x86_64-linux-gnu and i686-linux-gnu.

    (cherry picked from commit 9d7c5cc38e58fb0923e88901f87174a511b61552)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/27651] Performance regression after updating to 2.33

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27651

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P2                          |P1
           Severity|normal                      |critical
              Flags|                            |security-
                 CC|                            |fweimer at redhat dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.