[Bug target/112548] New: 5% exec time regression in 429.mcf on AMD zen4 CPU

[Bug target/112548] New: 5% exec time regression in 429.mcf on AMD zen4 CPU

[fkastl at suse dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

            Bug ID: 112548
           Summary: 5% exec time regression in 429.mcf on AMD zen4 CPU
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: fkastl at suse dot cz
  Target Milestone: ---
              Host: x86_64-linux
            Target: x86_64-linux

As seen here:

https://lnt.opensuse.org/db_default/v4/SPEC/graph?plot.0=991.60.0

between commits

g:4ea36076d66eea0f (2023-11-02 02:54)

and

g:c3847ca0571e5ace (2023-11-03 03:35)

there's about 5% slowdown.

The test is compiled with -Ofast -march=native -mtune=native lto and pgo. It is
run on an amd zen4 machine.

[Bug target/112548] 5% exec time regression in 429.mcf on AMD zen4 CPU

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |missed-optimization,
                   |                            |needs-bisection

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
Possibly r14-5076-g01c18f58d37865

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Filip Kastl <pheeck at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|needs-bisection             |
                 CC|                            |rdapp at gcc dot gnu.org
            Summary|[14 regression] 5% exec     |[14 regression] 5% exec
                   |time regression in 429.mcf  |time regression in 429.mcf
                   |on AMD zen4 CPU             |on AMD zen4 CPU (since
                   |                            |r14-5076-g01c18f58d37865)

--- Comment #2 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Indeed, I bisected the slowdown to r14-5076-g01c18f58d37865

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #3 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Btw, the slowdown seems specific to PGO+LTO, with PGO or LTO by itself the
benchmarks execution times are relatively stable:
https://lnt.opensuse.org/db_default/v4/SPEC/graph?plot.0=991.60.0&plot.1=992.60.0&plot.2=962.60.0

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #4 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Judging by the graph it looks like it was slow before, then got faster and now
slower again.  Is there some more info on why it got faster in the first place?
 Did the patch reverse something or is it rather a secondary effect?  I don't
have a zen4 handy to check.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #5 from Filip Kastl <pheeck at gcc dot gnu.org> ---
(In reply to Robin Dapp from comment #4)
> Judging by the graph it looks like it was slow before, then got faster and
> now slower again.  Is there some more info on why it got faster in the first
> place?  Did the patch reverse something or is it rather a secondary effect? 
> I don't have a zen4 handy to check.

Don't know if this helps but I bisected the speedup to
g:b583a2940af90d03f535648fef111cb158933f7d.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |14.0

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #6 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Honestly, I don't know how to analyze/debug this without a zen4, in particular
as it only seems to happen with PGO.  I tried locally but of course the
execution time doesn't change (same as with zen3 according to the database).
Is there a way to obtain the binaries in order to tell a difference?

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[law at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Jeffrey A. Law <law at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |law at gcc dot gnu.org
           Priority|P3                          |P2

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #7 from Robin Dapp <rdapp at gcc dot gnu.org> ---
I built executables with and without the commit (-Ofast -march=znver4 -flto). 
There is no difference so it must really be something that happens with PGO.
I'd really need access to a zen4 box or the pgo executables at least.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #8 from Filip Kastl <pheeck at gcc dot gnu.org> ---
(In reply to Robin Dapp from comment #7)
> I built executables with and without the commit (-Ofast -march=znver4
> -flto).  There is no difference so it must really be something that happens
> with PGO.
> I'd really need access to a zen4 box or the pgo executables at least.

I'd like to help but I'm afraid I cannot send you the SPEC binaries with PGO
applied since SPEC is licensed nor can I give you access to a Zen4 computer. I
suppose someone else will have to analyze this bug.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #9 from Sam James <sjames at gcc dot gnu.org> ---
(In reply to Filip Kastl from comment #8)
> I'd like to help but I'm afraid I cannot send you the SPEC binaries with PGO
> applied since SPEC is licensed nor can I give you access to a Zen4 computer.
> I suppose someone else will have to analyze this bug.

Could you perhaps send only the gcda files so Robin can build again with
-fprofile-use?

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #10 from Robin Dapp <rdapp at gcc dot gnu.org> ---
(In reply to Sam James from comment #9)
> (In reply to Filip Kastl from comment #8)
> > I'd like to help but I'm afraid I cannot send you the SPEC binaries with PGO
> > applied since SPEC is licensed nor can I give you access to a Zen4 computer.
> > I suppose someone else will have to analyze this bug.
> 
> Could you perhaps send only the gcda files so Robin can build again with
> -fprofile-use?

Yes, that would be helpful.

Or Filip builds the executables himself and posts (some of) the difference
here.  Maybe that also gets us a bit closer to the problem.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #11 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Created attachment 57698
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57698&action=edit
gcda data for the commit g:c3847ca0571e5ace - after the change

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #12 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Created attachment 57699
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57699&action=edit
gcda data for the commit g:4ea36076d66eea0f - before the change

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #13 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Hmm. It will be better to have the gcda data for the Robin's commit and the
commit before it. I'll go generate those.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Filip Kastl <pheeck at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #57698|0                           |1
        is obsolete|                            |

--- Comment #14 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Created attachment 57700
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57700&action=edit
gcda data for robin's commit

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Filip Kastl <pheeck at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #57699|0                           |1
        is obsolete|                            |

--- Comment #15 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Created attachment 57701
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57701&action=edit
gcda data for the commit before robin's commit

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #16 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Thank you!

I'm having a problem with the data, though.
Compiling with -Ofast -march=znver4 -mtune=znver4 -flto -fprofile-use=/tmp.
Would you mind showing your exact final options for compilation of e.g.
pbeampp.cc?

I see, similar-ish for both commits:
pbeampp.c:119:8: error: number of counters in profile data for function
'primal_bea_mpp' does not match its profile data (counter 'arcs', expected 20
and have 22) [-Werror=coverage-mismatch]

output.c:87:1: error: corrupted profile info: number of executions for edge 3-4
thought to be 1
output.c:87:1: error: corrupted profile info: number of executions for edge 3-5
thought to be -1
output.c:87:1: error: corrupted profile info: number of iterations for basic
block 5 thought to be -1

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #17 from Filip Kastl <pheeck at gcc dot gnu.org> ---
I have this in the SPEC .cfg file:

OPTIMIZE     = -Ofast -g -march=native -mtune=native  -flto=32

So the only difference I see is the inclusion of -g.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #18 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Hmm, doesn't help unfortunately.  A full command line for me looks like:

x86_64-pc-linux-gnu-gcc -c -o pbeampp.o -DSPEC_CPU -DNDEBUG  -DWANT_STDC_PROTO 
-Ofast -march=znver4 -mtune=znver4 -flto=32 -g -fprofile-use=/tmp
-SPEC_CPU_LP64         pbeampp.c.

Could you verify if it's exactly the same for you?  Maybe it would also help if
you explicitly specified znver4?

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #19 from Filip Kastl <pheeck at gcc dot gnu.org> ---
There's indeed another difference. In my case, gcc gets called with -std=gnu99.
Otherwise, I think the options are the same.

gcc -std=gnu99 -c -o pbeampp.o -DSPEC_CPU -DNDEBUG  -DWANT_STDC_PROTO 
-fprofile-use    -Ofast -g -march=native -mtune=native  -flto=32       
-DSPEC_CPU_LP64         pbeampp.c

If adding -std=gnu99 doesn't help, I can try to generate the gcda with znver4
instead of native as you suggested.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #20 from Robin Dapp <rdapp at gcc dot gnu.org> ---
No change with -std=gnu99 unfortunately.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #21 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Created attachment 57703
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57703&action=edit
gcda data for the commit before robin's commit (v2)

Here are the gcda files generated with -march=znver4 -mtune=znver4 instead of
native. Only for the earlier commit. Will send data for the Robin's commit if
this works.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #22 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Still the same problem unfortunately.

I'm a bit out of ideas - maybe your compiler executables could help?

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #23 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Yeah I also don't know what else to do to make the gcda files work for you :-/

I can send you my compiler binaries but you should have exactly the same if you
compile from the same commit (if I'm not mistaken). I checked that I don't
compile with any special ./configure options:

../src/configure --disable-bootstrap --enable-checking --disable-libsanitizer
--prefix=/home/fkastl/gcc/inst --enable-languages=c,lto

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #24 from Robin Dapp <rdapp at gcc dot gnu.org> ---
I rebuilt GCC from scratch with your options but still have the same problem. 
Could our sources differ?  My SPEC version might not be the most recent but I'm
not aware that mcf changed at some point.

Just to be sure: I'm using r14-5075-gc05f748218a0d5 as the "before" commit.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[law at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #25 from Jeffrey A. Law <law at gcc dot gnu.org> ---
Well, at least in theory SPEC isn't supposed to be changing the sources or
validation criteria on us.  So while our copy may be old, I would expect it's
still the same as Filip's.

That doesn't resolve any issues here though.  It's not clear how best to
proceed.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #26 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Yes, the "before" is r14-5075-gc05f748218a0d5.

----

I just tried to take the gcda data and use them to compile mcf on another
machine. I also ran into

output.c:87:1: error: corrupted profile info: number of executions for edge 3-4
thought to be 1
output.c:87:1: error: corrupted profile info: number of executions for edge 3-5
thought to be -1
output.c:87:1: error: corrupted profile info: number of iterations for basic
block 5 thought to be -1

so the problem probably isn't that our sources or compilation options differ. I
used the exact same config on both machines.

I'm wondering if I'm generating the gcda files right. I have this in my SPEC
.cfg file

OPTIMIZE     = -Ofast -g -march=native -mtune=native  -flto=32

PASS1_CFLAGS     = -fprofile-generate
PASS1_CXXFLAGS   = -fprofile-generate
PASS1_FFLAGS     = -fprofile-generate
PASS1_LDFLAGS    = -fprofile-generate

PASS2_CFLAGS     = -fprofile-use
PASS2_CXXFLAGS   = -fprofile-use
PASS2_FFLAGS     = -fprofile-use
PASS2_LDFLAGS    = -fprofile-use

I run the benchmark and afterwards I go to

benchspec/CPU2006/429.mcf/build/build_peak_<name of the cfg>.0000

and copy gcda files from there.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #27 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Can you try it with a simpler (non SPEC) test?  Maybe there is still something
weird happening with SPEC's scripting.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[pheeck at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #28 from Filip Kastl <pheeck at gcc dot gnu.org> ---
Created attachment 57710
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57710&action=edit
gcda data for himeno

I've tried sharing non-SPEC gcda data between machines. I used this benchmark 

https://raw.githubusercontent.com/kowsalyaChidambaram/Himeno-Benchmark/master/himenobmtxpa.c

I generated the gcda data on a zen4 machine

gcc.sh himenobmtxpa.c -std=c99 -Ofast -march=znver4 -mtune=znver4 -g
-fprofile-generate

and used them on another machine

gcc.sh himenobmtxpa.c -std=c99 -Ofast -march=znver4 -mtune=znver4 -g
-fprofile-use

and everything went fine.

[Bug target/112548] [14 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

--- Comment #29 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Yes, that also appears to work here.  There was no lto involved this time?
Now we need to figure out what's different with SPEC.

[Bug target/112548] [14/15 regression] 5% exec time regression in 429.mcf on AMD zen4 CPU (since r14-5076-g01c18f58d37865)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112548

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|14.0                        |14.2

--- Comment #30 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 14.1 is being released, retargeting bugs to GCC 14.2.