[Bug other/113575] New: [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[Bug other/113575] New: [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[doko at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

            Bug ID: 113575
           Summary: [14 Regression] memory hog building insn-opinit.o
                    (i686-linux-gnu -> riscv64-linux-gnu)
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
          Assignee: unassigned at gcc dot gnu.org
          Reporter: doko at gcc dot gnu.org
  Target Milestone: ---

seen with trunk 20240121, building a riscv64-linux-gnu cross compiler on
i686-linux-gnu:

cc1plus: out of memory allocating 65536 bytes after a total of 3543261184 bytes
make[5]: *** [Makefile:1198: insn-opinit.o] Error 1
make[5]: *** Waiting for unfinished jobs....

Configured with: -v
         --with-pkgversion='Debian 14-20240121-1'
         --with-bugurl='file:///usr/share/doc/gcc-14/README.Bugs'
         --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2,rust
         --prefix=/usr
         --with-gcc-major-version-only
         --program-suffix=-14
         --enable-shared
         --enable-linker-build-id
         --libexecdir=/usr/libexec
         --without-included-gettext
         --enable-threads=posix
         --libdir=/usr/lib
         --enable-nls
         --with-sysroot=/
         --enable-clocale=gnu
         --enable-libstdcxx-debug
         --enable-libstdcxx-time=yes
         --with-default-libstdcxx-abi=new
         --enable-libstdcxx-backtrace
         --enable-gnu-unique-object
         --disable-libitm
         --disable-libquadmath
         --disable-libquadmath-support
         --enable-plugin
         --enable-default-pie
         --with-system-zlib
         --enable-libphobos-checking=release
         --without-target-system-zlib
         --enable-multiarch
         --disable-werror
         --disable-multilib
         --with-arch=rv64gc
         --with-abi=lp64d
         --enable-checking=yes
         --build=i686-linux-gnu
         --host=i686-linux-gnu
         --target=riscv64-linux-gnu
         --program-prefix=riscv64-linux-gnu-
         --includedir=/usr/riscv64-linux-gnu/include

Other cross builds on i686-linux-gnu targeting amd64 arm64 s390x ppc64el armhf
built ok.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Which version is your host compiler?

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=111600

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
riscv has many many rtl patterns which definitely does not help insn-opinit
size.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Sam James <sjames at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |84402
                 CC|                            |sjames at gcc dot gnu.org

--- Comment #3 from Sam James <sjames at gcc dot gnu.org> ---
yeah, the split needs doing anyway, but it's really especially bad on riscv..


Referenced Bugs:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84402
[Bug 84402] [meta] GCC build system: parallelism bottleneck

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[doko at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #4 from Matthias Klose <doko at gcc dot gnu.org> ---
same version, r14-8314-g29f931e39f2

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Robin Dapp <rdapp at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rdapp at gcc dot gnu.org

--- Comment #5 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Yes, this is a known issue and it's due to our large number of patterns. 
Contrary to insn-emit insn-opinit cannot be split that easily.  It would
probably need a tree-like approach or similar.
I wouldn't see this as a regression in the classical sense as we just have many
more patterns because of the vector extension.
Is increasing the available memory an option in the meantime or does this
urgently require fixing?

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |14.0

--- Comment #6 from Richard Biener <rguenth at gcc dot gnu.org> ---
The source isn't unreasonable, we should see what takes all the memory there.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #7 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Ok, I'm going to check.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #8 from Richard Biener <rguenth at gcc dot gnu.org> ---
My host compiler (x86_64, older trunk) uses "just" 800MB.  3.5GB looks like a
runaway?  What uarch is your i586 compiler targeting?

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2024-01-24
     Ever confirmed|0                           |1

--- Comment #9 from Richard Biener <rguenth at gcc dot gnu.org> ---
Ah, I can reproduce - there are allocation spikes with a i586 compiler,
always during DF which oddly enough do not happen with a x86_64 compiler.

The course of action would be to reduce the testcase to a single function
(I've starting from a x86_64->riscv cross tree preprocessed source with -m32
added, so a bit of a "wrong" testcase).  Just keeping init_all_optabs
seems to reproduce it.

Reducing the number of stores to ena[] makes it eventually fit - cutting
in half seems to work for me but I still see a 1.5GB peak.

Oddly enough a checking enabled x86_64 hosted compiler shows similar bad
performance.

IIRC DF issues with many adjacent stores are not unheard of.  Possibly
on x86_64 store-merging helps to avoid this.

That said, a "fix" could be to adjust the insn-opinit generator to
emit multiple init_all_optabs functions, doing 1000 at a time.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #10 from Richard Biener <rguenth at gcc dot gnu.org> ---
btw, -fno-var-tracking also greatly improves compile-time (but does nothing to
memory use).  Compiling with -O1 reduces memory use to 300MB even when
var-tracking is enabled.  So an option might be to force building this
generator file with -O1 (I can hardly see anything in there that would
require more).  On the DF side this replaces LIVE with LR_IN IIRC.
It's also in line with us suggesting -O1 for (large) machine generated code ...

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[doko at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #11 from Matthias Klose <doko at gcc dot gnu.org> ---
> Is increasing the available memory an option
> in the meantime or does this urgently require fixing?

there is a buffer of 500mb, but it's already using 3.5gb.  That probably would
work building without any parallelism with some Makefile changes, but building
the whole compiler sequentially because of that is not a good option.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #12 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Created attachment 57209
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57209&action=edit
Tentative

I tested the attached "fix".  On my machine with 13.2 host compiler it reduced
the build time for insn-opinit.cc from > 4 mins to < 2 mins and the memory
usage from >1G to 600ish M.  I didn't observe 3.5G before, though.

For now I just went with an arbitrary threshold of 5000 patterns and splitting
into 10 functions.  After testing on x86 and aarch64 I realized that both have
<3000 patterns so right now it would only split riscv's init function.

Or rather the other way, i.e. splitting into fixed-size chunks (of 1000)
instead?

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #13 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Robin Dapp from comment #12)
> Created attachment 57209 [details]
> Tentative
> 
> I tested the attached "fix".  On my machine with 13.2 host compiler it
> reduced the build time for insn-opinit.cc from > 4 mins to < 2 mins and the
> memory usage from >1G to 600ish M.  I didn't observe 3.5G before, though.
> 
> For now I just went with an arbitrary threshold of 5000 patterns and
> splitting into 10 functions.  After testing on x86 and aarch64 I realized
> that both have <3000 patterns so right now it would only split riscv's init
> function.
> 
> Or rather the other way, i.e. splitting into fixed-size chunks (of 1000)
> instead?

Yeah, I'd simplify it by doing exactly that.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[rdapp at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #14 from Robin Dapp <rdapp at gcc dot gnu.org> ---
Ok, running tests with the adjusted version and going to post a patch
afterwards.

However, during a recent run compiling insn-recog took 2G and insn-emit-7 as
well as insn-emit-10 required > 1.5G each.  Looks like they could cause
problems as well then?  The insn-emit files can be split into 20 instead of 10
which might help but insn-recog I haven't had a look at yet.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

--- Comment #15 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Robin Dapp <rdapp@gcc.gnu.org>:

https://gcc.gnu.org/g:861997a9c7088da25ed4dc0bd339060ca063514f

commit r14-8457-g861997a9c7088da25ed4dc0bd339060ca063514f
Author: Robin Dapp <rdapp@ventanamicro.com>
Date:   Wed Jan 24 17:28:31 2024 +0100

    genopinit: Split init_all_optabs [PR113575].

    init_all_optabs initializes > 10000 patterns for riscv targets.  This
    leads to pathological situations in dataflow analysis (which can occur
    with many adjacent stores).
    To alleviate this this patch makes genopinit split the init_all_optabs
    function into several init_optabs_xx functions that each initialize 1000
    patterns.

    With this change insn-opinit.cc's compilation time is reduced from 4+
    minutes to 1:30 and memory consumption decreases from 1.2G to 630M.

    gcc/ChangeLog:

            PR other/113575

            * genopinit.cc (main): Split init_all_optabs into functions
            of 1000 patterns each.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[law at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Jeffrey A. Law <law at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |law at gcc dot gnu.org

--- Comment #16 from Jeffrey A. Law <law at gcc dot gnu.org> ---
Fixed on the trunk.

[Bug other/113575] [14 Regression] memory hog building insn-opinit.o (i686-linux-gnu -> riscv64-linux-gnu)

[law at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113575

Jeffrey A. Law <law at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #17 from Jeffrey A. Law <law at gcc dot gnu.org> ---
Forgot to change state.  Fixed on the trunk.