[Bug middle-end/102519] New: [12 Regression] VRP Jump threader memory explosion

[Bug middle-end/102519] New: [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

            Bug ID: 102519
           Summary: [12 Regression] VRP Jump threader memory explosion
           Product: gcc
           Version: 12.0
            Status: UNCONFIRMED
          Keywords: memory-hog
          Severity: normal
          Priority: P3
         Component: middle-end
          Assignee: unassigned at gcc dot gnu.org
          Reporter: dje at gcc dot gnu.org
                CC: aldyh at gcc dot gnu.org, law at gcc dot gnu.org
  Target Milestone: ---

After the VRP jump threader replacement on Monday, September 27, memory usage
for some testcases, such as gcc.target/powerpc/rlwimi-1.c has doubled.  This is
causing new testsuite failures on AIX because of the default lower memory
limits on AIX.  The testcases previously compiled with 1GB data size for the
compiler, and now require 2GB data size.

Similar to the problem exposed by Martin Sebor's patch a few weeks ago, this
likely is a problem with memory allocation in the new pass.  A data structure
not placed in GC or not correctly tracked by GC.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

David Edelsohn <dje at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-09-28
             Status|UNCONFIRMED                 |NEW

--- Comment #1 from David Edelsohn <dje at gcc dot gnu.org> ---
Confirmed.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

Aldy Hernandez <aldyh at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |amacleod at redhat dot com

--- Comment #2 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
I'm going to guess this has something to do with extremely large basic blocks.
This has bitten us in the past, and is likely to be an issue for the solver. 
Regardless, I'll take a look.  Thanks for reporting David.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |12.0

--- Comment #3 from Richard Biener <rguenth at gcc dot gnu.org> ---
You can always also try to watch
https://gcc.opensuse.org/gcc-old/c++bench-czerny/random/ which is where we
gathered testcases from compile-time-hog and memory-hog bugzillas and they are
compiled at various optimization levels.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #4 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
Is there any way of reproducing this on a cross?  I've been waiting 15 minutes
for a "git fetch" on gcc111.fsffrance.org or gcc119.fsffrance.org.  I'll
continue trying in the background just in case.

FWIW, on a cross to --target=powerpc-ibm-aix7.1.2.0 --enable-languages=c I
don't see much of a difference on cc1 with -ftime-report:

 TOTAL                              :  67.25          5.84         73.41       
  518M

 TOTAL                              :  70.15          6.23         76.74       
  533M

That looks like a 2.9% increase.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #5 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Aldy Hernandez from comment #4)
> Is there any way of reproducing this on a cross?  I've been waiting 15
> minutes for a "git fetch" on gcc111.fsffrance.org or gcc119.fsffrance.org. 
> I'll continue trying in the background just in case.
> 
> FWIW, on a cross to --target=powerpc-ibm-aix7.1.2.0 --enable-languages=c I
> don't see much of a difference on cc1 with -ftime-report:
> 
>  TOTAL                              :  67.25          5.84         73.41    
> 518M
> 
>  TOTAL                              :  70.15          6.23         76.74    
> 533M
> 
> That looks like a 2.9% increase.

Memory usage increased rather than compile time :).

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #6 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #5)
> (In reply to Aldy Hernandez from comment #4)
> > Is there any way of reproducing this on a cross?  I've been waiting 15
> > minutes for a "git fetch" on gcc111.fsffrance.org or gcc119.fsffrance.org. 
> > I'll continue trying in the background just in case.
> > 
> > FWIW, on a cross to --target=powerpc-ibm-aix7.1.2.0 --enable-languages=c I
> > don't see much of a difference on cc1 with -ftime-report:
> > 
> >  TOTAL                              :  67.25          5.84         73.41    
> > 518M
> > 
> >  TOTAL                              :  70.15          6.23         76.74    
> > 533M
> > 
> > That looks like a 2.9% increase.
> 
> Memory usage increased rather than compile time :).

But isn't that what was reported?  A 2X increase in memory usage?  Or is the
518M / 533M displayed here not an accurate measure of memory usage?

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #7 from David Edelsohn <dje at gcc dot gnu.org> ---
Have you tried a native PPC64LE Linux build?

AIX defaults to 32 bit, and it's big endian. I wouldn't expect that to affect
the memory usage, but I'm mentioning it.

Did you try to compiler gcc/testsuite/gcc.target/powerpc/rlwimi-1.c ?

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #8 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
(In reply to David Edelsohn from comment #7)
> Have you tried a native PPC64LE Linux build?
> 
> AIX defaults to 32 bit, and it's big endian. I wouldn't expect that to
> affect the memory usage, but I'm mentioning it.
> 
> Did you try to compiler gcc/testsuite/gcc.target/powerpc/rlwimi-1.c ?

Yes.  That was the first thing I did.

I've also tried ppc32 with setarch ppc32 on gcc135.

ppc64le:
 TOTAL                              :  34.50         11.83         46.33       
  540M (OLD)
 TOTAL                              :  31.62         11.72         43.34       
  555M (NEW)

-ftime-report -O2 -m32 -mlittle-endian
 TOTAL                              :  32.17         11.35         43.53       
  518M (OLD)
 TOTAL                              :  31.89         11.89         43.79       
  533M (NEW)

-ftime-report -O2 -m32 -mbig-endian
 TOTAL                              :  29.07         11.24         40.31       
  518M (OLD)
 TOTAL                              :  33.68         12.66         46.34       
  533M (NEW)

These are all on a stage1 compiler

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #9 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
Created attachment 51519
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=51519&action=edit
patch to help diagnose issue with -ftime-report

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #10 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
The attached patch adds a separate TV_* timer to see the actual break down for
VRP and the VRP threader.

Could you incorporate this patch and run the problematic file with ./cc1
-ftime-report -O2?  I'd like to see the table at the end.  If you're so
inclined, could you also email me the entire -ftime-report so I can see the GC
output as it compiles?

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #11 from David Edelsohn <dje at gcc dot gnu.org> ---
tree VRP threader                  :   0.25 (  2%)   0.03 (  1%)   0.76 (  1%) 
7804k (  2%)

Despite that report output, prior to the two patches ending with
ef1e524fd87a679f5da06116029c66a84daac80 "Remove old VRP jump threader code" and
"Replace VRP threader with a hybrid forward threader.", I was able to compile
the testcase with an AIX limit of 1.25GB data size and now I need 2.25GB.

In other words, I don't need the entire patch series or even the cleanup patch,
just the replace VRP and remove old VRP patches.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #12 from David Edelsohn <dje at gcc dot gnu.org> ---
GCC non-quiet mode shows:

...
g_31_31_16 f_31_31_17 g_31_31_17 f_31_31_23 g_31_31_23 f_31_31_24 g_31_31_24
f_31_31_25 g_31_31_25 f_31_31_29 g_31_31_29 f_31_31_30 g_31_31_30 f_31_31_31
g_31_31_31
Analyzing compilation unit
Performing interprocedural optimizations
 <*free_lang_data> {heap 4160k} <visibility> {heap 4288k} <build_ssa_passes>
{heap 4288k} <opt_local_passes> {heap 4416k} <remove_symbols> {heap 9985k}
<targetclone> {heap 9985k} <free-fnsummary> {heap 9985k}Streaming LTO
 <whole-program> {heap 12M} <profile_estimate> {heap 12M} <icf> {heap 15M}
<devirt> {heap 15M} <cp> {heap 15M} <sra> {heap 15M} <cdtor> {heap 15M}
<fnsummary> {heap 15M} <inline> {heap 15M} <pure-const> {heap 16M} <modref>
{heap 16M} <free-fnsummary> {heap 16M} <static-var> {heap 16M} <single-use>
{heap 16M}
Assembling functions:
 f_0_0_0 g_0_0_0 f_0_0_1 g_0_0_1 f_0_0_2 g_0_0_2 f_0_0_7 g_0_0_7 f_0_0_8
g_0_0_8 f_0_0_9 g_0_0_9 f_0_0_15 g_0_0_15 f_0_0_16 g_0_0_16 f_0_0_17 g_0_0_17
f_0_0_23 
...
f_17_15_9 g_17_15_9 f_17_15_15 g_17_15_15 f_17_15_16 g_17_15_16 f_17_15_17
g_17_15_17 f_17_15_23
cc1: out of memory allocating 65536 bytes after a total of 1071778432 bytes

Somehow the VRP threader change provoked this change in memory usage and
failure mode.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #13 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
Created attachment 51520
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=51520&action=edit
avoid CFG  and SSA updates when possible

The VRP threader is updating SSAs and CFG even if nothing changes.  This could
have an effect outside the time/memory budget for the threader, but it's just a
guess.

I am testing this patch and will commit it shortly.  Sorry to keep shooting in
the dark here, but could you see if this helps?

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #14 from David Edelsohn <dje at gcc dot gnu.org> ---
I tried the patch, but, unfortunately, it exceeds the memory limit in the same
manner.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #15 from David Edelsohn <dje at gcc dot gnu.org> ---
I annotated execute_vrp_threader() to call getrusage() and print the size of
RSS around each call to threader.thread_jumps().  It consistently is growing,
but not in the threader itself.  Was the former VPR Threader intentionally or
implicitly freeing some other data allocated by the compiler that the new
threader is not cleaning up?

Assembling  functions:
 f_0_0_0
RSS = 569788
RSS = 569788
RSS = 569792
RSS = 569792
 g_0_0_0
RSS = 569876
RSS = 569876
RSS = 569880
RSS = 569880
 f_0_0_1
RSS = 569880
RSS = 569880
RSS = 569884
RSS = 569884
...
g_17_15_17
RSS = 783824
RSS = 783824
RSS = 783836
RSS = 783836
 f_17_15_23
RSS = 783892
RSS = 783892
RSS = 783896
RSS = 783896

cc1: out of memory allocating 65536 bytes after a total of 1071909664 bytes

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at redhat dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #16 from Aldy Hernandez <aldyh at redhat dot com> ---
On Wed, Sep 29, 2021 at 10:46 PM dje at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519
>
> --- Comment #15 from David Edelsohn <dje at gcc dot gnu.org> ---
> I annotated execute_vrp_threader() to call getrusage() and print the size of
> RSS around each call to threader.thread_jumps().  It consistently is growing,
> but not in the threader itself.  Was the former VPR Threader intentionally or
> implicitly freeing some other data allocated by the compiler that the new
> threader is not cleaning up?

Huh.  Very good insight.

:-)

diff --git a/gcc/tree-vrp.c b/gcc/tree-vrp.c
index 69a3ab0ea9d..c24c67f8874 100644
--- a/gcc/tree-vrp.c
+++ b/gcc/tree-vrp.c
@@ -4408,6 +4408,7 @@ hybrid_threader::~hybrid_threader ()
   delete m_threader;
   delete m_state;
   delete m_ranger;
+  delete m_query;

   scev_finalize ();
   loop_optimizer_finalize ();

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[law at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #17 from Jeffrey A. Law <law at gcc dot gnu.org> ---
Consider that pre-approved.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #18 from David Edelsohn <dje at gcc dot gnu.org> ---
Yea!  That fixes the *worst* of the memory growth problem for me.

It still is growing, but much more slowly.  RSS now grows from 569788 to 642076
for the final function, instead of 783896 before it died on an intermediate
function.

Definitely an improvement and good catch.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldot at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

Bernhard Reutner-Fischer <aldot at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aldot at gcc dot gnu.org

--- Comment #19 from Bernhard Reutner-Fischer <aldot at gcc dot gnu.org> ---
so as asked in
https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580485.html what about
dominance_info?

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at redhat dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #20 from Aldy Hernandez <aldyh at redhat dot com> ---
Doesn't make a difference.  If the blocks are stale, they need to be
reconstructed anyhow.  It's preexisting behavior in VRP anyhow.

I heard you the first time ;-).

On Thu, Sep 30, 2021 at 7:49 AM aldot at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519
>
> Bernhard Reutner-Fischer <aldot at gcc dot gnu.org> changed:
>
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |aldot at gcc dot gnu.org
>
> --- Comment #19 from Bernhard Reutner-Fischer <aldot at gcc dot gnu.org> ---
> so as asked in
> https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580485.html what about
> dominance_info?
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at redhat dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #21 from Aldy Hernandez <aldyh at redhat dot com> ---
However, if you care to test a patch, I'd be happy to review it.

On Thu, Sep 30, 2021 at 7:49 AM aldot at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519
>
> Bernhard Reutner-Fischer <aldot at gcc dot gnu.org> changed:
>
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |aldot at gcc dot gnu.org
>
> --- Comment #19 from Bernhard Reutner-Fischer <aldot at gcc dot gnu.org> ---
> so as asked in
> https://gcc.gnu.org/pipermail/gcc-patches/2021-September/580485.html what about
> dominance_info?
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #22 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Aldy Hernandez <aldyh@gcc.gnu.org>:

https://gcc.gnu.org/g:64dd46dbc682fbbc03a74e0298f7ac471c5e80f2

commit r12-3974-g64dd46dbc682fbbc03a74e0298f7ac471c5e80f2
Author: Aldy Hernandez <aldyh@redhat.com>
Date:   Thu Sep 30 02:19:36 2021 +0200

    Plug memory leak in hybrid_threader.

    Tested on x86-64 Linux.

    gcc/ChangeLog:

            PR middle-end/102519
            * tree-vrp.c (hybrid_threader::~hybrid_threader): Free m_query.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[aldyh at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

--- Comment #23 from Aldy Hernandez <aldyh at gcc dot gnu.org> ---
I have built a cross to ppcle on gcc135 (ppc64le) and then bisected the lowest
amount of memory ./cc1 -O2 can compile rlwimi-1.c (via ulimit -v).

Before the VRP threader replacement it could run with 271megs.  Current trunk
can run it with 289megs for a 6.6% increase.

I also tried valgrind both on a cross from x86-64 and on ppc64le.  Nothing
comes up except some existing sparseset_bit_p stuff.

I still cannot reproduce :-(.

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #24 from Richard Biener <rguenth at gcc dot gnu.org> ---
David, I believe the issue is fixed meanwhile.  Can you please confirm that?

[Bug middle-end/102519] [12 Regression] VRP Jump threader memory explosion

[dje at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102519

David Edelsohn <dje at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|WAITING                     |RESOLVED

--- Comment #25 from David Edelsohn <dje at gcc dot gnu.org> ---
Fixed