[Bug tree-optimization/114557] ehcleanup cleanup_empty_eh_merge_phis eats a lot of memory

["rguenth at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114557

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2024-04-02
     Ever confirmed|0                           |1

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
The main reason is that the "large" PHI nodes are never re-used for large
allocations and that the GTY ((deletable (""))) for the free_phinodes
array doesn't seem to be effective for this testcase.

diff --git a/gcc/tree-phinodes.cc b/gcc/tree-phinodes.cc
index ddd731323e1..44dc345a3b7 100644
--- a/gcc/tree-phinodes.cc
+++ b/gcc/tree-phinodes.cc
@@ -223,6 +223,12 @@ release_phi_node (gimple *phi)
       delink_imm_use (imm);
     }

+  if (len - 2 >= NUM_BUCKETS - 2)
+    {
+      ggc_free (phi);
+      return;
+    }
+
   bucket = len > NUM_BUCKETS - 1 ? NUM_BUCKETS - 1 : len;
   bucket -= 2;
   vec_safe_push (free_phinodes[bucket], phi);

cuts memory usage to 10GB.  The free_phinodes buckets are also badly
designed - we always allocate power-of-two overall PHI node memory
through ideal_phi_node_len but have buckets for [2, ..., 10] actual
PHI nodes which likely makes us have exactly a single (maybe two)
free PHI node buckets ...

After ehcleanup (and with the above patch) we have

(gdb) p/r free_phinodes
$5 = {0x0, 0x0, 0x7ffff668e000, 0x0, 0x0, 0x0, 0x0, 0x7ffe8b3df000}

so as I said.  Two slots are used only.

(gdb) p/r (free_phinodes)[2].m_vecpfx
$7 = {m_alloc = 524287, m_using_auto_storage = 0, m_num = 265598}
(gdb) p/r (free_phinodes)[7].m_vecpfx
$8 = {m_alloc = 65535, m_using_auto_storage = 0, m_num = 59610}

After the next collection the array is actually empty:

(gdb) p/r free_phinodes
$9 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}

but memory usage doesn't shrink, likely either due to fragmentation or
due to stale references.  Maybe the 'deleteable' only gets effective
upon next collection but we do not collect anymore after hitting 10GB,
likely because we no longer expand the heap (hitting this peak is the
bug anyway).  We do re-collect in final(), but that doesn't reduce the
memory.  We do have a lot of blocks and edges and thus PHIs.