[Bug target/112411] ICE: SIGSEGV with --param=min-nondebug-insn-uid=2147483647 on powerpc64le-unknown-linux-gnu

["jakub at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112411

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |vmakarov at gcc dot gnu.org

--- Comment #8 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Note, on
void
foo (void)
{
}
./cc1 -quiet --param min-nondebug-insn-uid=0x2aaaa000 pr112411.c
still works but is fairly slow (various passes index arrays by INSN_UID and if
the uids are huge, it takes a lot of memory and compile time to clear them).
But
./cc1 -quiet --param min-nondebug-insn-uid=0x2aaaaaaa a.c

cc1: out of memory allocating 18446744065119617112 bytes after a total of
1626112 bytes
Seems lra does some vec handling by hand and uses an int variable for it:
lra.cc (check_and_expand_insn_recog_data):
  lra_insn_recog_data_len = index * 3 / 2 + 1;
Shall we change lra_insn_recog_data_len to size_t (and tweak comparisons to
avoid signed vs. unsigned compares)?
Or at least adjust the above such that the computation is say done in unsigned
HOST_WIDE_INT and use INT_MAX for lra_insn_recog_data_len on overflow rather
than negative value (including UB at compile time)?

With the following patch it compiles even with
./cc1 -quiet --param min-nondebug-insn-uid=0x40000000 a.c
but my 32GB RAM + 24GB swap was almost full with that.

--- gcc/params.opt.jj   2023-11-02 07:49:18.010852541 +0100
+++ gcc/params.opt      2023-12-06 18:55:57.045420935 +0100
@@ -779,7 +779,7 @@ Common Joined UInteger Var(param_min_loo
 The minimum threshold for probability of semi-invariant condition statement to
trigger loop split.

 -param=min-nondebug-insn-uid=
-Common Joined UInteger Var(param_min_nondebug_insn_uid) Param
+Common Joined UInteger Var(param_min_nondebug_insn_uid) Param IntegerRange(0,
1073741824)
 The minimum UID to be used for a nondebug insn.

 -param=min-size-for-stack-sharing=
--- gcc/lra.cc.jj       2023-12-05 13:17:29.642260866 +0100
+++ gcc/lra.cc  2023-12-06 19:14:52.117499420 +0100
@@ -764,11 +764,13 @@ static void
 check_and_expand_insn_recog_data (int index)
 {
   int i, old;
+  HOST_WIDE_INT len;

   if (lra_insn_recog_data_len > index)
     return;
   old = lra_insn_recog_data_len;
-  lra_insn_recog_data_len = index * 3 / 2 + 1;
+  len = index * HOST_WIDE_INT_C (3) / 2 + 1;
+  lra_insn_recog_data_len = MIN (len, INT_MAX);
   lra_insn_recog_data = XRESIZEVEC (lra_insn_recog_data_t,
                                    lra_insn_recog_data,
                                    lra_insn_recog_data_len);