[Bug target/98491] New: [MIPS] ICE: in mode_size_inline, with -mmsa

["xry111 at mengyan1223 dot wang" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98491

            Bug ID: 98491
           Summary: [MIPS] ICE: in mode_size_inline, with -mmsa
           Product: gcc
           Version: 10.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: xry111 at mengyan1223 dot wang
  Target Milestone: ---

I'm building a system with Linux From Scratch approach on a Loongson-3A4000
(mips64el, with MSA support).  I tried to build GCC-10.2.0 but it crashes
building other packages, with `-mmsa`.

I investigated a little and it shown a simple program could trigger the ICE:

$ cat bug.c
void foo()
{
  double x = 1.0;
}
$ cc bug.c -c
bug.c:3:10: internal compiler error: in mode_size_inline, at
./insn-modes-inline.h:18

The problem is pinpointed at gcc/config/mips/mips.c line 2895:

    return mips_symbol_insns (symbol_type, MAX_MACHINE_MODE);

In mips_symbol_insns:

    if (MSA_SUPPORTED_MODE_P (mode))
      return 0;

MSA_SUPPORTED_MODE_P is defined as:

#define MSA_SUPPORTED_MODE_P(MODE)          \
  (ISA_HAS_MSA                      \
   && GET_MODE_SIZE (MODE) == UNITS_PER_MSA_REG     \
   && (GET_MODE_CLASS (MODE) == MODE_VECTOR_INT     \
       || GET_MODE_CLASS (MODE) == MODE_VECTOR_FLOAT))

When -mmsa is used, ISA_HAS_MSA is expanded to `true`.  And GET_MODE_SIZE is
expanded to a call to mode_to_bytes, which is defined:

ALWAYS_INLINE poly_uint16
mode_to_bytes (machine_mode mode)
{
#if GCC_VERSION >= 4001
  return (__builtin_constant_p (mode)
      ? mode_size_inline (mode) : mode_size[mode]);
#else
  return mode_size[mode];
#endif
}

Here `mode` is MAX_MACHINE_MODE, which equals to NUM_MACHINE_MODES, the size of
array `mode_size`.  And, there is an assertion in mode_size_inline:

gcc_assert (mode >= 0 && mode < NUM_MACHINE_MODES);

So, if __builtin_constant_p is evaluated `true`, the assertion will be
triggered.  Otherwise, we have an out-of-bound array access.  Anyway it is
wrong.