[Bug tree-optimization/104931] New: wrong-code with number_of_iterations_lt_to_ne

["rguenth at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104931

            Bug ID: 104931
           Summary: wrong-code with number_of_iterations_lt_to_ne
           Product: gcc
           Version: 11.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rguenth at gcc dot gnu.org
  Target Milestone: ---

The premake tool is miscompiled when building it with LTO on i586-linux,
resulting in it immediately segfaulting via

==9912== Invalid read of size 4
==9912==    at 0x8162378: UnknownInlinedFun (lapi.c:197)
==9912==    by 0x8162378: lua_rotate.constprop.0 (lapi.c:217)
==9912==    by 0x8063881: luaL_requiref (lauxlib.c:983)
==9912==    by 0x807DF76: luaL_openlibs (linit.c:64)
==9912==    by 0x8061128: main (premake_main.c:15)
==9912==  Address 0x43816dc is 12 bytes before a block of size 408 alloc'd
==9912==    at 0x4035EDB: realloc (in
/usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so)
==9912==    by 0x80806A3: luaM_realloc_ (lmem.c:86)
==9912==    by 0x807727E: luaD_reallocstack (ldo.c:182)

this can be reproduced with GCC 10 and GCC 11 but not on trunk.  After checking
out https://github.com/premake/premake-core.git do

make -f Bootstrap.mak linux CC="gcc-11 -m32" CFLAGS="-O2 -flto -g"

and ./bin/release/premake will then segfault.

I've narrowed this down to the first IPA CP clone of lua_rotate being
miscompiled, we enter number_of_iterations_lt_to_ne for

  exit condition [(struct TValue *) (_2 + 4294967272) + 12, + ,
24](no_overflow) < _2 + 4294967272

with delta being 4294967284, the step type is unsigned int.  The problem
is that for pointer IVs the step type has to be interpreted as signed, but
the code uses an unsigned FLOOR_MOD to compute the condition under which
the loop will not iterate which it computes to

  result:
    zero if (struct TValue *) (_2 + 4294967272) + 12 > _2 + 4294967292
    # of iterations 178956971, bounded by 0

which is always false (but not folded).  When making sure to use a signed
type to compute the modulo the miscompile is gone and we manage to
compute the correct

  result:
    zero if (struct TValue *) (_2 + 4294967272) + 12 > _2 + 4294967284(OVF)
    # of iterations 0(OVF), bounded by 0

I've failed to create a small testcase - there seem to be special circumstances
required that make us enter niter analysis with exactly this SCEV.  The
simplified testcase

struct X { int x[3]; };
static void reverse (struct X *from, struct X *to)
{
  do
    {
      struct X temp = *from;
      *from = *to;
      *to = temp;
      from++;
      to--;
    }
  while (from < to);
}
void
lua_rotate (struct X **L)
{
  struct X *y = *L;
  struct X *to = y - 1;
  struct X *from = y - 2;
  reverse (from, to);
} 

does not exhibit this problem.