[Bug rtl-optimization/109187] [13 Regression] ICE: qsort checking failed: qsort comparator non-negative on sorted output: 1736258160 at -O2 since r13-5154-g733a1b777f16cd

["amonakov at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109187

--- Comment #2 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
This is caused by overflowing subtraction in autopref_rank_for_schedule:

      if (!irrel1 && !irrel2)
        /* Sort memory references from lowest offset to the largest.  */
        r = data1->offset - data2->offset;

When offsets are arbitrary this pattern (anti-pattern?) is inappropriate for
producing a less/equal/greater comparison result. The following (or variants)
is safe:

diff --git a/gcc/haifa-sched.cc b/gcc/haifa-sched.cc
index 4efaa9445..11bf10645 100644
--- a/gcc/haifa-sched.cc
+++ b/gcc/haifa-sched.cc
@@ -5686,7 +5686,7 @@ autopref_rank_for_schedule (const rtx_insn *insn1, const
rtx_insn *insn2)

       if (!irrel1 && !irrel2)
        /* Sort memory references from lowest offset to the largest.  */
-       r = data1->offset - data2->offset;
+       r = (data1->offset > data2->offset) - (data1->offset < data2->offset);
       else if (write)
        /* Schedule "irrelevant" insns before memory stores to resolve
           as many producer dependencies of stores as possible.  */