[Bug middle-end/50708] Infinite loop between rshift_double and lshift_double if count is LONG_MIN

["mikpe at it dot uu.se" <gcc-bugzilla@gcc.gnu.org>]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50708

--- Comment #5 from Mikael Pettersson <mikpe at it dot uu.se> 2011-10-30 20:35:03 UTC ---
The regression started with r158372:
http://gcc.gnu.org/ml/gcc-cvs/2010-04/msg00478.html

Here's what seems to have happened:
1. Anatoly Sokolov suggested to add a bunch of double_int_ functions:
http://gcc.gnu.org/ml/gcc-patches/2010-04/msg00482.html.

2. Eric Botcazou replied about the added double_int_rshift (a wrapper around
rshift_double) that the comment about the COUNT parameter was wrong, and that
rshift_double requires it to be positive (double_int_rshift stated that a
negative COUNT turned it into a left-shift):
http://gcc.gnu.org/ml/gcc-patches/2010-04/msg00557.html.

3. In the final version of Anatoly's patch committed in r158360:
http://gcc.gnu.org/ml/gcc-cvs/2010-04/msg00466.html, the comment above
rshift_double was changed to allow a negative COUNT as meaning a left-shift. 
However, the code in rshift_double was not changed, and still assumed that
COUNT would be positive.

4. Richard Guenther followed up with a cleanup patch to move rshift_double and
other related functions from fold-const.c to double-int.c in r158372:
http://gcc.gnu.org/ml/gcc-cvs/2010-04/msg00478.html.  However, rshift_double
was not copied verbatim, as a check for a negative COUNT was added at the
beginning followed by a conditional call to lshift_double with -COUNT.  This
made the code consistent with the comment, but also caused the loop on HWI32
platforms.

I suspect that r158360 shouldn't have allowed for a negative COUNT to
rshift_double or double_int_rshift.  Removing the check for negative COUNT from
the start of rshift_double does fix the test case on i686 with gcc-4.6.

However, I'm worried that parts of gcc now depend on the extended semantics of
rshift_double and double_int_rshift, so reverting that may be problematic. 
Instead we can robustify rshift_double a bit: if COUNT < 0, check if COUNT
equals INTTYPE_MINIMUM (HOST_WIDE_INT), and if so don't negate COUNT and call
lshift_double but instead set COUNT to 2 * HOST_BITS_PER_WIDE_INT.  The
following code will then handle this like other too-large shifts.  This fixes
the test case too with no testsuite regressions on i686 with gcc-4.6.

gcc-4.7 doesn't have the issue on i686 because i686 is now HWI64 (right?), but
other HWI32 platforms like m68k and arm-oabi still have the problem.  I'm
currently testing the 2nd patch with gcc-4.7 on i686 and x86_64.