[Bug tree-optimization/99079] Maybe a wrong code since r6-1462-g4ab1e111ef0669bb

["jakub at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99079

--- Comment #2 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 50175
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=50175&action=edit
gcc11-pr99079.patch

Untested fix.
The problem is that the tree_nop_conversion_p (type, TREE_TYPE (@3))
test actually didn't test what it was meant to test, in
(mod @0 (convert?@3 (...)))
TREE_TYPE (@3) is the type of the second argument of the modulo, so
necessarily compatible with type (compatible also with TREE_TYPE (@0)).
But, I think we actually don't need to require only nop conversions, which are
certainly fine.
If the conversion is narrowing, i.e. TREE_TYPE (@1) is wider than type, then
we know that @1 is some power of two and after the cast, it will be either 0
(but then UB, so we can rule that out), or that power of two.
For widening conversion where TREE_TYPE (@1) is unsigned it is also fine,
@1 will be some power of two (0 being UB) and @1 - 1 is a mask we can use after
widening it to the type of the mod.
The only problematic case is widening conversion where TREE_TYPE (@1) is
signed,
as the testcase shows, if the power of two is negative, i.e. msb set and all
other bits clear, then the widening cast turns that into 0xffffffff80000000
(if type is unsigned) and as that is not a power of two, we can't transform it
into a mask by 0x7fffffff.