[Bug tree-optimization/105533] New: UBSAN: gcc/expmed.cc:3272:26: runtime error: signed integer overflow: -9223372036854775808 - 1 cannot be represented in type 'long int'

["zsojka at seznam dot cz" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105533

            Bug ID: 105533
           Summary: UBSAN: gcc/expmed.cc:3272:26: runtime error: signed
                    integer overflow: -9223372036854775808 - 1 cannot be
                    represented in type 'long int'
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: zsojka at seznam dot cz
  Target Milestone: ---
              Host: x86_64-pc-linux-gnu

Created attachment 52943
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52943&action=edit
reduced testcase

Compiler output:
$ x86_64-pc-linux-gnu-gcc -O testcase.c -c
/repo/gcc-trunk/gcc/expmed.cc:3272:26: runtime error: signed integer overflow:
-9223372036854775808 - 1 cannot be represented in type 'long int'

(gdb) bt
#0  __ubsan::Diag::~Diag (this=this@entry=0x7fffffffc250, __in_chrg=<optimized
out>) at /repo/gcc-trunk/libsanitizer/ubsan/ubsan_diag.cpp:343
#1  0x00000000074e14ec in handleIntegerOverflowImpl<__ubsan::Value>
(Data=Data@entry=0x85fd8e0, LHS=LHS@entry=9223372036854775808,
Operator=Operator@entry=0x7b7c28d "-", RHS=..., Opts=...)
    at /repo/gcc-trunk/libsanitizer/ubsan/ubsan_handlers.cpp:227
#2  0x00000000074e478d in __ubsan::__ubsan_handle_sub_overflow
(Data=Data@entry=0x85fd8e0, LHS=LHS@entry=9223372036854775808, RHS=RHS@entry=1)
at /repo/gcc-trunk/libsanitizer/ubsan/ubsan_handlers.cpp:244
#3  0x0000000001384d5d in choose_mult_variant (mode=E_DImode, val=<optimized
out>, alg=0x7fffffffc6b0, variant=0x7fffffffc6ac, mult_cost=16) at
/repo/gcc-trunk/gcc/expmed.cc:3272
#4  0x0000000001385946 in mult_by_coeff_cost (coeff=-9223372036854775808,
mode=E_DImode, speed=<optimized out>) at /repo/gcc-trunk/gcc/expmed.cc:3610
#5  0x00000000068ae3ac in create_add_ssa_cand (gs=0x7ffff7851a50,
base_in=0x7ffff7852ea0, addend_in=<optimized out>, subtract_p=false,
speed=true) at /repo/gcc-trunk/gcc/gimple-ssa-strength-reduction.cc:1311
#6  0x00000000068b4668 in slsr_process_add (speed=true, rhs2=0x7ffff7852ea0,
rhs1=<optimized out>, gs=<optimized out>) at
/repo/gcc-trunk/gcc/gimple-ssa-strength-reduction.cc:1484
#7  find_candidates_dom_walker::before_dom_children (this=<optimized out>,
bb=<optimized out>) at
/repo/gcc-trunk/gcc/gimple-ssa-strength-reduction.cc:1803
#8  0x000000000665b9e5 in dom_walker::walk (this=<optimized out>,
bb=0x7ffff76f4478) at /repo/gcc-trunk/gcc/domwalk.cc:309
#9  0x000000000689d07a in (anonymous
namespace)::pass_strength_reduction::execute (this=<optimized out>,
fun=0x7ffff784c0b8) at
/repo/gcc-trunk/gcc/gimple-ssa-strength-reduction.cc:4039
#10 0x00000000021af318 in execute_one_pass (pass=0xc6701b0) at
/repo/gcc-trunk/gcc/passes.cc:2638
#11 0x00000000021b2902 in execute_pass_list_1 (pass=0xc6701b0) at
/repo/gcc-trunk/gcc/passes.cc:2738
#12 0x00000000021b2945 in execute_pass_list_1 (pass=0xc66dcc0) at
/repo/gcc-trunk/gcc/passes.cc:2739
#13 0x00000000021b29f9 in execute_pass_list (fn=0x7ffff784c0b8, pass=<optimized
out>) at /repo/gcc-trunk/gcc/passes.cc:2749
#14 0x0000000000f8cebc in cgraph_node::expand (this=0x7ffff784f110) at
/repo/gcc-trunk/gcc/cgraphunit.cc:1835
#15 0x0000000000f92108 in expand_all_functions () at
/repo/gcc-trunk/gcc/cgraphunit.cc:1999
#16 symbol_table::compile (this=0x7ffff76ec000) at
/repo/gcc-trunk/gcc/cgraphunit.cc:2349
#17 0x0000000000fa02ad in symbol_table::compile (this=0x7ffff76ec000) at
/repo/gcc-trunk/gcc/cgraphunit.cc:2262
#18 symbol_table::finalize_compilation_unit (this=0x7ffff76ec000) at
/repo/gcc-trunk/gcc/cgraphunit.cc:2530
#19 0x00000000026d34a3 in compile_file () at /repo/gcc-trunk/gcc/toplev.cc:479
#20 0x0000000000917d45 in do_compile (no_backend=false) at
/repo/gcc-trunk/gcc/toplev.cc:2168
#21 toplev::main (this=this@entry=0x7fffffffd27e, argc=<optimized out>,
argc@entry=17, argv=<optimized out>, argv@entry=0x7fffffffd3c8) at
/repo/gcc-trunk/gcc/toplev.cc:2320
#22 0x000000000091bb6b in main (argc=17, argv=0x7fffffffd3c8) at
/repo/gcc-trunk/gcc/main.cc:39

$ x86_64-pc-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=/repo/gcc-trunk/binary-latest-amd64/bin/x86_64-pc-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/repo/gcc-trunk/binary-trunk-r13-167-20220507103325-g0c723bb4be2-checking-release-bootstrap-ubsan-amd64/bin/../libexec/gcc/x86_64-pc-linux-gnu/13.0.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /repo/gcc-trunk//configure --enable-languages=c,c++
--enable-valgrind-annotations --disable-nls --enable-checking=release
--with-build-config=bootstrap-ubsan --with-cloog --with-ppl --with-isl
--build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
--target=x86_64-pc-linux-gnu --with-ld=/usr/bin/x86_64-pc-linux-gnu-ld
--with-as=/usr/bin/x86_64-pc-linux-gnu-as --disable-libstdcxx-pch
--prefix=/repo/gcc-trunk//binary-trunk-r13-167-20220507103325-g0c723bb4be2-checking-release-bootstrap-ubsan-amd64
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 13.0.0 20220507 (experimental) (GCC)