[Bug target/111815] New: VAX: ICE in 'print_operand_address' while building 'elf_zstd_decompress' from libbacktrace/elf.c

["macro at orcam dot me.uk" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111815

            Bug ID: 111815
           Summary: VAX: ICE in 'print_operand_address' while building
                    'elf_zstd_decompress' from libbacktrace/elf.c
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: blocker
          Priority: P3
         Component: target
          Assignee: macro at orcam dot me.uk
          Reporter: macro at orcam dot me.uk
  Target Milestone: ---
            Target: vax-*-*

Created attachment 56113
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=56113&action=edit
VAX: Only accept the index scaler as the RHS operand to ASHIFT

As from commit 9df1ba9a35b8 ("libbacktrace: support zstd decompression")
GCC for the `vax-netbsdelf' target fails to complete building, with an
ICE:

during RTL pass: final
.../libbacktrace/elf.c: In function 'elf_zstd_decompress':
.../libbacktrace/elf.c:5006:1: internal compiler error: in
print_operand_address, at config/vax/vax.cc:514
 5006 | }
      | ^
0x1113df97 print_operand_address(_IO_FILE*, rtx_def*)
        .../gcc/config/vax/vax.cc:514
0x10c2489b default_print_operand_address(_IO_FILE*, machine_mode, rtx_def*)
        .../gcc/targhooks.cc:373
0x106ddd0b output_address(machine_mode, rtx_def*)
        .../gcc/final.cc:3648
0x106ddd0b output_asm_insn(char const*, rtx_def**)
        .../gcc/final.cc:3505
0x106e2143 output_asm_insn(char const*, rtx_def**)
        .../gcc/final.cc:3421
0x106e2143 final_scan_insn_1
        .../gcc/final.cc:2841
0x106e28e3 final_scan_insn(rtx_insn*, _IO_FILE*, int, int, int*)
        .../gcc/final.cc:2887
0x106e2bf7 final_1
        .../gcc/final.cc:1979
0x106e3c67 rest_of_handle_final
        .../gcc/final.cc:4240
0x106e3c67 execute
        .../gcc/final.cc:4318
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.

This is due to combine producing an invalid address RTX:

(plus:SI (ashift:SI (const_int 1 [0x1])
        (reg:QI 3 %r3 [1232]))
    (reg/v:SI 10 %r10 [orig:736 weight_mask ] [736]))

where the expression is ((1 << R3) + R10), which does not match a valid
machine address mode.  Consequently `print_operand_address' chokes.

The ultimate cause has been commit c605a8bf9270 ("VAX: Accept ASHIFT in
address expressions"), where a shift of an immediate value by a register
has been mistakenly allowed as an index expression as if the shift
operation was commutative such as multiplication is.  So with ASHIFT
the scaler in an index expression has to be the right-hand operand, and
the backend has to enforce that, whereas with MULT the scaler can be
either operand.