[Bug target/107863] New: ICE with unrecognizable insn when using -funsigned-char with some AVX builtins

[Bug target/107863] New: ICE with unrecognizable insn when using -funsigned-char with some AVX builtins

[bouanto at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

            Bug ID: 107863
           Summary: ICE with unrecognizable insn when using
                    -funsigned-char with some AVX builtins
           Product: gcc
           Version: 12.2.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: bouanto at zoho dot com
  Target Milestone: ---

Hi.
When I compile the following code:

#include <immintrin.h>

int main(int argc, char* argv[]) {
    __m256i a = _mm256_set1_epi8(4);
    __m256i b = _mm256_set1_epi8(2);
    __m256i mask = _mm256_insert_epi8(_mm256_set1_epi8(0), -1, 2);
    __m256i r = (__m256i) __builtin_ia32_pblendvb256 ((__v32qi)a, (__v32qi)b,
(__v32qi)mask);
    return 0;
}

with the following command:

gcc main.c -o main -mavx512f -funsigned-char

I get the following error:

main.c: In function ‘main’:
main.c:9:1: error: unrecognizable insn:
    9 | }
      | ^
(insn 655 654 656 2 (set (reg:QI 607)
        (const_int 255 [0xff])) "main.c":6:20 -1
     (nil))
during RTL pass: vregs
main.c:9:1: internal compiler error: in extract_insn, at recog.cc:2791
0x1840d78 internal_error(char const*, ...)
        ???:0
0x62a3ac fancy_abort(char const*, int, char const*)
        ???:0
0x60555b _fatal_insn(char const*, rtx_def const*, char const*, int, char
const*)
        ???:0
0x60557d _fatal_insn_not_found(rtx_def const*, char const*, int, char const*)
        ???:0

The code compiles when not using -funsigned-char.

I'm not sure what would be the fix for this. Would it make sense that builtins
never use the char type, but instead use either unsigned char or signed char?

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |10.5
            Summary|ICE with unrecognizable     |[10/11/12/13 Regression]
                   |insn when using             |ICE with unrecognizable
                   |-funsigned-char with some   |insn when using
                   |AVX builtins                |-funsigned-char with some
                   |                            |SSE/AVX builtins
      Known to work|                            |4.4.7
           Keywords|                            |ice-on-valid-code
      Known to fail|                            |4.5.3
   Last reconfirmed|                            |2022-11-24
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Reduced to just:
#include <immintrin.h>

__m128i f(__m128i a) {
   return _mm_insert_epi8(a, -1, 2);
}

This only requires -msse4.1 -funsigned-char to reproduce the ICE.

;; _4 = __builtin_ia32_vec_set_v16qi (_1, 255, 2);

(insn 7 6 8 (set (reg:QI 86)
        (const_int 255 [0xff])) "/app/example.cpp":4:11 -1
     (nil))

Without -funsigned-char:
;; _4 = __builtin_ia32_vec_set_v16qi (_1, -1, 2);

(insn 7 6 8 (set (reg:QI 86)
        (const_int -1 [0xffffffffffffffff])) "/app/example.cpp":4:11 -1
     (nil))


I suspect the issue is the definition of __builtin_ia32_vec_set_v16qi uses char
type rather than signed/unsigned char here ...

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |FIXME

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
From i386-builtin-types.def:
# ??? Logically this should be intQI_type_node, but that maps to "signed char"
# which is a different type than "char" even if "char" is signed.  This must
# match the usage in emmintrin.h and changing this would change name mangling
# and so is not advisable.
DEF_PRIMITIVE_TYPE (QI, char_type_node)

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Target|                            |x86_64-*-*

--- Comment #3 from Richard Biener <rguenth at gcc dot gnu.org> ---
Not sure how name mangling is a concern for intrinsics...

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

Hongtao.liu <crazylht at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hjl.tools at gmail dot com

--- Comment #4 from Hongtao.liu <crazylht at gmail dot com> ---
Git blame show it start from
-----cut from git blame--------
Richard Henderson 2009-11-26 10:39 i386-builtin-types.awk(DEF_VCETOR_TYPE):
Allow an optinal 3rd argument to define the mode
-------cut end-------------

Intrinsics are inlines, and usally mapping to one or serverl instructions which
are not related to name mangling.

HJ do you know why?

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #5 from Hongtao.liu <crazylht at gmail dot com> ---
Also I get below from build_common_tree_nodes

  /* Define `char', which is like either `signed char' or `unsigned char'
     but not the same as either.  */
  char_type_node
    = (signed_char
       ? make_signed_type (CHAR_TYPE_SIZE)
       : make_unsigned_type (CHAR_TYPE_SIZE));

So using char_type_node should be ok with -funsigned_char?

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #6 from Hongtao.liu <crazylht at gmail dot com> ---
For pattern
(set (reg:QI 607)
    (const_int 255 [0xff]))

general_operand return false for op const_int 255 QImode since
trunc_int_for_mode (INTVAL (op), mode) return -1, INVAL (op) is 255.

---cut from general_operand (rtx, machine_mode)----------
  if (CONST_INT_P (op)
      && mode != VOIDmode
      && trunc_int_for_mode (INTVAL (op), mode) != INTVAL (op))
    return false;
---cut end-----------------


and in trunc_int_for_mode, it does signed extend, not unsigned_extend for
!flag_signed_char.

----cut from trunc_int_for_mode------------
  /* Sign-extend for the requested mode.  */

  if (width < HOST_BITS_PER_WIDE_INT)
    {
      HOST_WIDE_INT sign = 1;
      sign <<= width - 1;
      c &= (sign << 1) - 1;
      c ^= sign;
      c -= sign;
    }

  return c;
--------------cut end--------------


Should we do something like 


modified   gcc/explow.cc
@@ -64,7 +64,8 @@ trunc_int_for_mode (HOST_WIDE_INT c, machine_mode mode)

   /* Sign-extend for the requested mode.  */

-  if (width < HOST_BITS_PER_WIDE_INT)
+  if (width < HOST_BITS_PER_WIDE_INT
+      && (mode != QImode || !flag_signed_char))
     {
       HOST_WIDE_INT sign = 1;
       sign <<= width - 1;

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #7 from Hongtao.liu <crazylht at gmail dot com> ---

> -  if (width < HOST_BITS_PER_WIDE_INT)
> +  if (width < HOST_BITS_PER_WIDE_INT
> +      && (mode != QImode || !flag_signed_char))
typo should be 
+      && (mode != QImode || flag_signed_char))

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #8 from Hongtao.liu <crazylht at gmail dot com> ---
(In reply to Hongtao.liu from comment #7)
> > -  if (width < HOST_BITS_PER_WIDE_INT)
> > +  if (width < HOST_BITS_PER_WIDE_INT
> > +      && (mode != QImode || !flag_signed_char))
> typo should be 
> +      && (mode != QImode || flag_signed_char))

I guess not, flag_signed_char is not an exact map to QImode.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #9 from Hongtao.liu <crazylht at gmail dot com> ---
expand_expr_real_1 generates (const_int 255) without considering the target
mode.
I guess it's on purpose, so I'll leave that alone and only change the expander
in the backend. After applying convert_modes to (const_int 255), it's
transformed to (const_int -1) which should fix the issue.


-----------cut from expand_expr_real_1--------------
11010    case INTEGER_CST:
11011      {
11012        /* Given that TYPE_PRECISION (type) is not always equal to
11013           GET_MODE_PRECISION (TYPE_MODE (type)), we need to extend from
11014           the former to the latter according to the signedness of the
11015           type.  */
11016        scalar_int_mode int_mode = SCALAR_INT_TYPE_MODE (type);
11017        temp = immed_wide_int_const
11018          (wi::to_wide (exp, GET_MODE_PRECISION (int_mode)), int_mode);
11019        return temp;
11020      }
-----------cut ends------------------------


Proposed patch:

diff --git a/gcc/config/i386/i386-expand.cc b/gcc/config/i386/i386-expand.cc
index 0373c3614a4..c639ee3a9f7 100644
--- a/gcc/config/i386/i386-expand.cc
+++ b/gcc/config/i386/i386-expand.cc
@@ -12475,7 +12475,7 @@ ix86_expand_vec_set_builtin (tree exp)
   op1 = expand_expr (arg1, NULL_RTX, mode1, EXPAND_NORMAL);
   elt = get_element_number (TREE_TYPE (arg0), arg2);

-  if (GET_MODE (op1) != mode1 && GET_MODE (op1) != VOIDmode)
+  if (GET_MODE (op1) != mode1)
     op1 = convert_modes (mode1, GET_MODE (op1), op1, true);

   op0 = force_reg (tmode, op0);

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #10 from Hongtao.liu <crazylht at gmail dot com> ---
I notice there's TARGET_PROMOTE_PROTOTYPES which can prevent unsigend char 255
be extended to int 255 which is a more perfect solution to this problem. But we
can only get fntype in this hook, ideally we should check fndecl to make sure
it's target specific builtins since we don't want to prevent promotion for args
in other normal functions.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #11 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by hongtao Liu <liuhongt@gcc.gnu.org>:

https://gcc.gnu.org/g:cda29c540037fbcf00a377196050953aab1d3d5b

commit r13-4432-gcda29c540037fbcf00a377196050953aab1d3d5b
Author: liuhongt <hongtao.liu@intel.com>
Date:   Mon Nov 28 09:59:47 2022 +0800

    Fix unrecognizable insn due to illegal immediate_operand (const_int 255) of
QImode.

    For __builtin_ia32_vec_set_v16qi (a, -1, 2) with
    !flag_signed_char. it's transformed to
    __builtin_ia32_vec_set_v16qi (_4, 255, 2) in the gimple,
    and expanded to (const_int 255) in the rtl. But for immediate_operand,
    it expects (const_int 255) to be signed extended to
    (const_int -1). The mismatch caused an unrecognizable insn error.

    The patch converts (const_int 255) to (const_int -1) in the backend
    expander.

    gcc/ChangeLog:

            PR target/107863
            * config/i386/i386-expand.cc (ix86_expand_vec_set_builtin):
            Convert op1 to target mode whenever mode mismatch.

    gcc/testsuite/ChangeLog:

            * gcc.target/i386/pr107863.c: New test.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #12 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-12 branch has been updated by hongtao Liu
<liuhongt@gcc.gnu.org>:

https://gcc.gnu.org/g:b7306f02da33695bec90f153f6725a51d7c0ac71

commit r12-8954-gb7306f02da33695bec90f153f6725a51d7c0ac71
Author: liuhongt <hongtao.liu@intel.com>
Date:   Mon Nov 28 09:59:47 2022 +0800

    Fix unrecognizable insn due to illegal immediate_operand (const_int 255) of
QImode.

    For __builtin_ia32_vec_set_v16qi (a, -1, 2) with
    !flag_signed_char. it's transformed to
    __builtin_ia32_vec_set_v16qi (_4, 255, 2) in the gimple,
    and expanded to (const_int 255) in the rtl. But for immediate_operand,
    it expects (const_int 255) to be signed extended to
    (const_int -1). The mismatch caused an unrecognizable insn error.

    The patch converts (const_int 255) to (const_int -1) in the backend
    expander.

    gcc/ChangeLog:

            PR target/107863
            * config/i386/i386-expand.cc (ix86_expand_vec_set_builtin):
            Convert op1 to target mode whenever mode mismatch.

    gcc/testsuite/ChangeLog:

            * gcc.target/i386/pr107863.c: New test.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #13 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-11 branch has been updated by hongtao Liu
<liuhongt@gcc.gnu.org>:

https://gcc.gnu.org/g:e6d28f7fd4573988b2417a52acd0a27b7ee91681

commit r11-10404-ge6d28f7fd4573988b2417a52acd0a27b7ee91681
Author: liuhongt <hongtao.liu@intel.com>
Date:   Mon Nov 28 09:59:47 2022 +0800

    Fix unrecognizable insn due to illegal immediate_operand (const_int 255) of
QImode.

    For __builtin_ia32_vec_set_v16qi (a, -1, 2) with
    !flag_signed_char. it's transformed to
    __builtin_ia32_vec_set_v16qi (_4, 255, 2) in the gimple,
    and expanded to (const_int 255) in the rtl. But for immediate_operand,
    it expects (const_int 255) to be signed extended to
    (const_int -1). The mismatch caused an unrecognizable insn error.

    The patch converts (const_int 255) to (const_int -1) in the backend
    expander.

    gcc/ChangeLog:

            PR target/107863
            * config/i386/i386-expand.c (ix86_expand_vec_set_builtin):
            Convert op1 to target mode whenever mode mismatch.

    gcc/testsuite/ChangeLog:

            * gcc.target/i386/pr107863.c: New test.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #14 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-10 branch has been updated by hongtao Liu
<liuhongt@gcc.gnu.org>:

https://gcc.gnu.org/g:ac30c91a1002ae4049a4773d07d5da41e7bd3138

commit r10-11105-gac30c91a1002ae4049a4773d07d5da41e7bd3138
Author: liuhongt <hongtao.liu@intel.com>
Date:   Mon Nov 28 09:59:47 2022 +0800

    Fix unrecognizable insn due to illegal immediate_operand (const_int 255) of
QImode.

    For __builtin_ia32_vec_set_v16qi (a, -1, 2) with
    !flag_signed_char. it's transformed to
    __builtin_ia32_vec_set_v16qi (_4, 255, 2) in the gimple,
    and expanded to (const_int 255) in the rtl. But for immediate_operand,
    it expects (const_int 255) to be signed extended to
    (const_int -1). The mismatch caused an unrecognizable insn error.

    The patch converts (const_int 255) to (const_int -1) in the backend
    expander.

    gcc/ChangeLog:

            PR target/107863
            * config/i386/i386-expand.c (ix86_expand_vec_set_builtin):
            Convert op1 to target mode whenever mode mismatch.

    gcc/testsuite/ChangeLog:

            * gcc.target/i386/pr107863.c: New test.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

--- Comment #15 from Hongtao.liu <crazylht at gmail dot com> ---
Fixed in GCC10.5, GCC11.4,GCC12.3 and GCC13.

[Bug target/107863] [10/11/12/13 Regression] ICE with unrecognizable insn when using -funsigned-char with some SSE/AVX builtins

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107863

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED
                 CC|                            |jakub at gcc dot gnu.org

--- Comment #16 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
.