[Bug target/102143] New: ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[Bug target/102143] New: ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

            Bug ID: 102143
           Summary: ABI incompatibility with clang when passing 32bit
                    vectors on 32bit i686
           Product: gcc
           Version: 12.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: ubizjak at gmail dot com
  Target Milestone: ---

Following testcase:

--cut here--
typedef short __v2hi __attribute__((__vector_size__ (4)));

__v2hi
foo (__v2hi x, __v2hi y)
{
  return x + y;
}
--cut here--

exhibits ABI incompatibility with clang when passing 32bit vectors on 32bit
i686.

gcc-12 compiles with "-O2 -msse2" to:

foo:
        movd    4(%esp), %xmm0
        movd    8(%esp), %xmm1
        paddw   %xmm1, %xmm0
        movd    %xmm0, %eax
        ret

(gcc before version 12 uses the same ABI).

while clang-11 compiles the testacse to:

foo:
        paddw   %xmm1, %xmm0
        retl

So, clang is passing 32bit vectors via vector registers, while gcc is passing
them like integer values in memory and returns them in integer return register.

[Bug target/102143] ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-08-31
             Status|UNCONFIRMED                 |NEW

--- Comment #1 from H.J. Lu <hjl.tools at gmail dot com> ---
16-bit and 32-bit vector pass and return are not specified in i386 psABI.
64-bit vector is specified, not really usable.  Any suggestions?

[Bug target/102143] ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

--- Comment #2 from Uroš Bizjak <ubizjak at gmail dot com> ---
(In reply to H.J. Lu from comment #1)
> 16-bit and 32-bit vector pass and return are not specified in i386 psABI.
> 64-bit vector is specified, not really usable.  Any suggestions?

With -mno-sse, clang creates:

foo:
        movzwl  16(%esp), %edx
        movzwl  12(%esp), %eax
        addw    4(%esp), %ax
        addw    8(%esp), %dx
        retl

So, it is incompatible with itself for -msse/-mno-sse differences. The ABI
should be ISA agnostic, so using integer ABI as implemented by gcc is IMO the
way to go.

[Bug target/102143] ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

--- Comment #3 from Uroš Bizjak <ubizjak at gmail dot com> ---
gcc has also some problems in this area. With -mregparm=3, one would expect
arguments obeying integer ABI to be passed in registers, but regparm flag has
no effect and the same code is produced:

foo:
        movd    4(%esp), %xmm0
        movd    8(%esp), %xmm1
        paddw   %xmm1, %xmm0
        movd    %xmm0, %eax
        ret

[Bug target/102143] ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

--- Comment #4 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Well, we already have the ABI ISA dependent,
typedef int __v8si __attribute__((__vector_size__ (32)));

__v8si
foo (__v8si x, __v8si y)
{
  return x + y;
}
has different ABI based on -mavx or -mno-avx, etc.  For -mno-avx we emit a
warning:
/tmp/test.c: In function ‘foo’:
/tmp/test.c:5:1: warning: AVX vector return without AVX enabled changes the ABI
[-Wpsabi]
    5 | {
      | ^
/tmp/test.c:4:1: note: the ABI for passing parameters with 32-byte alignment
has changed in GCC 4.6
    4 | foo (__v8si x, __v8si y)
      | ^~~
/tmp/test.c:4:1: warning: AVX vector argument without AVX enabled changes the
ABI [-Wpsabi]

So, depending on what we decide, if the ABI will be ISA dependent, we want a
warning like the above one.

[Bug target/102143] ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

--- Comment #5 from Uroš Bizjak <ubizjak at gmail dot com> ---
(In reply to Uroš Bizjak from comment #3)
> gcc has also some problems in this area. With -mregparm=3, one would expect
> arguments obeying integer ABI to be passed in registers, but regparm flag
> has no effect and the same code is produced:
> 
> foo:
>         movd    4(%esp), %xmm0
>         movd    8(%esp), %xmm1
>         paddw   %xmm1, %xmm0
>         movd    %xmm0, %eax
>         ret

(sorry, sent the message too fast)

However, with -mno-sse -mregparm=3, regparm does have effect and produces:

foo:
        leal    (%eax,%edx), %ecx
        sarl    $16, %eax
        sarl    $16, %edx
        addl    %eax, %edx
        movzwl  %cx, %eax
        sall    $16, %edx
        orl     %edx, %eax
        ret

vs -mno-sse -mregparm=0:

foo:
        movl    4(%esp), %edx
        movl    8(%esp), %ecx
        leal    (%edx,%ecx), %eax
        sarl    $16, %edx
        sarl    $16, %ecx
        addl    %ecx, %edx
        movzwl  %ax, %eax
        sall    $16, %edx
        orl     %edx, %eax
        ret

[Bug target/102143] ABI incompatibility with clang when passing 32bit vectors on 32bit i686

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102143

--- Comment #6 from H.J. Lu <hjl.tools at gmail dot com> ---
I think psABIs should specify how to pass and return 8-bit, 16-bit and 32-bit
vectors.  We can treat them as

struct vectorN
{
   intN
};