[Bug target/104188] New: gcc omitting AVX-512 broadcast instruction

[Bug target/104188] New: gcc omitting AVX-512 broadcast instruction

[kvr000 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

            Bug ID: 104188
           Summary: gcc omitting AVX-512 broadcast instruction
           Product: gcc
           Version: 11.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: kvr000 at gmail dot com
  Target Milestone: ---

Hi,
there is a bug when generating AVX-512 instructions from intrinsics.  The code
is generated correctly in gcc-10 but gcc-11 completely omits the
vbroadcastf32x4 .

gcc version: 11.2.0-7ubuntu2 - 11.2.0

Source code of minimal working example:
// Matrix 4*4 multiplication:

#ifndef NO_VECTORIZE
#ifdef __x86_64__
#include <immintrin.h>
#include <x86intrin.h>
#endif
#ifdef __aarch64__
#include <arm_neon.h>
#endif
#endif

union Mat44 {
        float m[4][4];
#ifndef NO_VECTORIZE
#ifdef __x86_64__
        __m128 row[4];
        __m256 rowDuet[2];
        __m512 rowQuad;
#endif
#ifdef __aarch64__
        float32x4_t row[4];
#endif
#endif
};

void matmult_avx512(union Mat44 *out, union Mat44 *a, union Mat44 *b)
{
        __m512 a0123 = _mm512_loadu_ps(a->m[0]);
        __m512 b0000 = _mm512_broadcast_f32x4(b->row[0]);
        __m512 b1111 = _mm512_broadcast_f32x4(b->row[1]);
        __m512 b2222 = _mm512_broadcast_f32x4(b->row[2]);
        __m512 b3333 = _mm512_broadcast_f32x4(b->row[3]);

        __m512 result = _mm512_mul_ps(_mm512_permute_ps(a0123, 0x00), b0000);
        result = _mm512_fmadd_ps(_mm512_permute_ps(a0123, 0x55), b1111,
result);
        result = _mm512_fmadd_ps(_mm512_permute_ps(a0123, 0xaa), b2222,
result);
        result = _mm512_fmadd_ps(_mm512_permute_ps(a0123, 0xff), b3333,
result);

        _mm512_storeu_ps(out->m[0], result);
}


gcc-10 (correct):

        endbr64
        vmovups (%rsi), %zmm0
        vbroadcastf32x4 (%rdx), %zmm6           // note here
        vpermilps       $0, %zmm0, %zmm1
        vmulps  %zmm6, %zmm1, %zmm1
        vbroadcastf32x4 16(%rdx), %zmm5         // note here
        vpermilps       $85, %zmm0, %zmm2
        vbroadcastf32x4 32(%rdx), %zmm4         // note here
        vbroadcastf32x4 48(%rdx), %zmm3         // note here
        vfmadd132ps     %zmm5, %zmm1, %zmm2
        vpermilps       $170, %zmm0, %zmm1
        vpermilps       $255, %zmm0, %zmm0
        vfmadd132ps     %zmm4, %zmm2, %zmm1
        vfmadd132ps     %zmm3, %zmm1, %zmm0
        vmovups %zmm0, (%rdi)
        vzeroupper
        ret


gcc-11 (missing vbroadcasatf32x4) :

        endbr64
        vmovups (%rsi), %zmm0
        vpermilps       $0, %zmm0, %zmm1
        vmulps  (%rdx){1to16}, %zmm1, %zmm1
        vpermilps       $85, %zmm0, %zmm2
        vfmadd132ps     16(%rdx){1to16}, %zmm1, %zmm2
        vpermilps       $170, %zmm0, %zmm1
        vpermilps       $255, %zmm0, %zmm0
        vfmadd132ps     32(%rdx){1to16}, %zmm2, %zmm1
        vfmadd132ps     48(%rdx){1to16}, %zmm1, %zmm0
        vmovups %zmm0, (%rdi)
        vzeroupper
        ret

[Bug target/104188] gcc omitting AVX-512 broadcast instruction

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2022-01-22
     Ever confirmed|0                           |1
           Keywords|                            |wrong-code

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
There is a broadcast as part of the other instruction, optimizing it correctly:
        vfmadd132ps     16(%rdx){1to16}, %zmm1, %zmm2


{1to16} says to broadcast  from first element to all 16.

Why do you think this is wrong code?

[Bug target/104188] gcc omitting AVX-512 broadcast instruction

[kvr000 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #2 from Zbynek Vyskovsky <kvr000 at gmail dot com> ---
> {1to16} says to broadcast  from first element to all 16.

The vbroadcastf32x4 is supposed to copy first four elmenents to 4-7, 8-11 and
12-15 .

> Why do you think this is wrong code?

It doesn't work.  It produces the same number for each column for the same row,
likely as a result of above as it uses single element to multiply instead of
four different elements.

[Bug target/104188] gcc omitting AVX-512 broadcast instruction

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
So, can you please turn add main to your testcase, add __attribute__((noipa))
on
the function and add a caller that prepares arguments and check the result?

[Bug target/104188] gcc omitting AVX-512 broadcast instruction

[kvr000 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #4 from Zbynek Vyskovsky <kvr000 at gmail dot com> ---
Sure, the code:

#include <stdio.h>

#ifndef NO_VECTORIZE
#ifdef __x86_64__
#include <immintrin.h>
#include <x86intrin.h>
#endif
#ifdef __aarch64__
#include <arm_neon.h>
#endif
#endif

typedef union Mat44 {
        float m[4][4];
#ifndef NO_VECTORIZE
#ifdef __x86_64__
        __m128 row[4];
        __m256 rowDuet[2];
        __m512 rowQuad;
#endif
#ifdef __aarch64__
        float32x4_t row[4];
#endif
#endif
} Mat44;

__attribute__((noipa)) void matmult_avx512(union Mat44 *out, const Mat44 *a,
const Mat44 *b)
{
        __m512 a0123 = _mm512_loadu_ps(a->m[0]);
        __m512 b0000 = _mm512_broadcast_f32x4(b->row[0]);
        __m512 b1111 = _mm512_broadcast_f32x4(b->row[1]);
        __m512 b2222 = _mm512_broadcast_f32x4(b->row[2]);
        __m512 b3333 = _mm512_broadcast_f32x4(b->row[3]);

        __m512 result = _mm512_mul_ps(_mm512_permute_ps(a0123, 0x00), b0000);
        result = _mm512_fmadd_ps(_mm512_permute_ps(a0123, 0x55), b1111,
result);
        result = _mm512_fmadd_ps(_mm512_permute_ps(a0123, 0xaa), b2222,
result);
        result = _mm512_fmadd_ps(_mm512_permute_ps(a0123, 0xff), b3333,
result);

        _mm512_storeu_ps(out->m[0], result);
}

__attribute__((noipa)) void matmult_ref(Mat44 *out, const Mat44 *a, const Mat44
*b)
{
        Mat44 t;
        for (int i = 0; i < 4; i++) {
                for (int j = 0; j < 4; j++) {
                        t.m[i][j] = a->m[i][0]*b->m[0][j] +
a->m[i][1]*b->m[1][j] + a->m[i][2]*b->m[2][j] + a->m[i][3]*b->m[3][j];
                }
        }

        *out = t;
}

int main(void)
{
        Mat44 in = { m: { { 1, 2, 3, 4 }, { 5, 6, 7, 8 }, { 9, 10, 11, 12 }, {
13, 14, 15, 16 } } };
        Mat44 avx512_out;
        Mat44 ref_out;
        matmult_ref(&ref_out, &in, &in);
        matmult_avx512(&avx512_out, &in, &in);
        for (int r = 0; r < 4; ++r) {
                printf("%5.0f %5.0f %5.0f %5.0f      %5.0f %5.0f %5.0f
%5.0f\n", avx512_out.m[r][0], avx512_out.m[r][1], avx512_out.m[r][2],
avx512_out.m[r][3], ref_out.m[r][0], ref_out.m[r][1], ref_out.m[r][2],
ref_out.m[r][3]);
        }
        return 0;
}


Output (note the repeating first column on first side, caused by duplicating
single element instead of four):

  90    90    90    90         90   100   110   120
 202   202   202   202        202   228   254   280
 314   314   314   314        314   356   398   440
 426   426   426   426        426   484   542   600

[Bug target/104188] gcc omitting AVX-512 broadcast instruction

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
I came up with:
#include <x86intrin.h>

union U {
  float m[4][4];
  __m128 r[4];
  __m512 s;
};

__attribute__((noipa)) void
foo (union U *x, union U *a, union U *b)
{
  __m512 c = _mm512_loadu_ps (&a->s);
  __m512 d = _mm512_broadcast_f32x4 (b->r[0]);
  __m512 e = _mm512_broadcast_f32x4 (b->r[1]);
  __m512 f = _mm512_broadcast_f32x4 (b->r[2]);
  __m512 g = _mm512_broadcast_f32x4 (b->r[3]);
  __m512 h = _mm512_mul_ps (_mm512_permute_ps (c, 0x00), d);
  h = _mm512_fmadd_ps (_mm512_permute_ps (c, 0x55), e, h);
  h = _mm512_fmadd_ps (_mm512_permute_ps (c, 0xaa), f, h);
  h = _mm512_fmadd_ps (_mm512_permute_ps (c, 0xff), g, h);
  _mm512_storeu_ps (&x->s, h);
}

int
main ()
{
  union U a = { .m = { { 1.0f, 2.0f, 3.0f, 4.0f },
                       { 5.0f, 6.0f, 7.0f, 8.0f },
                       { 9.0f, 10.0f, 11.0f, 12.0f },
                       { 13.0f, 14.0f, 15.0f, 16.0f } } };
  union U b = { .m = { { 17.0f, 18.0f, 19.0f, 20.0f },
                       { 21.0f, 22.0f, 23.0f, 24.0f },
                       { 25.0f, 26.0f, 27.0f, 28.0f },
                       { 29.0f, 30.0f, 31.0f, 32.0f } } };
  union U c;
  foo (&c, &a, &b);
  if (c.m[0][0] != 250.0f || c.m[0][1] != 260.0f || c.m[0][2] != 270.0f ||
c.m[0][3] != 280.0f)
    __builtin_abort ();
  if (c.m[1][0] != 618.0f || c.m[1][1] != 644.0f || c.m[1][2] != 670.0f ||
c.m[1][3] != 696.0f)
    __builtin_abort ();
  if (c.m[2][0] != 986.0f || c.m[2][1] != 1028.0f || c.m[2][2] != 1070.0f ||
c.m[2][3] != 1112.0f)
    __builtin_abort ();
  if (c.m[3][0] != 1354.0f || c.m[3][1] != 1412.0f || c.m[3][2] != 1470.0f ||
c.m[3][3] != 1528.0f)
    __builtin_abort ();
  return 0;
}

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P3                          |P2
   Target Milestone|---                         |11.3
            Summary|gcc omitting AVX-512        |[11/12 Regression] gcc
                   |broadcast instruction       |omitting AVX-512 broadcast
                   |                            |instruction

--- Comment #6 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Started with r11-4203-g7026bb9504eb0f95e114f832cd6dd14302376861

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #7 from H.J. Lu <hjl.tools at gmail dot com> ---
Since vbroadcastf32x4 != vbroadcastss, we can't replace vbroadcastf32x4
with {1to16} broadcast.

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #8 from H.J. Lu <hjl.tools at gmail dot com> ---
Created attachment 52269
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52269&action=edit
A patch

This this.

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch

--- Comment #9 from H.J. Lu <hjl.tools at gmail dot com> ---
A patch is posted at

https://gcc.gnu.org/pipermail/gcc-patches/2022-January/589109.html

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[kvr000 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #10 from Zbynek Vyskovsky <kvr000 at gmail dot com> ---
Thanks for quick fix!

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #11 from H.J. Lu <hjl.tools at gmail dot com> ---
The v2 patch is posted at

https://gcc.gnu.org/pipermail/gcc-patches/2022-January/589125.html

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #12 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by H.J. Lu <hjl@gcc.gnu.org>:

https://gcc.gnu.org/g:4d2321314a656dd3e30117e2a5266cbacb1e60eb

commit r12-6831-g4d2321314a656dd3e30117e2a5266cbacb1e60eb
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Sat Jan 22 12:04:30 2022 -0800

    x86: Also check mode of memory broadcast in bcst_mem_operand

    Return false for invalid mode on memory broadcast in bcst_mem_operand:

    (vec_duplicate:V16SF (mem/j:V4SF (reg/v/f:DI 109 [ b ])))

    gcc/

            PR target/104188
            * config/i386/predicates.md (bcst_mem_operand): Also check mode
            of memory broadcast.

    gcc/testsuite/

            PR target/104188
            * gcc.target/i386/pr104188.c: New test.

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #13 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-11 branch has been updated by H.J. Lu <hjl@gcc.gnu.org>:

https://gcc.gnu.org/g:520147ba19db8034b1f911326beee104da606daa

commit r11-9489-g520147ba19db8034b1f911326beee104da606daa
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Sat Jan 22 12:04:30 2022 -0800

    x86: Also check mode of memory broadcast in bcst_mem_operand

    Return false for invalid mode on memory broadcast in bcst_mem_operand:

    (vec_duplicate:V16SF (mem/j:V4SF (reg/v/f:DI 109 [ b ])))

    gcc/

            PR target/104188
            * config/i386/predicates.md (bcst_mem_operand): Also check mode
            of memory broadcast.

    gcc/testsuite/

            PR target/104188
            * gcc.target/i386/pr104188.c: New test.

    (cherry picked from commit 4d2321314a656dd3e30117e2a5266cbacb1e60eb)

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #14 from H.J. Lu <hjl.tools at gmail dot com> ---
Fixed for GCC 12 and GCC 11.3.

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #15 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:192e4a9fa0880ef153631d5b0de864bc4bbb6a91

commit r12-6874-g192e4a9fa0880ef153631d5b0de864bc4bbb6a91
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Wed Jan 26 11:58:27 2022 +0100

    testsuite: Fix up pr104188.c testcase for i686-linux [PR104188]

    On i686-linux this new testcase FAILs with:
    cc1: warning: SSE instruction set disabled, using 387 arithmetics
    FAIL: gcc.target/i386/pr104188.c (test for excess errors)
    Excess errors:
    cc1: warning: SSE instruction set disabled, using 387 arithmetics
    This is because it uses -mfpmath=sse, but -msse2 isn't on.  Fixed
    by adding -msse2 to dg-options and requiring sse2_runtime effective
    target.

    2022-01-26  Jakub Jelinek  <jakub@redhat.com>

            PR target/104188
            * gcc.target/i386/pr104188.c: Add dg-require-effective-target
            sse2_runtime.  Add -msse2 to dg-options.

[Bug target/104188] [11/12 Regression] gcc omitting AVX-512 broadcast instruction

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104188

--- Comment #16 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-11 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:44d28c67b484fe3be7b693108fc03cbe6aa05329

commit r11-9512-g44d28c67b484fe3be7b693108fc03cbe6aa05329
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Wed Jan 26 11:58:27 2022 +0100

    testsuite: Fix up pr104188.c testcase for i686-linux [PR104188]

    On i686-linux this new testcase FAILs with:
    cc1: warning: SSE instruction set disabled, using 387 arithmetics
    FAIL: gcc.target/i386/pr104188.c (test for excess errors)
    Excess errors:
    cc1: warning: SSE instruction set disabled, using 387 arithmetics
    This is because it uses -mfpmath=sse, but -msse2 isn't on.  Fixed
    by adding -msse2 to dg-options and requiring sse2_runtime effective
    target.

    2022-01-26  Jakub Jelinek  <jakub@redhat.com>

            PR target/104188
            * gcc.target/i386/pr104188.c: Add dg-require-effective-target
            sse2_runtime.  Add -msse2 to dg-options.

    (cherry picked from commit 192e4a9fa0880ef153631d5b0de864bc4bbb6a91)