[Bug c/56068] New: -march=native creates Illegal instruction on KVM guests

[Bug c/56068] New: -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

             Bug #: 56068
           Summary: -march=native creates Illegal instruction on KVM
                    guests
    Classification: Unclassified
           Product: gcc
           Version: 4.4.6
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: jpyeron@pdinc.us


Created attachment 29238
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29238
gcc -dM -E -

gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)

My guess is that it is creating SSE3 instructions.

discovered while working https://issues.asterisk.org/jira/browse/ASTERISK-20128

[Bug c/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #1 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:08:47 UTC ---
Created attachment 29239
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29239
gcc -dM -E - -march=native

[Bug c/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #2 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:10:34 UTC ---
Created attachment 29240
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29240
diff of defines

[Bug c/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #3 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:12:15 UTC ---
mockbuild@centos6-64bit-builder ~/build/BUILD/tmp (mock-chroot)
$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 13
model name      : QEMU Virtual CPU version (cpu64-rhel6)
stepping        : 3
cpu MHz         : 2194.498
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 4
wp              : yes
flags           : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm up unfair_spinlock pni cx16
hypervisor lahf_lm abm sse4a
bogomips        : 4388.99
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management:

[Bug c/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #4 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:13:25 UTC ---
Created attachment 29241
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29241
gcc -v test.c output

[Bug c/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #5 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:14:05 UTC ---
Created attachment 29242
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29242
gcc test.c  -march=native -v

[Bug c/56068] -march=native creates Illegal instruction on KVM guests

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #6 from Andrew Pinski <pinskia at gcc dot gnu.org> 2013-01-21 17:14:54 UTC ---
> sse4a

There is the issue I think.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Target|                            |x86_64-*-*
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2013-01-21
          Component|c                           |target
     Ever Confirmed|0                           |1

--- Comment #7 from Andrew Pinski <pinskia at gcc dot gnu.org> 2013-01-21 17:19:29 UTC ---
Please try a newer version of GCC, 4.4 is no longer supported.  Also since this
is 4.4 is going to be modified by RedHat, please report it to them instead.  I
don't think bdver2 support was in the official 4.4 release.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #8 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:26:06 UTC ---
(In reply to comment #7)
> Please try a newer version of GCC, 4.4 is no longer supported.  Also since which version is the oldest supported version, I will use that version for testing.

I will also open a vendor issue. to support the RH GCC 4.4.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #9 from Jonathan Wakely <redi at gcc dot gnu.org> 2013-01-21 17:30:38 UTC ---
What is the illegal instruction? What version of glibc are you using?

I debugged a very similar problem last week where GCC was generating vmovsd on
a host without AVX support. /proc/cpuinfo showed no avx flag, but glibc's
__cpuid reported the AVX bit set which makes GCC use -mavx.  This is a glibc
bug, it should not set the AVX bit if the OSXSAVE bit is not also set. This
seems to predominantly affect VMs, but in my case was not a VM. The glibc bug
is fixed in 2.15, see http://sourceware.org/bugzilla/show_bug.cgi?id=14059 and
http://sourceware.org/bugzilla/show_bug.cgi?id=13753

Using -mno-avx (or replacing -march=cirei7) worked for me.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #10 from Jonathan Wakely <redi at gcc dot gnu.org> 2013-01-21 17:32:52 UTC ---
(In reply to comment #9)
> Using -mno-avx (or replacing -march=cirei7) worked for me.

Bah, that should be "replacing -march=native with -march=corei7"

My problem was with GCC 4.6, and the avx-detection code is the same in current
releases, so if your illegal instruction is an AVX one then I doubt trying a
newer GCC will help.  The problem is in glibc's __cpuid() function not GCC.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #11 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 17:42:06 UTC ---
(In reply to comment #10)
> (In reply to comment #9)
> > Using -mno-avx (or replacing -march=cirei7) worked for me.
> Bah, that should be "replacing -march=native with -march=corei7"
> My problem was with GCC 4.6, and the avx-detection code is the same in current
> releases, so if your illegal instruction is an AVX one then I doubt trying a
> newer GCC will help.  The problem is in glibc's __cpuid() function not GCC.

I will try 4.6 to prove that. Off to build 4.6...

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #12 from Jonathan Wakely <redi at gcc dot gnu.org> 2013-01-21 19:14:14 UTC ---
(In reply to comment #11)
> I will try 4.6 to prove that. Off to build 4.6...

Thanks for checking.  See http://gcc.gnu.org/wiki/InstallingGCC for the
foolproof way to do build it.

You could also try this with any GCC version on the KVM guest:

#include <stdio.h>
#include "cpuid.h"

int main()
{
      unsigned bit_osxsave = bit_AVX >> 1;

      unsigned int eax, ebx, ecx, edx;

      __cpuid (1, eax, ebx, ecx, edx);

      printf("bit_OSXSAVE (%u) = %u\n", bit_osxsave, ecx & bit_osxsave);
      printf("bit_AVX (%u) = %u\n", bit_AVX, ecx & bit_AVX);
}

If that prints 0 for OSXSAVE and non-zero for AVX then it's the same problem I
had.

Maybe GCC could work around it by checking both flags in the AVX detection
logic.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[ubizjak at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #13 from Uros Bizjak <ubizjak at gmail dot com> 2013-01-21 19:31:47 UTC ---
(In reply to comment #12)
> (In reply to comment #11)
> > I will try 4.6 to prove that. Off to build 4.6...
> 
> Thanks for checking.  See http://gcc.gnu.org/wiki/InstallingGCC for the
> foolproof way to do build it.
> 
> You could also try this with any GCC version on the KVM guest:
> 
> #include <stdio.h>
> #include "cpuid.h"
> 
> int main()
> {
>       unsigned bit_osxsave = bit_AVX >> 1;
> 
>       unsigned int eax, ebx, ecx, edx;
> 
>       __cpuid (1, eax, ebx, ecx, edx);
> 
>       printf("bit_OSXSAVE (%u) = %u\n", bit_osxsave, ecx & bit_osxsave);
>       printf("bit_AVX (%u) = %u\n", bit_AVX, ecx & bit_AVX);
> }
> 
> If that prints 0 for OSXSAVE and non-zero for AVX then it's the same problem I
> had.
> 
> Maybe GCC could work around it by checking both flags in the AVX detection
> logic.

Recent 4.6+ does. Please see driver-i386.c around line 470:

--snip--
  /* Get XCR_XFEATURE_ENABLED_MASK register with xgetbv.  */
#define XCR_XFEATURE_ENABLED_MASK    0x0
#define XSTATE_FP            0x1
#define XSTATE_SSE            0x2
#define XSTATE_YMM            0x4
  if (has_osxsave)
    asm (".byte 0x0f; .byte 0x01; .byte 0xd0"
     : "=a" (eax), "=d" (edx)
     : "c" (XCR_XFEATURE_ENABLED_MASK));

  /* Check if SSE and YMM states are supported.  */
  if (!has_osxsave
      || (eax & (XSTATE_SSE | XSTATE_YMM)) != (XSTATE_SSE | XSTATE_YMM))
    {
      has_avx = 0;
      has_fma = 0;
      has_fma4 = 0;
      has_xop = 0;
    }
--snip--

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #14 from Jonathan Wakely <redi at gcc dot gnu.org> 2013-01-21 19:58:48 UTC ---
(In reply to comment #13)
> > Maybe GCC could work around it by checking both flags in the AVX detection
> > logic.
> 
> Recent 4.6+ does. Please see driver-i386.c around line 470:

Ah, thank you! I only looked in the 4.6.3 and 4.7.2 releases, not the branch
heads.

That's good to know.

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[jpyeron at pdinc dot us]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

--- Comment #15 from Jason Pyeron <jpyeron at pdinc dot us> 2013-01-21 20:14:10 UTC ---
(In reply to comment #12)
> (In reply to comment #11)
> > I will try 4.6 to prove that. Off to build 4.6...
> ...
> If that prints 0 for OSXSAVE and non-zero for AVX then it's the same problem I
> had.
> Maybe GCC could work around it by checking both flags in the AVX detection
> logic.

mockbuild@centos6-64bit-builder ~/build/BUILD/gcc/test (mock-chroot)
$ gcc test.c -march=native

mockbuild@centos6-64bit-builder ~/build/BUILD/gcc/test (mock-chroot)
$ ./a.out
bit_OSXSAVE (134217728) = 0
bit_AVX (268435456) = 0

mockbuild@centos6-64bit-builder ~/build/BUILD/gcc/test (mock-chroot)
$ gcc test.c

mockbuild@centos6-64bit-builder ~/build/BUILD/gcc/test (mock-chroot)
$ ./a.out
bit_OSXSAVE (134217728) = 0
bit_AVX (268435456) = 0

on the other note, I am currently building
svn://gcc.gnu.org/svn/gcc/branches/gcc-4_6-branch

[Bug target/56068] -march=native creates Illegal instruction on KVM guests

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |INVALID

--- Comment #16 from Richard Biener <rguenth at gcc dot gnu.org> 2013-01-23 16:42:19 UTC ---
mockbuild@centos6-64bit-builder ~/build/BUILD/tmp (mock-chroot)
$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 13

I think that's the old qemu bug of providing a non-sensical CPUID
family/model/vendor combination by default on x86_64.  AMD family 6
is 32bits only.  Intel family 6 has 64bit support.

You'll hit funny issues with GMP as well.  Thus, fix your KVM/QEMU
config.