[Bug bootstrap/111768] New: Bootstrap failure with -march=native on Intel Alder Lake CPUs because of differing cache sizes

[Bug bootstrap/111768] New: Bootstrap failure with -march=native on Intel Alder Lake CPUs because of differing cache sizes

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

            Bug ID: 111768
           Summary: Bootstrap failure with -march=native on Intel Alder
                    Lake CPUs because of differing cache sizes
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: bootstrap
          Assignee: unassigned at gcc dot gnu.org
          Reporter: sjames at gcc dot gnu.org
  Target Milestone: ---

Please bare with me on this one.

We've had a bunch of reports downstream of bootstrap failures on Intel Alder
Lake CPUs with -march=native:
* https://bugs.gentoo.org/904426
* https://bugs.gentoo.org/908523
* https://bugs.gentoo.org/915389 (this has a bit more detail)

Alder Lake has a big.little architecture.

We end up with:
```
$ diffoscope ./stage2-x86_64-pc-linux-gnu/32/libgcc/eqhf2.o
./stage3-x86_64-pc-linux-gnu/32/libgcc/eqhf2.o
```
[...]
│ -  [    33]  GNU C17 13.2.1 20230826 -march=alderlake -mmmx -mpopcnt -msse
-msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -mno-sse4a -mno-fma4 -mno-xop
-mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw
-mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi
-mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq
-mno-avx512vbmi2 -mgfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg
-mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote
-mclflushopt -mclwb -mno-clzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr
-mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mmovdir64b -mmovdiri -mno-mwaitx
-mpconfig -mpku -mno-prefetchwt1 -mprfchw -mptwrite -mrdpid -mrdrnd -mrdseed
-mno-rtm -mserialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes
-mwaitpkg -mno-wbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile
-mno-amx-int8 -mno-amx-bf16 -mno-uintr -mhreset -mno-kl -mno-widekl -mavxvnni
-mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd
-mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex
--param=l1-cache-size=48 --param=l1-cache-line-size=64
--param=l2-cache-size=24576 -mtune=alderlake -m32 -mlong-double-80 -msse2 -g -g
-g -O2 -O3 -O2 -O2 -O3 -fbuilding-libgcc -fno-stack-protector
-fno-stack-clash-protection -fpic -fvisibility=hidden
│ -  [   578]  _Float16
│ -  [   581]  _FP_UNPACK_RAW_1_flo
│ -  [   596]  long long unsigned int
│ -  [   5ad]  sign
│ -  [   5b2]  _fcw
│ +  [    33]  _Float16
│ +  [    3c]  _FP_UNPACK_RAW_1_flo
│ +  [    51]  long long unsigned int
│ +  [    68]  sign
│ +  [    6d]  _fcw
│ +  [    72]  GNU C17 13.2.1 20230826 -march=alderlake -mmmx -mpopcnt -msse
-msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -mno-sse4a -mno-fma4 -mno-xop
-mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw
-mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi
-mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq
-mno-avx512vbmi2 -mgfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg
-mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote
-mclflushopt -mclwb -mno-clzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr
-mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mmovdir64b -mmovdiri -mno-mwaitx
-mpconfig -mpku -mno-prefetchwt1 -mprfchw -mptwrite -mrdpid -mrdrnd -mrdseed
-mno-rtm -mserialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes
-mwaitpkg -mno-wbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile
-mno-amx-int8 -mno-amx-bf16 -mno-uintr -mhreset -mno-kl -mno-widekl -mavxvnni
-mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd
-mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex
--param=l1-cache-size=32 --param=l1-cache-line-size=64
--param=l2-cache-size=24576 -mtune=alderlake -m32 -mlong-double-80 -msse2 -g -g
-g -O2 -O3 -O2 -O2 -O3 -fbuilding-libgcc -fno-stack-protector
-fno-stack-clash-protection -fpic -fvisibility=hidden
[...]
```

$ wdiff /tmp/a /tmp/b
<d>   DW_AT_producer    : (strp) (offset: [-0x33):-] {+0x72):+} GNU C17 13.2.1
20230826 -march=alderlake -mmmx -mpopcnt -msse -msse3 -mssse3 -msse4.1 -msse4.2
-mavx -mavx2 -mno-sse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2
-maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd
-mno-avx512er -mno-avx512pf -mno-avx512vbmi -mno-avx512ifma -mno-avx5124vnniw
-mno-avx5124fmaps -mno-avx512vpopcntdq -mno-avx512vbmi2 -mgfni -mvpclmulqdq
-mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect
-mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mno-clzero -mcx16
-mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe
-mmovdir64b -mmovdiri -mno-mwaitx -mpconfig -mpku -mno-prefetchwt1 -mprfchw
-mptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mserialize -mno-sgx -msha -mshstk
-mno-tbm -mno-tsxldtrk -mvaes -mwaitpkg -mno-wbnoinvd -mxsave -mxsavec
-mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr
-mhreset -mno-kl -mno-widekl -mavxvnni -mno-avx512fp16 -mno-avxifma
-mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi
-mno-raoint -mno-amx-complex [---param=l1-cache-size=48-]
{+--param=l1-cache-size=32+} --param=l1-cache-line-size=64
--param=l2-cache-size=24576 -mtune=alderlake -m32 -mlong-double-80 -msse2 -g -g
-g -O2 -O3 -O2 -O2 -O3 -fbuilding-libgcc -fno-stack-protector
-fno-stack-clash-protection -fpic -fvisibility=hidden

We have two differences:
* the offset (let's ignore that)
* l1 cache size: [---param=l1-cache-size=48-] {+--param=l1-cache-size=32+}

All affected users have been able to workaround it by just using a constant
expansion of -march=native, and users who have run the expansion repeatedly to
check the results see occasional =32 popping up.

[Bug bootstrap/111768] Bootstrap failure with -march=native on Intel Alder Lake CPUs because of differing cache sizes

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

Sam James <sjames at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |arsen at gcc dot gnu.org

--- Comment #1 from Sam James <sjames at gcc dot gnu.org> ---
It's unclear to me if GCC should mangle -march=native on Alder Lake CPUs, if it
should ignore cache sizes during bootstrap comparison (ew), or if Intel need to
be consulted to find out what sort of "consistency" guarantee is even provided
here.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
I think on those soc we should ignore the cache info or set it to some common
value between the 2.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #3 from Richard Biener <rguenth at gcc dot gnu.org> ---
I'd say "don't do this" (bootstrap with -march=native).  Alternatively use a
taskset to confine to either big or little cores.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #4 from Hongtao.liu <crazylht at gmail dot com> ---
I checked Alderlake's L1 cachesize and it is indeed 48, and L1 cachesize in
alderlake_cost is set to 32.
But then again, we have a lot of different platforms that share the same cost 
and they may have different L1 cachesizes, but from a micro-architecture tuning
point of view, it doesn't make a difference. A separate cost if only the L1
cachesize is different is quite unnecessary（the size itself is just a parameter
for the software prefetch, it doesn't have to be real hardware cachesize)

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[amonakov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #5 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
I think it's similar to attempting -march=native under distcc, which is already
warned about on Gentoo wiki: https://wiki.gentoo.org/wiki/Distcc

The difference here is that Intel so far decided to make ISA feature set the
same between 'performance' and 'power-efficient' cores, so the differences for
-march=native detection are minimal.

Intel also added a cpuid bit for hybrid CPUs, so in principle native arch
detection could inspect that bit and then override l1-cache-size to 32 KiB
(having the exact size in the param is not important, specifying a lower value
is ok), or just drop it and let cc1 fall back to the default value (64) from
params.opt.

Short term, I would advise users to add --param=l1-cache-size=32 after
-march=native in CFLAGS.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[arsen at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #6 from Arsen Arsenović <arsen at gcc dot gnu.org> ---
this poses another problem too, though: should big and little cores ever differ
in ISA support levels, building on big cores (which seems like a reasonable
thing to do) with -march=native could lead to generating code incompatible with
little cores.  perhaps it'd be reasonable to always probe all cores (is that
possible?) and pick a common subset?

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[amonakov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

Alexander Monakov <amonakov at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |amonakov at gcc dot gnu.org

--- Comment #7 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
I'm afraid hybrid CPUs with varying ISA feature sets are not practical for the
current ecosystem: you wouldn't be able to reschedule from a higher- to
lower-capable core. Not to mention scenarios like Mesa on-disk llvmpipe shader
cache.

"Always" probing all cores is a not a good idea (the compiler would have to
manually reschedule itself to all cores, of which there could be hundreds).
Plus, portable API for such probing across available cores does not exist
afaik.

I think releasing an x86 hybrid CPU with varying capabilities across cores
would require substantial preparatory work in the kernel and likely in the
userland as well, so probably best to leave it until the time comes and
specifics of what can differ are known.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[arsen at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #8 from Arsen Arsenović <arsen at gcc dot gnu.org> ---
(In reply to Alexander Monakov from comment #7)
> I'm afraid hybrid CPUs with varying ISA feature sets are not practical for
> the current ecosystem: you wouldn't be able to reschedule from a higher- to
> lower-capable core. Not to mention scenarios like Mesa on-disk llvmpipe
> shader cache.

indeed (but I believe it did happen with Alder Lake already, by accident, with
AVX512 on P-cores but not on E-cores).

> "Always" probing all cores is a not a good idea (the compiler would have to
> manually reschedule itself to all cores, of which there could be hundreds).
> Plus, portable API for such probing across available cores does not exist
> afaik.

I'd consider this close enough to 'not possible' ;P

my thinking was does cpuid provide a way to query cross-CPU (or CPU 'group' I
suppose).  if not, we're definitely better off just using a common, smaller
cache size for intel hybrid CPUs (at least for now)

> I think releasing an x86 hybrid CPU with varying capabilities across cores
> would require substantial preparatory work in the kernel and likely in the
> userland as well, so probably best to leave it until the time comes and
> specifics of what can differ are known.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[amonakov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #9 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
(In reply to Arsen Arsenović from comment #8)
> indeed (but I believe it did happen with Alder Lake already, by accident,
> with AVX512 on P-cores but not on E-cores).

AFAIK on those Alder Lake CPUs you could only get AVX-512 by disabling E-cores
in the BIOS, so you couldn't boot in a configuration when both E-cores are
available and AVX-512 on P-cores is available.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #10 from Hongtao.liu <crazylht at gmail dot com> ---
> indeed (but I believe it did happen with Alder Lake already, by accident,
> with AVX512 on P-cores but not on E-cores).

AVX512 is physically fused off for Alderlake P-core, P-core and E-core share
the same ISA level(AVX2).

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[amonakov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #11 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
(In reply to Hongtao.liu from comment #10)
> > indeed (but I believe it did happen with Alder Lake already, by accident,
> > with AVX512 on P-cores but not on E-cores).
> 
> AVX512 is physically fused off for Alderlake P-core, P-core and E-core share
> the same ISA level(AVX2).

I think Arsen means initial Alder Lake batches, where AVX-512 wasn't yet fused
off (but BIOS support was unofficial/experimental anyway).

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2023-10-12

--- Comment #12 from Richard Biener <rguenth at gcc dot gnu.org> ---
Looking at the 'hybrid' flag in cpuid sounds like the most reasonable thing to
do, possibly simply skipping auto-detection for the problematical parts
(L1 and L2 cache sizes) as Alex suggests.

[Bug target/111768] X86: -march=native does not support alder lake big.little cache infor correctly

[stormbyte at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111768

--- Comment #13 from David C. Manuelda <stormbyte at gmail dot com> ---
I'd suggest for now to pick a common value in order to prevent the compilation
failure (in stage comparison) while a proper fix/workaround is picked.