[Bug target/114487] New: [14 regression] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[Bug target/114487] New: [14 regression] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

            Bug ID: 114487
           Summary: [14 regression] ICE when building libsdl2 on
                    -mfpmath=sse x86 with LTO
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: sjames at gcc dot gnu.org
  Target Milestone: ---

I'm not sure if this is really a target bug, or an LTO option merging bug. I
suspect the issue comes down to mismatch of -mfpmath=sse and -msse -msse2 - but
I may be wrong.

Here's a reduction but it might be a bad one, as there's conflicting return
types here.

```
$ cat SDL_test_fuzzer.i
void SDLTest_RandomUnitFloat() { }
```

```
$ cat testautomation-testautomation_pixels.i
void SDLTest_RunSuites();

void SDLTest_AssertPass();
float SDLTest_RandomUnitFloat();
typedef struct {
  int testCase;
} SDLTest_TestCaseReference;
typedef struct {
  char *name;
  void *testSetUp;
  SDLTest_TestCaseReference *testCases;
  void *testTearDown;
} SDLTest_TestSuiteReference;
void pixels_calcGammaRamp() {
  float gamma = SDLTest_RandomUnitFloat();
  SDLTest_AssertPass(gamma);
}
SDLTest_TestCaseReference pixelsTests[] = {(int)pixels_calcGammaRamp};
SDLTest_TestSuiteReference pixelsTestSuite = {0, pixelsTests};

int main() {
  SDLTest_TestSuiteReference *testSuites = &pixelsTestSuite;
  SDLTest_RunSuites(testSuites);
}
```

```
$ gcc -O3 -pipe -march=i686 -mfpmath=sse -flto -mmmx -msse -msse2
-fno-strict-aliasing SDL_test_fuzzer.i -fPIC -o SDL_test_fuzzer.o -c
$ gcc -O3 -march=i686 -flto -o a testautomation-testautomation_pixels.i
SDL_test_fuzzer.o
testautomation-testautomation_pixels.i: In function ‘pixels_calcGammaRamp’:
testautomation-testautomation_pixels.i:17:1: error: unrecognizable insn:
   17 | }
      | ^
(insn 6 5 7 2 (set (reg/v:SF 100 [ gamma ])
        (reg:SF 20 xmm0)) "testautomation-testautomation_pixels.i":15:17 -1
     (nil))
during RTL pass: vregs
testautomation-testautomation_pixels.i:17:1: internal compiler error: in
extract_insn, at recog.cc:2812
0x5722510f _fatal_insn(char const*, rtx_def const*, char const*, int, char
const*)
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/rtl-error.cc:108
0x57225146 _fatal_insn_not_found(rtx_def const*, char const*, int, char const*)
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/rtl-error.cc:116
0x56d46375 extract_insn(rtx_insn*)
        /usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/recog.cc:2812
0x56d46375 instantiate_virtual_regs_in_insn
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/function.cc:1611
0x56d46375 instantiate_virtual_regs
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/function.cc:1994
0x56d46375 execute
       
/usr/src/debug/sys-devel/gcc-14.0.9999/gcc-14.0.9999/gcc/function.cc:2041
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
lto-wrapper: fatal error: gcc returned 1 exit status
compilation terminated.
/usr/lib/gcc/i686-pc-linux-gnu/14/../../../../i686-pc-linux-gnu/bin/ld: error:
lto-wrapper failed
collect2: error: ld returned 1 exit status
```

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[14 regression] ICE when    |ICE when building libsdl2
                   |building libsdl2 on         |on -mfpmath=sse x86 with
                   |-mfpmath=sse x86 with LTO   |LTO
      Known to fail|                            |12.3.1, 13.2.1, 14.0, 7.5.0
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2024-03-27
           Keywords|                            |lto
     Ever confirmed|0                           |1

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
I can confirm the ICE.  We're expanding

;; gamma_4 = SDLTest_RandomUnitFloat ();

(call_insn/u 5 4 6 (set (reg:SF 20 xmm0)
        (call (mem:QI (symbol_ref:SI ("SDLTest_RandomUnitFloat") [flags 0x3] 
<function_decl 0x7f8b97072200 SDLTest_RandomUnitFloat>) [0
SDLTest_RandomUnitFloat S1 A8])
            (const_int 0 [0]))) "testautomation-testautomation_pixels.i":15:17
-1
     (expr_list:REG_CALL_DECL (symbol_ref:SI ("SDLTest_RandomUnitFloat") [flags
0x3]  <function_decl 0x7f8b97072200 SDLTest_RandomUnitFloat>)
        (expr_list:REG_EH_REGION (const_int 0 [0])
            (nil)))
    (nil))

(insn 6 5 0 (set (reg/v:SF 99 [ gamma ])
        (reg:SF 20 xmm0)) "testautomation-testautomation_pixels.i":15:17 -1
     (nil))

I'm not sure what's wrong - looks like a target issue to me.

I'll note -mmmx -msse isn't necessary, -msse2 is enough.  Likewise -fPIC
isn't required.  I've reproduced with -m32 added on x86_64.

GCC 13/12 are also broken the same way and GCC 7 doesn't terminate compiling.

So I'm not sure whether this is a regression (the reduced testcase, that is).
I can't find a version that works.

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #2 from Uroš Bizjak <ubizjak at gmail dot com> ---
(In reply to Richard Biener from comment #1)

> (insn 6 5 0 (set (reg/v:SF 99 [ gamma ])
>         (reg:SF 20 xmm0)) "testautomation-testautomation_pixels.i":15:17 -1
>      (nil))
> 
> I'm not sure what's wrong - looks like a target issue to me.

We are working with SFmode, so -msse is enough to trigger the bug.

This is known issue. GCC assumes that at least moves of all hard registers are
working, which is not the case when LTO-compiling
testautomation-testautomation_pixels.i with SDL_test_fuzzer.o (that enables and
uses XMM registers via -msse -mfpmath=sse).

It looks to me that the compiler hits this part in function_value_32 when
LTO-compiling:

--cut here--
  /* Override FP return register with %xmm0 for local functions when
     SSE math is enabled or for functions with sseregparm attribute.  */
  if ((fn || fntype) && (mode == SFmode || mode == DFmode))
    {
      int sse_level = ix86_function_sseregparm (fntype, fn, false);
      if (sse_level == -1)
        {
          error ("calling %qD with SSE calling convention without "
                 "SSE/SSE2 enabled", fn);
          sorry ("this is a GCC bug that can be worked around by adding "
                 "attribute used to function called");
        }
      else if ((sse_level >= 1 && mode == SFmode)
               || (sse_level == 2 && mode == DFmode))
        regno = FIRST_SSE_REG;
    }
--cut here--

Adding -msse to the second compilation works OK, removing -mfpmath=sse from the
first compilation also works OK.

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #3 from Uroš Bizjak <ubizjak at gmail dot com> ---
(In reply to Uroš Bizjak from comment #2)
> Adding -msse to the second compilation works OK, removing -mfpmath=sse from
> the first compilation also works OK.

Which makes this PR a LTO reincarnation of PR66047.

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[ubizjak at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #4 from Uroš Bizjak <ubizjak at gmail dot com> ---
(In reply to Uroš Bizjak from comment #3)
> (In reply to Uroš Bizjak from comment #2)
> > Adding -msse to the second compilation works OK, removing -mfpmath=sse from
> > the first compilation also works OK.
> 
> Which makes this PR a LTO reincarnation of PR66047.

Please see the FIXME in ix86_function_sseregparm:

          /* Refuse to produce wrong code when local function with SSE enabled
             is called from SSE disabled function.
             FIXME: We need a way to detect these cases cross-ltrans partition
             and avoid using SSE calling conventions on local functions called
             from function with SSE disabled.  For now at least delay the
             warning until we know we are going to produce wrong code.
             See PR66047  */

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org,
                   |                            |rguenth at gcc dot gnu.org

--- Comment #5 from Richard Biener <rguenth at gcc dot gnu.org> ---
Ah.  But note in this case we only have a single ltrans unit.  We might be
confused by the fact that wrong prototype will result in a
call through a float(*)() type while the actual local function doesn't
use any FP registers and has void return.

OTOH the 'void' can be exchanged for 'float' in SDL_test_fuzzer.i and the
issue still reproduces (with a single LTRANS unit).

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #6 from Richard Biener <rguenth at gcc dot gnu.org> ---
I'd say ix86_function_sseregparm should be decided at a specific point and
recorded for later use.  Alternatively there needs to be a (target) IPA
phase where we can mark functions we cannot turn into sseregparm.

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #7 from Sam James <sjames at gcc dot gnu.org> ---
(You're right, by the way. The original fails with older GCC too. Sorry!)

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #8 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Created attachment 57823
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57823&action=edit
testcase for non-lto

`-W -Wall -m32 -fwhole-program -march=i686  -O1` is enough

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #9 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #8)
> Created attachment 57823 [details]
> testcase for non-lto
> 
> `-W -Wall -m32 -fwhole-program -march=i686  -O1` is enough

Note the only difference between this and the reduced LTO case is fix up of
SDLTest_RandomUnitFloat return type so it would be more defined and then the
additions of the #pragma and the use of `-fwhole-program` option instead of LTO
(since LTO does a similar thing).

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #10 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #9)
> (In reply to Andrew Pinski from comment #8)
> > Created attachment 57823 [details]
> > testcase for non-lto
> > 
> > `-W -Wall -m32 -fwhole-program -march=i686  -O1` is enough
> 
> Note the only difference between this and the reduced LTO case is fix up of
> SDLTest_RandomUnitFloat return type so it would be more defined and then the
> additions of the #pragma and the use of `-fwhole-program` option instead of
> LTO (since LTO does a similar thing).

You can remove -fwhole-program if you make SDLTest_RandomUnitFloat static too
(since that is basically what -fwhole-program is doing after all) and you still
get the failure.

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #11 from Sam James <sjames at gcc dot gnu.org> ---
I'm going to upload the originals in case it offers more insight because the
return type isn't mismatched there.

[Bug target/114487] ICE when building libsdl2 on -mfpmath=sse x86 with LTO

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114487

--- Comment #12 from Sam James <sjames at gcc dot gnu.org> ---
Ah, wait, no point with andrew's nicer testcase ;)