[Bug lto/110710] New: LTO linker on Windows creates an invalid Makefile

[Bug lto/110710] New: LTO linker on Windows creates an invalid Makefile

[cz.finn.cz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

            Bug ID: 110710
           Summary: LTO linker on Windows creates an invalid Makefile
           Product: gcc
           Version: 12.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: cz.finn.cz at gmail dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

In MS/Windows, when linking an executable file in a single thread (-flto=1),
the linker issues a warning:

lto-wrapper.exe: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper.exe: note: see the '-flto' option documentation for more
information

When linking using -flto=auto, a temporary makefile is created in a %TEMP%
directory. This makefile contains something like this:

C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.ltrans.o:
        @arm-none-eabi-gcc  '-xlto' '-c' '-fno-openmp' '-fno-openacc'
'-fno-pie' '-fcf-protection=none' '-mcpu=cortex-r5' '-mfloat-abi=hard'
'-mfpu=vfpv3-d16' '-mbig-endian' '-mthumb' '-mthumb-interwork'
'-mlibarch=armv7-r+fp+idiv' '-march=armv7-r+fp+idiv' '-O2' '-mcpu=cortex-r5'
'-mfloat-abi=hard' '-mfpu=vfpv3-d16' '-mbig-endian' '-mthumb'
'-mthumb-interwork' '-g' '-nostartfiles' '-gdwarf-2' '-O2'
'-falign-functions=8' '-mbe32' '-L' 'build/release' '-mlibarch=armv7-r+fp+idiv'
'-march=armv7-r+fp+idiv' '-dumpdir'
'dist/release/CA002734_2QC_1.0.0_20230714.elf.' '-dumpbase'
'dist/release/CA002734_2QC_1.0.0_20230714.elf.ltrans0.ltrans' '-fltrans' '-o'
'C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.ltrans.o'
'C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.o'
        @-touch -r C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.o
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.o.tem > /dev/null 2>&1 &&
mv C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.o.tem
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.o
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.ltrans.o:
        @arm-none-eabi-gcc  '-xlto' '-c' '-fno-openmp' '-fno-openacc'
'-fno-pie' '-fcf-protection=none' '-mcpu=cortex-r5' '-mfloat-abi=hard'
'-mfpu=vfpv3-d16' '-mbig-endian' '-mthumb' '-mthumb-interwork'
'-mlibarch=armv7-r+fp+idiv' '-march=armv7-r+fp+idiv' '-O2' '-mcpu=cortex-r5'
'-mfloat-abi=hard' '-mfpu=vfpv3-d16' '-mbig-endian' '-mthumb'
'-mthumb-interwork' '-g' '-nostartfiles' '-gdwarf-2' '-O2'
'-falign-functions=8' '-mbe32' '-L' 'build/release' '-mlibarch=armv7-r+fp+idiv'
'-march=armv7-r+fp+idiv' '-dumpdir'
'dist/release/CA002734_2QC_1.0.0_20230714.elf.' '-dumpbase'
'dist/release/CA002734_2QC_1.0.0_20230714.elf.ltrans1.ltrans' '-fltrans' '-o'
'C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.ltrans.o'
'C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.o'
        @-touch -r C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.o
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.o.tem > /dev/null 2>&1 &&
mv C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.o.tem
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.o
.PHONY: all
all: \
        C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.ltrans.o \
        C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.ltrans.o

This file contains constructions, which obviously cannot work in Windows SHELL,
such as redirection to /dev/null, using of tools/commands, which generally may
not be available (mv, touch). As a result, the make execution fails, and the
top-level make issues errors, such as

The system cannot find the file specified.
make[2]: [C:\Users\K.GOTT\AppData\Local\Temp\1\ccxCkgV8.mk:6:
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans1.ltrans.o] Error 1
(ignored)
The system cannot find the file specified.
make[2]: [C:\Users\K.GOTT\AppData\Local\Temp\1\ccxCkgV8.mk:3:
C:\Users\K.GOTT\AppData\Local\Temp\1\cc071ciG.ltrans0.ltrans.o] Error 1
(ignored)

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
In a mingw env, I thought most of the posix simple posix commands are there.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2023-07-18
             Status|UNCONFIRMED                 |WAITING
     Ever confirmed|0                           |1

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Even make is not normally on windows.

So what env are you using to build programs? Gcc assumes simple commands are
working in general.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[cz.finn.cz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #3 from Jan Nárovec <cz.finn.cz at gmail dot com> ---
The commands (mv, touch) are not the main problem, these can be indeed somehow
provided, but redirecion to /dev/null cannot work at all in the windows SHELL
(cmd.exe).

We use gcc for a cross compilation, primary development platform is MS/Windows
and the target platform is ARM Cortex-R.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #4 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
You cannot use posix make without a posix shell. That means /dev/null support
too.

https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap10.html

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[cz.finn.cz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #5 from Jan Nárovec <cz.finn.cz at gmail dot com> ---
We are using GNU/make (which is a superset of POSIX make, I don't know, whether
it makes some difference) with SHELL=cmd.exe. If GCC intends to support only
POSIX shells, it is OK for us (we will condsider using WSL) and you can close
this issue as invalid.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[amy at amyspark dot me]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

Amyspark <amy at amyspark dot me> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |amy at amyspark dot me

--- Comment #6 from Amyspark <amy at amyspark dot me> ---
This still happens in GCC 13.2, and breaks standalone usage of GCC outside of a
MSYS2 (or similar) shell.

See my report (I did it way before finding this bug, will cross link now):
https://github.com/StephanTLavavej/mingw-distro/issues/102

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

peter0x44 at disroot dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |peter0x44 at disroot dot org

--- Comment #7 from peter0x44 at disroot dot org ---
The way make works is it prefers using an sh.exe if it finds it on the PATH.

https://git.savannah.gnu.org/cgit/make.git/tree/src/variable.c#n1637
(see variable.c of GNU make)

Otherwise, cmd.exe is used. This is probably why many people don't notice, as
most "mingw" environments do have this. But "mingw" does not imply "has an
sh.exe shell". You can totally add gcc to your cmd.exe PATH and use it fine,
and you can also write Makefiles with GNU Make that expect to use cmd.exe to
execute their build rules.

If GCC's generated temporary Makefiles expect to run under a "posix sh" always,
I believe they should have on the first line:

SHELL = /bin/sh

For a workaround, I would recommend busybox-w32. It's what w64devkit uses.
You can just put it on your PATH and see if it solves that issue.

It's what w64devkit uses, and I have used `-flto=auto` with it successfully
before.

As for actually reviewing this issue in the generated Makefiles, there are a
few approaches that can be used.

One of them it to utilize a "polyglot test" to tell which shell the build rules
are being executed under.

Something like:

SHELLTYPE := posix
ifeq ($(shell echo "test", "test"))
    SHELLTYPE := cmd
endif

This works because cmd.exe's echo command writes quotes.
under cmd.exe echo "test" will write:
"test"
and under a posix:
test (without quotes)

Then, the Makefile can later do something like:
ifeq (posix, $(SHELLTYPE))
    gcc whatever.c 2>/dev/null
else
    gcc whatever.c 2>NUL
endif

Now, this is an enormous HACK, but it worked okay in some cases, for me.
I implemented it in a different Makefile generating project here:
https://github.com/premake/premake-core/pull/2039

But is probably a terrible idea for GCC, since I believe ifeq is a GNU Make
extension. So is $(shell), afaik. I'm only presenting this option for
completeness.

I think the better solution for this case would be for --host=*-w64-mingw32,
write out Makefiles with:

SHELL = cmd
and then proceed to use all the cmd.exe shell equivalents, like 2>NUL, move
instead of mv, etc.

I've spent quite a bit of time before thinking and exploring this particular
"problem space", so it's just my analysis here. Unfortunately, I think there
are no "nice" ways to solve this problem.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #8 from peter0x44 at disroot dot org ---
The way make works is it prefers using an sh.exe if it finds it on the PATH.

https://git.savannah.gnu.org/cgit/make.git/tree/src/variable.c#n1637
(see variable.c of GNU make)

Otherwise, cmd.exe is used. This is probably why many people don't notice, as
most "mingw" environments do have this. But "mingw" does not imply "has an
sh.exe shell". You can totally add gcc to your cmd.exe PATH and use it fine,
and you can also write Makefiles with GNU Make that expect to use cmd.exe to
execute their build rules.

If GCC's generated temporary Makefiles expect to run under a "posix sh" always,
I believe they should have on the first line:

SHELL = /bin/sh

For a workaround, I would recommend busybox-w32. It's what w64devkit uses.
You can just put it on your PATH and see if it solves that issue.

It's what w64devkit uses, and I have used `-flto=auto` with it successfully
before.

As for actually reviewing this issue in the generated Makefiles, there are a
few approaches that can be used.

One of them it to utilize a "polyglot test" to tell which shell the build rules
are being executed under.

Something like:

SHELLTYPE := posix
ifeq ($(shell echo "test", "test"))
    SHELLTYPE := cmd
endif

This works because cmd.exe's echo command writes quotes.
under cmd.exe echo "test" will write:
"test"
and under a posix:
test (without quotes)

Then, the Makefile can later do something like:
ifeq (posix, $(SHELLTYPE))
    gcc whatever.c 2>/dev/null
else
    gcc whatever.c 2>NUL
endif

Now, this is an enormous HACK, but it worked okay in some cases, for me.
I implemented it in a different Makefile generating project here:
https://github.com/premake/premake-core/pull/2039

But is probably a terrible idea for GCC, since I believe ifeq is a GNU Make
extension. So is $(shell), afaik. I'm only presenting this option for
completeness.

I think the better solution for this case would be for --host=*-w64-mingw32,
write out Makefiles with:

SHELL = cmd
and then proceed to use all the cmd.exe shell equivalents, like 2>NUL, move
instead of mv, etc.

I've spent quite a bit of time before thinking and exploring this particular
"problem space", so it's just my analysis here. Unfortunately, I think there
are no "nice" ways to solve this problem.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #9 from peter0x44 at disroot dot org ---
That sent twice... sorry

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #10 from peter0x44 at disroot dot org ---

> For a workaround, I would recommend busybox-w32. It's what w64devkit uses.
> You can just put it on your PATH and see if it solves that issue.

Just to be completely explicit, what I mean to do is:
Download busybox.exe from here:
https://frippery.org/busybox/

Rename it to "sh.exe", and place it on your PATH.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #11 from peter0x44 at disroot dot org ---
.I did some digging into why lto-wrapper.cc is emitting these commands
It seems that they are not essential.

/* If we are not preserving the ltrans input files then
   truncate them as soon as we have processed it.  This
   reduces temporary disk-space usage.  */
if (! save_temps)
  fprintf (mstream, "\t@-touch -r \"%s\" \"%s.tem\" > /dev/null "
           "2>&1 && mv \"%s.tem\" \"%s\"\n",
           input_name, input_name, input_name, input_name);

This is the only thing in the Makefile that actually needs the posix shell
commands.

When emitted, it looks like this:
        @-touch -r "/tmp/ccprwrlE.ltrans0.o" "/tmp/ccprwrlE.ltrans0.o.tem" >
/dev/null 2>&1 && mv "/tmp/ccprwrlE.ltrans0.o.tem" "/tmp/ccprwrlE.ltrans0.o"

I think what this is doing is copying the timestamp of the ".o" file onto a
".o.tmp", then renaming the ".o.tmp" to be the ".o". This is a truncation
because the file is not supposed to exist in advance, so touch is creating it.

So, the purpose is to truncate a file without updating its modified timestamp.

I checked how this could be done in cmd, and it seems it is rather deficient
here. There is no direct equivalent of "touch", but some commands can update
the timestamp of files to the current time. There is no way to copy a timestamp
from another file without using powershell it seems, which make can't use.

So, the questions I'm left with are:
Does it matter if the timestamp doesn't get preserved?
Does it matter if the file doesn't get truncated?
And lastly, is it possible to copy the timestamp of one file to another using
cmd commands?

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #12 from peter0x44 at disroot dot org ---
diff --git a/gcc/lto-wrapper.cc b/gcc/lto-wrapper.cc
index 5186d040ce0..7119157d830 100644
--- a/gcc/lto-wrapper.cc
+++ b/gcc/lto-wrapper.cc
@@ -2024,9 +2024,7 @@ cont:
                 truncate them as soon as we have processed it.  This
                 reduces temporary disk-space usage.  */
              if (! save_temps)
-               fprintf (mstream, "\t@-touch -r \"%s\" \"%s.tem\" > /dev/null "
-                        "2>&1 && mv \"%s.tem\" \"%s\"\n",
-                        input_name, input_name, input_name, input_name);
+               fprintf (mstream, "\t@-copy /y nul \"%s\"" input_name);
            }
          else
            {

This experimental patch works for me, it doesn't seem to matter if the
timestamp gets updated. I guess what's left to do is write code that runs "sh
-c "true"" to determine if there is an sh.exe present, and then emit the
correct fragment to truncate the file.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[peter0x44 at disroot dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #13 from peter0x44 at disroot dot org ---
https://gcc.gnu.org/pipermail/gcc-patches/2024-March/648427.html

Patch submitted. Couldn't figure out how to assign myself in bugzilla.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[egallager at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

Eric Gallager <egallager at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch
                 CC|                            |egallager at gcc dot gnu.org
                URL|                            |https://gcc.gnu.org/piperma
                   |                            |il/gcc-patches/2024-March/6
                   |                            |48427.html
             Status|WAITING                     |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |peter0x44 at disroot dot org

--- Comment #14 from Eric Gallager <egallager at gcc dot gnu.org> ---
(In reply to Peter Damianov from comment #13)
> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/648427.html
> 
> Patch submitted. Couldn't figure out how to assign myself in bugzilla.

There's a privilege for it; I just assigned you.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #15 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Richard Biener <rguenth@gcc.gnu.org>:

https://gcc.gnu.org/g:985b5a90f70c7376c771317c6c8c3bc5ef05e227

commit r15-107-g985b5a90f70c7376c771317c6c8c3bc5ef05e227
Author: Peter Damianov <peter0x44@disroot.org>
Date:   Sun Apr 28 16:16:11 2024 -0700

    Driver: Add new -truncate option

    This commit adds a new option to the driver that truncates one file after
    linking.

    Tested likeso:

    $ gcc hello.c -c
    $ du -h hello.o
    4.0K  hello.o
    $ gcc hello.o -truncate hello.o
    $ ./a.out
    Hello world
    $ du -h hello.o
    $ 0   hello.o

    $ gcc hello.o -truncate
    gcc: error: missing filename after '-truncate'

    The motivation for adding this is PR110710. It is used by lto-wrapper to
    truncate files in a shell-independent manner.

    Signed-off-by: Peter Damianov <peter0x44@disroot.org>

            PR lto/110710
            * common.opt (truncate): New internal option.
            * gcc.cc (totruncate_file): New global.
            (driver_handle_option): Handle -truncate <file>.
            (driver::final_actions): Truncate the file indicated.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

--- Comment #16 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Richard Biener <rguenth@gcc.gnu.org>:

https://gcc.gnu.org/g:a704554d2e798e2e1b74b9fea4baf3477180bd9d

commit r15-108-ga704554d2e798e2e1b74b9fea4baf3477180bd9d
Author: Peter Damianov <peter0x44@disroot.org>
Date:   Sun Apr 28 16:16:12 2024 -0700

    lto-wrapper: Truncate files using -truncate driver option [PR110710]

    This commit changes the Makefiles generated by lto-wrapper to no longer use
    the "mv" and "touch" shell commands. These don't exist on Windows, so when
the
    Makefile attempts to call them, it results in errors like:
    The system cannot find the file specified.

    This problem only manifested when calling gcc from cmd.exe, and having no
    sh.exe present on the PATH. The Windows port of GNU Make searches the PATH
for
    an sh.exe, and uses it if present.

    I have tested this in environments with and without sh.exe on the PATH and
    confirmed it works as expected.

    Signed-off-by: Peter Damianov <peter0x44@disroot.org>

            PR lto/110710
            * lto-wrapper.cc (run_gcc): Instead of truncating a processed
            ltrans input from the Makefile use the new -truncate option
            to accomplish the same.

[Bug lto/110710] LTO linker on Windows creates an invalid Makefile

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110710

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to work|                            |15.0

--- Comment #17 from Richard Biener <rguenth at gcc dot gnu.org> ---
This has now been fixed on trunk, it's too late for 14.1 but we can backport
later if the patch doesn't cause any issues.