[Bug target/105144] New: [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[Bug target/105144] New: [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

            Bug ID: 105144
           Summary: [12 Regression] Bootstrap + install failure on aarch64
                    due to aarch64-tune.md likely since r12-7842
           Product: gcc
           Version: 12.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: jakub at gcc dot gnu.org
  Target Milestone: ---

Latest Fedora package builds fail on aarch64, I believe it is caused by
r12-7842-g9f37d31324f89d0b7b2abac988a976d121ae29c6 (but haven't actually
bisected).  See e.g.
https://kojipkgs.fedoraproject.org//work/tasks/8070/85048070/build.log

We are doing a --with-build-config=bootstrap-lto --enable-link-serialization=1
profiledbootstrap with system gcc less than 4 weeks old gcc trunk snapshot.
What I see in the log is that it bootstraps just fine, then during make install
does:
/bin/sh ../../gcc/config/aarch64/gentune.sh \
        ../../gcc/config/aarch64/aarch64-cores.def > \
        tmp-aarch64-tune.md
/bin/sh ../../gcc/../move-if-change tmp-aarch64-tune.md \
        ../../gcc/config/aarch64/aarch64-tune.md
echo timestamp > s-aarch64-tune-md
which hasn't been done during the bootstrap time, this changes aarch64-tune.md
and so it rebuilds quite a few but not all *.o files but with the system g++
rather than the new compiler and ICEs on lto1 because there were silent changes
in the LTO bytecode between compiler from ~ 4 weeks ago and current one.
There are multiple issues related to this:
1) the r12-7842 change changed aarch64-cores.def (moved neoverse-n2 entry
later)
   but didn't regenerate and commit aarch64-tune.md
2) the reason why gentune.sh isn't invoked during bootstrap but is during
   install is I believe caused by:
# Dependencies for the md file.  The first time through, we just assume
# the md file itself and the generated dependency file (in order to get
# it built).  The second time through we have the dependency file.
-include mddeps.mk
MD_DEPS = s-mddeps $(md_file) $(MD_INCLUDES)

s-mddeps: $(md_file) $(MD_INCLUDES) build/genmddeps$(build_exeext)
        $(RUN_GEN) build/genmddeps$(build_exeext) $(md_file) > tmp-mddeps
        $(SHELL) $(srcdir)/../move-if-change tmp-mddeps mddeps.mk
        $(STAMP) s-mddeps
   When bootstrapping gcc into a fresh new object directory (well, set of them,
   like stage{1,2,3}-gcc, stageprofile-gcc, prev-gcc, gcc etc.),
   mddeps.mk doesn't
   exist and so isn't included and everything that depends on the *.md file(s)
   depends just on s-mddeps that creates the mddeps.mk, on the main $(md_file)
   (gcc/config/aarch64/aarch64.md in this case) and that is it.  Not a big deal
   the first time, as everything is being rebuild.  But for aarch64 it means
   that gentune.sh which is from:
$(srcdir)/config/aarch64/aarch64-tune.md: s-aarch64-tune-md; @true
s-aarch64-tune-md: $(srcdir)/config/aarch64/gentune.sh \
        $(srcdir)/config/aarch64/aarch64-cores.def
        $(SHELL) $(srcdir)/config/aarch64/gentune.sh \
                $(srcdir)/config/aarch64/aarch64-cores.def > \
                tmp-aarch64-tune.md
        $(SHELL) $(srcdir)/../move-if-change tmp-aarch64-tune.md \
                $(srcdir)/config/aarch64/aarch64-tune.md
        $(STAMP) s-aarch64-tune-md
   isn't invoked at that point.  Next during make install, mddeps.mk already
   exists and so it invokes it, regenerates it and as it is different,
   move-if-change overwrites it
3) I think it is wrong to change files in $(srcdir) or its subdirs during
   build except for --enable-maintainer-mode, users could have the tree
   in read-only location etc.

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |jakub at gcc dot gnu.org
           Priority|P3                          |P1
   Last reconfirmed|                            |2022-04-04
     Ever confirmed|0                           |1
   Target Milestone|---                         |12.0
             Status|UNCONFIRMED                 |ASSIGNED

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 52742
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52742&action=edit
gcc12-pr105144.patch

Fix for 1) and 2).

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #2 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 52743
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52743&action=edit
gcc12-pr105144-2.patch

And incremental fix for 3).

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #3 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:376cf6d19cdcb791a4257bc1cafabfae89227683

commit r12-7981-g376cf6d19cdcb791a4257bc1cafabfae89227683
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Mon Apr 4 12:09:26 2022 +0200

    aarch64: Fix aarch64-tune.md (re)generation [PR105144]

    As I wrote in the PR, our Fedora trunk gcc builds likely after r12-7842
    change are now failing (lto1 crashes).
    What happens is that when one bootstraps into an empty build directory
    (or set of them), mddeps.mk doesn't exist yet and so Makefile doesn't
    include it.  When building from an empty dir, that is usually not a big
    issue, it is enough when various build directory files depend on just
    $(srcdir)/config/aarch64/aarch64.md, those files don't exist and
    aarch64.md does, so they are built, so is mddeps.mk.
    But because the other dependencies aren't there (in particular
    $(srcdir)/config/aarch64/aarch64-tune.md ), the
    s-aarch64-tune-md rule isn't invoked to regenerate that file and the
    r12-7842 commit reordered aarch64-cores.def entries but didn't commit
    regenerated aarch64-tune.md.  Because it is just reordering in
    aarch64-tune.md, it actually doesn't matter and bootstraps succeeds.
    But then during make install, mddeps.mk exists already in gcc/ directory,
    it sees that aarch64-cores.def is newer than aarch64-tune.md (unless
    gen_update is used, that just touches aarch64-tune.md to make sure it
    is newer) and regenerates it and as it is different, make install rebuilds
    a large subset of the *.o files, but this time with the system g++
    rather than previous stage one.  And during lto linking of it there
    are differences in LTO bytecode between the compilers and we crash.

    The following patch fixes that by regenerating aarch64-tune.md
    (what was forgotten in r12-7842) and by adding a dependency from
    s-mddeps to s-aarch64-tune-md, which makes sure that even when mddeps.mk
    doesn't exist yet make sees the dependency and regenerates aarch64-tune.md
    if needed.

    2022-04-04  Jakub Jelinek  <jakub@redhat.com>

            PR target/105144
            * config/aarch64/t-aarch64 (s-mddeps): Depend on s-aarch64-tune-md.
            * config/aarch64/aarch64-tune.md: Regenerated.

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #4 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:2f0610acbc056052a108e4a46911fc21d0dca2ab

commit r12-7983-g2f0610acbc056052a108e4a46911fc21d0dca2ab
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Mon Apr 4 13:56:32 2022 +0200

    aarch64: Restrict aarch64-tune.md regeneration to --enable-maintainer-mode
[PR105144]

    Normally updates to the source directory files are guarded with
    --enable-maintainer-mode, e.g. we don't regenerate configure, config.h,
    Makefile.in in directories that use automake etc. unless gcc is configured
    that way.  Otherwise the source tree can't be e.g. stored on a read-only
    filesystem etc.
    In gcc/Makefile.in we use @MAINT@ for that but that works because
    gcc/Makefile is generated by configure.  In config/*/t-* files we need to
    check $(ENABLE_MAINTAINER_RULES):
     # The following provides the variable ENABLE_MAINTAINER_RULES that can
     # be used in language Make-lang.in makefile fragments to enable
     # maintainer rules.  So, ENABLE_MAINTAINER_RULES is 'true' in
     # maintainer mode, and '' otherwise.
     @MAINT@ ENABLE_MAINTAINER_RULES = true

    On Mon, Apr 04, 2022 at 11:10:14AM +0100, Richard Sandiford wrote:
    > I guess the risk is that it will become even easier to forget
    > to commit an updated aarch64-tune.md.  Perhaps we should have a
    > non-maintainer rule to build aarch64-tune.md locally and check it
    > against the source-directory version, and fail the build if there's
    > a mismatch.  Or maybe we should just generate aarch64-tune.md in the
    > build directory and remove the source directory version.

    I've tried if aarch64-tune.md will be read from the build dir, but it is
    not.  The gen* files can use -I options to add additional directories, but
    they don't use them.

    Here is a variant patch which will complain and fail if there is a change
    and --enable-maintainer-mode is not enabled.

    2022-04-04  Jakub Jelinek  <jakub@redhat.com>

            PR target/105144
            * config/aarch64/t-aarch64 (s-aarch64-tune-md): Do move-if-change
            only if configured with --enable-maintainer-mode, otherwise compare
            tmp-aarch64-tune.md with $(srcdir)/config/aarch64/aarch64-tune.md
and
            if they differ, emit a message and fail.

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Fixed.

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[raj.khem at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

Khem Raj <raj.khem at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |raj.khem at gmail dot com

--- Comment #6 from Khem Raj <raj.khem at gmail dot com> ---
I am seeing a build failure which looks like a race condition to me. Perhaps
related to this or related commits in this series.

| g++  -c  
-isystem/mnt/b/yoe/master/build/tmp/work/x86_64-linux/gcc-cross-aarch64/13.0.1-r0/recipe-sysroot-native/usr/include
-O2 -pipe   -DIN_GCC  -DCROSS_DIRECTORY_STRUCTURE   -fno-exceptions -fno-rtti
-fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings
-Wcast-qual -Wmissing-format-attribute -Wconditionally-supported
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros
-Wno-overlength-strings   -DHAVE_CONFIG_H  -DGENERATOR_FILE -I. -Ibuild
-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc
-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/build
-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/../include

-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/../libcpp/include
 \
|       -o build/gencheck.o
../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/gencheck.cc
| yes
| checking for gethostbyname... yes
|
| checking for exported symbols... In file included from ./tm.h:34,
|                  from
../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/gencheck.cc:23:
| ./config/aarch64/aarch64.h:164:10: fatal error:
aarch64-option-extensions.def: No such file or directory
|   164 | #include "aarch64-option-extensions.def"
|       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| compilation terminated.
| make[1]: *** [Makefile:2855: build/gencheck.o] Error 1
| make[1]: *** Waiting for unfinished jobs....

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[raj.khem at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #7 from Khem Raj <raj.khem at gmail dot com> ---
in Yocto we build outside the source tree. And during cross-build it will copy
aarch64.h into builddir and use it from there but then build can not find files
it includes since its being included from builddir. This patch below helps

Subject: [PATCH] aarch64: Fix include paths when S != B

aarch64.h gets copied into build directory when built out of tree, in
this case build uses this file but does not find the includes inside it
since they are not found in any of include paths specified in compiler
cmdline.

Fixes build errors like

% g++ -c  
-isystem/mnt/b/yoe/master/build/tmp/work/x86_64-linux/gcc-cross-aarch64/13.0.1-r0/recipe-sysroot-native/usr/include
-O2 -pipe   -DIN_GCC  -DCROSS_DIRECTORY_STRUCTURE   -fno-exceptions -fno-rtti
-fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings
-Wcast-qual -Wmissing-format-attribute -Wconditionally-supported
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros
-Wno-overlength-strings   -DHAVE_CONFIG_H  -DGENERATOR_FILE -I. -Ibuild
-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc
-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/build
-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/../include

-I../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/../libcpp/include
-o build/gencheck.o
../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/gencheck.cc
In file included from ./tm.h:34,
                 from
../../../../../../../work-shared/gcc-13.0.1-r0/gcc-b2ec2504af77b35e748067eeb846821d12a6b6b4/gcc/gencheck.cc:23:
./config/aarch64/aarch64.h:164:10: fatal error: aarch64-option-extensions.def:
No such file or directory
  164 | #include "aarch64-option-extensions.def"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Upstream-Status: Pending

Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
 gcc/config/aarch64/aarch64.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/gcc/config/aarch64/aarch64.h b/gcc/config/aarch64/aarch64.h
index 7129ed1ff37..3ca7068b4cb 100644
--- a/gcc/config/aarch64/aarch64.h
+++ b/gcc/config/aarch64/aarch64.h
@@ -161,8 +161,8 @@
 enum class aarch64_feature : unsigned char {
 #define AARCH64_OPT_EXTENSION(A, IDENT, C, D, E, F) IDENT,
 #define AARCH64_ARCH(A, B, IDENT, D, E) IDENT,
-#include "aarch64-option-extensions.def"
-#include "aarch64-arches.def"
+#include "config/aarch64/aarch64-option-extensions.def"
+#include "config/aarch64/aarch64-arches.def"
 };

 /* Define unique flags for each of the above.  */
@@ -171,8 +171,8 @@ enum class aarch64_feature : unsigned char {
     = aarch64_feature_flags (1) << int (aarch64_feature::IDENT);
 #define AARCH64_OPT_EXTENSION(A, IDENT, C, D, E, F) HANDLE (IDENT)
 #define AARCH64_ARCH(A, B, IDENT, D, E) HANDLE (IDENT)
-#include "aarch64-option-extensions.def"
-#include "aarch64-arches.def"
+#include "config/aarch64/aarch64-option-extensions.def"
+#include "config/aarch64/aarch64-arches.def"
 #undef HANDLE

 #endif

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #8 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Khem Raj from comment #7)
> in Yocto we build outside the source tree. And during cross-build it will
> copy aarch64.h into builddir and use it from there but then build can not
> find files it includes since its being included from builddir. This patch
> below helps

aarch64.h is never copied into the builddir.

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #9 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #8)
> (In reply to Khem Raj from comment #7)
> > in Yocto we build outside the source tree. And during cross-build it will
> > copy aarch64.h into builddir and use it from there but then build can not
> > find files it includes since its being included from builddir. This patch
> > below helps
> 
> aarch64.h is never copied into the builddir.

Even for cross builds.
I think something else is wrong.  Please file a different bug with the full
log? Including the part where the copy happens because I just did a:
find . -name aarch64.h
in the build directory of the cross build of GCC 12 I had around and there was
no such file happening.

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[raj.khem at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #10 from Khem Raj <raj.khem at gmail dot com> ---
(In reply to Andrew Pinski from comment #9)
> (In reply to Andrew Pinski from comment #8)
> > (In reply to Khem Raj from comment #7)
> > > in Yocto we build outside the source tree. And during cross-build it will
> > > copy aarch64.h into builddir and use it from there but then build can not
> > > find files it includes since its being included from builddir. This patch
> > > below helps
> > 
> > aarch64.h is never copied into the builddir.
> 
> Even for cross builds.
> I think something else is wrong.  Please file a different bug with the full
> log? Including the part where the copy happens because I just did a:
> find . -name aarch64.h
> in the build directory of the cross build of GCC 12 I had around and there
> was no such file happening.

you are right. I should have mentioned that these copying is done by yocto
builds  for working though its notion of multilibs [1]. However, I think if we
apply this fix it should not regress builds and we will have one less patch to
carry in yocto.

[1]
https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/gcc/gcc-multilib-config.inc#n30

[Bug target/105144] [12 Regression] Bootstrap + install failure on aarch64 due to aarch64-tune.md likely since r12-7842

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105144

--- Comment #11 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Khem Raj from comment #10)
> 
> you are right. I should have mentioned that these copying is done by yocto
> builds  for working though its notion of multilibs [1]. However, I think if
> we apply this fix it should not regress builds and we will have one less
> patch to carry in yocto.
> 
> [1]
> https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/gcc/gcc-
> multilib-config.inc#n30

Yocto notion of multilib is just broken then. Why not fix that instead?