[PATCH] Allow parallel toplevel bootstrap

[PATCH] Allow parallel toplevel bootstrap

[Richard Earnshaw]

This patch breaks building with a unified build tree.  We now get:

arm-elf-ar  rc ./libgcc.a libgcc/./_udivsi3.o libgcc/./_divsi3.o
libgcc/./_umodsi3.o libgcc/./_modsi3.o libgcc/./_dvmd_tls.o [...]
libgcc/./gthr-gnat.o libgcc/./unwind-c.o
gnumake[3]: arm-elf-ar: Command not found
gnumake[3]: *** [libgcc.a] Error 127

The problem is in the top-level makefile with the extra level of
recursion that is now used.

AR_FOR_TARGET in this case is intended to expand to ..../binutils/ar,
but the macro to do this is in RECURSE_FLAGS_TO_PASS, and that's
expanded before we've built binutils, so instead of pointing to the
newly created ar it points to the non-existant arm-elf-ar.

Restarting the make after binutils has been built should work ok, but
that's not really the point.

R.

[PATCH] Fix unified tree breakage

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]


>Restarting the make after binutils has been built should work ok, but
>that's not really the point.
>  
>
The attached patch should fix it.

With the patch, we get this sequence for bootstrap:

        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        if [ ! -f stage_last ]; then \
          $(MAKE) $(RECURSE_FLAGS_TO_PASS) bootstrap; \
        fi
        @$(unstage)
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(TARGET_FLAGS_TO_PASS) all-host
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-target
        @$(stage)

for GCC, no bootstrap:

        @$(unstage)
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-prebootstrap
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-host
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-target
        @$(stage)

And for no GCC:

        @$(unstage)
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-host
        @r=`${PWD_COMMAND}`; export r; \
        s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
        $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-target
        @$(stage)

Richard, does it work for you?

Paolo

[-- Attachment #2: fix-breakage-for-rearnshaw.patch --]
[-- Type: text/plain, Size: 1266 bytes --]

2005-10-05  Paolo Bonzini  <bonzini@gnu.org>

	* Makefile.tpl (all) [gcc-no-bootstrap]: Make prebootstrap packages
	before other host packages.  Do not use stage_last to find the
	flags to pass to all-host.

Index: Makefile.tpl
===================================================================
RCS file: /cvs/gcc/gcc/Makefile.tpl,v
retrieving revision 1.143
diff -p -u -u -r1.143 Makefile.tpl
--- Makefile.tpl	5 Oct 2005 07:32:13 -0000	1.143
+++ Makefile.tpl	5 Oct 2005 13:56:11 -0000
@@ -709,13 +709,19 @@ all:
 	fi
 @endif gcc-bootstrap
 	@$(unstage)
+@if gcc-no-bootstrap
 	@r=`${PWD_COMMAND}`; export r; \
 	s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
-	if [ -f stage_last ]; then \
-	  $(MAKE) $(TARGET_FLAGS_TO_PASS) all-host; \
-	else \
-	  $(MAKE) $(RECURSE_FLAGS_TO_PASS) all-host; \
-	fi
+	$(MAKE) $(RECURSE_FLAGS_TO_PASS) all-prebootstrap
+@endif gcc-no-bootstrap
+	@r=`${PWD_COMMAND}`; export r; \
+	s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
+@if gcc-bootstrap
+	$(MAKE) $(TARGET_FLAGS_TO_PASS) all-host
+@endif gcc-bootstrap
+@if gcc-no-bootstrap
+	$(MAKE) $(RECURSE_FLAGS_TO_PASS) all-host
+@endif gcc-no-bootstrap
 	@r=`${PWD_COMMAND}`; export r; \
 	s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
 	$(MAKE) $(RECURSE_FLAGS_TO_PASS) all-target

Re: [PATCH] Fix unified tree breakage

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 32 bytes --]

This is the correct one.

Paolo

[-- Attachment #2: fix-breakage-for-rearnshaw.patch --]
[-- Type: text/plain, Size: 787 bytes --]

2005-10-05  Paolo Bonzini  <bonzini@gnu.org>

	* Makefile.tpl (all) [gcc-no-bootstrap]: Make prebootstrap packages
	before other host packages.

Index: Makefile.tpl
===================================================================
RCS file: /cvs/gcc/gcc/Makefile.tpl,v
retrieving revision 1.143
diff -p -u -u -r1.143 Makefile.tpl
--- Makefile.tpl	5 Oct 2005 07:32:13 -0000	1.143
+++ Makefile.tpl	5 Oct 2005 14:08:40 -0000
@@ -709,6 +709,11 @@ all:
 	fi
 @endif gcc-bootstrap
 	@$(unstage)
+@if gcc-no-bootstrap
+	@r=`${PWD_COMMAND}`; export r; \
+	s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
+	$(MAKE) $(RECURSE_FLAGS_TO_PASS) all-prebootstrap
+@endif gcc-no-bootstrap
 	@r=`${PWD_COMMAND}`; export r; \
 	s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
 	if [ -f stage_last ]; then \

Re: [PATCH] Fix unified tree breakage

[Richard Earnshaw]

On Wed, 2005-10-05 at 15:09, Paolo Bonzini wrote:
> This is the correct one.
> 
> Paolo
> 
> ______________________________________________________________________
> 2005-10-05  Paolo Bonzini  <bonzini@gnu.org>
> 
> 	* Makefile.tpl (all) [gcc-no-bootstrap]: Make prebootstrap packages
> 	before other host packages.

That appears to work, thanks.  However, why do you assume this would
only be needed when not doing a bootstrap?  OK, it's not common to want
to use a unified tree for that, but there's no reason why it shouldn't
work.

R.

Re: [PATCH] Fix unified tree breakage

[Paolo Bonzini]

>>2005-10-05  Paolo Bonzini  <bonzini@gnu.org>
>>
>>	* Makefile.tpl (all) [gcc-no-bootstrap]: Make prebootstrap packages
>>	before other host packages.
>>    
>>
>That appears to work, thanks.  However, why do you assume this would
>only be needed when not doing a bootstrap?  OK, it's not common to want
>to use a unified tree for that, but there's no reason why it shouldn't
>work.
>  
>
I'm sorry if this is confusing.  It will go away as soon as GCC 4.1 
branches.

In 4.2, we'll have *toplevel* bootstrap enabled by default.  This is 
what you saw indicated as "@if gcc-bootstrap", and it means that the 
toplevel orchestrates a 3-stage build of libiberty, binutils, and 
everything else up to gcc.  In this case it will just work because 
stage1 will use the system assembler/linker, and stage2/3 will use the 
previous stage of the in-tree assembler/linker: there is no need to do 
"all-prebootstrap", because the packages in "all-prebootstrap" are all 
bootstrapped (together with GCC).

BTW, with toplevel bootstrap, it could make much more sense to use a 
unified tree in a native configuration.

For 4.1, you need --enable-bootstrap to enable toplevel bootstrap.  But 
the default is still non-toplevel bootstrap, and in that case making 
"all-prebootstrap" is still handled with dependencies:

    $(GCC_STRAP_TARGETS): all-prebootstrap configure-gcc
             @r=`pwd`; export r; \
             etc.

Paolo

Re: [PATCH] Fix unified tree breakage

[Richard Earnshaw]

On Wed, 2005-10-05 at 16:23, Paolo Bonzini wrote:
> >>2005-10-05  Paolo Bonzini  <bonzini@gnu.org>
> >>
> >>	* Makefile.tpl (all) [gcc-no-bootstrap]: Make prebootstrap packages
> >>	before other host packages.
> >>    
> >>
> >That appears to work, thanks.  However, why do you assume this would
> >only be needed when not doing a bootstrap?  OK, it's not common to want
> >to use a unified tree for that, but there's no reason why it shouldn't
> >work.
> >  
> >
> I'm sorry if this is confusing.  It will go away as soon as GCC 4.1 
> branches.

:-)

> In 4.2, we'll have *toplevel* bootstrap enabled by default.  This is 
> what you saw indicated as "@if gcc-bootstrap", and it means that the 
> toplevel orchestrates a 3-stage build of libiberty, binutils, and 
> everything else up to gcc.  In this case it will just work because 
> stage1 will use the system assembler/linker, and stage2/3 will use the 
> previous stage of the in-tree assembler/linker: there is no need to do 
> "all-prebootstrap", because the packages in "all-prebootstrap" are all 
> bootstrapped (together with GCC).

Ah!  I get it now.  Sorry, I was confusing gcc-bootstrap with 'make
bootstrap'

This is OK.

R.

Re: [PATCH] Fix unified tree breakage

[Alexandre Oliva]

On Oct  5, 2005, Paolo Bonzini <paolo.bonzini@lu.unisi.ch> wrote:

> In 4.2, we'll have *toplevel* bootstrap enabled by default.  This is
> what you saw indicated as "@if gcc-bootstrap", and it means that the
> toplevel orchestrates a 3-stage build of libiberty, binutils, and
> everything else up to gcc.  In this case it will just work because
> stage1 will use the system assembler/linker, and stage2/3 will use the
> previous stage of the in-tree assembler/linker:

This is not good.  It means the libgcc build uses one pair of
assembler/linker different from the one that gets installed.  Each
stage must use its own assembler/linker to build the GCC libraries.

> there is no need to do "all-prebootstrap", because the packages in
> "all-prebootstrap" are all bootstrapped (together with GCC).

I think this has to be done for all stages, and every stage must use
its own assembler/linker pair to build any target libraries (libgcc
included), otherwise the point of bootstrapping is sort of missed.

> BTW, with toplevel bootstrap, it could make much more sense to use a
> unified tree in a native configuration.

Too bad the unified tree won't be as trivial to create as checking out
uberbaum as soon as GCC moves to subversion :-(

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [PATCH] Fix unified tree breakage

[Alexandre Oliva]

On Oct  5, 2005, Alexandre Oliva <aoliva@redhat.com> wrote:

> On Oct  5, 2005, Paolo Bonzini <paolo.bonzini@lu.unisi.ch> wrote:
>> In 4.2, we'll have *toplevel* bootstrap enabled by default.  This is
>> what you saw indicated as "@if gcc-bootstrap", and it means that the
>> toplevel orchestrates a 3-stage build of libiberty, binutils, and
>> everything else up to gcc.  In this case it will just work because
>> stage1 will use the system assembler/linker, and stage2/3 will use the
>> previous stage of the in-tree assembler/linker:

> This is not good.  It means the libgcc build uses one pair of
> assembler/linker different from the one that gets installed.  Each
> stage must use its own assembler/linker to build the GCC libraries.

And, just to clarify why it's so important to get this right, when
bootstrapping GCC, starting without GNU binutils, but configuring GCC
to use binutils, you'd likely get assembler/linker failures while
building libgcc for stage1.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [PATCH] Fix unified tree breakage

[Alexandre Oliva]

On Oct  6, 2005, Paolo Bonzini <bonzini@gnu.org> wrote:

> The stage2 and stage3 binaries should be identical (and even the
> stage1 and stage2 binaries should behave the same).  Thus, using the
> previous stage of the in-tree assembler/linker could even be correct,
> but it is not what it is done.

Except for the stage1 case, that's exactly where relying on the
previous stage's assembler/linker could cause things to fall apart.

Anyhow, good to know it's not broken.

That said, the other day I ran into another recent breakage in a
unified tree: gas/as-new is *hard-linked* to gcc/as, as opposed to
being symlinked.  This means that, if I modify the gas sources and
rebuild them, gcc will still use the old assembler.  Oops.

Ditto for linker and binutils.

If hard-linking is a must for whatever reason (such as simplifying
stage pushing and popping), then we must add dependencies to make sure
these binaries remain up to date with respect to what they were
hard-linked to.

> run-tool --target='$$r/$(HOST_SUBDIR)/gas/as-new' \
>           --target='$$r/$(HOST_SUBDIR)/gcc/as' \
>           --host='$(AS)' \
>           '$(CONFIGURED_AS_FOR_TARGET)'"

> finds an in-tree target assembler, falling back to the system host
> assembler in a native configuration (target = host) or then to the
> $(CONFIGURED_AS_FOR_TARGET).

Neat.

> The first parameter specifies the desired triplet for the tool, so
> --host is only considered if the host and target triplets are
> identical.

I think you need some means to avoid running the wrong binary in
Canadian crosses and cross-back tools.  E.g., consider --build=x
--host=y --target=z.  How can run-tool --target=whatever distinguish
between gas/as-new, that's built for the host y and thus won't run on
the build machine x, from gcc/as, presumably copied from a pre-build
--host=x --target=z toolchain, specified in
$(CONFIGURED_AS_FOR_TARGET), or, as a potential nice improvement,
built in the build-x directory.

And then there's --build=x --host=y --target=x, for which you *could*
use the system assembler.

Just something to think about :-)

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: [PATCH] Fix unified tree breakage

[Bernd Jendrissek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Oct 07, 2005 at 11:08:38AM -0300, Alexandre Oliva wrote:
> I think you need some means to avoid running the wrong binary in
> Canadian crosses and cross-back tools.  E.g., consider --build=x
> --host=y --target=z.  How can run-tool --target=whatever distinguish
> between gas/as-new, that's built for the host y and thus won't run on
> the build machine x, from gcc/as, presumably copied from a pre-build
> --host=x --target=z toolchain, specified in
> $(CONFIGURED_AS_FOR_TARGET), or, as a potential nice improvement,
> built in the build-x directory.

I ran into something like this the other day, while building GCC as
- --build=i686-pc-linux-gnu --host=i386-mingw32 --target=arm-unknown-elf.
The resulting GCC wants to run just "as"; if I specify
- --with-as=c:/gnutools/cross/bin/arm-unknown-elf.exe (which is what I
really wanted), the configure script got confused about what exactly
that --with-as (and same for ld) is: a tool running on the build
machine, or the host machine.  At runtime GCC wants to use it as
DEFAULT_ASSEMBLER to rewrite in the place of "as" in invoke_as, so from
that perspective it is a host tool.  OTOH the configure script expects
that specified assembler to exist and be test -x, which is just
confused.

Eventually I just got my colleague to do gcc -dumpspecs >specs, edit
invoke_as, and then add -specs=specs to $(CC) in the makefile.  Does GCC
really need --with-host-as=/nonexistent/on/build/as, or do I need to
RTFM on --with-sysroot?

For the archives...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Please fetch my new key 804177F8 from hkp://wwwkeys.eu.pgp.net/
Comment: Key D7CBA633 will soon expire, check its signature of 804177F8 now!

iD8DBQFDRtGawyMv24BBd/gRAoP5AJ0QN3cWbH7mqNRQA4nIRTJteqfBogCeO/9s
ka58VHMt8ZkshNcH+l2aaPY=
=fCj7
-----END PGP SIGNATURE-----

Re: [PATCH] Fix unified tree breakage

[Alexandre Oliva]

On Oct  5, 2005, Paolo Bonzini <paolo.bonzini@lu.unisi.ch> wrote:

> 2005-10-05  Paolo Bonzini  <bonzini@gnu.org>

> 	* Makefile.tpl (all) [gcc-no-bootstrap]: Make prebootstrap packages
> 	before other host packages.

Ok, thanks, but please consider the points in the e-mail I've just
posted downthread and try to address them.  If this means just taking
out the conditionals, it's pre-approved.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}