Re: (toplevel patch) Fix multilib.out dependencies and related problems

Re: (toplevel patch) Fix multilib.out dependencies and related problems

[Nathanael Nerode]

>> Uh, yeah.  Make isn't designed to realize that you can change a file
>> and not change the generated file depending on it; this is not an
>> unreasonable assumption in most cases.  To put it another way, I
>> don't care very much.
>
>Except that gcc's makefile does this all over the place, and it works
>just fine.  I was told about this trick in school - in 1986, on a
>Gould mainframe.  It works.
>
>multilib.out : gcc/xgcc
>        gcc/xgcc --print-multilibs > multilib.tmp
>        $(srcdir)/move-if-change multilib.tmp multilib.out
Right.  I'm happy to do this.

>> Of course it should.  But it can't, it actually can't, because then
>> we have a real target depending on a phony target, and every target
>> subdir gets reconfigured every single time.  Sorry.
>
>Right, except for the move-if-change trick, which short circuits it if
>there is no real change.  That's the whole purpose of the
>move-if-change script.

Hmm.  So the trick is that *multilib.out* gets hit every single time 
because it depends on all-gcc, but foo/Makefile *doesn't*.

In other words, when multilib.out is considered 'out of date' and needs 
to be rebuilt, its rule is run.  But if that rule doesn't change the 
datestamp on multilib.out, Make decides that the things depending on 
multilib.out, such as foo/Makefile, *don't* need to be rebuilt.

How sad for Make; it can't assume that because it rebuilt something, 
that that something changed. :-)  I didn't know that POSIX makes were 
required to deal with that sort of seeming paradox...

OK, I get it.  Revised patch coming in a minute or two.

--Nathanael

Re: (toplevel patch) Fix multilib.out dependencies and related problems

[DJ Delorie]

> In other words, when multilib.out is considered 'out of date' and
> needs to be rebuilt, its rule is run.  But if that rule doesn't
> change the datestamp on multilib.out, Make decides that the things
> depending on multilib.out, such as foo/Makefile, *don't* need to be
> rebuilt.

Yup.  See gcc's Makefile for examples of how to do this.
They do it like this:

multilib.out : multilib.ts ; @true
multilib.ts : [dependencies]
	foo > multilib.tmp
	$(SHELL) $(srcdir)/move-if-change multilib.tmp multilib.out
	echo timestamp > multilib.ts

The other multilib.out targets just depend on multilib.out:

$target/[module]/multlib.out : multilib.out
	cp multilib.out $target/[module]/multilib.out

Re: (toplevel patch) Fix multilib.out dependencies and related problems

[Zack Weinberg]

Nathanael Nerode <neroden@twcny.rr.com> writes:

>>> Uh, yeah.  Make isn't designed to realize that you can change a file
>>> and not change the generated file depending on it; this is not an
>>> unreasonable assumption in most cases.  To put it another way, I
>>> don't care very much.
>>
>>Except that gcc's makefile does this all over the place, and it works
>>just fine.  I was told about this trick in school - in 1986, on a
>>Gould mainframe.  It works.
>>
>>multilib.out : gcc/xgcc
>>        gcc/xgcc --print-multilibs > multilib.tmp
>>        $(srcdir)/move-if-change multilib.tmp multilib.out
> Right.  I'm happy to do this.

The Right Thing is ever so slightly more complicated:

STAMP = echo timestamp >

multilib.out : s-multilib ; @true
s-multilib : gcc/xgcc
           gcc/xgcc --print-multilibs > multilib.tmp
           $(srcdir)/move-if-change multilib.tmp multilib.out
           $(STAMP) s-multilib

[plus, make sure to delete s-multilib as well as multilib.out on 
'make clean'.]

What this does for you is avoid running the actual rule for
multilib.out if gcc/xgcc hasn't changed.  Instead, it only runs the
rule for s-multilib, which is a no-op.

zw

Re: (toplevel patch) Fix multilib.out dependencies and related problems

[Nathanael Nerode]

I said:
> Removed xgcc and ran 'make'; gcc rebuilt it, and target libs were 
> reconfigured and built. 

DJ quoth:
>That shouldn't happen.  Use move-if-change to update one common
>multilib.out, so that if the list of multilibs doesn't change,
>*nothing* gets reconfigured or rebuilt.  We want to trigger a *check*
>when gcc changes, but only actually *reconfigure* if needed.
Uh, yeah.  Make isn't designed to realize that you can change a file and 
not change the generated file depending on it; this is not an 
unreasonable assumption in most cases.  To put it another way, I don't 
care very much.

>I'm wondering if the common multilib.out target should depend on the
>all-gcc phony target; we kinda want it to run every time we check the
Of course it should.  But it can't, it actually can't, because then we 
have a real target depending on a phony target, and every target subdir 
gets reconfigured every single time.  Sorry.  

You could try to do this with a phony multilib.out target.  Then you'd 
need to make configure-target-foo depend on it instead of 
(TARGET_SUBDIR)/foo/Makefile.  And you'd find that target subdirs would 
never get reconfigured.  Sorry.

You could patch that problem up by putting the code for foo/Makefile 
back into configure-target-foo, and hard-coding yet more things Make 
should do, such as determining whether foo/Makefile is out of date.
Maybe this is the preferred method, but I will have nothing to do with 
it, since I'm certain it's the wrong direction to go.

>gcc subdir, because we don't know if something else besides xgcc
>affects the specs (which affects multilib.out).
Eeech.  We should actually know what affects the specs, you know. :-/

>  We can get away with
>this if we use move-if-change.
Kinda, sorta.

Here's a scheme, which might do the trick.

(First note that a common multilib.out is a separate issue and 
can and should be done however we fix these other things.)

Have the common multilib.out depend on nothing.
Have the build process for gcc (I hope there's nothing else which 
can affect multilib.out...) do the testing and move-if-changing of 
multilib.out.  Effectively, this is a preinstall; building all-gcc 
should build multilib.out directly into the location where the target 
subdirs look for it, and the rules for regenerating it will live in the 
gcc Makefile.

This should mean that a change in gcc, provided the build process is 
used, will change multilib.out.

Happy with this?

If not, here are some other ideas, thrown out at least partly in 
frustration:
* Whatever determines the contents of multilib.out should just get its 
own damn rule at the top level, rather than messing around with building 
the whole of gcc to determine that it's ".;".

* Force every target subdir which actually cares about this to produce 
its own copy of multilib.out and to test whether it's out of date on its 
own, so that it's out of the hair of the top level.

* Don't support toplevel Make at all.

* Convert the configuration and Makefiles for all subdirectories to
fragmentary Makefile fragments, which are plugged into the top level.  
In other words, only support the top level.

* Make all-gcc a real target, similarly for all the others, and require 
the subdirs to create the corresponding files name 'all'.  This is 
actually a pretty good idea, although it's quite non-standard; it 
amounts to the total elimination of all phony targets.

* Use existing real targets: in place of all-gcc, use gcc/xgcc gcc/cc1 
etc.  This of course requires massive and ugly changes to the top level.

--
Incidentally, I now know the one feature I want added to all versions of 
Make, and it's not one which exists in any version of make I've seen.  
Plus, it's very very simple.

For any target, in addition to being able to specify a command and 
dependencies, you should be able to specify an external program to run 
in a shell, which gives exit status 0 if the target is up to date and 
exit status 1 if the target needs to be rebuilt.  (For total 
generality, it might require a shell variable to be passed in indicating 
whether the dependencies were up to date.)  If it returned 1, the 
dependencies would still have to be examined to see if they needed to be 
rebuilt first, however.

This would deal with virtually all "problems" with Make in a generic 
way.  For instance, we simply annotate multilib.out with its test and a 
dependency on all-gcc.  All-gcc is still considered updated every 
time, but this doesn't force multilib.out to be.

This actually solves every real issue I have ever seen anyone have with 
Make ever, and I am surprised that it hasn't been implemented in any 
version I've met.  I'm not sure which make variants to push this idea 
at...

--Nathanael

Re: (toplevel patch) Fix multilib.out dependencies and related problems

[DJ Delorie]

> Uh, yeah.  Make isn't designed to realize that you can change a file
> and not change the generated file depending on it; this is not an
> unreasonable assumption in most cases.  To put it another way, I
> don't care very much.

Except that gcc's makefile does this all over the place, and it works
just fine.  I was told about this trick in school - in 1986, on a
Gould mainframe.  It works.

multilib.out : gcc/xgcc
	gcc/xgcc --print-multilibs > multilib.tmp
	$(srcdir)/move-if-change multilib.tmp multilib.out

> Of course it should.  But it can't, it actually can't, because then
> we have a real target depending on a phony target, and every target
> subdir gets reconfigured every single time.  Sorry.

Right, except for the move-if-change trick, which short circuits it if
there is no real change.  That's the whole purpose of the
move-if-change script.

> >gcc subdir, because we don't know if something else besides xgcc
> >affects the specs (which affects multilib.out).
> Eeech.  We should actually know what affects the specs, you know. :-/

We should, but gcc may read from an external specs file, and we don't
always know (easily) where that comes from.  Especially if the gcc
we're using isn't the one we just built.

> Have the build process for gcc (I hope there's nothing else which
> can affect multilib.out...) do the testing and move-if-changing of
> multilib.out.

Unless we're not building gcc.

> * Whatever determines the contents of multilib.out should just get its 
> own damn rule at the top level, rather than messing around with building 
> the whole of gcc to determine that it's ".;".

gcc determines the multilib list, and it's a very complicated process
to do so.

> * Force every target subdir which actually cares about this to produce 
> its own copy of multilib.out and to test whether it's out of date on its 
> own, so that it's out of the hair of the top level.

This won't work because the targets are configured once for each
multilib.  By the time the target subdir has control, it's too late.

(toplevel patch) Fix multilib.out dependencies and related problems.

[Nathanael Nerode]

I think this finally fixes the various problems related to phony depdencies,
real dependencies, and multilib.out.  DJ, please review. :-)

Bootstrapped on i686-pc-linux-gnu.  Ran 'make'; no reconfigurations.  
Removed xgcc and ran 'make'; gcc rebuilt it, and target libs were 
reconfigured and built. 
Ran 'make'; no reconfigurations.  

Configured and built with a preset CC_FOR_TARGET; no errors from missing
dependencies.

So it all works.

Woo hoo.

I know I have some leftover BASE_CC_FOR_TARGET junk, but I'll get that later.

	* configure.in: Issue 'maybe_compiler_driver' 
	(rather than BASE_CC_FOR_TARGET) as the dependence for multilib.out
	* Makefile.tpl: Use it; add regen rule for gcc/xgcc.
	* Makefile.in: Regenerate.

Index: Makefile.tpl
===================================================================
RCS file: /cvs/gcc/gcc/Makefile.tpl,v
retrieving revision 1.21
diff -u -r1.21 Makefile.tpl
--- Makefile.tpl	19 Dec 2002 20:23:50 -0000	1.21
+++ Makefile.tpl	19 Dec 2002 21:16:18 -0000
@@ -784,7 +784,6 @@
 # built are.
 TAGS: do-TAGS
 
-
 # --------------------------------------
 # Modules which run on the build machine
 # --------------------------------------
@@ -970,7 +969,7 @@
 
 # Don't manually override CC_FOR_TARGET at make time; get it set right
 # at configure time.  Otherwise multilibs may be wrong.
-$(TARGET_SUBDIR)/[+module+]/multilib.out: maybe-all-gcc
+$(TARGET_SUBDIR)/[+module+]/multilib.out: @maybe_compiler_driver@
 	@[ -d $(TARGET_SUBDIR)/[+module+] ] || mkdir $(TARGET_SUBDIR)/[+module+];\
 	r=`${PWD}`; export r; \
 	echo "Configuring multilibs for [+module+]"; \
@@ -1264,6 +1263,14 @@
 	  rm -rf gcc/include; \
 	  mv gcc/tmp-include gcc/include 2>/dev/null; \
 	else true; fi
+
+# Special multilib.out dependence support.
+# When in-tree gcc is being used as CC_FOR_TARGET, we make
+# multilib.out depend on gcc/xgcc.  So we need a rule for what to
+# do if gcc/xgcc is in fact missing.  (Normally it would be either
+# up to date or newer than multilib.out, not missing.)
+gcc/xgcc:
+	$(MAKE) all-gcc
 
 # --------------------------------------
 # Dependencies between different modules
Index: configure.in
===================================================================
RCS file: /cvs/gcc/gcc/configure.in,v
retrieving revision 1.196
diff -u -r1.196 configure.in
--- configure.in	19 Dec 2002 20:23:50 -0000	1.196
+++ configure.in	19 Dec 2002 21:16:21 -0000
@@ -1698,11 +1698,23 @@
 qqCXX_FOR_TARGET=`echo "$qCXX_FOR_TARGET" | sed -e 's,[$][$],$$$$,g'`
 qqRAW_CXX_FOR_TARGET=`echo "$qRAW_CXX_FOR_TARGET" | sed -e 's,[$][$],$$$$,g'`
 
-# BASE_CC_FOR_TARGET is the actual program file of CC_FOR_TARGET,
-# for use in dependencies.  Replace $$r with a relative
-# path if it's present.
-set dummy $CC_FOR_TARGET; cc_for_target_first_word=$2
-BASE_CC_FOR_TARGET=`echo "$cc_for_target_first_word" | sed -e 's,[$][$]r,.,'`
+# maybe_compiler_driver is the program file on which multilib.out depends.
+
+# If we're not using the configured GCC, for now we just punt on the
+# dependency.  If you change the external compiler you're using, you
+# should know enough to reconfigure from scratch.
+maybe_compiler_driver=
+
+# On the other hand, if you are using the configured GCC...
+case "${configdirs}" in
+  *gcc*) # We're configuring GCC.
+    case "${CC_FOR_TARGET}" in
+      *xgcc*)
+        # And we are (probably) using the configured GCC.
+        maybe_compiler_driver=gcc/xgcc
+        ;;
+    esac ;;
+esac
 
 sedtemp=sed.$$
 cat >$sedtemp <<EOF
@@ -1722,7 +1734,7 @@
 s%@target_configargs@%${targargs}%
 s%@FLAGS_FOR_TARGET@%${FLAGS_FOR_TARGET}%
 s%@CC_FOR_TARGET@%\$(STAGE_CC_WRAPPER) ${CC_FOR_TARGET}%
-s%@BASE_CC_FOR_TARGET@%${BASE_CC_FOR_TARGET}%
+s%@maybe_compiler_driver@%${maybe_compiler_driver}%
 s%@GCJ_FOR_TARGET@%\$(STAGE_CC_WRAPPER) ${GCJ_FOR_TARGET}%
 s%@CXX_FOR_TARGET@%\$(STAGE_CC_WRAPPER) ${qCXX_FOR_TARGET}%
 s%@RAW_CXX_FOR_TARGET@%\$(STAGE_CC_WRAPPER) ${qRAW_CXX_FOR_TARGET}%

Re: (toplevel patch) Fix multilib.out dependencies and related problems.

[DJ Delorie]

> Removed xgcc and ran 'make'; gcc rebuilt it, and target libs were 
> reconfigured and built. 

That shouldn't happen.  Use move-if-change to update one common
multilib.out, so that if the list of multilibs doesn't change,
*nothing* gets reconfigured or rebuilt.  We want to trigger a *check*
when gcc changes, but only actually *reconfigure* if needed.

I'm wondering if the common multilib.out target should depend on the
all-gcc phony target; we kinda want it to run every time we check the
gcc subdir, because we don't know if something else besides xgcc
affects the specs (which affects multilib.out).  We can get away with
this if we use move-if-change.

Re: (toplevel patch) Fix multilib.out dependencies and related problems.

[Alexandre Oliva]

On Dec 19, 2002, DJ Delorie <dj@redhat.com> wrote:

> I'm wondering if the common multilib.out target should depend on the
> all-gcc phony target; we kinda want it to run every time we check the
> gcc subdir, because we don't know if something else besides xgcc
> affects the specs (which affects multilib.out).  We can get away with
> this if we use move-if-change.

Yup, this sounds like the best approach.

Having an explicit rule to build gcc/xgcc would wreak havoc in
parallel builds, should we attempt to build both all-gcc and gcc/xgcc
in parallel.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer