Serialization dependencies muck up configure-on-demand

Serialization dependencies muck up configure-on-demand

[Daniel Jacobowitz]

The serializiation dependencies are causing all-gdb to have to configure
things like the assembler.  Consider --target=alpha-elf.  This isn't
supported in gas but the top level doesn't know that (a separate problem,
for another day, and not the real cause of trouble here).  The top level
emits serialization dependencies like:

# Serialization dependencies.  Host configures don't work well in parallel to
# each other, due to contention over config.cache.  Target configures and 
# build configures are similar.

mmalloc/Makefile: intl/Makefile
libiberty/Makefile: mmalloc/Makefile
opcodes/Makefile: libiberty/Makefile
bfd/Makefile: opcodes/Makefile
readline/Makefile: bfd/Makefile
tcl/Makefile: readline/Makefile
tk/Makefile: tcl/Makefile
itcl/Makefile: tk/Makefile
tix/Makefile: itcl/Makefile
libgui/Makefile: tix/Makefile
binutils/Makefile: libgui/Makefile
gas/Makefile: binutils/Makefile
ld/Makefile: gas/Makefile
sid/Makefile: ld/Makefile
sim/Makefile: sid/Makefile
gdb/Makefile: sim/Makefile
etc/Makefile: gdb/Makefile
expect/Makefile: etc/Makefile
dejagnu/Makefile: expect/Makefile
utils/Makefile: dejagnu/Makefile
$(TARGET_SUBDIR)/rda/Makefile: $(TARGET_SUBDIR)/libiberty/Makefile


This means that to build GDB in this tree requires configuring gas, and I've
lost all that I hoped to gain from doing the configury in the makefile.

Do we need the serialization dependencies?  While we support all sorts of
versions of Make, do we actually support parallel make on anything that
doesn't recognize .NOTPARALLEL?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Serialization dependencies muck up configure-on-demand

[DJ Delorie]

> Do we need the serialization dependencies?

We need to make sure that no two configures run at the same time.  If
you can figure out a better way to do that, please do.

> While we support all sorts of versions of Make, do we actually
> support parallel make on anything that doesn't recognize
> .NOTPARALLEL?

We want to run the toplevel makefile in parallel, because we want to
do the builds and tests in parallel.  So, we want to get rid of
.NOTPARALLEL, not depend on it.

Re: Serialization dependencies muck up configure-on-demand

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 01:33:04PM -0500, DJ Delorie wrote:
> 
> > Do we need the serialization dependencies?
> 
> We need to make sure that no two configures run at the same time.  If
> you can figure out a better way to do that, please do.
> 
> > While we support all sorts of versions of Make, do we actually
> > support parallel make on anything that doesn't recognize
> > .NOTPARALLEL?
> 
> We want to run the toplevel makefile in parallel, because we want to
> do the builds and tests in parallel.  So, we want to get rid of
> .NOTPARALLEL, not depend on it.

Well, I'm not sure how useful of a goal that is; I suppose it would
speed up re-making when most directories don't need to be updated or
updated substantially.  I don't see a better way without being truly
gross about it... no, even having a combinatorial number of
serialization dependencies won't work since they have to be on real
files.

However, GNU make has the answer.  We'd probably have to autoconf for
this feature, it's quite new:
    * A new feature exists: order-only prerequisites.  These prerequisites
      affect the order in which targets are built, but they do not impact
      the rebuild/no-rebuild decision of their dependents.  That is to say,
      they allow you to require target B be built before target A, without
      requiring that target A will always be rebuilt if target B is updated.
      Patch for this feature provided by Greg McGary <greg@mcgary.org>.

The syntax is:
TARGETS : NORMAL-PREREQUISITES | ORDER-ONLY-PREREQUISITES

So if that works in a test at autoconf time, we could use it.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Serialization dependencies muck up configure-on-demand

[DJ Delorie]

> > We want to run the toplevel makefile in parallel, because we want to
> > do the builds and tests in parallel.  So, we want to get rid of
> > .NOTPARALLEL, not depend on it.
> 
> Well, I'm not sure how useful of a goal that is;

It lets you start building libiberty while you're still configuring
gcc, for example.  Or you can reconfigure a single subdirectory by
deleting its Makefile and running "make".  Or you can test gcc, gdb,
and binutils all at once with "make -j3 check".  You can also add or
remove source directories to/from your build, or edit the configury,
and reconfigure in a fraction of the time.

> I don't see a better way without being truly gross about it... no,
> even having a combinatorial number of serialization dependencies
> won't work since they have to be on real files.

Try the attached patch.  We'll need to explicitly add back in true
configure dependencies (like bfd:libiberty) based on the real files.

> However, GNU make has the answer.

We cannot rely on using GNU make features.

Index: configure.in
===================================================================
RCS file: /cvs/gcc/gcc/configure.in,v
retrieving revision 1.198
diff -p -3 -r1.198 configure.in
*** configure.in	23 Dec 2002 16:54:15 -0000	1.198
--- configure.in	23 Dec 2002 20:49:18 -0000
*************** olditem=
*** 1458,1464 ****
  for item in ${build_configdirs} ; do
    case ${olditem} in
      "") ;;
!     *) echo "\$(BUILD_SUBDIR)/${item}/Makefile: \$(BUILD_SUBDIR)/${olditem}/Makefile" >> serdep.tmp ;;
    esac
    olditem=${item}
  done
--- 1458,1464 ----
  for item in ${build_configdirs} ; do
    case ${olditem} in
      "") ;;
!     *) echo "configure-build-${item}: configure-build-${olditem}" >> serdep.tmp ;;
    esac
    olditem=${item}
  done
*************** olditem=
*** 1466,1472 ****
  for item in ${configdirs} ; do
    case ${olditem} in
      "") ;;
!     *) echo "${item}/Makefile: ${olditem}/Makefile" >> serdep.tmp ;;
    esac
    olditem=${item}
  done
--- 1466,1472 ----
  for item in ${configdirs} ; do
    case ${olditem} in
      "") ;;
!     *) echo "configure-${item}: configure-${olditem}" >> serdep.tmp ;;
    esac
    olditem=${item}
  done
*************** olditem=
*** 1474,1480 ****
  for item in ${target_configdirs} ; do
    case ${olditem} in
      "") ;;
!     *) echo "\$(TARGET_SUBDIR)/${item}/Makefile: \$(TARGET_SUBDIR)/${olditem}/Makefile" >> serdep.tmp ;;
    esac
    olditem=${item}
  done
--- 1474,1480 ----
  for item in ${target_configdirs} ; do
    case ${olditem} in
      "") ;;
!     *) echo "configure-target-${item}: configure-target-${olditem}" >> serdep.tmp ;;
    esac
    olditem=${item}
  done

Re: Serialization dependencies muck up configure-on-demand

[Alexandre Oliva]

On Dec 23, 2002, DJ Delorie <dj@redhat.com> wrote:

>> Do we need the serialization dependencies?

> We need to make sure that no two configures run at the same time.  If
> you can figure out a better way to do that, please do.

We can use shell locking (*) in the top level to make sure no two
configures run at the same time, if we really need that.  Serialized
dependencies are exactly the wrong way to solve the problem, IMO.  I
hadn't realized they had been implemented like that, otherwise I'd
have objected.


(*) attempt to create a directory say configure.lock and sleep until
creation succeeds, then run sub-configure, then rmdir it.  This can
fail in the rare advent of the build directory being on an unreliable
NFS server, such that the reply of mkdir fails to get to the client,
that retries the operation and then gets a failure because the
directory now exists.  We can probably live with that.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: Serialization dependencies muck up configure-on-demand

[DJ Delorie]

> We can use shell locking (*) in the top level to make sure no two
> configures run at the same time, if we really need that.

I really don't want to create a spinlock system just for a build.
There are too many weird ways that can fail.

>  Serialized dependencies are exactly the wrong way to solve the
> problem, IMO.  I hadn't realized they had been implemented like
> that, otherwise I'd have objected.

We've done it before.  We're using the "can't do this until you do
that" half of the dependency logic, not the "I need this built for my
build" half.

Re: Serialization dependencies muck up configure-on-demand

[Alexandre Oliva]

On Dec 23, 2002, DJ Delorie <dj@redhat.com> wrote:

>> We can use shell locking (*) in the top level to make sure no two
>> configures run at the same time, if we really need that.

> I really don't want to create a spinlock system just for a build.
> There are too many weird ways that can fail.

If you print a message saying `waiting for [lockdir] to go away...'
before sleeping, the user will probably be able to figure it out if
something goes wrong.  And things will only go wrong if the user
actually aborts some running configure, which has always been a bad
idea anyway (especially when started from ./config.status --recheck).

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: Serialization dependencies muck up configure-on-demand

[DJ Delorie]

> If you print a message saying `waiting for [lockdir] to go away...'
> before sleeping, the user will probably be able to figure it out if
> something goes wrong. 

Unless your cron job mysteriously runs forever...

Re: Serialization dependencies muck up configure-on-demand

[Nathanael Nerode]

Daniel mentioned:
>However, GNU make has the answer.  We'd probably have to autoconf for
>this feature, it's quite new:
>    * A new feature exists: order-only prerequisites.  These 
>prerequisites
>      affect the order in which targets are built, but they do not 
>impact
>      the rebuild/no-rebuild decision of their dependents.  That is to 
>say,
>      they allow you to require target B be built before target A, 
>without
>      requiring that target A will always be rebuilt if target B is 
>updated.
>      Patch for this feature provided by Greg McGary <greg@mcgary.org>.
>
>The syntax is:
>TARGETS : NORMAL-PREREQUISITES | ORDER-ONLY-PREREQUISITES
>
>So if that works in a test at autoconf time, we could use it.

Quite right; if configure detects that this is available, we should use 
this in the Makefile rather than the existing scheme.

Unfortuntately, I'm not feeling up to coding this for a while. :-(

Would there be interest in temporarily disabling the serialization 
dependencies (since .NOTPARALLEL is still there) until we can get a 
better solution for all this?  I'm just thinking of 
"if (0)"ing them out in 'configure.in' for the time being.

--Nathanael

Re: Serialization dependencies muck up configure-on-demand

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 03:37:51PM -0500, Nathanael Nerode wrote:
> Daniel mentioned:
> >However, GNU make has the answer.  We'd probably have to autoconf for
> >this feature, it's quite new:
> >    * A new feature exists: order-only prerequisites.  These 
> >prerequisites
> >      affect the order in which targets are built, but they do not 
> >impact
> >      the rebuild/no-rebuild decision of their dependents.  That is to 
> >say,
> >      they allow you to require target B be built before target A, 
> >without
> >      requiring that target A will always be rebuilt if target B is 
> >updated.
> >      Patch for this feature provided by Greg McGary <greg@mcgary.org>.
> >
> >The syntax is:
> >TARGETS : NORMAL-PREREQUISITES | ORDER-ONLY-PREREQUISITES
> >
> >So if that works in a test at autoconf time, we could use it.
> 
> Quite right; if configure detects that this is available, we should use 
> this in the Makefile rather than the existing scheme.
> 
> Unfortuntately, I'm not feeling up to coding this for a while. :-(

Oh, I can do it; in fact the autoconfery is trivial.  I need to go
figure out why it isn't working as I expect though.  Right now saying
"a : | b" appears to mean that b out-of-date does not trigger remaking
a, but "make a" still causes b to be remade.  In other words it's still
a dependency but not used for rebuild calculation.  Serialization wants
something different so this may not be our answer.

Make folks, am I reading the documentation right?  Do we need another
feature to express this concept?  Background: we run a collection of
configure scripts from the makefile; due to the current handling of
config.cache we only want one of them to run at a time, without having
to .NOTPARALLEL the entire makefile; they are not actually as strictly
dependent as that implies.

> Would there be interest in temporarily disabling the serialization 
> dependencies (since .NOTPARALLEL is still there) until we can get a 
> better solution for all this?  I'm just thinking of 
> "if (0)"ing them out in 'configure.in' for the time being.

(Well, I like it.)

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Serialization dependencies muck up configure-on-demand

[Paul D. Smith]

%% Daniel Jacobowitz <drow@mvista.com> writes:

  dj> Oh, I can do it; in fact the autoconfery is trivial.  I need to go
  dj> figure out why it isn't working as I expect though.  Right now
  dj> saying "a : | b" appears to mean that b out-of-date does not
  dj> trigger remaking a, but "make a" still causes b to be remade.  In
  dj> other words it's still a dependency but not used for rebuild
  dj> calculation.

Correct.  If b is a prerequisite, order-only or not, then it will be
remade.  The only difference in an order-only prerequisite is that
it is ignored when deciding whether or not a needs to be rebuilt.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Serialization dependencies muck up configure-on-demand

[Nathanael Nerode]

Daniel said:
>Well, I'm not sure how useful of a goal that is; I suppose it would
>speed up re-making when most directories don't need to be updated or
>updated substantially.  I don't see a better way without being truly

Specifically, when nothing needs to be reconfigured, but lots of things 
do need to be rebuilt, this is very useful. :-)  This is quite a common 
situation.

--Nathanael

Re: Serialization dependencies muck up configure-on-demand

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 03:39:32PM -0500, Nathanael Nerode wrote:
> Daniel said:
> >Well, I'm not sure how useful of a goal that is; I suppose it would
> >speed up re-making when most directories don't need to be updated or
> >updated substantially.  I don't see a better way without being truly
> 
> Specifically, when nothing needs to be reconfigured, but lots of things 
> do need to be rebuilt, this is very useful. :-)  This is quite a common 
> situation.

Nah; if lots of things need to be rebuilt we can just do one thing at a
time, since each directory would keep all the available make jobs busy. 
The improvement is pretty marginal.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer