using distcc to speed up gcc bootstraps

using distcc to speed up gcc bootstraps

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

A while ago, we've been talking about using distcc to speed up gcc
bootstraps.  I've made some progress today towards this goal, but
there's still some way to go to get there.

I've written a wrapper for distcc that goes over the command line
arguments looking for relative pathnames and, if it finds any, it
turns them into network-neutral full pathnames, using the netpwd
script below, and then runs the real distcc to do the work.

Unfortunately, this solution is not complete yet, since libstdc++-v3
fails to configure with this patch.  I'm yet to investigate why, but I
wanted heads up that I've made some progress.

I enclose below the patch for GCC I'm using, that, if you set
STAGE_CC_WRAPPER=distcc, will let you take advantage of distcc for the
several stages of gcc.  I'd like to put this patch in as soon as I get
it into a functional state, but I'm open to suggestions for the name
of the variable.

I also enclose the distcc wrapper script and the netpwd script it uses
to turn pwd into a network-neutral full pathname.

Besides the problem mentioned above, I've run into two other problems:

- the bootstrap check fails, even though I have every reason to
  believe it bootstrapped correctly.  I believe the cause of the
  failure is the problem documented in distcc's manual: that gcc adds
  the preprocessed filename to the debugging info emitted for each
  file.  I believe it shouldn't do this, but rather just use the name
  in the first #line directive in the preprocessed file.  I'll look
  into trying to fix this problem, if I confirm it is indeed the
  reason for the failure.

- the load on my build machine skyrockets while building the ada
  compiler and the Java library.  It would be nice if distcc could
  throttle other instances thereof when it realizes it can't send
  builds to remote build servers.  A lock on localhost, to be used in
  such situations, would be ideal IMO.  Is this an acceptable change
  for distcc?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: stage-cc-wrapper.patch --]
[-- Type: text/x-patch, Size: 4028 bytes --]

Index: Makefile.in
===================================================================
RCS file: /cvs/gcc/egcs/Makefile.in,v
retrieving revision 1.110
diff -u -p -r1.110 Makefile.in
--- Makefile.in 8 Jul 2002 21:40:41 -0000 1.110
+++ Makefile.in 14 Aug 2002 02:49:35 -0000
@@ -251,7 +251,7 @@ GCJ_FOR_TARGET =
 # variable is passed down to the gcc Makefile, where it is used to
 # build libgcc2.a.  We define it here so that it can itself be
 # overridden on the command line.
-GCC_FOR_TARGET = $$r/gcc/xgcc -B$$r/gcc/ $(FLAGS_FOR_TARGET)
+GCC_FOR_TARGET = $(STAGE_CC_WRAPPER) $$r/gcc/xgcc -B$$r/gcc/ $(FLAGS_FOR_TARGET)
 
 AS_FOR_TARGET = ` \
   if [ -f $$r/gas/as-new ] ; then \
Index: configure.in
===================================================================
RCS file: /cvs/gcc/egcs/configure.in,v
retrieving revision 1.176
diff -u -p -r1.176 configure.in
--- configure.in 6 Aug 2002 09:26:29 -0000 1.176
+++ configure.in 14 Aug 2002 02:49:38 -0000
@@ -1554,10 +1554,10 @@ cat >$sedtemp <<EOF
 s:^TARGET_CONFIGDIRS[ 	]*=.*$:TARGET_CONFIGDIRS = ${target_configdirs}:
 s%^TARGET_CONFIGARGS[ 	]*=.*$%TARGET_CONFIGARGS = ${targargs}%
 s%^FLAGS_FOR_TARGET[ 	]*=.*$%FLAGS_FOR_TARGET = ${FLAGS_FOR_TARGET}%
-s%^CC_FOR_TARGET[ 	]*=.*$%CC_FOR_TARGET = ${CC_FOR_TARGET}%
-s%^GCJ_FOR_TARGET[      ]*=.*$%GCJ_FOR_TARGET = ${GCJ_FOR_TARGET}%
-s%^CXX_FOR_TARGET[ 	]*=.*$%CXX_FOR_TARGET = ${qCXX_FOR_TARGET}%
-s%^CXX_FOR_TARGET_FOR_RECURSIVE_MAKE[ 	]*=.*$%CXX_FOR_TARGET_FOR_RECURSIVE_MAKE = ${qqCXX_FOR_TARGET}%
+s%^CC_FOR_TARGET[ 	]*=.*$%CC_FOR_TARGET = \$(STAGE_CC_WRAPPER) ${CC_FOR_TARGET}%
+s%^GCJ_FOR_TARGET[      ]*=.*$%GCJ_FOR_TARGET = \$(STAGE_CC_WRAPPER) ${GCJ_FOR_TARGET}%
+s%^CXX_FOR_TARGET[ 	]*=.*$%CXX_FOR_TARGET = \$(STAGE_CC_WRAPPER) ${qCXX_FOR_TARGET}%
+s%^CXX_FOR_TARGET_FOR_RECURSIVE_MAKE[ 	]*=.*$%CXX_FOR_TARGET_FOR_RECURSIVE_MAKE = \$(STAGE_CC_WRAPPER) ${qqCXX_FOR_TARGET}%
 s%^TARGET_SUBDIR[ 	]*=.*$%TARGET_SUBDIR = ${target_subdir}%
 s%^BUILD_SUBDIR[ 	]*=.*$%BUILD_SUBDIR = ${build_subdir}%
 s%^BUILD_CONFIGARGS[ 	]*=.*$%BUILD_CONFIGARGS = ${buildargs}%
Index: gcc/Makefile.in
===================================================================
RCS file: /cvs/gcc/egcs/gcc/Makefile.in,v
retrieving revision 1.933
diff -u -p -r1.933 Makefile.in
--- gcc/Makefile.in 13 Aug 2002 23:11:37 -0000 1.933
+++ gcc/Makefile.in 14 Aug 2002 02:49:43 -0000
@@ -175,7 +175,7 @@ USER_H = $(srcdir)/ginclude/stdarg.h $(s
 # The GCC to use for compiling libgcc.a and crt*.o.
 # Usually the one we just built.
 # Don't use this as a dependency--use $(GCC_PASSES) or $(GCC_PARTS).
-GCC_FOR_TARGET = ./xgcc -B./ -B$(build_tooldir)/bin/ -isystem $(build_tooldir)/include -isystem $(build_tooldir)/sys-include
+GCC_FOR_TARGET = $(STAGE_CC_WRAPPER) ./xgcc -B./ -B$(build_tooldir)/bin/ -isystem $(build_tooldir)/include -isystem $(build_tooldir)/sys-include
 
 # This is used instead of ALL_CFLAGS when compiling with GCC_FOR_TARGET.
 # It omits XCFLAGS, and specifies -B./.
@@ -3330,7 +3330,7 @@ stage1_copy: stage1_build
 	echo stage2_build > stage_last
 
 stage2_build: stage1_copy
-	$(MAKE) CC="stage1/xgcc$(exeext) -Bstage1/ -B$(build_tooldir)/bin/" \
+	$(MAKE) CC="$(STAGE_CC_WRAPPER) stage1/xgcc$(exeext) -Bstage1/ -B$(build_tooldir)/bin/" \
 		 STAGE_PREFIX=stage1/ \
 		 $(STAGE2_FLAGS_TO_PASS)
 	$(STAMP) stage2_build
@@ -3342,7 +3342,7 @@ stage2_copy: stage2_build
 	echo stage3_build > stage_last
 
 stage3_build: stage2_copy
-	$(MAKE) CC="stage2/xgcc$(exeext) -Bstage2/ -B$(build_tooldir)/bin/" \
+	$(MAKE) CC="$(STAGE_CC_WRAPPER) stage2/xgcc$(exeext) -Bstage2/ -B$(build_tooldir)/bin/" \
 		 STAGE_PREFIX=stage2/ \
 		 $(STAGE2_FLAGS_TO_PASS)
 	$(STAMP) stage3_build
@@ -3355,7 +3355,7 @@ stage3_copy: stage3_build
 	echo stage4_build > stage_last
 
 stage4_build: stage3_copy
-	$(MAKE) CC="stage3/xgcc$(exeext) -Bstage3/ -B$(build_tooldir)/bin/" \
+	$(MAKE) CC="$(STAGE_CC_WRAPPER) stage3/xgcc$(exeext) -Bstage3/ -B$(build_tooldir)/bin/" \
 		 STAGE_PREFIX=stage3/ \
 		 $(STAGE2_FLAGS_TO_PASS)
 	$(STAMP) stage4_build

[-- Attachment #3: distcc --]
[-- Type: application/x-sh, Size: 1286 bytes --]

[-- Attachment #4: netpwd --]
[-- Type: application/x-sh, Size: 790 bytes --]

[-- Attachment #5: Type: text/plain, Size: 289 bytes --]


-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Martin Pool]

On 14 Aug 2002, Alexandre Oliva <aoliva@redhat.com> wrote:
> A while ago, we've been talking about using distcc to speed up gcc
> bootstraps.  I've made some progress today towards this goal, but
> there's still some way to go to get there.

Great!

> - the bootstrap check fails, even though I have every reason to
>   believe it bootstrapped correctly.  I believe the cause of the
>   failure is the problem documented in distcc's manual: that gcc adds
>   the preprocessed filename to the debugging info emitted for each
>   file.  I believe it shouldn't do this, but rather just use the name
>   in the first #line directive in the preprocessed file.  I'll look
>   into trying to fix this problem, if I confirm it is indeed the
>   reason for the failure.

I can see how including the pwd is useful for making gdb more
friendly, but it seems kind of tacky at the same time, by putting a
kind of "hidden variable" into the .o file.  

If you have to do it, it seems to me that it would be cleaner to make
cpp resolve the absolute path of the input file, and put that in to a
# directive, either replacing the existing ones or otherwise.

Presumably you could also sometimes get false failures of other tools
that try to compare .o files built at different times.

> - the load on my build machine skyrockets while building the ada
>   compiler and the Java library.  It would be nice if distcc could
>   throttle other instances thereof when it realizes it can't send
>   builds to remote build servers.  A lock on localhost, to be used in
>   such situations, would be ideal IMO.  Is this an acceptable change
>   for distcc?

Yes, I think that would be very appropriate.  I would also like to
make it take into account the load caused by jobs which turn out to be
impossible to distribute, such as preprocessor or linker invocations.
I have started on that for the next version.

-- 
Martin 

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 14, 2002, Martin Pool <mbp@samba.org> wrote:

> I can see how including the pwd is useful for making gdb more
> friendly, but it seems kind of tacky at the same time, by putting a
> kind of "hidden variable" into the .o file.  

I agree.  I have confirmed that this is the reason for the
differences, and I've managed to avoid them by making sure I start
distccd on all of the build servers while being in the same directory
(say /tmp), and then *not* listing localhost in DISTCC_HOSTS (using
the local host name instead).

> If you have to do it, it seems to me that it would be cleaner to make
> cpp resolve the absolute path of the input file, and put that in to a
> # directive, either replacing the existing ones or otherwise.

An alternative I'm considering is to have cpp emit some different #
directive that contains the full pathname of the source file, or
perhaps just the directory name.  I.e., something like:

# 1 "filename"
# 1 "<directory> /path/name"
# 1 "filename"
...

I find this somewhat icky, but it's fully backward-compatible.  Neil,
Zack, anyone else, what do you think?


The other problem in libstdc++-v3 seems to be a bug in distcc:
libstdc++-v3 tries to runs .../xgcc -B.../ conftest.C -c -S.  The
distcc client writes the assembly output to conftest.o.  Curiously, if
I add `-o conftest.s' to the command line, it is the server that
fails, claiming it couldn't find conftest.C.  I haven't tracked it
down yet, but found a work-around: adding `ln -s conftest.o
conftest.s' just before the compilation in libstdc++-v3/configure.
With that, I managed to get a full bootstrap of binutils+gcc+gdb to
succeed.  Yay!

>> - the load on my build machine skyrockets while building the ada
>> compiler and the Java library. [...]  A lock on localhost, to be
>> used in such situations, would be ideal IMO.  Is this an acceptable
>> change for distcc?

> Yes, I think that would be very appropriate.  I would also like to
> make it take into account the load caused by jobs which turn out to be
> impossible to distribute, such as preprocessor or linker invocations.
> I have started on that for the next version.

Cool!  So, is there anything left for me to do regarding this feature,
or are you already taking care of all of it?  Let me know how I can
help.  I can't promise I'll have time for it, but I can always try :-)

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Phil Edwards]

On Fri, Aug 16, 2002 at 05:19:12AM -0300, Alexandre Oliva wrote:
> The other problem in libstdc++-v3 seems to be a bug in distcc:
> libstdc++-v3 tries to runs .../xgcc -B.../ conftest.C -c -S.

How odd.  Why do we do that?


Phil

-- 
I would therefore like to posit that computing's central challenge, viz. "How
not to make a mess of it," has /not/ been met.
                                                 - Edsger Dijkstra, 1930-2002

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 16, 2002, Phil Edwards <phil@jaj.com> wrote:

> On Fri, Aug 16, 2002 at 05:19:12AM -0300, Alexandre Oliva wrote:
>> The other problem in libstdc++-v3 seems to be a bug in distcc:
>> libstdc++-v3 tries to runs .../xgcc -B.../ conftest.C -c -S.

> How odd.  Why do we do that?

To check the exception model used by default in GCC.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: using distcc to speed up gcc bootstraps

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 2046 bytes --]

On Aug 14, 2002, Alexandre Oliva <aoliva@redhat.com> wrote:

> A while ago, we've been talking about using distcc to speed up gcc
> bootstraps.  I've made some progress today towards this goal, but
> there's still some way to go to get there.

I'm checking this patch in, that is useful to bootstrap gcc, but not
build libstdc++-v3 nor libjava yet: this will require some fixing in
distcc.  So, there's no reason to not install this right away.  I also
enclose newer versions of the helper scripts, that I'm considering
adding to contrib.  Does anyone think it's worth it?

> I've written a wrapper for distcc that goes over the command line
> arguments looking for relative pathnames and, if it finds any, it
> turns them into network-neutral full pathnames, using the netpwd
> script below, and then runs the real distcc to do the work.

> Unfortunately, this solution is not complete yet, since libstdc++-v3
> fails to configure with this patch.  I'm yet to investigate why, but I
> wanted heads up that I've made some progress.

> I enclose below the patch for GCC I'm using, that, if you set
> STAGE_CC_WRAPPER=distcc, will let you take advantage of distcc for the
> several stages of gcc.  I'd like to put this patch in as soon as I get
> it into a functional state, but I'm open to suggestions for the name
> of the variable.

> I also enclose the distcc wrapper script and the netpwd script it uses
> to turn pwd into a network-neutral full pathname.

> Besides the problem mentioned above, I've run into two other problems:

> - the bootstrap check fails, even though I have every reason to
>   believe it bootstrapped correctly.  I believe the cause of the
>   failure is the problem documented in distcc's manual: that gcc adds
>   the preprocessed filename to the debugging info emitted for each
>   file.  I believe it shouldn't do this, but rather just use the name
>   in the first #line directive in the preprocessed file.  I'll look
>   into trying to fix this problem, if I confirm it is indeed the
>   reason for the failure.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: stage-cc-wrapper.patch --]
[-- Type: text/x-patch, Size: 4447 bytes --]

Index: ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* Makefile.in (GCC_FOR_TARGET): Prepend STAGE_CC_WRAPPER.
	* configure.in (CC_FOR_TARGET, GCJ_FOR_TARGET, CXX_FOR_TARGET,
	CXX_FOR_TARGET_FOR_RECURSIVE_MAKE): Likewise.

Index: gcc/ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* Makefile.in (GCC_FOR_TARGET): Prepend STAGE_CC_WRAPPER.
	(stage2_build, stage3_build, stage4_build): Likewise, to CC.

Index: Makefile.in
===================================================================
RCS file: /cvs/gcc/egcs/Makefile.in,v
retrieving revision 1.110
diff -u -p -r1.110 Makefile.in
--- Makefile.in 8 Jul 2002 21:40:41 -0000 1.110
+++ Makefile.in 14 Aug 2002 02:49:35 -0000
@@ -251,7 +251,7 @@ GCJ_FOR_TARGET =
 # variable is passed down to the gcc Makefile, where it is used to
 # build libgcc2.a.  We define it here so that it can itself be
 # overridden on the command line.
-GCC_FOR_TARGET = $$r/gcc/xgcc -B$$r/gcc/ $(FLAGS_FOR_TARGET)
+GCC_FOR_TARGET = $(STAGE_CC_WRAPPER) $$r/gcc/xgcc -B$$r/gcc/ $(FLAGS_FOR_TARGET)
 
 AS_FOR_TARGET = ` \
   if [ -f $$r/gas/as-new ] ; then \
Index: configure.in
===================================================================
RCS file: /cvs/gcc/egcs/configure.in,v
retrieving revision 1.176
diff -u -p -r1.176 configure.in
--- configure.in 6 Aug 2002 09:26:29 -0000 1.176
+++ configure.in 14 Aug 2002 02:49:38 -0000
@@ -1554,10 +1554,10 @@ cat >$sedtemp <<EOF
 s:^TARGET_CONFIGDIRS[ 	]*=.*$:TARGET_CONFIGDIRS = ${target_configdirs}:
 s%^TARGET_CONFIGARGS[ 	]*=.*$%TARGET_CONFIGARGS = ${targargs}%
 s%^FLAGS_FOR_TARGET[ 	]*=.*$%FLAGS_FOR_TARGET = ${FLAGS_FOR_TARGET}%
-s%^CC_FOR_TARGET[ 	]*=.*$%CC_FOR_TARGET = ${CC_FOR_TARGET}%
-s%^GCJ_FOR_TARGET[      ]*=.*$%GCJ_FOR_TARGET = ${GCJ_FOR_TARGET}%
-s%^CXX_FOR_TARGET[ 	]*=.*$%CXX_FOR_TARGET = ${qCXX_FOR_TARGET}%
-s%^CXX_FOR_TARGET_FOR_RECURSIVE_MAKE[ 	]*=.*$%CXX_FOR_TARGET_FOR_RECURSIVE_MAKE = ${qqCXX_FOR_TARGET}%
+s%^CC_FOR_TARGET[ 	]*=.*$%CC_FOR_TARGET = \$(STAGE_CC_WRAPPER) ${CC_FOR_TARGET}%
+s%^GCJ_FOR_TARGET[      ]*=.*$%GCJ_FOR_TARGET = \$(STAGE_CC_WRAPPER) ${GCJ_FOR_TARGET}%
+s%^CXX_FOR_TARGET[ 	]*=.*$%CXX_FOR_TARGET = \$(STAGE_CC_WRAPPER) ${qCXX_FOR_TARGET}%
+s%^CXX_FOR_TARGET_FOR_RECURSIVE_MAKE[ 	]*=.*$%CXX_FOR_TARGET_FOR_RECURSIVE_MAKE = \$(STAGE_CC_WRAPPER) ${qqCXX_FOR_TARGET}%
 s%^TARGET_SUBDIR[ 	]*=.*$%TARGET_SUBDIR = ${target_subdir}%
 s%^BUILD_SUBDIR[ 	]*=.*$%BUILD_SUBDIR = ${build_subdir}%
 s%^BUILD_CONFIGARGS[ 	]*=.*$%BUILD_CONFIGARGS = ${buildargs}%
Index: gcc/Makefile.in
===================================================================
RCS file: /cvs/gcc/egcs/gcc/Makefile.in,v
retrieving revision 1.933
diff -u -p -r1.933 Makefile.in
--- gcc/Makefile.in 13 Aug 2002 23:11:37 -0000 1.933
+++ gcc/Makefile.in 14 Aug 2002 02:49:43 -0000
@@ -175,7 +175,7 @@ USER_H = $(srcdir)/ginclude/stdarg.h $(s
 # The GCC to use for compiling libgcc.a and crt*.o.
 # Usually the one we just built.
 # Don't use this as a dependency--use $(GCC_PASSES) or $(GCC_PARTS).
-GCC_FOR_TARGET = ./xgcc -B./ -B$(build_tooldir)/bin/ -isystem $(build_tooldir)/include -isystem $(build_tooldir)/sys-include
+GCC_FOR_TARGET = $(STAGE_CC_WRAPPER) ./xgcc -B./ -B$(build_tooldir)/bin/ -isystem $(build_tooldir)/include -isystem $(build_tooldir)/sys-include
 
 # This is used instead of ALL_CFLAGS when compiling with GCC_FOR_TARGET.
 # It omits XCFLAGS, and specifies -B./.
@@ -3330,7 +3330,7 @@ stage1_copy: stage1_build
 	echo stage2_build > stage_last
 
 stage2_build: stage1_copy
-	$(MAKE) CC="stage1/xgcc$(exeext) -Bstage1/ -B$(build_tooldir)/bin/" \
+	$(MAKE) CC="$(STAGE_CC_WRAPPER) stage1/xgcc$(exeext) -Bstage1/ -B$(build_tooldir)/bin/" \
 		 STAGE_PREFIX=stage1/ \
 		 $(STAGE2_FLAGS_TO_PASS)
 	$(STAMP) stage2_build
@@ -3342,7 +3342,7 @@ stage2_copy: stage2_build
 	echo stage3_build > stage_last
 
 stage3_build: stage2_copy
-	$(MAKE) CC="stage2/xgcc$(exeext) -Bstage2/ -B$(build_tooldir)/bin/" \
+	$(MAKE) CC="$(STAGE_CC_WRAPPER) stage2/xgcc$(exeext) -Bstage2/ -B$(build_tooldir)/bin/" \
 		 STAGE_PREFIX=stage2/ \
 		 $(STAGE2_FLAGS_TO_PASS)
 	$(STAMP) stage3_build
@@ -3355,7 +3355,7 @@ stage3_copy: stage3_build
 	echo stage4_build > stage_last
 
 stage4_build: stage3_copy
-	$(MAKE) CC="stage3/xgcc$(exeext) -Bstage3/ -B$(build_tooldir)/bin/" \
+	$(MAKE) CC="$(STAGE_CC_WRAPPER) stage3/xgcc$(exeext) -Bstage3/ -B$(build_tooldir)/bin/" \
 		 STAGE_PREFIX=stage3/ \
 		 $(STAGE2_FLAGS_TO_PASS)
 	$(STAMP) stage4_build

[-- Attachment #3: distccrel --]
[-- Type: application/octet-stream, Size: 1810 bytes --]

#! /bin/sh
# Copyright 2002 Free Software Foundation
# by Alexandre Oliva <aoliva@redhat.com>

# This script is Free Software, and it can be copied, distributed and
# modified as defined in the GNU General Public License.  A copy of
# its license can be downloaded from http://www.gnu.org/copyleft/gpl.html

# This is a wrapper for distcc that turns relative pathnames into
# network-neutral ones.  It modifies anything containing a slash and
# not starting with dash or slash, as well as -B flags.  We need not
# be concerned about -I and -L, since distcc always does preprocessing
# and linking locally.

# To use it, set the STAGE_CC_WRAPPER to the name of this script, and
# the GCC build machinery will use it when buildling bootstrap stages
# and target libraries.  You will also need to point NETPWD to the
# netpwd script, and perhaps customize it to map pathnames properly
# for your environment.


# If the first pathname is relative (.../xgcc, ./libtool, etc),
case "$1" in
[^/]*/*)
  nargs=$#
  basedir=`${NETPWD-netpwd}`
  # then process arguments
  for arg
  do
    case $arg in
    -B[^/]*)
      arg=-B$basedir/`echo "X$arg" | sed -e '1s/^X-B//'`
      ;;
    [^-/]*/*)
      arg=$basedir/$arg
      ;;
    esac
    set fnord ${1+"$@"} "$arg"
  done
  shift $nargs # take list of fnords out
  shift $nargs # take original args out
  ;;
esac

# Since GCC adds the current working directory to the object file's
# debugging info, you have to start distcc on all machines in the same
# directory (say /tmp) and make sure you never use the localhost
# compiler, using the current machine as if it were remote, if at all.
case " $DISTCC_HOSTS " in
*" localhost "*)
  HOSTNAME=${HOSTNAME-`uname -n`}
  DISTCC_HOSTS=`echo $DISTCC_HOSTS | sed "s,localhost,$HOSTNAME,"`
  ;;
esac

exec distcc ${1+"$@"}

[-- Attachment #4: netpwd --]
[-- Type: application/octet-stream, Size: 790 bytes --]

#! /bin/sh
# Copyright 2002 Free Software Foundation
# by Alexandre Oliva <aoliva@redhat.com>

# This script is Free Software, and it can be copied, distributed and
# modified as defined in the GNU General Public License.  A copy of
# its license can be downloaded from http://www.gnu.org/copyleft/gpl.html

# Turn a local pathname into a network-neutral one.  It assumes
# /net/$HOSTNAME/pathname can access $HOSTNAME's /pathname from other
# machines, and that /net/$HOSTNAME is actually mounted (by amd)
# inside /.automount/$HOSTNAME/root, so it compensates for that.

dir=`${REAL_PWD-pwd}`
case $dir in
/net/*)
  ;;
/.automount/*/root/*)
  dir=/net/`echo "$dir" |  sed -e '1s,^/\.automount/,,' -e 's,/root/,/,'`
  ;;
*)
  dir=/net/${HOSTNAME-`uname -n`}$dir
  ;;
esac
echo $dir
exit 0

[-- Attachment #5: Type: text/plain, Size: 289 bytes --]


-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 1526 bytes --]

On Aug 16, 2002, Alexandre Oliva <aoliva@redhat.com> wrote:

> An alternative I'm considering is to have cpp emit some different #
> directive that contains the full pathname of the source file, or
> perhaps just the directory name.  I.e., something like:

> # 1 "filename"
> # 1 "<directory> /path/name"
> # 1 "filename"
> ...

> I find this somewhat icky, but it's fully backward-compatible.  Neil,
> Zack, anyone else, what do you think?

Not having got any complaints about this proposal, I went ahead and
implemented it.  Patch below.  Ok to install?  Bootstrapped on
athlon-pc-linux-gnu.

Unfortunately, the bootstrap compare test still failed :-(

The reason is that gcc generates different debugging information for a
source file such as:

#define FOO(X)
int FOO((x, y,
         z, w)) bar;
int baz;

If we preprocess this code and compile it separately (e.g.,
-save-temps), the result is:

# 2 "<filename>"
int bar;

int baz;

i.e., the declaration of bar is moved from line 3 to line 2.  However,
with integrated preprocessing, the declaration of bar is correctly
flagged as in line 3 in debugging information.  This difference is
enough for the bootstrap compare to fail, since at least varray.h
contains a number of union member declarations like above, with
multi-line GTY() macro invocations between the types and the
declarators.  Oh, well...

This should probably be fixed.  I don't think it's good that compiling
with or without -save-temps can result in different assembly output.
Neil, Zack, any comments?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: distcc-preproc-directory.patch --]
[-- Type: text/x-patch, Size: 10442 bytes --]

Index: gcc/ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* cppinit.c (read_original_filename): Return bool to tell
	whether the current directory was present in the input stream.
	Call...
	(read_original_directory): New function.  Look for a # line
	"<directory>..." right after the original filename.
	(cpp_read_main_file): Emit <directory> directive if it was not
	emitted by read_original_directory.
	(cpp_finish_options): Don't translate <built-in> nor <command
	line>.
	* dbxout.c (dbxout_init): Call get_src_pwd() instead of getpwd().
	* dwarf2out.c (gen_compile_unit_die): Likewise.
	* dwarfout.c (output_compile_unit_die, dwarfout_init): Likewise.
	* toplev.c (src_pwd): New static variable.
	(set_src_pwd, get_src_pwd): New functions.
	* toplev.h (get_src_pwd, set_src_pwd): Declare.
	* c-lex.c (cb_file_change): Recognize <directory> markers.
	Rename main_input_file if it's the current input_file and an
	LC_RENAME directive comes up.

Index: gcc/c-lex.c
===================================================================
RCS file: /cvs/uberbaum/gcc/c-lex.c,v
retrieving revision 1.187
diff -u -p -r1.187 c-lex.c
--- gcc/c-lex.c 12 Aug 2002 06:02:53 -0000 1.187
+++ gcc/c-lex.c 23 Aug 2002 09:14:35 -0000
@@ -295,6 +295,14 @@ cb_file_change (pfile, new_map)
       
       (*debug_hooks->end_source_file) (to_line);
     }
+  else if (new_map->reason == LC_RENAME)
+    {
+      if (strncmp (new_map->to_file, "<directory>",
+		   strlen ("<directory>")) == 0)
+	set_src_pwd (new_map->to_file + strlen ("<directory>"));
+      if (input_filename == main_input_filename)
+	main_input_filename = new_map->to_file;
+    }
 
   update_header_times (new_map->to_file);
   in_system_header = new_map->sysp != 0;
Index: gcc/cppinit.c
===================================================================
RCS file: /cvs/uberbaum/gcc/cppinit.c,v
retrieving revision 1.260
diff -u -p -r1.260 cppinit.c
--- gcc/cppinit.c 21 Aug 2002 17:07:26 -0000 1.260
+++ gcc/cppinit.c 23 Aug 2002 09:14:36 -0000
@@ -106,7 +106,8 @@ static bool push_include		PARAMS ((cpp_r
 						 struct pending_option *));
 static void free_chain			PARAMS ((struct pending_option *));
 static void init_standard_includes	PARAMS ((cpp_reader *));
-static void read_original_filename	PARAMS ((cpp_reader *));
+static bool read_original_filename	PARAMS ((cpp_reader *));
+static bool read_original_directory	PARAMS ((cpp_reader *));
 static void new_pending_directive	PARAMS ((struct cpp_pending *,
 						 const char *,
 						 cl_directive_handler));
@@ -950,6 +951,8 @@ cpp_read_main_file (pfile, fname, table)
      const char *fname;
      hash_table *table;
 {
+  char *pwd;
+
   sanity_checks (pfile);
 
   post_options (pfile);
@@ -999,8 +1002,22 @@ cpp_read_main_file (pfile, fname, table)
 
   /* For foo.i, read the original filename foo.c now, for the benefit
      of the front ends.  */
-  if (CPP_OPTION (pfile, preprocessed))
-    read_original_filename (pfile);
+  if (CPP_OPTION (pfile, preprocessed)
+      && read_original_filename (pfile))
+    pwd = NULL;
+  else
+    pwd = getpwd ();
+
+  if (pwd)
+    {
+      char *dir = concat ("<directory>", pwd, NULL);
+      const char *name = pfile->map->to_file;
+      unsigned int line = pfile->line;
+      int sysp = pfile->map->sysp;
+      
+      _cpp_do_file_change (pfile, LC_RENAME, dir, 1, 0);
+      _cpp_do_file_change (pfile, LC_RENAME, name, line, sysp);
+    }
 
   return pfile->map->to_file;
 }
@@ -1009,7 +1026,7 @@ cpp_read_main_file (pfile, fname, table)
    handle the directive so we know the original file name.  This will
    generate file_change callbacks, which the front ends must handle
    appropriately given their state of initialization.  */
-static void
+static bool
 read_original_filename (pfile)
      cpp_reader *pfile;
 {
@@ -1027,12 +1044,61 @@ read_original_filename (pfile)
       if (token1->type == CPP_NUMBER)
 	{
 	  _cpp_handle_directive (pfile, token->flags & PREV_WHITE);
-	  return;
+	  return read_original_directory (pfile);
+	}
+    }
+
+  /* Backup as if nothing happened.  */
+  _cpp_backup_tokens (pfile, 1);
+  return false;
+}
+
+/* For preprocessed files, if the tokens following the first filename
+   line are of the form # NUM "<directory>...", handle the directive
+   so we know the original current directory.  This will generate
+   file_change callbacks, which the front ends must handle
+   appropriately given their state of initialization.  */
+static bool
+read_original_directory (pfile)
+     cpp_reader *pfile;
+{
+  const cpp_token *token, *token1, *token2;
+
+  /* Lex ahead; if the first tokens are of the form # NUM, then
+     process the directive, otherwise back up.  */
+  token = _cpp_lex_direct (pfile);
+  if (token->type == CPP_HASH)
+    {
+      token1 = _cpp_lex_direct (pfile);
+
+      /* If it's a #line directive, handle it.  */
+      if (token1->type == CPP_NUMBER)
+	{
+	  token2 = _cpp_lex_direct (pfile);
+	  _cpp_backup_tokens (pfile, 2);
+
+	  if (token2->type == CPP_STRING
+	      && strncmp ((const char *)token2->val.str.text, "<directory>",
+			  strlen ("<directory>")) == 0)
+	    {
+	      const char *name = pfile->map->to_file;
+	      unsigned int line = pfile->line;
+	      int sysp = pfile->map->sysp;
+
+	      _cpp_handle_directive (pfile, token->flags & PREV_WHITE);
+
+	      _cpp_do_file_change (pfile, LC_RENAME, name, line, sysp);
+	      return true;
+	    }
 	}
+      else
+	_cpp_backup_tokens (pfile, 1);
     }
 
   /* Backup as if nothing happened.  */
   _cpp_backup_tokens (pfile, 1);
+
+  return false;
 }
 
 /* Handle pending command line options: -D, -U, -A, -imacros and
@@ -1052,9 +1118,9 @@ cpp_finish_options (pfile)
     {
       struct pending_option *p;
 
-      _cpp_do_file_change (pfile, LC_RENAME, _("<built-in>"), 1, 0);
+      _cpp_do_file_change (pfile, LC_RENAME, "<built-in>", 1, 0);
       init_builtins (pfile);
-      _cpp_do_file_change (pfile, LC_RENAME, _("<command line>"), 1, 0);
+      _cpp_do_file_change (pfile, LC_RENAME, "<command line>", 1, 0);
       for (p = CPP_OPTION (pfile, pending)->directive_head; p; p = p->next)
 	(*p->handler) (pfile, p->arg);
 
Index: gcc/dbxout.c
===================================================================
RCS file: /cvs/uberbaum/gcc/dbxout.c,v
retrieving revision 1.124
diff -u -p -r1.124 dbxout.c
--- gcc/dbxout.c 15 Aug 2002 19:59:28 -0000 1.124
+++ gcc/dbxout.c 23 Aug 2002 09:14:38 -0000
@@ -431,7 +431,8 @@ dbxout_init (input_file_name)
   if (use_gnu_debug_info_extensions)
 #endif
     {
-      if (!cwd && (cwd = getpwd ()) && (!*cwd || cwd[strlen (cwd) - 1] != '/'))
+      if (!cwd && (cwd = get_src_pwd ())
+	  && (!*cwd || cwd[strlen (cwd) - 1] != '/'))
 	cwd = concat (cwd, FILE_NAME_JOINER, NULL);
       if (cwd)
 	{
Index: gcc/dwarf2out.c
===================================================================
RCS file: /cvs/uberbaum/gcc/dwarf2out.c,v
retrieving revision 1.383
diff -u -p -r1.383 dwarf2out.c
--- gcc/dwarf2out.c 21 Aug 2002 02:41:43 -0000 1.383
+++ gcc/dwarf2out.c 23 Aug 2002 09:14:45 -0000
@@ -10719,7 +10719,7 @@ gen_compile_unit_die (filename)
 {
   dw_die_ref die;
   char producer[250];
-  const char *wd = getpwd ();
+  const char *wd = get_src_pwd ();
   const char *language_string = lang_hooks.name;
   int language;
 
Index: gcc/dwarfout.c
===================================================================
RCS file: /cvs/uberbaum/gcc/dwarfout.c,v
retrieving revision 1.115
diff -u -p -r1.115 dwarfout.c
--- gcc/dwarfout.c 15 Aug 2002 19:59:29 -0000 1.115
+++ gcc/dwarfout.c 23 Aug 2002 09:14:49 -0000
@@ -4146,7 +4146,7 @@ output_compile_unit_die (arg)
     stmt_list_attribute (LINE_BEGIN_LABEL);
 
   {
-    const char *wd = getpwd ();
+    const char *wd = get_src_pwd ();
     if (wd)
       comp_dir_attribute (wd);
   }
@@ -6271,7 +6271,7 @@ dwarfout_init (main_input_filename)
 	  ASM_OUTPUT_PUSH_SECTION (asm_out_file, DEBUG_SFNAMES_SECTION);
 	  ASM_OUTPUT_LABEL (asm_out_file, SFNAMES_BEGIN_LABEL);
 	  {
-	    const char *pwd = getpwd ();
+	    const char *pwd = get_src_pwd ();
 	    char *dirname;
 
 	    if (!pwd)
Index: gcc/toplev.c
===================================================================
RCS file: /cvs/uberbaum/gcc/toplev.c,v
retrieving revision 1.668
diff -u -p -r1.668 toplev.c
--- gcc/toplev.c 14 Aug 2002 06:36:51 -0000 1.668
+++ gcc/toplev.c 23 Aug 2002 09:14:53 -0000
@@ -1558,7 +1558,43 @@ static const lang_independent_options W_
   {"missing-noreturn", &warn_missing_noreturn, 1,
    N_("Warn about functions which might be candidates for attribute noreturn") }
 };
+\f
+/* The current working directory of a translation.  It's generally the
+   directory from which compilation was initiated, but a preprocessed
+   file may specify the original directory in which it was
+   created.  */
 
+static const char *src_pwd;
+
+/* Initialize src_pwd with the given string, and return true.  If it
+   was already initialized, return false.  As a special case, it may
+   be called with a NULL argument to test whether src_pwd has NOT been
+   initialized yet.  */
+
+bool
+set_src_pwd (pwd)
+     const char *pwd;
+{
+  if (src_pwd)
+    return false;
+
+  src_pwd = xstrdup (pwd);
+  return true;
+}
+
+/* Return the directory from which the translation unit was initiated,
+   in case set_src_pwd() was not called before to assign it a
+   different value.  */
+
+const char *
+get_src_pwd ()
+{
+  if (! src_pwd)
+    src_pwd = getpwd ();
+
+  return src_pwd;
+}
+\f
 void
 set_Wunused (setting)
      int setting;
Index: gcc/toplev.h
===================================================================
RCS file: /cvs/uberbaum/gcc/toplev.h,v
retrieving revision 1.89
diff -u -p -r1.89 toplev.h
--- gcc/toplev.h 4 Aug 2002 16:21:02 -0000 1.89
+++ gcc/toplev.h 23 Aug 2002 09:14:53 -0000
@@ -1,5 +1,6 @@
 /* toplev.h - Various declarations for functions found in toplev.c
-   Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc.
+   Copyright 1998, 1999, 2000, 2001, 2002
+   Free Software Foundation, Inc.
 
 This file is part of GCC.
 
@@ -132,5 +133,8 @@ extern bool fast_math_flags_set_p	PARAMS
 #endif
 extern int exact_log2_wide             PARAMS ((unsigned HOST_WIDE_INT));
 extern int floor_log2_wide             PARAMS ((unsigned HOST_WIDE_INT));
+
+extern const char *get_src_pwd	       PARAMS ((void));
+extern bool set_src_pwd		       PARAMS ((const char *));
 
 #endif /* ! GCC_TOPLEV_H */

[-- Attachment #3: Type: text/plain, Size: 289 bytes --]


-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> > I find this somewhat icky, but it's fully backward-compatible.  Neil,
> > Zack, anyone else, what do you think?
> 
> Not having got any complaints about this proposal, I went ahead and
> implemented it.  Patch below.  Ok to install?  Bootstrapped on
> athlon-pc-linux-gnu.

I don't like it.  This area is messy enough as-is without adding more
random metadata to it.

If this does go in, I don't think it should go in until 3.4 branches,
and I don't think it should be in cpplib (it should be in cc1).

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 23, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> Alexandre Oliva wrote:-
>> > I find this somewhat icky, but it's fully backward-compatible.  Neil,
>> > Zack, anyone else, what do you think?
>> 
>> Not having got any complaints about this proposal, I went ahead and
>> implemented it.  Patch below.  Ok to install?  Bootstrapped on
>> athlon-pc-linux-gnu.

> I don't like it.  This area is messy enough as-is without adding more
> random metadata to it.

So you're saying you don't care about distcc and you think it's not
important to preserve the directory where the source file was
preprocessed in the debugging info?  Or are you just saying you have
some better way to achieve this goal that you failed to mention?

> If this does go in, I don't think it should go in until 3.4 branches,

Why?  It's fully backward-compatible, and it adds information that
helps GCC do a better job.

> and I don't think it should be in cpplib (it should be in cc1).

So how do you suggest it to emit a single <directory> directive right
after the initial source file name?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> So you're saying you don't care about distcc and you think it's not
> important to preserve the directory where the source file was
> preprocessed in the debugging info?  Or are you just saying you have
> some better way to achieve this goal that you failed to mention?

I've not thought about it; I just don't like the idea.

> > If this does go in, I don't think it should go in until 3.4 branches,
> 
> Why?  It's fully backward-compatible, and it adds information that
> helps GCC do a better job.

Because we're fixing bugs now.  I've not be doing stuff that I want
to do, because we're in stage 3.

> > and I don't think it should be in cpplib (it should be in cc1).
> 
> So how do you suggest it to emit a single <directory> directive right
> after the initial source file name?

As part of cc1's preprocessed output code.

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 23, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> I don't like it.  This area is messy enough as-is without adding more
> random metadata to it.

Oh, and did I mention that this patch actually fixes a bug?

Or do you disagree that the compiler, when given the same preprocessed
input and the same command-line flags, should produce the same output?

Currently, it doesn't, if you happen to change your current working
directory.  This patch fixes this very bug.  It's not a regression,
but it is a bug.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> On Aug 23, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:
> 
> > I don't like it.  This area is messy enough as-is without adding more
> > random metadata to it.
> 
> Oh, and did I mention that this patch actually fixes a bug?
> 
> Or do you disagree that the compiler, when given the same preprocessed
> input and the same command-line flags, should produce the same output?
> 
> Currently, it doesn't, if you happen to change your current working
> directory.  This patch fixes this very bug.  It's not a regression,
> but it is a bug.

Let's wait for zack's input.  Zack?  8-)

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 23, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

>> So how do you suggest it to emit a single <directory> directive right
>> after the initial source file name?

> As part of cc1's preprocessed output code.

But how does it tell that it's time to emit the <directory> directive?

Oh, I think I see what you mean.  Since we currently don't handle at
all the case of `gcc -E -x cpp-output' to emit an unmodified copy of
the input file, it's just a matter of the preprocessor output emitting
<directory>getpwd() after the initial line directive, and handling
this directive entirely in cc1.  Does this sound acceptable to you?
If so, I'll give it a try tonight.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Zack Weinberg]

On Fri, Aug 23, 2002 at 10:19:47PM +0100, Neil Booth wrote:
> Alexandre Oliva wrote:-
> 
> > On Aug 23, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:
> > 
> > > I don't like it.  This area is messy enough as-is without adding more
> > > random metadata to it.
> > 
> > Oh, and did I mention that this patch actually fixes a bug?
> > 
> > Or do you disagree that the compiler, when given the same preprocessed
> > input and the same command-line flags, should produce the same output?
> > 
> > Currently, it doesn't, if you happen to change your current working
> > directory.  This patch fixes this very bug.  It's not a regression,
> > but it is a bug.
> 
> Let's wait for zack's input.  Zack?  8-)

I agree that this is a bug; I'm not sure that Alexandre's patch is the
appropriate fix.  I do not fully understand the problem; what's the
executive summary?

zw

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 23, 2002, Zack Weinberg <zack@codesourcery.com> wrote:

> I do not fully understand the problem; what's the executive summary?

Debugging output routines use getpwd() to emit the `current directory'
bit.  So, if you preprocess a program in one directory, take the
preprocessed output elsewhere, and run the compiler on the
preprocessed file, you get different debugging info depending on
the cwd at the time of the translation to assembly.

If instead we included the cwd of the preprocessor at the time it
started the translation, the compilation result would be the same
regardless of the cwd of the execution of cc1.

My reasoning is that, if we handle things like __DATE__ and __TIME__
in the preprocessor (that could change should they be deferred to the
compiler proper), so should the cwd.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Zack Weinberg]

On Fri, Aug 23, 2002 at 10:59:55PM -0300, Alexandre Oliva wrote:
> On Aug 23, 2002, Zack Weinberg <zack@codesourcery.com> wrote:
> 
> > I do not fully understand the problem; what's the executive summary?
> 
> Debugging output routines use getpwd() to emit the `current directory'
> bit.  So, if you preprocess a program in one directory, take the
> preprocessed output elsewhere, and run the compiler on the
> preprocessed file, you get different debugging info depending on
> the cwd at the time of the translation to assembly.

Okay.  Why is information about the current directory in the object
file at all?

zw

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 23, 2002, Zack Weinberg <zack@codesourcery.com> wrote:

> Okay.  Why is information about the current directory in the object
> file at all?

It's DW_AT_comp_dir in DWARF-2.  Debuggers rely on it to find source
files.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> > As part of cc1's preprocessed output code.
> 
> But how does it tell that it's time to emit the <directory> directive?

cc1 should be doing the preprocessed output.  Currently it's in
cppmain.c, but it should be moved to a "c-" file in the front end.
This is all part of the work to make cpplib a proper library.
If cc1 is doing the preprocessed output, it's easy for it to dump
whatever it wants in there.

Handling the preprocessed input (with extra lines like yours) is a
little harder, but should be easy once we read a token line at a
time.

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> But how does it tell that it's time to emit the <directory> directive?

I'm not sure, but if we do this it might be better to keep all the info
on the first line, so it contains the original file name and the
original current directory.  I'm not keen on having more # lines.

Why did your patch remove the gettext calls on the pseudo file names?

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 24, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> Alexandre Oliva wrote:-
>> But how does it tell that it's time to emit the <directory> directive?

> I'm not sure, but if we do this it might be better to keep all the info
> on the first line, so it contains the original file name and the
> original current directory.  I'm not keen on having more # lines.

How do you know what is the current directory and what is the original
file name?  Are you suggesting more than one string or something like
that?  *That* would be backward-incompatible, which is something I've
tried to avoid.

> Why did your patch remove the gettext calls on the pseudo file names?

Because other parts of the compiler compare source filenames with
<built-in>, so those strings shouldn't be subject to translation.
Also, you wouldn't want them to be handled differently if you get a
file preprocessed with another language enabled, would you?

My stance is that preprocessed files are not user-readable content,
but rather another form of source code.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> How do you know what is the current directory and what is the original
> file name?  Are you suggesting more than one string or something like
> that?  *That* would be backward-incompatible, which is something I've
> tried to avoid.

Yes, have two strings, with the second one optional and only valid at
the start of a preprocessed file.  Just an idea; I think it's cleaner.
It is backwards compatible.

> > Why did your patch remove the gettext calls on the pseudo file names?
> 
> Because other parts of the compiler compare source filenames with
> <built-in>, so those strings shouldn't be subject to translation.

I don't think they should be doing that.  Why do they feel the need
to do that?  

> Also, you wouldn't want them to be handled differently if you get a
> file preprocessed with another language enabled, would you?

No, but remember that file names appear in diagnostics.

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 24, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> Yes, have two strings, with the second one optional and only valid at
> the start of a preprocessed file.  Just an idea; I think it's cleaner.
> It is backwards compatible.

So are additional strings allowed at all?  I thought the filename was
supposed to be followed by flags only.  In fact, the CPP manual seems
to support my understanding.  If the documented format does not
support this, I don't see how it can be considered backward
compatible.

>> > Why did your patch remove the gettext calls on the pseudo file names?
>> 
>> Because other parts of the compiler compare source filenames with
>> <built-in>, so those strings shouldn't be subject to translation.

> I don't think they should be doing that.  Why do they feel the need
> to do that?  

dwarf2out.c maps that to filename 0, whereas tree.c uses "<built-in>"
as the source file name in case input_filename is NULL (should never
happen, right?).  cp/decl.c, in the debugging function
print_binding_leval, doesn't print info about declarations associated
with files named <built-in> either.

The only relevant use of <built-in> seems to be in dwarf2out.c, and
that's actually an important one.  In case you want to get built-in
macros through to debugging info, you need it to be able to detect the
proper name.  But then, I don't know what the problem would be if the
macros were marked as defined in a file named, say, <pré-definido>.
Perhaps it could instead take any names enclosed in angle brackets as
filename 0, instead?  Or just do away with this special case?  I don't
know.  Any DWARF-2 debugging info experts around?

>> Also, you wouldn't want them to be handled differently if you get a
>> file preprocessed with another language enabled, would you?

> No, but remember that file names appear in diagnostics.

Good point.  And we definitely don't want to be translating filenames
:-)

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 24, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> Alexandre Oliva wrote:-
>> > As part of cc1's preprocessed output code.
>> 
>> But how does it tell that it's time to emit the <directory> directive?

> cc1 should be doing the preprocessed output.  Currently it's in
> cppmain.c, but it should be moved to a "c-" file in the front end.
> This is all part of the work to make cpplib a proper library.
> If cc1 is doing the preprocessed output, it's easy for it to dump
> whatever it wants in there.

The problem is not just about dumping, it's also about initializing
properly when reading back.  I've just tried to move the outputting of
<directory> to cppmain.c.  It doesn't work.  When c-lex.c's
cb_file_change get to see the <directory> directive, we're already
past debug_hooks->init, where the die of the compile unit is
generated.

Barring major reworking in the rest of the compiler, it is
read_original_filename() that must find out the current directory, for
the same reason that it has to find out the main input filename in
advance.

As for adding another string to the initial pathname line, I suppose I
could do it, but how would this info get passed into cb_file_change
anyway?  It's not use to have the info in the preprocessed file if it
doesn't reach the interested parties :-)

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> So are additional strings allowed at all?  I thought the filename was
> supposed to be followed by flags only.  In fact, the CPP manual seems
> to support my understanding.  If the documented format does not
> support this, I don't see how it can be considered backward
> compatible.

Of course, but that doesn't prevent you adding it for the first line
of the original file now, does it?  And it would still be backwards
compatible.

> > I don't think they should be doing that.  Why do they feel the need
> > to do that?  
> 
> dwarf2out.c maps that to filename 0, whereas tree.c uses "<built-in>"
> as the source file name in case input_filename is NULL (should never
> happen, right?).  cp/decl.c, in the debugging function
> print_binding_leval, doesn't print info about declarations associated
> with files named <built-in> either.

I think this is old code before integrated CPP.

> The only relevant use of <built-in> seems to be in dwarf2out.c, and
> that's actually an important one.  In case you want to get built-in
> macros through to debugging info, you need it to be able to detect the
> proper name.

This is another case where I don't think we can reasonably expect
to do as well with preprocessed files as with original files.  Original
files should be easy to get right.  We can only get debug info for
macros for .i files with -dD anyway.

As far as I'm concerned, the only reason for "<built-in>" and "<command-
line>" is diagnostics; I did the work to get them right between 3.0 and
3.1 I think.

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Neil Booth]

Alexandre Oliva wrote:-

> > cc1 should be doing the preprocessed output.  Currently it's in
> > cppmain.c, but it should be moved to a "c-" file in the front end.
> > This is all part of the work to make cpplib a proper library.
> > If cc1 is doing the preprocessed output, it's easy for it to dump
> > whatever it wants in there.
> 
> The problem is not just about dumping, it's also about initializing
> properly when reading back.

I know; this was hard to fix when we went to integrated CPP (and
the reason for a lot of my toplev.c and langhooks work).

To clean up the cpplib / front end interface further, I was continuing
to work on this but I was stopped on Aug 15th with the freeze.

Note that the difficulty is caused by wanting to get debug info right
for .i files; nothing else introduces such nasty dependences on
initialization ordering.  I'm far from convinced that earlier versions
of GCC had it as correct as 3.x does.

Neil.

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 25, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> Of course, but that doesn't prevent you adding it for the first line
> of the original file now, does it?  And it would still be backwards
> compatible.

I don't see how introducing a change that was not forewarned in older
docs be regarded as backward-compatible.  Any other program that
claimed to support the old GCC preprocessed output could legally throw
the towel when seeing that.

> This is another case where I don't think we can reasonably expect
> to do as well with preprocessed files as with original files.  Original
> files should be easy to get right.  We can only get debug info for
> macros for .i files with -dD anyway.

Perhaps -ggdb should imply -dD, now that we can emit debugging info
about macros?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [distcc] using distcc to speed up gcc bootstraps

[Alexandre Oliva]

On Aug 25, 2002, Neil Booth <neil@daikokuya.co.uk> wrote:

> Note that the difficulty is caused by wanting to get debug info right
> for .i files

Precisely.  And this is more and more important as people adopt tools
such as ccache and distcc.  It's sad that all the work you did to
integrate cpp into gcc can't be used by these tools, but they offer
such amazing speed ups to developers in general that we shouldn't just
ignore them.  ON the contrary: we should make every reasonable effort
to make sure they don't produce results that might be regarded as
incorrect.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer