Slow compilation with many files

Slow compilation with many files

[Zed]

I've tried to make my c++ source files small in my current project. Compiling
my approx. 20 files takes very long time. I created a new source file that
just includes all my original ones with #include statements, and compiled
it. This method was much quicker. Compiling all files included in one takes
approx. double the time compared to that of compiling just one of the small
source files. So If I make a change to some header that is included by many
of the source files, the last described method that includes the files to
one is much faster. 

It seems there is a lot of compilation time overhead if the code is compiled
in many small parts - this is without taking linking into account. Is there
any settings or some trick that can reduce this overhead, except my ugly
inclusion method?

This is the command executed per source file, created by autoconf:

g++ -DHAVE_CONFIG_H -I. -I.. -DPACKAGE_LOCALE_DIR=\""/usr/local//locale"\"
-DPACKAGE_SRC_DIR=\""."\" -DPACKAGE_DATA_DIR=\""/usr/local/share"\"
-I/usr/include/opencv   -pthread -I/usr/include/glib-2.0
-I/usr/lib/glib-2.0/include   -I/usr/include/gtk-2.0
-I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo
-I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include
-I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/pixman-1  
-I/usr/local/include/ltilib -D_GNU_SOURCE -fpic -D_DEBUG    -g -O2 -MT
pick-vertex-manipulator.o -MD -MP -MF .deps/pick-vertex-manipulator.Tpo -c
-o pick-vertex-manipulator.o pick-vertex-manipulator.cpp

Which is essentially after removing unimportant stuff:

g++ -DHAVE_CONFIG_H -D_GNU_SOURCE -fpic -D_DEBUG -g -O2 -MT
pick-vertex-manipulator.o -MD -MP -MF .deps/pick-vertex-manipulator.Tpo -c
-o pick-vertex-manipulator.o pick-vertex-manipulator.cpp

Thank you!
-- 
View this message in context: http://www.nabble.com/Slow-compilation-with-many-files-tp16691342p16691342.html
Sent from the gcc - Help mailing list archive at Nabble.com.

Re: Slow compilation with many files

[Brian Dessent]

Zed wrote:

> I've tried to make my c++ source files small in my current project. Compiling
> my approx. 20 files takes very long time. I created a new source file that
> just includes all my original ones with #include statements, and compiled
> it. This method was much quicker. Compiling all files included in one takes
> approx. double the time compared to that of compiling just one of the small
> source files. So If I make a change to some header that is included by many
> of the source files, the last described method that includes the files to
> one is much faster.

This is essentially what the -combine switch does, except that switch
only supports C.

> It seems there is a lot of compilation time overhead if the code is compiled
> in many small parts - this is without taking linking into account. Is there
> any settings or some trick that can reduce this overhead, except my ugly
> inclusion method?

Yes, there's the startup/cleanup overhead of the compiler itself, plus
the overhead of parsing all the various headers 20 times instead of
once.  You can try using a precompiled header to reduce the cost of the
latter, see
<http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html>.  Note that
this isn't something you can just switch on, you have to think a little
bit about how to implement it, but the manual gives some good
suggestions about how to do that.

Brian

Re: Slow compilation with many files

[Per Rosengren]

Thank you! Precompiled headers is the solution.

My only problem now is how to get my buildsystem to use it. I am
currently using autoconf and automake. If I run "gcc -c myheader.hh", I
get a myheader.hh.gch file. I have tried to add the headers to
<target>SOURCES in Makefile.am, but the resulting Makefile doesn't
compile them.

I have also looked into switching to CMake, but it doesn't seem to
support precompiled headers, without adding some hack script for an
ADD_PRECOMPILED_HEADERS macro.

I use Kdevelop as my programming environment, so if there is anything
there that enables the use of precompiled headers, it would be very
convenient.

This is probably the wrong forum for discussing build systems, but maybe
you know if and how they support this feature of gcc anyway.


Per

Brian Dessent wrote:
> Zed wrote:
> 
>> I've tried to make my c++ source files small in my current project. Compiling
>> my approx. 20 files takes very long time. I created a new source file that
>> just includes all my original ones with #include statements, and compiled
>> it. This method was much quicker. Compiling all files included in one takes
>> approx. double the time compared to that of compiling just one of the small
>> source files. So If I make a change to some header that is included by many
>> of the source files, the last described method that includes the files to
>> one is much faster.
> 
> This is essentially what the -combine switch does, except that switch
> only supports C.
> 
>> It seems there is a lot of compilation time overhead if the code is compiled
>> in many small parts - this is without taking linking into account. Is there
>> any settings or some trick that can reduce this overhead, except my ugly
>> inclusion method?
> 
> Yes, there's the startup/cleanup overhead of the compiler itself, plus
> the overhead of parsing all the various headers 20 times instead of
> once.  You can try using a precompiled header to reduce the cost of the
> latter, see
> <http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html>.  Note that
> this isn't something you can just switch on, you have to think a little
> bit about how to implement it, but the manual gives some good
> suggestions about how to do that.
> 
> Brian

precompiled headers (was: Slow compilation with many files)

[Brian Dessent]

Per Rosengren wrote:

> My only problem now is how to get my buildsystem to use it. I am
> currently using autoconf and automake. If I run "gcc -c myheader.hh", I
> get a myheader.hh.gch file. I have tried to add the headers to
> <target>SOURCES in Makefile.am, but the resulting Makefile doesn't
> compile them.

Anything you write in Makefile.am is simply passed through through
verbatim to the output Makefile in the absence of any automake-specific
syntax, so just write whatever you need as standard make rules.  I'd
start with something like:

BUILT_SOURCES = myheader.hh.gch
%.hh.gch: %.hh
	$(CXX) -c $<

(The %-pattern rule syntax is GNU make specific, so if you need
portability to other makes you would probably need to either spell it
out as "myheader.hh.gch: myheader.hh" or add .hh.gch to .SUFFIXES and
write it as an old-style suffix rule.)

You could also do it at the end of configure using e.g.
AC_CONFIG_COMMANDS or AC_CONFIG_COMMANDS_POST, assuring that it's the
first thing done before compiling anything.  The problem with that is
that it won't capture any dependency info, so the .gch will never be
remade if myheader.hh or something it includes is modified.  

But honestly the above is just a random stab at how to start, you'd be
much better asking on the automake list (in CC:.)

Brian

Re: precompiled headers

[Ralf Wildenhues]

* Brian Dessent wrote on Fri, Apr 18, 2008 at 11:28:07AM CEST:
> BUILT_SOURCES = myheader.hh.gch
> %.hh.gch: %.hh
> 	$(CXX) -c $<
[...] 

The rule is a good start, but likely lacking dependency information.

> But honestly the above is just a random stab at how to start, you'd be
> much better asking on the automake list (in CC:.)

My humble musings about precompiled headers support are mostly in
<http://thread.gmane.org/gmane.comp.gcc.devel/60371>, and those
affect GCC-type precompiled headers only.

Cheers,
Ralf

Re: precompiled headers

[Per Rosengren]

The compiler flags given when compiling a source file that uses a 
precompiled header must match those used when the precompiled header was 
compiled. Because of this I altered Brian's code to:

BUILT_SOURCES = myheader.hh.gch
%.hh.gch: %.hh
	$(CXXCOMPILE) -c $<

The only difference here is that CXXCOMPILE - which is used to compile 
my sources, adds some flags to CXX.
For debugging purposes, I added

AM_CXXFLAGS = -H -Winvalid-pch

where -H prints all header files included, and -Winvalid-pch warns if a 
precompiled header was found but not used.

I think all dependencies are correct.


Thank you!
Per

Ralf Wildenhues wrote:
> * Brian Dessent wrote on Fri, Apr 18, 2008 at 11:28:07AM CEST:
>> BUILT_SOURCES = myheader.hh.gch
>> %.hh.gch: %.hh
>> 	$(CXX) -c $<
> [...] 
> 
> The rule is a good start, but likely lacking dependency information.
> 
>> But honestly the above is just a random stab at how to start, you'd be
>> much better asking on the automake list (in CC:.)
> 
> My humble musings about precompiled headers support are mostly in
> <http://thread.gmane.org/gmane.comp.gcc.devel/60371>, and those
> affect GCC-type precompiled headers only.
> 
> Cheers,
> Ralf

Re: Slow compilation with many files

[J.C. Pizarro]

Zed <perrose@kth.se> wrote:
> I've tried to make my c++ source files small in my current project. Compiling
> my approx. 20 files takes very long time. I created a new source file that
> just includes all my original ones with #include statements, and compiled
> it. This method was much quicker. Compiling all files included in one takes
> approx. double the time compared to that of compiling just one of the small
> source files. So If I make a change to some header that is included by many
> of the source files, the last described method that includes the files to
> one is much faster.
>
> It seems there is a lot of compilation time overhead if the code is compiled
> in many small parts - this is without taking linking into account. Is there
> any settings or some trick that can reduce this overhead, except my ugly
> inclusion method?
>
> This is the command executed per source file, created by autoconf:
>
> g++ -DHAVE_CONFIG_H -I. -I.. -DPACKAGE_LOCALE_DIR=\""/usr/local//locale"\"
> -DPACKAGE_SRC_DIR=\""."\" -DPACKAGE_DATA_DIR=\""/usr/local/share"\"
> -I/usr/include/opencv   -pthread -I/usr/include/glib-2.0
> -I/usr/lib/glib-2.0/include   -I/usr/include/gtk-2.0
> -I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo
> -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include
> -I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/pixman-1
> -I/usr/local/include/ltilib -D_GNU_SOURCE -fpic -D_DEBUG    -g -O2 -MT
> pick-vertex-manipulator.o -MD -MP -MF .deps/pick-vertex-manipulator.Tpo -c
> -o pick-vertex-manipulator.o pick-vertex-manipulator.cpp
>
> Which is essentially after removing unimportant stuff:
>
> g++ -DHAVE_CONFIG_H -D_GNU_SOURCE -fpic -D_DEBUG -g -O2 -MT
> pick-vertex-manipulator.o -MD -MP -MF .deps/pick-vertex-manipulator.Tpo -c
> -o pick-vertex-manipulator.o pick-vertex-manipulator.cpp
>
> Thank you!

I think that GCC lacks
http://en.wikipedia.org/wiki/Memoization

With Memoization and gcc-kernel daemon, it can compile faster bigger apps.