Re: Suggestions for improving gcc 3.2 compilation speed?

Re: Suggestions for improving gcc 3.2 compilation speed?

[Benjamin Kosnik]

> I have a g++ compiler that is 5.6x faster than the current g++ when
> compiling large C++ code bases. That's when compared to a g++ PCH,
> without it, more like 8x to 16x faster when PCH isn't used.

Your PCH numbers are interesting to me.

I cannot get PCH to work with g++. How are you able to do this? Are you
generating individual .pch files for each header (don't see any
significant speed improvement), or are you using -include
big_globbed_header.h.pch (and if so, can you port the patches to do this
to mainline g++, because it currently doesn't work), or is there another
approach?

See this thread, and follow-ups.
http://gcc.gnu.org/ml/libstdc++/2003-03/msg00025.html

curious, 
benjamin

Re: Suggestions for improving gcc 3.2 compilation speed?

[Karel Gardas]

On Fri, 7 Mar 2003, Benjamin Kosnik wrote:

> > I have a g++ compiler that is 5.6x faster than the current g++ when
> > compiling large C++ code bases. That's when compared to a g++ PCH,
> > without it, more like 8x to 16x faster when PCH isn't used.
>
> Your PCH numbers are interesting to me.
>
> I cannot get PCH to work with g++. How are you able to do this? Are you
> generating individual .pch files for each header (don't see any
> significant speed improvement), or are you using -include
> big_globbed_header.h.pch (and if so, can you port the patches to do this
> to mainline g++, because it currently doesn't work), or is there another
> approach?
>

I'm also interested since I still see while using PCH (at least on 2 weeks
old main-trunk sources) already defined symbol assembler error messages.
Problem reported: http://gcc.gnu.org/ml/gcc/2003-01/msg00419.html

Thanks,

Karel
--
Karel Gardas                  kgardas@objectsecurity.com
ObjectSecurity Ltd.           http://www.objectsecurity.com

Re: Suggestions for improving gcc 3.2 compilation speed?

[Geoff Keating]

Karel Gardas <kgardas@objectsecurity.com> writes:

> On Fri, 7 Mar 2003, Benjamin Kosnik wrote:
> 
> > > I have a g++ compiler that is 5.6x faster than the current g++ when
> > > compiling large C++ code bases. That's when compared to a g++ PCH,
> > > without it, more like 8x to 16x faster when PCH isn't used.
> >
> > Your PCH numbers are interesting to me.
> >
> > I cannot get PCH to work with g++. How are you able to do this? Are you
> > generating individual .pch files for each header (don't see any
> > significant speed improvement), or are you using -include
> > big_globbed_header.h.pch (and if so, can you port the patches to do this
> > to mainline g++, because it currently doesn't work), or is there another
> > approach?
> >
> 
> I'm also interested since I still see while using PCH (at least on 2 weeks
> old main-trunk sources) already defined symbol assembler error messages.
> Problem reported: http://gcc.gnu.org/ml/gcc/2003-01/msg00419.html

Darwin uses stabs at the moment, so Mike won't see the DWARF problems
you're seeing.

-- 
- Geoffrey Keating <geoffk@geoffk.org>

Re: Suggestions for improving gcc 3.2 compilation speed?

[Mike Stump]

On Friday, March 7, 2003, at 11:12 AM, Benjamin Kosnik wrote:
>> I have a g++ compiler that is 5.6x faster than the current g++ when
>> compiling large C++ code bases. That's when compared to a g++ PCH,
>> without it, more like 8x to 16x faster when PCH isn't used.
>
> Your PCH numbers are interesting to me.
>
> I cannot get PCH to work with g++. How are you able to do this? Are you
> generating individual .pch files for each header (don't see any
> significant speed improvement), or are you using -include
> big_globbed_header.h.pch (and if so, can you port the patches to do 
> this
> to mainline g++, because it currently doesn't work), or is there 
> another
> approach?

We use -include.  We generate one large application .gch file that 
include tons of things in it.

bash-2.05a$ wc MillenniumFE.ii
   152201  333117 4298858 MillenniumFE.ii

That is the .ii file of the file that is -included.  This generates a 
30-40 MB .gch file.  The application has 279 translation units in it.

I don't know offhand why it works for us and not you.  Random guess, I 
thought I saw something about TAKES_ARG go by recently in another 
context...  Found it, in cppspec.c Devang added 
DEFFAULT_WORD_TAKES_ARG, is this related to why -include doesn't work?

The best win I can think of for g++ would be to create all, and include 
all the headers one can, and change all headers to inlcude all, and 
then precompile all, and link it around under all the various names 
that can be used and fix the can't do more than abs names problem in 
PCH.

This doesn't rely upon -include, works out of the box for all the std 
include files, maximal win, easy to use, just #include one of the 
normal C++ headers before anything else in the .cc files.

Re: Suggestions for improving gcc 3.2 compilation speed?

[Devang Patel]

On Friday, March 7, 2003, at 03:24 PM, Mike Stump wrote:

> I don't know offhand why it works for us and not you.  Random guess, I 
> thought I saw something about TAKES_ARG go by recently in another 
> context...  Found it, in cppspec.c Devang added 
> DEFFAULT_WORD_TAKES_ARG, is this related to why -include doesn't work?

-include does not work with g++. (PR 9394). I have the fix. I will post 
patch when bootstrap and regression test is complete.

-Devang

Re: Suggestions for improving gcc 3.2 compilation speed?

[Benjamin Kosnik]

>I don't know offhand why it works for us and not you.  Random guess, I 
>thought I saw something about TAKES_ARG go by recently in another 
>context...  Found it, in cppspec.c Devang added 
>DEFFAULT_WORD_TAKES_ARG, is this related to why -include doesn't work?

c++/9394 c++ -include doesn't work

says:

Copy the DEFAULT_WORD_SWITCH_TAKES_ARG from gcc.c into gcc/cp/g++spec.c
and use it instead.

-benjamin

Re: Suggestions for improving gcc 3.2 compilation speed?

[Geoff Keating]

Mike Stump <mrs@apple.com> writes:

> On Friday, March 7, 2003, at 11:12 AM, Benjamin Kosnik wrote:
> >> I have a g++ compiler that is 5.6x faster than the current g++ when
> >> compiling large C++ code bases. That's when compared to a g++ PCH,
> >> without it, more like 8x to 16x faster when PCH isn't used.
> >
> > Your PCH numbers are interesting to me.
> >
> > I cannot get PCH to work with g++. How are you able to do this? Are you
> > generating individual .pch files for each header (don't see any
> > significant speed improvement), or are you using -include
> > big_globbed_header.h.pch (and if so, can you port the patches to do
> > this
> > to mainline g++, because it currently doesn't work), or is there
> > another
> > approach?
> 
> We use -include.  We generate one large application .gch file that
> include tons of things in it.
> 
> bash-2.05a$ wc MillenniumFE.ii
>    152201  333117 4298858 MillenniumFE.ii
> 
> That is the .ii file of the file that is -included.  This generates a
> 30-40 MB .gch file.  The application has 279 translation units in it.
> 
> I don't know offhand why it works for us and not you.  Random guess, I
> thought I saw something about TAKES_ARG go by recently in another
> context...  Found it, in cppspec.c Devang added
> DEFFAULT_WORD_TAKES_ARG, is this related to why -include doesn't work?

It could be that problem; we don't hit it because the prefix headers
in much of our source aren't named like *.h and the problem only
occurs if you do '-include xxx.h'.

-- 
- Geoffrey Keating <geoffk@geoffk.org>

Re: Suggestions for improving gcc 3.2 compilation speed?

[Benjamin Kosnik]

>The best win I can think of for g++ would be to create all, and include 
>all the headers one can, and change all headers to inlcude all, and 
>then precompile all, and link it around under all the various names 
>that can be used and fix the can't do more than abs names problem in 
>PCH.
>
>This doesn't rely upon -include, works out of the box for all the std 
>include files, maximal win, easy to use, just #include one of the 
>normal C++ headers before anything else in the .cc files.

Well....

With Devang's patch I can do this, only I used std++.h.pch as "include
all." I'm not quite sure what you mean by the "link it around under all the
various names that can be used and fix the..." bits.

However, still.

baseline cvs, no pch
1031.380u 68.120s 23:47.94 76.9%        0+0k 0+0io 1687168pf+0w

cvs with stamp-std-precompile,  C++ includes as individual pch files
1005.580u 67.070s 23:43.49 75.3%        0+0k 0+0io 1678954pf+0w

-include stdc++.h, stdc++.h.pch generated
1563.800u 103.100s 34:36.41 80.2%       0+0k 0+0io 1784752pf+0w

#include stdc++.h, stdc++.h.pch generated
1017.940u 66.770s 24:34.06 73.5%        0+0k 0+0io 1684457pf+0w

When I run the compiler with "-v -H" I get

#include "..." search starts here:
#include <...> search starts here:
 /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include
 /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/i686-pc-linux-gnu
 /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/backward
 /mnt/hd/src/gcc/libstdc++-v3/libsupc++
 /mnt/hd/src/gcc/libstdc++-v3/testsuite
 /mnt/hd/bld/gcc/gcc/include
 /usr/local/include
 /mnt/hd/bld/H-x86-gcc/include
 /usr/include
End of search list.
GNU C++ version 3.4 20030308 (experimental) (i686-pc-linux-gnu)
        compiled by GNU C version 3.4 20030308 (experimental).
GGC heuristics: --param ggc-min-expand=64 --param ggc-min-heapsize=64154

. /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/string
.. /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/stdc++.h
.. /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/i686-pc-linux-gnu/bits/c++config.h

etc, etc, etc. Tell me if I'm doing something wrong, but I thought this
was how it was supposed to work...

This is with

<bkoz@belmont> /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include
%ll stdc++.h.pch
-rw-rw-r--    1 bkoz     bkoz          13M Mar  8 01:43 stdc++.h.pch


Where stdc++.h includes all the C++ standard includes.

Is the .pch just not being used?

-benjamin

Re: Suggestions for improving gcc 3.2 compilation speed?

[Geoff Keating]

Benjamin Kosnik <bkoz@redhat.com> writes:

> >The best win I can think of for g++ would be to create all, and include 
> >all the headers one can, and change all headers to inlcude all, and 
> >then precompile all, and link it around under all the various names 
> >that can be used and fix the can't do more than abs names problem in 
> >PCH.
> >
> >This doesn't rely upon -include, works out of the box for all the std 
> >include files, maximal win, easy to use, just #include one of the 
> >normal C++ headers before anything else in the .cc files.
> 
> Well....
> 
> With Devang's patch I can do this, only I used std++.h.pch as "include
> all." I'm not quite sure what you mean by the "link it around under all the
> various names that can be used and fix the..." bits.
> 
> However, still.
> 
> baseline cvs, no pch
> 1031.380u 68.120s 23:47.94 76.9%        0+0k 0+0io 1687168pf+0w
> 
> cvs with stamp-std-precompile,  C++ includes as individual pch files
> 1005.580u 67.070s 23:43.49 75.3%        0+0k 0+0io 1678954pf+0w
> 
> -include stdc++.h, stdc++.h.pch generated
> 1563.800u 103.100s 34:36.41 80.2%       0+0k 0+0io 1784752pf+0w
> 
> #include stdc++.h, stdc++.h.pch generated
> 1017.940u 66.770s 24:34.06 73.5%        0+0k 0+0io 1684457pf+0w
> 
> When I run the compiler with "-v -H" I get
> 
> #include "..." search starts here:
> #include <...> search starts here:
>  /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include
>  /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/i686-pc-linux-gnu
>  /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/backward
>  /mnt/hd/src/gcc/libstdc++-v3/libsupc++
>  /mnt/hd/src/gcc/libstdc++-v3/testsuite
>  /mnt/hd/bld/gcc/gcc/include
>  /usr/local/include
>  /mnt/hd/bld/H-x86-gcc/include
>  /usr/include
> End of search list.
> GNU C++ version 3.4 20030308 (experimental) (i686-pc-linux-gnu)
>         compiled by GNU C version 3.4 20030308 (experimental).
> GGC heuristics: --param ggc-min-expand=64 --param ggc-min-heapsize=64154
> 
> . /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/string
> .. /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/stdc++.h
> .. /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/i686-pc-linux-gnu/bits/c++config.h
> 
> etc, etc, etc. Tell me if I'm doing something wrong, but I thought this
> was how it was supposed to work...
> 
> This is with
> 
> <bkoz@belmont> /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include
> %ll stdc++.h.pch
> -rw-rw-r--    1 bkoz     bkoz          13M Mar  8 01:43 stdc++.h.pch
> 
> 
> Where stdc++.h includes all the C++ standard includes.
> 
> Is the .pch just not being used?

It's possible.  

You might want to check whether you're using a compiler that uses .pch
or .gch as the PCH extension.

Also, consider using the -Winvalid-pch flag.
-- 
- Geoffrey Keating <geoffk@geoffk.org>

Re: Suggestions for improving gcc 3.2 compilation speed?

[Benjamin Kosnik]

>> Is the .pch just not being used?
>
>It's possible. 

I'm hoping that it's not being used, just from the timings.

I was hoping for a bit more info than this if possible. Like, should the
.pch file show up in -v -H? How is this debugged?

>You might want to check whether you're using a compiler that uses .pch
>or .gch as the PCH extension.

Apparently, mainline g++ on linux uses .pch by default. I tried to just
generate a pch as you've previously instructed and the output file has
.pch, not .gch. 

Perhaps you could try this yourself?

>Also, consider using the -Winvalid-pch flag.

Ummmm, yeah. Done and done. 

On mainline gcc, you can generate a precompiled C++ include for
libstdc++-v3 with relatively small effort. To recap:

cd bld/target/libstdc++-v3/include
make stamp-std-precompile

Also, at the suggestion of Mike Stump, I re-did the C++ includes to
first include stdc++.h. Then, I generated a stdc++.h.pch. That approach,
which is only a stylistic variation, can be found here:

http://gcc.gnu.org/ml/gcc-patches/2003-03/msg00694.html

I'm not quite sure what to do now.

-benjamin

Re: Suggestions for improving gcc 3.2 compilation speed?

[Geoff Keating]

> Date: Sat, 8 Mar 2003 15:12:57 -0600
> From: Benjamin Kosnik <bkoz@redhat.com>
> Cc: gcc@gcc.gnu.org

> >> Is the .pch just not being used?
> >
> >It's possible. 
> 
> I'm hoping that it's not being used, just from the timings.
> 
> I was hoping for a bit more info than this if possible. Like, should the
> .pch file show up in -v -H? How is this debugged?

It doesn't seem to show up in -H, not sure why (or what the correct
behaviour might be).  I usually debug using -MD.

> >You might want to check whether you're using a compiler that uses .pch
> >or .gch as the PCH extension.
> 
> Apparently, mainline g++ on linux uses .pch by default. I tried to just
> generate a pch as you've previously instructed and the output file has
> .pch, not .gch. 

Aargh.  That's a bug.  It creates with .pch, but is looking for .gch.
I'm testing the one-character fix now.

> Perhaps you could try this yourself?

I will.

-- 
- Geoffrey Keating <geoffk@geoffk.org>

Re: Suggestions for improving gcc 3.2 compilation speed?

[Benjamin Kosnik]

>It doesn't seem to show up in -H, not sure why (or what the correct
>behaviour might be).  I usually debug using -MD.

Hmmm. 

It looks to me like -MD and -H output is identical. 

In terms of default behavior, I'd hope for the following:

- no .gch file is generated if it is incorrect (The present behavior,
where a flawed .gch file is emitted still baffles me)

- -H to show that the precompiled header file is being used. Perhaps
something like:

insert.o: 21_strings/insert.cc \
  /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/string \
  /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/stdc++.h \
  (using) /mnt/hd/bld/gcc/i686-pc-linux-gnu/libstdc++-v3/include/stdc++.h.pch \
...

The current behavior, on both of these points, is kind of confusing.

>Aargh.  That's a bug.  It creates with .pch, but is looking for .gch.
>I'm testing the one-character fix now.
>
>> Perhaps you could try this yourself?
>
>I will.

I've tried mainline after your patch (thanks) and still see don't see
any improvement. Perhaps it's something else?

-benjamin

Re: Suggestions for improving gcc 3.2 compilation speed?

[Mike Stump]

On Saturday, March 8, 2003, at 12:20 AM, Benjamin Kosnik wrote:
> I'm not quite sure what you mean by the "link it around under all the
> various names that can be used and fix the..." bits.

old .h contents:

map:
[ contents of map ]

list:
[ contents of list ]

new scheme:

map:
#include <all>
list:
#include <all>

all:
#include <old_map>
#include <old_list>

old_map:
[ contents of map ]

old_list:
[ contnets of list ]

and then precompile all, then ln all.gch map.gch; ln all.gch list.gch.  
Total disk space, sizeof(all) + just a little overhead * number of 
files in the C++ standard.  Benefit can be made to kick in for any 
source files that are like:

#include <all>
[ ... ]

or
#include <map>

or
#include <list>

and the benefits extend through to all standard C++ headers, for 
example:

#include <list>
#include "myproject"
#include <map>

has cost of mmap (all) + groking (myproject), that's it.  Currently, it 
has cost groking (list) + groking (myproject) + groking (map).

> However, still.
>
> baseline cvs, no pch
> 1031.380u 68.120s 23:47.94 76.9%        0+0k 0+0io 1687168pf+0w
>
> cvs with stamp-std-precompile,  C++ includes as individual pch files
> 1005.580u 67.070s 23:43.49 75.3%        0+0k 0+0io 1678954pf+0w
>
> -include stdc++.h, stdc++.h.pch generated
> 1563.800u 103.100s 34:36.41 80.2%       0+0k 0+0io 1784752pf+0w

I suspect this isn't using PCH!  Only absolute names work currently?!

try -include `pwd`/stdc++.h or whatever the dir is.

> #include stdc++.h, stdc++.h.pch generated
> 1017.940u 66.770s 24:34.06 73.5%        0+0k 0+0io 1684457pf+0w

This looks better?  Not sure, as it is only 1-2 seconds faster than the 
first.

I think -v should say if a pch file is used, otherwise, it is hard to 
tell when the mechanism kicks in, well, unless you have a huge .h file 
and you know exactly how long it takes to parse, and mmap.  Wanna put a 
fprintf (stderr, "PCH file %s being used\n", name); in cpp_read_state?  
I think that will make it easy for the user to understand when PCH is 
used, and more importantly, to spot when it isn't being used.

pch messages [was: Re: Suggestions for improving gcc 3.2 compilation speed?]

[Karel Gardas]

On Mon, 10 Mar 2003, Mike Stump wrote:

> I think -v should say if a pch file is used, otherwise, it is hard to
> tell when the mechanism kicks in, well, unless you have a huge .h file
> and you know exactly how long it takes to parse, and mmap.  Wanna put a
> fprintf (stderr, "PCH file %s being used\n", name); in cpp_read_state?
> I think that will make it easy for the user to understand when PCH is
> used, and more importantly, to spot when it isn't being used.

JFYI: como prints such info too. E.g.

como  -I../include --pch --pch_dir /home/karel/arch/sl3/sl3/./pch
-DFAST_PCH -g  -a --long_long --implicit_extern_c_type_conversion
--diag_suppress=997     -c poa_base.cc -o poa_base.o

"poa_base.cc": using precompiled header file "/home/karel/arch/sl3/sl3/./pch/current.pch"
como  -I../include --pch --pch_dir /home/karel/arch/sl3/sl3/./pch
-DFAST_PCH -g  -a --long_long --implicit_extern_c_type_conversion
--diag_suppress=997     -c poa_impl.cc -o poa_impl.o

"poa_impl.cc": using precompiled header file "/home/karel/arch/sl3/sl3/./pch/os-unix.pch"
como  -I../include --pch --pch_dir /home/karel/arch/sl3/sl3/./pch
-DFAST_PCH -g  -a --long_long --implicit_extern_c_type_conversion
--diag_suppress=997     -c dynany.cc -o dynany.o

"dynany.cc": using precompiled header file "/home/karel/arch/sl3/sl3/./pch/current.pch"


Cheers,

Karel
--
Karel Gardas                  kgardas@objectsecurity.com
ObjectSecurity Ltd.           http://www.objectsecurity.com

Re: pch messages [was: Re: Suggestions for improving gcc 3.2 compilation speed?]

[B. Kosnik]

>"poa_impl.cc": using precompiled header file "/home/karel/arch/sl3/sl3/./pch/os-unix.pch"

I'd like something like this too!

>  Wanna put a 
>fprintf (stderr, "PCH file %s being used\n", name); in cpp_read_state?  
>I think that will make it easy for the user to understand when PCH is 
>used, and more importantly, to spot when it isn't being used.

Yes, but I think precompiled file is better than PCH. Do you think this
is the right place for it?

-benjamin

Re: pch messages [was: Re: Suggestions for improving gcc 3.2 compilation speed?]

[Mike Stump]

On Monday, March 10, 2003, at 04:40 PM, B. Kosnik wrote:
> Yes, but I think precompiled file is better than PCH.

Agreed.

> Do you think this is the right place for it?

I don't see that it is terribly wrong.  Geoff may have a better 
suggestion.

Re: pch messages [was: Re: Suggestions for improving gcc 3.2 compilation speed?]

[Geoff Keating]

Mike Stump <mrs@apple.com> writes:

> On Monday, March 10, 2003, at 04:40 PM, B. Kosnik wrote:
> > Yes, but I think precompiled file is better than PCH.
> 
> Agreed.
> 
> > Do you think this is the right place for it?
> 
> I don't see that it is terribly wrong.  Geoff may have a better
> suggestion.

Someone (maybe even bkoz) suggested that -H should provide this
information; that certainly sounds like the right place to have
debugging info for header-inclusion problems.

-- 
- Geoffrey Keating <geoffk@geoffk.org>

Re: pch messages

[Zack Weinberg]

Geoff Keating <geoffk@geoffk.org> writes:

> Someone (maybe even bkoz) suggested that -H should provide this
> information; that certainly sounds like the right place to have
> debugging info for header-inclusion problems.

Agree.  Patches are welcome.  (If we do this, we should make -H more
prominently documented.)

zw