Precompiled headers

Precompiled headers

[Robert Vazan]

Hi,

I've got tired of lenghty compilations and decided to do something about
it. As expected, precompiled headers have been discussed several times,
some attempts were made and something gets into 3.4 release.

From various comments, I judge that it just dumps compiler state into gch
file and reloads it for next source file. It is rumoured to give six times
faster compilations. Improvement from 12 to 2 minutes is not enough for me,
not to mention that simplified header structure means 2-3 times more
recompiling. I need it under one second.

My strategy is to track dependencies down to inidividual function
declarations and structures. Then diff changes in parsed headers and
rebuild only those files that really depend on modified statements. If
there is just one typedef added without name conflict, nothing is rebuilt.

One option is to work over plain sources, but here I get into complexities
of various languages and I cannot make it work anywhere but in C. Other
option is to use one of compiler's internal structures, that abstract away
most of C++ mess into easy to trap lookups. I am not Gcc developer, but I
have weird feeling that current sources aren't doing such nice database
lookups, that I could hang on. I am not going to rewrite half the compiler,
of course.

So what would you recommend to me? Is there any small part of compiler that
"sees" all dependencies as the code is compiled? I can make dependency
tracker for C easily as external program. Are there any libs that could
help me? I know about incremental linking and constant linking, but
dependencies are going to make biggest difference right now.

Re: Precompiled headers

[Per Bothner]

Robert Vazan wrote:

> My strategy is to track dependencies down to inidividual function
> declarations and structures.

Have you looked at the compiler-server branch?

For background see:
http://www.gccsummit.org/2003/view_abstract.php?talk=28
http://per.bothner.com/papers/GccSummit03/gcc-server.pdf
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Precompiled headers

[Mike Stump]

On Sunday, November 2, 2003, at 02:20 AM, Robert Vazan wrote:
> It is rumoured to give six times faster compilations. Improvement from 
> 12 to 2 minutes is not enough for me, not to mention that simplified 
> header structure means 2-3 times more
> recompiling. I need it under one second.

12 minutes to 1 second isn't on the horizon.  If you want that this 
year, you will either have to work very hard or go someplace else.  If 
you want to work on it, the compile server is a natural fit.  The 
approach you cite is a very natural fit for it.  Currently,  if you go 
from 12 to 2, you might make 1 minute with the compile server.  On 
changes that invalidate most coarse gained dependency checks but 
preserve the fine gain checks, you should be able to get additional 
speed ups with some work.  I'd love to see this case benchmarked, I'd 
guess that an additional 4-10x (over PCH) might be possible, but even 
at 10x, that'd only be 6 seconds, and only in an odd sort of case.

I have code in there now for C++ at least that for code that doesn't 
change, can consume and spit it out very fast the second time 
(dependency check plus fprintf speeds).  It uses fine grain dependency 
checking and will redo just the parts that have changed or failed the 
dependency check.

Re: Precompiled headers

[Robert Vazan]

On Mon, 3 Nov 2003 12:14:09 -0800 Mike Stump <mrs@apple.com> wrote:

> 12 minutes to 1 second isn't on the horizon.  If you want that this 
> year, you will either have to work very hard or go someplace else.  If 
> you want to work on it, the compile server is a natural fit.

I read the pdf and looked at some code. It seems fairly easy to do what I
want, but I have no way to make sure that it works in all cases. You guys
probably know better what parts of code need changes. So I see it as
independent project.

On the other hand, I don't quite understand where does your scepticism come
from. If you invert dependency map, you can find failed dependencies in
time proportional to size of changed code. If no part of source file has to
be rebuilt, you can avoid file operations and just touch the output.

Negative dependencies can be solved by depending on overloaded symbol and
making symbol depend on list of its declarations. Similarly with other
negative dependencies. I am not sure how much does it complicate gcc's
structures.

> I have code in there now for C++ at least that for code that doesn't 
> change, can consume and spit it out very fast the second time 
> (dependency check plus fprintf speeds).  It uses fine grain dependency 
> checking and will redo just the parts that have changed or failed the 
> dependency check.

I don't see anything about coarse/fine gain checks in the pdf.

Re: Precompiled headers

[Mike Stump]

On Friday, November 7, 2003, at 05:02  AM, Robert Vazan wrote:
> On the other hand, I don't quite understand where does your scepticism 
> come
> from.

Are you talking batch build from scratch?  I was.  From what you say, 
it sounds like you're thinking about cases in which not much changes 
and we've compiled it all before.  Yes, in that case, we should be able 
to get _lots_ of speed improvements.

> I don't see anything about coarse/fine gain checks in the pdf.

The pdf is a starting point.  C++ is much advanced over the pdf.

Re: Precompiled headers

[Mike Stump]

On Friday, November 7, 2003, at 05:02  AM, Robert Vazan wrote:
> On the other hand, I don't quite understand where does your scepticism 
> come
> from.

Are you talking batch build from scratch?  I was.  From what you say, 
it sounds like you're thinking about cases in which not much changes 
and we've compiled it all before.  Yes, in that case, we should be able 
to get _lots_ of speed improvements.

> I don't see anything about coarse/fine gain checks in the pdf.

The pdf is a starting point.  C++ is much advanced over the pdf.

Re: Precompiled headers

[Robert Vazan]

On Fri, 7 Nov 2003 12:12:50 -0800 Mike Stump <mrs@apple.com> wrote:

> Are you talking batch build from scratch?  I was.  From what you say, 
> it sounds like you're thinking about cases in which not much changes 
> and we've compiled it all before.

Yes, I am concerned about modify-compile-test cycle. I don't mind length of
nightly rebuild and releases.

>  Yes, in that case, we should be able 
> to get _lots_ of speed improvements.

Do you mean that your project will be able to eliminate unnecessary
rebuilds? Can your code answer the question whether given object file will
change after rebuild without actually rebuilding it? Makefiles answer that
question wrongly in most cases and that's where the waste comes from.

If you already plan this, then I better wait for your results instead of
starting overlapping project. What do you think?

Re: Precompiled headers

[Mike Stump]

On Saturday, November 8, 2003, at 04:41 AM, Robert Vazan wrote:
> Yes, I am concerned about modify-compile-test cycle. I don't mind 
> length of
> nightly rebuild and releases.

Ah, ok.  A very important detail.

>>  Yes, in that case, we should be able
>> to get _lots_ of speed improvements.
>
> Do you mean that your project will be able to eliminate unnecessary
> rebuilds?

No, what it means is that if you rebuild, that those rebuilds will go 
faster, compile time
should be close to the amount changed + things that are dependent upon 
those things that changed.  While we don't have all the code 
implemented to do this, yet, the design so far, permits it.

> Can your code answer the question whether given object file will
> change after rebuild without actually rebuilding it? Makefiles answer 
> that
> question wrongly in most cases and that's where the waste comes from.

They change.

> If you already plan this, then I better wait for your results instead 
> of
> starting overlapping project. What do you think?

No, you don't have to wait, but use it (the compile server) as a 
starting point.  See how well it works, see what problems you find, 
report on the results you see, including the bugs.  Try your hand at 
fixing the easier ones that you find.

If you want pointers into what needs doing, see below.  The first three 
are trivial and will need doing.  0 is hard, and I am doing that now.  
3, 4, 5 and 6 need testing and maybe doing.

Unintended problems:
1   add if (server_mode) to output fragments in toplev.c
1   asm_out_file = 0 when we close it (crash on linux)
1   fix fragment end code to not ftell when asm_out_file is 0.
Laguage fidelity:
0   -D on client cmd line
1   C++: cpp macros reset/restored per fragment
       As this is done, -include is only done once, so ensure that all 
fragments are active
	 for the -include.
     Unintented smashing of data (TREE_ASM_WRITTEN) causing errors
2   Fixup overloading to be fragment aware
     #import/#pragma once files will not be included in other TUs.  See 
should_stack_file,
	reset once_only.
Asm fidelity:
3   debugging information
5   EH information
4   RTX const pool
6   Ln being ok on frag reuse
     dup .stabs "t5.cc",130,... info
     debugging info for builtins (bool:) (marked as output, undo/redo)
	IDENTIFIER_TYPENAME_P needs to be set/reset upon fragment activation 
and bewteen units.
	TREE_DEPRACATED needs to be set/reset upon fragment activation and 
bewteen units, also
	should be called
        IDENTIFIER_DEPRACATED
        cpp_hashnode->flags & NODE_POISONED
Usage:
     Streamline server starting
     Default to on (at least for testing)
Determinism:
     Return exit code after we reregister with load balancer
Scalability:
     Running out of vm
     Expiring old fragments when memory is tight
GC:
	Restore useful dynamic type checking in gt_pch_note_object.

Re: Precompiled headers

[Robert Vazan]

On Mon, 10 Nov 2003 14:34:59 -0800 Mike Stump <mrs@apple.com> wrote:

> On Saturday, November 8, 2003, at 04:41 AM, Robert Vazan wrote:
> > Can your code answer the question whether given object file will
> > change after rebuild without actually rebuilding it?
> 
> They change.

Of course they don't change, short of time stamp. Assuming gcc doesn't use
random number generator.

> No, you don't have to wait, but use it (the compile server) as a 
> starting point.  See how well it works, see what problems you find, 
> report on the results you see, including the bugs.  Try your hand at 
> fixing the easier ones that you find.

I will be happy to test it. Although you said just a while back in October
that a lot of work must be done before testing is most useful.

> If you want pointers into what needs doing, see below.

I have a broader plan that goes beyond gcc and I think I better invest time
into areas that nobody works on yet. I just wanted to avoid duplicate work,
which I have mostly accomplished. I can look into it again when compile
server project cools down.

Precompiled headers

[Noel Yap]

I found in the archive talk about precompiled headers
but this talk looks like it died out 2 1/2 years ago. 
Is the work continuing behind the scenes?

I'm thinking of the following:

1. If there were an option, say --perform-includes,
that would process only the #includes, one header file
could be generated for each sub-lattice in the include
hierarchy.

Pros:
  A. Would greatly decrease the number of opened files
within a large project.
  B. Transparent to programmers but may need changing
of makefiles (see below).
  C. Since makefiles can know the dependencies of
header files, they can have the responsibility of
updating these new header files.  I think this would
imply that another flag, say --preparsed-include=path,
would need to exist so that the build may tell cpp
where these files are.  In the end, cpp doesn't need
to know about dependencies -- it'll use the preparsed
header if and only if it's there.
  D. System headers that are preparsed in this way can
be shared by everyone.

Cons:
  A. Requires a lot of disk space.

2. If there were an option, say
--keep-macro-definitions, that does everything that
cpp already does except that it keeps #define's, a
cheap form of pre-compiled headers can be achieved.

Pros:
  A. This shouldn't be a difficult change since it
should just be a matter of not doing something.

Cons:
  A. Requires a lot of disk space.

The above two have gone by the requirements:
1. Coding conventions must not be changed.
2. Large-project builds should improve in speed.
3. Small-project builds should not be affected.

Any comments?

Thanks,
Noel

__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

Precompiled headers...

[Rob Taylor]

I seem to have lost track.. who is doing the integration of the PCH code into
the main tree? how's it going?

Thanks,
Rob Taylor

Re: precompiled headers

[Jeff Sturm]

On Thu, 5 Apr 2001, Szasz Pal wrote:
> The gcc doesn't have something like 'precompiled headers' like borland & 
> watcom c++ had?? It would spead up the compilation of a a lots of programs

PCH is a work-in-progress:

http://gcc.gnu.org/cgi-bin/htsearch?words=precompiled+headers

Jeff

Re: precompiled headers

[Neil Booth]

Szasz Pal wrote:-

> I started to develop a program using Qt and Kde, but the compilation is very 
> slow. For example for a source it takes 10 secs, although the sorce isn't big.
> I observed that it takes about 9 secs just to compile the header files.
> The gcc doesn't have something like 'precompiled headers' like borland & 
> watcom c++ had?? It would spead up the compilation of a a lots of programs

Not at present.

Neil.

precompiled headers

[Szasz Pal]

Hello!

I started to develop a program using Qt and Kde, but the compilation is very 
slow. For example for a source it takes 10 secs, although the sorce isn't big.
I observed that it takes about 9 secs just to compile the header files.
The gcc doesn't have something like 'precompiled headers' like borland & 
watcom c++ had?? It would spead up the compilation of a a lots of programs

bye
Pali



______________________________________________________________________
Do you want a free e-mail for life ? Get it at http://www.email.ro/

Precompiled Headers

[Chad Gatesman]

I was wondering if there has been any talk of adding the ability to
create pre-compiled headers to egcs?
If not, may I make such a suggestion?  What do others think of this?

Please CC me any responses, for I have trouble keeping up with this list
with all the other lists I am a part of.  Thank You.

--
Chad Gatesman
Software Engineer
Bristol Technology