Apple's PFE for GCC is now available

Apple's PFE for GCC is now available

[Stan Shebs]

Last year, I posted a message describing Apple's Persistent Front End
(PFE) project (http://gcc.gnu.org/ml/gcc/2001-09/msg01210.html).  I'm
now pleased to announce that the complete source code is now both
available and portable; it runs well on GNU/Linux, and likely other
types of systems as well.

When used to implement precompiled headers, PFE offers a considerable
speedup; typically 6x for 100KLoc Mac headers, and as much as 12x on
Linux using comparable synthetic headers.  (Of course, your results
will vary considerably, depending on everything from system RAM to
source file size.)  PFE has been in production use for building
parts of Jaguar for some months now, so it's known to be reliable.

The most convenient way to get to current sources is via our CVS
repository.  It's not a true anoncvs setup, but you can register at
http://www.opensource.apple.com/cgi-bin/apslregfm to get login id,
then use :pserver:<id>@anoncvs.opensource.apple.com:/cvs/Darwin as
your CVSROOT, module "gcc3".  (More details may be found at
http://developer.apple.com/darwin/tools/cvs/read-only.html .)  If the
registration thing is too onerous, we can probably set up a tarball
somewhere.

This is an exact mirror of our working repository, and as such is not
guaranteed to work at all times, such as right after importing an FSF
GCC snapshot. :-) The tag "pfe-test-2002-07-30" identifies sources
known to work on my vanilla Red Hat 7.1 system.

The toplevel file README.Apple is a summary of building and using.
PFE precomps are simple to use:

        gcc --dump-pch foo -x c incl.h
        gcc --dump-pch foo -x c++ incl.h

creates the directory foo that holds compiler states (one for
each language desired), and

        gcc --load-pch foo file.c
        gcc --load-pch foo otherfile.cc

loads state from foo, and compiles in the usual way.  Semantically
it's identical to saying

        gcc -include incl.h file.c
        gcc -include incl.h otherfile.cc

This version of GCC includes many other features, most of them
specific to Mac OS X programming.  The one that may be of general
interest is Objective-C++, which is C++ extended to work with the
Objective-C extensions.

There is a majordomo mailing list darwin-gcc3@opensource.apple.com
that gets CVS messages and is also used for discussion; everybody is
welcome to join.

Finally, if you're wondering how this relates to the PCH branch, it's
being actively discussed.  PFE has both pluses and minuses relative to
the current PCH work, and ideally we'll be able to use both in coming
up with superior precompiled header support for FSF GCC.

Stan

Re: Apple's PFE for GCC is now available

[Neil Booth]

> Finally, if you're wondering how this relates to the PCH branch, it's
> being actively discussed.  PFE has both pluses and minuses relative to
> the current PCH work, and ideally we'll be able to use both in coming
> up with superior precompiled header support for FSF GCC.

Where is it being actively discussed?  It's a shame this stuff isn't
out in the open; I'd like to know what the plans are as I have (had)
ideas of my own.

Neil.

Re: Apple's PFE for GCC is now available

[Mike Stump]

On Tuesday, July 30, 2002, at 11:43 PM, Neil Booth wrote:

>> Finally, if you're wondering how this relates to the PCH branch, it's
>> being actively discussed.  PFE has both pluses and minuses relative to
>> the current PCH work, and ideally we'll be able to use both in coming
>> up with superior precompiled header support for FSF GCC.
>
> Where is it being actively discussed?  It's a shame this stuff isn't
> out in the open; I'd like to know what the plans are as I have (had)
> ideas of my own.

This is the right place.  This is the place where we are discussing it.

The plan is to make gcc/g+ 6x faster.  Exactly how we do that, is up 
for discussion.  We have an initial implementation that gets us there 
that we are using in our production compiler.  If it meets the various 
standards for the tree, would be nice to get it into the tree.  Once 
in, we can then refine it further.

Also, if people that are interested would grab a copy and try it out, 
say on linux or apple boxes, find/fix problems, try it out and just 
give feedback on the performance enhancement they see, say for GTK+, 
mozilla or pick your favorite monster, that would help.  Also, let us 
know if it `feels' ready for prime time.  Let us know if it breaks your 
builds...  Are there any problems with it that should be addressed 
before it goes into the tree?

Re: Apple's PFE for GCC is now available

[Daniel Berlin]

On Wednesday, July 31, 2002, at 12:04  PM, Mike Stump wrote:

> On Tuesday, July 30, 2002, at 11:43 PM, Neil Booth wrote:
>
>>> Finally, if you're wondering how this relates to the PCH branch, it's
>>> being actively discussed.  PFE has both pluses and minuses relative 
>>> to
>>> the current PCH work, and ideally we'll be able to use both in coming
>>> up with superior precompiled header support for FSF GCC.
>>
>> Where is it being actively discussed?  It's a shame this stuff isn't
>> out in the open; I'd like to know what the plans are as I have (had)
>> ideas of my own.
>
> This is the right place.  This is the place where we are discussing it.
>
> The plan is to make gcc/g+ 6x faster.  Exactly how we do that, is up 
> for discussion.  We have an initial implementation that gets us there 
> that we are using in our production compiler.  If it meets the various 
> standards for the tree, would be nice to get it into the tree.  Once 
> in, we can then refine it further.
(A note: I've been following PFE since it's existed in apple's tree, so 
I'm intimately familiar with it.)
PFE's implementation, as it stands, is somewhat invasive, with the 
GGC_ALLOC macro, etc.
While I understand many of these invasions are necessary to restore the 
state properly (like intercepting the debug hooks), there is one 
invasion that *should* be completely unnecessary if PFE was in the 
mainline.

This is the new walker functions to freeze/thaw various allocated 
structures.

These should be gengtype generated.  It would reduce the size of the 
changes needed for PFE by a large amount.

If gengtype can't do something you need it to do in order to be able to 
produce the walker code, then please, point this out.

IIRC, it was Geoff's intention to use it for generating the walkers 
necessary for PCH anyway.

>
> Also, if people that are interested would grab a copy and try it out, 
> say on linux or apple boxes, find/fix problems, try it out and just 
> give feedback on the performance enhancement they see, say for GTK+, 
> mozilla or pick your favorite monster, that would help.  Also, let us 
> know if it `feels' ready for prime time.  Let us know if it breaks 
> your builds...  Are there any problems with it that should be 
> addressed before it goes into the tree?
>
I'd really like to see the freeze/thaw walkers auto-generated before it 
goes into the tree.

Re: Apple's PFE for GCC is now available

[Stan Shebs]

Neil Booth wrote:

>>Finally, if you're wondering how this relates to the PCH branch, it's
>>being actively discussed.  PFE has both pluses and minuses relative to
>>the current PCH work, and ideally we'll be able to use both in coming
>>up with superior precompiled header support for FSF GCC.
>>
>
>Where is it being actively discussed?  It's a shame this stuff isn't
>out in the open; I'd like to know what the plans are as I have (had)
>ideas of my own.
>
"Actively" is perhaps an exaggeration; at this point it's been mostly
in-person conversation.  Now that working PFE sources are available
for people to look at and try, it's  worthwhile to move to more
specific discussion online.

So nobody feels in the dark, I'll put up my current favorite idea for
target practice :-) ; use the PFE code for allocation to make sure that
everything goes into large mmap-able blobs, use the tree-marking code
to autogen walkers for freezing/thawing if mmap can't go back to the
same place, use PFE code for managing headers and date checking since
it all exists now, and have a big free-for-all argument about what to
call the dump/load switches. 1/2 :-)

One downside of the big mmap is that it doesn't allow for incremental
loading, but incremental would be a useful fallback to keep the
mmap failure case efficient.  Our experience so far is that while
a particular system's mmap wins are predictable, different systems
are randomly different in the size and location of areas to use.

Stan

>

Re: Apple's PFE for GCC is now available

[Christoph Hellwig]

On Tue, Jul 30, 2002 at 09:47:43PM -0700, Stan Shebs wrote:
> The most convenient way to get to current sources is via our CVS
> repository.  It's not a true anoncvs setup, but you can register at
> http://www.opensource.apple.com/cgi-bin/apslregfm to get login id,
> then use :pserver:<id>@anoncvs.opensource.apple.com:/cvs/Darwin as
> your CVSROOT, module "gcc3".  (More details may be found at
> http://developer.apple.com/darwin/tools/cvs/read-only.html .)  If the
> registration thing is too onerous, we can probably set up a tarball
> somewhere.

I'd be interested in testing it for Linux kernel compiles but I don't
really want to signup first.  Providing a tarball would be very nice,
yes.

Re: Apple's PFE for GCC is now available

[Devang Patel]

On Wednesday, July 31, 2002, at 09:19  AM, Daniel Berlin wrote:

>
> On Wednesday, July 31, 2002, at 12:04  PM, Mike Stump wrote:
>
>> On Tuesday, July 30, 2002, at 11:43 PM, Neil Booth wrote:
>>
>>>> Finally, if you're wondering how this relates to the PCH branch, 
>>>> it's
>>>> being actively discussed.  PFE has both pluses and minuses relative 
>>>> to
>>>> the current PCH work, and ideally we'll be able to use both in 
>>>> coming
>>>> up with superior precompiled header support for FSF GCC.
>>>
>>> Where is it being actively discussed?  It's a shame this stuff isn't
>>> out in the open; I'd like to know what the plans are as I have (had)
>>> ideas of my own.
>>
>> This is the right place.  This is the place where we are discussing 
>> it.
>>
>> The plan is to make gcc/g+ 6x faster.  Exactly how we do that, is up 
>> for discussion.  We have an initial implementation that gets us there 
>> that we are using in our production compiler.  If it meets the 
>> various standards for the tree, would be nice to get it into the 
>> tree.  Once in, we can then refine it further.
> (A note: I've been following PFE since it's existed in apple's tree, 
> so I'm intimately familiar with it.)
> PFE's implementation, as it stands, is somewhat invasive, with the 
> GGC_ALLOC macro, etc.
> While I understand many of these invasions are necessary to restore 
> the state properly (like intercepting the debug hooks), there is one 
> invasion that *should* be completely unnecessary if PFE was in the 
> mainline.

We have taken great care (with lots of extra pain) to make sure that we 
can continue
importing from FSF while we continue to work on PFE (and do not disturb 
strict Mac OS X
schedule). Most of the macro invasion is to keep things separate so 
that with one
configure switch we can entirely disable or enable PFE in gcc3. If PFE 
is accepted
in FSF, most of it can go away.

PFE implementation code is everywhere but it is, IMO, very clean and 
separate.
For example,
- Many places we are using PFE_MALLOC macro instead of xmalloc () call.
   Which is just doing the job of diverting to actual xmalloc or PFE 
mem. manager.
- All PFE operations are controlled by #ifdef PFE. (again no need to 
keep this
   once PFE goes into FSF)
- All new routines required for PFE are always appended at the end of 
source file.
- All pfe routines use pfe_ prefix and many place it can be actually 
replaced
   and/or merged with original routine.
- We have gone through lots of pain, because otherwise we would need to
   reorganize FSF source in minor way. We refrained to do that, because 
it would
   have made our regular FSF imports very difficult.
- PFE memory manager and tree walker is kept in separate directory 
gcc/pfe.
   These two components, in some sense collide/replicate, with garbage 
collector and
other tree walkers. It would be nice, if people more familiar with 
garbage
collector and other tree walkers look into these and see what we are 
trying
to do. The freeze/throw tree walk usage is somewhat different than other
tree walks (even we had to rewrite our own tree walker to get it 
correct)
so, in my opinion, we can not just replace it with other tree walker 
unless
we update other tree walks. But I can be wrong.

In any case, more experienced eye balls can help us to identify what
we missed and what we could have done in better way. And we are ready 
to do
all necessary to have one real good precomp model in FSF gcc.

In short, I just wanted to say that some of the code clutter is not 
necessary
and it is there because it would be easier to see what PFE is trying to 
do
and ultimately make it easier to merge with FSF.

- Devang

Re: Apple's PFE for GCC is now available

[Alex Hornby]

On Wed, 2002-07-31 at 05:47, Stan Shebs wrote:
> Last year, I posted a message describing Apple's Persistent Front End
> (PFE) project (http://gcc.gnu.org/ml/gcc/2001-09/msg01210.html).  I'm
> now pleased to announce that the complete source code is now both
> available and portable; it runs well on GNU/Linux, and likely other
> types of systems as well.
> 
> When used to implement precompiled headers, PFE offers a considerable
> speedup; typically 6x for 100KLoc Mac headers, and as much as 12x on
> Linux using comparable synthetic headers.  (Of course, your results
> will vary considerably, depending on everything from system RAM to
> source file size.)  PFE has been in production use for building
> parts of Jaguar for some months now, so it's known to be reliable.
> 

Hi Stan,

As a GCC user, would I need to restructure my project's includes to
benefit from PFE?

The reason I ask is that some pch implementations I've used (e.g.
borland's) seem to require a "one big header" approach with a pch header
that includes all the other headers. This means that if you change one
header file everything ends up rebuilding, which obviously negates some
of the build speedup benefits. Of course perhaps I'm just using the
borland pch incorrectly :)

Regards,
Alex.