GCC IMA & general future proposal

GCC IMA & general future proposal

[Geoff Keating]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

I've put a quick draft of my idea of what the future of GCC should look 
like at

<http://www.geoffk.org/gcc/future/gccfuture.html>

It's short, but has a complicated diagram.  It is intended for 
discussion at the meeting in a few hours.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2410 bytes --]

Re: GCC IMA & general future proposal

[Kenneth Zadeck]

There are significant advantages to do doing a fair amount of 
optimization while you still have only a single file before you dump the 
il to the object file.

1) The techniques that one would use for a single file are different 
than those that one can consider over an entire program.  There are 
significant power vs scaling tradeoffs in algorithm selection.  For 
instance at the single file level, almost all of the algorithms one 
would use are flow sensitive.  At the link level almost all of the 
techniques will have to be flow insensitive.

2) The more of the computation time you are able to push to the earlier 
compiler, the better the user experience will be, especially in the late 
debugging and change cycle.  Anytime anything changes, you are starting 
the link level from scratch, in a development only one module changes, 
having pushed most of the heavy lifting to the earlier compile will be a 
big win.  While some of the compiler will surely be common between the 
link level and the front level, you have to assume that there are going 
to be a lot of differences. 

I would also point out that a significant number of people are not going 
to be able to use the full power of the link level stuff, so putting all 
smarts there may not be the best plan.  Programs that require dynamic 
libraries or that dynamicly load modules (like operating systems) may 
not work with this without significant engineering on their build 
scripts to describe where the boundaries are and what can pass over 
them.  The structure stuff that is being done at Apple is based on a 
completely closed world assumption that the only library is the standard 
c library.  Fine for benchmarks and the compiler, but may not be so good 
for a big gooey applications.

I should point out that the intel compiler is a bad example to base 
anything on.  It is simply a marketing tool to sell more processors and 
as such is designed to make benchmarks run particularly well on intel 
hardware.  Anything else is not so good.  Remember that this is the 
compiler that generates delibertly bad code to be used when you run on 
an amd chip.
It detects the cpu model and refuses to use any vector instructions even 
though it knows the amd chips have them.  As such, they have guaranteed 
that no one will use the compiler for anything except benchmarking. The 
ibm compiler is a better model.  This is the compiler that is used to 
develop all ibm applications.

Kenny

Geoff Keating wrote:

> I've put a quick draft of my idea of what the future of GCC should 
> look like at
>
> <http://www.geoffk.org/gcc/future/gccfuture.html>
>
> It's short, but has a complicated diagram.  It is intended for 
> discussion at the meeting in a few hours.

Re: GCC IMA & general future proposal

[Joseph S. Myers]

On Wed, 27 Oct 2004, Geoff Keating wrote:

> I've put a quick draft of my idea of what the future of GCC should look like
> at
> 
> <http://www.geoffk.org/gcc/future/gccfuture.html>

It is still useful in such a framework for front ends to be able to handle 
multiple translation units at once as C does, and pass off GENERIC for all 
of them to the next level at once, so that various cross-translation-unit 
consistency checks can be made which depend on language-specific rules.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: GCC IMA & general future proposal

[Jan Hubicka]

> On Wed, 27 Oct 2004, Geoff Keating wrote:
> 
> > I've put a quick draft of my idea of what the future of GCC should look like
> > at
> > 
> > <http://www.geoffk.org/gcc/future/gccfuture.html>
> 
> It is still useful in such a framework for front ends to be able to handle 
> multiple translation units at once as C does, and pass off GENERIC for all 
> of them to the next level at once, so that various cross-translation-unit 
> consistency checks can be made which depend on language-specific rules.

Also for C++ you don't need to end up with duplicated template bodies
everywhere. I think that KDE use the trick of including multiple C++
files together to reduce this overhead so they might give us some
statistics...

I will look into the draft tonight

Honza
> 
> -- 
> Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
>     jsm@polyomino.org.uk (personal mail)
>     joseph@codesourcery.com (CodeSourcery mail)
>     jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: GCC IMA & general future proposal

[Richard Guenther]

On Wed, 27 Oct 2004 01:52:10 -0700, Geoff Keating <geoffk@geoffk.org> wrote:
> I've put a quick draft of my idea of what the future of GCC should look
> like at
> 
> <http://www.geoffk.org/gcc/future/gccfuture.html>
> 
> It's short, but has a complicated diagram.  It is intended for
> discussion at the meeting in a few hours.

One thing to add would be unit testing, i.e. to be able to feed
specific input to parts of the compiler (like optimization passes),
dump the output and check the outcome.

Like one could (in theory) add GENERIC, GIMPLE and RTL front- and
back-ends to the
current GCC infrastructure.  Of course these IL are all not complete,
but one could work
around this.

Richard.

Re: GCC IMA & general future proposal

[Frank Ch. Eigler]

Geoff Keating <geoffk@geoffk.org> writes:

> I've put a quick draft of my idea of what the future of GCC should
> look like at <http://www.geoffk.org/gcc/future/gccfuture.html>
> [...]

Is it tantamount to adopting LLVM?

- FChE

Re: GCC IMA & general future proposal

[Scott Robert Ladd]

Kenneth Zadeck wrote:
> I should point out that the intel compiler is a bad example to base 
> anything on.  It is simply a marketing tool to sell more processors and 
> as such is designed to make benchmarks run particularly well on intel 
> hardware.  Anything else is not so good.  Remember that this is the 
> compiler that generates delibertly bad code to be used when you run on 
> an amd chip.

Is Intel's compiler relevant to GCC? Yes, particularly in the sense that 
without competition, software tends to stagnate. We all demand 
competition for Microsoft; it seems only fair that the GCC community set 
a calm and intelligent example of competition.

Certainly Intel's compilers are marketing tools -- but they also produce 
code that is faster than GCC for Real World(tm) applications, not just 
benchmarks. The BBC's Dirac video codec, for example, runs 10-15% faster 
when compiled with Intel's compiler.

I love GCC, which is why I'm working to make it better. Intel's compiler 
produces valid, working programs that is in many cases faster than GCC; 
it's a fact, whether we like it or not.

> It detects the cpu model and refuses to use any vector instructions even 
> though it knows the amd chips have them.  As such, they have guaranteed 
> that no one will use the compiler for anything except benchmarking.

Incorrect. I know many people who use Intel's compiler for production 
code, especially in the high-performance and Fortran communities.

Yes, I'm frustrated that Intel's EM64T compiler won't generate code that 
runs on my AMD Opterons. However, given the realities of the 
marketplace, I can't get angry at Intel for not supporting their 
competitor's hardware.

GCC's greatest strengths are its portability and openness. That does not 
mean we should not generate the finest possible code on individual 
platforms.

> The 
> ibm compiler is a better model.  This is the compiler that is used to 
> develop all ibm applications.

You may be correct. But most people do not have access to the IBM 
compiler, since it is not (to the best of my knowledge) available for 
non-commercial use. Anyone can download Intel's compilers and explore 
them, which is why it gets more consideration.

-- 
Scott Robert Ladd
site: http://www.coyotegulch.com
blog: http://chaoticcoyote.blogspot.com

Re: GCC IMA & general future proposal

[Scott Robert Ladd]

Geoff Keating wrote:
> I've put a quick draft of my idea of what the future of GCC should look 
> like at
> 
> <http://www.geoffk.org/gcc/future/gccfuture.html>
> 
> It's short, but has a complicated diagram.  It is intended for 
> discussion at the meeting in a few hours.

Fascinating. Is anyone making a transcript of this meeting?

..Scott

Re: GCC IMA & general future proposal

[Daniel Berlin]

On Oct 27, 2004, at 10:54 AM, Scott Robert Ladd wrote:

> Geoff Keating wrote:
>> I've put a quick draft of my idea of what the future of GCC should 
>> look like at
>> <http://www.geoffk.org/gcc/future/gccfuture.html>
>> It's short, but has a complicated diagram.  It is intended for 
>> discussion at the meeting in a few hours.
>
> Fascinating. Is anyone making a transcript of this meeting?

I am.
I'm also working on marking up the transcript of the last meeting so i 
can post it.
I'm just a bit slow at it :)

>
> ..Scott

Re: GCC IMA & general future proposal

[David Edelsohn]

>>>>> Scott Robert Ladd writes:

Scott> Kenneth Zadeck wrote:
>> I should point out that the intel compiler is a bad example to base 
>> anything on.  It is simply a marketing tool to sell more processors and 
>> as such is designed to make benchmarks run particularly well on intel 
>> hardware.  Anything else is not so good.  Remember that this is the 
>> compiler that generates delibertly bad code to be used when you run on 
>> an amd chip.

Scott> Is Intel's compiler relevant to GCC? Yes, particularly in the sense that 
Scott> without competition, software tends to stagnate. We all demand 
Scott> competition for Microsoft; it seems only fair that the GCC community set 
Scott> a calm and intelligent example of competition.

Scott> Certainly Intel's compilers are marketing tools -- but they also produce 
Scott> code that is faster than GCC for Real World(tm) applications, not just 
Scott> benchmarks. The BBC's Dirac video codec, for example, runs 10-15% faster 
Scott> when compiled with Intel's compiler.

	That is not the point of Ken's comment.  The issue is not whether
ICC produces faster, production code, but Intel's intended market and how
they expect customers to use ICC.

	You could counter Ken's comments by saying that Intel and ICC will
train compiler customers to expect different behavior from the compiler
when building an application (moving most of the compilation time to the
link stage).  This is not a question about performance benefit from and
importance of IMA, which will produce better code using either method.

	It would help if we could stay focussed on the issue about the IMA
design being questioned: how much work to do during the initial
compilation and how much to do at link time.

	Ken has made some very good and important technical and customer
experience points about doing more work during the initial compilation.  I
am firmly in that camp.  There has not been much disagreement about that
in the replies that I have seen.

>> It detects the cpu model and refuses to use any vector instructions even 
>> though it knows the amd chips have them.  As such, they have guaranteed 
>> that no one will use the compiler for anything except benchmarking.

Scott> Incorrect. I know many people who use Intel's compiler for production 
Scott> code, especially in the high-performance and Fortran communities.

Scott> Yes, I'm frustrated that Intel's EM64T compiler won't generate code that 
Scott> runs on my AMD Opterons. However, given the realities of the 
Scott> marketplace, I can't get angry at Intel for not supporting their 
Scott> competitor's hardware.

Scott> GCC's greatest strengths are its portability and openness. That does not 
Scott> mean we should not generate the finest possible code on individual 
Scott> platforms.

	Again, that is not the point of Ken's comment.  Intel isn't the
only vendor of x86 processors.  Because of ICC's behavior, ISVs and
programmers who distribute code for generic x86 installations cannot and
will not use ICC.  ICC is fine if one only has Intel-based systems and
wants to compile one's own applications or one wants to compile
applications only for Intel-based systems.

	The behavior limits ICC's usefulness as a general x86 compiler
and, thereby, limits it's usefulness for comparison.  It does not make ICC
irrelevant and Ken never implied that, but it does decrease ICC's market
and, therefore, it's importance for comparison.

	ICC is targeted at a very specific, niche market of Intel-only
users that may be less representative of the compile-time behavior that
most customer want and would use.

>> The 
>> ibm compiler is a better model.  This is the compiler that is used to 
>> develop all ibm applications.

Scott> You may be correct. But most people do not have access to the IBM 
Scott> compiler, since it is not (to the best of my knowledge) available for 
Scott> non-commercial use. Anyone can download Intel's compilers and explore 
Scott> them, which is why it gets more consideration.

	It depends what customer base you are trying to target.  ICC may
have more people playing with it, but that does not mean that it has
greater use among customers who want and need these types of high-end IMA
features.

	Also, IBM compilers are available for free trial.

David

Re: GCC IMA & general future proposal

[Scott Robert Ladd]

I'll keep this short, then let you have the final word if you want it.

David Edelsohn wrote:
>>> they have guaranteed that no one will use the compiler for 
>>> anything except benchmarking.
> 
> Scott> Incorrect. I know many people who use Intel's compiler for 
> production code, especially in the high-performance and Fortran
> communities.
> 
> Again, that is not the point of Ken's comment.

Ken could have made his point without all the anti-Intel rhetoric.
Geoff's proposal doesn't even mention Intel by name, so I wonder why Ken
felt it necessary to use this as a platform for assailing Intel?

As quoted above, he stated that "no one will use the [Intel] compiler
for anything other than benchmarking." That is demonstrably *not* true.
We don't like it when people say false things about GCC...

As it is, Geoff's proposal makes sense, and I'll be interested to see
what the outcome of the meeting is.

> Also, IBM compilers are available for free trial.

A 60-day trial, which is better than some, but not as good as Intel's
non-commercial license (and certainly both are worse, in license terms,
than free software like GCC!)

I agree that Intel is using their compiler as a marketing tool. Such is
*not* incompatible with their compiler also being quite useful in
circles where IBM and GCC are not viable options.

The best thing for GCC is to be viable in as many circles as is practical.

..Scott

Re: GCC IMA & general future proposal

[Giovanni Bajo]

Jan Hubicka wrote:

>> It is still useful in such a framework for front ends to be able to
>> handle multiple translation units at once as C does, and pass off
>> GENERIC for all
>> of them to the next level at once, so that various
>> cross-translation-unit consistency checks can be made which depend
>> on language-specific rules.
>
> Also for C++ you don't need to end up with duplicated template bodies
> everywhere.

This is going to be complex though. If you look at the EDG frontend, they use
an iterative system to generate the minimum set of template instantiations (at
least, by default):

- When a file is first compiled, no templates are instantiated. Instead, a
separate text file (.ti IIRC) is generated with a list of the instantiations
that the file requires.
- At link time, all these files are collected and merged, getting an unique
lists of all the template instantiations needed.
- A new set of text files (.to IIRC) is generated, which contains the list of
instantiations which are requested for each file.
- The files with a .to list are recompiled. The presence of the .to file
automatically triggers the requested template instantiations. The .ti file is
generated again: it can be a superset of the previous one because the requested
instantiations might generate new dependencies on missing instantiations.
- The process is re-iterated: the .ti files are merged again, and if there are
missing instantiations, new .to files are generated, and so on.

Of course, special care must be taken at link time when assigning
instantiations to files, to avoid endless loops, and minimize recompilations.
EDG used to have bugs like this which are very hard to reduce and reproduce.

An additional good point of this system is that it is kind of "stable": if you
modify a single file, assuming you don't ask for more instantiations, you do
not have to recompile anything when you link. My experience is that building
big applications from scratch might requires several instantiation loops (up to
3 in my applications), but after that the compilation times after touching a
small set of files is much decreased.

I think that such a system could be added to GCC as well, by using a new option
(-fcool-template-processing) and a special collect2-like program to be run at
link time.

In other words, I believe a solution to the problem of minimizing the template
instantiations is orthogonal to IMA and its design.

Giovanni Bajo

Re: GCC IMA & general future proposal

[Scott Robert Ladd]

Giovanni Bajo wrote:
> I think that such a system could be added to GCC as well, by using a new option
> (-fcool-template-processing) and a special collect2-like program to be run at
> link time.

We might also consider something like (I hope the meaning is obvious):

    -fcool-template-processing=cautious
    -fcool-template-processing=aggressive

The compiler could default to "cautious", thus avoiding some of those 
bugs encountered by EDG; for people who perform complex template 
gynastics, "aggressive" might not work, but "cautious" would.

-- 
Scott Robert Ladd
site: http://www.coyotegulch.com
blog: http://chaoticcoyote.blogspot.com

Re: GCC IMA & general future proposal

[Kenneth Zadeck]

Scott Robert Ladd wrote:

> I'll keep this short, then let you have the final word if you want it.
>
> David Edelsohn wrote:
>
>>>> they have guaranteed that no one will use the compiler for anything 
>>>> except benchmarking.
>>>
>>
>> Scott> Incorrect. I know many people who use Intel's compiler for 
>> production code, especially in the high-performance and Fortran
>> communities.
>>
>> Again, that is not the point of Ken's comment.
>
>
> Ken could have made his point without all the anti-Intel rhetoric.
> Geoff's proposal doesn't even mention Intel by name, so I wonder why Ken
> felt it necessary to use this as a platform for assailing Intel?
>
I have nothing against intel.  However their compiler is designed to 
PRIMARILY highlight their hardware.  They will be the first to admit 
that they are a hardware company and they primarily support software 
with the eye towards it enhancing their hardware.  There is nothing 
wrong with this attitude and over the years I have talked several Intel 
execs who are quite open about this policy.

Having said that my main reason for mentioning them was because at the 
ipa meeting the intel compiler was used as a model of a compiler that 
did almost nothing to produce the .o file and did all of the work at 
link time.  Furthemore the proposal was made that we should emulate 
this.  This model works if your customers are benchmark runners but is 
not the best model if your customers are software developers.

While the ibm compiler does a good job on benchmarks, it also does a 
very good job on a wide variety of applications that do not fit into the 
compile the whole world model, like their operating systems, their data 
base engines and their java virtual machines.

Re: GCC IMA & general future proposal

[Jan Hubicka]

> Jan Hubicka wrote:
> 
> >> It is still useful in such a framework for front ends to be able to
> >> handle multiple translation units at once as C does, and pass off
> >> GENERIC for all
> >> of them to the next level at once, so that various
> >> cross-translation-unit consistency checks can be made which depend
> >> on language-specific rules.
> >
> > Also for C++ you don't need to end up with duplicated template bodies
> > everywhere.
> 
> This is going to be complex though. If you look at the EDG frontend, they use
> an iterative system to generate the minimum set of template instantiations (at
> least, by default):
> 
> - When a file is first compiled, no templates are instantiated. Instead, a
> separate text file (.ti IIRC) is generated with a list of the instantiations
> that the file requires.
> - At link time, all these files are collected and merged, getting an unique
> lists of all the template instantiations needed.
> - A new set of text files (.to IIRC) is generated, which contains the list of
> instantiations which are requested for each file.
> - The files with a .to list are recompiled. The presence of the .to file
> automatically triggers the requested template instantiations. The .ti file is
> generated again: it can be a superset of the previous one because the requested
> instantiations might generate new dependencies on missing instantiations.
> - The process is re-iterated: the .ti files are merged again, and if there are
> missing instantiations, new .to files are generated, and so on.
> 
> Of course, special care must be taken at link time when assigning
> instantiations to files, to avoid endless loops, and minimize recompilations.
> EDG used to have bugs like this which are very hard to reduce and reproduce.
> 
> An additional good point of this system is that it is kind of "stable": if you
> modify a single file, assuming you don't ask for more instantiations, you do
> not have to recompile anything when you link. My experience is that building
> big applications from scratch might requires several instantiation loops (up to
> 3 in my applications), but after that the compilation times after touching a
> small set of files is much decreased.
> 
> I think that such a system could be added to GCC as well, by using a new option
> (-fcool-template-processing) and a special collect2-like program to be run at
> link time.
> 
> In other words, I believe a solution to the problem of minimizing the template
> instantiations is orthogonal to IMA and its design.

I've had someting easier in my mind here - you simply parse multiple C++
files to memory like we can do now for C and then do template
instantiation same way we are doing now but taking into acount that you
don't need to instantiate same template multiple times for different
files.

Of course something that scales to bigger units better than this "easy"
trick is desirable.  The way EDG does this looks pretty amusing to me ;))
I wonder if they do have some paper or something with experimental
results of this approach (ie how often the reduction in templates
outweight the recompilation costs)

Honza
> 
> Giovanni Bajo
> 
> 

Re: GCC IMA & general future proposal

[Jan Hubicka]

> Jan Hubicka wrote:
> >>Jan Hubicka wrote:
> >>
> >>
> >>>>It is still useful in such a framework for front ends to be able to
> >>>>handle multiple translation units at once as C does, and pass off
> >>>>GENERIC for all
> >>>>of them to the next level at once, so that various
> >>>>cross-translation-unit consistency checks can be made which depend
> >>>>on language-specific rules.
> >>>
> >>>Also for C++ you don't need to end up with duplicated template bodies
> >>>everywhere.
> >>
> >>This is going to be complex though. If you look at the EDG frontend, they 
> >>use
> >>an iterative system to generate the minimum set of template 
> >>instantiations (at
> >>least, by default):
> >>
> >>- When a file is first compiled, no templates are instantiated. Instead, a
> >>separate text file (.ti IIRC) is generated with a list of the 
> >>instantiations
> >>that the file requires.
> >>- At link time, all these files are collected and merged, getting an 
> >>unique
> >>lists of all the template instantiations needed.
> >>- A new set of text files (.to IIRC) is generated, which contains the 
> >>list of
> >>instantiations which are requested for each file.
> >>- The files with a .to list are recompiled. The presence of the .to file
> >>automatically triggers the requested template instantiations. The .ti 
> >>file is
> >>generated again: it can be a superset of the previous one because the 
> >>requested
> >>instantiations might generate new dependencies on missing instantiations.
> >>- The process is re-iterated: the .ti files are merged again, and if 
> >>there are
> >>missing instantiations, new .to files are generated, and so on.
> >>
> >>Of course, special care must be taken at link time when assigning
> >>instantiations to files, to avoid endless loops, and minimize 
> >>recompilations.
> >>EDG used to have bugs like this which are very hard to reduce and 
> >>reproduce.
> >>
> >>An additional good point of this system is that it is kind of "stable": 
> >>if you
> >>modify a single file, assuming you don't ask for more instantiations, you 
> >>do
> >>not have to recompile anything when you link. My experience is that 
> >>building
> >>big applications from scratch might requires several instantiation loops 
> >>(up to
> >>3 in my applications), but after that the compilation times after 
> >>touching a
> >>small set of files is much decreased.
> >>
> >>I think that such a system could be added to GCC as well, by using a new 
> >>option
> >>(-fcool-template-processing) and a special collect2-like program to be 
> >>run at
> >>link time.
> >>
> >>In other words, I believe a solution to the problem of minimizing the 
> >>template
> >>instantiations is orthogonal to IMA and its design.
> >
> >
> >I've had someting easier in my mind here - you simply parse multiple C++
> >files to memory like we can do now for C and then do template
> >instantiation same way we are doing now but taking into acount that you
> >don't need to instantiate same template multiple times for different
> >files.
> >
> >Of course something that scales to bigger units better than this "easy"
> >trick is desirable.  The way EDG does this looks pretty amusing to me ;))
> >I wonder if they do have some paper or something with experimental
> >results of this approach (ie how often the reduction in templates
> >outweight the recompilation costs)
> 
> This may be a stupid suggestion and is probably not as general
> as what EDG is doing but couldn't the template instantiations be
> put into special sections and then let ld eliminate the duplicates
> as it merges the object files into the executable?

This is what GCC does currently, but it results in very many
instantiations being compiled just to be thrown away that consumes time
and disc space.

Honza
> 
> >Honza
> >
> >>Giovanni Bajo
> >>
> >>
> 
> 
> -- 
> Joel Sherrill, Ph.D.             Director of Research & Development
> joel@OARcorp.com                 On-Line Applications Research
> Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>    Support Available             (256) 722-9985
> 

Re: GCC IMA & general future proposal

[Joel Sherrill <joel@OARcorp.com>]

Jan Hubicka wrote:
>>Jan Hubicka wrote:
>>
>>
>>>>It is still useful in such a framework for front ends to be able to
>>>>handle multiple translation units at once as C does, and pass off
>>>>GENERIC for all
>>>>of them to the next level at once, so that various
>>>>cross-translation-unit consistency checks can be made which depend
>>>>on language-specific rules.
>>>
>>>Also for C++ you don't need to end up with duplicated template bodies
>>>everywhere.
>>
>>This is going to be complex though. If you look at the EDG frontend, they use
>>an iterative system to generate the minimum set of template instantiations (at
>>least, by default):
>>
>>- When a file is first compiled, no templates are instantiated. Instead, a
>>separate text file (.ti IIRC) is generated with a list of the instantiations
>>that the file requires.
>>- At link time, all these files are collected and merged, getting an unique
>>lists of all the template instantiations needed.
>>- A new set of text files (.to IIRC) is generated, which contains the list of
>>instantiations which are requested for each file.
>>- The files with a .to list are recompiled. The presence of the .to file
>>automatically triggers the requested template instantiations. The .ti file is
>>generated again: it can be a superset of the previous one because the requested
>>instantiations might generate new dependencies on missing instantiations.
>>- The process is re-iterated: the .ti files are merged again, and if there are
>>missing instantiations, new .to files are generated, and so on.
>>
>>Of course, special care must be taken at link time when assigning
>>instantiations to files, to avoid endless loops, and minimize recompilations.
>>EDG used to have bugs like this which are very hard to reduce and reproduce.
>>
>>An additional good point of this system is that it is kind of "stable": if you
>>modify a single file, assuming you don't ask for more instantiations, you do
>>not have to recompile anything when you link. My experience is that building
>>big applications from scratch might requires several instantiation loops (up to
>>3 in my applications), but after that the compilation times after touching a
>>small set of files is much decreased.
>>
>>I think that such a system could be added to GCC as well, by using a new option
>>(-fcool-template-processing) and a special collect2-like program to be run at
>>link time.
>>
>>In other words, I believe a solution to the problem of minimizing the template
>>instantiations is orthogonal to IMA and its design.
> 
> 
> I've had someting easier in my mind here - you simply parse multiple C++
> files to memory like we can do now for C and then do template
> instantiation same way we are doing now but taking into acount that you
> don't need to instantiate same template multiple times for different
> files.
> 
> Of course something that scales to bigger units better than this "easy"
> trick is desirable.  The way EDG does this looks pretty amusing to me ;))
> I wonder if they do have some paper or something with experimental
> results of this approach (ie how often the reduction in templates
> outweight the recompilation costs)

This may be a stupid suggestion and is probably not as general
as what EDG is doing but couldn't the template instantiations be
put into special sections and then let ld eliminate the duplicates
as it merges the object files into the executable?

> Honza
> 
>>Giovanni Bajo
>>
>>


-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel@OARcorp.com                 On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
    Support Available             (256) 722-9985

Re: GCC IMA & general future proposal

[Jeffrey A Law]

On Wed, 2004-10-27 at 19:49 +0200, Jan Hubicka wrote:
> > 
> > This may be a stupid suggestion and is probably not as general
> > as what EDG is doing but couldn't the template instantiations be
> > put into special sections and then let ld eliminate the duplicates
> > as it merges the object files into the executable?
> 
> This is what GCC does currently, but it results in very many
> instantiations being compiled just to be thrown away that consumes time
> and disc space.
Right.  And I get customers who complain about this on a pretty
consistent basis.  This scheme slows down compilation time, link time
and eats a ton of disk space.

jeff

Re: GCC IMA & general future proposal

[Daniel Berlin]

>
> This may be a stupid suggestion and is probably not as general
> as what EDG is doing but couldn't the template instantiations be
> put into special sections and then let ld eliminate the duplicates
> as it merges the object files into the executable?
>

This is what we do now.
However, it doesn't save the work of doing the instantiation, only the 
space it costs in the final executable.
They seem to be talking about saving the work of doing duplicate 
instantiation in the first place.

Re: GCC IMA & general future proposal

[Matt Austern]

On Oct 27, 2004, at 10:50 AM, Daniel Berlin wrote:

>>
>> This may be a stupid suggestion and is probably not as general
>> as what EDG is doing but couldn't the template instantiations be
>> put into special sections and then let ld eliminate the duplicates
>> as it merges the object files into the executable?
>>
>
> This is what we do now.
> However, it doesn't save the work of doing the instantiation, only the 
> space it costs in the final executable.
> They seem to be talking about saving the work of doing duplicate 
> instantiation in the first place.

We have to be careful, though.  The Sun model, where you try to 
instantiate a template only once, is more fragile than our model, it 
can sometimes result in even longer compile times (my first impression, 
when I moved from the EDG-based SGI compiler to gcc, was how much 
faster gcc was), and it introduces dependencies that aren't reflected 
in the makefile.  The Sun model has some advantages, but it is 
definitely not a uniform win.

We also have to be careful about accumulating more and more template 
instantiation models without ever throwing any away.  We've now got the 
instantiate-everything-and-merge model, the 
explicit-instantiations-only model, and the repository model.  Let's 
try not to go from three models to four.

Now what I *could* get behind, if we want to think about radical 
changes to our template instantiation model, is either implementing 
export or implementing something that does a better job of solving the 
problems that export was designed to solve.

			--Matt

Re: GCC IMA & general future proposal

[Joe Buck]

On Wed, Oct 27, 2004 at 11:55:49AM -0600, Jeffrey A Law wrote:
> On Wed, 2004-10-27 at 19:49 +0200, Jan Hubicka wrote:
> > > 
> > > This may be a stupid suggestion and is probably not as general
> > > as what EDG is doing but couldn't the template instantiations be
> > > put into special sections and then let ld eliminate the duplicates
> > > as it merges the object files into the executable?
> > 
> > This is what GCC does currently, but it results in very many
> > instantiations being compiled just to be thrown away that consumes time
> > and disc space.
> Right.  And I get customers who complain about this on a pretty
> consistent basis.  This scheme slows down compilation time, link time
> and eats a ton of disk space.

But the alternative of repeatedly recompiling until no more templates
are instantiated can also be wasteful and slow.  I haven't yet seen a
scheme that really wins over what GCC is doing now, if you consider the
effect of corrupt repository databases, library closure issues and the
like.

Re: GCC IMA & general future proposal

[Mike Stump]

On Oct 27, 2004, at 9:48 AM, Giovanni Bajo wrote:
> I think that such a system could be added to GCC as well, by using a 
> new option (-fcool-template-processing) and a special collect2-like 
> program to be run at link time.

What are the significant differences between what you want and -frepo?

Re: GCC IMA & general future proposal

[Jeffrey A Law]

On Wed, 2004-10-27 at 14:04 -0700, Joe Buck wrote:
> On Wed, Oct 27, 2004 at 11:55:49AM -0600, Jeffrey A Law wrote:
> > On Wed, 2004-10-27 at 19:49 +0200, Jan Hubicka wrote:
> > > > 
> > > > This may be a stupid suggestion and is probably not as general
> > > > as what EDG is doing but couldn't the template instantiations be
> > > > put into special sections and then let ld eliminate the duplicates
> > > > as it merges the object files into the executable?
> > > 
> > > This is what GCC does currently, but it results in very many
> > > instantiations being compiled just to be thrown away that consumes time
> > > and disc space.
> > Right.  And I get customers who complain about this on a pretty
> > consistent basis.  This scheme slows down compilation time, link time
> > and eats a ton of disk space.
> 
> But the alternative of repeatedly recompiling until no more templates
> are instantiated can also be wasteful and slow. 
Yes.

>  I haven't yet seen a
> scheme that really wins over what GCC is doing now, if you consider the
> effect of corrupt repository databases, library closure issues and the
> like.
Well, actually a number of the organizations I've talked to actually
prefer the Sun method -- corrupt repositories and all.  Basically
they blast away their repo once a week or so to deal with that
little problem.

That's not to say we should emulate the losing behaviors found in
the Sun system -- just to say that even with its flaws there are
organizations which prefer it over how GCC handles instantiation.

Jeff

Re: GCC IMA & general future proposal

[Joe Buck]

On Wed, Oct 27, 2004 at 08:38:23PM -0600, Jeffrey A Law wrote:
> Well, actually a number of the organizations I've talked to actually
> prefer the Sun method -- corrupt repositories and all.  Basically
> they blast away their repo once a week or so to deal with that
> little problem.

I have spent about five years of my life fighting Sun's broken repo.
(I've had to produce code that works with Sun's compiler, g++, HP aCC,
and sometimes MS).

You've got to blast it away a lot more than once a week, and it
regularly gets corrupt.  I hate it with a passion.  I've had more
trouble with it than with any of the other compilers.  The object
file bloat is a small price to pay for something that Just Works.

Re: GCC IMA & general future proposal

[Benjamin Kosnik]

>> 	Is LLVM going to be licensed under the GPL?

> If that were seriously the issue, I suspect that relicensing would be an
> option.

.. then release a version of LLVM under the GPL so that it can seriously
be considered.

All else is just talk.

-benjamin

Re: GCC IMA & general future proposal

[Chris Lattner]

On Wed, 27 Oct 2004, Steven Bosscher wrote:

> On Wednesday 27 October 2004 21:03, Chris Lattner wrote:
> > To put it another way, which has more missing features? :)  LOC isn't
> > everything, as we've shown many times over.
>
> It's not LOC.
>
> It's 52 targets with a bunch of subtargets against a compiler that so
> far can't tell the difference between a Pentium 2 and a Pentium 3.
>
> Redoing all the backends for LLVM is a huge amount of work.  Think
> about the S/390 for example, or AMD64, Itanium, etc.  Those are not
> targets you just rewrite a good backend for in a matter of weeks or
> months.

Getting back to technical issues, let me point out that using LLVM to do
IPO does not require throwing away all of the RTL backends!

-Chris

-- 
http://llvm.org/
http://nondot.org/sabre/

Re: GCC IMA & general future proposal

[Steven Bosscher]

On Wednesday 27 October 2004 21:03, Chris Lattner wrote:
> To put it another way, which has more missing features? :)  LOC isn't
> everything, as we've shown many times over.

It's not LOC.

It's 52 targets with a bunch of subtargets against a compiler that so
far can't tell the difference between a Pentium 2 and a Pentium 3.

Redoing all the backends for LLVM is a huge amount of work.  Think
about the S/390 for example, or AMD64, Itanium, etc.  Those are not
targets you just rewrite a good backend for in a matter of weeks or
months.

Gr.
Steven

Re: GCC IMA & general future proposal

[Chris Lattner]

On Wed, 27 Oct 2004, Geoff Keating wrote:
> >>> Has anyone seriously considered using or adopting LLVM
> >>> (http://llvm.org)?
> >>> It seems to provide just about everything on your diagram, and is
> >>> also
> >>> quite efficient.
> >>>
> >>> Even if it is not an option, I would really be interested to hear why
> >>> not and what is missing.  In particular, LLVM is mutable, and it's
> >>> easier to add things to it than to reinvent the whole thing.  :)
> >>
> >> It's the part where you have to add everything that GCC has that LLVM
> >> doesn't that seems to be the problem.
> >
> > Heh, likewise, you're talking about adding everything that LLVM has to
> > GCC.  :)
>
> Right.  So, which is the larger source base, LLVM or GCC?

To put it another way, which has more missing features? :)  LOC isn't
everything, as we've shown many times over.

-Chris

-- 
http://llvm.org/
http://nondot.org/sabre/

Re: GCC IMA & general future proposal

[Daniel Berlin]

>
>> On Wed, 27 Oct 2004, Geoff Keating wrote:
>>> It's the part where you have to add everything that GCC has that LLVM
>>> doesn't that seems to be the problem.
>> 
>> Heh, likewise, you're talking about adding everything that LLVM has to
>> GCC.  :)
>
> Right.  So, which is the larger source base, LLVM or GCC?
>
This sounds like it will quickly degenerate into an e-penis contest.

The issues in using LLVM are not technical for the most part, i believe. 
It's a fine piece of software.
The issues are political and legal.

Re: GCC IMA & general future proposal

[Geoff Keating]

[-- Attachment #1: Type: text/plain, Size: 882 bytes --]


On 27/10/2004, at 11:54 AM, Chris Lattner wrote:

> On Wed, 27 Oct 2004, Geoff Keating wrote:
>>>> It's short, but has a complicated diagram. It is intended for
>>>> discussion
>>>> at the meeting in a few hours.
>>>
>>> Has anyone seriously considered using or adopting LLVM
>>> (http://llvm.org)?
>>> It seems to provide just about everything on your diagram, and is 
>>> also
>>> quite efficient.
>>>
>>> Even if it is not an option, I would really be interested to hear why
>>> not and what is missing.  In particular, LLVM is mutable, and it's
>>> easier to add things to it than to reinvent the whole thing.  :)
>>
>> It's the part where you have to add everything that GCC has that LLVM
>> doesn't that seems to be the problem.
>
> Heh, likewise, you're talking about adding everything that LLVM has to
> GCC.  :)

Right.  So, which is the larger source base, LLVM or GCC?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2410 bytes --]

Re: GCC IMA & general future proposal

[Chris Lattner]

On Wed, 27 Oct 2004, Geoff Keating wrote:
> >> It's short, but has a complicated diagram. It is intended for
> >> discussion
> >> at the meeting in a few hours.
> >
> > Has anyone seriously considered using or adopting LLVM
> > (http://llvm.org)?
> > It seems to provide just about everything on your diagram, and is also
> > quite efficient.
> >
> > Even if it is not an option, I would really be interested to hear why
> > not and what is missing.  In particular, LLVM is mutable, and it's
> > easier to add things to it than to reinvent the whole thing.  :)
>
> It's the part where you have to add everything that GCC has that LLVM
> doesn't that seems to be the problem.

Heh, likewise, you're talking about adding everything that LLVM has to
GCC.  :)

-Chris

-- 
http://llvm.org/
http://nondot.org/sabre/

Re: GCC IMA & general future proposal

[Chris Lattner]

On Wed, 27 Oct 2004, David Edelsohn wrote:
> >>>>> Chris Lattner writes:
> Chris> Has anyone seriously considered using or adopting LLVM (http://llvm.org)?
> Chris> Even if it is not an option, I would really be interested to hear why not
> Chris> and what is missing.  In particular, LLVM is mutable, and it's easier to
> Chris> add things to it than to reinvent the whole thing.  :)
>
> 	Are you referring to the LLVM format and design or LLVM itself?

LLVM itself.

> 	Is LLVM going to be licensed under the GPL?

If that were seriously the issue, I suspect that relicensing would be an
option.

-Chris

-- 
http://llvm.org/
http://nondot.org/sabre/

Re: GCC IMA & general future proposal

[David Edelsohn]

>>>>> Chris Lattner writes:

Chris> Has anyone seriously considered using or adopting LLVM (http://llvm.org)?
Chris> Even if it is not an option, I would really be interested to hear why not
Chris> and what is missing.  In particular, LLVM is mutable, and it's easier to
Chris> add things to it than to reinvent the whole thing.  :)

	Are you referring to the LLVM format and design or LLVM itself?

	Is LLVM going to be licensed under the GPL?

David

Re: GCC IMA & general future proposal

[Geoff Keating]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]


On 27/10/2004, at 11:45 AM, Chris Lattner wrote:

>
> Geoff Keating wrote:
>> I've put a quick draft of my idea of what the future of GCC should 
>> look
>> like at
>> <http://www.geoffk.org/gcc/future/gccfuture.html>
>>
>> It's short, but has a complicated diagram. It is intended for 
>> discussion
>> at the meeting in a few hours.
>
> Has anyone seriously considered using or adopting LLVM 
> (http://llvm.org)?
> It seems to provide just about everything on your diagram, and is also
> quite efficient.
>
> Even if it is not an option, I would really be interested to hear why 
> not
> and what is missing.  In particular, LLVM is mutable, and it's easier 
> to
> add things to it than to reinvent the whole thing.  :)

It's the part where you have to add everything that GCC has that LLVM 
doesn't that seems to be the problem.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2410 bytes --]

RE: GCC IMA & general future proposal

[Chris Lattner]

Geoff Keating wrote:
> I've put a quick draft of my idea of what the future of GCC should look
> like at
> <http://www.geoffk.org/gcc/future/gccfuture.html>
>
> It's short, but has a complicated diagram. It is intended for discussion
> at the meeting in a few hours.

Has anyone seriously considered using or adopting LLVM (http://llvm.org)?
It seems to provide just about everything on your diagram, and is also
quite efficient.

Even if it is not an option, I would really be interested to hear why not
and what is missing.  In particular, LLVM is mutable, and it's easier to
add things to it than to reinvent the whole thing.  :)

Thoughts?

-Chris

-- 
http://llvm.org/
http://nondot.org/sabre/