GCC internal re-architecture proposal

GCC internal re-architecture proposal

[Andrew MacLeod]

[ I foolishly sent this with the document as an attachment... hopefully 
it gets rejected and anyone interested can simply download the document 
from the wiki..]

Over the past couple of months, I've slowly been putting together an 
action plan to help modernize GCC's source base.  We've had various 
ideas put forth over the years, and a few of the more meritorious ones 
have been incorporated.  My primary goal is to disentangle the front end 
from the middle/back ends, giving us better control over their data 
structures.

The biggest challenge is having an executable plan which allows the 
source to be updated incrementally.   Ie, we need a way to get from 
'Here' to 'There' that does not impact ongoing normal activities, nor 
noticeably affect the performance of the compiler.  This is a huge 
effort and it will require a large time commitment on my part to see it 
through.

I've attached a link to a document which outlines my proposal. Since 
putting it together, I've performed a couple of actual file conversions 
(albeit minor ones) beyond the simple examples I used in the document 
from the original proof of concept.  I've been refining some of the 
actual implementation details and logged the various design decisions 
that I encounter for later discussions  This document is a work in 
progress, and should be considered a strategic guide rather than an 
implementation bible.

There will be a BOF at Cauldron for anyone interested in discussing 
aspects of this.  I should also have a list of the interesting issues 
I've encountered for discussion there.

Im on vacation for about half the time between now and Cauldron, so I 
wanted to get this out now.  As for a time line, I wont be able to get 
seriously started on it for a couple of months since I have the C11 
_Atomic and gimple-atomic implementations to get delivered. That gives 
us a reasonable amount of time to address concerns and sort out the 
overall approach as well other nitty gritty before any serious work 
commences.    Once underway, I expect there would be plenty to do for 
anyone else interesting in volunteering some time.

Here's the proposal document... happy reading! Feedback is welcome.

I hope this link will work  :-)...
http://gcc.gnu.org/wiki/AndrewMacLeod?action=AttachFile&do=view&target=gccrestructureplan2.1.odt

Highlander

Re: GCC internal re-architecture proposal

[Andrew MacLeod]

On 06/21/2013 11:27 AM, Andrew MacLeod wrote:
>
> I hope this link will work  :-)...
> http://gcc.gnu.org/wiki/AndrewMacLeod?action=AttachFile&do=view&target=gccrestructureplan2.1.odt 
>
>
> Highlander
or you can use this PDF file...
http://gcc.gnu.org/wiki/AndrewMacLeod?action=AttachFile&do=view&target=gccrestructureplan2.1.pdf

Re: GCC internal re-architecture proposal

[Joseph S. Myers]

A few observations:

* You don't mention anything about separating host-side and target-side 
configuration, which are also entangled.  A slightly out-of-date list of 
target macros used in target-side code is at 
<http://gcc.gnu.org/wiki/Top-Level_Libgcc_Migration> and a clean back-end 
class would involve eliminating those (so libgcc only includes libgcc_tm.h 
and not the host-side tm.h at all).  (Various of the other things listed 
on that page were finished by Rainer, but not the target macros.)

* I don't see the advantages of separating out the host information into a 
different API.  Any particular compiler binary can only meaningfully run 
on one host, whereas in principle it should be possible to have multiple 
instances of a target class in one process so separating out the target 
interface does make sense.

* It's not clear to me what's supposed to be different about the 
target_info class compared to the existing target structure.  We "just" 
need to convert several hundred more target macros into hooks; that's the 
major work rather than any change from a global object to C++ classes or 
similar.  (The transitional conversion whereby targhooks.c contains a hook 
definition that uses a target macro is, to my mind, of very limited value 
as an end-point; such conversions need finishing by eliminating all direct 
users of the macro, then converting all back ends to define the hook 
directly, then poisoning the macro and allowing and documenting only the 
hook.)

Target macros used in front ends are indeed a good priority for converting 
to hooks.  See previous discussions (and in some cases old patches by 
Joern) of how to convert some particular cases, e.g. INT_TYPE_SIZE or 
SIZE_TYPE (roughly, it seems to make sense to have hooks covering groups 
of related macros rather than one hook per macro in all cases).

(Yes, a clean separation needs some new hook structures as well, e.g. in 
the driver, where the approach for defining the hooks would probably be 
different from the present one given how they tend to depend on the target 
OS as much as the target architecture.)

* Quite a lot of target interfaces would be used by the front ends 
specifically for defining macros for building target libraries only; see 
the "probably predefine macros if -fbuilding-libgcc in most cases" list on 
that wiki page.  (Thus, given a clean separation of target-side and 
host-side configuration, c-cppbuiltin.c is inevitably going to use a lot 
of target macros / hooks.)

* Various cases of front-end folding are to get better warnings, e.g. to 
detect that in a signed-to-unsigned conversion the converted value is in 
fact never negative.  I don't see this issue mentioned in the document.  
But maybe the GIMPLE generated by the front end should contain some form 
of note "give this warning if that condition is satisfied" and then the 
warnings would get given after some GIMPLE optimizations.

* Option handling uses a single OPT_* enumeration with options from all 
parts of the compiler; for a full clean separation, you can no longer just 
have one array of all the options generated when GCC is built, and each 
bit of the compiler would need to see just a subset of the options.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: GCC internal re-architecture proposal

[Andrew MacLeod]

On 06/21/2013 12:16 PM, Joseph S. Myers wrote:
> A few observations:
>
> * You don't mention anything about separating host-side and target-side
> configuration, which are also entangled.  A slightly out-of-date list of
> target macros used in target-side code is at
> <http://gcc.gnu.org/wiki/Top-Level_Libgcc_Migration> and a clean back-end
> class would involve eliminating those (so libgcc only includes libgcc_tm.h
> and not the host-side tm.h at all).  (Various of the other things listed
> on that page were finished by Rainer, but not the target macros.)
yeah, I didn't go into any configuration stuff... I don't understand it 
well enough (and avoid it like the plague :-)  so that will require some 
outside assistance :-)

Thats something that can be flushed out better in the future for sure.


>
> * I don't see the advantages of separating out the host information into a
> different API.  Any particular compiler binary can only meaningfully run
> on one host, whereas in principle it should be possible to have multiple
> instances of a target class in one process so separating out the target
> interface does make sense.
That has been brought up, and It may not be worth any effort at all.   I 
added it more for completeness and consideration for getting that 
information "all in one place".  It may well come to nothing.
>
> * It's not clear to me what's supposed to be different about the
> target_info class compared to the existing target structure.  We "just"

There may not be any difference.  The term target_info was just taken 
out of my hat. It may well be that we just extend the existing 
structure.   It just need to be flushed out fully and completed. Im not 
terribly familiar with those bits, so i didn't give it much detail :-)
> need to convert several hundred more target macros into hooks; that's the
> major work rather than any change from a global object to C++ classes or
> similar.  (The transitional conversion whereby targhooks.c contains a hook
> definition that uses a target macro is, to my mind, of very limited value
> as an end-point; such conversions need finishing by eliminating all direct
> users of the macro, then converting all back ends to define the hook
> directly, then poisoning the macro and allowing and documenting only the
> hook.)
I think that sounds like a most reasonable way to proceed.
>
> Target macros used in front ends are indeed a good priority for converting
> to hooks.  See previous discussions (and in some cases old patches by
> Joern) of how to convert some particular cases, e.g. INT_TYPE_SIZE or
> SIZE_TYPE (roughly, it seems to make sense to have hooks covering groups
> of related macros rather than one hook per macro in all cases).
indeed, thats exactly the sort of low-hanging simplifications I was 
thinking about, and I'm sure there will be others
>
>
>
> * Quite a lot of target interfaces would be used by the front ends
> specifically for defining macros for building target libraries only; see
> the "probably predefine macros if -fbuilding-libgcc in most cases" list on
> that wiki page.  (Thus, given a clean separation of target-side and
> host-side configuration, c-cppbuiltin.c is inevitably going to use a lot
> of target macros / hooks.)
If thats where most of it lies, maybe there is some way to "push" some 
of that activity into the backend?...   I'll investigate c-cppbuiltin.c 
at some point when I get some time.   "Heavy" consumers like that would 
show up after the initial conversion stages and we can consider how to 
improve it.

>
> * Various cases of front-end folding are to get better warnings, e.g. to
> detect that in a signed-to-unsigned conversion the converted value is in
> fact never negative.  I don't see this issue mentioned in the document.
> But maybe the GIMPLE generated by the front end should contain some form
> of note "give this warning if that condition is satisfied" and then the
> warnings would get given after some GIMPLE optimizations.

Indeed, a previous incarnation of the document took that idea to further 
extremes, but that also sounds highly reasonable.   The gimple fodler 
may be able to issue those sorts of warnings as it "optimizes" by 
folding.   I'll add something back in about that.
>
> * Option handling uses a single OPT_* enumeration with options from all
> parts of the compiler; for a full clean separation, you can no longer just
> have one array of all the options generated when GCC is built, and each
> bit of the compiler would need to see just a subset of the options.
>
Good observation, I'll add something about that too.

Thanks!

Andrew

Re: GCC internal re-architecture proposal

[Tobias Burnus]

Andrew MacLeod wrote:
>> * Quite a lot of target interfaces would be used by the front ends
>> specifically for defining macros for building target libraries only; see
>> the "probably predefine macros if -fbuilding-libgcc in most cases"
>> list on
>> that wiki page.  (Thus, given a clean separation of target-side and
>> host-side configuration, c-cppbuiltin.c is inevitably going to use a lot
>> of target macros / hooks.)
> If thats where most of it lies, maybe there is some way to "push" some
> of that activity into the backend?...   I'll investigate c-cppbuiltin.c
> at some point when I get some time.   "Heavy" consumers like that would
> show up after the initial conversion stages and we can consider how to
> improve it.

When looking at CPP, please don't forget that also gfortran uses CPP 
(libcpp).

(Currently, gfortran does not provide all target macro definitions as 
gcc/config is too C/C++ centric; the idea was to move to hooks and then 
to change the definitions into a generic and C/C++ only part - such that 
the generic part can be also used for Fortran. See PR42954.)

Tobias

Re: GCC internal re-architecture proposal

[Eric Botcazou]

> * Various cases of front-end folding are to get better warnings, e.g. to
> detect that in a signed-to-unsigned conversion the converted value is in
> fact never negative.  I don't see this issue mentioned in the document.

Tree front-end folding, or more precisely folding of expressions generated for 
types, is required when types with variable size are first-class citizens like 
in Ada.  In practice, something even more powerful is required, i.e. outlining 
(called from stor-layout.c:variable_size) and back-inlining of the expressions 
(tree-inline.c:maybe_inline_call_in_expr), if you want to be able to compile 
big codebases with a reasonable amount of resources.

-- 
Eric Botcazou

Re: GCC internal re-architecture proposal

[Richard Biener]

On Fri, Jun 21, 2013 at 5:27 PM, Andrew MacLeod <amacleod@redhat.com> wrote:
> [ I foolishly sent this with the document as an attachment... hopefully it
> gets rejected and anyone interested can simply download the document from
> the wiki..]
>
> Over the past couple of months, I've slowly been putting together an action
> plan to help modernize GCC's source base.  We've had various ideas put forth
> over the years, and a few of the more meritorious ones have been
> incorporated.  My primary goal is to disentangle the front end from the
> middle/back ends, giving us better control over their data structures.
>
> The biggest challenge is having an executable plan which allows the source
> to be updated incrementally.   Ie, we need a way to get from 'Here' to
> 'There' that does not impact ongoing normal activities, nor noticeably
> affect the performance of the compiler.  This is a huge effort and it will
> require a large time commitment on my part to see it through.

Sorry for following up on this old e-mail but it just occured to me that
the (unfinished) GIMPLE frontend would be a great place to bootstrap
the new and shiny GIMPLE data structures.

Of course for it to work you need to translate that back to fat-and-old GIMPLE
and eventually finish the textual format dumping as well - not sure what
the state of the gimple frontend branch is.

Richard.

Re: GCC internal re-architecture proposal

[Diego Novillo]

On Tue, Mar 18, 2014 at 10:19 AM, Richard Biener
<richard.guenther@gmail.com> wrote:
> On Fri, Jun 21, 2013 at 5:27 PM, Andrew MacLeod <amacleod@redhat.com> wrote:
>> [ I foolishly sent this with the document as an attachment... hopefully it
>> gets rejected and anyone interested can simply download the document from
>> the wiki..]
>>
>> Over the past couple of months, I've slowly been putting together an action
>> plan to help modernize GCC's source base.  We've had various ideas put forth
>> over the years, and a few of the more meritorious ones have been
>> incorporated.  My primary goal is to disentangle the front end from the
>> middle/back ends, giving us better control over their data structures.
>>
>> The biggest challenge is having an executable plan which allows the source
>> to be updated incrementally.   Ie, we need a way to get from 'Here' to
>> 'There' that does not impact ongoing normal activities, nor noticeably
>> affect the performance of the compiler.  This is a huge effort and it will
>> require a large time commitment on my part to see it through.
>
> Sorry for following up on this old e-mail but it just occured to me that
> the (unfinished) GIMPLE frontend would be a great place to bootstrap
> the new and shiny GIMPLE data structures.
>
> Of course for it to work you need to translate that back to fat-and-old GIMPLE
> and eventually finish the textual format dumping as well - not sure what
> the state of the gimple frontend branch is.

Work on it stalled sometime last year.

I'm not sure if I would mix the re-arch work with the GIMPLE FE work.
The parser will take the most benefit out of the new structure, but I
think it may be better to make the front end be a sub-branch of the
re-architect work so we don't conflate the two.

Some work will be independent, but it will be easier to maintain both
branches if they are separate.

I do like the idea of tying the GIMPLE front end to the re-arch
branch, however. So, making it a sub-branch of re-arch is definitely a
good idea.


Diego.

Re: GCC internal re-architecture proposal

[Richard Biener]

On Tue, Mar 18, 2014 at 5:34 PM, Diego Novillo <dnovillo@google.com> wrote:
> On Tue, Mar 18, 2014 at 10:19 AM, Richard Biener
> <richard.guenther@gmail.com> wrote:
>> On Fri, Jun 21, 2013 at 5:27 PM, Andrew MacLeod <amacleod@redhat.com> wrote:
>>> [ I foolishly sent this with the document as an attachment... hopefully it
>>> gets rejected and anyone interested can simply download the document from
>>> the wiki..]
>>>
>>> Over the past couple of months, I've slowly been putting together an action
>>> plan to help modernize GCC's source base.  We've had various ideas put forth
>>> over the years, and a few of the more meritorious ones have been
>>> incorporated.  My primary goal is to disentangle the front end from the
>>> middle/back ends, giving us better control over their data structures.
>>>
>>> The biggest challenge is having an executable plan which allows the source
>>> to be updated incrementally.   Ie, we need a way to get from 'Here' to
>>> 'There' that does not impact ongoing normal activities, nor noticeably
>>> affect the performance of the compiler.  This is a huge effort and it will
>>> require a large time commitment on my part to see it through.
>>
>> Sorry for following up on this old e-mail but it just occured to me that
>> the (unfinished) GIMPLE frontend would be a great place to bootstrap
>> the new and shiny GIMPLE data structures.
>>
>> Of course for it to work you need to translate that back to fat-and-old GIMPLE
>> and eventually finish the textual format dumping as well - not sure what
>> the state of the gimple frontend branch is.
>
> Work on it stalled sometime last year.
>
> I'm not sure if I would mix the re-arch work with the GIMPLE FE work.
> The parser will take the most benefit out of the new structure, but I
> think it may be better to make the front end be a sub-branch of the
> re-architect work so we don't conflate the two.
>
> Some work will be independent, but it will be easier to maintain both
> branches if they are separate.
>
> I do like the idea of tying the GIMPLE front end to the re-arch
> branch, however. So, making it a sub-branch of re-arch is definitely a
> good idea.

Well, I thought that the GIMPLE FE itself is the easiest place where
you can try out things given that it has to represent all features
of GIMPLE but doesn't (necessarily) use the GIMPLE IL as its AST.
Especially the no-trees part should be easy there.  And we can be
convinced of the beauty of the new AST before munging all these
wrappers into trunk with the fear that only the wrappers will stay
and the project itself will never finish ... thus we get the ugliness
while with the GIMPLE FE we'd at least get the GIMPLE FE ...

Richard.

>
> Diego.

Re: GCC internal re-architecture proposal

[Andrew MacLeod]

On 03/19/2014 05:08 AM, Richard Biener wrote:
> On Tue, Mar 18, 2014 at 5:34 PM, Diego Novillo <dnovillo@google.com> wrote:
>> On Tue, Mar 18, 2014 at 10:19 AM, Richard Biener
>> <richard.guenther@gmail.com> wrote:
>>> On Fri, Jun 21, 2013 at 5:27 PM, Andrew MacLeod <amacleod@redhat.com> wrote:
>>>> [ I foolishly sent this with the document as an attachment... hopefully it
>>>> gets rejected and anyone interested can simply download the document from
>>>> the wiki..]
>>>>
>>>> Over the past couple of months, I've slowly been putting together an action
>>>> plan to help modernize GCC's source base.  We've had various ideas put forth
>>>> over the years, and a few of the more meritorious ones have been
>>>> incorporated.  My primary goal is to disentangle the front end from the
>>>> middle/back ends, giving us better control over their data structures.
>>>>
>>>> The biggest challenge is having an executable plan which allows the source
>>>> to be updated incrementally.   Ie, we need a way to get from 'Here' to
>>>> 'There' that does not impact ongoing normal activities, nor noticeably
>>>> affect the performance of the compiler.  This is a huge effort and it will
>>>> require a large time commitment on my part to see it through.
>>> Sorry for following up on this old e-mail but it just occured to me that
>>> the (unfinished) GIMPLE frontend would be a great place to bootstrap
>>> the new and shiny GIMPLE data structures.
>>>
>>> Of course for it to work you need to translate that back to fat-and-old GIMPLE
>>> and eventually finish the textual format dumping as well - not sure what
>>> the state of the gimple frontend branch is.
>> Work on it stalled sometime last year.
>>
>> I'm not sure if I would mix the re-arch work with the GIMPLE FE work.
>> The parser will take the most benefit out of the new structure, but I
>> think it may be better to make the front end be a sub-branch of the
>> re-architect work so we don't conflate the two.
>>
>> Some work will be independent, but it will be easier to maintain both
>> branches if they are separate.
>>
>> I do like the idea of tying the GIMPLE front end to the re-arch
>> branch, however. So, making it a sub-branch of re-arch is definitely a
>> good idea.
> Well, I thought that the GIMPLE FE itself is the easiest place where
> you can try out things given that it has to represent all features
> of GIMPLE but doesn't (necessarily) use the GIMPLE IL as its AST.
> Especially the no-trees part should be easy there.  And we can be
> convinced of the beauty of the new AST before munging all these
> wrappers into trunk with the fear that only the wrappers will stay
> and the project itself will never finish ... thus we get the ugliness
> while with the GIMPLE FE we'd at least get the GIMPLE FE ...
>
>

Well, there are no new ASTs...The gimple IL is still the same, except 
we'll have an opportunity to better formalize it, and it will be 
statically type-safe rather than dynamic like today.  Its not really 
practical to maintain it on a branch all the way until a completely 
non-tree front end can be produced. There are too many "unrelated" 
issues to progress from where we are today to a separate front end.

That said, It has become necessary to start separating out the front end 
interface components shortly as part of the work.  This can use the 
gimple FE  as a litmus test.  Maybe we can drum up some interest and 
finish development of it.

In any case, no decisions have to be made yet, and I'm not planning to 
rush it.  I'm still refining things and it'll be a couple of months 
before its worth discussing any sort of integration plan or options.  I 
plan to put together some documentation during that time and we can 
discuss its merits and fate when that is ready.

I figure we can hash things out during the month leading up to Cauldron.

Andrew