[pph] Adapting LTO streaming for front end AST saving

[pph] Adapting LTO streaming for front end AST saving

[Diego Novillo]

Over at the PPH branch we are starting to re-use the LTO streaming
routines to save front end trees.  Clearly, there are things that need
to be extended and/or replaced since LTO streaming assumes that we are
in GIMPLE.  However, there is a large intersection that I think can be
commoned out.

- ASTs do not need to concern themselves with language differences.
They are generated and consumed by cc1plus, so saving
language-dependent information is fine.
- LTO streaming has several checks and assumptions that prevent
non-gimple trees (e.g., DECL_SAVED_TREE must be NULL).

I'm looking for opinions on what would be the best approach to factor
out the common code.  So far, I have created a layer of routines and
data structures that the front end calls to manipulate PPH files.
These are wrappers on top of LTO streaming that deal with all the
sections, buffers and streams used by LTO.

I was thinking of using langhooks to do things like checks (like the
check for DECL_SAVED_TREE in
lto_output_ts_decl_non_common_tree_pointers or the asserts in
lto_get_common_nodes).  I'm expecting that there will be other things,
like handling more tree nodes in the tree streaming routines.  But
everything else seems to be already sufficiently flexible for our
needs.

Thoughts?


Thanks.  Diego.

Re: [pph] Adapting LTO streaming for front end AST saving

[Richard Guenther]

On Wed, 23 Mar 2011, Diego Novillo wrote:

> Over at the PPH branch we are starting to re-use the LTO streaming
> routines to save front end trees.  Clearly, there are things that need
> to be extended and/or replaced since LTO streaming assumes that we are
> in GIMPLE.  However, there is a large intersection that I think can be
> commoned out.
> 
> - ASTs do not need to concern themselves with language differences.
> They are generated and consumed by cc1plus, so saving
> language-dependent information is fine.
> - LTO streaming has several checks and assumptions that prevent
> non-gimple trees (e.g., DECL_SAVED_TREE must be NULL).
> 
> I'm looking for opinions on what would be the best approach to factor
> out the common code.  So far, I have created a layer of routines and
> data structures that the front end calls to manipulate PPH files.
> These are wrappers on top of LTO streaming that deal with all the
> sections, buffers and streams used by LTO.
> 
> I was thinking of using langhooks to do things like checks (like the
> check for DECL_SAVED_TREE in
> lto_output_ts_decl_non_common_tree_pointers or the asserts in
> lto_get_common_nodes).  I'm expecting that there will be other things,
> like handling more tree nodes in the tree streaming routines.  But
> everything else seems to be already sufficiently flexible for our
> needs.
> 
> Thoughts?

Yes, Micha has a load of patches cleaning up streaming and removing
unecessary abstraction.  So, why'd you need to share any of it?  I
think it would be much easier if you worked with a copy (ugh,
streaming trees again....).

Richard.

Re: [pph] Adapting LTO streaming for front end AST saving

[Diego Novillo]

On Wed, Mar 23, 2011 at 10:53, Richard Guenther <rguenther@suse.de> wrote:

> Yes, Micha has a load of patches cleaning up streaming and removing
> unecessary abstraction.  So, why'd you need to share any of it?

Removing unnecessary abstraction is fine.  But there is a bunch of
code that will be common, in particular: building string index tables,
symbol tables (the decl index), reading/writing hooks to use other
file formats (pph are not elf files) and basic trees (particularly,
decls, constants, expressions).

We could add anything that is specific to the front end we can add
with callbacks.  But duplicating all that common code seems wasteful.


Diego.

Re: [pph] Adapting LTO streaming for front end AST saving

[Jan Hubicka]

> On Wed, 23 Mar 2011, Diego Novillo wrote:
> 
> > Over at the PPH branch we are starting to re-use the LTO streaming
> > routines to save front end trees.  Clearly, there are things that need
> > to be extended and/or replaced since LTO streaming assumes that we are
> > in GIMPLE.  However, there is a large intersection that I think can be
> > commoned out.
> > 
> > - ASTs do not need to concern themselves with language differences.
> > They are generated and consumed by cc1plus, so saving
> > language-dependent information is fine.
> > - LTO streaming has several checks and assumptions that prevent
> > non-gimple trees (e.g., DECL_SAVED_TREE must be NULL).
> > 
> > I'm looking for opinions on what would be the best approach to factor
> > out the common code.  So far, I have created a layer of routines and
> > data structures that the front end calls to manipulate PPH files.
> > These are wrappers on top of LTO streaming that deal with all the
> > sections, buffers and streams used by LTO.
> > 
> > I was thinking of using langhooks to do things like checks (like the
> > check for DECL_SAVED_TREE in
> > lto_output_ts_decl_non_common_tree_pointers or the asserts in
> > lto_get_common_nodes).  I'm expecting that there will be other things,
> > like handling more tree nodes in the tree streaming routines.  But
> > everything else seems to be already sufficiently flexible for our
> > needs.
> > 
> > Thoughts?
> 
> Yes, Micha has a load of patches cleaning up streaming and removing
> unecessary abstraction.  So, why'd you need to share any of it?  I

Cool, I also have some.

> think it would be much easier if you worked with a copy (ugh,
> streaming trees again....).

I also think using same machinery for FE/gimple is a mistake.  Trees are making
life hard since they are interface in between FE<->gimplifier, part of gimple,
interface for RTL expansion and way we represent debug info.

Those are not neccesarily related things and tying them together makes 
things harder to optimize & cleanup.

I would rather see well define Gimple bitcode rather than designing common
format to handle PCHs, LTO and external templates on the top of existing trees.

Honza

> 
> Richard.

Re: [pph] Adapting LTO streaming for front end AST saving

[Diego Novillo]

On Wed, Mar 23, 2011 at 12:38, Jan Hubicka <hubicka@ucw.cz> wrote:

>> think it would be much easier if you worked with a copy (ugh,
>> streaming trees again....).
>
> I also think using same machinery for FE/gimple is a mistake.  Trees are making
> life hard since they are interface in between FE<->gimplifier, part of gimple,
> interface for RTL expansion and way we represent debug info.

Yes, but this is only part of the common machinery.  I'm actually more
interested in the other functionality.  Keeping string tables, symbol
tables, file sections, ability to stream shared pointers by creating
references, etc.

As gimple gets rid of trees, we can transition that code into the
front end.  The common code that remains, we can add hooks so each
user implements its own functionality.

> Those are not neccesarily related things and tying them together makes
> things harder to optimize & cleanup.

No, that's not the intent.  I simply want to re-use common code.  We
will have streamers for debug info, and passes have their own
streamers.  PPH is adding another streamer in the front end.  All I'm
looking for is to have a set of common streaming code I can use.

> I would rather see well define Gimple bitcode rather than designing common
> format to handle PCHs, LTO and external templates on the top of existing trees.

In as much as gimple is still using tree codes, those would remain in
common code.  As gimple gets rid of trees, we handle those in the
front end.  So, I don't think that will be a problem.


Diego.

Re: [pph] Adapting LTO streaming for front end AST saving

[Diego Novillo]

This thread spilled into IRC chatter.  I think we stopped talking past
each other now:

(2011-03-23 12:51:34) froydnj: dnovillo: gimple gets rid of trees?
how does that work?
(2011-03-23 12:52:29) dnovillo: froydnj: we've been talking about
tuplifying more, but i don't think it makes sense past certain point
(say, _CST, IDENTIFIERS, DECLs, etc)
(2011-03-23 12:53:05) dnovillo: i wasn't clear in my initial message,
i think that's what confused honza and richi
(2011-03-23 12:58:26) honza: dnovillo: What would be really nice to
have is an sane (from middle end POV) type representation.  Tree types
makes life quite hard, especially at WPA time.
(2011-03-23 12:58:35) honza: because they tie way too many things together
(2011-03-23 12:59:03) honza: but indeed gimplified/tuplified types are
quite intrusive change.
(2011-03-23 12:59:18) dnovillo: honza: agreed.  but i want to make
sure my point is clear.  i want to reuse common streaming code.  it's
more than just pickling trees.
(2011-03-23 13:00:06) dnovillo: honza: there will be things we will
never change out of trees, so that will stay in common code.  if/when
we start getting rid of trees in gimple, it will make sense to take
that code out of lto-streamer*.c and move it into tree-streamer*c or
some such.
(2011-03-23 13:00:30) dnovillo: does that clarify things?
(2011-03-23 13:02:53) honza: dnovillo: Well, I am trying to get
cleaner idea where we want to go with streaming WRT LTO and WPA in
longer term.  It is obvious that gread deal of ineffecitvity coes from
fact that we stream blindly all the data we currently have in our tree
nodes = a lot of things that is never really used in middle-end.
(2011-03-23 13:03:34) honza: plus I really think we want defined
bytecode in longer run and trees makes this hard too.
(2011-03-23 13:03:35) matz: honza: I'm currently working quite hard to
do merging while reading in the global state, and to reduce the stuff
we have to put into global state.
(2011-03-23 13:03:44) dnovillo: honza: sure.  and we want to get rid
of as much as we can there.
(2011-03-23 13:04:06) matz: But now the IPA passes are quite a road-block :/
(2011-03-23 13:04:17) dnovillo: honza: but i don't want to simply toss
out that pickling code.  i want to move it to tree-streamer*
(2011-03-23 13:04:18) honza: matz: Why?
(2011-03-23 13:04:38) dnovillo: honza: so that we can use from FEs
that want to pickle trees.
(2011-03-23 13:04:40) matz: honza: I managed to get quite far with
merging symbols before reading in the cgraph.
(2011-03-23 13:04:58) honza: matz: and now you need to merge summaries, I see ;)
(2011-03-23 13:05:19) matz: honza: Exactly.  And the info in those
summaries is currently not always self-contained.
(2011-03-23 13:05:59) honza: dnovillo: hmm, I really don't know here.
On one hand clearly I would like to see more separation of gimple and
frontend ASTs. on the other side with current state of affairs I see
that not sharing streamer code is just impractical code duplication...
(2011-03-23 13:06:02) matz: honza: For instance the ipa-prop info
doesn't stream out the number of params, and also depends on the
callee list at stream-in to be exactly as when streaming out.
(2011-03-23 13:06:24) dnovillo: honza: now you lost me.  what?
(2011-03-23 13:08:11) honza: dnovillo: for LTO it makes a lot sense
when the way you stream function bodies is well defined VM (well, like
LLVM). We can't reach this when we will just keep saving our trees
without any more strict specification
(2011-03-23 13:08:38) honza: for AST in FE you have different
requirements, even when they happens to be represented by same tree *
structure in GCC.
(2011-03-23 13:08:51) dnovillo: honza: sure.  i agree with that completely.
(2011-03-23 13:09:18) honza: that is flawed design.  So i am just
concerned that merging the streamer implemetnation still goes
indirection of reusing tree for both purposes.
(2011-03-23 13:09:55) dnovillo: honza: i am not talking about merging
anything.  i don't know how to explain this.  let me try again.
(2011-03-23 13:10:04) dnovillo: honza: there's a bunch of streaming
code that has nothing to do with pickling.
(2011-03-23 13:10:05) honza: but I guess without going all the way and
designing gimple way of implementing tuples/debug info etc. there is
not much help.
(2011-03-23 13:10:28) dnovillo: honza: that code is commonable and
re-usable from various places.
(2011-03-23 13:10:30) dnovillo: right?
(2011-03-23 13:10:59) honza: OK. Depends on how high level of
streaming logic you think of.
(2011-03-23 13:11:21) dnovillo: (keeping of string tables, streams,
sections, hooks to read/write, etc)
(2011-03-23 13:11:35) honza: yep, that is fine for sure.
(2011-03-23 13:11:39) dnovillo: honza: the *pickling* code is what you
are worried about.
(2011-03-23 13:11:50) honza: OK, I guess we are in sync then :0
(2011-03-23 13:12:00) honza: Yes, getting this separated and with
clean API is good idea.
(2011-03-23 13:12:17) matz: And even the pickling code currently can
be shared.  It's not that we currently have anything else than the
tree pickling code we currently have :)
(2011-03-23 13:12:21) dnovillo: honza: what i propose there is to
reduce gimple's dependence on it.  move it to a different place (say
tree-pickle* or tree-streamer*)
(2011-03-23 13:12:31) dnovillo: for now i can just share it since
gimple also uses it
(2011-03-23 13:12:37) dnovillo: if gimple stops using it, that's fine.
(2011-03-23 13:12:48) dnovillo: all i ask is for it not to be tossed out.
mattyb matz
(2011-03-23 13:13:01) matz: We'd have to have a replacement for it first :)
(2011-03-23 13:13:19) dnovillo: matz: yes :)  and there is some tree
pickling that gimple will always want/need to do.
(2011-03-23 13:13:27) matz: most probably, yes.
(2011-03-23 13:13:30) honza: dnovillo: Yep, sharing these things and
crafting it somewhat better is definitely an improvmenet.
(2011-03-23 13:13:38) dnovillo: everything i add to it will be via
streamer hooks.
(2011-03-23 13:14:03) dnovillo: so, i will not be modifying LTO
behaviour in any way.
(2011-03-23 13:15:10) honza: well, OK. Current LTO behaviour is
definitely not very good, but as Micha mentions, we need replacement
for that first anyway.
(2011-03-23 13:15:33) matz: We're all at GROW, right?
(2011-03-23 13:15:45) dnovillo: OK, then i guess we are on the same
page now.  yes, i'll be at GROW.
(2011-03-23 13:16:02) matz: the suse bunch too.
(2011-03-23 13:16:07) dnovillo: great.