* [pph] Adapting LTO streaming for front end AST saving @ 2011-03-23 14:47 Diego Novillo 2011-03-23 14:53 ` Richard Guenther 0 siblings, 1 reply; 6+ messages in thread From: Diego Novillo @ 2011-03-23 14:47 UTC (permalink / raw) To: gcc; +Cc: Richard Guenther, Jan Hubicka, Lawrence Crowl Over at the PPH branch we are starting to re-use the LTO streaming routines to save front end trees. Clearly, there are things that need to be extended and/or replaced since LTO streaming assumes that we are in GIMPLE. However, there is a large intersection that I think can be commoned out. - ASTs do not need to concern themselves with language differences. They are generated and consumed by cc1plus, so saving language-dependent information is fine. - LTO streaming has several checks and assumptions that prevent non-gimple trees (e.g., DECL_SAVED_TREE must be NULL). I'm looking for opinions on what would be the best approach to factor out the common code. So far, I have created a layer of routines and data structures that the front end calls to manipulate PPH files. These are wrappers on top of LTO streaming that deal with all the sections, buffers and streams used by LTO. I was thinking of using langhooks to do things like checks (like the check for DECL_SAVED_TREE in lto_output_ts_decl_non_common_tree_pointers or the asserts in lto_get_common_nodes). I'm expecting that there will be other things, like handling more tree nodes in the tree streaming routines. But everything else seems to be already sufficiently flexible for our needs. Thoughts? Thanks. Diego. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pph] Adapting LTO streaming for front end AST saving 2011-03-23 14:47 [pph] Adapting LTO streaming for front end AST saving Diego Novillo @ 2011-03-23 14:53 ` Richard Guenther 2011-03-23 15:11 ` Diego Novillo 2011-03-23 16:39 ` Jan Hubicka 0 siblings, 2 replies; 6+ messages in thread From: Richard Guenther @ 2011-03-23 14:53 UTC (permalink / raw) To: Diego Novillo; +Cc: gcc, Jan Hubicka, Lawrence Crowl On Wed, 23 Mar 2011, Diego Novillo wrote: > Over at the PPH branch we are starting to re-use the LTO streaming > routines to save front end trees. Clearly, there are things that need > to be extended and/or replaced since LTO streaming assumes that we are > in GIMPLE. However, there is a large intersection that I think can be > commoned out. > > - ASTs do not need to concern themselves with language differences. > They are generated and consumed by cc1plus, so saving > language-dependent information is fine. > - LTO streaming has several checks and assumptions that prevent > non-gimple trees (e.g., DECL_SAVED_TREE must be NULL). > > I'm looking for opinions on what would be the best approach to factor > out the common code. So far, I have created a layer of routines and > data structures that the front end calls to manipulate PPH files. > These are wrappers on top of LTO streaming that deal with all the > sections, buffers and streams used by LTO. > > I was thinking of using langhooks to do things like checks (like the > check for DECL_SAVED_TREE in > lto_output_ts_decl_non_common_tree_pointers or the asserts in > lto_get_common_nodes). I'm expecting that there will be other things, > like handling more tree nodes in the tree streaming routines. But > everything else seems to be already sufficiently flexible for our > needs. > > Thoughts? Yes, Micha has a load of patches cleaning up streaming and removing unecessary abstraction. So, why'd you need to share any of it? I think it would be much easier if you worked with a copy (ugh, streaming trees again....). Richard. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pph] Adapting LTO streaming for front end AST saving 2011-03-23 14:53 ` Richard Guenther @ 2011-03-23 15:11 ` Diego Novillo 2011-03-23 16:39 ` Jan Hubicka 1 sibling, 0 replies; 6+ messages in thread From: Diego Novillo @ 2011-03-23 15:11 UTC (permalink / raw) To: Richard Guenther; +Cc: gcc, Jan Hubicka, Lawrence Crowl On Wed, Mar 23, 2011 at 10:53, Richard Guenther <rguenther@suse.de> wrote: > Yes, Micha has a load of patches cleaning up streaming and removing > unecessary abstraction. So, why'd you need to share any of it? Removing unnecessary abstraction is fine. But there is a bunch of code that will be common, in particular: building string index tables, symbol tables (the decl index), reading/writing hooks to use other file formats (pph are not elf files) and basic trees (particularly, decls, constants, expressions). We could add anything that is specific to the front end we can add with callbacks. But duplicating all that common code seems wasteful. Diego. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pph] Adapting LTO streaming for front end AST saving 2011-03-23 14:53 ` Richard Guenther 2011-03-23 15:11 ` Diego Novillo @ 2011-03-23 16:39 ` Jan Hubicka 2011-03-23 16:49 ` Diego Novillo 1 sibling, 1 reply; 6+ messages in thread From: Jan Hubicka @ 2011-03-23 16:39 UTC (permalink / raw) To: Richard Guenther; +Cc: Diego Novillo, gcc, Jan Hubicka, Lawrence Crowl > On Wed, 23 Mar 2011, Diego Novillo wrote: > > > Over at the PPH branch we are starting to re-use the LTO streaming > > routines to save front end trees. Clearly, there are things that need > > to be extended and/or replaced since LTO streaming assumes that we are > > in GIMPLE. However, there is a large intersection that I think can be > > commoned out. > > > > - ASTs do not need to concern themselves with language differences. > > They are generated and consumed by cc1plus, so saving > > language-dependent information is fine. > > - LTO streaming has several checks and assumptions that prevent > > non-gimple trees (e.g., DECL_SAVED_TREE must be NULL). > > > > I'm looking for opinions on what would be the best approach to factor > > out the common code. So far, I have created a layer of routines and > > data structures that the front end calls to manipulate PPH files. > > These are wrappers on top of LTO streaming that deal with all the > > sections, buffers and streams used by LTO. > > > > I was thinking of using langhooks to do things like checks (like the > > check for DECL_SAVED_TREE in > > lto_output_ts_decl_non_common_tree_pointers or the asserts in > > lto_get_common_nodes). I'm expecting that there will be other things, > > like handling more tree nodes in the tree streaming routines. But > > everything else seems to be already sufficiently flexible for our > > needs. > > > > Thoughts? > > Yes, Micha has a load of patches cleaning up streaming and removing > unecessary abstraction. So, why'd you need to share any of it? I Cool, I also have some. > think it would be much easier if you worked with a copy (ugh, > streaming trees again....). I also think using same machinery for FE/gimple is a mistake. Trees are making life hard since they are interface in between FE<->gimplifier, part of gimple, interface for RTL expansion and way we represent debug info. Those are not neccesarily related things and tying them together makes things harder to optimize & cleanup. I would rather see well define Gimple bitcode rather than designing common format to handle PCHs, LTO and external templates on the top of existing trees. Honza > > Richard. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pph] Adapting LTO streaming for front end AST saving 2011-03-23 16:39 ` Jan Hubicka @ 2011-03-23 16:49 ` Diego Novillo 2011-03-23 17:58 ` Diego Novillo 0 siblings, 1 reply; 6+ messages in thread From: Diego Novillo @ 2011-03-23 16:49 UTC (permalink / raw) To: Jan Hubicka; +Cc: Richard Guenther, gcc, Jan Hubicka, Lawrence Crowl On Wed, Mar 23, 2011 at 12:38, Jan Hubicka <hubicka@ucw.cz> wrote: >> think it would be much easier if you worked with a copy (ugh, >> streaming trees again....). > > I also think using same machinery for FE/gimple is a mistake. Trees are making > life hard since they are interface in between FE<->gimplifier, part of gimple, > interface for RTL expansion and way we represent debug info. Yes, but this is only part of the common machinery. I'm actually more interested in the other functionality. Keeping string tables, symbol tables, file sections, ability to stream shared pointers by creating references, etc. As gimple gets rid of trees, we can transition that code into the front end. The common code that remains, we can add hooks so each user implements its own functionality. > Those are not neccesarily related things and tying them together makes > things harder to optimize & cleanup. No, that's not the intent. I simply want to re-use common code. We will have streamers for debug info, and passes have their own streamers. PPH is adding another streamer in the front end. All I'm looking for is to have a set of common streaming code I can use. > I would rather see well define Gimple bitcode rather than designing common > format to handle PCHs, LTO and external templates on the top of existing trees. In as much as gimple is still using tree codes, those would remain in common code. As gimple gets rid of trees, we handle those in the front end. So, I don't think that will be a problem. Diego. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pph] Adapting LTO streaming for front end AST saving 2011-03-23 16:49 ` Diego Novillo @ 2011-03-23 17:58 ` Diego Novillo 0 siblings, 0 replies; 6+ messages in thread From: Diego Novillo @ 2011-03-23 17:58 UTC (permalink / raw) To: Jan Hubicka; +Cc: Richard Guenther, gcc, Jan Hubicka, Lawrence Crowl This thread spilled into IRC chatter. I think we stopped talking past each other now: (2011-03-23 12:51:34) froydnj: dnovillo: gimple gets rid of trees? how does that work? (2011-03-23 12:52:29) dnovillo: froydnj: we've been talking about tuplifying more, but i don't think it makes sense past certain point (say, _CST, IDENTIFIERS, DECLs, etc) (2011-03-23 12:53:05) dnovillo: i wasn't clear in my initial message, i think that's what confused honza and richi (2011-03-23 12:58:26) honza: dnovillo: What would be really nice to have is an sane (from middle end POV) type representation. Tree types makes life quite hard, especially at WPA time. (2011-03-23 12:58:35) honza: because they tie way too many things together (2011-03-23 12:59:03) honza: but indeed gimplified/tuplified types are quite intrusive change. (2011-03-23 12:59:18) dnovillo: honza: agreed. but i want to make sure my point is clear. i want to reuse common streaming code. it's more than just pickling trees. (2011-03-23 13:00:06) dnovillo: honza: there will be things we will never change out of trees, so that will stay in common code. if/when we start getting rid of trees in gimple, it will make sense to take that code out of lto-streamer*.c and move it into tree-streamer*c or some such. (2011-03-23 13:00:30) dnovillo: does that clarify things? (2011-03-23 13:02:53) honza: dnovillo: Well, I am trying to get cleaner idea where we want to go with streaming WRT LTO and WPA in longer term. It is obvious that gread deal of ineffecitvity coes from fact that we stream blindly all the data we currently have in our tree nodes = a lot of things that is never really used in middle-end. (2011-03-23 13:03:34) honza: plus I really think we want defined bytecode in longer run and trees makes this hard too. (2011-03-23 13:03:35) matz: honza: I'm currently working quite hard to do merging while reading in the global state, and to reduce the stuff we have to put into global state. (2011-03-23 13:03:44) dnovillo: honza: sure. and we want to get rid of as much as we can there. (2011-03-23 13:04:06) matz: But now the IPA passes are quite a road-block :/ (2011-03-23 13:04:17) dnovillo: honza: but i don't want to simply toss out that pickling code. i want to move it to tree-streamer* (2011-03-23 13:04:18) honza: matz: Why? (2011-03-23 13:04:38) dnovillo: honza: so that we can use from FEs that want to pickle trees. (2011-03-23 13:04:40) matz: honza: I managed to get quite far with merging symbols before reading in the cgraph. (2011-03-23 13:04:58) honza: matz: and now you need to merge summaries, I see ;) (2011-03-23 13:05:19) matz: honza: Exactly. And the info in those summaries is currently not always self-contained. (2011-03-23 13:05:59) honza: dnovillo: hmm, I really don't know here. On one hand clearly I would like to see more separation of gimple and frontend ASTs. on the other side with current state of affairs I see that not sharing streamer code is just impractical code duplication... (2011-03-23 13:06:02) matz: honza: For instance the ipa-prop info doesn't stream out the number of params, and also depends on the callee list at stream-in to be exactly as when streaming out. (2011-03-23 13:06:24) dnovillo: honza: now you lost me. what? (2011-03-23 13:08:11) honza: dnovillo: for LTO it makes a lot sense when the way you stream function bodies is well defined VM (well, like LLVM). We can't reach this when we will just keep saving our trees without any more strict specification (2011-03-23 13:08:38) honza: for AST in FE you have different requirements, even when they happens to be represented by same tree * structure in GCC. (2011-03-23 13:08:51) dnovillo: honza: sure. i agree with that completely. (2011-03-23 13:09:18) honza: that is flawed design. So i am just concerned that merging the streamer implemetnation still goes indirection of reusing tree for both purposes. (2011-03-23 13:09:55) dnovillo: honza: i am not talking about merging anything. i don't know how to explain this. let me try again. (2011-03-23 13:10:04) dnovillo: honza: there's a bunch of streaming code that has nothing to do with pickling. (2011-03-23 13:10:05) honza: but I guess without going all the way and designing gimple way of implementing tuples/debug info etc. there is not much help. (2011-03-23 13:10:28) dnovillo: honza: that code is commonable and re-usable from various places. (2011-03-23 13:10:30) dnovillo: right? (2011-03-23 13:10:59) honza: OK. Depends on how high level of streaming logic you think of. (2011-03-23 13:11:21) dnovillo: (keeping of string tables, streams, sections, hooks to read/write, etc) (2011-03-23 13:11:35) honza: yep, that is fine for sure. (2011-03-23 13:11:39) dnovillo: honza: the *pickling* code is what you are worried about. (2011-03-23 13:11:50) honza: OK, I guess we are in sync then :0 (2011-03-23 13:12:00) honza: Yes, getting this separated and with clean API is good idea. (2011-03-23 13:12:17) matz: And even the pickling code currently can be shared. It's not that we currently have anything else than the tree pickling code we currently have :) (2011-03-23 13:12:21) dnovillo: honza: what i propose there is to reduce gimple's dependence on it. move it to a different place (say tree-pickle* or tree-streamer*) (2011-03-23 13:12:31) dnovillo: for now i can just share it since gimple also uses it (2011-03-23 13:12:37) dnovillo: if gimple stops using it, that's fine. (2011-03-23 13:12:48) dnovillo: all i ask is for it not to be tossed out. mattyb matz (2011-03-23 13:13:01) matz: We'd have to have a replacement for it first :) (2011-03-23 13:13:19) dnovillo: matz: yes :) and there is some tree pickling that gimple will always want/need to do. (2011-03-23 13:13:27) matz: most probably, yes. (2011-03-23 13:13:30) honza: dnovillo: Yep, sharing these things and crafting it somewhat better is definitely an improvmenet. (2011-03-23 13:13:38) dnovillo: everything i add to it will be via streamer hooks. (2011-03-23 13:14:03) dnovillo: so, i will not be modifying LTO behaviour in any way. (2011-03-23 13:15:10) honza: well, OK. Current LTO behaviour is definitely not very good, but as Micha mentions, we need replacement for that first anyway. (2011-03-23 13:15:33) matz: We're all at GROW, right? (2011-03-23 13:15:45) dnovillo: OK, then i guess we are on the same page now. yes, i'll be at GROW. (2011-03-23 13:16:02) matz: the suse bunch too. (2011-03-23 13:16:07) dnovillo: great. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-03-23 17:18 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-03-23 14:47 [pph] Adapting LTO streaming for front end AST saving Diego Novillo 2011-03-23 14:53 ` Richard Guenther 2011-03-23 15:11 ` Diego Novillo 2011-03-23 16:39 ` Jan Hubicka 2011-03-23 16:49 ` Diego Novillo 2011-03-23 17:58 ` Diego Novillo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).