[gc-improv] Permanent vs function RTL obstack fix

[gc-improv] Permanent vs function RTL obstack fix

[Laurynas Biveinis]

[-- Attachment #1: Type: text/plain, Size: 187 bytes --]

Fixes a bunch of C testsuite failures. Committed to gc-improv.

2011-04-07  Laurynas Biveinis  <laurynas.biveinis@gmail.com>

	* stmt.c (label_rtx): Allocate RTX in permanent RTL memory.

[-- Attachment #2: 10-label-rtx.patch --]
[-- Type: text/x-patch, Size: 458 bytes --]

Index: stmt.c
===================================================================
--- stmt.c	(revision 170593)
+++ stmt.c	(working copy)
@@ -139,7 +139,10 @@
 
   if (!DECL_RTL_SET_P (label))
     {
-      rtx r = gen_label_rtx ();
+      rtx r;
+      use_rtl_permanent_mem ();
+      r = gen_label_rtx ();
+      use_rtl_function_mem ();
       SET_DECL_RTL (label, r);
       if (FORCED_LABEL (label) || DECL_NONLOCAL (label))
 	LABEL_PRESERVE_P (r) = 1;

Re: [gc-improv] Permanent vs function RTL obstack fix

[Steven Bosscher]

On Thu, Apr 7, 2011 at 8:17 AM, Laurynas Biveinis
<laurynas.biveinis@gmail.com> wrote:
> Fixes a bunch of C testsuite failures. Committed to gc-improv.
>
> 2011-04-07  Laurynas Biveinis  <laurynas.biveinis@gmail.com>
>
>        * stmt.c (label_rtx): Allocate RTX in permanent RTL memory.

That looks strange, labels should be function specific, except
non-local labels. Maybe you can get away with allocating DECL_NONLOCAL
labels on the permanent rtl obstack?

Perhaps a third, per-translation-unit obstack is necessary?

Ciao!
Steven

Re: [gc-improv] Permanent vs function RTL obstack fix

[Laurynas Biveinis]

2011/4/8 Steven Bosscher <stevenb.gcc@gmail.com>:
>>        * stmt.c (label_rtx): Allocate RTX in permanent RTL memory.
>
> That looks strange, labels should be function specific, except
> non-local labels. Maybe you can get away with allocating DECL_NONLOCAL
> labels on the permanent rtl obstack?

That's a good idea, I will try it once things are stable on the branch.

> Perhaps a third, per-translation-unit obstack is necessary?

Perhaps. After I finish with permanent rtl obstack, I will measure how
large it gets and if it's worthwhile to split out
per-translation-obstack out of it.

Thanks,
-- 
Laurynas

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/08/11 07:21, Laurynas Biveinis wrote:
> 2011/4/8 Steven Bosscher <stevenb.gcc@gmail.com>:
>>>        * stmt.c (label_rtx): Allocate RTX in permanent RTL memory.
>>
>> That looks strange, labels should be function specific, except
>> non-local labels. Maybe you can get away with allocating DECL_NONLOCAL
>> labels on the permanent rtl obstack?
> 
> That's a good idea, I will try it once things are stable on the branch.
> 
>> Perhaps a third, per-translation-unit obstack is necessary?
> 
> Perhaps. After I finish with permanent rtl obstack, I will measure how
> large it gets and if it's worthwhile to split out
> per-translation-obstack out of it.
And then you'll want a per-statement obstack, then per-expression
obstack, and before you know it, GCC looks much like it did 20 years ago.

Sigh.

jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNnx1iAAoJEBRtltQi2kC7YJwIAI02Q9vDzyqmvcZGe25jb93A
/esh3wWlDSV8TKKwBJkRt6D8PDlRqqaNa5owcEm0iKAYN3lcBo+Q0nYGqoYnN5b1
YVIiajfR809L2NeQOCjVwYpZotQa0MOGuWRrpIRYRCMEOFgljGzwa1utj5qOUXEr
s27d0vNnE5ShuJAOt+uJlDU9xgaNTHrCcJBwacKGBBxKRba8nV2wQ9uJPjypoAeX
Fza9uJQRqe9rmOheUZsPIthNeZbUUCb0jVzGbivUzpa6gyLQgLry66IvnuAg7BPP
Xo2lQh3CDZY3QJnktuiDA201RvPRlxhoA+8jTfMMPi+/NexS3+GrLA9fTC+QkE0=
=hQs1
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Richard Guenther]

On Fri, Apr 8, 2011 at 4:36 PM, Jeff Law <law@redhat.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 04/08/11 07:21, Laurynas Biveinis wrote:
>> 2011/4/8 Steven Bosscher <stevenb.gcc@gmail.com>:
>>>>        * stmt.c (label_rtx): Allocate RTX in permanent RTL memory.
>>>
>>> That looks strange, labels should be function specific, except
>>> non-local labels. Maybe you can get away with allocating DECL_NONLOCAL
>>> labels on the permanent rtl obstack?
>>
>> That's a good idea, I will try it once things are stable on the branch.
>>
>>> Perhaps a third, per-translation-unit obstack is necessary?
>>
>> Perhaps. After I finish with permanent rtl obstack, I will measure how
>> large it gets and if it's worthwhile to split out
>> per-translation-obstack out of it.
> And then you'll want a per-statement obstack, then per-expression
> obstack, and before you know it, GCC looks much like it did 20 years ago.
>
> Sigh.

I remember it was fast at that time though ;)

Richard.

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/08/11 08:39, Richard Guenther wrote:
> On Fri, Apr 8, 2011 at 4:36 PM, Jeff Law <law@redhat.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 04/08/11 07:21, Laurynas Biveinis wrote:
>>> 2011/4/8 Steven Bosscher <stevenb.gcc@gmail.com>:
>>>>>        * stmt.c (label_rtx): Allocate RTX in permanent RTL memory.
>>>>
>>>> That looks strange, labels should be function specific, except
>>>> non-local labels. Maybe you can get away with allocating DECL_NONLOCAL
>>>> labels on the permanent rtl obstack?
>>>
>>> That's a good idea, I will try it once things are stable on the branch.
>>>
>>>> Perhaps a third, per-translation-unit obstack is necessary?
>>>
>>> Perhaps. After I finish with permanent rtl obstack, I will measure how
>>> large it gets and if it's worthwhile to split out
>>> per-translation-obstack out of it.
>> And then you'll want a per-statement obstack, then per-expression
>> obstack, and before you know it, GCC looks much like it did 20 years ago.
>>
>> Sigh.
> 
> I remember it was fast at that time though ;)
Yea, fast to segfault dereferencing a dangling pointer due to placing an
object on the wrong obstack.  And fast because we didn't have a good way
to keep most expressions beyond the lifetime of a single statement thus
preventing any kind of higher level analysis and optimization.

Jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNnx7OAAoJEBRtltQi2kC7cLQH/RDU86grC6BTnVc9iG5KE4zu
B+YP0+42dJfrinjL0cqkPeEprTr5j144XJYujsxT49f7G+qu1qX+gzyfowgTdOwb
dIfhnWXoC9d09E0nXD+mLVwHwNwsrkv1loN501wryPksSOsgjxjhwc7TCW8WnW93
2sjrpKLg0qvy+/8KH0RTA80q9xehGkfqux3y4ZbNLsPBreYp+jhTzsVqjxk+T0F7
Ges5PFLjXwQdbx+iFBStQhlZHf8A/3TRXnrfllPRRl37bzR56Jnza2ag8EkVHwXi
5bNQ+YWqJXHiKG2nInARTNGnJXQU8XMGYYaZTx1Myl0c8osAlfu0QwLHrJn+Few=
=4X+7
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Steven Bosscher]

On Fri, Apr 8, 2011 at 4:36 PM, Jeff Law <law@redhat.com> wrote:
>>> Perhaps a third, per-translation-unit obstack is necessary?
>>
>> Perhaps. After I finish with permanent rtl obstack, I will measure how
>> large it gets and if it's worthwhile to split out
>> per-translation-obstack out of it.
> And then you'll want a per-statement obstack, then per-expression
> obstack, and before you know it, GCC looks much like it did 20 years ago.

I don't think that's a fair comment, or a real concern. Unlike 20
years ago, GCC now has a reasonable idea of how long RTX objects live.
I can think of only four states for RTX in memory:

1. RTL that lives throughout the compilation, things like shared
constants. RTX objects in this group are only in GC memory now because
one RTX object on GC-memory cannot point to a non-GC RTX object (the
mark-and-sweep process expects all objects to be in GC memory).

2. RTL from back-end (re-)initialization. Stuff in this group should
go on a permanent obstack that is only removed if the back end is
re-initialized (e.g. for the MIPS back end, or for future multi-target
support).

3. RTL per translation unit. I am not sure what falls in this class.
Things like RTL for global variables maybe? RTL that may escape from
functions? Apparently the latter happens with labels, I hope it only
happens with nested functions, really. Got some insights to share
here? ;-)

4. RTL per function. GCC expands one GIMPLE function at a time, and
the idea is to initialize the RTL obstack once when expanding starts,
let it grow until final, and blow it away after final. Unlike 20 years
ago, this obstack is never rolled back during RTL passes. This relies
on generating not too much garbage, but memory for per-function RTL
should be dwarfed by per-translation unit GIMPLE anyway.

Jeff, few people from 20 years ago are still around, so perhaps you
can help a bit to help make things are going in the right direction.
;-)

Ciao!
Steven

Re: [gc-improv] Permanent vs function RTL obstack fix

[Laurynas Biveinis]

2011/4/9 Steven Bosscher <stevenb.gcc@gmail.com>:
> 4. RTL per function. GCC expands one GIMPLE function at a time, and
> the idea is to initialize the RTL obstack once when expanding starts,
> let it grow until final, and blow it away after final. Unlike 20 years
> ago, this obstack is never rolled back during RTL passes. This relies
> on generating not too much garbage, but memory for per-function RTL
> should be dwarfed by per-translation unit GIMPLE anyway.

Well, I have plans to see if it is worthwhile for pass like combine to
rollback the function obstack to do away with scratch RTL. Of course
this depends, on how much memory can be saved by doing this - in
comparison to current GC.

-- 
Laurynas

Re: [gc-improv] Permanent vs function RTL obstack fix

[Basile Starynkevitch]

On Sun, 10 Apr 2011 21:27:10 +0300
Laurynas Biveinis <laurynas.biveinis@gmail.com> wrote:

> 2011/4/9 Steven Bosscher <stevenb.gcc@gmail.com>:
> > 4. RTL per function. GCC expands one GIMPLE function at a time, and
> > the idea is to initialize the RTL obstack once when expanding starts,
> > let it grow until final, and blow it away after final. Unlike 20 years
> > ago, this obstack is never rolled back during RTL passes. This relies
> > on generating not too much garbage, but memory for per-function RTL
> > should be dwarfed by per-translation unit GIMPLE anyway.
> 
> Well, I have plans to see if it is worthwhile for pass like combine to
> rollback the function obstack to do away with scratch RTL. Of course
> this depends, on how much memory can be saved by doing this - in
> comparison to current GC.


I respect a lot Laurynas' work, but my general personal feeling & wish
is on the contrary that in the long term, more GCC data should be
garbage collected, and that GCC's garbage collector should be better.

However, I tend to believe that Laurynas cleanup on RTL could be
helpful.

And in the long run, I would imagine that making RTL data garbage
collectable again would just be a matter of adding GTY annotations
somewhere.

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/10/11 12:27, Laurynas Biveinis wrote:
> 2011/4/9 Steven Bosscher <stevenb.gcc@gmail.com>:
>> 4. RTL per function. GCC expands one GIMPLE function at a time, and
>> the idea is to initialize the RTL obstack once when expanding starts,
>> let it grow until final, and blow it away after final. Unlike 20 years
>> ago, this obstack is never rolled back during RTL passes. This relies
>> on generating not too much garbage, but memory for per-function RTL
>> should be dwarfed by per-translation unit GIMPLE anyway.
> 
> Well, I have plans to see if it is worthwhile for pass like combine to
> rollback the function obstack to do away with scratch RTL. Of course
> this depends, on how much memory can be saved by doing this - in
> comparison to current GC.
One of the fundamental problems you have to watch out for when dealing
with scratch objects is how to handle the case when you belatedly
realize you want the object to have a longer lifetime.

The obvious solution is you copy the object, but then you have to be
able to distinguish within the object, what fields point to other
temporary objects vs permanent objects so that you can copy the
referenced temporary objects, but not the permanent objects (other parts
of the compiler may expect those permanent objects to be unique/shared).
   Not fun, not at all fun.  Been there, done that.

jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNo150AAoJEBRtltQi2kC7Yb0H+gPB2ub86sbkGx0ee5ry1YYc
ww222sMb+YP6wQ/fIxi/tYXdfxcouJ4/SuiC03tYNAwvfONZuQNrZKyEwu5cPXIh
OMQYhV5pxDLfaRmpklBZWfYndStlWUYrmZAHPLI0zO5BCxgQaiZx/A6zjg6lPNY/
VnMKpdF1Tp0M03tJ1JNqMlTKrP5mkV/gAsjQVyjAM1DJntLYIqxqmm4tinaAJXUf
pYDHecQgV/ZvngzCI8XydNZXEk/GnrcVCnyByO1BBLOzCom63+WXzXAGE9HPTOvj
fNncNhOFz5rAzowiddngkoxlnPNGJKbuypprt/U5u17j91MlGADRDTkcapvI21A=
=PwyX
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Mike Stump]

On Apr 11, 2011, at 1:03 PM, Jeff Law wrote:
> The obvious solution is you copy the object, but then you have to be
> able to distinguish within the object, what fields point to other
> temporary objects vs permanent objects so that you can copy the
> referenced temporary objects, but not the permanent objects (other parts
> of the compiler may expect those permanent objects to be unique/shared).
>   Not fun, not at all fun.  Been there, done that.

And then someone tells you that you can't copy...  Usually a nasty bug report much later.  So the choice is then, not solve a problem, or make everything permanent or add GC (back).  :-)  Been there, done that too.

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/11/11 18:21, Mike Stump wrote:
> On Apr 11, 2011, at 1:03 PM, Jeff Law wrote:
>> The obvious solution is you copy the object, but then you have to be
>> able to distinguish within the object, what fields point to other
>> temporary objects vs permanent objects so that you can copy the
>> referenced temporary objects, but not the permanent objects (other parts
>> of the compiler may expect those permanent objects to be unique/shared).
>>   Not fun, not at all fun.  Been there, done that.
> 
> And then someone tells you that you can't copy...  Usually a nasty bug report much later.  So the choice is then, not solve a problem, or make everything permanent or add GC (back).  :-)  Been there, done that too.
Right.  Hence the old hack to mark the temporary obstacks and make them
permanent (wasting an enormous amount of space in the process).  I don't
recall the name of that function, but it certainly made me barf.

And yes, I remember all too well the problems with deep copying having
written one of the earliest tree deep copy routines to support
- -fcombine-statics .  It was the source of more problems than I could
ever count -- all related to objects allocated to different obstacks
hanging off a single tree node or objects of two different types being
stored into the same field within tree nodes.  That was circa 1992/1993.


These were the kind of problems that ultimately led to the GC system we
have today.  There are clearly things that can be better handled with
different allocation models and I'll support moving to better models
where it makes sense.  Where it doesn't, obviously I won't support it.

jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNo77qAAoJEBRtltQi2kC7X0UH/0n1ydyhcvfjjoqhQCIwjf9J
jp3MOel6I1RTjAMm69N0ZqCP5t8AuHIIJdlrSQ1Aworx0gNl4+VjbEXJEM6dcQB4
enJ5mYmPXQ3EMDEi6C/uPIGwPcPsqO9sFTe91plQIsI7m6OTRjM4I/aS9XmRQ/uQ
8qBhDTSY3sTn4rOTUvqKfvdStP9B/Sf2tYpYbHgm/tfLRQ5UFHWpZFckcyntHFs8
rTxcA3xKlVNSg2D9+3CfM8KVWqMVXeHgB+tlB2Q6L7/TAeCqGHABE5MxeB1+HDJN
QoB0HCMCt7t6sqGYk6S5xayyfJUrWvt5XBiVa8oXrP9sVh1iNM8mY33zzwPE2M4=
=9inU
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Steven Bosscher]

On Tue, Apr 12, 2011 at 4:54 AM, Jeff Law <law@redhat.com> wrote:
> On 04/11/11 18:21, Mike Stump wrote:
>> On Apr 11, 2011, at 1:03 PM, Jeff Law wrote:
>>> The obvious solution is you copy the object, but then you have to be
>>> able to distinguish within the object, what fields point to other
>>> temporary objects vs permanent objects so that you can copy the
>>> referenced temporary objects, but not the permanent objects (other parts
>>> of the compiler may expect those permanent objects to be unique/shared).
>>>   Not fun, not at all fun.  Been there, done that.
>>
>> And then someone tells you that you can't copy...  Usually a nasty bug report much later.  So the choice is then, not solve a problem, or make everything permanent or add GC (back).  :-)  Been there, done that too.
> Right.  Hence the old hack to mark the temporary obstacks and make them
> permanent (wasting an enormous amount of space in the process).  I don't
> recall the name of that function, but it certainly made me barf.
>
> And yes, I remember all too well the problems with deep copying having
> written one of the earliest tree deep copy routines to support
> - -fcombine-statics .  It was the source of more problems than I could
> ever count -- all related to objects allocated to different obstacks
> hanging off a single tree node or objects of two different types being
> stored into the same field within tree nodes.  That was circa 1992/1993.
>
>
> These were the kind of problems that ultimately led to the GC system we
> have today.  There are clearly things that can be better handled with
> different allocation models and I'll support moving to better models
> where it makes sense.  Where it doesn't, obviously I won't support it.

I am unsure how to interpret that last sentence. Do you currently
believe that the approach of 2 or 3 obstacks for RTL is a better model
that makes sense, or not?

I think all these comments from you "old guys" ;-) are more
discouraging than fair. What Laurynas and Bernd have done, is nothing
like the old obstacks of GCC2. I am also a bit surprised that all
these fears only come up now that someone actually tries to do what
Zack already suggested in 2003, around the time that unit-at-a-time
compilation became the default mode.

Anyway, please do give this idea a chance at least. There is a big
difference between the pre-GC obstacks and what Laurynas is now trying
(not just in terms of memory management, but in the entire compilation
process!) and apart from the name "obstacks" this idea and the pre-GC
memory management in GCC have nothing in common.

Ciao!
Steven

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jakub Jelinek]

On Tue, Apr 12, 2011 at 08:33:56AM +0200, Steven Bosscher wrote:
> I think all these comments from you "old guys" ;-) are more
> discouraging than fair. What Laurynas and Bernd have done, is nothing

It is IMHO completely fair to point that the risks this brings in
a huge maintainance nightmare are very high.

	Jakub

Re: [gc-improv] Permanent vs function RTL obstack fix

[Steven Bosscher]

On Tue, Apr 12, 2011 at 9:01 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Tue, Apr 12, 2011 at 08:33:56AM +0200, Steven Bosscher wrote:
>> I think all these comments from you "old guys" ;-) are more
>> discouraging than fair. What Laurynas and Bernd have done, is nothing
>
> It is IMHO completely fair to point that the risks this brings in
> a huge maintainance nightmare are very high.

And IM-equally-HO it is completely unfair to talk about risks in any
situation where there is nothing yet to talk about! Give it a chance
and wait for something that's more than just an idea, and then assess
the risks based on an implementation.

Or just say "this won't fly" now so that people who would like to work
on this can turn their attention to something else. Also fine. Really.

Ciao!
Steven

Re: [gc-improv] Permanent vs function RTL obstack fix

[Mike Stump]

On Apr 12, 2011, at 1:45 AM, Steven Bosscher wrote:
> On Tue, Apr 12, 2011 at 9:01 AM, Jakub Jelinek <jakub@redhat.com> wrote:
>> On Tue, Apr 12, 2011 at 08:33:56AM +0200, Steven Bosscher wrote:
>>> I think all these comments from you "old guys" ;-) are more
>>> discouraging than fair. What Laurynas and Bernd have done, is nothing
>> 
>> It is IMHO completely fair to point that the risks this brings in
>> a huge maintainance nightmare are very high.
> 
> And IM-equally-HO it is completely unfair to talk about risks in any
> situation where there is nothing yet to talk about! Give it a chance
> and wait for something that's more than just an idea, and then assess
> the risks based on an implementation.

The problem is that in the olden days, people thought it would work, then, they ran into, oh, but the lifetime is wrong.  Oh, but I don't know what the lifetime will be before I start allocating.  Oh, what do you mean I can't copy it...  All these things happened after real bug reports and great difficulty in tracking down real problems.  A review might not be able to spot the things that won't work.  We don't want to discourage, only to let everyone know what our experience is with these sorts of problems.  So, for example, if one is predicating copying objects to solve lifetime problems, we can expound what some of the problems were in the past.  They could be relevant, they might not be not relevant; also, they just might cause people to explore an area that might be a problem, find and address it sooner.

Also, actual patches have been posted since Jan 27th...  I don't see that all the problems are cleverly avoided.  So, let's take one:

>   - If objects stored in PCH have pointers pointing outside of PCH-able/GC-managed memory, these become wild pointers on PCH read even with GTY((skip)) applied properly. However, not all GTY((skip)) pointers point outside of PCH-able memory, so I have overloaded GTY((deletable)) option to reset such fields to NULL on PCH write. This is only a part of the fix as logic needs to be reviewed what to do with these NULLs after the PCH read. In case of DECL_RTL, the NULL in the field causes re-creation of rtx anew, which I think is the right thing.

Not an example of a warm fuzzy.  When you combine this with the entire idea of PCH, and that is to slosh to disk and back, the entire state of the compiler, essentially, unmodified, we run into a conceptual gap.  Now, why did PCH have that design, so that we could limit the number of pch bugs and come up with a robust design that just works.  This is a useful property.

Now, one can audit DECL_RTL, to see if it matters, but there are roughly 344 places to check.  I glanced at a few, to see if I can spot problems.  I find it hard to know if any particular one will be a problem.  At least some of them can't be found by testing.  Presently we do what I'd call smoke testing for pch.  Deeper issues aren't covered.  So, who then will spot the issues?

Re: [gc-improv] Permanent vs function RTL obstack fix

[Steven Bosscher]

On Tue, Apr 12, 2011 at 12:44 PM, Mike Stump <mikestump@comcast.net> wrote:
>>   - If objects stored in PCH have pointers pointing outside of PCH-able/GC-managed memory, these become wild pointers on PCH read even with GTY((skip)) applied properly. However, not all GTY((skip)) pointers point outside of PCH-able memory, so I have overloaded GTY((deletable)) option to reset such fields to NULL on PCH write. This is only a part of the fix as logic needs to be reviewed what to do with these NULLs after the PCH read. In case of DECL_RTL, the NULL in the field causes re-creation of rtx anew, which I think is the right thing.
>
> Not an example of a warm fuzzy.  When you combine this with the entire idea of PCH, and that is to slosh to disk and back, the entire state of the compiler, essentially, unmodified, we run into a conceptual gap.  Now, why did PCH have that design, so that we could limit the number of pch bugs and come up with a robust design that just works.  This is a useful property.

I've debuged PCH bugs and they were usually a _result_ of the poor
design. Missing GTY on non-pointer objects for example. There is
nothing robust about the design of PCH and it certainly does not "just
work".

But, two things:

1. RTL and PCH should be completely separate. No DECL_RTL should ever
end up in a PCH because DECL_RTL is not created in the front end (or
should not be, it'd be a bug)

2. PCH as we know it will probably be gone soon (Google's pph stuff)

Ciao!
Steven

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/11 02:45, Steven Bosscher wrote:
> On Tue, Apr 12, 2011 at 9:01 AM, Jakub Jelinek <jakub@redhat.com> wrote:
>> On Tue, Apr 12, 2011 at 08:33:56AM +0200, Steven Bosscher wrote:
>>> I think all these comments from you "old guys" ;-) are more
>>> discouraging than fair. What Laurynas and Bernd have done, is nothing
>>
>> It is IMHO completely fair to point that the risks this brings in
>> a huge maintainance nightmare are very high.
> 
> And IM-equally-HO it is completely unfair to talk about risks in any
> situation where there is nothing yet to talk about! Give it a chance
> and wait for something that's more than just an idea, and then assess
> the risks based on an implementation.
Given that we've already got a goodly amount of experience with obstack
and GC based mechanisms for allocation, I think it is completely fair
and wise to discuss the known risk/reward for both.
> 
> Or just say "this won't fly" now so that people who would like to work
> on this can turn their attention to something else. Also fine. Really.
I have serious concerns about reverting to obstacks as our main memory
allocation approach for tree & rtl data.  However, I also realize that
there are objects where obstacks make sense and I realize some of those
objects may currently be hanging off tree or RTL structures.

With that in mind, I'm all for a critical examination of our data
structures, their lifetimes and what allocation model works best.  From
that I would expect that we'll find some cases where an obstack model
works better and the structures will move to that model.

jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNpGmEAAoJEBRtltQi2kC7cjUIALtwCRbcsxABIx6xtWlZIHRy
vPfB8pc6u+IhIrbu/T2qZjUoP6bq5UQIgCPIy3o7qSgJ5Qd8NvYJQ5nMHMyPZvX2
MDzyTJcMB1OUsoZhgVwTdZoL1aSp3ARopruDM2DNc9zo8DXP5YFcn2w6bXaASjr5
jENRoiqTe6hLgJXQZT7QQObusR6gM4Off78Hs/vGlaOmeXMSONfMTxms1ya0ROQe
h+YyXpQuLsUDdIO9wSHfeD0/73er8fLYqlcJ77GPUDK907oVtr4bKUWOirwX9QL2
vRwMur93cVjPvHuNUPZdNxsrpozJH+G/iAIPJVa3K+AYXBvvzWtSpDHb4ttvy48=
=pN2J
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Bernd Schmidt]

On 04/11/2011 10:03 PM, Jeff Law wrote:
> One of the fundamental problems you have to watch out for when dealing
> with scratch objects is how to handle the case when you belatedly
> realize you want the object to have a longer lifetime.

Historically, our problems with obstacks were a consequence of
line-by-line processing of the input file and never being quite certain
where a given object would have to end up. Almost all of that is simply
no longer an issue, which means RTL memory management is conceptually a
lot simpler than it used to be. A few things (constants, mostly) are
shared and must be permanent, the rest can go a single per-function
obstack which is freed after every function. I don't think we should (or
need to) try to have more obstacks for RTL.

I also don't see why a deep copy to the permanent obstack would be a
source of problems these days. More common than wanting to do that
however is the opposite case where you can speculatively generate RTL on
the function obstack, see if it's valid, and if not reset the obstack to
the earlier point to reclaim the memory. I know that reload and combine
could be made more efficient this way, and I'd guess the same is true
for fwprop.

I'd also like to point out that I share Steven's experience that the GC
stuff doesn't "just work"; I've certainly had objects collected from
under me and needed to puzzle out where to sprinkle the necessary GTY
markers.


Bernd

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/11 05:54, Bernd Schmidt wrote:
> On 04/11/2011 10:03 PM, Jeff Law wrote:
>> One of the fundamental problems you have to watch out for when dealing
>> with scratch objects is how to handle the case when you belatedly
>> realize you want the object to have a longer lifetime.
> 
> Historically, our problems with obstacks were a consequence of
> line-by-line processing of the input file and never being quite certain
> where a given object would have to end up.
That was certainly the cause of the bulk of the problems.  And to be
honest, the multi-obstack design made sense when it was written,
assuming one could keep track of which obstack was the right one to use.
 As we wanted to solve new problems, the obstack approach really fell apart.

Also note that trees were a *much* bigger problem than RTL WRT memory
management.  That's not to say there weren't any RTL problems, but that
we ran into far more problems with tree allocations.


 Almost all of that is simply
> no longer an issue, which means RTL memory management is conceptually a
> lot simpler than it used to be. 
In some respects, yes, memory management in general is conceptually a
hell of a lot simpler.  I'll note that some of the simplifications came
as a side effect of moving to GC allocation, but aren't inherent to
using GC allocation.  For example, removal of storing RTL into tree
structures without a suitable union :-)

In other respects I think memory management has become more difficult,
at least to change because usage of GC has allowed us to allocate and
mostly forget.  One can certainly argue that model has problems, but I
would claim the problems with allocate and forget were smaller than the
problems we had with explicit management via obstacks.



A few things (constants, mostly) are
> shared and must be permanent, the rest can go a single per-function
> obstack which is freed after every function. I don't think we should (or
> need to) try to have more obstacks for RTL.
To have two obstacks for RTL, you really have to have a bright line that
everyone knows and understands and that you can codify as well so that
things like deep copying can DTRT.  If you can't codify it for a deep
copy, then, IMHO, it's a non-starter.  Also note that having
verification capabilities helps considerably.  You also need rules for
if/when/how a single RTL object might validly reference RTL objects with
differing lifetimes.



> 
> I also don't see why a deep copy to the permanent obstack would be a
> source of problems these days.
For RTL, the biggest problem is making sure you do the right thing with
RTL that is expected to be unique & shared (trees are much more
difficult), which as you note is primarily going to be constants and the
like.



 More common than wanting to do that
> however is the opposite case where you can speculatively generate RTL on
> the function obstack, see if it's valid, and if not reset the obstack to
> the earlier point to reclaim the memory. I know that reload and combine
> could be made more efficient this way, and I'd guess the same is true
> for fwprop.
There are definitely things that obstacks handle well, I won't dispute that.


> 
> I'd also like to point out that I share Steven's experience that the GC
> stuff doesn't "just work"; I've certainly had objects collected from
> under me and needed to puzzle out where to sprinkle the necessary GTY
> markers.
In my experience, I've seen far fewer allocation mistakes with the GC
system than with the multi-obstack system we had prior to the
introduction of GC.  Neither is going to be 100% perfect as programmers
can certainly goof things up.  The question is which is easier to deal
with in the general case and I'd say GC wins by a wide margin.  Of
course, that comes with a penalty (performance), thus I'm open to
exploring what objects really need to be part of the GC system and which
can be handled by other allocation schemes such as obstacks.

jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNpHAeAAoJEBRtltQi2kC7HpQIAIchzQ/uBei7FaIJIbOK7dFn
kOey7eqhySdb1WTY4mMKfPPRrqD9rCfR1PPftH/x4250lRMLTKgN3WvCtUmSef8o
tAM4hwViadQUvOlJPjIve3VqNg2fV6CmG/mtnChuwAknsUe7z/xm+C+J6lPd0Mts
LWOqya+JGYrGVuSP9+BdANEVUM4mkoWaZ2pXOzml7pkvW2JQBloaJLlyN7lobShQ
Kw2wTnpzVlnyeCT8Ow8AVHk1Yw4osZkKmN5VrjRmGaKaOtl2CeQk+LODFmHp+YrE
Jkc/YTNtYev6Pt30QfE26qylcjWqJte3me0Ux5J8KR5fxNc2NFz+Vs20m6zy+M4=
=q93W
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Laurynas Biveinis]

2011/4/8 Jeff Law <law@redhat.com>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 04/08/11 07:21, Laurynas Biveinis wrote:
>> 2011/4/8 Steven Bosscher <stevenb.gcc@gmail.com>:
>>>>        * stmt.c (label_rtx): Allocate RTX in permanent RTL memory.
>>>
>>> That looks strange, labels should be function specific, except
>>> non-local labels. Maybe you can get away with allocating DECL_NONLOCAL
>>> labels on the permanent rtl obstack?
>>
>> That's a good idea, I will try it once things are stable on the branch.
>>
>>> Perhaps a third, per-translation-unit obstack is necessary?
>>
>> Perhaps. After I finish with permanent rtl obstack, I will measure how
>> large it gets and if it's worthwhile to split out
>> per-translation-obstack out of it.
> And then you'll want a per-statement obstack, then per-expression
> obstack, and before you know it, GCC looks much like it did 20 years ago.

It is certainly true that moving away from GC will make some kinds of
bugs possible again, but I hope that not enough to be an unmanageable
concern. The RTL object lifetimes seem to be clear in most of the
instances and so far I am going with only two of them: permanent and
function. After the initial conversion is done, I don't expect much
trouble for any new RTL allocations introduced to be decided which
memory area they belong to.

Adding a third area, e.g. per-TU, of course would complicate this, but
I still believe this is manageable.

--
Laurynas

Re: [gc-improv] Permanent vs function RTL obstack fix

[Steven Bosscher]

On Sun, Apr 10, 2011 at 8:22 PM, Laurynas Biveinis
<laurynas.biveinis@gmail.com> wrote:
> It is certainly true that moving away from GC will make some kinds of
> bugs possible again, but I hope that not enough to be an unmanageable
> concern.

One thing that may help, is to poison parts of released obstacks,
instead of actually releasing the memory. A bit like the poison
mechanism of GGC.

An obstack memory checking mechanism similar to gcac would also be
nice. Something that walks all RTL and verifies that RTX objects are
allocated on the obstacks where you'd expect them to be (via
_obstack_allocated_p).

I don't think GCC had any checking like that 20 years ago...

Ciao!
Steven

Re: [gc-improv] Permanent vs function RTL obstack fix

[Jeff Law]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/10/11 12:22, Laurynas Biveinis wrote:
> 
> It is certainly true that moving away from GC will make some kinds of
> bugs possible again, but I hope that not enough to be an unmanageable
> concern.
It's definitely a huge concern.  More for tree objects than RTL objects
though.



 The RTL object lifetimes seem to be clear in most of the
> instances and so far I am going with only two of them: permanent and
> function. After the initial conversion is done, I don't expect much
> trouble for any new RTL allocations introduced to be decided which
> memory area they belong to.
> 
> Adding a third area, e.g. per-TU, of course would complicate this, but
> I still believe this is manageable.
So what's the plan for the case where you need to change the lifetime of
an object?    What's the plan for building some kind of consistency
checking to ensure that we aren't referencing dangling pointers.

I ask these questions because they were a serious source of problems in
the past and any significant revamping of memory management needs to
have a reasonable story for how to deal with them, else we're taking a
rather significant step backwards.

jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNo1+YAAoJEBRtltQi2kC7LeMH/2pxPMnlJAsjiwHApURV/sxX
5XyGBvawPs1W6zobBRUeOhIrfn8hSm6P2QywZvj7EpfTbiD/aKdVXz8zBOC2J1IG
69TDQwXY4YhqEW5WxTsBK6YAoFTALebZe6dbLRuN5795X+d5rSZmlyiX/GgICB7M
2iMqqkH6kv9wO2k6pfeN6k+hIHZmpVHRg3KeADTWvO5+3FKkVWFizA3LHhPf/pDM
7sG5o6CB8AI7PBNgh6A7xNs045NexIhEdkQ/R7jQNpySk3XHpfOPhjKh135hWwnw
6UoR8xqned5nr1sj6n07i+hSvYDLT6Izm68ZnFe5E09lPemVe4rZnO6Sb/+fhOA=
=gD9u
-----END PGP SIGNATURE-----

Re: [gc-improv] Permanent vs function RTL obstack fix

[Laurynas Biveinis]

> So what's the plan for the case where you need to change the lifetime of
> an object?

Copying it. Frankly at the moment I don't how much trouble does deep
copying from scratch to function entails, as mentioned in your other
e-mail. ATM I am working at separating permanent from function. If it
turns out to be unsurmountable, then, well, lesson learned.

> What's the plan for building some kind of consistency
> checking to ensure that we aren't referencing dangling pointers.

I haven't thought them out in detail at this moment, but I had some
ideas, very close to what Steven suggested: poisoning RTL obstacks
instead of freeing/reusing. Walking existing GC memory at RTL function
memory freeing time to see that no pointers point to it. The latter is
virtually free even.

> I ask these questions because they were a serious source of problems in
> the past and any significant revamping of memory management needs to
> have a reasonable story for how to deal with them, else we're taking a
> rather significant step backwards.

I plan to propose merging the branch once such concerns will be
adequately addressed. Personally I believe that it is possible to
address them. In the case it is impossible to address them, I will
propose merging very lite version of the branch that changes RTL
allocation interface, keeps GC behind it, and leaves the rest for
presumably brighter future. Hopefully this will not be the case.

Regarding PCH: so far whenever I touch it, frankly I hate it. In the
current context: the RTL output there was early-generated RTL which
IMHO itself is a sign of a poor IL separation in GCC. So in that
respect I believe my work is a step in a right direction, even if
painful.

Thanks for your feedback,
-- 
Laurynas

Re: [gc-improv] Permanent vs function RTL obstack fix

[Laurynas Biveinis]

[Resending with the correct Mike's address, sorry for the spam]

> So what's the plan for the case where you need to change the lifetime of
> an object?

Copying it. Frankly at the moment I don't how much trouble does deep
copying from scratch to function entails, as mentioned in your other
e-mail. ATM I am working at separating permanent from function. If it
turns out to be unsurmountable, then, well, lesson learned.

> What's the plan for building some kind of consistency
> checking to ensure that we aren't referencing dangling pointers.

I haven't thought them out in detail at this moment, but I had some
ideas, very close to what Steven suggested: poisoning RTL obstacks
instead of freeing/reusing. Walking existing GC memory at RTL function
memory freeing time to see that no pointers point to it. The latter is
virtually free even.

> I ask these questions because they were a serious source of problems in
> the past and any significant revamping of memory management needs to
> have a reasonable story for how to deal with them, else we're taking a
> rather significant step backwards.

I plan to propose merging the branch once such concerns will be
adequately addressed. Personally I believe that it is possible to
address them. In the case it is impossible to address them, I will
propose merging very lite version of the branch that changes RTL
allocation interface, keeps GC behind it, and leaves the rest for
presumably brighter future. Hopefully this will not be the case.

Regarding PCH: so far whenever I touch it, frankly I hate it. In the
current context: the RTL output there was early-generated RTL which
IMHO itself is a sign of a poor IL separation in GCC. So in that
respect I believe my work is a step in a right direction, even if
painful.

Thanks for your feedback,
--
Laurynas

Re: [gc-improv] Permanent vs function RTL obstack fix

[Mike Stump]

On Apr 12, 2011, at 4:46 AM, Laurynas Biveinis wrote:
>> So what's the plan for the case where you need to change the lifetime of
>> an object?
> 
> Copying it.  Frankly at the moment I don't how much trouble does deep
> copying from scratch to function entails,

The code to copy isn't too hard and if fairly simple to understand, test and do.  Roughly, you just need to understand how deep you have to copy, part of this is ensuring you iterator over all the pointers in the objects that might point to the shorter lifetime.

The problems of the past would be, all the other folks that have live pointers into anything which is copied.  You have the first pointer and that is trivial to change, but all the other pointers could be problematic.  Some of the problems were when some of the pointers pointed to the new and some of the pointers pointed to the old, then, for all the data hanging off those data structures that were supposed to be single pieces of data, there were now two copies of that data.  Imagine if you had two TREE_ASM_WRITTENs for a `single' decl.  RTL is vastly simpler than a _DECL node.  For just generated RTL (combine temp rtl building), I would not expect any problems.  Likewise scratch to function, those should be easy enough.  Harder are function to anything longer, say, permanent.