GSOC Proposal

GSOC Proposal

[nick]

Greetings All,

I've already done most of the work required for signing up for GSoC
as of last year i.e. reading getting started, being signed up legally
for contributions.

My only real concern would be the proposal which I started writing here:
https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing

The biography and success section I'm fine with my bigger concern would be the project and roadmap
section. The roadmap is there and I will go into more detail about it in the projects section as
need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
paragraphs discussing it in the Projects Section.

Any other comments are welcome as well as I write it there,
Nick

Re: GSOC Proposal

[Richard Biener]

On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>
> Greetings All,
>
> I've already done most of the work required for signing up for GSoC
> as of last year i.e. reading getting started, being signed up legally
> for contributions.
>
> My only real concern would be the proposal which I started writing here:
> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>
> The biography and success section I'm fine with my bigger concern would be the project and roadmap
> section. The roadmap is there and I will go into more detail about it in the projects section as
> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
> paragraphs discussing it in the Projects Section.

I'm not sure I understand either the problem analysis nor the project
goal parts.  What
shared state with respect to garbage collection are you talking about?

Richard.

> Any other comments are welcome as well as I write it there,
> Nick

Re: GSOC Proposal

[nick]

On 2019-03-28 4:59 a.m., Richard Biener wrote:
> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>>
>> Greetings All,
>>
>> I've already done most of the work required for signing up for GSoC
>> as of last year i.e. reading getting started, being signed up legally
>> for contributions.
>>
>> My only real concern would be the proposal which I started writing here:
>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>>
>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
>> section. The roadmap is there and I will go into more detail about it in the projects section as
>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
>> paragraphs discussing it in the Projects Section.
> 
> I'm not sure I understand either the problem analysis nor the project
> goal parts.  What
> shared state with respect to garbage collection are you talking about?
> 
> Richard.
> 
I just fixed it. Seems we were discussing RTL itself. I edited it to reflect those
changes. Let me know if it's unclear or you would actually like me to discuss
some changes that may occur in the RTL layer itself.


I'm glad to be more exact if that's better but seems your confusion was just what
layer we were touching.

Nick
>> Any other comments are welcome as well as I write it there,
>> Nick

Re: GSOC Proposal

[Richard Biener]

[-- Attachment #1: Type: text/plain, Size: 2818 bytes --]

On Thu, 28 Mar 2019, nick wrote:

> 
> 
> On 2019-03-28 4:59 a.m., Richard Biener wrote:
> > On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
> >>
> >> Greetings All,
> >>
> >> I've already done most of the work required for signing up for GSoC
> >> as of last year i.e. reading getting started, being signed up legally
> >> for contributions.
> >>
> >> My only real concern would be the proposal which I started writing here:
> >> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
> >>
> >> The biography and success section I'm fine with my bigger concern would be the project and roadmap
> >> section. The roadmap is there and I will go into more detail about it in the projects section as
> >> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
> >> paragraphs discussing it in the Projects Section.
> > 
> > I'm not sure I understand either the problem analysis nor the project
> > goal parts.  What
> > shared state with respect to garbage collection are you talking about?
> > 
> > Richard.
> > 
> I just fixed it. Seems we were discussing RTL itself. I edited it to 
> reflect those changes. Let me know if it's unclear or you would actually 
> like me to discuss some changes that may occur in the RTL layer itself.
> 
> 
> I'm glad to be more exact if that's better but seems your confusion was 
> just what layer we were touching.

Let me just throw in some knowledge here.  The issue with RTL
is that we currently can only have a single function in this
intermediate language state since a function in RTL has some
state in global variables that would differ if it were another
function.  We can have multiple functions in GIMPLE intermediate
language state since all such state is in a function-specific
data structure (struct function).  The hard thing about moving
all this "global" state of RTL into the same place is that
there's global state in the various backends (and there's
already a struct funtion 'machine' part for such state, so there's
hope the issue isn't as big as it could be) and that some of
the global state is big and only changes very rarely.
That said, I'm not sure if anybody knows the full details here.

So as far as I understand you'd like to tackle this as project
with the goal to be able to have multiple functions in RTL
state.

That's laudable but IMHO also quite ambitious for a GSoC
project.  It's also an area I am not very familiar with so
I opt out of being a mentor for this project.

Richard.

> Nick
> >> Any other comments are welcome as well as I write it there,
> >> Nick
> 

-- 
Richard Biener <rguenther@suse.de>
SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany;
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah; HRB 21284 (AG Nürnberg)

Re: GSOC Proposal

[nick]

On 2019-03-29 5:08 a.m., Richard Biener wrote:
> On Thu, 28 Mar 2019, nick wrote:
> 
>>
>>
>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>>>>
>>>> Greetings All,
>>>>
>>>> I've already done most of the work required for signing up for GSoC
>>>> as of last year i.e. reading getting started, being signed up legally
>>>> for contributions.
>>>>
>>>> My only real concern would be the proposal which I started writing here:
>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>>>>
>>>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
>>>> section. The roadmap is there and I will go into more detail about it in the projects section as
>>>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
>>>> paragraphs discussing it in the Projects Section.
>>>
>>> I'm not sure I understand either the problem analysis nor the project
>>> goal parts.  What
>>> shared state with respect to garbage collection are you talking about?
>>>
>>> Richard.
>>>
>> I just fixed it. Seems we were discussing RTL itself. I edited it to 
>> reflect those changes. Let me know if it's unclear or you would actually 
>> like me to discuss some changes that may occur in the RTL layer itself.
>>
>>
>> I'm glad to be more exact if that's better but seems your confusion was 
>> just what layer we were touching.
> 
> Let me just throw in some knowledge here.  The issue with RTL
> is that we currently can only have a single function in this
> intermediate language state since a function in RTL has some
> state in global variables that would differ if it were another
> function.  We can have multiple functions in GIMPLE intermediate
> language state since all such state is in a function-specific
> data structure (struct function).  The hard thing about moving
> all this "global" state of RTL into the same place is that
> there's global state in the various backends (and there's
> already a struct funtion 'machine' part for such state, so there's
> hope the issue isn't as big as it could be) and that some of
> the global state is big and only changes very rarely.
> That said, I'm not sure if anybody knows the full details here.
> 
> So as far as I understand you'd like to tackle this as project
> with the goal to be able to have multiple functions in RTL
> state.
> 
> That's laudable but IMHO also quite ambitious for a GSoC
> project.  It's also an area I am not very familiar with so
> I opt out of being a mentor for this project.
> 
While I'm aware of three areas where the shared state is an issue
currently:
1, Compiler's Proper
2. The expand_functions 
3. RTL
4.Garbage Collector

Or maybe a project to be more
explicit about regions of the code that assume that the garbage-
collector can't run within them?[3] (since the GC is state that would
be shared by the threads).

This is what we were discussing previously and I wrote my proposal for
that. You however seem confused about what parts of the garbage collector
would be touched. That's fine with me, however seems you want be to
be more exact about which part  is touched.

My questions would be as it's changed back to the garbage collector project:
https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit

1. Your confusion about which part of the garbage collector is touched doesn't
really make sense s it's for the whole garbage collector as related to shared
state?
2. Injection was my code here in phase 3 for the callers of the new functions or
macros, perhaps this is not needed as the work with the garbage collector is enough?
3. Am I not understanding this project as I thought I was in the proposal I wrote?

Seems your more confusing my wording probably so I'm going to suggest one of 
two things here:
a) I'm going to allow you to make comments with what's confusing you and
it needs that's the issue here more than anything else so I sent you 
a link and please comment where you are having issues with this not
be clear for you:
Or maybe a project to be more
explicit about regions of the code that assume that the garbage-
collector can't run within them?[3] (since the GC is state that would
be shared by the threads).
as that's the actual project

b) Just comment here about the wording that's an issue for you or
where you want more exact wording

Sorry and hopefully this is helps you understand where I'm going,
Nick

> Richard.
> 
>> Nick
>>>> Any other comments are welcome as well as I write it there,
>>>> Nick
>>
> 

Re: GSOC Proposal

[nick]

On 2019-03-29 10:28 a.m., nick wrote:
> 
> 
> On 2019-03-29 5:08 a.m., Richard Biener wrote:
>> On Thu, 28 Mar 2019, nick wrote:
>>
>>>
>>>
>>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
>>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>>>>>
>>>>> Greetings All,
>>>>>
>>>>> I've already done most of the work required for signing up for GSoC
>>>>> as of last year i.e. reading getting started, being signed up legally
>>>>> for contributions.
>>>>>
>>>>> My only real concern would be the proposal which I started writing here:
>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>>>>>
>>>>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
>>>>> section. The roadmap is there and I will go into more detail about it in the projects section as
>>>>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
>>>>> paragraphs discussing it in the Projects Section.
>>>>
>>>> I'm not sure I understand either the problem analysis nor the project
>>>> goal parts.  What
>>>> shared state with respect to garbage collection are you talking about?
>>>>
>>>> Richard.
>>>>
>>> I just fixed it. Seems we were discussing RTL itself. I edited it to 
>>> reflect those changes. Let me know if it's unclear or you would actually 
>>> like me to discuss some changes that may occur in the RTL layer itself.
>>>
>>>
>>> I'm glad to be more exact if that's better but seems your confusion was 
>>> just what layer we were touching.
>>
>> Let me just throw in some knowledge here.  The issue with RTL
>> is that we currently can only have a single function in this
>> intermediate language state since a function in RTL has some
>> state in global variables that would differ if it were another
>> function.  We can have multiple functions in GIMPLE intermediate
>> language state since all such state is in a function-specific
>> data structure (struct function).  The hard thing about moving
>> all this "global" state of RTL into the same place is that
>> there's global state in the various backends (and there's
>> already a struct funtion 'machine' part for such state, so there's
>> hope the issue isn't as big as it could be) and that some of
>> the global state is big and only changes very rarely.
>> That said, I'm not sure if anybody knows the full details here.
>>
>> So as far as I understand you'd like to tackle this as project
>> with the goal to be able to have multiple functions in RTL
>> state.
>>
>> That's laudable but IMHO also quite ambitious for a GSoC
>> project.  It's also an area I am not very familiar with so
>> I opt out of being a mentor for this project.
>>
> While I'm aware of three areas where the shared state is an issue
> currently:
> 1, Compiler's Proper
> 2. The expand_functions 
> 3. RTL
> 4.Garbage Collector
> 
> Or maybe a project to be more
> explicit about regions of the code that assume that the garbage-
> collector can't run within them?[3] (since the GC is state that would
> be shared by the threads).
> 
> This is what we were discussing previously and I wrote my proposal for
> that. You however seem confused about what parts of the garbage collector
> would be touched. That's fine with me, however seems you want be to
> be more exact about which part  is touched.
> 
> My questions would be as it's changed back to the garbage collector project:
> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> 
> 1. Your confusion about which part of the garbage collector is touched doesn't
> really make sense s it's for the whole garbage collector as related to shared
> state?
> 2. Injection was my code here in phase 3 for the callers of the new functions or
> macros, perhaps this is not needed as the work with the garbage collector is enough?
> 3. Am I not understanding this project as I thought I was in the proposal I wrote?
> 
> Seems your more confusing my wording probably so I'm going to suggest one of 
> two things here:
> a) I'm going to allow you to make comments with what's confusing you and
> it needs that's the issue here more than anything else so I sent you 
> a link and please comment where you are having issues with this not
> be clear for you:
> Or maybe a project to be more
> explicit about regions of the code that assume that the garbage-
> collector can't run within them?[3] (since the GC is state that would
> be shared by the threads).
> as that's the actual project
> 
> b) Just comment here about the wording that's an issue for you or
> where you want more exact wording
> 
> Sorry and hopefully this is helps you understand where I'm going,
> Nick
> 
>> Richard.
>>
>>> Nick
>>>>> Any other comments are welcome as well as I write it there,
>>>>> Nick
>>>

Richard,

Seems your right touching the complete garbage collector is too much. I'm
just looking at the users of the garbage collector and it seems one of the
major ones is pre compiled headers.

I've narrowed it down to that. My own real final concern is two things:
1. Does it make sense to you in my writing?
2. Should callers inject the information for state sharing as required
as that seems better or is it better for the garbage collector to  store
the state sharing flags,marcos and functions internally for this.

Thanks and seems I was over thinking the last proposal it's too much:),
Nick
>>

Re: GSOC Proposal

[Eric Gallager]

On 3/29/19, nick <xerofoify@gmail.com> wrote:
>
>
> On 2019-03-29 10:28 a.m., nick wrote:
>>
>>
>> On 2019-03-29 5:08 a.m., Richard Biener wrote:
>>> On Thu, 28 Mar 2019, nick wrote:
>>>
>>>>
>>>>
>>>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
>>>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>>>>>>
>>>>>> Greetings All,
>>>>>>
>>>>>> I've already done most of the work required for signing up for GSoC
>>>>>> as of last year i.e. reading getting started, being signed up legally
>>>>>> for contributions.
>>>>>>
>>>>>> My only real concern would be the proposal which I started writing
>>>>>> here:
>>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>>>>>>
>>>>>> The biography and success section I'm fine with my bigger concern
>>>>>> would be the project and roadmap
>>>>>> section. The roadmap is there and I will go into more detail about it
>>>>>> in the projects section as
>>>>>> need be. Just wanted to known if the roadmap is detailed enough or can
>>>>>> I just write out a few
>>>>>> paragraphs discussing it in the Projects Section.
>>>>>
>>>>> I'm not sure I understand either the problem analysis nor the project
>>>>> goal parts.  What
>>>>> shared state with respect to garbage collection are you talking about?
>>>>>
>>>>> Richard.
>>>>>
>>>> I just fixed it. Seems we were discussing RTL itself. I edited it to
>>>> reflect those changes. Let me know if it's unclear or you would actually
>>>>
>>>> like me to discuss some changes that may occur in the RTL layer itself.
>>>>
>>>>
>>>> I'm glad to be more exact if that's better but seems your confusion was
>>>>
>>>> just what layer we were touching.
>>>
>>> Let me just throw in some knowledge here.  The issue with RTL
>>> is that we currently can only have a single function in this
>>> intermediate language state since a function in RTL has some
>>> state in global variables that would differ if it were another
>>> function.  We can have multiple functions in GIMPLE intermediate
>>> language state since all such state is in a function-specific
>>> data structure (struct function).  The hard thing about moving
>>> all this "global" state of RTL into the same place is that
>>> there's global state in the various backends (and there's
>>> already a struct funtion 'machine' part for such state, so there's
>>> hope the issue isn't as big as it could be) and that some of
>>> the global state is big and only changes very rarely.
>>> That said, I'm not sure if anybody knows the full details here.
>>>
>>> So as far as I understand you'd like to tackle this as project
>>> with the goal to be able to have multiple functions in RTL
>>> state.
>>>
>>> That's laudable but IMHO also quite ambitious for a GSoC
>>> project.  It's also an area I am not very familiar with so
>>> I opt out of being a mentor for this project.
>>>
>> While I'm aware of three areas where the shared state is an issue
>> currently:
>> 1, Compiler's Proper
>> 2. The expand_functions
>> 3. RTL
>> 4.Garbage Collector
>>
>> Or maybe a project to be more
>> explicit about regions of the code that assume that the garbage-
>> collector can't run within them?[3] (since the GC is state that would
>> be shared by the threads).
>>
>> This is what we were discussing previously and I wrote my proposal for
>> that. You however seem confused about what parts of the garbage collector
>> would be touched. That's fine with me, however seems you want be to
>> be more exact about which part  is touched.
>>
>> My questions would be as it's changed back to the garbage collector
>> project:
>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>
>> 1. Your confusion about which part of the garbage collector is touched
>> doesn't
>> really make sense s it's for the whole garbage collector as related to
>> shared
>> state?
>> 2. Injection was my code here in phase 3 for the callers of the new
>> functions or
>> macros, perhaps this is not needed as the work with the garbage collector
>> is enough?
>> 3. Am I not understanding this project as I thought I was in the proposal
>> I wrote?
>>
>> Seems your more confusing my wording probably so I'm going to suggest one
>> of
>> two things here:
>> a) I'm going to allow you to make comments with what's confusing you and
>> it needs that's the issue here more than anything else so I sent you
>> a link and please comment where you are having issues with this not
>> be clear for you:
>> Or maybe a project to be more
>> explicit about regions of the code that assume that the garbage-
>> collector can't run within them?[3] (since the GC is state that would
>> be shared by the threads).
>> as that's the actual project
>>
>> b) Just comment here about the wording that's an issue for you or
>> where you want more exact wording
>>
>> Sorry and hopefully this is helps you understand where I'm going,
>> Nick
>>
>>> Richard.
>>>
>>>> Nick
>>>>>> Any other comments are welcome as well as I write it there,
>>>>>> Nick
>>>>
>
> Richard,
>
> Seems your right touching the complete garbage collector is too much. I'm
> just looking at the users of the garbage collector and it seems one of the
> major ones is pre compiled headers.
>

The thing about pre-compiled headers is that I seem to remember some
GCC devs saying they wanted to rip out pre-compiled headers completely
once the C++ modules branch is merged, so I'm not sure if it's worth
putting that much work into something that might be removed soon,
anyways... I'm pretty sure Nathan Sidwell is the main person working
on the C++ modules branch, so I'm cc-ing him.

> I've narrowed it down to that. My own real final concern is two things:
> 1. Does it make sense to you in my writing?
> 2. Should callers inject the information for state sharing as required
> as that seems better or is it better for the garbage collector to  store
> the state sharing flags,marcos and functions internally for this.
>
> Thanks and seems I was over thinking the last proposal it's too much:),
> Nick
>>>
>

Re: GSOC Proposal

[Nathan Sidwell]

On 4/1/19 1:24 AM, Eric Gallager wrote:
> On 3/29/19, nick <xerofoify@gmail.com> wrote:

>> Seems your right touching the complete garbage collector is too much. I'm
>> just looking at the users of the garbage collector and it seems one of the
>> major ones is pre compiled headers.
>>
> 
> The thing about pre-compiled headers is that I seem to remember some
> GCC devs saying they wanted to rip out pre-compiled headers completely
> once the C++ modules branch is merged, so I'm not sure if it's worth
> putting that much work into something that might be removed soon,
> anyways... I'm pretty sure Nathan Sidwell is the main person working
> on the C++ modules branch, so I'm cc-ing him.

The use of the GC machinery for PCH is needed by the front ends and is 
orthogonal to that for GC during code generation.

nathan

-- 
Nathan Sidwell

Re: GSOC Proposal

[Richard Biener]

[-- Attachment #1: Type: text/plain, Size: 6461 bytes --]

On Fri, 29 Mar 2019, nick wrote:

> 
> 
> On 2019-03-29 10:28 a.m., nick wrote:
> > 
> > 
> > On 2019-03-29 5:08 a.m., Richard Biener wrote:
> >> On Thu, 28 Mar 2019, nick wrote:
> >>
> >>>
> >>>
> >>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
> >>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
> >>>>>
> >>>>> Greetings All,
> >>>>>
> >>>>> I've already done most of the work required for signing up for GSoC
> >>>>> as of last year i.e. reading getting started, being signed up legally
> >>>>> for contributions.
> >>>>>
> >>>>> My only real concern would be the proposal which I started writing here:
> >>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
> >>>>>
> >>>>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
> >>>>> section. The roadmap is there and I will go into more detail about it in the projects section as
> >>>>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
> >>>>> paragraphs discussing it in the Projects Section.
> >>>>
> >>>> I'm not sure I understand either the problem analysis nor the project
> >>>> goal parts.  What
> >>>> shared state with respect to garbage collection are you talking about?
> >>>>
> >>>> Richard.
> >>>>
> >>> I just fixed it. Seems we were discussing RTL itself. I edited it to 
> >>> reflect those changes. Let me know if it's unclear or you would actually 
> >>> like me to discuss some changes that may occur in the RTL layer itself.
> >>>
> >>>
> >>> I'm glad to be more exact if that's better but seems your confusion was 
> >>> just what layer we were touching.
> >>
> >> Let me just throw in some knowledge here.  The issue with RTL
> >> is that we currently can only have a single function in this
> >> intermediate language state since a function in RTL has some
> >> state in global variables that would differ if it were another
> >> function.  We can have multiple functions in GIMPLE intermediate
> >> language state since all such state is in a function-specific
> >> data structure (struct function).  The hard thing about moving
> >> all this "global" state of RTL into the same place is that
> >> there's global state in the various backends (and there's
> >> already a struct funtion 'machine' part for such state, so there's
> >> hope the issue isn't as big as it could be) and that some of
> >> the global state is big and only changes very rarely.
> >> That said, I'm not sure if anybody knows the full details here.
> >>
> >> So as far as I understand you'd like to tackle this as project
> >> with the goal to be able to have multiple functions in RTL
> >> state.
> >>
> >> That's laudable but IMHO also quite ambitious for a GSoC
> >> project.  It's also an area I am not very familiar with so
> >> I opt out of being a mentor for this project.
> >>
> > While I'm aware of three areas where the shared state is an issue
> > currently:
> > 1, Compiler's Proper
> > 2. The expand_functions 
> > 3. RTL
> > 4.Garbage Collector
> > 
> > Or maybe a project to be more
> > explicit about regions of the code that assume that the garbage-
> > collector can't run within them?[3] (since the GC is state that would
> > be shared by the threads).
> > 
> > This is what we were discussing previously and I wrote my proposal for
> > that. You however seem confused about what parts of the garbage collector
> > would be touched. That's fine with me, however seems you want be to
> > be more exact about which part  is touched.
> > 
> > My questions would be as it's changed back to the garbage collector project:
> > https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> > 
> > 1. Your confusion about which part of the garbage collector is touched doesn't
> > really make sense s it's for the whole garbage collector as related to shared
> > state?
> > 2. Injection was my code here in phase 3 for the callers of the new functions or
> > macros, perhaps this is not needed as the work with the garbage collector is enough?
> > 3. Am I not understanding this project as I thought I was in the proposal I wrote?
> > 
> > Seems your more confusing my wording probably so I'm going to suggest one of 
> > two things here:
> > a) I'm going to allow you to make comments with what's confusing you and
> > it needs that's the issue here more than anything else so I sent you 
> > a link and please comment where you are having issues with this not
> > be clear for you:
> > Or maybe a project to be more
> > explicit about regions of the code that assume that the garbage-
> > collector can't run within them?[3] (since the GC is state that would
> > be shared by the threads).
> > as that's the actual project
> > 
> > b) Just comment here about the wording that's an issue for you or
> > where you want more exact wording
> > 
> > Sorry and hopefully this is helps you understand where I'm going,
> > Nick
> > 
> >> Richard.
> >>
> >>> Nick
> >>>>> Any other comments are welcome as well as I write it there,
> >>>>> Nick
> >>>
> 
> Richard,
> 
> Seems your right touching the complete garbage collector is too much. I'm
> just looking at the users of the garbage collector and it seems one of the
> major ones is pre compiled headers.
> 
> I've narrowed it down to that. My own real final concern is two things:
> 1. Does it make sense to you in my writing?
> 2. Should callers inject the information for state sharing as required
> as that seems better or is it better for the garbage collector to  store
> the state sharing flags,marcos and functions internally for this.
> 
> Thanks and seems I was over thinking the last proposal it's too much:),

Nick,

as I've previously explained the garbage collector (and 
precompiled-headers) workings are understood well and its state
is already annotated everywhere.  What I do not understand is
what "global state" with respect to parallelization in GCC you
refer to when you write about the garbage collector and how
annotating helps parallelization.  The garbage collector itself
is only an issue for parallelization as far as it is not
thread-safe at the moment.  Collection already only happens
at very specific points where it is known to be safe.

Richard.

-- 
Richard Biener <rguenther@suse.de>
SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany;
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah; HRB 21284 (AG Nürnberg)

Re: GSOC Proposal

[nick]

On 2019-04-01 5:56 a.m., Richard Biener wrote:
> On Fri, 29 Mar 2019, nick wrote:
> 
>>
>>
>> On 2019-03-29 10:28 a.m., nick wrote:
>>>
>>>
>>> On 2019-03-29 5:08 a.m., Richard Biener wrote:
>>>> On Thu, 28 Mar 2019, nick wrote:
>>>>
>>>>>
>>>>>
>>>>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
>>>>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>>>>>>>
>>>>>>> Greetings All,
>>>>>>>
>>>>>>> I've already done most of the work required for signing up for GSoC
>>>>>>> as of last year i.e. reading getting started, being signed up legally
>>>>>>> for contributions.
>>>>>>>
>>>>>>> My only real concern would be the proposal which I started writing here:
>>>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>>>>>>>
>>>>>>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
>>>>>>> section. The roadmap is there and I will go into more detail about it in the projects section as
>>>>>>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
>>>>>>> paragraphs discussing it in the Projects Section.
>>>>>>
>>>>>> I'm not sure I understand either the problem analysis nor the project
>>>>>> goal parts.  What
>>>>>> shared state with respect to garbage collection are you talking about?
>>>>>>
>>>>>> Richard.
>>>>>>
>>>>> I just fixed it. Seems we were discussing RTL itself. I edited it to 
>>>>> reflect those changes. Let me know if it's unclear or you would actually 
>>>>> like me to discuss some changes that may occur in the RTL layer itself.
>>>>>
>>>>>
>>>>> I'm glad to be more exact if that's better but seems your confusion was 
>>>>> just what layer we were touching.
>>>>
>>>> Let me just throw in some knowledge here.  The issue with RTL
>>>> is that we currently can only have a single function in this
>>>> intermediate language state since a function in RTL has some
>>>> state in global variables that would differ if it were another
>>>> function.  We can have multiple functions in GIMPLE intermediate
>>>> language state since all such state is in a function-specific
>>>> data structure (struct function).  The hard thing about moving
>>>> all this "global" state of RTL into the same place is that
>>>> there's global state in the various backends (and there's
>>>> already a struct funtion 'machine' part for such state, so there's
>>>> hope the issue isn't as big as it could be) and that some of
>>>> the global state is big and only changes very rarely.
>>>> That said, I'm not sure if anybody knows the full details here.
>>>>
>>>> So as far as I understand you'd like to tackle this as project
>>>> with the goal to be able to have multiple functions in RTL
>>>> state.
>>>>
>>>> That's laudable but IMHO also quite ambitious for a GSoC
>>>> project.  It's also an area I am not very familiar with so
>>>> I opt out of being a mentor for this project.
>>>>
>>> While I'm aware of three areas where the shared state is an issue
>>> currently:
>>> 1, Compiler's Proper
>>> 2. The expand_functions 
>>> 3. RTL
>>> 4.Garbage Collector
>>>
>>> Or maybe a project to be more
>>> explicit about regions of the code that assume that the garbage-
>>> collector can't run within them?[3] (since the GC is state that would
>>> be shared by the threads).
>>>
>>> This is what we were discussing previously and I wrote my proposal for
>>> that. You however seem confused about what parts of the garbage collector
>>> would be touched. That's fine with me, however seems you want be to
>>> be more exact about which part  is touched.
>>>
>>> My questions would be as it's changed back to the garbage collector project:
>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>>
>>> 1. Your confusion about which part of the garbage collector is touched doesn't
>>> really make sense s it's for the whole garbage collector as related to shared
>>> state?
>>> 2. Injection was my code here in phase 3 for the callers of the new functions or
>>> macros, perhaps this is not needed as the work with the garbage collector is enough?
>>> 3. Am I not understanding this project as I thought I was in the proposal I wrote?
>>>
>>> Seems your more confusing my wording probably so I'm going to suggest one of 
>>> two things here:
>>> a) I'm going to allow you to make comments with what's confusing you and
>>> it needs that's the issue here more than anything else so I sent you 
>>> a link and please comment where you are having issues with this not
>>> be clear for you:
>>> Or maybe a project to be more
>>> explicit about regions of the code that assume that the garbage-
>>> collector can't run within them?[3] (since the GC is state that would
>>> be shared by the threads).
>>> as that's the actual project
>>>
>>> b) Just comment here about the wording that's an issue for you or
>>> where you want more exact wording
>>>
>>> Sorry and hopefully this is helps you understand where I'm going,
>>> Nick
>>>
>>>> Richard.
>>>>
>>>>> Nick
>>>>>>> Any other comments are welcome as well as I write it there,
>>>>>>> Nick
>>>>>
>>
>> Richard,
>>
>> Seems your right touching the complete garbage collector is too much. I'm
>> just looking at the users of the garbage collector and it seems one of the
>> major ones is pre compiled headers.
>>
>> I've narrowed it down to that. My own real final concern is two things:
>> 1. Does it make sense to you in my writing?
>> 2. Should callers inject the information for state sharing as required
>> as that seems better or is it better for the garbage collector to  store
>> the state sharing flags,marcos and functions internally for this.
>>
>> Thanks and seems I was over thinking the last proposal it's too much:),
> 
> Nick,
> 
> as I've previously explained the garbage collector (and 
> precompiled-headers) workings are understood well and its state
> is already annotated everywhere.  What I do not understand is
> what "global state" with respect to parallelization in GCC you
> refer to when you write about the garbage collector and how
> annotating helps parallelization.  The garbage collector itself
> is only an issue for parallelization as far as it is not
> thread-safe at the moment.  Collection already only happens
> at very specific points where it is known to be safe.
> 
> Richard.
> 
Well I'm talking about the shared roots of this garbage collector core state 
data structure or just struct ggc_root_tab.

But also this seems that this to be no longer shared globally if I'm not mistaken 
or this:
static vec<const_ggc_root_tab_t> extra_root_vec;

Not sure after reading the code which is a bigger deal through so I wrote
my proposal not just asking which is a better issue for not being thread
safe. Sorry about that.

As for the second question injection seems to not be the issue or outside
callers but just internal so phase 3 or step 3 would now be:
Find internal callers or users of x where x is one of the above rather
than injecting outside callers. Which answers my second question about
external callers being a issue still.

Let me know which  of the two is a better issue:
1. struct ggc_root_tabs being shared
2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
vector of root nodes for each type of allocation

and I will gladly rewrite my proposal sections for that
as needs to be reedited.

Thanks,
Nick

Re: GSOC Proposal

[Richard Biener]

[-- Attachment #1: Type: text/plain, Size: 8280 bytes --]

On Mon, 1 Apr 2019, nick wrote:

> 
> 
> On 2019-04-01 5:56 a.m., Richard Biener wrote:
> > On Fri, 29 Mar 2019, nick wrote:
> > 
> >>
> >>
> >> On 2019-03-29 10:28 a.m., nick wrote:
> >>>
> >>>
> >>> On 2019-03-29 5:08 a.m., Richard Biener wrote:
> >>>> On Thu, 28 Mar 2019, nick wrote:
> >>>>
> >>>>>
> >>>>>
> >>>>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
> >>>>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Greetings All,
> >>>>>>>
> >>>>>>> I've already done most of the work required for signing up for GSoC
> >>>>>>> as of last year i.e. reading getting started, being signed up legally
> >>>>>>> for contributions.
> >>>>>>>
> >>>>>>> My only real concern would be the proposal which I started writing here:
> >>>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
> >>>>>>>
> >>>>>>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
> >>>>>>> section. The roadmap is there and I will go into more detail about it in the projects section as
> >>>>>>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
> >>>>>>> paragraphs discussing it in the Projects Section.
> >>>>>>
> >>>>>> I'm not sure I understand either the problem analysis nor the project
> >>>>>> goal parts.  What
> >>>>>> shared state with respect to garbage collection are you talking about?
> >>>>>>
> >>>>>> Richard.
> >>>>>>
> >>>>> I just fixed it. Seems we were discussing RTL itself. I edited it to 
> >>>>> reflect those changes. Let me know if it's unclear or you would actually 
> >>>>> like me to discuss some changes that may occur in the RTL layer itself.
> >>>>>
> >>>>>
> >>>>> I'm glad to be more exact if that's better but seems your confusion was 
> >>>>> just what layer we were touching.
> >>>>
> >>>> Let me just throw in some knowledge here.  The issue with RTL
> >>>> is that we currently can only have a single function in this
> >>>> intermediate language state since a function in RTL has some
> >>>> state in global variables that would differ if it were another
> >>>> function.  We can have multiple functions in GIMPLE intermediate
> >>>> language state since all such state is in a function-specific
> >>>> data structure (struct function).  The hard thing about moving
> >>>> all this "global" state of RTL into the same place is that
> >>>> there's global state in the various backends (and there's
> >>>> already a struct funtion 'machine' part for such state, so there's
> >>>> hope the issue isn't as big as it could be) and that some of
> >>>> the global state is big and only changes very rarely.
> >>>> That said, I'm not sure if anybody knows the full details here.
> >>>>
> >>>> So as far as I understand you'd like to tackle this as project
> >>>> with the goal to be able to have multiple functions in RTL
> >>>> state.
> >>>>
> >>>> That's laudable but IMHO also quite ambitious for a GSoC
> >>>> project.  It's also an area I am not very familiar with so
> >>>> I opt out of being a mentor for this project.
> >>>>
> >>> While I'm aware of three areas where the shared state is an issue
> >>> currently:
> >>> 1, Compiler's Proper
> >>> 2. The expand_functions 
> >>> 3. RTL
> >>> 4.Garbage Collector
> >>>
> >>> Or maybe a project to be more
> >>> explicit about regions of the code that assume that the garbage-
> >>> collector can't run within them?[3] (since the GC is state that would
> >>> be shared by the threads).
> >>>
> >>> This is what we were discussing previously and I wrote my proposal for
> >>> that. You however seem confused about what parts of the garbage collector
> >>> would be touched. That's fine with me, however seems you want be to
> >>> be more exact about which part  is touched.
> >>>
> >>> My questions would be as it's changed back to the garbage collector project:
> >>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> >>>
> >>> 1. Your confusion about which part of the garbage collector is touched doesn't
> >>> really make sense s it's for the whole garbage collector as related to shared
> >>> state?
> >>> 2. Injection was my code here in phase 3 for the callers of the new functions or
> >>> macros, perhaps this is not needed as the work with the garbage collector is enough?
> >>> 3. Am I not understanding this project as I thought I was in the proposal I wrote?
> >>>
> >>> Seems your more confusing my wording probably so I'm going to suggest one of 
> >>> two things here:
> >>> a) I'm going to allow you to make comments with what's confusing you and
> >>> it needs that's the issue here more than anything else so I sent you 
> >>> a link and please comment where you are having issues with this not
> >>> be clear for you:
> >>> Or maybe a project to be more
> >>> explicit about regions of the code that assume that the garbage-
> >>> collector can't run within them?[3] (since the GC is state that would
> >>> be shared by the threads).
> >>> as that's the actual project
> >>>
> >>> b) Just comment here about the wording that's an issue for you or
> >>> where you want more exact wording
> >>>
> >>> Sorry and hopefully this is helps you understand where I'm going,
> >>> Nick
> >>>
> >>>> Richard.
> >>>>
> >>>>> Nick
> >>>>>>> Any other comments are welcome as well as I write it there,
> >>>>>>> Nick
> >>>>>
> >>
> >> Richard,
> >>
> >> Seems your right touching the complete garbage collector is too much. I'm
> >> just looking at the users of the garbage collector and it seems one of the
> >> major ones is pre compiled headers.
> >>
> >> I've narrowed it down to that. My own real final concern is two things:
> >> 1. Does it make sense to you in my writing?
> >> 2. Should callers inject the information for state sharing as required
> >> as that seems better or is it better for the garbage collector to  store
> >> the state sharing flags,marcos and functions internally for this.
> >>
> >> Thanks and seems I was over thinking the last proposal it's too much:),
> > 
> > Nick,
> > 
> > as I've previously explained the garbage collector (and 
> > precompiled-headers) workings are understood well and its state
> > is already annotated everywhere.  What I do not understand is
> > what "global state" with respect to parallelization in GCC you
> > refer to when you write about the garbage collector and how
> > annotating helps parallelization.  The garbage collector itself
> > is only an issue for parallelization as far as it is not
> > thread-safe at the moment.  Collection already only happens
> > at very specific points where it is known to be safe.
> > 
> > Richard.
> > 
> Well I'm talking about the shared roots of this garbage collector core state 
> data structure or just struct ggc_root_tab.
> 
> But also this seems that this to be no longer shared globally if I'm not mistaken 
> or this:
> static vec<const_ggc_root_tab_t> extra_root_vec;
> 
> Not sure after reading the code which is a bigger deal through so I wrote
> my proposal not just asking which is a better issue for not being thread
> safe. Sorry about that.
> 
> As for the second question injection seems to not be the issue or outside
> callers but just internal so phase 3 or step 3 would now be:
> Find internal callers or users of x where x is one of the above rather
> than injecting outside callers. Which answers my second question about
> external callers being a issue still.
> 
> Let me know which  of the two is a better issue:
> 1. struct ggc_root_tabs being shared
> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
> vector of root nodes for each type of allocation
> 
> and I will gladly rewrite my proposal sections for that
> as needs to be reedited.

I don't think working on the garbage collector as a separate
GSoC project is useful at this point.  Doing locking around
allocation seems like a good short-term solution and if that
turns out to be a performance issue for the threaded part
using per-thread freelists is likely an easy to deploy
solution.

Richard.

-- 
Richard Biener <rguenther@suse.de>
SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany;
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah; HRB 21284 (AG Nürnberg)

Re: GSOC Proposal

[nick]

On 2019-04-01 9:47 a.m., Richard Biener wrote:
> On Mon, 1 Apr 2019, nick wrote:
> 
>>
>>
>> On 2019-04-01 5:56 a.m., Richard Biener wrote:
>>> On Fri, 29 Mar 2019, nick wrote:
>>>
>>>>
>>>>
>>>> On 2019-03-29 10:28 a.m., nick wrote:
>>>>>
>>>>>
>>>>> On 2019-03-29 5:08 a.m., Richard Biener wrote:
>>>>>> On Thu, 28 Mar 2019, nick wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2019-03-28 4:59 a.m., Richard Biener wrote:
>>>>>>>> On Wed, Mar 27, 2019 at 6:31 PM nick <xerofoify@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Greetings All,
>>>>>>>>>
>>>>>>>>> I've already done most of the work required for signing up for GSoC
>>>>>>>>> as of last year i.e. reading getting started, being signed up legally
>>>>>>>>> for contributions.
>>>>>>>>>
>>>>>>>>> My only real concern would be the proposal which I started writing here:
>>>>>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit?usp=sharing
>>>>>>>>>
>>>>>>>>> The biography and success section I'm fine with my bigger concern would be the project and roadmap
>>>>>>>>> section. The roadmap is there and I will go into more detail about it in the projects section as
>>>>>>>>> need be. Just wanted to known if the roadmap is detailed enough or can I just write out a few
>>>>>>>>> paragraphs discussing it in the Projects Section.
>>>>>>>>
>>>>>>>> I'm not sure I understand either the problem analysis nor the project
>>>>>>>> goal parts.  What
>>>>>>>> shared state with respect to garbage collection are you talking about?
>>>>>>>>
>>>>>>>> Richard.
>>>>>>>>
>>>>>>> I just fixed it. Seems we were discussing RTL itself. I edited it to 
>>>>>>> reflect those changes. Let me know if it's unclear or you would actually 
>>>>>>> like me to discuss some changes that may occur in the RTL layer itself.
>>>>>>>
>>>>>>>
>>>>>>> I'm glad to be more exact if that's better but seems your confusion was 
>>>>>>> just what layer we were touching.
>>>>>>
>>>>>> Let me just throw in some knowledge here.  The issue with RTL
>>>>>> is that we currently can only have a single function in this
>>>>>> intermediate language state since a function in RTL has some
>>>>>> state in global variables that would differ if it were another
>>>>>> function.  We can have multiple functions in GIMPLE intermediate
>>>>>> language state since all such state is in a function-specific
>>>>>> data structure (struct function).  The hard thing about moving
>>>>>> all this "global" state of RTL into the same place is that
>>>>>> there's global state in the various backends (and there's
>>>>>> already a struct funtion 'machine' part for such state, so there's
>>>>>> hope the issue isn't as big as it could be) and that some of
>>>>>> the global state is big and only changes very rarely.
>>>>>> That said, I'm not sure if anybody knows the full details here.
>>>>>>
>>>>>> So as far as I understand you'd like to tackle this as project
>>>>>> with the goal to be able to have multiple functions in RTL
>>>>>> state.
>>>>>>
>>>>>> That's laudable but IMHO also quite ambitious for a GSoC
>>>>>> project.  It's also an area I am not very familiar with so
>>>>>> I opt out of being a mentor for this project.
>>>>>>
>>>>> While I'm aware of three areas where the shared state is an issue
>>>>> currently:
>>>>> 1, Compiler's Proper
>>>>> 2. The expand_functions 
>>>>> 3. RTL
>>>>> 4.Garbage Collector
>>>>>
>>>>> Or maybe a project to be more
>>>>> explicit about regions of the code that assume that the garbage-
>>>>> collector can't run within them?[3] (since the GC is state that would
>>>>> be shared by the threads).
>>>>>
>>>>> This is what we were discussing previously and I wrote my proposal for
>>>>> that. You however seem confused about what parts of the garbage collector
>>>>> would be touched. That's fine with me, however seems you want be to
>>>>> be more exact about which part  is touched.
>>>>>
>>>>> My questions would be as it's changed back to the garbage collector project:
>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>>>>
>>>>> 1. Your confusion about which part of the garbage collector is touched doesn't
>>>>> really make sense s it's for the whole garbage collector as related to shared
>>>>> state?
>>>>> 2. Injection was my code here in phase 3 for the callers of the new functions or
>>>>> macros, perhaps this is not needed as the work with the garbage collector is enough?
>>>>> 3. Am I not understanding this project as I thought I was in the proposal I wrote?
>>>>>
>>>>> Seems your more confusing my wording probably so I'm going to suggest one of 
>>>>> two things here:
>>>>> a) I'm going to allow you to make comments with what's confusing you and
>>>>> it needs that's the issue here more than anything else so I sent you 
>>>>> a link and please comment where you are having issues with this not
>>>>> be clear for you:
>>>>> Or maybe a project to be more
>>>>> explicit about regions of the code that assume that the garbage-
>>>>> collector can't run within them?[3] (since the GC is state that would
>>>>> be shared by the threads).
>>>>> as that's the actual project
>>>>>
>>>>> b) Just comment here about the wording that's an issue for you or
>>>>> where you want more exact wording
>>>>>
>>>>> Sorry and hopefully this is helps you understand where I'm going,
>>>>> Nick
>>>>>
>>>>>> Richard.
>>>>>>
>>>>>>> Nick
>>>>>>>>> Any other comments are welcome as well as I write it there,
>>>>>>>>> Nick
>>>>>>>
>>>>
>>>> Richard,
>>>>
>>>> Seems your right touching the complete garbage collector is too much. I'm
>>>> just looking at the users of the garbage collector and it seems one of the
>>>> major ones is pre compiled headers.
>>>>
>>>> I've narrowed it down to that. My own real final concern is two things:
>>>> 1. Does it make sense to you in my writing?
>>>> 2. Should callers inject the information for state sharing as required
>>>> as that seems better or is it better for the garbage collector to  store
>>>> the state sharing flags,marcos and functions internally for this.
>>>>
>>>> Thanks and seems I was over thinking the last proposal it's too much:),
>>>
>>> Nick,
>>>
>>> as I've previously explained the garbage collector (and 
>>> precompiled-headers) workings are understood well and its state
>>> is already annotated everywhere.  What I do not understand is
>>> what "global state" with respect to parallelization in GCC you
>>> refer to when you write about the garbage collector and how
>>> annotating helps parallelization.  The garbage collector itself
>>> is only an issue for parallelization as far as it is not
>>> thread-safe at the moment.  Collection already only happens
>>> at very specific points where it is known to be safe.
>>>
>>> Richard.
>>>
>> Well I'm talking about the shared roots of this garbage collector core state 
>> data structure or just struct ggc_root_tab.
>>
>> But also this seems that this to be no longer shared globally if I'm not mistaken 
>> or this:
>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>
>> Not sure after reading the code which is a bigger deal through so I wrote
>> my proposal not just asking which is a better issue for not being thread
>> safe. Sorry about that.
>>
>> As for the second question injection seems to not be the issue or outside
>> callers but just internal so phase 3 or step 3 would now be:
>> Find internal callers or users of x where x is one of the above rather
>> than injecting outside callers. Which answers my second question about
>> external callers being a issue still.
>>
>> Let me know which  of the two is a better issue:
>> 1. struct ggc_root_tabs being shared
>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
>> vector of root nodes for each type of allocation
>>
>> and I will gladly rewrite my proposal sections for that
>> as needs to be reedited.
> 
> I don't think working on the garbage collector as a separate
> GSoC project is useful at this point.  Doing locking around
> allocation seems like a good short-term solution and if that
> turns out to be a performance issue for the threaded part
> using per-thread freelists is likely an easy to deploy
> solution.
> 
> Richard.
> 
I agree but we were discussing this:
Or maybe a project to be more
explicit about regions of the code that assume that the garbage-
collector can't run within them?[3] (since the GC is state that would
be shared by the threads).

In addition I moved my paper back to our discussion about garbage collector
state with outside callers.Seems we really need to do something about
my wording as the idea of my project in a nutshell was to figure
out how to mark shared state by callers and inject it into the
garbage collector letting it known that the state was not shared between
threads or shared. Seems that was on the GSoc page and in our discussions the issue
is marking outside code for shared state. If that's correct then my
wording of outside callers is incorrect it should have been shared
state between threads on outside callers to the garbage collector.
If the state is that in your wording above then great as I understand
where we are going and will gladly change my wording.

Also freelists don't work here as the state is shared at the caller's end which
would need two major issues:
1. Locking on nodes of the freelists when two threads allocate at the same thing
which can be a problem if the shared state is shared a lot
2. Locking allocation with large numbers of callers can starve threads

Seems that working on the garbage collector itself isn't the issue but the callers
as I just figured out as related to your state idea. Let me know if that's correct
and if the wording change I mentioned is fine with you as that's the state it
seems that needs to be changed.
Nick

Re: GSOC Proposal

[Richard Biener]

[-- Attachment #1: Type: text/plain, Size: 5086 bytes --]

On Mon, 1 Apr 2019, nick wrote:

> 
> 
> On 2019-04-01 9:47 a.m., Richard Biener wrote:
> > On Mon, 1 Apr 2019, nick wrote:
> > 
> >> Well I'm talking about the shared roots of this garbage collector core state 
> >> data structure or just struct ggc_root_tab.
> >>
> >> But also this seems that this to be no longer shared globally if I'm not mistaken 
> >> or this:
> >> static vec<const_ggc_root_tab_t> extra_root_vec;
> >>
> >> Not sure after reading the code which is a bigger deal through so I wrote
> >> my proposal not just asking which is a better issue for not being thread
> >> safe. Sorry about that.
> >>
> >> As for the second question injection seems to not be the issue or outside
> >> callers but just internal so phase 3 or step 3 would now be:
> >> Find internal callers or users of x where x is one of the above rather
> >> than injecting outside callers. Which answers my second question about
> >> external callers being a issue still.
> >>
> >> Let me know which  of the two is a better issue:
> >> 1. struct ggc_root_tabs being shared
> >> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
> >> vector of root nodes for each type of allocation
> >>
> >> and I will gladly rewrite my proposal sections for that
> >> as needs to be reedited.
> > 
> > I don't think working on the garbage collector as a separate
> > GSoC project is useful at this point.  Doing locking around
> > allocation seems like a good short-term solution and if that
> > turns out to be a performance issue for the threaded part
> > using per-thread freelists is likely an easy to deploy
> > solution.
> > 
> > Richard.
> > 
> I agree but we were discussing this:
> Or maybe a project to be more
> explicit about regions of the code that assume that the garbage-
> collector can't run within them?[3] (since the GC is state that would
> be shared by the threads).

The process of collecting garbage is not the only issue (and that
very issue is easiest mitigated by collecting only at specific
points - which is what we do - and have those be serializing points).
The main issue is the underlying memory allocator (GCC uses memory
that is garbage collected plus regular heap memory).

> In addition I moved my paper back to our discussion about garbage collector
> state with outside callers.Seems we really need to do something about
> my wording as the idea of my project in a nutshell was to figure
> out how to mark shared state by callers and inject it into the
> garbage collector letting it known that the state was not shared between
> threads or shared. Seems that was on the GSoc page and in our discussions the issue
> is marking outside code for shared state. If that's correct then my
> wording of outside callers is incorrect it should have been shared
> state between threads on outside callers to the garbage collector.
> If the state is that in your wording above then great as I understand
> where we are going and will gladly change my wording.

I'm still not sure what you are shooting at, the above sentences do
not make any sense to me.

> Also freelists don't work here as the state is shared at the caller's 
> end which would need two major issues:
> 1. Locking on nodes of the 
> freelists when two threads allocate at the same thing which can be a 
> problem if the shared state is shared a lot
> 2. Locking allocation with 
> large numbers of callers can starve threads

First of all allocating memory from the GC pool is not the main
work of GIMPLE passes so simply serializing at allocation time might
work out.  Second free lists of course do work.  What you'd do is
have a fast path in allocation using a thread-local "free list"
which you can allocate from without taking any lock.  Maybe I should
explain "free list" since that term doesn't make too much sense in
a garbage collector world.  What I'd do is when a client thread
asks for memory of size N allocate M objects of that size but put
M - 1 on the client thread local "free list" to be allocated lock-free
from for the next M - 1 calls.  Note that garbage collected memory
objects are only handed out in fixed chunks (powers of two plus
a few special sizes) so you'd have one "free list" per chunk size
per thread.

The collection itself (mark & sweep) would be fully serialized still
(and not return to any threads local "free list").

ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
_do_ have ggc_free ...).

As said, I don't see GC or the memory allocator as sth interesting
to work on for parallelization until the basic setup works and it
proves to be a bottleneck.

> Seems that working on the garbage collector itself isn't the issue but 
> the callers as I just figured out as related to your state idea. Let me 
> know if that's correct and if the wording change I mentioned is fine 
> with you as that's the state it seems that needs to be changed.
> Nick 

Richard.

-- 
Richard Biener <rguenther@suse.de>
SUSE Linux GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany;
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah; HRB 21284 (AG Nürnberg)

Re: GSOC Proposal

[nick]

On 2019-04-03 7:30 a.m., Richard Biener wrote:
> On Mon, 1 Apr 2019, nick wrote:
> 
>>
>>
>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
>>> On Mon, 1 Apr 2019, nick wrote:
>>>
>>>> Well I'm talking about the shared roots of this garbage collector core state 
>>>> data structure or just struct ggc_root_tab.
>>>>
>>>> But also this seems that this to be no longer shared globally if I'm not mistaken 
>>>> or this:
>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>>>
>>>> Not sure after reading the code which is a bigger deal through so I wrote
>>>> my proposal not just asking which is a better issue for not being thread
>>>> safe. Sorry about that.
>>>>
>>>> As for the second question injection seems to not be the issue or outside
>>>> callers but just internal so phase 3 or step 3 would now be:
>>>> Find internal callers or users of x where x is one of the above rather
>>>> than injecting outside callers. Which answers my second question about
>>>> external callers being a issue still.
>>>>
>>>> Let me know which  of the two is a better issue:
>>>> 1. struct ggc_root_tabs being shared
>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
>>>> vector of root nodes for each type of allocation
>>>>
>>>> and I will gladly rewrite my proposal sections for that
>>>> as needs to be reedited.
>>>
>>> I don't think working on the garbage collector as a separate
>>> GSoC project is useful at this point.  Doing locking around
>>> allocation seems like a good short-term solution and if that
>>> turns out to be a performance issue for the threaded part
>>> using per-thread freelists is likely an easy to deploy
>>> solution.
>>>
>>> Richard.
>>>
>> I agree but we were discussing this:
>> Or maybe a project to be more
>> explicit about regions of the code that assume that the garbage-
>> collector can't run within them?[3] (since the GC is state that would
>> be shared by the threads).
> 
> The process of collecting garbage is not the only issue (and that
> very issue is easiest mitigated by collecting only at specific
> points - which is what we do - and have those be serializing points).
> The main issue is the underlying memory allocator (GCC uses memory
> that is garbage collected plus regular heap memory).
> 
>> In addition I moved my paper back to our discussion about garbage collector
>> state with outside callers.Seems we really need to do something about
>> my wording as the idea of my project in a nutshell was to figure
>> out how to mark shared state by callers and inject it into the
>> garbage collector letting it known that the state was not shared between
>> threads or shared. Seems that was on the GSoc page and in our discussions the issue
>> is marking outside code for shared state. If that's correct then my
>> wording of outside callers is incorrect it should have been shared
>> state between threads on outside callers to the garbage collector.
>> If the state is that in your wording above then great as I understand
>> where we are going and will gladly change my wording.
> 
> I'm still not sure what you are shooting at, the above sentences do
> not make any sense to me.
> 
>> Also freelists don't work here as the state is shared at the caller's 
>> end which would need two major issues:
>> 1. Locking on nodes of the 
>> freelists when two threads allocate at the same thing which can be a 
>> problem if the shared state is shared a lot
>> 2. Locking allocation with 
>> large numbers of callers can starve threads
> 
> First of all allocating memory from the GC pool is not the main
> work of GIMPLE passes so simply serializing at allocation time might
> work out.  Second free lists of course do work.  What you'd do is
> have a fast path in allocation using a thread-local "free list"
> which you can allocate from without taking any lock.  Maybe I should
> explain "free list" since that term doesn't make too much sense in
> a garbage collector world.  What I'd do is when a client thread
> asks for memory of size N allocate M objects of that size but put
> M - 1 on the client thread local "free list" to be allocated lock-free
> from for the next M - 1 calls.  Note that garbage collected memory
> objects are only handed out in fixed chunks (powers of two plus
> a few special sizes) so you'd have one "free list" per chunk size
> per thread.
> 
> The collection itself (mark & sweep) would be fully serialized still
> (and not return to any threads local "free list").
> 
> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
> _do_ have ggc_free ...).
> 
> As said, I don't see GC or the memory allocator as sth interesting
> to work on for parallelization until the basic setup works and it
> proves to be a bottleneck.
> 
>> Seems that working on the garbage collector itself isn't the issue but 
>> the callers as I just figured out as related to your state idea. Let me 
>> know if that's correct and if the wording change I mentioned is fine 
>> with you as that's the state it seems that needs to be changed.
>> Nick 
> 
> Richard.
> 

That's fine and it's my fault for not understanding you better. I was aware 
of the expand_functions_all being taken for passes.c. However it seems
two other issues are these sets as related to threads:
1.finalize_compilation_unit
2.and the ipa set of pass functions

If I'm understanding it correctly number 1 seems to be a early version of
expand_all_functions for the GENERIC representation if that's the case
it really should be fixed. Not sure which is a better issue as both
seem to have issues either at the GENERIC level or GIMPLE level with shared
state.

Let me know if this is better as it seems now that I really think about 
it GIMPLE or GENERIC functions in passes.c are the main issue. 

Sorry for the misunderstanding and hopefully one of functions listed is better
for moving forward with my proposal,
Nick

Re: GSOC Proposal

[Richard Biener]

On Wed, 3 Apr 2019, nick wrote:

> 
> 
> On 2019-04-03 7:30 a.m., Richard Biener wrote:
> > On Mon, 1 Apr 2019, nick wrote:
> > 
> >>
> >>
> >> On 2019-04-01 9:47 a.m., Richard Biener wrote:
> >>> On Mon, 1 Apr 2019, nick wrote:
> >>>
> >>>> Well I'm talking about the shared roots of this garbage collector core state 
> >>>> data structure or just struct ggc_root_tab.
> >>>>
> >>>> But also this seems that this to be no longer shared globally if I'm not mistaken 
> >>>> or this:
> >>>> static vec<const_ggc_root_tab_t> extra_root_vec;
> >>>>
> >>>> Not sure after reading the code which is a bigger deal through so I wrote
> >>>> my proposal not just asking which is a better issue for not being thread
> >>>> safe. Sorry about that.
> >>>>
> >>>> As for the second question injection seems to not be the issue or outside
> >>>> callers but just internal so phase 3 or step 3 would now be:
> >>>> Find internal callers or users of x where x is one of the above rather
> >>>> than injecting outside callers. Which answers my second question about
> >>>> external callers being a issue still.
> >>>>
> >>>> Let me know which  of the two is a better issue:
> >>>> 1. struct ggc_root_tabs being shared
> >>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
> >>>> vector of root nodes for each type of allocation
> >>>>
> >>>> and I will gladly rewrite my proposal sections for that
> >>>> as needs to be reedited.
> >>>
> >>> I don't think working on the garbage collector as a separate
> >>> GSoC project is useful at this point.  Doing locking around
> >>> allocation seems like a good short-term solution and if that
> >>> turns out to be a performance issue for the threaded part
> >>> using per-thread freelists is likely an easy to deploy
> >>> solution.
> >>>
> >>> Richard.
> >>>
> >> I agree but we were discussing this:
> >> Or maybe a project to be more
> >> explicit about regions of the code that assume that the garbage-
> >> collector can't run within them?[3] (since the GC is state that would
> >> be shared by the threads).
> > 
> > The process of collecting garbage is not the only issue (and that
> > very issue is easiest mitigated by collecting only at specific
> > points - which is what we do - and have those be serializing points).
> > The main issue is the underlying memory allocator (GCC uses memory
> > that is garbage collected plus regular heap memory).
> > 
> >> In addition I moved my paper back to our discussion about garbage collector
> >> state with outside callers.Seems we really need to do something about
> >> my wording as the idea of my project in a nutshell was to figure
> >> out how to mark shared state by callers and inject it into the
> >> garbage collector letting it known that the state was not shared between
> >> threads or shared. Seems that was on the GSoc page and in our discussions the issue
> >> is marking outside code for shared state. If that's correct then my
> >> wording of outside callers is incorrect it should have been shared
> >> state between threads on outside callers to the garbage collector.
> >> If the state is that in your wording above then great as I understand
> >> where we are going and will gladly change my wording.
> > 
> > I'm still not sure what you are shooting at, the above sentences do
> > not make any sense to me.
> > 
> >> Also freelists don't work here as the state is shared at the caller's 
> >> end which would need two major issues:
> >> 1. Locking on nodes of the 
> >> freelists when two threads allocate at the same thing which can be a 
> >> problem if the shared state is shared a lot
> >> 2. Locking allocation with 
> >> large numbers of callers can starve threads
> > 
> > First of all allocating memory from the GC pool is not the main
> > work of GIMPLE passes so simply serializing at allocation time might
> > work out.  Second free lists of course do work.  What you'd do is
> > have a fast path in allocation using a thread-local "free list"
> > which you can allocate from without taking any lock.  Maybe I should
> > explain "free list" since that term doesn't make too much sense in
> > a garbage collector world.  What I'd do is when a client thread
> > asks for memory of size N allocate M objects of that size but put
> > M - 1 on the client thread local "free list" to be allocated lock-free
> > from for the next M - 1 calls.  Note that garbage collected memory
> > objects are only handed out in fixed chunks (powers of two plus
> > a few special sizes) so you'd have one "free list" per chunk size
> > per thread.
> > 
> > The collection itself (mark & sweep) would be fully serialized still
> > (and not return to any threads local "free list").
> > 
> > ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
> > _do_ have ggc_free ...).
> > 
> > As said, I don't see GC or the memory allocator as sth interesting
> > to work on for parallelization until the basic setup works and it
> > proves to be a bottleneck.
> > 
> >> Seems that working on the garbage collector itself isn't the issue but 
> >> the callers as I just figured out as related to your state idea. Let me 
> >> know if that's correct and if the wording change I mentioned is fine 
> >> with you as that's the state it seems that needs to be changed.
> >> Nick 
> > 
> > Richard.
> > 
> 
> That's fine and it's my fault for not understanding you better. I was aware 
> of the expand_functions_all being taken for passes.c. However it seems
> two other issues are these sets as related to threads:
> 1.finalize_compilation_unit
> 2.and the ipa set of pass functions
> 
> If I'm understanding it correctly number 1 seems to be a early version of
> expand_all_functions for the GENERIC representation if that's the case
> it really should be fixed. Not sure which is a better issue as both
> seem to have issues either at the GENERIC level or GIMPLE level with shared
> state.
> 
> Let me know if this is better as it seems now that I really think about 
> it GIMPLE or GENERIC functions in passes.c are the main issue. 
> 
> Sorry for the misunderstanding and hopefully one of functions listed is better
> for moving forward with my proposal,

Sorry, but guessing at useful projects by skimming through GCC code
at this point isn't the way to go forward - this new "idea" lacks
both detail and understanding.  Please try to stick to one of the
suggested projects or do more thorough research in case you want
to work on a new project idea next year.

Thanks,
Richard.

Re: GSOC Proposal

[nick]

On 2019-04-05 6:25 a.m., Richard Biener wrote:
> On Wed, 3 Apr 2019, nick wrote:
> 
>>
>>
>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
>>> On Mon, 1 Apr 2019, nick wrote:
>>>
>>>>
>>>>
>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>
>>>>>> Well I'm talking about the shared roots of this garbage collector core state 
>>>>>> data structure or just struct ggc_root_tab.
>>>>>>
>>>>>> But also this seems that this to be no longer shared globally if I'm not mistaken 
>>>>>> or this:
>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>>>>>
>>>>>> Not sure after reading the code which is a bigger deal through so I wrote
>>>>>> my proposal not just asking which is a better issue for not being thread
>>>>>> safe. Sorry about that.
>>>>>>
>>>>>> As for the second question injection seems to not be the issue or outside
>>>>>> callers but just internal so phase 3 or step 3 would now be:
>>>>>> Find internal callers or users of x where x is one of the above rather
>>>>>> than injecting outside callers. Which answers my second question about
>>>>>> external callers being a issue still.
>>>>>>
>>>>>> Let me know which  of the two is a better issue:
>>>>>> 1. struct ggc_root_tabs being shared
>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared heap or
>>>>>> vector of root nodes for each type of allocation
>>>>>>
>>>>>> and I will gladly rewrite my proposal sections for that
>>>>>> as needs to be reedited.
>>>>>
>>>>> I don't think working on the garbage collector as a separate
>>>>> GSoC project is useful at this point.  Doing locking around
>>>>> allocation seems like a good short-term solution and if that
>>>>> turns out to be a performance issue for the threaded part
>>>>> using per-thread freelists is likely an easy to deploy
>>>>> solution.
>>>>>
>>>>> Richard.
>>>>>
>>>> I agree but we were discussing this:
>>>> Or maybe a project to be more
>>>> explicit about regions of the code that assume that the garbage-
>>>> collector can't run within them?[3] (since the GC is state that would
>>>> be shared by the threads).
>>>
>>> The process of collecting garbage is not the only issue (and that
>>> very issue is easiest mitigated by collecting only at specific
>>> points - which is what we do - and have those be serializing points).
>>> The main issue is the underlying memory allocator (GCC uses memory
>>> that is garbage collected plus regular heap memory).
>>>
>>>> In addition I moved my paper back to our discussion about garbage collector
>>>> state with outside callers.Seems we really need to do something about
>>>> my wording as the idea of my project in a nutshell was to figure
>>>> out how to mark shared state by callers and inject it into the
>>>> garbage collector letting it known that the state was not shared between
>>>> threads or shared. Seems that was on the GSoc page and in our discussions the issue
>>>> is marking outside code for shared state. If that's correct then my
>>>> wording of outside callers is incorrect it should have been shared
>>>> state between threads on outside callers to the garbage collector.
>>>> If the state is that in your wording above then great as I understand
>>>> where we are going and will gladly change my wording.
>>>
>>> I'm still not sure what you are shooting at, the above sentences do
>>> not make any sense to me.
>>>
>>>> Also freelists don't work here as the state is shared at the caller's 
>>>> end which would need two major issues:
>>>> 1. Locking on nodes of the 
>>>> freelists when two threads allocate at the same thing which can be a 
>>>> problem if the shared state is shared a lot
>>>> 2. Locking allocation with 
>>>> large numbers of callers can starve threads
>>>
>>> First of all allocating memory from the GC pool is not the main
>>> work of GIMPLE passes so simply serializing at allocation time might
>>> work out.  Second free lists of course do work.  What you'd do is
>>> have a fast path in allocation using a thread-local "free list"
>>> which you can allocate from without taking any lock.  Maybe I should
>>> explain "free list" since that term doesn't make too much sense in
>>> a garbage collector world.  What I'd do is when a client thread
>>> asks for memory of size N allocate M objects of that size but put
>>> M - 1 on the client thread local "free list" to be allocated lock-free
>>> from for the next M - 1 calls.  Note that garbage collected memory
>>> objects are only handed out in fixed chunks (powers of two plus
>>> a few special sizes) so you'd have one "free list" per chunk size
>>> per thread.
>>>
>>> The collection itself (mark & sweep) would be fully serialized still
>>> (and not return to any threads local "free list").
>>>
>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
>>> _do_ have ggc_free ...).
>>>
>>> As said, I don't see GC or the memory allocator as sth interesting
>>> to work on for parallelization until the basic setup works and it
>>> proves to be a bottleneck.
>>>
>>>> Seems that working on the garbage collector itself isn't the issue but 
>>>> the callers as I just figured out as related to your state idea. Let me 
>>>> know if that's correct and if the wording change I mentioned is fine 
>>>> with you as that's the state it seems that needs to be changed.
>>>> Nick 
>>>
>>> Richard.
>>>
>>
>> That's fine and it's my fault for not understanding you better. I was aware 
>> of the expand_functions_all being taken for passes.c. However it seems
>> two other issues are these sets as related to threads:
>> 1.finalize_compilation_unit
>> 2.and the ipa set of pass functions
>>
>> If I'm understanding it correctly number 1 seems to be a early version of
>> expand_all_functions for the GENERIC representation if that's the case
>> it really should be fixed. Not sure which is a better issue as both
>> seem to have issues either at the GENERIC level or GIMPLE level with shared
>> state.
>>
>> Let me know if this is better as it seems now that I really think about 
>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
>>
>> Sorry for the misunderstanding and hopefully one of functions listed is better
>> for moving forward with my proposal,
> 
> Sorry, but guessing at useful projects by skimming through GCC code
> at this point isn't the way to go forward - this new "idea" lacks
> both detail and understanding.  Please try to stick to one of the
> suggested projects or do more thorough research in case you want
> to work on a new project idea next year.
> 
> Thanks,
> Richard.
> 

I was talking about cgraphunits.c and it seems that according to this:
Parallelize compilation using threads. GCC currently has an awful lot of truly global state and even more per-pass global state which makes this somewhat hard. The idea is to avoid issues with global state as much as possible by partitioning the compilation pipeline in pieces that share as little global state as possible and ensure each thread only works in one of those partitions. The biggest roadblock will be the not thread-safe memory allocator of GCC garbage collector. The goal of this project is to have a compilation pipeline driven by a scheduler assigning functions to be optimized to the partitions in the pipeline. This project would be mentored by Richard Biener. Required skills include: C/C++, ability to analyze big complex code base, parallelization

We are trying to create a rendering pipeline if I'm correct and it seems that the GENERIC level needs finalize_compilation_unit
to be fixed like expand_all_functions at the GIMPLE. That's my point it still is within that project. Here is what I wrote
as I figured out that it was shared state related to GENERIC passing to GIMPLE which is a bottleneck or would be in the 
threaded pipeline.

https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit

Nick

Re: GSOC Proposal

[Richard Biener]

On April 5, 2019 6:11:15 PM GMT+02:00, nick <xerofoify@gmail.com> wrote:
>
>
>On 2019-04-05 6:25 a.m., Richard Biener wrote:
>> On Wed, 3 Apr 2019, nick wrote:
>> 
>>>
>>>
>>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>
>>>>>
>>>>>
>>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
>>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>>
>>>>>>> Well I'm talking about the shared roots of this garbage
>collector core state 
>>>>>>> data structure or just struct ggc_root_tab.
>>>>>>>
>>>>>>> But also this seems that this to be no longer shared globally if
>I'm not mistaken 
>>>>>>> or this:
>>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>>>>>>
>>>>>>> Not sure after reading the code which is a bigger deal through
>so I wrote
>>>>>>> my proposal not just asking which is a better issue for not
>being thread
>>>>>>> safe. Sorry about that.
>>>>>>>
>>>>>>> As for the second question injection seems to not be the issue
>or outside
>>>>>>> callers but just internal so phase 3 or step 3 would now be:
>>>>>>> Find internal callers or users of x where x is one of the above
>rather
>>>>>>> than injecting outside callers. Which answers my second question
>about
>>>>>>> external callers being a issue still.
>>>>>>>
>>>>>>> Let me know which  of the two is a better issue:
>>>>>>> 1. struct ggc_root_tabs being shared
>>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared
>heap or
>>>>>>> vector of root nodes for each type of allocation
>>>>>>>
>>>>>>> and I will gladly rewrite my proposal sections for that
>>>>>>> as needs to be reedited.
>>>>>>
>>>>>> I don't think working on the garbage collector as a separate
>>>>>> GSoC project is useful at this point.  Doing locking around
>>>>>> allocation seems like a good short-term solution and if that
>>>>>> turns out to be a performance issue for the threaded part
>>>>>> using per-thread freelists is likely an easy to deploy
>>>>>> solution.
>>>>>>
>>>>>> Richard.
>>>>>>
>>>>> I agree but we were discussing this:
>>>>> Or maybe a project to be more
>>>>> explicit about regions of the code that assume that the garbage-
>>>>> collector can't run within them?[3] (since the GC is state that
>would
>>>>> be shared by the threads).
>>>>
>>>> The process of collecting garbage is not the only issue (and that
>>>> very issue is easiest mitigated by collecting only at specific
>>>> points - which is what we do - and have those be serializing
>points).
>>>> The main issue is the underlying memory allocator (GCC uses memory
>>>> that is garbage collected plus regular heap memory).
>>>>
>>>>> In addition I moved my paper back to our discussion about garbage
>collector
>>>>> state with outside callers.Seems we really need to do something
>about
>>>>> my wording as the idea of my project in a nutshell was to figure
>>>>> out how to mark shared state by callers and inject it into the
>>>>> garbage collector letting it known that the state was not shared
>between
>>>>> threads or shared. Seems that was on the GSoc page and in our
>discussions the issue
>>>>> is marking outside code for shared state. If that's correct then
>my
>>>>> wording of outside callers is incorrect it should have been shared
>>>>> state between threads on outside callers to the garbage collector.
>>>>> If the state is that in your wording above then great as I
>understand
>>>>> where we are going and will gladly change my wording.
>>>>
>>>> I'm still not sure what you are shooting at, the above sentences do
>>>> not make any sense to me.
>>>>
>>>>> Also freelists don't work here as the state is shared at the
>caller's 
>>>>> end which would need two major issues:
>>>>> 1. Locking on nodes of the 
>>>>> freelists when two threads allocate at the same thing which can be
>a 
>>>>> problem if the shared state is shared a lot
>>>>> 2. Locking allocation with 
>>>>> large numbers of callers can starve threads
>>>>
>>>> First of all allocating memory from the GC pool is not the main
>>>> work of GIMPLE passes so simply serializing at allocation time
>might
>>>> work out.  Second free lists of course do work.  What you'd do is
>>>> have a fast path in allocation using a thread-local "free list"
>>>> which you can allocate from without taking any lock.  Maybe I
>should
>>>> explain "free list" since that term doesn't make too much sense in
>>>> a garbage collector world.  What I'd do is when a client thread
>>>> asks for memory of size N allocate M objects of that size but put
>>>> M - 1 on the client thread local "free list" to be allocated
>lock-free
>>>> from for the next M - 1 calls.  Note that garbage collected memory
>>>> objects are only handed out in fixed chunks (powers of two plus
>>>> a few special sizes) so you'd have one "free list" per chunk size
>>>> per thread.
>>>>
>>>> The collection itself (mark & sweep) would be fully serialized
>still
>>>> (and not return to any threads local "free list").
>>>>
>>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
>>>> _do_ have ggc_free ...).
>>>>
>>>> As said, I don't see GC or the memory allocator as sth interesting
>>>> to work on for parallelization until the basic setup works and it
>>>> proves to be a bottleneck.
>>>>
>>>>> Seems that working on the garbage collector itself isn't the issue
>but 
>>>>> the callers as I just figured out as related to your state idea.
>Let me 
>>>>> know if that's correct and if the wording change I mentioned is
>fine 
>>>>> with you as that's the state it seems that needs to be changed.
>>>>> Nick 
>>>>
>>>> Richard.
>>>>
>>>
>>> That's fine and it's my fault for not understanding you better. I
>was aware 
>>> of the expand_functions_all being taken for passes.c. However it
>seems
>>> two other issues are these sets as related to threads:
>>> 1.finalize_compilation_unit
>>> 2.and the ipa set of pass functions
>>>
>>> If I'm understanding it correctly number 1 seems to be a early
>version of
>>> expand_all_functions for the GENERIC representation if that's the
>case
>>> it really should be fixed. Not sure which is a better issue as both
>>> seem to have issues either at the GENERIC level or GIMPLE level with
>shared
>>> state.
>>>
>>> Let me know if this is better as it seems now that I really think
>about 
>>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
>>>
>>> Sorry for the misunderstanding and hopefully one of functions listed
>is better
>>> for moving forward with my proposal,
>> 
>> Sorry, but guessing at useful projects by skimming through GCC code
>> at this point isn't the way to go forward - this new "idea" lacks
>> both detail and understanding.  Please try to stick to one of the
>> suggested projects or do more thorough research in case you want
>> to work on a new project idea next year.
>> 
>> Thanks,
>> Richard.
>> 
>
>I was talking about cgraphunits.c and it seems that according to this:
>Parallelize compilation using threads. GCC currently has an awful lot
>of truly global state and even more per-pass global state which makes
>this somewhat hard. The idea is to avoid issues with global state as
>much as possible by partitioning the compilation pipeline in pieces
>that share as little global state as possible and ensure each thread
>only works in one of those partitions. The biggest roadblock will be
>the not thread-safe memory allocator of GCC garbage collector. The goal
>of this project is to have a compilation pipeline driven by a scheduler
>assigning functions to be optimized to the partitions in the pipeline.
>This project would be mentored by Richard Biener. Required skills
>include: C/C++, ability to analyze big complex code base,
>parallelization
>
>We are trying to create a rendering pipeline if I'm correct and it
>seems that the GENERIC level needs finalize_compilation_unit
>to be fixed like expand_all_functions at the GIMPLE. That's my point it
>still is within that project. Here is what I wrote
>as I figured out that it was shared state related to GENERIC passing to
>GIMPLE which is a bottleneck or would be in the 
>threaded pipeline.
>
>https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit

The pre- post-IPA parts cannot be easily parallelized and that includes GENERIC to GIMPLE translation. This is why the project should focus on the much easier post-IPA and pre-RTL parts of the compilation pipeline since there interaction between functions is minimal. 

Richard. 

>
>Nick

Re: GSOC Proposal

[nick]

On 2019-04-07 5:31 a.m., Richard Biener wrote:
> On April 5, 2019 6:11:15 PM GMT+02:00, nick <xerofoify@gmail.com> wrote:
>>
>>
>> On 2019-04-05 6:25 a.m., Richard Biener wrote:
>>> On Wed, 3 Apr 2019, nick wrote:
>>>
>>>>
>>>>
>>>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
>>>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>>>
>>>>>>>> Well I'm talking about the shared roots of this garbage
>> collector core state 
>>>>>>>> data structure or just struct ggc_root_tab.
>>>>>>>>
>>>>>>>> But also this seems that this to be no longer shared globally if
>> I'm not mistaken 
>>>>>>>> or this:
>>>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>>>>>>>
>>>>>>>> Not sure after reading the code which is a bigger deal through
>> so I wrote
>>>>>>>> my proposal not just asking which is a better issue for not
>> being thread
>>>>>>>> safe. Sorry about that.
>>>>>>>>
>>>>>>>> As for the second question injection seems to not be the issue
>> or outside
>>>>>>>> callers but just internal so phase 3 or step 3 would now be:
>>>>>>>> Find internal callers or users of x where x is one of the above
>> rather
>>>>>>>> than injecting outside callers. Which answers my second question
>> about
>>>>>>>> external callers being a issue still.
>>>>>>>>
>>>>>>>> Let me know which  of the two is a better issue:
>>>>>>>> 1. struct ggc_root_tabs being shared
>>>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared
>> heap or
>>>>>>>> vector of root nodes for each type of allocation
>>>>>>>>
>>>>>>>> and I will gladly rewrite my proposal sections for that
>>>>>>>> as needs to be reedited.
>>>>>>>
>>>>>>> I don't think working on the garbage collector as a separate
>>>>>>> GSoC project is useful at this point.  Doing locking around
>>>>>>> allocation seems like a good short-term solution and if that
>>>>>>> turns out to be a performance issue for the threaded part
>>>>>>> using per-thread freelists is likely an easy to deploy
>>>>>>> solution.
>>>>>>>
>>>>>>> Richard.
>>>>>>>
>>>>>> I agree but we were discussing this:
>>>>>> Or maybe a project to be more
>>>>>> explicit about regions of the code that assume that the garbage-
>>>>>> collector can't run within them?[3] (since the GC is state that
>> would
>>>>>> be shared by the threads).
>>>>>
>>>>> The process of collecting garbage is not the only issue (and that
>>>>> very issue is easiest mitigated by collecting only at specific
>>>>> points - which is what we do - and have those be serializing
>> points).
>>>>> The main issue is the underlying memory allocator (GCC uses memory
>>>>> that is garbage collected plus regular heap memory).
>>>>>
>>>>>> In addition I moved my paper back to our discussion about garbage
>> collector
>>>>>> state with outside callers.Seems we really need to do something
>> about
>>>>>> my wording as the idea of my project in a nutshell was to figure
>>>>>> out how to mark shared state by callers and inject it into the
>>>>>> garbage collector letting it known that the state was not shared
>> between
>>>>>> threads or shared. Seems that was on the GSoc page and in our
>> discussions the issue
>>>>>> is marking outside code for shared state. If that's correct then
>> my
>>>>>> wording of outside callers is incorrect it should have been shared
>>>>>> state between threads on outside callers to the garbage collector.
>>>>>> If the state is that in your wording above then great as I
>> understand
>>>>>> where we are going and will gladly change my wording.
>>>>>
>>>>> I'm still not sure what you are shooting at, the above sentences do
>>>>> not make any sense to me.
>>>>>
>>>>>> Also freelists don't work here as the state is shared at the
>> caller's 
>>>>>> end which would need two major issues:
>>>>>> 1. Locking on nodes of the 
>>>>>> freelists when two threads allocate at the same thing which can be
>> a 
>>>>>> problem if the shared state is shared a lot
>>>>>> 2. Locking allocation with 
>>>>>> large numbers of callers can starve threads
>>>>>
>>>>> First of all allocating memory from the GC pool is not the main
>>>>> work of GIMPLE passes so simply serializing at allocation time
>> might
>>>>> work out.  Second free lists of course do work.  What you'd do is
>>>>> have a fast path in allocation using a thread-local "free list"
>>>>> which you can allocate from without taking any lock.  Maybe I
>> should
>>>>> explain "free list" since that term doesn't make too much sense in
>>>>> a garbage collector world.  What I'd do is when a client thread
>>>>> asks for memory of size N allocate M objects of that size but put
>>>>> M - 1 on the client thread local "free list" to be allocated
>> lock-free
>>>>> from for the next M - 1 calls.  Note that garbage collected memory
>>>>> objects are only handed out in fixed chunks (powers of two plus
>>>>> a few special sizes) so you'd have one "free list" per chunk size
>>>>> per thread.
>>>>>
>>>>> The collection itself (mark & sweep) would be fully serialized
>> still
>>>>> (and not return to any threads local "free list").
>>>>>
>>>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
>>>>> _do_ have ggc_free ...).
>>>>>
>>>>> As said, I don't see GC or the memory allocator as sth interesting
>>>>> to work on for parallelization until the basic setup works and it
>>>>> proves to be a bottleneck.
>>>>>
>>>>>> Seems that working on the garbage collector itself isn't the issue
>> but 
>>>>>> the callers as I just figured out as related to your state idea.
>> Let me 
>>>>>> know if that's correct and if the wording change I mentioned is
>> fine 
>>>>>> with you as that's the state it seems that needs to be changed.
>>>>>> Nick 
>>>>>
>>>>> Richard.
>>>>>
>>>>
>>>> That's fine and it's my fault for not understanding you better. I
>> was aware 
>>>> of the expand_functions_all being taken for passes.c. However it
>> seems
>>>> two other issues are these sets as related to threads:
>>>> 1.finalize_compilation_unit
>>>> 2.and the ipa set of pass functions
>>>>
>>>> If I'm understanding it correctly number 1 seems to be a early
>> version of
>>>> expand_all_functions for the GENERIC representation if that's the
>> case
>>>> it really should be fixed. Not sure which is a better issue as both
>>>> seem to have issues either at the GENERIC level or GIMPLE level with
>> shared
>>>> state.
>>>>
>>>> Let me know if this is better as it seems now that I really think
>> about 
>>>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
>>>>
>>>> Sorry for the misunderstanding and hopefully one of functions listed
>> is better
>>>> for moving forward with my proposal,
>>>
>>> Sorry, but guessing at useful projects by skimming through GCC code
>>> at this point isn't the way to go forward - this new "idea" lacks
>>> both detail and understanding.  Please try to stick to one of the
>>> suggested projects or do more thorough research in case you want
>>> to work on a new project idea next year.
>>>
>>> Thanks,
>>> Richard.
>>>
>>
>> I was talking about cgraphunits.c and it seems that according to this:
>> Parallelize compilation using threads. GCC currently has an awful lot
>> of truly global state and even more per-pass global state which makes
>> this somewhat hard. The idea is to avoid issues with global state as
>> much as possible by partitioning the compilation pipeline in pieces
>> that share as little global state as possible and ensure each thread
>> only works in one of those partitions. The biggest roadblock will be
>> the not thread-safe memory allocator of GCC garbage collector. The goal
>> of this project is to have a compilation pipeline driven by a scheduler
>> assigning functions to be optimized to the partitions in the pipeline.
>> This project would be mentored by Richard Biener. Required skills
>> include: C/C++, ability to analyze big complex code base,
>> parallelization
>>
>> We are trying to create a rendering pipeline if I'm correct and it
>> seems that the GENERIC level needs finalize_compilation_unit
>> to be fixed like expand_all_functions at the GIMPLE. That's my point it
>> still is within that project. Here is what I wrote
>> as I figured out that it was shared state related to GENERIC passing to
>> GIMPLE which is a bottleneck or would be in the 
>> threaded pipeline.
>>
>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> 
> The pre- post-IPA parts cannot be easily parallelized and that includes GENERIC to GIMPLE translation. This is why the project should focus on the much easier post-IPA and pre-RTL parts of the compilation pipeline since there interaction between functions is minimal. 
> 
> Richard. 
> 
>>
>> Nick
> 
Richard,
Thanks seems we finally got somewhere I rewrote it for that. Please just doubt check tomorrow before I submit later tomorrow
as I would just want to make sure it's OK with you. Here is the link:
https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit

The only thing is if you want more detail in the project part as I just think it's
enough but let me know if not.

Thanks for your patience,
Nick

Re: GSOC Proposal

[Richard Biener]

On Sun, 7 Apr 2019, nick wrote:

> 
> 
> On 2019-04-07 5:31 a.m., Richard Biener wrote:
> > On April 5, 2019 6:11:15 PM GMT+02:00, nick <xerofoify@gmail.com> wrote:
> >>
> >>
> >> On 2019-04-05 6:25 a.m., Richard Biener wrote:
> >>> On Wed, 3 Apr 2019, nick wrote:
> >>>
> >>>>
> >>>>
> >>>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
> >>>>> On Mon, 1 Apr 2019, nick wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
> >>>>>>> On Mon, 1 Apr 2019, nick wrote:
> >>>>>>>
> >>>>>>>> Well I'm talking about the shared roots of this garbage
> >> collector core state 
> >>>>>>>> data structure or just struct ggc_root_tab.
> >>>>>>>>
> >>>>>>>> But also this seems that this to be no longer shared globally if
> >> I'm not mistaken 
> >>>>>>>> or this:
> >>>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
> >>>>>>>>
> >>>>>>>> Not sure after reading the code which is a bigger deal through
> >> so I wrote
> >>>>>>>> my proposal not just asking which is a better issue for not
> >> being thread
> >>>>>>>> safe. Sorry about that.
> >>>>>>>>
> >>>>>>>> As for the second question injection seems to not be the issue
> >> or outside
> >>>>>>>> callers but just internal so phase 3 or step 3 would now be:
> >>>>>>>> Find internal callers or users of x where x is one of the above
> >> rather
> >>>>>>>> than injecting outside callers. Which answers my second question
> >> about
> >>>>>>>> external callers being a issue still.
> >>>>>>>>
> >>>>>>>> Let me know which  of the two is a better issue:
> >>>>>>>> 1. struct ggc_root_tabs being shared
> >>>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared
> >> heap or
> >>>>>>>> vector of root nodes for each type of allocation
> >>>>>>>>
> >>>>>>>> and I will gladly rewrite my proposal sections for that
> >>>>>>>> as needs to be reedited.
> >>>>>>>
> >>>>>>> I don't think working on the garbage collector as a separate
> >>>>>>> GSoC project is useful at this point.  Doing locking around
> >>>>>>> allocation seems like a good short-term solution and if that
> >>>>>>> turns out to be a performance issue for the threaded part
> >>>>>>> using per-thread freelists is likely an easy to deploy
> >>>>>>> solution.
> >>>>>>>
> >>>>>>> Richard.
> >>>>>>>
> >>>>>> I agree but we were discussing this:
> >>>>>> Or maybe a project to be more
> >>>>>> explicit about regions of the code that assume that the garbage-
> >>>>>> collector can't run within them?[3] (since the GC is state that
> >> would
> >>>>>> be shared by the threads).
> >>>>>
> >>>>> The process of collecting garbage is not the only issue (and that
> >>>>> very issue is easiest mitigated by collecting only at specific
> >>>>> points - which is what we do - and have those be serializing
> >> points).
> >>>>> The main issue is the underlying memory allocator (GCC uses memory
> >>>>> that is garbage collected plus regular heap memory).
> >>>>>
> >>>>>> In addition I moved my paper back to our discussion about garbage
> >> collector
> >>>>>> state with outside callers.Seems we really need to do something
> >> about
> >>>>>> my wording as the idea of my project in a nutshell was to figure
> >>>>>> out how to mark shared state by callers and inject it into the
> >>>>>> garbage collector letting it known that the state was not shared
> >> between
> >>>>>> threads or shared. Seems that was on the GSoc page and in our
> >> discussions the issue
> >>>>>> is marking outside code for shared state. If that's correct then
> >> my
> >>>>>> wording of outside callers is incorrect it should have been shared
> >>>>>> state between threads on outside callers to the garbage collector.
> >>>>>> If the state is that in your wording above then great as I
> >> understand
> >>>>>> where we are going and will gladly change my wording.
> >>>>>
> >>>>> I'm still not sure what you are shooting at, the above sentences do
> >>>>> not make any sense to me.
> >>>>>
> >>>>>> Also freelists don't work here as the state is shared at the
> >> caller's 
> >>>>>> end which would need two major issues:
> >>>>>> 1. Locking on nodes of the 
> >>>>>> freelists when two threads allocate at the same thing which can be
> >> a 
> >>>>>> problem if the shared state is shared a lot
> >>>>>> 2. Locking allocation with 
> >>>>>> large numbers of callers can starve threads
> >>>>>
> >>>>> First of all allocating memory from the GC pool is not the main
> >>>>> work of GIMPLE passes so simply serializing at allocation time
> >> might
> >>>>> work out.  Second free lists of course do work.  What you'd do is
> >>>>> have a fast path in allocation using a thread-local "free list"
> >>>>> which you can allocate from without taking any lock.  Maybe I
> >> should
> >>>>> explain "free list" since that term doesn't make too much sense in
> >>>>> a garbage collector world.  What I'd do is when a client thread
> >>>>> asks for memory of size N allocate M objects of that size but put
> >>>>> M - 1 on the client thread local "free list" to be allocated
> >> lock-free
> >>>>> from for the next M - 1 calls.  Note that garbage collected memory
> >>>>> objects are only handed out in fixed chunks (powers of two plus
> >>>>> a few special sizes) so you'd have one "free list" per chunk size
> >>>>> per thread.
> >>>>>
> >>>>> The collection itself (mark & sweep) would be fully serialized
> >> still
> >>>>> (and not return to any threads local "free list").
> >>>>>
> >>>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
> >>>>> _do_ have ggc_free ...).
> >>>>>
> >>>>> As said, I don't see GC or the memory allocator as sth interesting
> >>>>> to work on for parallelization until the basic setup works and it
> >>>>> proves to be a bottleneck.
> >>>>>
> >>>>>> Seems that working on the garbage collector itself isn't the issue
> >> but 
> >>>>>> the callers as I just figured out as related to your state idea.
> >> Let me 
> >>>>>> know if that's correct and if the wording change I mentioned is
> >> fine 
> >>>>>> with you as that's the state it seems that needs to be changed.
> >>>>>> Nick 
> >>>>>
> >>>>> Richard.
> >>>>>
> >>>>
> >>>> That's fine and it's my fault for not understanding you better. I
> >> was aware 
> >>>> of the expand_functions_all being taken for passes.c. However it
> >> seems
> >>>> two other issues are these sets as related to threads:
> >>>> 1.finalize_compilation_unit
> >>>> 2.and the ipa set of pass functions
> >>>>
> >>>> If I'm understanding it correctly number 1 seems to be a early
> >> version of
> >>>> expand_all_functions for the GENERIC representation if that's the
> >> case
> >>>> it really should be fixed. Not sure which is a better issue as both
> >>>> seem to have issues either at the GENERIC level or GIMPLE level with
> >> shared
> >>>> state.
> >>>>
> >>>> Let me know if this is better as it seems now that I really think
> >> about 
> >>>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
> >>>>
> >>>> Sorry for the misunderstanding and hopefully one of functions listed
> >> is better
> >>>> for moving forward with my proposal,
> >>>
> >>> Sorry, but guessing at useful projects by skimming through GCC code
> >>> at this point isn't the way to go forward - this new "idea" lacks
> >>> both detail and understanding.  Please try to stick to one of the
> >>> suggested projects or do more thorough research in case you want
> >>> to work on a new project idea next year.
> >>>
> >>> Thanks,
> >>> Richard.
> >>>
> >>
> >> I was talking about cgraphunits.c and it seems that according to this:
> >> Parallelize compilation using threads. GCC currently has an awful lot
> >> of truly global state and even more per-pass global state which makes
> >> this somewhat hard. The idea is to avoid issues with global state as
> >> much as possible by partitioning the compilation pipeline in pieces
> >> that share as little global state as possible and ensure each thread
> >> only works in one of those partitions. The biggest roadblock will be
> >> the not thread-safe memory allocator of GCC garbage collector. The goal
> >> of this project is to have a compilation pipeline driven by a scheduler
> >> assigning functions to be optimized to the partitions in the pipeline.
> >> This project would be mentored by Richard Biener. Required skills
> >> include: C/C++, ability to analyze big complex code base,
> >> parallelization
> >>
> >> We are trying to create a rendering pipeline if I'm correct and it
> >> seems that the GENERIC level needs finalize_compilation_unit
> >> to be fixed like expand_all_functions at the GIMPLE. That's my point it
> >> still is within that project. Here is what I wrote
> >> as I figured out that it was shared state related to GENERIC passing to
> >> GIMPLE which is a bottleneck or would be in the 
> >> threaded pipeline.
> >>
> >> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> > 
> > The pre- post-IPA parts cannot be easily parallelized and that includes GENERIC to GIMPLE translation. This is why the project should focus on the much easier post-IPA and pre-RTL parts of the compilation pipeline since there interaction between functions is minimal. 
> > 
> > Richard. 
> > 
> >>
> >> Nick
> > 
> Richard,
> Thanks seems we finally got somewhere I rewrote it for that. Please just doubt check tomorrow before I submit later tomorrow
> as I would just want to make sure it's OK with you. Here is the link:
> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> 
> The only thing is if you want more detail in the project part as I just think it's
> enough but let me know if not.

It seems the project part lacks any specifics and those are to be
researched and discovered in the Mid may - Late May timeframe of
your Roadmap?

I do not understand what "Create optimized functions ... for avoiding
sharing of state ..." is about - what kind of functions do you intend
to write?  Often examples help.

Richard.

> Thanks for your patience,
> Nick

Re: GSOC Proposal

[nick]

On 2019-04-08 3:29 a.m., Richard Biener wrote:
> On Sun, 7 Apr 2019, nick wrote:
> 
>>
>>
>> On 2019-04-07 5:31 a.m., Richard Biener wrote:
>>> On April 5, 2019 6:11:15 PM GMT+02:00, nick <xerofoify@gmail.com> wrote:
>>>>
>>>>
>>>> On 2019-04-05 6:25 a.m., Richard Biener wrote:
>>>>> On Wed, 3 Apr 2019, nick wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
>>>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
>>>>>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>>>>>
>>>>>>>>>> Well I'm talking about the shared roots of this garbage
>>>> collector core state 
>>>>>>>>>> data structure or just struct ggc_root_tab.
>>>>>>>>>>
>>>>>>>>>> But also this seems that this to be no longer shared globally if
>>>> I'm not mistaken 
>>>>>>>>>> or this:
>>>>>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>>>>>>>>>
>>>>>>>>>> Not sure after reading the code which is a bigger deal through
>>>> so I wrote
>>>>>>>>>> my proposal not just asking which is a better issue for not
>>>> being thread
>>>>>>>>>> safe. Sorry about that.
>>>>>>>>>>
>>>>>>>>>> As for the second question injection seems to not be the issue
>>>> or outside
>>>>>>>>>> callers but just internal so phase 3 or step 3 would now be:
>>>>>>>>>> Find internal callers or users of x where x is one of the above
>>>> rather
>>>>>>>>>> than injecting outside callers. Which answers my second question
>>>> about
>>>>>>>>>> external callers being a issue still.
>>>>>>>>>>
>>>>>>>>>> Let me know which  of the two is a better issue:
>>>>>>>>>> 1. struct ggc_root_tabs being shared
>>>>>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared
>>>> heap or
>>>>>>>>>> vector of root nodes for each type of allocation
>>>>>>>>>>
>>>>>>>>>> and I will gladly rewrite my proposal sections for that
>>>>>>>>>> as needs to be reedited.
>>>>>>>>>
>>>>>>>>> I don't think working on the garbage collector as a separate
>>>>>>>>> GSoC project is useful at this point.  Doing locking around
>>>>>>>>> allocation seems like a good short-term solution and if that
>>>>>>>>> turns out to be a performance issue for the threaded part
>>>>>>>>> using per-thread freelists is likely an easy to deploy
>>>>>>>>> solution.
>>>>>>>>>
>>>>>>>>> Richard.
>>>>>>>>>
>>>>>>>> I agree but we were discussing this:
>>>>>>>> Or maybe a project to be more
>>>>>>>> explicit about regions of the code that assume that the garbage-
>>>>>>>> collector can't run within them?[3] (since the GC is state that
>>>> would
>>>>>>>> be shared by the threads).
>>>>>>>
>>>>>>> The process of collecting garbage is not the only issue (and that
>>>>>>> very issue is easiest mitigated by collecting only at specific
>>>>>>> points - which is what we do - and have those be serializing
>>>> points).
>>>>>>> The main issue is the underlying memory allocator (GCC uses memory
>>>>>>> that is garbage collected plus regular heap memory).
>>>>>>>
>>>>>>>> In addition I moved my paper back to our discussion about garbage
>>>> collector
>>>>>>>> state with outside callers.Seems we really need to do something
>>>> about
>>>>>>>> my wording as the idea of my project in a nutshell was to figure
>>>>>>>> out how to mark shared state by callers and inject it into the
>>>>>>>> garbage collector letting it known that the state was not shared
>>>> between
>>>>>>>> threads or shared. Seems that was on the GSoc page and in our
>>>> discussions the issue
>>>>>>>> is marking outside code for shared state. If that's correct then
>>>> my
>>>>>>>> wording of outside callers is incorrect it should have been shared
>>>>>>>> state between threads on outside callers to the garbage collector.
>>>>>>>> If the state is that in your wording above then great as I
>>>> understand
>>>>>>>> where we are going and will gladly change my wording.
>>>>>>>
>>>>>>> I'm still not sure what you are shooting at, the above sentences do
>>>>>>> not make any sense to me.
>>>>>>>
>>>>>>>> Also freelists don't work here as the state is shared at the
>>>> caller's 
>>>>>>>> end which would need two major issues:
>>>>>>>> 1. Locking on nodes of the 
>>>>>>>> freelists when two threads allocate at the same thing which can be
>>>> a 
>>>>>>>> problem if the shared state is shared a lot
>>>>>>>> 2. Locking allocation with 
>>>>>>>> large numbers of callers can starve threads
>>>>>>>
>>>>>>> First of all allocating memory from the GC pool is not the main
>>>>>>> work of GIMPLE passes so simply serializing at allocation time
>>>> might
>>>>>>> work out.  Second free lists of course do work.  What you'd do is
>>>>>>> have a fast path in allocation using a thread-local "free list"
>>>>>>> which you can allocate from without taking any lock.  Maybe I
>>>> should
>>>>>>> explain "free list" since that term doesn't make too much sense in
>>>>>>> a garbage collector world.  What I'd do is when a client thread
>>>>>>> asks for memory of size N allocate M objects of that size but put
>>>>>>> M - 1 on the client thread local "free list" to be allocated
>>>> lock-free
>>>>>>> from for the next M - 1 calls.  Note that garbage collected memory
>>>>>>> objects are only handed out in fixed chunks (powers of two plus
>>>>>>> a few special sizes) so you'd have one "free list" per chunk size
>>>>>>> per thread.
>>>>>>>
>>>>>>> The collection itself (mark & sweep) would be fully serialized
>>>> still
>>>>>>> (and not return to any threads local "free list").
>>>>>>>
>>>>>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
>>>>>>> _do_ have ggc_free ...).
>>>>>>>
>>>>>>> As said, I don't see GC or the memory allocator as sth interesting
>>>>>>> to work on for parallelization until the basic setup works and it
>>>>>>> proves to be a bottleneck.
>>>>>>>
>>>>>>>> Seems that working on the garbage collector itself isn't the issue
>>>> but 
>>>>>>>> the callers as I just figured out as related to your state idea.
>>>> Let me 
>>>>>>>> know if that's correct and if the wording change I mentioned is
>>>> fine 
>>>>>>>> with you as that's the state it seems that needs to be changed.
>>>>>>>> Nick 
>>>>>>>
>>>>>>> Richard.
>>>>>>>
>>>>>>
>>>>>> That's fine and it's my fault for not understanding you better. I
>>>> was aware 
>>>>>> of the expand_functions_all being taken for passes.c. However it
>>>> seems
>>>>>> two other issues are these sets as related to threads:
>>>>>> 1.finalize_compilation_unit
>>>>>> 2.and the ipa set of pass functions
>>>>>>
>>>>>> If I'm understanding it correctly number 1 seems to be a early
>>>> version of
>>>>>> expand_all_functions for the GENERIC representation if that's the
>>>> case
>>>>>> it really should be fixed. Not sure which is a better issue as both
>>>>>> seem to have issues either at the GENERIC level or GIMPLE level with
>>>> shared
>>>>>> state.
>>>>>>
>>>>>> Let me know if this is better as it seems now that I really think
>>>> about 
>>>>>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
>>>>>>
>>>>>> Sorry for the misunderstanding and hopefully one of functions listed
>>>> is better
>>>>>> for moving forward with my proposal,
>>>>>
>>>>> Sorry, but guessing at useful projects by skimming through GCC code
>>>>> at this point isn't the way to go forward - this new "idea" lacks
>>>>> both detail and understanding.  Please try to stick to one of the
>>>>> suggested projects or do more thorough research in case you want
>>>>> to work on a new project idea next year.
>>>>>
>>>>> Thanks,
>>>>> Richard.
>>>>>
>>>>
>>>> I was talking about cgraphunits.c and it seems that according to this:
>>>> Parallelize compilation using threads. GCC currently has an awful lot
>>>> of truly global state and even more per-pass global state which makes
>>>> this somewhat hard. The idea is to avoid issues with global state as
>>>> much as possible by partitioning the compilation pipeline in pieces
>>>> that share as little global state as possible and ensure each thread
>>>> only works in one of those partitions. The biggest roadblock will be
>>>> the not thread-safe memory allocator of GCC garbage collector. The goal
>>>> of this project is to have a compilation pipeline driven by a scheduler
>>>> assigning functions to be optimized to the partitions in the pipeline.
>>>> This project would be mentored by Richard Biener. Required skills
>>>> include: C/C++, ability to analyze big complex code base,
>>>> parallelization
>>>>
>>>> We are trying to create a rendering pipeline if I'm correct and it
>>>> seems that the GENERIC level needs finalize_compilation_unit
>>>> to be fixed like expand_all_functions at the GIMPLE. That's my point it
>>>> still is within that project. Here is what I wrote
>>>> as I figured out that it was shared state related to GENERIC passing to
>>>> GIMPLE which is a bottleneck or would be in the 
>>>> threaded pipeline.
>>>>
>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>>
>>> The pre- post-IPA parts cannot be easily parallelized and that includes GENERIC to GIMPLE translation. This is why the project should focus on the much easier post-IPA and pre-RTL parts of the compilation pipeline since there interaction between functions is minimal. 
>>>
>>> Richard. 
>>>
>>>>
>>>> Nick
>>>
>> Richard,
>> Thanks seems we finally got somewhere I rewrote it for that. Please just doubt check tomorrow before I submit later tomorrow
>> as I would just want to make sure it's OK with you. Here is the link:
>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>
>> The only thing is if you want more detail in the project part as I just think it's
>> enough but let me know if not.
> 
> It seems the project part lacks any specifics and those are to be
> researched and discovered in the Mid may - Late May timeframe of
> your Roadmap?
Yes this is because after looking into ipa functions it would take
some time to really understand it not just at a surface level. It's
not the functions themselves but thinking about how the pipeline
would work in the POST IPA and pre RTL functions. 

> I do not understand what "Create optimized functions ... for avoiding
> sharing of state ..." is about - what kind of functions do you intend
> to write?  Often examples help.
> 
Fix functions at both the post IPA  and pre RTL levels for avoiding sharing of state and increasing 
throughput with multiple threads in the compiler is what I actually meant. There aren't any new 
functions it's optimizing the paths. So it was a small typo. Thanks for noticing it.

I will submit either later today or early tomorrow so if there's anything else let me know, 
Nick
> Richard.
> 
>> Thanks for your patience,
>> Nick

Re: GSOC Proposal

[Richard Biener]

On Mon, 8 Apr 2019, nick wrote:

> 
> 
> On 2019-04-08 3:29 a.m., Richard Biener wrote:
> > On Sun, 7 Apr 2019, nick wrote:
> > 
> >>
> >>
> >> On 2019-04-07 5:31 a.m., Richard Biener wrote:
> >>> On April 5, 2019 6:11:15 PM GMT+02:00, nick <xerofoify@gmail.com> wrote:
> >>>>
> >>>>
> >>>> On 2019-04-05 6:25 a.m., Richard Biener wrote:
> >>>>> On Wed, 3 Apr 2019, nick wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
> >>>>>>> On Mon, 1 Apr 2019, nick wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
> >>>>>>>>> On Mon, 1 Apr 2019, nick wrote:
> >>>>>>>>>
> >>>>>>>>>> Well I'm talking about the shared roots of this garbage
> >>>> collector core state 
> >>>>>>>>>> data structure or just struct ggc_root_tab.
> >>>>>>>>>>
> >>>>>>>>>> But also this seems that this to be no longer shared globally if
> >>>> I'm not mistaken 
> >>>>>>>>>> or this:
> >>>>>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
> >>>>>>>>>>
> >>>>>>>>>> Not sure after reading the code which is a bigger deal through
> >>>> so I wrote
> >>>>>>>>>> my proposal not just asking which is a better issue for not
> >>>> being thread
> >>>>>>>>>> safe. Sorry about that.
> >>>>>>>>>>
> >>>>>>>>>> As for the second question injection seems to not be the issue
> >>>> or outside
> >>>>>>>>>> callers but just internal so phase 3 or step 3 would now be:
> >>>>>>>>>> Find internal callers or users of x where x is one of the above
> >>>> rather
> >>>>>>>>>> than injecting outside callers. Which answers my second question
> >>>> about
> >>>>>>>>>> external callers being a issue still.
> >>>>>>>>>>
> >>>>>>>>>> Let me know which  of the two is a better issue:
> >>>>>>>>>> 1. struct ggc_root_tabs being shared
> >>>>>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared
> >>>> heap or
> >>>>>>>>>> vector of root nodes for each type of allocation
> >>>>>>>>>>
> >>>>>>>>>> and I will gladly rewrite my proposal sections for that
> >>>>>>>>>> as needs to be reedited.
> >>>>>>>>>
> >>>>>>>>> I don't think working on the garbage collector as a separate
> >>>>>>>>> GSoC project is useful at this point.  Doing locking around
> >>>>>>>>> allocation seems like a good short-term solution and if that
> >>>>>>>>> turns out to be a performance issue for the threaded part
> >>>>>>>>> using per-thread freelists is likely an easy to deploy
> >>>>>>>>> solution.
> >>>>>>>>>
> >>>>>>>>> Richard.
> >>>>>>>>>
> >>>>>>>> I agree but we were discussing this:
> >>>>>>>> Or maybe a project to be more
> >>>>>>>> explicit about regions of the code that assume that the garbage-
> >>>>>>>> collector can't run within them?[3] (since the GC is state that
> >>>> would
> >>>>>>>> be shared by the threads).
> >>>>>>>
> >>>>>>> The process of collecting garbage is not the only issue (and that
> >>>>>>> very issue is easiest mitigated by collecting only at specific
> >>>>>>> points - which is what we do - and have those be serializing
> >>>> points).
> >>>>>>> The main issue is the underlying memory allocator (GCC uses memory
> >>>>>>> that is garbage collected plus regular heap memory).
> >>>>>>>
> >>>>>>>> In addition I moved my paper back to our discussion about garbage
> >>>> collector
> >>>>>>>> state with outside callers.Seems we really need to do something
> >>>> about
> >>>>>>>> my wording as the idea of my project in a nutshell was to figure
> >>>>>>>> out how to mark shared state by callers and inject it into the
> >>>>>>>> garbage collector letting it known that the state was not shared
> >>>> between
> >>>>>>>> threads or shared. Seems that was on the GSoc page and in our
> >>>> discussions the issue
> >>>>>>>> is marking outside code for shared state. If that's correct then
> >>>> my
> >>>>>>>> wording of outside callers is incorrect it should have been shared
> >>>>>>>> state between threads on outside callers to the garbage collector.
> >>>>>>>> If the state is that in your wording above then great as I
> >>>> understand
> >>>>>>>> where we are going and will gladly change my wording.
> >>>>>>>
> >>>>>>> I'm still not sure what you are shooting at, the above sentences do
> >>>>>>> not make any sense to me.
> >>>>>>>
> >>>>>>>> Also freelists don't work here as the state is shared at the
> >>>> caller's 
> >>>>>>>> end which would need two major issues:
> >>>>>>>> 1. Locking on nodes of the 
> >>>>>>>> freelists when two threads allocate at the same thing which can be
> >>>> a 
> >>>>>>>> problem if the shared state is shared a lot
> >>>>>>>> 2. Locking allocation with 
> >>>>>>>> large numbers of callers can starve threads
> >>>>>>>
> >>>>>>> First of all allocating memory from the GC pool is not the main
> >>>>>>> work of GIMPLE passes so simply serializing at allocation time
> >>>> might
> >>>>>>> work out.  Second free lists of course do work.  What you'd do is
> >>>>>>> have a fast path in allocation using a thread-local "free list"
> >>>>>>> which you can allocate from without taking any lock.  Maybe I
> >>>> should
> >>>>>>> explain "free list" since that term doesn't make too much sense in
> >>>>>>> a garbage collector world.  What I'd do is when a client thread
> >>>>>>> asks for memory of size N allocate M objects of that size but put
> >>>>>>> M - 1 on the client thread local "free list" to be allocated
> >>>> lock-free
> >>>>>>> from for the next M - 1 calls.  Note that garbage collected memory
> >>>>>>> objects are only handed out in fixed chunks (powers of two plus
> >>>>>>> a few special sizes) so you'd have one "free list" per chunk size
> >>>>>>> per thread.
> >>>>>>>
> >>>>>>> The collection itself (mark & sweep) would be fully serialized
> >>>> still
> >>>>>>> (and not return to any threads local "free list").
> >>>>>>>
> >>>>>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
> >>>>>>> _do_ have ggc_free ...).
> >>>>>>>
> >>>>>>> As said, I don't see GC or the memory allocator as sth interesting
> >>>>>>> to work on for parallelization until the basic setup works and it
> >>>>>>> proves to be a bottleneck.
> >>>>>>>
> >>>>>>>> Seems that working on the garbage collector itself isn't the issue
> >>>> but 
> >>>>>>>> the callers as I just figured out as related to your state idea.
> >>>> Let me 
> >>>>>>>> know if that's correct and if the wording change I mentioned is
> >>>> fine 
> >>>>>>>> with you as that's the state it seems that needs to be changed.
> >>>>>>>> Nick 
> >>>>>>>
> >>>>>>> Richard.
> >>>>>>>
> >>>>>>
> >>>>>> That's fine and it's my fault for not understanding you better. I
> >>>> was aware 
> >>>>>> of the expand_functions_all being taken for passes.c. However it
> >>>> seems
> >>>>>> two other issues are these sets as related to threads:
> >>>>>> 1.finalize_compilation_unit
> >>>>>> 2.and the ipa set of pass functions
> >>>>>>
> >>>>>> If I'm understanding it correctly number 1 seems to be a early
> >>>> version of
> >>>>>> expand_all_functions for the GENERIC representation if that's the
> >>>> case
> >>>>>> it really should be fixed. Not sure which is a better issue as both
> >>>>>> seem to have issues either at the GENERIC level or GIMPLE level with
> >>>> shared
> >>>>>> state.
> >>>>>>
> >>>>>> Let me know if this is better as it seems now that I really think
> >>>> about 
> >>>>>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
> >>>>>>
> >>>>>> Sorry for the misunderstanding and hopefully one of functions listed
> >>>> is better
> >>>>>> for moving forward with my proposal,
> >>>>>
> >>>>> Sorry, but guessing at useful projects by skimming through GCC code
> >>>>> at this point isn't the way to go forward - this new "idea" lacks
> >>>>> both detail and understanding.  Please try to stick to one of the
> >>>>> suggested projects or do more thorough research in case you want
> >>>>> to work on a new project idea next year.
> >>>>>
> >>>>> Thanks,
> >>>>> Richard.
> >>>>>
> >>>>
> >>>> I was talking about cgraphunits.c and it seems that according to this:
> >>>> Parallelize compilation using threads. GCC currently has an awful lot
> >>>> of truly global state and even more per-pass global state which makes
> >>>> this somewhat hard. The idea is to avoid issues with global state as
> >>>> much as possible by partitioning the compilation pipeline in pieces
> >>>> that share as little global state as possible and ensure each thread
> >>>> only works in one of those partitions. The biggest roadblock will be
> >>>> the not thread-safe memory allocator of GCC garbage collector. The goal
> >>>> of this project is to have a compilation pipeline driven by a scheduler
> >>>> assigning functions to be optimized to the partitions in the pipeline.
> >>>> This project would be mentored by Richard Biener. Required skills
> >>>> include: C/C++, ability to analyze big complex code base,
> >>>> parallelization
> >>>>
> >>>> We are trying to create a rendering pipeline if I'm correct and it
> >>>> seems that the GENERIC level needs finalize_compilation_unit
> >>>> to be fixed like expand_all_functions at the GIMPLE. That's my point it
> >>>> still is within that project. Here is what I wrote
> >>>> as I figured out that it was shared state related to GENERIC passing to
> >>>> GIMPLE which is a bottleneck or would be in the 
> >>>> threaded pipeline.
> >>>>
> >>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> >>>
> >>> The pre- post-IPA parts cannot be easily parallelized and that includes GENERIC to GIMPLE translation. This is why the project should focus on the much easier post-IPA and pre-RTL parts of the compilation pipeline since there interaction between functions is minimal. 
> >>>
> >>> Richard. 
> >>>
> >>>>
> >>>> Nick
> >>>
> >> Richard,
> >> Thanks seems we finally got somewhere I rewrote it for that. Please just doubt check tomorrow before I submit later tomorrow
> >> as I would just want to make sure it's OK with you. Here is the link:
> >> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
> >>
> >> The only thing is if you want more detail in the project part as I just think it's
> >> enough but let me know if not.
> > 
> > It seems the project part lacks any specifics and those are to be
> > researched and discovered in the Mid may - Late May timeframe of
> > your Roadmap?
> Yes this is because after looking into ipa functions it would take
> some time to really understand it not just at a surface level. It's
> not the functions themselves but thinking about how the pipeline
> would work in the POST IPA and pre RTL functions. 
> 
> > I do not understand what "Create optimized functions ... for avoiding
> > sharing of state ..." is about - what kind of functions do you intend
> > to write?  Often examples help.
> > 
> Fix functions at both the post IPA  and pre RTL levels for avoiding sharing of state and increasing 
> throughput with multiple threads in the compiler is what I actually meant. There aren't any new 
> functions it's optimizing the paths. So it was a small typo. Thanks for noticing it.
> 
> I will submit either later today or early tomorrow so if there's anything else let me know, 

I think you will have to explain how you want to avoid sharing of state
and at least vaguely _what_ global state you want to avoid.  The other
proposal side-steps global state of individual GIMPLE passes by never
executing the same GIMPLE pass twice at the same time for example,
so it's the scheduler avoiding shared state.  What other global state
do you have in mind?

Richard.

Re: GSOC Proposal

[nick]

On 2019-04-08 9:42 a.m., Richard Biener wrote:
> On Mon, 8 Apr 2019, nick wrote:
> 
>>
>>
>> On 2019-04-08 3:29 a.m., Richard Biener wrote:
>>> On Sun, 7 Apr 2019, nick wrote:
>>>
>>>>
>>>>
>>>> On 2019-04-07 5:31 a.m., Richard Biener wrote:
>>>>> On April 5, 2019 6:11:15 PM GMT+02:00, nick <xerofoify@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> On 2019-04-05 6:25 a.m., Richard Biener wrote:
>>>>>>> On Wed, 3 Apr 2019, nick wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2019-04-03 7:30 a.m., Richard Biener wrote:
>>>>>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2019-04-01 9:47 a.m., Richard Biener wrote:
>>>>>>>>>>> On Mon, 1 Apr 2019, nick wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Well I'm talking about the shared roots of this garbage
>>>>>> collector core state 
>>>>>>>>>>>> data structure or just struct ggc_root_tab.
>>>>>>>>>>>>
>>>>>>>>>>>> But also this seems that this to be no longer shared globally if
>>>>>> I'm not mistaken 
>>>>>>>>>>>> or this:
>>>>>>>>>>>> static vec<const_ggc_root_tab_t> extra_root_vec;
>>>>>>>>>>>>
>>>>>>>>>>>> Not sure after reading the code which is a bigger deal through
>>>>>> so I wrote
>>>>>>>>>>>> my proposal not just asking which is a better issue for not
>>>>>> being thread
>>>>>>>>>>>> safe. Sorry about that.
>>>>>>>>>>>>
>>>>>>>>>>>> As for the second question injection seems to not be the issue
>>>>>> or outside
>>>>>>>>>>>> callers but just internal so phase 3 or step 3 would now be:
>>>>>>>>>>>> Find internal callers or users of x where x is one of the above
>>>>>> rather
>>>>>>>>>>>> than injecting outside callers. Which answers my second question
>>>>>> about
>>>>>>>>>>>> external callers being a issue still.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know which  of the two is a better issue:
>>>>>>>>>>>> 1. struct ggc_root_tabs being shared
>>>>>>>>>>>> 2.static vec<const_ggc_root_tab_t> extra_root_vec; as a shared
>>>>>> heap or
>>>>>>>>>>>> vector of root nodes for each type of allocation
>>>>>>>>>>>>
>>>>>>>>>>>> and I will gladly rewrite my proposal sections for that
>>>>>>>>>>>> as needs to be reedited.
>>>>>>>>>>>
>>>>>>>>>>> I don't think working on the garbage collector as a separate
>>>>>>>>>>> GSoC project is useful at this point.  Doing locking around
>>>>>>>>>>> allocation seems like a good short-term solution and if that
>>>>>>>>>>> turns out to be a performance issue for the threaded part
>>>>>>>>>>> using per-thread freelists is likely an easy to deploy
>>>>>>>>>>> solution.
>>>>>>>>>>>
>>>>>>>>>>> Richard.
>>>>>>>>>>>
>>>>>>>>>> I agree but we were discussing this:
>>>>>>>>>> Or maybe a project to be more
>>>>>>>>>> explicit about regions of the code that assume that the garbage-
>>>>>>>>>> collector can't run within them?[3] (since the GC is state that
>>>>>> would
>>>>>>>>>> be shared by the threads).
>>>>>>>>>
>>>>>>>>> The process of collecting garbage is not the only issue (and that
>>>>>>>>> very issue is easiest mitigated by collecting only at specific
>>>>>>>>> points - which is what we do - and have those be serializing
>>>>>> points).
>>>>>>>>> The main issue is the underlying memory allocator (GCC uses memory
>>>>>>>>> that is garbage collected plus regular heap memory).
>>>>>>>>>
>>>>>>>>>> In addition I moved my paper back to our discussion about garbage
>>>>>> collector
>>>>>>>>>> state with outside callers.Seems we really need to do something
>>>>>> about
>>>>>>>>>> my wording as the idea of my project in a nutshell was to figure
>>>>>>>>>> out how to mark shared state by callers and inject it into the
>>>>>>>>>> garbage collector letting it known that the state was not shared
>>>>>> between
>>>>>>>>>> threads or shared. Seems that was on the GSoc page and in our
>>>>>> discussions the issue
>>>>>>>>>> is marking outside code for shared state. If that's correct then
>>>>>> my
>>>>>>>>>> wording of outside callers is incorrect it should have been shared
>>>>>>>>>> state between threads on outside callers to the garbage collector.
>>>>>>>>>> If the state is that in your wording above then great as I
>>>>>> understand
>>>>>>>>>> where we are going and will gladly change my wording.
>>>>>>>>>
>>>>>>>>> I'm still not sure what you are shooting at, the above sentences do
>>>>>>>>> not make any sense to me.
>>>>>>>>>
>>>>>>>>>> Also freelists don't work here as the state is shared at the
>>>>>> caller's 
>>>>>>>>>> end which would need two major issues:
>>>>>>>>>> 1. Locking on nodes of the 
>>>>>>>>>> freelists when two threads allocate at the same thing which can be
>>>>>> a 
>>>>>>>>>> problem if the shared state is shared a lot
>>>>>>>>>> 2. Locking allocation with 
>>>>>>>>>> large numbers of callers can starve threads
>>>>>>>>>
>>>>>>>>> First of all allocating memory from the GC pool is not the main
>>>>>>>>> work of GIMPLE passes so simply serializing at allocation time
>>>>>> might
>>>>>>>>> work out.  Second free lists of course do work.  What you'd do is
>>>>>>>>> have a fast path in allocation using a thread-local "free list"
>>>>>>>>> which you can allocate from without taking any lock.  Maybe I
>>>>>> should
>>>>>>>>> explain "free list" since that term doesn't make too much sense in
>>>>>>>>> a garbage collector world.  What I'd do is when a client thread
>>>>>>>>> asks for memory of size N allocate M objects of that size but put
>>>>>>>>> M - 1 on the client thread local "free list" to be allocated
>>>>>> lock-free
>>>>>>>>> from for the next M - 1 calls.  Note that garbage collected memory
>>>>>>>>> objects are only handed out in fixed chunks (powers of two plus
>>>>>>>>> a few special sizes) so you'd have one "free list" per chunk size
>>>>>>>>> per thread.
>>>>>>>>>
>>>>>>>>> The collection itself (mark & sweep) would be fully serialized
>>>>>> still
>>>>>>>>> (and not return to any threads local "free list").
>>>>>>>>>
>>>>>>>>> ggc_free'd objects _might_ go to the threads "free list"s (yeah, we
>>>>>>>>> _do_ have ggc_free ...).
>>>>>>>>>
>>>>>>>>> As said, I don't see GC or the memory allocator as sth interesting
>>>>>>>>> to work on for parallelization until the basic setup works and it
>>>>>>>>> proves to be a bottleneck.
>>>>>>>>>
>>>>>>>>>> Seems that working on the garbage collector itself isn't the issue
>>>>>> but 
>>>>>>>>>> the callers as I just figured out as related to your state idea.
>>>>>> Let me 
>>>>>>>>>> know if that's correct and if the wording change I mentioned is
>>>>>> fine 
>>>>>>>>>> with you as that's the state it seems that needs to be changed.
>>>>>>>>>> Nick 
>>>>>>>>>
>>>>>>>>> Richard.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That's fine and it's my fault for not understanding you better. I
>>>>>> was aware 
>>>>>>>> of the expand_functions_all being taken for passes.c. However it
>>>>>> seems
>>>>>>>> two other issues are these sets as related to threads:
>>>>>>>> 1.finalize_compilation_unit
>>>>>>>> 2.and the ipa set of pass functions
>>>>>>>>
>>>>>>>> If I'm understanding it correctly number 1 seems to be a early
>>>>>> version of
>>>>>>>> expand_all_functions for the GENERIC representation if that's the
>>>>>> case
>>>>>>>> it really should be fixed. Not sure which is a better issue as both
>>>>>>>> seem to have issues either at the GENERIC level or GIMPLE level with
>>>>>> shared
>>>>>>>> state.
>>>>>>>>
>>>>>>>> Let me know if this is better as it seems now that I really think
>>>>>> about 
>>>>>>>> it GIMPLE or GENERIC functions in passes.c are the main issue. 
>>>>>>>>
>>>>>>>> Sorry for the misunderstanding and hopefully one of functions listed
>>>>>> is better
>>>>>>>> for moving forward with my proposal,
>>>>>>>
>>>>>>> Sorry, but guessing at useful projects by skimming through GCC code
>>>>>>> at this point isn't the way to go forward - this new "idea" lacks
>>>>>>> both detail and understanding.  Please try to stick to one of the
>>>>>>> suggested projects or do more thorough research in case you want
>>>>>>> to work on a new project idea next year.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Richard.
>>>>>>>
>>>>>>
>>>>>> I was talking about cgraphunits.c and it seems that according to this:
>>>>>> Parallelize compilation using threads. GCC currently has an awful lot
>>>>>> of truly global state and even more per-pass global state which makes
>>>>>> this somewhat hard. The idea is to avoid issues with global state as
>>>>>> much as possible by partitioning the compilation pipeline in pieces
>>>>>> that share as little global state as possible and ensure each thread
>>>>>> only works in one of those partitions. The biggest roadblock will be
>>>>>> the not thread-safe memory allocator of GCC garbage collector. The goal
>>>>>> of this project is to have a compilation pipeline driven by a scheduler
>>>>>> assigning functions to be optimized to the partitions in the pipeline.
>>>>>> This project would be mentored by Richard Biener. Required skills
>>>>>> include: C/C++, ability to analyze big complex code base,
>>>>>> parallelization
>>>>>>
>>>>>> We are trying to create a rendering pipeline if I'm correct and it
>>>>>> seems that the GENERIC level needs finalize_compilation_unit
>>>>>> to be fixed like expand_all_functions at the GIMPLE. That's my point it
>>>>>> still is within that project. Here is what I wrote
>>>>>> as I figured out that it was shared state related to GENERIC passing to
>>>>>> GIMPLE which is a bottleneck or would be in the 
>>>>>> threaded pipeline.
>>>>>>
>>>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>>>>
>>>>> The pre- post-IPA parts cannot be easily parallelized and that includes GENERIC to GIMPLE translation. This is why the project should focus on the much easier post-IPA and pre-RTL parts of the compilation pipeline since there interaction between functions is minimal. 
>>>>>
>>>>> Richard. 
>>>>>
>>>>>>
>>>>>> Nick
>>>>>
>>>> Richard,
>>>> Thanks seems we finally got somewhere I rewrote it for that. Please just doubt check tomorrow before I submit later tomorrow
>>>> as I would just want to make sure it's OK with you. Here is the link:
>>>> https://docs.google.com/document/d/1BKVeh62IpigsQYf_fJqkdu_js0EeGdKtXInkWZ-DtU0/edit
>>>>
>>>> The only thing is if you want more detail in the project part as I just think it's
>>>> enough but let me know if not.
>>>
>>> It seems the project part lacks any specifics and those are to be
>>> researched and discovered in the Mid may - Late May timeframe of
>>> your Roadmap?
>> Yes this is because after looking into ipa functions it would take
>> some time to really understand it not just at a surface level. It's
>> not the functions themselves but thinking about how the pipeline
>> would work in the POST IPA and pre RTL functions. 
>>
>>> I do not understand what "Create optimized functions ... for avoiding
>>> sharing of state ..." is about - what kind of functions do you intend
>>> to write?  Often examples help.
>>>
>> Fix functions at both the post IPA  and pre RTL levels for avoiding sharing of state and increasing 
>> throughput with multiple threads in the compiler is what I actually meant. There aren't any new 
>> functions it's optimizing the paths. So it was a small typo. Thanks for noticing it.
>>
>> I will submit either later today or early tomorrow so if there's anything else let me know, 
> 
> I think you will have to explain how you want to avoid sharing of state
> and at least vaguely _what_ global state you want to avoid.  The other
> proposal side-steps global state of individual GIMPLE passes by never
> executing the same GIMPLE pass twice at the same time for example,
> so it's the scheduler avoiding shared state.  What other global state
> do you have in mind?
> 
> Richard.
> 

If I understanding correctly there are two at the IPA late pass we can pass over the program
partition multiple times that would be fixed. For pre RTL seems cgraph_function_versioning
has issues with copying functions more than once and then doing the transformation on different
threads multiple times. 

If that's enough info I will just edit the roadmap, introduction and project parts to explain
the details.

Nick

GSoC Proposal

[Abhigyan Kashyap]

GSOC proposal

[Ismael El Houas Ghouddana]

Dear Mr./Mrs,

First of all, I really appreciate your time and attention. I am Ismael El
Houas an aerospace engineer student with knowledge of Google Cloud Platform
and I want to express my interest in working on your project.

Secondly, I want to ask if I am still at a time to apply to this project,
unfortunately, I was not aware of GSOC until yesterday. In the case, I am
still able to apply for it, I will make the proposal as soon as possible.

Finally, many thanks for your attention.

Yours faithfully,

Ismael El Houas

Re: GSOC proposal

[Martin Jambor]

Hello Ismael,

On Wed, Mar 21 2018, Ismael El Houas Ghouddana wrote:
> Dear Mr./Mrs,
>
> First of all, I really appreciate your time and attention. I am Ismael El
> Houas an aerospace engineer student with knowledge of Google Cloud Platform
> and I want to express my interest in working on your project.

I am sorry to reply only now, mainly because of traveling, I was not
reading my email in the second half of last week.  

>
> Secondly, I want to ask if I am still at a time to apply to this project,
> unfortunately, I was not aware of GSOC until yesterday. In the case, I am
> still able to apply for it, I will make the proposal as soon as possible.

Strictly speaking, the deadline is tomorrow, as decided by the GSoC
organizers.  If you have been working on a proposal despite not hearing
from us, we would sill like to see it and encourage you to submit it
before the deadline.  If you haven't, it is really getting rather late,
unless you have a very clear idea of what you want to do (in that case
we should still try!).

My apologies again for missing you message, I hope GSoC works out for
you one way or another.

Martin

GSoC Proposal

[Sai kiran]

Dear GCC,

I have been told that the Project - "Implement regular expressions in
c++" mentored by Sir Benjamin De Kosnik is not completed and is
available for this year GSoC project also by the the Mentor.


Since my course of study for this semester is Generic Programming
using C++ and i am working on that project for last one and half
months are so.... i would like to complete that project if the you
give me a chance...

waiting for your reply..

thank you..
-- 
Saikiran A

6th Semester,
Computer Science and Engineering,
P.E.S. Institute of Technology,
Bangalore.

Re: GSoC Proposal

[Benjamin De Kosnik]

> I have been told that the Project - "Implement regular expressions in
> c++" mentored by Sir Benjamin De Kosnik is not completed and is
> available for this year GSoC project also by the the Mentor.

Sorry, there still appears to be some confusion here.

I am not mentoring GSOC this year. Here is some helpful information for
people who want to propose projects:
http://gcc.gnu.org/ml/gcc/2013-03/msg00082.html

-benjamin