State of AutoFDO in GCC

State of AutoFDO in GCC

[Eugene Rozenfeld]

GCC documentation for AutoFDO points to create_gcov tool that converts perf.data file into gcov format that can be consumed by gcc with -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, https://gcc.gnu.org/wiki/AutoFDO/Tutorial).

I noticed that the source code for create_gcov has been deleted from https://github.com/google/autofdo on April 7. I asked about that change in that repo and got the following reply:

https://github.com/google/autofdo/pull/107#issuecomment-819108738

"Actually we didn't use create_gcov and havn't updated create_gcov for years, and we also didn't have enough tests to guarantee it works (It was gcc-4.8 when we used and verified create_gcov). If you need it, it is welcomed to update create_gcov and add it to the respository."

Does this mean that AutoFDO is currently dead in gcc?

Thanks,

Eugene

Re: State of AutoFDO in GCC

[Martin Liška]

On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> GCC documentation for AutoFDO points to create_gcov tool that converts perf.data file into gcov format that can be consumed by gcc with -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> 
> I noticed that the source code for create_gcov has been deleted from https://github.com/google/autofdo on April 7. I asked about that change in that repo and got the following reply:
> 
> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> 
> "Actually we didn't use create_gcov and havn't updated create_gcov for years, and we also didn't have enough tests to guarantee it works (It was gcc-4.8 when we used and verified create_gcov). If you need it, it is welcomed to update create_gcov and add it to the respository."
> 
> Does this mean that AutoFDO is currently dead in gcc?

Hello.

Yes. I know that even basic test cases have been broken for years in the GCC.
It's new to me that create_gcov was removed.

I tend to send patch to GCC that will remove AutoFDO from GCC.
I known Bin spent some time working on AutoFDO, has he came up to something?

Martin

> 
> Thanks,
> 
> Eugene
> 

Re: State of AutoFDO in GCC

[Jan Hubicka]

> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > GCC documentation for AutoFDO points to create_gcov tool that converts perf.data file into gcov format that can be consumed by gcc with -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > 
> > I noticed that the source code for create_gcov has been deleted from https://github.com/google/autofdo on April 7. I asked about that change in that repo and got the following reply:
> > 
> > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > 
> > "Actually we didn't use create_gcov and havn't updated create_gcov for years, and we also didn't have enough tests to guarantee it works (It was gcc-4.8 when we used and verified create_gcov). If you need it, it is welcomed to update create_gcov and add it to the respository."
> > 
> > Does this mean that AutoFDO is currently dead in gcc?
> 
> Hello.
> 
> Yes. I know that even basic test cases have been broken for years in the GCC.
> It's new to me that create_gcov was removed.
> 
> I tend to send patch to GCC that will remove AutoFDO from GCC.
> I known Bin spent some time working on AutoFDO, has he came up to something?

The GCC side of auto-FDO is not that hard.  We have most of
infrastructure in place, but stopping point for me was always difficulty
to get gcov-tool working.  If some maintainer steps up, I think I can
fix GCC side.

I am bit unsure how important feature it is - we have FDO that works
quite well for most users but I know there are some users of the LLVM
implementation and there is potential to tie this with other hardware
events to asist i.e. if conversion (where one wants to know how well CPU
predicts the jump rather than just the jump probability) which I always
found potentially interesting.

Honza
> 
> Martin
> 
> > 
> > Thanks,
> > 
> > Eugene
> > 
> 

Re: State of AutoFDO in GCC

[Bin.Cheng]

On Fri, Apr 23, 2021 at 4:16 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > GCC documentation for AutoFDO points to create_gcov tool that converts perf.data file into gcov format that can be consumed by gcc with -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html, https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> >
> > I noticed that the source code for create_gcov has been deleted from https://github.com/google/autofdo on April 7. I asked about that change in that repo and got the following reply:
> >
> > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> >
> > "Actually we didn't use create_gcov and havn't updated create_gcov for years, and we also didn't have enough tests to guarantee it works (It was gcc-4.8 when we used and verified create_gcov). If you need it, it is welcomed to update create_gcov and add it to the respository."
> >
> > Does this mean that AutoFDO is currently dead in gcc?
>
> Hello.
>
> Yes. I know that even basic test cases have been broken for years in the GCC.
> It's new to me that create_gcov was removed.
>
> I tend to send patch to GCC that will remove AutoFDO from GCC.
> I known Bin spent some time working on AutoFDO, has he came up to something?
Hi Martin,
I haven't touched this part for quite some time.  I have no objection
to removing it from GCC.  However, I do have general concern that
because of fewer users/developers, it's less likely and harder for new
features to land in GCC.  I have no idea if this is a real problem or
how to fix it.  OTOH, maybe removing rotten features, making GCC
more(?) concise, and improving existing features that GCC is doing
well is the right thing.

Thanks,
bin
>
> Martin
>
> >
> > Thanks,
> >
> > Eugene
> >
>

Re: State of AutoFDO in GCC

[Xinliang David Li]

Hi, the create_gcov tool was probably removed with the assumption that it
was only used with Google GCC branch, but it is actually used with GCC
trunk as well.

Given that, the tool will be restored in the github repo. It seems to build
and work fine with the regression test.

The tool may ust work as it is right now, but there is no guarantee it
won't break in the future unless someone in the GCC community tries to
maintain it.

Thanks,

David

On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:

> > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > GCC documentation for AutoFDO points to create_gcov tool that converts
> perf.data file into gcov format that can be consumed by gcc with
> -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > >
> > > I noticed that the source code for create_gcov has been deleted from
> https://github.com/google/autofdo on April 7. I asked about that change
> in that repo and got the following reply:
> > >
> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > >
> > > "Actually we didn't use create_gcov and havn't updated create_gcov for
> years, and we also didn't have enough tests to guarantee it works (It was
> gcc-4.8 when we used and verified create_gcov). If you need it, it is
> welcomed to update create_gcov and add it to the respository."
> > >
> > > Does this mean that AutoFDO is currently dead in gcc?
> >
> > Hello.
> >
> > Yes. I know that even basic test cases have been broken for years in the
> GCC.
> > It's new to me that create_gcov was removed.
> >
> > I tend to send patch to GCC that will remove AutoFDO from GCC.
> > I known Bin spent some time working on AutoFDO, has he came up to
> something?
>
> The GCC side of auto-FDO is not that hard.  We have most of
> infrastructure in place, but stopping point for me was always difficulty
> to get gcov-tool working.  If some maintainer steps up, I think I can
> fix GCC side.
>
> I am bit unsure how important feature it is - we have FDO that works
> quite well for most users but I know there are some users of the LLVM
> implementation and there is potential to tie this with other hardware
> events to asist i.e. if conversion (where one wants to know how well CPU
> predicts the jump rather than just the jump probability) which I always
> found potentially interesting.
>
> Honza
> >
> > Martin
> >
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> >
>

Re: State of AutoFDO in GCC

[Richard Biener]

On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
<gcc@gcc.gnu.org> wrote:
>
> Hi, the create_gcov tool was probably removed with the assumption that it
> was only used with Google GCC branch, but it is actually used with GCC
> trunk as well.
>
> Given that, the tool will be restored in the github repo. It seems to build
> and work fine with the regression test.
>
> The tool may ust work as it is right now, but there is no guarantee it
> won't break in the future unless someone in the GCC community tries to
> maintain it.

I think if we want to keep the feature it makes sense to provide create_gcov
functionality either directly from perf (input data producer) or from gcc
(data consumer).  Of course I have no idea about its complexity, license
or implementation language ...

Having the tool third-party makes keeping the whole chain working more
difficult.

Richard.

> Thanks,
>
> David
>
> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
>
> > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > GCC documentation for AutoFDO points to create_gcov tool that converts
> > perf.data file into gcov format that can be consumed by gcc with
> > -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > >
> > > > I noticed that the source code for create_gcov has been deleted from
> > https://github.com/google/autofdo on April 7. I asked about that change
> > in that repo and got the following reply:
> > > >
> > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > >
> > > > "Actually we didn't use create_gcov and havn't updated create_gcov for
> > years, and we also didn't have enough tests to guarantee it works (It was
> > gcc-4.8 when we used and verified create_gcov). If you need it, it is
> > welcomed to update create_gcov and add it to the respository."
> > > >
> > > > Does this mean that AutoFDO is currently dead in gcc?
> > >
> > > Hello.
> > >
> > > Yes. I know that even basic test cases have been broken for years in the
> > GCC.
> > > It's new to me that create_gcov was removed.
> > >
> > > I tend to send patch to GCC that will remove AutoFDO from GCC.
> > > I known Bin spent some time working on AutoFDO, has he came up to
> > something?
> >
> > The GCC side of auto-FDO is not that hard.  We have most of
> > infrastructure in place, but stopping point for me was always difficulty
> > to get gcov-tool working.  If some maintainer steps up, I think I can
> > fix GCC side.
> >
> > I am bit unsure how important feature it is - we have FDO that works
> > quite well for most users but I know there are some users of the LLVM
> > implementation and there is potential to tie this with other hardware
> > events to asist i.e. if conversion (where one wants to know how well CPU
> > predicts the jump rather than just the jump probability) which I always
> > found potentially interesting.
> >
> > Honza
> > >
> > > Martin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Eugene
> > > >
> > >
> >

Re: State of AutoFDO in GCC

[Martin Liška]

On 4/23/21 9:00 AM, Richard Biener via Gcc wrote:
> On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
> <gcc@gcc.gnu.org> wrote:
>>
>> Hi, the create_gcov tool was probably removed with the assumption that it
>> was only used with Google GCC branch, but it is actually used with GCC
>> trunk as well.
>>
>> Given that, the tool will be restored in the github repo. It seems to build
>> and work fine with the regression test.
>>
>> The tool may ust work as it is right now, but there is no guarantee it
>> won't break in the future unless someone in the GCC community tries to
>> maintain it.

Hi.

The current situation is that AutoFDO doesn't work with pretty simple test-cases
we have in testsuite:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379

These are ~5 years old and nothing has happened.

I'm pretty sure the current autofdo can't emit a .gcda file format that
I've changed in the recent years.

> 
> I think if we want to keep the feature it makes sense to provide create_gcov
> functionality either directly from perf (input data producer) or from gcc
> (data consumer).  Of course I have no idea about its complexity, license
> or implementation language ...

For me, it's just an i386 feature (maybe aarch64 has perf counters too?), supported
only by vendor (Intel) and I'm not planning working on that.
I don't like having a feature that is obviously broken and potential GCC users get
bad experience every time they try to use it.

Can we at least deprecate the feature for GCC 11? If these is enough interest,
we can fix it, if not, I would remove it in GCC 13 timeframe.

Thoughts?
Martin

> 
> Having the tool third-party makes keeping the whole chain working more
> difficult.
> 
> Richard.
> 
>> Thanks,
>>
>> David
>>
>> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
>>
>>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
>>>>> GCC documentation for AutoFDO points to create_gcov tool that converts
>>> perf.data file into gcov format that can be consumed by gcc with
>>> -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
>>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
>>>>>
>>>>> I noticed that the source code for create_gcov has been deleted from
>>> https://github.com/google/autofdo on April 7. I asked about that change
>>> in that repo and got the following reply:
>>>>>
>>>>> https://github.com/google/autofdo/pull/107#issuecomment-819108738
>>>>>
>>>>> "Actually we didn't use create_gcov and havn't updated create_gcov for
>>> years, and we also didn't have enough tests to guarantee it works (It was
>>> gcc-4.8 when we used and verified create_gcov). If you need it, it is
>>> welcomed to update create_gcov and add it to the respository."
>>>>>
>>>>> Does this mean that AutoFDO is currently dead in gcc?
>>>>
>>>> Hello.
>>>>
>>>> Yes. I know that even basic test cases have been broken for years in the
>>> GCC.
>>>> It's new to me that create_gcov was removed.
>>>>
>>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
>>>> I known Bin spent some time working on AutoFDO, has he came up to
>>> something?
>>>
>>> The GCC side of auto-FDO is not that hard.  We have most of
>>> infrastructure in place, but stopping point for me was always difficulty
>>> to get gcov-tool working.  If some maintainer steps up, I think I can
>>> fix GCC side.
>>>
>>> I am bit unsure how important feature it is - we have FDO that works
>>> quite well for most users but I know there are some users of the LLVM
>>> implementation and there is potential to tie this with other hardware
>>> events to asist i.e. if conversion (where one wants to know how well CPU
>>> predicts the jump rather than just the jump probability) which I always
>>> found potentially interesting.
>>>
>>> Honza
>>>>
>>>> Martin
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Eugene
>>>>>
>>>>
>>>

Re: State of AutoFDO in GCC

[Richard Biener]

On Fri, Apr 23, 2021 at 9:18 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 4/23/21 9:00 AM, Richard Biener via Gcc wrote:
> > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
> > <gcc@gcc.gnu.org> wrote:
> >>
> >> Hi, the create_gcov tool was probably removed with the assumption that it
> >> was only used with Google GCC branch, but it is actually used with GCC
> >> trunk as well.
> >>
> >> Given that, the tool will be restored in the github repo. It seems to build
> >> and work fine with the regression test.
> >>
> >> The tool may ust work as it is right now, but there is no guarantee it
> >> won't break in the future unless someone in the GCC community tries to
> >> maintain it.
>
> Hi.
>
> The current situation is that AutoFDO doesn't work with pretty simple test-cases
> we have in testsuite:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379
>
> These are ~5 years old and nothing has happened.
>
> I'm pretty sure the current autofdo can't emit a .gcda file format that
> I've changed in the recent years.
>
> >
> > I think if we want to keep the feature it makes sense to provide create_gcov
> > functionality either directly from perf (input data producer) or from gcc
> > (data consumer).  Of course I have no idea about its complexity, license
> > or implementation language ...
>
> For me, it's just an i386 feature (maybe aarch64 has perf counters too?), supported
> only by vendor (Intel) and I'm not planning working on that.
> I don't like having a feature that is obviously broken and potential GCC users get
> bad experience every time they try to use it.
>
> Can we at least deprecate the feature for GCC 11? If these is enough interest,
> we can fix it, if not, I would remove it in GCC 13 timeframe.

I have no opinion here and I'm not opposed to ripping out the feature
from GCC 12.
Since it doesn't work anyway there's no need for any deprecation.

Richard.

> Thoughts?
> Martin
>
> >
> > Having the tool third-party makes keeping the whole chain working more
> > difficult.
> >
> > Richard.
> >
> >> Thanks,
> >>
> >> David
> >>
> >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> >>
> >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> >>>>> GCC documentation for AutoFDO points to create_gcov tool that converts
> >>> perf.data file into gcov format that can be consumed by gcc with
> >>> -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> >>>>>
> >>>>> I noticed that the source code for create_gcov has been deleted from
> >>> https://github.com/google/autofdo on April 7. I asked about that change
> >>> in that repo and got the following reply:
> >>>>>
> >>>>> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> >>>>>
> >>>>> "Actually we didn't use create_gcov and havn't updated create_gcov for
> >>> years, and we also didn't have enough tests to guarantee it works (It was
> >>> gcc-4.8 when we used and verified create_gcov). If you need it, it is
> >>> welcomed to update create_gcov and add it to the respository."
> >>>>>
> >>>>> Does this mean that AutoFDO is currently dead in gcc?
> >>>>
> >>>> Hello.
> >>>>
> >>>> Yes. I know that even basic test cases have been broken for years in the
> >>> GCC.
> >>>> It's new to me that create_gcov was removed.
> >>>>
> >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> >>>> I known Bin spent some time working on AutoFDO, has he came up to
> >>> something?
> >>>
> >>> The GCC side of auto-FDO is not that hard.  We have most of
> >>> infrastructure in place, but stopping point for me was always difficulty
> >>> to get gcov-tool working.  If some maintainer steps up, I think I can
> >>> fix GCC side.
> >>>
> >>> I am bit unsure how important feature it is - we have FDO that works
> >>> quite well for most users but I know there are some users of the LLVM
> >>> implementation and there is potential to tie this with other hardware
> >>> events to asist i.e. if conversion (where one wants to know how well CPU
> >>> predicts the jump rather than just the jump probability) which I always
> >>> found potentially interesting.
> >>>
> >>> Honza
> >>>>
> >>>> Martin
> >>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Eugene
> >>>>>
> >>>>
> >>>
>

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Fri, Apr 23, 2021 at 12:00 AM Richard Biener <richard.guenther@gmail.com>
wrote:

> On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
> <gcc@gcc.gnu.org> wrote:
> >
> > Hi, the create_gcov tool was probably removed with the assumption that it
> > was only used with Google GCC branch, but it is actually used with GCC
> > trunk as well.
> >
> > Given that, the tool will be restored in the github repo. It seems to
> build
> > and work fine with the regression test.
> >
> > The tool may ust work as it is right now, but there is no guarantee it
> > won't break in the future unless someone in the GCC community tries to
> > maintain it.
>
> I think if we want to keep the feature it makes sense to provide
> create_gcov
> functionality either directly from perf (input data producer) or from gcc
> (data consumer).  Of course I have no idea about its complexity, license
> or implementation language ...
>
>
Right. What it takes is a perf data (can be text format) parser to produce
the format GCC needs, but someone in the community needs to take the lead.
It should not involve too much effort.

David

Having the tool third-party makes keeping the whole chain working more
> difficult.
>
> Richard.
>
> > Thanks,
> >
> > David
> >
> > On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> >
> > > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > > GCC documentation for AutoFDO points to create_gcov tool that
> converts
> > > perf.data file into gcov format that can be consumed by gcc with
> > > -fauto-profile (
> https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > > >
> > > > > I noticed that the source code for create_gcov has been deleted
> from
> > > https://github.com/google/autofdo on April 7. I asked about that
> change
> > > in that repo and got the following reply:
> > > > >
> > > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > > >
> > > > > "Actually we didn't use create_gcov and havn't updated create_gcov
> for
> > > years, and we also didn't have enough tests to guarantee it works (It
> was
> > > gcc-4.8 when we used and verified create_gcov). If you need it, it is
> > > welcomed to update create_gcov and add it to the respository."
> > > > >
> > > > > Does this mean that AutoFDO is currently dead in gcc?
> > > >
> > > > Hello.
> > > >
> > > > Yes. I know that even basic test cases have been broken for years in
> the
> > > GCC.
> > > > It's new to me that create_gcov was removed.
> > > >
> > > > I tend to send patch to GCC that will remove AutoFDO from GCC.
> > > > I known Bin spent some time working on AutoFDO, has he came up to
> > > something?
> > >
> > > The GCC side of auto-FDO is not that hard.  We have most of
> > > infrastructure in place, but stopping point for me was always
> difficulty
> > > to get gcov-tool working.  If some maintainer steps up, I think I can
> > > fix GCC side.
> > >
> > > I am bit unsure how important feature it is - we have FDO that works
> > > quite well for most users but I know there are some users of the LLVM
> > > implementation and there is potential to tie this with other hardware
> > > events to asist i.e. if conversion (where one wants to know how well
> CPU
> > > predicts the jump rather than just the jump probability) which I always
> > > found potentially interesting.
> > >
> > > Honza
> > > >
> > > > Martin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Eugene
> > > > >
> > > >
> > >
>

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Fri, Apr 23, 2021 at 12:18 AM Martin Liška <mliska@suse.cz> wrote:

> On 4/23/21 9:00 AM, Richard Biener via Gcc wrote:
> > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
> > <gcc@gcc.gnu.org> wrote:
> >>
> >> Hi, the create_gcov tool was probably removed with the assumption that
> it
> >> was only used with Google GCC branch, but it is actually used with GCC
> >> trunk as well.
> >>
> >> Given that, the tool will be restored in the github repo. It seems to
> build
> >> and work fine with the regression test.
> >>
> >> The tool may ust work as it is right now, but there is no guarantee it
> >> won't break in the future unless someone in the GCC community tries to
> >> maintain it.
>
> Hi.
>
> The current situation is that AutoFDO doesn't work with pretty simple
> test-cases
> we have in testsuite:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379
>
> These are ~5 years old and nothing has happened.
>
> I'm pretty sure the current autofdo can't emit a .gcda file format that
> I've changed in the recent years.
>

I have not not looked into the details, but is it possible the problem is
at GCC side (e.g, debug info quality etc)?

>
> >
> > I think if we want to keep the feature it makes sense to provide
> create_gcov
> > functionality either directly from perf (input data producer) or from gcc
> > (data consumer).  Of course I have no idea about its complexity, license
> > or implementation language ...
>
> For me, it's just an i386 feature (maybe aarch64 has perf counters too?),
> supported
> only by vendor (Intel) and I'm not planning working on that.
> I don't like having a feature that is obviously broken and potential GCC
> users get
> bad experience every time they try to use it.
>

It must be working at sometime, so perhaps a bisect of GCC revisions can
reveal when it regressed.


> Can we at least deprecate the feature for GCC 11? If these is enough
> interest,
> we can fix it, if not, I would remove it in GCC 13 timeframe.
>

This is a major feature in Clang. Deprecating it in GCC will be unfortunate.

David

>
> Thoughts?
> Martin
>
> >
> > Having the tool third-party makes keeping the whole chain working more
> > difficult.
> >
> > Richard.
> >
> >> Thanks,
> >>
> >> David
> >>
> >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> >>
> >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> >>>>> GCC documentation for AutoFDO points to create_gcov tool that
> converts
> >>> perf.data file into gcov format that can be consumed by gcc with
> >>> -fauto-profile (
> https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> >>>>>
> >>>>> I noticed that the source code for create_gcov has been deleted from
> >>> https://github.com/google/autofdo on April 7. I asked about that
> change
> >>> in that repo and got the following reply:
> >>>>>
> >>>>> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> >>>>>
> >>>>> "Actually we didn't use create_gcov and havn't updated create_gcov
> for
> >>> years, and we also didn't have enough tests to guarantee it works (It
> was
> >>> gcc-4.8 when we used and verified create_gcov). If you need it, it is
> >>> welcomed to update create_gcov and add it to the respository."
> >>>>>
> >>>>> Does this mean that AutoFDO is currently dead in gcc?
> >>>>
> >>>> Hello.
> >>>>
> >>>> Yes. I know that even basic test cases have been broken for years in
> the
> >>> GCC.
> >>>> It's new to me that create_gcov was removed.
> >>>>
> >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> >>>> I known Bin spent some time working on AutoFDO, has he came up to
> >>> something?
> >>>
> >>> The GCC side of auto-FDO is not that hard.  We have most of
> >>> infrastructure in place, but stopping point for me was always
> difficulty
> >>> to get gcov-tool working.  If some maintainer steps up, I think I can
> >>> fix GCC side.
> >>>
> >>> I am bit unsure how important feature it is - we have FDO that works
> >>> quite well for most users but I know there are some users of the LLVM
> >>> implementation and there is potential to tie this with other hardware
> >>> events to asist i.e. if conversion (where one wants to know how well
> CPU
> >>> predicts the jump rather than just the jump probability) which I always
> >>> found potentially interesting.
> >>>
> >>> Honza
> >>>>
> >>>> Martin
> >>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Eugene
> >>>>>
> >>>>
> >>>
>
>

Re: State of AutoFDO in GCC

[Jan Hubicka]

> On Fri, Apr 23, 2021 at 12:18 AM Martin Liška <mliska@suse.cz> wrote:
> 
> > On 4/23/21 9:00 AM, Richard Biener via Gcc wrote:
> > > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
> > > <gcc@gcc.gnu.org> wrote:
> > >>
> > >> Hi, the create_gcov tool was probably removed with the assumption that
> > it
> > >> was only used with Google GCC branch, but it is actually used with GCC
> > >> trunk as well.
> > >>
> > >> Given that, the tool will be restored in the github repo. It seems to
> > build
> > >> and work fine with the regression test.
> > >>
> > >> The tool may ust work as it is right now, but there is no guarantee it
> > >> won't break in the future unless someone in the GCC community tries to
> > >> maintain it.
> >
> > Hi.
> >
> > The current situation is that AutoFDO doesn't work with pretty simple
> > test-cases
> > we have in testsuite:
> >
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379
> >
> > These are ~5 years old and nothing has happened.

Basic functionality is to identify hot parts of programs and give basic
idea of branch probabilities.  Bugs above are about other optimizations
(indirect inlining, hot-cold partitioning and peeling). First one ought
to be working but still indirect call profiling accounts for relatively
minor portion of FDO benefits, so auto-FDO should be usefable even w/o
these working if tooling was fixed.  Expecting hot-cold partitioning and
peeling to work reliably for testcases designed for normalFDO is IMO bit
unrealistic.

Perf is not limited to x86, so we should get basic usability for all
major architectures.  David, what CPUs are supported with auto-FDO on
LLVM side?
> 
> This is a major feature in Clang. Deprecating it in GCC will be unfortunate.

I also like overall idea of auto-FDO just never had much time to get it
into shape.  Every stage1 I am trying to fix something that was broken
for a while (like ICF last stage1) and fix it, so option would be to
focus on autoFDO this stage1 even though I have quite few other things
in TODO list.  I never looked into the tools translating perf data to
gcc compatible format and how easy would be to keep this alive.

David, how does perf->LLVM path work these days?

Honza
> 
> David
> 
> >
> > Thoughts?
> > Martin
> >
> > >
> > > Having the tool third-party makes keeping the whole chain working more
> > > difficult.
> > >
> > > Richard.
> > >
> > >> Thanks,
> > >>
> > >> David
> > >>
> > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> > >>
> > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > >>>>> GCC documentation for AutoFDO points to create_gcov tool that
> > converts
> > >>> perf.data file into gcov format that can be consumed by gcc with
> > >>> -fauto-profile (
> > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > >>>>>
> > >>>>> I noticed that the source code for create_gcov has been deleted from
> > >>> https://github.com/google/autofdo on April 7. I asked about that
> > change
> > >>> in that repo and got the following reply:
> > >>>>>
> > >>>>> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > >>>>>
> > >>>>> "Actually we didn't use create_gcov and havn't updated create_gcov
> > for
> > >>> years, and we also didn't have enough tests to guarantee it works (It
> > was
> > >>> gcc-4.8 when we used and verified create_gcov). If you need it, it is
> > >>> welcomed to update create_gcov and add it to the respository."
> > >>>>>
> > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > >>>>
> > >>>> Hello.
> > >>>>
> > >>>> Yes. I know that even basic test cases have been broken for years in
> > the
> > >>> GCC.
> > >>>> It's new to me that create_gcov was removed.
> > >>>>
> > >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> > >>>> I known Bin spent some time working on AutoFDO, has he came up to
> > >>> something?
> > >>>
> > >>> The GCC side of auto-FDO is not that hard.  We have most of
> > >>> infrastructure in place, but stopping point for me was always
> > difficulty
> > >>> to get gcov-tool working.  If some maintainer steps up, I think I can
> > >>> fix GCC side.
> > >>>
> > >>> I am bit unsure how important feature it is - we have FDO that works
> > >>> quite well for most users but I know there are some users of the LLVM
> > >>> implementation and there is potential to tie this with other hardware
> > >>> events to asist i.e. if conversion (where one wants to know how well
> > CPU
> > >>> predicts the jump rather than just the jump probability) which I always
> > >>> found potentially interesting.
> > >>>
> > >>> Honza
> > >>>>
> > >>>> Martin
> > >>>>
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Eugene
> > >>>>>
> > >>>>
> > >>>
> >
> >

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Fri, Apr 23, 2021 at 9:54 AM Jan Hubicka <hubicka@ucw.cz> wrote:

> > On Fri, Apr 23, 2021 at 12:18 AM Martin Liška <mliska@suse.cz> wrote:
> >
> > > On 4/23/21 9:00 AM, Richard Biener via Gcc wrote:
> > > > On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
> > > > <gcc@gcc.gnu.org> wrote:
> > > >>
> > > >> Hi, the create_gcov tool was probably removed with the assumption
> that
> > > it
> > > >> was only used with Google GCC branch, but it is actually used with
> GCC
> > > >> trunk as well.
> > > >>
> > > >> Given that, the tool will be restored in the github repo. It seems
> to
> > > build
> > > >> and work fine with the regression test.
> > > >>
> > > >> The tool may ust work as it is right now, but there is no guarantee
> it
> > > >> won't break in the future unless someone in the GCC community tries
> to
> > > >> maintain it.
> > >
> > > Hi.
> > >
> > > The current situation is that AutoFDO doesn't work with pretty simple
> > > test-cases
> > > we have in testsuite:
> > >
> > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
> > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379
> > >
> > > These are ~5 years old and nothing has happened.
>
> Basic functionality is to identify hot parts of programs and give basic
> idea of branch probabilities.  Bugs above are about other optimizations
> (indirect inlining, hot-cold partitioning and peeling). First one ought
> to be working but still indirect call profiling accounts for relatively
> minor portion of FDO benefits, so auto-FDO should be usefable even w/o
> these working if tooling was fixed.  Expecting hot-cold partitioning and
> peeling to work reliably for testcases designed for normalFDO is IMO bit
> unrealistic.
>
> Perf is not limited to x86, so we should get basic usability for all
> major architectures.  David, what CPUs are supported with auto-FDO on
> LLVM side?
> >
> > This is a major feature in Clang. Deprecating it in GCC will be
> unfortunate.
>
> I also like overall idea of auto-FDO just never had much time to get it
> into shape.  Every stage1 I am trying to fix something that was broken
> for a while (like ICF last stage1) and fix it, so option would be to
> focus on autoFDO this stage1 even though I have quite few other things
> in TODO list.  I never looked into the tools translating perf data to
> gcc compatible format and how easy would be to keep this alive.
>
> David, how does perf->LLVM path work these days?
>

It uses create_llvm_prof tool which is in the same git repo. The data
parsing part is shared with create_gcov, but the writer is obviously
different for the two tools.

David


> Honza
> >
> > David
> >
> > >
> > > Thoughts?
> > > Martin
> > >
> > > >
> > > > Having the tool third-party makes keeping the whole chain working
> more
> > > > difficult.
> > > >
> > > > Richard.
> > > >
> > > >> Thanks,
> > > >>
> > > >> David
> > > >>
> > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> > > >>
> > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > >>>>> GCC documentation for AutoFDO points to create_gcov tool that
> > > converts
> > > >>> perf.data file into gcov format that can be consumed by gcc with
> > > >>> -fauto-profile (
> > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > >>>>>
> > > >>>>> I noticed that the source code for create_gcov has been deleted
> from
> > > >>> https://github.com/google/autofdo on April 7. I asked about that
> > > change
> > > >>> in that repo and got the following reply:
> > > >>>>>
> > > >>>>>
> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > >>>>>
> > > >>>>> "Actually we didn't use create_gcov and havn't updated
> create_gcov
> > > for
> > > >>> years, and we also didn't have enough tests to guarantee it works
> (It
> > > was
> > > >>> gcc-4.8 when we used and verified create_gcov). If you need it, it
> is
> > > >>> welcomed to update create_gcov and add it to the respository."
> > > >>>>>
> > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > > >>>>
> > > >>>> Hello.
> > > >>>>
> > > >>>> Yes. I know that even basic test cases have been broken for years
> in
> > > the
> > > >>> GCC.
> > > >>>> It's new to me that create_gcov was removed.
> > > >>>>
> > > >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> > > >>>> I known Bin spent some time working on AutoFDO, has he came up to
> > > >>> something?
> > > >>>
> > > >>> The GCC side of auto-FDO is not that hard.  We have most of
> > > >>> infrastructure in place, but stopping point for me was always
> > > difficulty
> > > >>> to get gcov-tool working.  If some maintainer steps up, I think I
> can
> > > >>> fix GCC side.
> > > >>>
> > > >>> I am bit unsure how important feature it is - we have FDO that
> works
> > > >>> quite well for most users but I know there are some users of the
> LLVM
> > > >>> implementation and there is potential to tie this with other
> hardware
> > > >>> events to asist i.e. if conversion (where one wants to know how
> well
> > > CPU
> > > >>> predicts the jump rather than just the jump probability) which I
> always
> > > >>> found potentially interesting.
> > > >>>
> > > >>> Honza
> > > >>>>
> > > >>>> Martin
> > > >>>>
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>>
> > > >>>>> Eugene
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> > >
>

Re: State of AutoFDO in GCC

[Jan Hubicka]

> 
> It uses create_llvm_prof tool which is in the same git repo. The data
> parsing part is shared with create_gcov, but the writer is obviously
> different for the two tools.

OK and what are the main differences between llvmand gcc format?

Honza
> 
> David
> 
> 
> > Honza
> > >
> > > David
> > >
> > > >
> > > > Thoughts?
> > > > Martin
> > > >
> > > > >
> > > > > Having the tool third-party makes keeping the whole chain working
> > more
> > > > > difficult.
> > > > >
> > > > > Richard.
> > > > >
> > > > >> Thanks,
> > > > >>
> > > > >> David
> > > > >>
> > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> > > > >>
> > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool that
> > > > converts
> > > > >>> perf.data file into gcov format that can be consumed by gcc with
> > > > >>> -fauto-profile (
> > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > > >>>>>
> > > > >>>>> I noticed that the source code for create_gcov has been deleted
> > from
> > > > >>> https://github.com/google/autofdo on April 7. I asked about that
> > > > change
> > > > >>> in that repo and got the following reply:
> > > > >>>>>
> > > > >>>>>
> > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > > >>>>>
> > > > >>>>> "Actually we didn't use create_gcov and havn't updated
> > create_gcov
> > > > for
> > > > >>> years, and we also didn't have enough tests to guarantee it works
> > (It
> > > > was
> > > > >>> gcc-4.8 when we used and verified create_gcov). If you need it, it
> > is
> > > > >>> welcomed to update create_gcov and add it to the respository."
> > > > >>>>>
> > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > > > >>>>
> > > > >>>> Hello.
> > > > >>>>
> > > > >>>> Yes. I know that even basic test cases have been broken for years
> > in
> > > > the
> > > > >>> GCC.
> > > > >>>> It's new to me that create_gcov was removed.
> > > > >>>>
> > > > >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> > > > >>>> I known Bin spent some time working on AutoFDO, has he came up to
> > > > >>> something?
> > > > >>>
> > > > >>> The GCC side of auto-FDO is not that hard.  We have most of
> > > > >>> infrastructure in place, but stopping point for me was always
> > > > difficulty
> > > > >>> to get gcov-tool working.  If some maintainer steps up, I think I
> > can
> > > > >>> fix GCC side.
> > > > >>>
> > > > >>> I am bit unsure how important feature it is - we have FDO that
> > works
> > > > >>> quite well for most users but I know there are some users of the
> > LLVM
> > > > >>> implementation and there is potential to tie this with other
> > hardware
> > > > >>> events to asist i.e. if conversion (where one wants to know how
> > well
> > > > CPU
> > > > >>> predicts the jump rather than just the jump probability) which I
> > always
> > > > >>> found potentially interesting.
> > > > >>>
> > > > >>> Honza
> > > > >>>>
> > > > >>>> Martin
> > > > >>>>
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>>
> > > > >>>>> Eugene
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > >
> > > >
> >

Re: State of AutoFDO in GCC

[Jan Hubicka]

> Hi.
> 
> The current situation is that AutoFDO doesn't work with pretty simple test-cases
> we have in testsuite:
> 
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379
> 
> These are ~5 years old and nothing has happened.
> 
> I'm pretty sure the current autofdo can't emit a .gcda file format that
> I've changed in the recent years.

I believe that the gcda files consumed by auto-FDO are not really
sharing the normal gcov file format, just the low level i/o.
See read_profile in auto-profile.c.

Honza
> 
> > 
> > I think if we want to keep the feature it makes sense to provide create_gcov
> > functionality either directly from perf (input data producer) or from gcc
> > (data consumer).  Of course I have no idea about its complexity, license
> > or implementation language ...
> 
> For me, it's just an i386 feature (maybe aarch64 has perf counters too?), supported
> only by vendor (Intel) and I'm not planning working on that.
> I don't like having a feature that is obviously broken and potential GCC users get
> bad experience every time they try to use it.
> 
> Can we at least deprecate the feature for GCC 11? If these is enough interest,
> we can fix it, if not, I would remove it in GCC 13 timeframe.
> 
> Thoughts?
> Martin
> 
> > 
> > Having the tool third-party makes keeping the whole chain working more
> > difficult.
> > 
> > Richard.
> > 
> >> Thanks,
> >>
> >> David
> >>
> >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
> >>
> >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> >>>>> GCC documentation for AutoFDO points to create_gcov tool that converts
> >>> perf.data file into gcov format that can be consumed by gcc with
> >>> -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> >>>>>
> >>>>> I noticed that the source code for create_gcov has been deleted from
> >>> https://github.com/google/autofdo on April 7. I asked about that change
> >>> in that repo and got the following reply:
> >>>>>
> >>>>> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> >>>>>
> >>>>> "Actually we didn't use create_gcov and havn't updated create_gcov for
> >>> years, and we also didn't have enough tests to guarantee it works (It was
> >>> gcc-4.8 when we used and verified create_gcov). If you need it, it is
> >>> welcomed to update create_gcov and add it to the respository."
> >>>>>
> >>>>> Does this mean that AutoFDO is currently dead in gcc?
> >>>>
> >>>> Hello.
> >>>>
> >>>> Yes. I know that even basic test cases have been broken for years in the
> >>> GCC.
> >>>> It's new to me that create_gcov was removed.
> >>>>
> >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> >>>> I known Bin spent some time working on AutoFDO, has he came up to
> >>> something?
> >>>
> >>> The GCC side of auto-FDO is not that hard.  We have most of
> >>> infrastructure in place, but stopping point for me was always difficulty
> >>> to get gcov-tool working.  If some maintainer steps up, I think I can
> >>> fix GCC side.
> >>>
> >>> I am bit unsure how important feature it is - we have FDO that works
> >>> quite well for most users but I know there are some users of the LLVM
> >>> implementation and there is potential to tie this with other hardware
> >>> events to asist i.e. if conversion (where one wants to know how well CPU
> >>> predicts the jump rather than just the jump probability) which I always
> >>> found potentially interesting.
> >>>
> >>> Honza
> >>>>
> >>>> Martin
> >>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Eugene
> >>>>>
> >>>>
> >>>
> 

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka <hubicka@ucw.cz> wrote:

> >
> > It uses create_llvm_prof tool which is in the same git repo. The data
> > parsing part is shared with create_gcov, but the writer is obviously
> > different for the two tools.
>
> OK and what are the main differences between llvmand gcc format?
>
> GCC expects GCOV format, I think while LLVM uses a newly designed binary
format.

David


> Honza
> >
> > David
> >
> >
> > > Honza
> > > >
> > > > David
> > > >
> > > > >
> > > > > Thoughts?
> > > > > Martin
> > > > >
> > > > > >
> > > > > > Having the tool third-party makes keeping the whole chain working
> > > more
> > > > > > difficult.
> > > > > >
> > > > > > Richard.
> > > > > >
> > > > > >> Thanks,
> > > > > >>
> > > > > >> David
> > > > > >>
> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz>
> wrote:
> > > > > >>
> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool that
> > > > > converts
> > > > > >>> perf.data file into gcov format that can be consumed by gcc
> with
> > > > > >>> -fauto-profile (
> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > > > >>>>>
> > > > > >>>>> I noticed that the source code for create_gcov has been
> deleted
> > > from
> > > > > >>> https://github.com/google/autofdo on April 7. I asked about
> that
> > > > > change
> > > > > >>> in that repo and got the following reply:
> > > > > >>>>>
> > > > > >>>>>
> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > > > >>>>>
> > > > > >>>>> "Actually we didn't use create_gcov and havn't updated
> > > create_gcov
> > > > > for
> > > > > >>> years, and we also didn't have enough tests to guarantee it
> works
> > > (It
> > > > > was
> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you need
> it, it
> > > is
> > > > > >>> welcomed to update create_gcov and add it to the respository."
> > > > > >>>>>
> > > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > > > > >>>>
> > > > > >>>> Hello.
> > > > > >>>>
> > > > > >>>> Yes. I know that even basic test cases have been broken for
> years
> > > in
> > > > > the
> > > > > >>> GCC.
> > > > > >>>> It's new to me that create_gcov was removed.
> > > > > >>>>
> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO from GCC.
> > > > > >>>> I known Bin spent some time working on AutoFDO, has he came
> up to
> > > > > >>> something?
> > > > > >>>
> > > > > >>> The GCC side of auto-FDO is not that hard.  We have most of
> > > > > >>> infrastructure in place, but stopping point for me was always
> > > > > difficulty
> > > > > >>> to get gcov-tool working.  If some maintainer steps up, I
> think I
> > > can
> > > > > >>> fix GCC side.
> > > > > >>>
> > > > > >>> I am bit unsure how important feature it is - we have FDO that
> > > works
> > > > > >>> quite well for most users but I know there are some users of
> the
> > > LLVM
> > > > > >>> implementation and there is potential to tie this with other
> > > hardware
> > > > > >>> events to asist i.e. if conversion (where one wants to know how
> > > well
> > > > > CPU
> > > > > >>> predicts the jump rather than just the jump probability) which
> I
> > > always
> > > > > >>> found potentially interesting.
> > > > > >>>
> > > > > >>> Honza
> > > > > >>>>
> > > > > >>>> Martin
> > > > > >>>>
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>>
> > > > > >>>>> Eugene
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > >
> > > > >
> > >
>

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Fri, Apr 23, 2021 at 10:27 AM Xinliang David Li <davidxl@google.com>
wrote:

>
>
> On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka <hubicka@ucw.cz> wrote:
>
>> >
>> > It uses create_llvm_prof tool which is in the same git repo. The data
>> > parsing part is shared with create_gcov, but the writer is obviously
>> > different for the two tools.
>>
>> OK and what are the main differences between llvmand gcc format?
>>
>> GCC expects GCOV format, I think while LLVM uses a newly designed binary
> format.
>
>
Sorry I missed your next reply.  I forgot about the details of GCC' format.

David
>
>
>> Honza
>> >
>> > David
>> >
>> >
>> > > Honza
>> > > >
>> > > > David
>> > > >
>> > > > >
>> > > > > Thoughts?
>> > > > > Martin
>> > > > >
>> > > > > >
>> > > > > > Having the tool third-party makes keeping the whole chain
>> working
>> > > more
>> > > > > > difficult.
>> > > > > >
>> > > > > > Richard.
>> > > > > >
>> > > > > >> Thanks,
>> > > > > >>
>> > > > > >> David
>> > > > > >>
>> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz>
>> wrote:
>> > > > > >>
>> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
>> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool
>> that
>> > > > > converts
>> > > > > >>> perf.data file into gcov format that can be consumed by gcc
>> with
>> > > > > >>> -fauto-profile (
>> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
>> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
>> > > > > >>>>>
>> > > > > >>>>> I noticed that the source code for create_gcov has been
>> deleted
>> > > from
>> > > > > >>> https://github.com/google/autofdo on April 7. I asked about
>> that
>> > > > > change
>> > > > > >>> in that repo and got the following reply:
>> > > > > >>>>>
>> > > > > >>>>>
>> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
>> > > > > >>>>>
>> > > > > >>>>> "Actually we didn't use create_gcov and havn't updated
>> > > create_gcov
>> > > > > for
>> > > > > >>> years, and we also didn't have enough tests to guarantee it
>> works
>> > > (It
>> > > > > was
>> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you need
>> it, it
>> > > is
>> > > > > >>> welcomed to update create_gcov and add it to the respository."
>> > > > > >>>>>
>> > > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
>> > > > > >>>>
>> > > > > >>>> Hello.
>> > > > > >>>>
>> > > > > >>>> Yes. I know that even basic test cases have been broken for
>> years
>> > > in
>> > > > > the
>> > > > > >>> GCC.
>> > > > > >>>> It's new to me that create_gcov was removed.
>> > > > > >>>>
>> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO from
>> GCC.
>> > > > > >>>> I known Bin spent some time working on AutoFDO, has he came
>> up to
>> > > > > >>> something?
>> > > > > >>>
>> > > > > >>> The GCC side of auto-FDO is not that hard.  We have most of
>> > > > > >>> infrastructure in place, but stopping point for me was always
>> > > > > difficulty
>> > > > > >>> to get gcov-tool working.  If some maintainer steps up, I
>> think I
>> > > can
>> > > > > >>> fix GCC side.
>> > > > > >>>
>> > > > > >>> I am bit unsure how important feature it is - we have FDO that
>> > > works
>> > > > > >>> quite well for most users but I know there are some users of
>> the
>> > > LLVM
>> > > > > >>> implementation and there is potential to tie this with other
>> > > hardware
>> > > > > >>> events to asist i.e. if conversion (where one wants to know
>> how
>> > > well
>> > > > > CPU
>> > > > > >>> predicts the jump rather than just the jump probability)
>> which I
>> > > always
>> > > > > >>> found potentially interesting.
>> > > > > >>>
>> > > > > >>> Honza
>> > > > > >>>>
>> > > > > >>>> Martin
>> > > > > >>>>
>> > > > > >>>>>
>> > > > > >>>>> Thanks,
>> > > > > >>>>>
>> > > > > >>>>> Eugene
>> > > > > >>>>>
>> > > > > >>>>
>> > > > > >>>
>> > > > >
>> > > > >
>> > >
>>
>

Re: State of AutoFDO in GCC

[Jan Hubicka]

> On Fri, Apr 23, 2021 at 10:27 AM Xinliang David Li <davidxl@google.com>
> wrote:
> 
> >
> >
> > On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka <hubicka@ucw.cz> wrote:
> >
> >> >
> >> > It uses create_llvm_prof tool which is in the same git repo. The data
> >> > parsing part is shared with create_gcov, but the writer is obviously
> >> > different for the two tools.
> >>
> >> OK and what are the main differences between llvmand gcc format?
> >>
> >> GCC expects GCOV format, I think while LLVM uses a newly designed binary
> > format.
> >
> >
> Sorry I missed your next reply.  I forgot about the details of GCC' format.

Thanks for explanation.  How hard would be to make GCC consume this
format? Is is reasonably stable and documented somewhere?

Does LLVM's auto-FDO support non-Intel CPUs these days?

Honza
> 
> David
> >
> >
> >> Honza
> >> >
> >> > David
> >> >
> >> >
> >> > > Honza
> >> > > >
> >> > > > David
> >> > > >
> >> > > > >
> >> > > > > Thoughts?
> >> > > > > Martin
> >> > > > >
> >> > > > > >
> >> > > > > > Having the tool third-party makes keeping the whole chain
> >> working
> >> > > more
> >> > > > > > difficult.
> >> > > > > >
> >> > > > > > Richard.
> >> > > > > >
> >> > > > > >> Thanks,
> >> > > > > >>
> >> > > > > >> David
> >> > > > > >>
> >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz>
> >> wrote:
> >> > > > > >>
> >> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> >> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool
> >> that
> >> > > > > converts
> >> > > > > >>> perf.data file into gcov format that can be consumed by gcc
> >> with
> >> > > > > >>> -fauto-profile (
> >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> >> > > > > >>>>>
> >> > > > > >>>>> I noticed that the source code for create_gcov has been
> >> deleted
> >> > > from
> >> > > > > >>> https://github.com/google/autofdo on April 7. I asked about
> >> that
> >> > > > > change
> >> > > > > >>> in that repo and got the following reply:
> >> > > > > >>>>>
> >> > > > > >>>>>
> >> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> >> > > > > >>>>>
> >> > > > > >>>>> "Actually we didn't use create_gcov and havn't updated
> >> > > create_gcov
> >> > > > > for
> >> > > > > >>> years, and we also didn't have enough tests to guarantee it
> >> works
> >> > > (It
> >> > > > > was
> >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you need
> >> it, it
> >> > > is
> >> > > > > >>> welcomed to update create_gcov and add it to the respository."
> >> > > > > >>>>>
> >> > > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> >> > > > > >>>>
> >> > > > > >>>> Hello.
> >> > > > > >>>>
> >> > > > > >>>> Yes. I know that even basic test cases have been broken for
> >> years
> >> > > in
> >> > > > > the
> >> > > > > >>> GCC.
> >> > > > > >>>> It's new to me that create_gcov was removed.
> >> > > > > >>>>
> >> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO from
> >> GCC.
> >> > > > > >>>> I known Bin spent some time working on AutoFDO, has he came
> >> up to
> >> > > > > >>> something?
> >> > > > > >>>
> >> > > > > >>> The GCC side of auto-FDO is not that hard.  We have most of
> >> > > > > >>> infrastructure in place, but stopping point for me was always
> >> > > > > difficulty
> >> > > > > >>> to get gcov-tool working.  If some maintainer steps up, I
> >> think I
> >> > > can
> >> > > > > >>> fix GCC side.
> >> > > > > >>>
> >> > > > > >>> I am bit unsure how important feature it is - we have FDO that
> >> > > works
> >> > > > > >>> quite well for most users but I know there are some users of
> >> the
> >> > > LLVM
> >> > > > > >>> implementation and there is potential to tie this with other
> >> > > hardware
> >> > > > > >>> events to asist i.e. if conversion (where one wants to know
> >> how
> >> > > well
> >> > > > > CPU
> >> > > > > >>> predicts the jump rather than just the jump probability)
> >> which I
> >> > > always
> >> > > > > >>> found potentially interesting.
> >> > > > > >>>
> >> > > > > >>> Honza
> >> > > > > >>>>
> >> > > > > >>>> Martin
> >> > > > > >>>>
> >> > > > > >>>>>
> >> > > > > >>>>> Thanks,
> >> > > > > >>>>>
> >> > > > > >>>>> Eugene
> >> > > > > >>>>>
> >> > > > > >>>>
> >> > > > > >>>
> >> > > > >
> >> > > > >
> >> > >
> >>
> >

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Fri, Apr 23, 2021 at 12:28 PM Jan Hubicka <hubicka@ucw.cz> wrote:

> > On Fri, Apr 23, 2021 at 10:27 AM Xinliang David Li <davidxl@google.com>
> > wrote:
> >
> > >
> > >
> > > On Fri, Apr 23, 2021 at 10:16 AM Jan Hubicka <hubicka@ucw.cz> wrote:
> > >
> > >> >
> > >> > It uses create_llvm_prof tool which is in the same git repo. The
> data
> > >> > parsing part is shared with create_gcov, but the writer is obviously
> > >> > different for the two tools.
> > >>
> > >> OK and what are the main differences between llvmand gcc format?
> > >>
> > >> GCC expects GCOV format, I think while LLVM uses a newly designed
> binary
> > > format.
> > >
> > >
> > Sorry I missed your next reply.  I forgot about the details of GCC'
> format.
>
> Thanks for explanation.  How hard would be to make GCC consume this
> format? Is is reasonably stable and documented somewhere?
>

The text format is documented here:
https://clang.llvm.org/docs/UsersManual.html
The binary format is not documented. The binary format is not guaranteed to
be backward compatible, so sharing the same format may not be the best way
as changes for clang may break GCC.

Since linux perf format does not change, the tool should be relatively
stable with low maintenance cost. Changes are needed only when some new
AutoFDO features are added to the compiler side.

Does LLVM's auto-FDO support non-Intel CPUs these days?
>

It supports LBR like events, so it is CPU vendor dependent. For ARM, using
ETM can achieve the goal, but I don't have detailed knowledge of it.

David

>
> Honza
> >
> > David
> > >
> > >
> > >> Honza
> > >> >
> > >> > David
> > >> >
> > >> >
> > >> > > Honza
> > >> > > >
> > >> > > > David
> > >> > > >
> > >> > > > >
> > >> > > > > Thoughts?
> > >> > > > > Martin
> > >> > > > >
> > >> > > > > >
> > >> > > > > > Having the tool third-party makes keeping the whole chain
> > >> working
> > >> > > more
> > >> > > > > > difficult.
> > >> > > > > >
> > >> > > > > > Richard.
> > >> > > > > >
> > >> > > > > >> Thanks,
> > >> > > > > >>
> > >> > > > > >> David
> > >> > > > > >>
> > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <
> hubicka@ucw.cz>
> > >> wrote:
> > >> > > > > >>
> > >> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > >> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool
> > >> that
> > >> > > > > converts
> > >> > > > > >>> perf.data file into gcov format that can be consumed by
> gcc
> > >> with
> > >> > > > > >>> -fauto-profile (
> > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > >> > > > > >>>>>
> > >> > > > > >>>>> I noticed that the source code for create_gcov has been
> > >> deleted
> > >> > > from
> > >> > > > > >>> https://github.com/google/autofdo on April 7. I asked
> about
> > >> that
> > >> > > > > change
> > >> > > > > >>> in that repo and got the following reply:
> > >> > > > > >>>>>
> > >> > > > > >>>>>
> > >> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > >> > > > > >>>>>
> > >> > > > > >>>>> "Actually we didn't use create_gcov and havn't updated
> > >> > > create_gcov
> > >> > > > > for
> > >> > > > > >>> years, and we also didn't have enough tests to guarantee
> it
> > >> works
> > >> > > (It
> > >> > > > > was
> > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you
> need
> > >> it, it
> > >> > > is
> > >> > > > > >>> welcomed to update create_gcov and add it to the
> respository."
> > >> > > > > >>>>>
> > >> > > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > >> > > > > >>>>
> > >> > > > > >>>> Hello.
> > >> > > > > >>>>
> > >> > > > > >>>> Yes. I know that even basic test cases have been broken
> for
> > >> years
> > >> > > in
> > >> > > > > the
> > >> > > > > >>> GCC.
> > >> > > > > >>>> It's new to me that create_gcov was removed.
> > >> > > > > >>>>
> > >> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO from
> > >> GCC.
> > >> > > > > >>>> I known Bin spent some time working on AutoFDO, has he
> came
> > >> up to
> > >> > > > > >>> something?
> > >> > > > > >>>
> > >> > > > > >>> The GCC side of auto-FDO is not that hard.  We have most
> of
> > >> > > > > >>> infrastructure in place, but stopping point for me was
> always
> > >> > > > > difficulty
> > >> > > > > >>> to get gcov-tool working.  If some maintainer steps up, I
> > >> think I
> > >> > > can
> > >> > > > > >>> fix GCC side.
> > >> > > > > >>>
> > >> > > > > >>> I am bit unsure how important feature it is - we have FDO
> that
> > >> > > works
> > >> > > > > >>> quite well for most users but I know there are some users
> of
> > >> the
> > >> > > LLVM
> > >> > > > > >>> implementation and there is potential to tie this with
> other
> > >> > > hardware
> > >> > > > > >>> events to asist i.e. if conversion (where one wants to
> know
> > >> how
> > >> > > well
> > >> > > > > CPU
> > >> > > > > >>> predicts the jump rather than just the jump probability)
> > >> which I
> > >> > > always
> > >> > > > > >>> found potentially interesting.
> > >> > > > > >>>
> > >> > > > > >>> Honza
> > >> > > > > >>>>
> > >> > > > > >>>> Martin
> > >> > > > > >>>>
> > >> > > > > >>>>>
> > >> > > > > >>>>> Thanks,
> > >> > > > > >>>>>
> > >> > > > > >>>>> Eugene
> > >> > > > > >>>>>
> > >> > > > > >>>>
> > >> > > > > >>>
> > >> > > > >
> > >> > > > >
> > >> > >
> > >>
> > >
>

Re: State of AutoFDO in GCC

[Jan Hubicka]

David,
> 
> The text format is documented here:
> https://clang.llvm.org/docs/UsersManual.html
> The binary format is not documented. The binary format is not guaranteed to
> be backward compatible, so sharing the same format may not be the best way
> as changes for clang may break GCC.
> 
> Since linux perf format does not change, the tool should be relatively
> stable with low maintenance cost. Changes are needed only when some new
> AutoFDO features are added to the compiler side.

I was under impression that it is indeed problem with the tool requiring
old format of linux perf. At least with opensuse distro the shipped tool
fails for me:
jan@skylake:~> create_llvm_prof --binary=./code --out=code.prof
E0425 21:01:55.038128 17977 perf_reader.cc:996] Unsupported event type
79
F0425 21:01:55.038295 17977 perf_parser.cc:240] Check failed:
reader_.ReadPerfSampleInfo(*parsed_event.raw_event, &sample_info) 
*** Check failure stack trace: ***
    @     0x55e6deb6058e  (unknown)
    @     0x55e6deb94a49  (unknown)
    ..
    Aborted (core dumped)

I collect data as intstructed here:
https://clang.llvm.org/docs/UsersManual.html

It is from package autofdo-0.18-4.4.x86_64 and perf 5.11.15.

Is there a way to get this working w/o using older perf?
Honza
> 
> Does LLVM's auto-FDO support non-Intel CPUs these days?
> >
> 
> It supports LBR like events, so it is CPU vendor dependent. For ARM, using
> ETM can achieve the goal, but I don't have detailed knowledge of it.
> 
> David
> 
> >
> > Honza
> > >
> > > David
> > > >
> > > >
> > > >> Honza
> > > >> >
> > > >> > David
> > > >> >
> > > >> >
> > > >> > > Honza
> > > >> > > >
> > > >> > > > David
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Thoughts?
> > > >> > > > > Martin
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > Having the tool third-party makes keeping the whole chain
> > > >> working
> > > >> > > more
> > > >> > > > > > difficult.
> > > >> > > > > >
> > > >> > > > > > Richard.
> > > >> > > > > >
> > > >> > > > > >> Thanks,
> > > >> > > > > >>
> > > >> > > > > >> David
> > > >> > > > > >>
> > > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <
> > hubicka@ucw.cz>
> > > >> wrote:
> > > >> > > > > >>
> > > >> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > >> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov tool
> > > >> that
> > > >> > > > > converts
> > > >> > > > > >>> perf.data file into gcov format that can be consumed by
> > gcc
> > > >> with
> > > >> > > > > >>> -fauto-profile (
> > > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> I noticed that the source code for create_gcov has been
> > > >> deleted
> > > >> > > from
> > > >> > > > > >>> https://github.com/google/autofdo on April 7. I asked
> > about
> > > >> that
> > > >> > > > > change
> > > >> > > > > >>> in that repo and got the following reply:
> > > >> > > > > >>>>>
> > > >> > > > > >>>>>
> > > >> > > https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> "Actually we didn't use create_gcov and havn't updated
> > > >> > > create_gcov
> > > >> > > > > for
> > > >> > > > > >>> years, and we also didn't have enough tests to guarantee
> > it
> > > >> works
> > > >> > > (It
> > > >> > > > > was
> > > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you
> > need
> > > >> it, it
> > > >> > > is
> > > >> > > > > >>> welcomed to update create_gcov and add it to the
> > respository."
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> Does this mean that AutoFDO is currently dead in gcc?
> > > >> > > > > >>>>
> > > >> > > > > >>>> Hello.
> > > >> > > > > >>>>
> > > >> > > > > >>>> Yes. I know that even basic test cases have been broken
> > for
> > > >> years
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > >>> GCC.
> > > >> > > > > >>>> It's new to me that create_gcov was removed.
> > > >> > > > > >>>>
> > > >> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO from
> > > >> GCC.
> > > >> > > > > >>>> I known Bin spent some time working on AutoFDO, has he
> > came
> > > >> up to
> > > >> > > > > >>> something?
> > > >> > > > > >>>
> > > >> > > > > >>> The GCC side of auto-FDO is not that hard.  We have most
> > of
> > > >> > > > > >>> infrastructure in place, but stopping point for me was
> > always
> > > >> > > > > difficulty
> > > >> > > > > >>> to get gcov-tool working.  If some maintainer steps up, I
> > > >> think I
> > > >> > > can
> > > >> > > > > >>> fix GCC side.
> > > >> > > > > >>>
> > > >> > > > > >>> I am bit unsure how important feature it is - we have FDO
> > that
> > > >> > > works
> > > >> > > > > >>> quite well for most users but I know there are some users
> > of
> > > >> the
> > > >> > > LLVM
> > > >> > > > > >>> implementation and there is potential to tie this with
> > other
> > > >> > > hardware
> > > >> > > > > >>> events to asist i.e. if conversion (where one wants to
> > know
> > > >> how
> > > >> > > well
> > > >> > > > > CPU
> > > >> > > > > >>> predicts the jump rather than just the jump probability)
> > > >> which I
> > > >> > > always
> > > >> > > > > >>> found potentially interesting.
> > > >> > > > > >>>
> > > >> > > > > >>> Honza
> > > >> > > > > >>>>
> > > >> > > > > >>>> Martin
> > > >> > > > > >>>>
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> Thanks,
> > > >> > > > > >>>>>
> > > >> > > > > >>>>> Eugene
> > > >> > > > > >>>>>
> > > >> > > > > >>>>
> > > >> > > > > >>>
> > > >> > > > >
> > > >> > > > >
> > > >> > >
> > > >>
> > > >
> >

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Sun, Apr 25, 2021 at 12:07 PM Jan Hubicka <hubicka@ucw.cz> wrote:

> David,
> >
> > The text format is documented here:
> > https://clang.llvm.org/docs/UsersManual.html
> > The binary format is not documented. The binary format is not guaranteed
> to
> > be backward compatible, so sharing the same format may not be the best
> way
> > as changes for clang may break GCC.
> >
> > Since linux perf format does not change, the tool should be relatively
> > stable with low maintenance cost. Changes are needed only when some new
> > AutoFDO features are added to the compiler side.
>
> I was under impression that it is indeed problem with the tool requiring
> old format of linux perf. At least with opensuse distro the shipped tool
> fails for me:
> jan@skylake:~> create_llvm_prof --binary=./code --out=code.prof
> E0425 21:01:55.038128 17977 perf_reader.cc:996] Unsupported event type
> 79
> F0425 21:01:55.038295 17977 perf_parser.cc:240] Check failed:
> reader_.ReadPerfSampleInfo(*parsed_event.raw_event, &sample_info)
> *** Check failure stack trace: ***
>     @     0x55e6deb6058e  (unknown)
>     @     0x55e6deb94a49  (unknown)
>     ..
>     Aborted (core dumped)
>
> I collect data as intstructed here:
> https://clang.llvm.org/docs/UsersManual.html
>
> It is from package autofdo-0.18-4.4.x86_64 and perf 5.11.15.
>
> Is there a way to get this working w/o using older perf?
> Honza
> >


Interesting. That means we will also see the same error when using the
latest perf.

Wei, are you aware of the issue?

David





>
> > Does LLVM's auto-FDO support non-Intel CPUs these days?
> > >
> >
> > It supports LBR like events, so it is CPU vendor dependent. For ARM,
> using
> > ETM can achieve the goal, but I don't have detailed knowledge of it.
> >
> > David
> >
> > >
> > > Honza
> > > >
> > > > David
> > > > >
> > > > >
> > > > >> Honza
> > > > >> >
> > > > >> > David
> > > > >> >
> > > > >> >
> > > > >> > > Honza
> > > > >> > > >
> > > > >> > > > David
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > Thoughts?
> > > > >> > > > > Martin
> > > > >> > > > >
> > > > >> > > > > >
> > > > >> > > > > > Having the tool third-party makes keeping the whole
> chain
> > > > >> working
> > > > >> > > more
> > > > >> > > > > > difficult.
> > > > >> > > > > >
> > > > >> > > > > > Richard.
> > > > >> > > > > >
> > > > >> > > > > >> Thanks,
> > > > >> > > > > >>
> > > > >> > > > > >> David
> > > > >> > > > > >>
> > > > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <
> > > hubicka@ucw.cz>
> > > > >> wrote:
> > > > >> > > > > >>
> > > > >> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > >> > > > > >>>>> GCC documentation for AutoFDO points to create_gcov
> tool
> > > > >> that
> > > > >> > > > > converts
> > > > >> > > > > >>> perf.data file into gcov format that can be consumed
> by
> > > gcc
> > > > >> with
> > > > >> > > > > >>> -fauto-profile (
> > > > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html,
> > > > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>> I noticed that the source code for create_gcov has
> been
> > > > >> deleted
> > > > >> > > from
> > > > >> > > > > >>> https://github.com/google/autofdo on April 7. I asked
> > > about
> > > > >> that
> > > > >> > > > > change
> > > > >> > > > > >>> in that repo and got the following reply:
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>>
> > > > >> > >
> https://github.com/google/autofdo/pull/107#issuecomment-819108738
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>> "Actually we didn't use create_gcov and havn't
> updated
> > > > >> > > create_gcov
> > > > >> > > > > for
> > > > >> > > > > >>> years, and we also didn't have enough tests to
> guarantee
> > > it
> > > > >> works
> > > > >> > > (It
> > > > >> > > > > was
> > > > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If you
> > > need
> > > > >> it, it
> > > > >> > > is
> > > > >> > > > > >>> welcomed to update create_gcov and add it to the
> > > respository."
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>> Does this mean that AutoFDO is currently dead in
> gcc?
> > > > >> > > > > >>>>
> > > > >> > > > > >>>> Hello.
> > > > >> > > > > >>>>
> > > > >> > > > > >>>> Yes. I know that even basic test cases have been
> broken
> > > for
> > > > >> years
> > > > >> > > in
> > > > >> > > > > the
> > > > >> > > > > >>> GCC.
> > > > >> > > > > >>>> It's new to me that create_gcov was removed.
> > > > >> > > > > >>>>
> > > > >> > > > > >>>> I tend to send patch to GCC that will remove AutoFDO
> from
> > > > >> GCC.
> > > > >> > > > > >>>> I known Bin spent some time working on AutoFDO, has
> he
> > > came
> > > > >> up to
> > > > >> > > > > >>> something?
> > > > >> > > > > >>>
> > > > >> > > > > >>> The GCC side of auto-FDO is not that hard.  We have
> most
> > > of
> > > > >> > > > > >>> infrastructure in place, but stopping point for me was
> > > always
> > > > >> > > > > difficulty
> > > > >> > > > > >>> to get gcov-tool working.  If some maintainer steps
> up, I
> > > > >> think I
> > > > >> > > can
> > > > >> > > > > >>> fix GCC side.
> > > > >> > > > > >>>
> > > > >> > > > > >>> I am bit unsure how important feature it is - we have
> FDO
> > > that
> > > > >> > > works
> > > > >> > > > > >>> quite well for most users but I know there are some
> users
> > > of
> > > > >> the
> > > > >> > > LLVM
> > > > >> > > > > >>> implementation and there is potential to tie this with
> > > other
> > > > >> > > hardware
> > > > >> > > > > >>> events to asist i.e. if conversion (where one wants to
> > > know
> > > > >> how
> > > > >> > > well
> > > > >> > > > > CPU
> > > > >> > > > > >>> predicts the jump rather than just the jump
> probability)
> > > > >> which I
> > > > >> > > always
> > > > >> > > > > >>> found potentially interesting.
> > > > >> > > > > >>>
> > > > >> > > > > >>> Honza
> > > > >> > > > > >>>>
> > > > >> > > > > >>>> Martin
> > > > >> > > > > >>>>
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>> Thanks,
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>> Eugene
> > > > >> > > > > >>>>>
> > > > >> > > > > >>>>
> > > > >> > > > > >>>
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >>
> > > > >
> > >
>

Re: State of AutoFDO in GCC

[Wei Mi]

On Sun, Apr 25, 2021 at 4:18 PM Xinliang David Li <davidxl@google.com>
wrote:

>
>
> On Sun, Apr 25, 2021 at 12:07 PM Jan Hubicka <hubicka@ucw.cz> wrote:
>
>> David,
>> >
>> > The text format is documented here:
>> > https://clang.llvm.org/docs/UsersManual.html
>> > The binary format is not documented. The binary format is not
>> guaranteed to
>> > be backward compatible, so sharing the same format may not be the best
>> way
>> > as changes for clang may break GCC.
>> >
>> > Since linux perf format does not change, the tool should be relatively
>> > stable with low maintenance cost. Changes are needed only when some new
>> > AutoFDO features are added to the compiler side.
>>
>> I was under impression that it is indeed problem with the tool requiring
>> old format of linux perf. At least with opensuse distro the shipped tool
>> fails for me:
>> jan@skylake:~> create_llvm_prof --binary=./code --out=code.prof
>> E0425 21:01:55.038128 17977 perf_reader.cc:996] Unsupported event type
>> 79
>> F0425 21:01:55.038295 17977 perf_parser.cc:240] Check failed:
>> reader_.ReadPerfSampleInfo(*parsed_event.raw_event, &sample_info)
>> *** Check failure stack trace: ***
>>     @     0x55e6deb6058e  (unknown)
>>     @     0x55e6deb94a49  (unknown)
>>     ..
>>     Aborted (core dumped)
>>
>> I collect data as intstructed here:
>> https://clang.llvm.org/docs/UsersManual.html
>>
>> It is from package autofdo-0.18-4.4.x86_64 and perf 5.11.15.
>>
>> Is there a way to get this working w/o using older perf?
>> Honza
>> >
>
>
> Interesting. That means we will also see the same error when using the
> latest perf.
>
> Wei, are you aware of the issue?
>
> David
>
>
>
My local perf is 4.13 so I cannot look at the problem. I remember we has
fixed such problem in quipper before. Could you try the latest version of
autofdo (using the guideline here: https://github.com/google/autofdo#readme)?
autofdo-0.18 is an old version and it is using an old quipper.

Thanks,
Wei.



>
>
>
>>
>> > Does LLVM's auto-FDO support non-Intel CPUs these days?
>> > >
>> >
>> > It supports LBR like events, so it is CPU vendor dependent. For ARM,
>> using
>> > ETM can achieve the goal, but I don't have detailed knowledge of it.
>> >
>> > David
>> >
>> > >
>> > > Honza
>> > > >
>> > > > David
>> > > > >
>> > > > >
>> > > > >> Honza
>> > > > >> >
>> > > > >> > David
>> > > > >> >
>> > > > >> >
>> > > > >> > > Honza
>> > > > >> > > >
>> > > > >> > > > David
>> > > > >> > > >
>> > > > >> > > > >
>> > > > >> > > > > Thoughts?
>> > > > >> > > > > Martin
>> > > > >> > > > >
>> > > > >> > > > > >
>> > > > >> > > > > > Having the tool third-party makes keeping the whole
>> chain
>> > > > >> working
>> > > > >> > > more
>> > > > >> > > > > > difficult.
>> > > > >> > > > > >
>> > > > >> > > > > > Richard.
>> > > > >> > > > > >
>> > > > >> > > > > >> Thanks,
>> > > > >> > > > > >>
>> > > > >> > > > > >> David
>> > > > >> > > > > >>
>> > > > >> > > > > >> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <
>> > > hubicka@ucw.cz>
>> > > > >> wrote:
>> > > > >> > > > > >>
>> > > > >> > > > > >>>> On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
>> > > > >> > > > > >>>>> GCC documentation for AutoFDO points to
>> create_gcov tool
>> > > > >> that
>> > > > >> > > > > converts
>> > > > >> > > > > >>> perf.data file into gcov format that can be consumed
>> by
>> > > gcc
>> > > > >> with
>> > > > >> > > > > >>> -fauto-profile (
>> > > > >> > > > > https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html
>> ,
>> > > > >> > > > > >>> https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>> I noticed that the source code for create_gcov has
>> been
>> > > > >> deleted
>> > > > >> > > from
>> > > > >> > > > > >>> https://github.com/google/autofdo on April 7. I
>> asked
>> > > about
>> > > > >> that
>> > > > >> > > > > change
>> > > > >> > > > > >>> in that repo and got the following reply:
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>>
>> > > > >> > >
>> https://github.com/google/autofdo/pull/107#issuecomment-819108738
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>> "Actually we didn't use create_gcov and havn't
>> updated
>> > > > >> > > create_gcov
>> > > > >> > > > > for
>> > > > >> > > > > >>> years, and we also didn't have enough tests to
>> guarantee
>> > > it
>> > > > >> works
>> > > > >> > > (It
>> > > > >> > > > > was
>> > > > >> > > > > >>> gcc-4.8 when we used and verified create_gcov). If
>> you
>> > > need
>> > > > >> it, it
>> > > > >> > > is
>> > > > >> > > > > >>> welcomed to update create_gcov and add it to the
>> > > respository."
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>> Does this mean that AutoFDO is currently dead in
>> gcc?
>> > > > >> > > > > >>>>
>> > > > >> > > > > >>>> Hello.
>> > > > >> > > > > >>>>
>> > > > >> > > > > >>>> Yes. I know that even basic test cases have been
>> broken
>> > > for
>> > > > >> years
>> > > > >> > > in
>> > > > >> > > > > the
>> > > > >> > > > > >>> GCC.
>> > > > >> > > > > >>>> It's new to me that create_gcov was removed.
>> > > > >> > > > > >>>>
>> > > > >> > > > > >>>> I tend to send patch to GCC that will remove
>> AutoFDO from
>> > > > >> GCC.
>> > > > >> > > > > >>>> I known Bin spent some time working on AutoFDO, has
>> he
>> > > came
>> > > > >> up to
>> > > > >> > > > > >>> something?
>> > > > >> > > > > >>>
>> > > > >> > > > > >>> The GCC side of auto-FDO is not that hard.  We have
>> most
>> > > of
>> > > > >> > > > > >>> infrastructure in place, but stopping point for me
>> was
>> > > always
>> > > > >> > > > > difficulty
>> > > > >> > > > > >>> to get gcov-tool working.  If some maintainer steps
>> up, I
>> > > > >> think I
>> > > > >> > > can
>> > > > >> > > > > >>> fix GCC side.
>> > > > >> > > > > >>>
>> > > > >> > > > > >>> I am bit unsure how important feature it is - we
>> have FDO
>> > > that
>> > > > >> > > works
>> > > > >> > > > > >>> quite well for most users but I know there are some
>> users
>> > > of
>> > > > >> the
>> > > > >> > > LLVM
>> > > > >> > > > > >>> implementation and there is potential to tie this
>> with
>> > > other
>> > > > >> > > hardware
>> > > > >> > > > > >>> events to asist i.e. if conversion (where one wants
>> to
>> > > know
>> > > > >> how
>> > > > >> > > well
>> > > > >> > > > > CPU
>> > > > >> > > > > >>> predicts the jump rather than just the jump
>> probability)
>> > > > >> which I
>> > > > >> > > always
>> > > > >> > > > > >>> found potentially interesting.
>> > > > >> > > > > >>>
>> > > > >> > > > > >>> Honza
>> > > > >> > > > > >>>>
>> > > > >> > > > > >>>> Martin
>> > > > >> > > > > >>>>
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>> Thanks,
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>> Eugene
>> > > > >> > > > > >>>>>
>> > > > >> > > > > >>>>
>> > > > >> > > > > >>>
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > >
>> > > > >>
>> > > > >
>> > >
>>
>

Re: State of AutoFDO in GCC

[Andi Kleen]

Jan Hubicka <hubicka@ucw.cz> writes:
>
> Is there a way to get this working w/o using older perf?

It's usually rather simple to fix up autofdo for new perf.
I did it before here

https://github.com/andikleen/autofdo/commits/perf4-3

I think it would work always if it just ignored unknown records
(which is quite possible). perf adds new records now and then,
but they can be usually ignored by old tools.
Only the assert happy coding style in autofdo prevents it.

BTW longer term my feeling is autofdo should be replaced with something
like https://lists.llvm.org/pipermail/llvm-dev/2020-August/144101.html
I think that would fix most of the weirdnesses in the current autofdo
implementation.

-Andi

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Mon, Apr 26, 2021 at 8:12 AM Andi Kleen <ak@linux.intel.com> wrote:

> Jan Hubicka <hubicka@ucw.cz> writes:
> >
> > Is there a way to get this working w/o using older perf?
>
> It's usually rather simple to fix up autofdo for new perf.
> I did it before here
>
> https://github.com/andikleen/autofdo/commits/perf4-3
>
> I think it would work always if it just ignored unknown records
> (which is quite possible). perf adds new records now and then,
> but they can be usually ignored by old tools.
> Only the assert happy coding style in autofdo prevents it.
>
> BTW longer term my feeling is autofdo should be replaced with something
> like https://lists.llvm.org/pipermail/llvm-dev/2020-August/144101.html
> I think that would fix most of the weirdnesses in the current autofdo
> implementation.
>

There are multiple directional changes in this new tool:
1) it uses perf-script trace output (in text) as input profile data;
2) it uses pseudo probe like instrumentation FDO to do profile matching;
3) it supports full profile context sensitivity (not just from inline
stacks).

#1 is most relevant to this discussion -- using text format reduces the
pain introduced by perf binary format change.

David



>
> -Andi
>

Re: State of AutoFDO in GCC

[Andi Kleen]

>    There are multiple directional changes in this new tool:
>    1) it uses perf-script trace output (in text) as input profile data; 

I suspect this will break regularly too

(I personally did numerous changes to perf script output, and also
wrote a lot of parsing scripts)

The perf script output has some bad problems, e.g. for file names or processes
with spaces and some other issues. To make it handleable would need some
redesign to actually generate in a machine friendly format.

A perf.data parser should be fine, just don't fill it up with asserts
and "be liberal what you accept" and ignore unknown records.

-Andi

Re: State of AutoFDO in GCC

[Xinliang David Li]

On Mon, Apr 26, 2021 at 11:00 AM Andi Kleen <ak@linux.intel.com> wrote:

> >    There are multiple directional changes in this new tool:
> >    1) it uses perf-script trace output (in text) as input profile data;
>
> I suspect this will break regularly too
>
> (I personally did numerous changes to perf script output, and also
> wrote a lot of parsing scripts)
>
> The perf script output has some bad problems, e.g. for file names or
> processes
> with spaces and some other issues. To make it handleable would need some
> redesign to actually generate in a machine friendly format.
>

Andi, thanks for the input.


+authors of the llvm-profgen tool for their experience with using perf
script output.

David




>
> A perf.data parser should be fine, just don't fill it up with asserts
> and "be liberal what you accept" and ignore unknown records.
>
> -Andi
>

Re: State of AutoFDO in GCC

[Hongtao Yu]

Andi, thanks for pointing out the perf script issues. Can you please elaborate a bit on the exact issue you have seen? We’ve been using specific output of perf script such as mmap, LBR and callstack events filtered by process id. It works fine so far but may certainly hit issues in the future with extended uses.

Thanks,
Hongtao

From: Xinliang David Li <davidxl@google.com>
Date: Monday, April 26, 2021 at 11:05 AM
To: Andi Kleen <ak@linux.intel.com>
Cc: Jan Hubicka <hubicka@ucw.cz>, gcc@gcc.gnu.org <gcc@gcc.gnu.org>, Wei Mi <wmi@google.com>, Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>, Wenlei He <wenlei@fb.com>, Hongtao Yu <hoy@fb.com>
Subject: Re: State of AutoFDO in GCC


On Mon, Apr 26, 2021 at 11:00 AM Andi Kleen <ak@linux.intel.com<mailto:ak@linux.intel.com>> wrote:
>    There are multiple directional changes in this new tool:
>    1) it uses perf-script trace output (in text) as input profile data;

I suspect this will break regularly too

(I personally did numerous changes to perf script output, and also
wrote a lot of parsing scripts)

The perf script output has some bad problems, e.g. for file names or processes
with spaces and some other issues. To make it handleable would need some
redesign to actually generate in a machine friendly format.

Andi, thanks for the input.


+authors of the llvm-profgen tool for their experience with using perf script output.

David




A perf.data parser should be fine, just don't fill it up with asserts
and "be liberal what you accept" and ignore unknown records.

-Andi

Re: State of AutoFDO in GCC

[Andi Kleen]

On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
>    Andi, thanks for pointing out the perf script issues. Can you please
>    elaborate a bit on the exact issue you have seen? We’ve been using
>    specific output of perf script such as mmap, LBR and callstack events
>    filtered by process id. It works fine so far but may certainly hit issues
>    in the future with extended uses.

The main problem is that you cannot just split the output in fields because
some fields (like file names or process names) can have spaces without
any kind of delimeter.

It's possible to work around this, but tends to be fragile and ugly.

Also there were occasional changes in the output the past.

-Andi

Re: State of AutoFDO in GCC

[Andi Kleen]

On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
>    Andi, thanks for pointing out the perf script issues. Can you please
>    elaborate a bit on the exact issue you have seen? We’ve been using
>    specific output of perf script such as mmap, LBR and callstack events
>    filtered by process id. It works fine so far but may certainly hit issues
>    in the future with extended uses.

Okay I took a look at the latest autofdo now. It seems to be basically
a LLVM project now that depends on LLVM to even build with all kinds
of dependency hell on some old LLVM version and other packages.

I guess gcc will really need a replacement that doesn't pull in
all of LLVM if it wants to continue supporting autofdo.

I'm myself unable to build now.

I'm using the old version I had a git fork of and that 
was before all of this. I added a patch to make it work
with the latest perf by ignoring increased perf_attr
and unknown perf events.

Honza please use

https://github.com/andikleen/autofdo -b perf-future

for testing.

-Andi

Re: Re: State of AutoFDO in GCC

[172060045]

Hi all, 

I`m using GCC 9.3 AutoFDO and the old version create_gcov on arm64 
and it works well. Actually it support not only LBR like mode but 
also inst_retired even cycles event, which`s the early implementation 
of AutoFDO[1]. There is no difference in output format of create_gcov 
between LBR mode and inst_retired. inst_retired is less accurate than 
LBR but still works.

I hope AutoFDO could work better on GCC and am willing to contribute 
to it, does anybody have suggestions for me?

[1] Ramasamy, Vinodha, et al. "Feedback-directed optimizations in gcc 
with estimated edge profiles from hardware event sampling." (2008).

> On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> >    Andi, thanks for pointing out the perf script issues. Can you please
> >    elaborate a bit on the exact issue you have seen? We’ve been using
> >    specific output of perf script such as mmap, LBR and callstack events
> >    filtered by process id. It works fine so far but may certainly hit issues
> >    in the future with extended uses.
> 
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
> 
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
> 
> I'm myself unable to build now.
> 
> I'm using the old version I had a git fork of and that 
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
> 
> Honza please use
> 
> https://github.com/andikleen/autofdo -b perf-future
> 
> for testing.
> 
> -Andi

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Is the format produced by create_gcov and expected by GCC under -fauto-rpofile documented somewhere? How is it different from .gcda used in FDO, e.g., as described here: http://src.gnu-darwin.org/src/contrib/gcc/gcov-io.h.html?
My input data is different from perf.data and I'd like to write a tool that produces the format needed for AutoFDO.

I would prefer that AutoFDO is not removed from GCC and it would be helpful if create_gcov were restored in google/autofdo. I checked out a revision before the recent merge and tried it on a simple example and it seems to work.
I'm also interested in contributing improvements for AutoFDO so will try to investigate https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379

Thanks,

Eugene

From: Xinliang David Li <davidxl@google.com>
Sent: Friday, April 23, 2021 9:36 AM
To: Richard Biener <richard.guenther@gmail.com>
Cc: Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC



On Fri, Apr 23, 2021 at 12:00 AM Richard Biener <richard.guenther@gmail.com<mailto:richard.guenther@gmail.com>> wrote:
On Fri, Apr 23, 2021 at 7:28 AM Xinliang David Li via Gcc
<gcc@gcc.gnu.org<mailto:gcc@gcc.gnu.org>> wrote:
>
> Hi, the create_gcov tool was probably removed with the assumption that it
> was only used with Google GCC branch, but it is actually used with GCC
> trunk as well.
>
> Given that, the tool will be restored in the github repo. It seems to build
> and work fine with the regression test.
>
> The tool may ust work as it is right now, but there is no guarantee it
> won't break in the future unless someone in the GCC community tries to
> maintain it.

I think if we want to keep the feature it makes sense to provide create_gcov
functionality either directly from perf (input data producer) or from gcc
(data consumer).  Of course I have no idea about its complexity, license
or implementation language ...

Right. What it takes is a perf data (can be text format) parser to produce the format GCC needs, but someone in the community needs to take the lead. It should not involve too much effort.

David

Having the tool third-party makes keeping the whole chain working more
difficult.

Richard.

> Thanks,
>
> David
>
> On Thu, Apr 22, 2021 at 3:29 PM Jan Hubicka <hubicka@ucw.cz<mailto:hubicka@ucw.cz>> wrote:
>
> > > On 4/22/21 9:58 PM, Eugene Rozenfeld via Gcc wrote:
> > > > GCC documentation for AutoFDO points to create_gcov tool that converts
> > perf.data file into gcov format that can be consumed by gcc with
> > -fauto-profile (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fonlinedocs%2Fgcc%2FOptimize-Options.html&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772845557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FAYvyG0%2BxQ%2BbgLgEHUckUngTXDLoNJ4AASMeMVAhHWE%3D&reserved=0>,
> > https://gcc.gnu.org/wiki/AutoFDO/Tutorial<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fwiki%2FAutoFDO%2FTutorial&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772855555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IqMDaxmMTM64eavwksTEXZKwlVhR5UZZiL4tfcyu3io%3D&reserved=0>).
> > > >
> > > > I noticed that the source code for create_gcov has been deleted from
> > https://github.com/google/autofdo<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772865552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6m86ZH8vmrXFIJOGO8WPYBxap1R4uULFZi5mE04dFbc%3D&reserved=0> on April 7. I asked about that change
> > in that repo and got the following reply:
> > > >
> > > > https://github.com/google/autofdo/pull/107#issuecomment-819108738<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fpull%2F107%23issuecomment-819108738&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C08a2bfb9135c41955a7708d90675e9ba%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637547925772875543%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9k945Ma9HRo4RzafSMzr%2FiKx8ochLr77GyIO7pzBVJ8%3D&reserved=0>
> > > >
> > > > "Actually we didn't use create_gcov and havn't updated create_gcov for
> > years, and we also didn't have enough tests to guarantee it works (It was
> > gcc-4.8 when we used and verified create_gcov). If you need it, it is
> > welcomed to update create_gcov and add it to the respository."
> > > >
> > > > Does this mean that AutoFDO is currently dead in gcc?
> > >
> > > Hello.
> > >
> > > Yes. I know that even basic test cases have been broken for years in the
> > GCC.
> > > It's new to me that create_gcov was removed.
> > >
> > > I tend to send patch to GCC that will remove AutoFDO from GCC.
> > > I known Bin spent some time working on AutoFDO, has he came up to
> > something?
> >
> > The GCC side of auto-FDO is not that hard.  We have most of
> > infrastructure in place, but stopping point for me was always difficulty
> > to get gcov-tool working.  If some maintainer steps up, I think I can
> > fix GCC side.
> >
> > I am bit unsure how important feature it is - we have FDO that works
> > quite well for most users but I know there are some users of the LLVM
> > implementation and there is potential to tie this with other hardware
> > events to asist i.e. if conversion (where one wants to know how well CPU
> > predicts the jump rather than just the jump probability) which I always
> > found potentially interesting.
> >
> > Honza
> > >
> > > Martin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Eugene
> > > >
> > >
> >

Re: State of AutoFDO in GCC

[Andi Kleen]

172060045@hdu.edu.cn writes:

> Hi all, 
>
> I`m using GCC 9.3 AutoFDO and the old version create_gcov on arm64 
> and it works well. Actually it support not only LBR like mode but 
> also inst_retired even cycles event, which`s the early implementation 
> of AutoFDO[1]. There is no difference in output format of create_gcov 
> between LBR mode and inst_retired. inst_retired is less accurate than 
> LBR but still works.
>
> I hope AutoFDO could work better on GCC and am willing to contribute 
> to it, does anybody have suggestions for me?

I think at a minimum we need someone to run the test suite regularly
and look at all the problem, at least filling PRs, so that regressions
could be fixed.

But I actually hope one of the automatic testers could at least do
the testing.

And perhaps try it on real applications and see how much you can
improve performance, and if there are problems also file PRs.

There's some other work that could be done (e.g. port some of the
missing changes from the old google autofdo tree to mainline), but that
might be more difficult depending on your gcc internals expertise.

-Andi

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Andi Kleen]

Eugene Rozenfeld via Gcc <gcc@gcc.gnu.org> writes:

> Is the format produced by create_gcov and expected by GCC under
> -fauto-rpofile documented somewhere? How is it different from .gcda
> used in FDO, e.g., as described here:
> http://src.gnu-darwin.org/src/contrib/gcc/gcov-io.h.html?

I believe it's very similar.

> I would prefer that AutoFDO is not removed from GCC and it would be
> helpful if create_gcov were restored in google/autofdo. I checked out
> a revision before the recent merge and tried it on a simple example
> and it seems to work.
> I'm also interested in contributing improvements for AutoFDO so will
> try to investigate https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71672
> and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81379

That would be great.

-Andi

Re: Re: State of AutoFDO in GCC

[172060045]

> > Hi all, 
> >
> > I`m using GCC 9.3 AutoFDO and the old version create_gcov on arm64 
> > and it works well. Actually it support not only LBR like mode but 
> > also inst_retired even cycles event, which`s the early implementation 
> > of AutoFDO[1]. There is no difference in output format of create_gcov 
> > between LBR mode and inst_retired. inst_retired is less accurate than 
> > LBR but still works.
> >
> > I hope AutoFDO could work better on GCC and am willing to contribute 
> > to it, does anybody have suggestions for me?
> 
> I think at a minimum we need someone to run the test suite regularly
> and look at all the problem, at least filling PRs, so that regressions
> could be fixed.
> 
> But I actually hope one of the automatic testers could at least do
> the testing.
> 

Hi Andi,
Thanks for your advice, I'm trying to start these jobs.

-Yancheng

> And perhaps try it on real applications and see how much you can
> improve performance, and if there are problems also file PRs.
> 

With my tests, AutoFDO could achieve almost half of the effect of 
instrumentation FDO on real applications such as MySQL 8.0.20 .

> There's some other work that could be done (e.g. port some of the
> missing changes from the old google autofdo tree to mainline), but that
> might be more difficult depending on your gcc internals expertise.
> 

get it, thanks.

> -Andi

Re: State of AutoFDO in GCC

[Andi Kleen]

> With my tests, AutoFDO could achieve almost half of the effect of
> instrumentation FDO on real applications such as MySQL 8.0.20 .

Likely this could be improved with some of the missing changes. 
Apparently discriminator support is worth quite a bit especially on 
dense C++ code bases. Without that, gcc autofdo only works on line 
numbers, which can be very limiting if single lines have a lot of basic 
blocks.

Sadly discriminator support is currently only on the old Google branch 
and not in gcc mainline

Longer term it would probably be best to replace all this with some 
custom specialized binary annotation instead of stretching DWARF beyond 
its limits.

-Andi

Re: State of AutoFDO in GCC

[Jan Hubicka]

> 
> > With my tests, AutoFDO could achieve almost half of the effect of
> > instrumentation FDO on real applications such as MySQL 8.0.20 .
> 
> Likely this could be improved with some of the missing changes. Apparently
> discriminator support is worth quite a bit especially on dense C++ code
> bases. Without that, gcc autofdo only works on line numbers, which can be
> very limiting if single lines have a lot of basic blocks.
> 
> Sadly discriminator support is currently only on the old Google branch and
> not in gcc mainline
> 
> Longer term it would probably be best to replace all this with some custom
> specialized binary annotation instead of stretching DWARF beyond its limits.

I think it makes sense to pick the AutoFDO to the lists of things that
would be nice to fix in GCC12.  I guess first we need to solve the
issues with the tool producing autofdo gcda files and once that works
setup some benchmarking so we know how things compare to FDO and they
get tested.

If you point me to the discriminator patches I can try to figure out how
hard would be to mainline them.  I am not too sure about custom
annotations though - storing info about regions of code segment is
always a pain and it is quite nice that dwarf provides support for that.
But I guess we could go step by step.  I first need a working setup ;)

Honza
> 
> -Andi
> 

Re: State of AutoFDO in GCC

[Andi Kleen]

On 5/9/2021 10:01 AM, Jan Hubicka wrote:
>>> With my tests, AutoFDO could achieve almost half of the effect of
>>> instrumentation FDO on real applications such as MySQL 8.0.20 .
>> Likely this could be improved with some of the missing changes. Apparently
>> discriminator support is worth quite a bit especially on dense C++ code
>> bases. Without that, gcc autofdo only works on line numbers, which can be
>> very limiting if single lines have a lot of basic blocks.
>>
>> Sadly discriminator support is currently only on the old Google branch and
>> not in gcc mainline
>>
>> Longer term it would probably be best to replace all this with some custom
>> specialized binary annotation instead of stretching DWARF beyond its limits.
> I think it makes sense to pick the AutoFDO to the lists of things that
> would be nice to fix in GCC12.  I guess first we need to solve the
> issues with the tool producing autofdo gcda files and once that works
> setup some benchmarking so we know how things compare to FDO and they
> get tested.

It should work with my updated branch and latest perf. This is an old 
version before the great LLVMification and the regressions, so it builds 
on its own. I removed all the checks that broke with new perf versions, 
at least as far as I could test.

https://github.com/andikleen/autofdo/tree/perf-future

Longer term I'm thinking the autofdo tools setup as currently done is 
not the right way anyways. The problem is that to get good results you 
need to keep autofdo running for a long time, but that causes perf to 
produce gigantic perf.data files on disk for every sample. For the gcc 
boot strap runs we had cases where it reached GBs. This is all quite 
unnecessary because all it needs to do is to keep some statistics on 
basic block edges, so keeping all the samples is not needed at all.

As a minimum we probably need to figure out how to run it in online in 
perf pipe mode to avoid the temporary files. But the actual code 
algorithms of create_gcov are rather simple, so maybe it could be 
converted into a simple online tool that does the profiling.

And then we need some tooling to find the profile data for a given 
binary in this combined output. Today it's rather difficult to patch 
build systems to always point to the right files. This would need some 
metadata or at least a file name convention for gcov. This would allow 
longer term profiling of full real systems with existing build systems.

> If you point me to the discriminator patches I can try to figure out how
> hard would be to mainline them.

It's difficult to find now because it was a branch in the old SVN that 
wasn't converted. Sadly the great git conversion was quite lossy.

IIRC it was separate patches in the google gcc_4_8 SVN branch (of which 
I don't seem to have a copy either), but in _4_9 they squashed 
everything in autofdo together. It could be gotten if someone has some 
backup of the old SVN repository and git convert the google_4_8 branch.

Here in the 4_9 version you can search for get_discriminator and diff it 
against newer versions:

https://github.com/andikleen/gcc-old-svn/blob/6ff70bb2ef3cc0a5c6940030a89546bf40e70891/gcc/auto-profile.c#L393

and all the changes to other files were in the complete autofdo patch:

https://github.com/andikleen/gcc-old-svn/commit/d71978a93358a397fb80b20f3a65caad3d9addf1#diff-94f0fa7f897ccce65856dc5a98bae4bf6957a346766613d79414c976d093aa4a

can also search for discriminator there

The basic ideas was quite simple. You have an unique value for the dwarf 
discriminator for each basic block, and then you include that in all the 
autofdo location comparisons.

>   I am not too sure about custom
> annotations though - storing info about regions of code segment is
> always a pain and it is quite nice that dwarf provides support for that.
> But I guess we could go step by step.  I first need a working setup ;)

The thing about custom annotations is that it would make autofdo 
independent of the early inliner, which is one of the main reasons for 
the strange structure and placement of the autofdo passes. If we had an 
independent stable identifier for code regions this could be all done at 
better places.

Maybe a similar thing could be done in dwarf, but it would seem a 
stretch because it's really designed around source code.

-Andi

Re: State of AutoFDO in GCC

[Joseph Myers]

On Mon, 10 May 2021, Andi Kleen via Gcc wrote:

> It's difficult to find now because it was a branch in the old SVN that wasn't
> converted. Sadly the great git conversion was quite lossy.

All branches and tags, including deleted ones, were converted (under 
not-fetched-by-default refs in some cases); the git repository has 
everything that might plausibly be useful, omitting only a few things that 
would have been meaningless to convert, such as mistaken branch creations 
in the root of the repository and the SVN hooks directory.  Use "git 
ls-remote git://gcc.gnu.org/git/gcc.git" to see the full list of over 5000 
refs available in the repository (or do a clone with --mirror to fetch 
them all).

> IIRC it was separate patches in the google gcc_4_8 SVN branch (of which I
> don't seem to have a copy either), but in _4_9 they squashed everything in

That branch is refs/vendors/google/heads/gcc-4_8 (refs/vendors/ isn't 
fetched by default, but it's there).  (The old SVN repository is also 
still available at svn://gcc.gnu.org/svn/gcc .)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: State of AutoFDO in GCC

[Andi Kleen]

On Mon, May 10, 2021 at 04:55:50PM +0000, Joseph Myers wrote:
> On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
> 
> > It's difficult to find now because it was a branch in the old SVN that wasn't
> > converted. Sadly the great git conversion was quite lossy.
> 
> All branches and tags, including deleted ones, were converted (under 
> not-fetched-by-default refs in some cases); the git repository has 
> everything that might plausibly be useful, omitting only a few things that 
> would have been meaningless to convert, such as mistaken branch creations 
> in the root of the repository and the SVN hooks directory.  Use "git 
> ls-remote git://gcc.gnu.org/git/gcc.git" to see the full list of over 5000 
> refs available in the repository (or do a clone with --mirror to fetch 
> them all).

Okay thanks. I don't see them in any of the web interfaces, neither on
https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git
nor on
https://github.com/gcc-mirror/gcc
but 
git fetch origin vendors/google/heads/gcc-4_8
does the trick for fetching the commits, but not the symbolic branches.

Anyways with that it looks like the discriminator changes are:

commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
Author: Dehao Chen <dehao@gcc.gnu.org>
Date:   Thu Oct 10 14:39:31 2013 +0000

    Use only lineno+discriminator (remove the callee function name) as the key to represent callsite. Because each callsite will have its discriminator if in the same line.

    2013-10-10  Dehao Chen  <dehao@google.com>

            * gcc/auto-profile.c (get_function_instance_by_decl): Remove
            callee_name from callsite.
            (read_function_instance): Likewise.

    From-SVN: r203379

commit 3987da76affbfbe7195c0a16b33beedc649ec14f
Author: Dehao Chen <dehao@gcc.gnu.org>
Date:   Tue Aug 27 16:46:49 2013 +0000

    Refactor AutoFDO to:

    1. Now that we have discriminator for inlined callsite, we do not need special handling for callsite location any more.
    2. If a source line is mapped to multiple BBs, only the first BB will be annotated.
    3. Before actual annotation, mark everythin BB/edge as not annotated.

    2013-08-27  Dehao Chen  <dehao@google.com>

            * gcc/auto-profile.c (location_set): New data structure.
            (get_count_info): Add new parameter.
            (get_combined_location): Remove unused parameter.
            (get_inline_stack): Remove unused parameter.
            (afdo_get_bb_count): Add new parameter.
            (afdo_annotate_cfg): Reset annotated flags.
commit 1e6c4a7a8fb8e20545bb9f9032d3854f3f794c18
Author: Dehao Chen <dehao@gcc.gnu.org>
Date:   Thu Aug 22 17:20:29 2013 +0000

    Set discriminator for call stmts within a same basic block.

    2013-08-22  Dehao Chen  <dehao@google.com>

            * gcc/tree-cfg.c (assign_discriminators): assign discriminator for
            call stmt in a same BB if it is mapped to a same line.

commit b0be0175f4cf18fdd77bb013b181eead3a0a4773
Author: Dehao Chen <dehao@gcc.gnu.org>
Date:   Mon Aug 19 21:26:33 2013 +0000

    Fix the discriminator assignment bug during hashing.

    2013-08-19  Dehao Chen  <dehao@google.com>

            * tree-cfg.c (next_discriminator_for_locus): Fix discriminator
            assignment bug.

    From-SVN: r201857
commit 9fa26998a63d4b22b637ed8702520819e408a694
Author: Dehao Chen <dehao@gcc.gnu.org>
Date:   Mon Aug 19 20:16:47 2013 +0000

    Add discrminator for inlined callsites.

    2013-08-19  Dehao Chen  (dehao@google.com)

            * include/dwarf2.def (DW_AT_GNU_discriminator): New attribute.
            * gcc/dwarf2out.c (add_call_src_coords_attributes): Emit discriminator
            attribute for inlined callsite.

    From-SVN: r201856

-Andi

Re: State of AutoFDO in GCC

[Wei Mi]

https://github.com/google/autofdo has been updated. Now
create_gcov/dump_gcov are added back and can be built separately.

Please look at "2.2 Build autofdo tool for gcc" in
https://github.com/google/autofdo#readme

On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <ak@linux.intel.com> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> >    Andi, thanks for pointing out the perf script issues. Can you please
> >    elaborate a bit on the exact issue you have seen? We’ve been using
> >    specific output of perf script such as mmap, LBR and callstack events
> >    filtered by process id. It works fine so far but may certainly hit
issues
> >    in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://github.com/andikleen/autofdo -b perf-future
>
> for testing.
>
> -Andi
>

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://github.com/google/autofdo#readme
and got build failures:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
/usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal error: perf_stat.pb.h: No such file or directory
#include "perf_stat.pb.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.

What is supposed to generate perf_stat.pb.h?

Thanks,

Eugene

From: Wei Mi <wmi@google.com>
Sent: Monday, May 10, 2021 4:47 PM
To: Andi Kleen <ak@linux.intel.com>
Cc: Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>; Wenlei He <wenlei@fb.com>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

https://github.com/google/autofdo<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C82398afe328045b86d9b08d9140de31a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562872153209962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9S2u2uxoHsNKqbZanANvvfcqaN3wQXOWwyezvAuiLFM%3D&reserved=0> has been updated. Now create_gcov/dump_gcov are added back and can be built separately.

Please look at "2.2 Build autofdo tool for gcc" in https://github.com/google/autofdo#readme
<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C82398afe328045b86d9b08d9140de31a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562872153209962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1jTlYoCwNKR8TX8uxXUJsnistwj8hOnXxas5dnW7UuU%3D&reserved=0>
On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <ak@linux.intel.com<mailto:ak@linux.intel.com>> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> >    Andi, thanks for pointing out the perf script issues. Can you please
> >    elaborate a bit on the exact issue you have seen? We've been using
> >    specific output of perf script such as mmap, LBR and callstack events
> >    filtered by process id. It works fine so far but may certainly hit issues
> >    in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://github.com/andikleen/autofdo<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C82398afe328045b86d9b08d9140de31a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562872153219952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FpyYs3I3ZVNtQ6zRlbyytt4THiAFKgkz1ImQb3EkhqE%3D&reserved=0> -b perf-future
>
> for testing.
>
> -Andi
>

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

It is a proto library build dependency issue which didn't expose on my
platform. I fix it at
https://github.com/google/autofdo/commit/98269aee9674cc885cc5eb1bd917eb2d12731710.
Please try again.

Thanks,
Wei.

On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <
Eugene.Rozenfeld@microsoft.com> wrote:

> I tried following the instructions in "2.2 Build autofdo tool for gcc" in
> https://github.com/google/autofdo#readme
> and got build failures:
>
>
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
>
> [1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
>
> FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
>
> /usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil
> -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF
> CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o
> CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
>
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>
>                  from ../sample_reader.h:18,
>
>                  from ../profile.h:15,
>
>                  from ../profile.cc:5:
>
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning:
> "DISALLOW_COPY_AND_ASSIGN" redefined
>
> #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from ../profile.h:14:0,
>
>                  from ../profile.cc:5:
>
> ../base/macros.h:114:0: note: this is the location of the previous
> definition
>
> #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>
>                  from ../sample_reader.h:18,
>
>                  from ../profile.h:15,
>
>                  from ../profile.cc:5:
>
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> warning: "arraysize" redefined
>
> #define arraysize(x) (sizeof(x) / sizeof(*x))
>
> In file included from ../profile.h:14:0,
>
>                  from ../profile.cc:5:
>
> ../base/macros.h:162:0: note: this is the location of the previous
> definition
>
> #define arraysize(array) (sizeof(ArraySizeHelper(array)))
>
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>
>                  from ../sample_reader.h:18,
>
>                  from ../profile.h:15,
>
>                  from ../profile.cc:5:
>
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal
> error: perf_stat.pb.h: No such file or directory
>
> #include "perf_stat.pb.h"
>
>           ^~~~~~~~~~~~~~~~
>
> compilation terminated.
>
>
>
> What is supposed to generate perf_stat.pb.h?
>
>
>
> Thanks,
>
>
>
> Eugene
>
>
>
> *From:* Wei Mi <wmi@google.com>
> *Sent:* Monday, May 10, 2021 4:47 PM
> *To:* Andi Kleen <ak@linux.intel.com>
> *Cc:* Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>;
> Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Eugene Rozenfeld <
> Eugene.Rozenfeld@microsoft.com>; Wenlei He <wenlei@fb.com>
> *Subject:* [EXTERNAL] Re: State of AutoFDO in GCC
>
>
>
> https://github.com/google/autofdo
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C82398afe328045b86d9b08d9140de31a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562872153209962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9S2u2uxoHsNKqbZanANvvfcqaN3wQXOWwyezvAuiLFM%3D&reserved=0>
> has been updated. Now create_gcov/dump_gcov are added back and can be built
> separately.
>
> Please look at "2.2 Build autofdo tool for gcc" in
> https://github.com/google/autofdo#readme
>
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C82398afe328045b86d9b08d9140de31a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562872153209962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1jTlYoCwNKR8TX8uxXUJsnistwj8hOnXxas5dnW7UuU%3D&reserved=0>
> On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <ak@linux.intel.com> wrote:
> >
> > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > >    Andi, thanks for pointing out the perf script issues. Can you please
> > >    elaborate a bit on the exact issue you have seen? We’ve been using
> > >    specific output of perf script such as mmap, LBR and callstack
> events
> > >    filtered by process id. It works fine so far but may certainly hit
> issues
> > >    in the future with extended uses.
> >
> > Okay I took a look at the latest autofdo now. It seems to be basically
> > a LLVM project now that depends on LLVM to even build with all kinds
> > of dependency hell on some old LLVM version and other packages.
> >
> > I guess gcc will really need a replacement that doesn't pull in
> > all of LLVM if it wants to continue supporting autofdo.
> >
> > I'm myself unable to build now.
> >
> > I'm using the old version I had a git fork of and that
> > was before all of this. I added a patch to make it work
> > with the latest perf by ignoring increased perf_attr
> > and unknown perf events.
> >
> > Honza please use
> >
> > https://github.com/andikleen/autofdo
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C82398afe328045b86d9b08d9140de31a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562872153219952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FpyYs3I3ZVNtQ6zRlbyytt4THiAFKgkz1ImQb3EkhqE%3D&reserved=0>
> -b perf-future
> >
> > for testing.
> >
> > -Andi
> >
>

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o 
/usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
 
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
 
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
 #define arraysize(x) (sizeof(x) / sizeof(*x))
 
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
 #define arraysize(array) (sizeof(ArraySizeHelper(array)))
 
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal error: perf_data.pb.h: No such file or directory
 #include "perf_data.pb.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.
[6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
ninja: build stopped: subcommand failed.

Thanks,

Eugene

From: Wei Mi <wmi@google.com> 
Sent: Saturday, May 22, 2021 9:37 AM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342270322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TxMivTXj9w0W5OFLunWcOpqf9wefrrUd0m3RqV8JLaA%3D&reserved=0. Please try again.

Thanks,
Wei.

On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342280318%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KBPIcf5k1v230tbLke9%2FdSt25nbPrHhpBvPb8PiebPY%3D&reserved=0and got build failures:
 
eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o 
/usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal error: perf_stat.pb.h: No such file or directory
#include "perf_stat.pb.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.
 
What is supposed to generate perf_stat.pb.h?
 
Thanks,
 
Eugene
 
From: Wei Mi <mailto:wmi@google.com> 
Sent: Monday, May 10, 2021 4:47 PM
To: Andi Kleen <mailto:ak@linux.intel.com>
Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>; mailto:gcc@gcc.gnu.org; Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He <mailto:wenlei@fb.com>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC
 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWJlvkaOdGT5p8ZLmeZYkZ6lPhSmdmzc0tEnBULcNsY%3D&reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.

Please look at "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fXLah0%2B0YHbo%2BHlbcgUFwlFQD9YwCUZApmZHoasiW0%3D&reserved=0
On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <mailto:ak@linux.intel.com> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> >    Andi, thanks for pointing out the perf script issues. Can you please
> >    elaborate a bit on the exact issue you have seen? We've been using
> >    specific output of perf script such as mmap, LBR and callstack events
> >    filtered by process id. It works fine so far but may certainly hit issues
> >    in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342300314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aNe2qZeEjsE2%2Fm7ZbaIFPFAcnF3wtQEkOJr0fG5tHWo%3D&reserved=0 -b perf-future
>
> for testing.
>
> -Andi
>

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

Sorry, I added dependency for create_gcov but missed it for dump_gcov.
Fixed it at
https://github.com/google/autofdo/commit/6ca36cdc30986f13583a3aef3e27746ca4fc5bf6
.

Thanks,
Wei.

On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <
Eugene.Rozenfeld@microsoft.com> wrote:

> Thank you Wei. Looks like something is still missing. This time
> perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04
> with cmake 3.12.1):
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> [1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil
> -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning:
> "DISALLOW_COPY_AND_ASSIGN" redefined
>  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:114:0: note: this is the location of the previous
> definition
>  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> warning: "arraysize" redefined
>  #define arraysize(x) (sizeof(x) / sizeof(*x))
>
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:162:0: note: this is the location of the previous
> definition
>  #define arraysize(array) (sizeof(ArraySizeHelper(array)))
>
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal
> error: perf_data.pb.h: No such file or directory
>  #include "perf_data.pb.h"
>           ^~~~~~~~~~~~~~~~
> compilation terminated.
> [6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> ninja: build stopped: subcommand failed.
>
> Thanks,
>
> Eugene
>
> From: Wei Mi <wmi@google.com>
> Sent: Saturday, May 22, 2021 9:37 AM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It is a proto library build dependency issue which didn't expose on my
> platform. I fix it at
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342270322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TxMivTXj9w0W5OFLunWcOpqf9wefrrUd0m3RqV8JLaA%3D&reserved=0.
> Please try again.
>
> Thanks,
> Wei.
>
> On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:
> Eugene.Rozenfeld@microsoft.com> wrote:
> I tried following the instructions in "2.2 Build autofdo tool for gcc" in
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342280318%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KBPIcf5k1v230tbLke9%2FdSt25nbPrHhpBvPb8PiebPY%3D&reserved=0and
> got build failures:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> [1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> /usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil
> -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF
> CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o
> CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning:
> "DISALLOW_COPY_AND_ASSIGN" redefined
> #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:114:0: note: this is the location of the previous
> definition
> #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> warning: "arraysize" redefined
> #define arraysize(x) (sizeof(x) / sizeof(*x))
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:162:0: note: this is the location of the previous
> definition
> #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal
> error: perf_stat.pb.h: No such file or directory
> #include "perf_stat.pb.h"
>           ^~~~~~~~~~~~~~~~
> compilation terminated.
>
> What is supposed to generate perf_stat.pb.h?
>
> Thanks,
>
> Eugene
>
> From: Wei Mi <mailto:wmi@google.com>
> Sent: Monday, May 10, 2021 4:47 PM
> To: Andi Kleen <mailto:ak@linux.intel.com>
> Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li <mailto:
> davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>; mailto:
> gcc@gcc.gnu.org; Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com>;
> Wenlei He <mailto:wenlei@fb.com>
> Subject: [EXTERNAL] Re: State of AutoFDO in GCC
>
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWJlvkaOdGT5p8ZLmeZYkZ6lPhSmdmzc0tEnBULcNsY%3D&reserved=0
> has been updated. Now create_gcov/dump_gcov are added back and can be built
> separately.
>
> Please look at "2.2 Build autofdo tool for gcc" in
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fXLah0%2B0YHbo%2BHlbcgUFwlFQD9YwCUZApmZHoasiW0%3D&reserved=0
> On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <mailto:ak@linux.intel.com>
> wrote:
> >
> > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > >    Andi, thanks for pointing out the perf script issues. Can you please
> > >    elaborate a bit on the exact issue you have seen? We've been using
> > >    specific output of perf script such as mmap, LBR and callstack
> events
> > >    filtered by process id. It works fine so far but may certainly hit
> issues
> > >    in the future with extended uses.
> >
> > Okay I took a look at the latest autofdo now. It seems to be basically
> > a LLVM project now that depends on LLVM to even build with all kinds
> > of dependency hell on some old LLVM version and other packages.
> >
> > I guess gcc will really need a replacement that doesn't pull in
> > all of LLVM if it wants to continue supporting autofdo.
> >
> > I'm myself unable to build now.
> >
> > I'm using the old version I had a git fork of and that
> > was before all of this. I added a patch to make it work
> > with the latest perf by ignoring increased perf_attr
> > and unknown perf events.
> >
> > Honza please use
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342300314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aNe2qZeEjsE2%2Fm7ZbaIFPFAcnF3wtQEkOJr0fG5tHWo%3D&reserved=0
> -b perf-future
> >
> > for testing.
> >
> > -Andi
> >
>

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

That fixed the error I saw before but the build still fails. The errors start with

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[2/217] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
/usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:144:26: error: 'unordered_map' in namespace 'std' does not name a template type
   using container = std::unordered_map<key_t, value_t>;
                          ^~~~~~~~~~~~~



From: Wei Mi <wmi@google.com>
Sent: Monday, May 24, 2021 8:12 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://github.com/google/autofdo/commit/6ca36cdc30986f13583a3aef3e27746ca4fc5bf6<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312421958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fWMYoqpMP9jRocxHKRKN5zv5DLIgCuLR%2FgczvA1rJc%3D&reserved=0>.

Thanks,
Wei.

On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>> wrote:
Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
/usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \

In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
 #define DISALLOW_COPY_AND_ASSIGN(TypeName) \

In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
 #define arraysize(x) (sizeof(x) / sizeof(*x))

In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
 #define arraysize(array) (sizeof(ArraySizeHelper(array)))

In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal error: perf_data.pb.h: No such file or directory
 #include "perf_data.pb.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.
[6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
ninja: build stopped: subcommand failed.

Thanks,

Eugene

From: Wei Mi <wmi@google.com<mailto:wmi@google.com>>
Sent: Saturday, May 22, 2021 9:37 AM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>>
Cc: Andi Kleen <ak@linux.intel.com<mailto:ak@linux.intel.com>>; Hongtao Yu <hoy@fb.com<mailto:hoy@fb.com>>; Xinliang David Li <davidxl@google.com<mailto:davidxl@google.com>>; Jan Hubicka <hubicka@ucw.cz<mailto:hubicka@ucw.cz>>; gcc@gcc.gnu.org<mailto:gcc@gcc.gnu.org>; Wenlei He <wenlei@fb.com<mailto:wenlei@fb.com>>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342270322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TxMivTXj9w0W5OFLunWcOpqf9wefrrUd0m3RqV8JLaA%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312431954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OdcAYo8mJD7ipadVeQq94T%2BGoBlO0AsJErGmeVKVJ7c%3D&reserved=0>. Please try again.

Thanks,
Wei.

On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>> wrote:
I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342280318%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KBPIcf5k1v230tbLke9%2FdSt25nbPrHhpBvPb8PiebPY%3D&reserved=0and<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312441950%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Nz6MH%2B2MFwbvgAAi6FcxJretHqF6SiH3P2oLbVqOJ%2B4%3D&reserved=0> got build failures:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
/usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
                 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
                 from ../sample_reader.h:18,
                 from ../profile.h:15,
                 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal error: perf_stat.pb.h: No such file or directory
#include "perf_stat.pb.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.

What is supposed to generate perf_stat.pb.h?

Thanks,

Eugene

From: Wei Mi <mailto:wmi@google.com<mailto:wmi@google.com>>
Sent: Monday, May 10, 2021 4:47 PM
To: Andi Kleen <mailto:ak@linux.intel.com<mailto:ak@linux.intel.com>>
Cc: Hongtao Yu <mailto:hoy@fb.com<mailto:hoy@fb.com>>; Xinliang David Li <mailto:davidxl@google.com<mailto:davidxl@google.com>>; Jan Hubicka <mailto:hubicka@ucw.cz<mailto:hubicka@ucw.cz>>; mailto:gcc@gcc.gnu.org<mailto:gcc@gcc.gnu.org>; Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com<mailto:Eugene.Rozenfeld@microsoft.com>>; Wenlei He <mailto:wenlei@fb.com<mailto:wenlei@fb.com>>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWJlvkaOdGT5p8ZLmeZYkZ6lPhSmdmzc0tEnBULcNsY%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312451945%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xGCwhFyre4ipz2xXZnkeIL1a%2Bd6x%2FbWLe58szmVYdO8%3D&reserved=0> has been updated. Now create_gcov/dump_gcov are added back and can be built separately.

Please look at "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fXLah0%2B0YHbo%2BHlbcgUFwlFQD9YwCUZApmZHoasiW0%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312461941%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4U%2B6DVDVtRVsfE%2FN41s2IaygGaFchfwVqvbhiC%2FSLTY%3D&reserved=0>
On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <mailto:ak@linux.intel.com<mailto:ak@linux.intel.com>> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> >    Andi, thanks for pointing out the perf script issues. Can you please
> >    elaborate a bit on the exact issue you have seen? We've been using
> >    specific output of perf script such as mmap, LBR and callstack events
> >    filtered by process id. It works fine so far but may certainly hit issues
> >    in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342300314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aNe2qZeEjsE2%2Fm7ZbaIFPFAcnF3wtQEkOJr0fG5tHWo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cac1d4d3757714d05fc1308d91f2ae1ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575091312471940%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4JAfSVhReYE9ITLr5IKzLucH5ckdFtvGG5AKtnFjBZQ%3D&reserved=0> -b perf-future
>
> for testing.
>
> -Andi
>

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

It isn't exposed on my platform either. Looks like a bug in
perf_data_converter (i.e., quipper). Could you try adding #include
<unordered_map> in
third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and
see if it fixes the problem? If it works, I will need to file a bug
against perf_data_converter.

Thanks,
Wei.

On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld
<Eugene.Rozenfeld@microsoft.com> wrote:
>
> That fixed the error I saw before but the build still fails. The errors start with
>
>
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
>
> [2/217] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
>
> FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
>
> /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
>
> ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:144:26: error: ‘unordered_map’ in namespace ‘std’ does not name a template type
>
>    using container = std::unordered_map<key_t, value_t>;
>
>                           ^~~~~~~~~~~~~
>
>
>
>
>
>
>
> From: Wei Mi <wmi@google.com>
> Sent: Monday, May 24, 2021 8:12 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
>
>
> Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://github.com/google/autofdo/commit/6ca36cdc30986f13583a3aef3e27746ca4fc5bf6.
>
>
>
> Thanks,
>
> Wei.
>
>
>
> On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
>
> Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> [1/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
>  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:114:0: note: this is the location of the previous definition
>  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
>  #define arraysize(x) (sizeof(x) / sizeof(*x))
>
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:162:0: note: this is the location of the previous definition
>  #define arraysize(array) (sizeof(ArraySizeHelper(array)))
>
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal error: perf_data.pb.h: No such file or directory
>  #include "perf_data.pb.h"
>           ^~~~~~~~~~~~~~~~
> compilation terminated.
> [6/241] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> ninja: build stopped: subcommand failed.
>
> Thanks,
>
> Eugene
>
> From: Wei Mi <wmi@google.com>
> Sent: Saturday, May 22, 2021 9:37 AM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342270322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TxMivTXj9w0W5OFLunWcOpqf9wefrrUd0m3RqV8JLaA%3D&reserved=0. Please try again.
>
> Thanks,
> Wei.
>
> On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342280318%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KBPIcf5k1v230tbLke9%2FdSt25nbPrHhpBvPb8PiebPY%3D&reserved=0and got build failures:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> [1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> /usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: "DISALLOW_COPY_AND_ASSIGN" redefined
> #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:114:0: note: this is the location of the previous definition
> #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: "arraysize" redefined
> #define arraysize(x) (sizeof(x) / sizeof(*x))
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:162:0: note: this is the location of the previous definition
> #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal error: perf_stat.pb.h: No such file or directory
> #include "perf_stat.pb.h"
>           ^~~~~~~~~~~~~~~~
> compilation terminated.
>
> What is supposed to generate perf_stat.pb.h?
>
> Thanks,
>
> Eugene
>
> From: Wei Mi <mailto:wmi@google.com>
> Sent: Monday, May 10, 2021 4:47 PM
> To: Andi Kleen <mailto:ak@linux.intel.com>
> Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>; mailto:gcc@gcc.gnu.org; Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He <mailto:wenlei@fb.com>
> Subject: [EXTERNAL] Re: State of AutoFDO in GCC
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWJlvkaOdGT5p8ZLmeZYkZ6lPhSmdmzc0tEnBULcNsY%3D&reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
>
> Please look at "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342290320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3fXLah0%2B0YHbo%2BHlbcgUFwlFQD9YwCUZApmZHoasiW0%3D&reserved=0
> On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen <mailto:ak@linux.intel.com> wrote:
> >
> > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > >    Andi, thanks for pointing out the perf script issues. Can you please
> > >    elaborate a bit on the exact issue you have seen? We've been using
> > >    specific output of perf script such as mmap, LBR and callstack events
> > >    filtered by process id. It works fine so far but may certainly hit issues
> > >    in the future with extended uses.
> >
> > Okay I took a look at the latest autofdo now. It seems to be basically
> > a LLVM project now that depends on LLVM to even build with all kinds
> > of dependency hell on some old LLVM version and other packages.
> >
> > I guess gcc will really need a replacement that doesn't pull in
> > all of LLVM if it wants to continue supporting autofdo.
> >
> > I'm myself unable to build now.
> >
> > I'm using the old version I had a git fork of and that
> > was before all of this. I added a patch to make it work
> > with the latest perf by ignoring increased perf_attr
> > and unknown perf events.
> >
> > Honza please use
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C414e79d9b075428ceffc08d91d3fd7ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637572982342300314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aNe2qZeEjsE2%2Fm7ZbaIFPFAcnF3wtQEkOJr0fG5tHWo%3D&reserved=0 -b perf-future
> >
> > for testing.
> >
> > -Andi
> >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

That eliminates the previous error but there is a new one:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[3/199] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o 
/usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
         nrcpus = proto_uint32_metadata.data()[0];

-----Original Message-----
From: Wei Mi <wmi@google.com> 
Sent: Monday, May 24, 2021 8:54 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.

Thanks,
Wei.

On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
>
> That fixed the error I saw before but the build still fails. The 
> errors start with
>
>
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
>
> [2/217] Building CXX object 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> r/huge_page_deducer.cc.o
>
> FAILED: 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> r/huge_page_deducer.cc.o
>
> /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
>
> ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:14
> 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> template type
>
>    using container = std::unordered_map<key_t, value_t>;
>
>                           ^~~~~~~~~~~~~
>
>
>
>
>
>
>
> From: Wei Mi <wmi@google.com>
> Sent: Monday, May 24, 2021 8:12 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang 
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; 
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
>
>
> Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816277204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=hwJ%2BG64Yw%2BEGMAV7kzilOjAOkHoOv3TQpdqkzcHGO%2FM%3D&amp;reserved=0.
>
>
>
> Thanks,
>
> Wei.
>
>
>
> On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
>
> Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] Building 
> CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: 
> warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define 
> DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:114:0: note: this is the location of the previous 
> definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
>
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: 
> warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) / 
> sizeof(*x))
>
> In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:162:0: note: this is the location of the previous 
> definition  #define arraysize(array) (sizeof(ArraySizeHelper(array)))
>
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: 
> fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
>           ^~~~~~~~~~~~~~~~
> compilation terminated.
> [6/241] Building CXX object 
> CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> ninja: build stopped: subcommand failed.
>
> Thanks,
>
> Eugene
>
> From: Wei Mi <wmi@google.com>
> Sent: Saturday, May 22, 2021 9:37 AM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang 
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; 
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PikMs8RtJHBsTUqdjCVWAKNSC5Rls%2B71pSEhq4z0XsM%3D&amp;reserved=0. Please try again.
>
> Thanks,
> Wei.
>
> On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp;reserved=0 got build failures:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] Building 
> CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> /usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil 
> -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: 
> warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define 
> DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:114:0: note: this is the location of the previous 
> definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file 
> included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: 
> warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / 
> sizeof(*x)) In file included from ../profile.h:14:0,
>                  from ../profile.cc:5:
> ../base/macros.h:162:0: note: this is the location of the previous 
> definition #define arraysize(array) (sizeof(ArraySizeHelper(array))) 
> In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
>                  from ../sample_reader.h:18,
>                  from ../profile.h:15,
>                  from ../profile.cc:5:
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: 
> fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
>           ^~~~~~~~~~~~~~~~
> compilation terminated.
>
> What is supposed to generate perf_stat.pb.h?
>
> Thanks,
>
> Eugene
>
> From: Wei Mi <mailto:wmi@google.com>
> Sent: Monday, May 10, 2021 4:47 PM
> To: Andi Kleen <mailto:ak@linux.intel.com>
> Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li 
> <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>; 
> mailto:gcc@gcc.gnu.org; Eugene Rozenfeld 
> <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He 
> <mailto:wenlei@fb.com>
> Subject: [EXTERNAL] Re: State of AutoFDO in GCC
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=a9gGf48MYY2jzbSucOAlhsmdLuYt%2BcM8V5hvGoA39Ms%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
>
> Please look at "2.2 Build autofdo tool for gcc" in 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld
> %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp;res
> erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen 
> <mailto:ak@linux.intel.com> wrote:
> >
> > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > >    Andi, thanks for pointing out the perf script issues. Can you please
> > >    elaborate a bit on the exact issue you have seen? We've been using
> > >    specific output of perf script such as mmap, LBR and callstack events
> > >    filtered by process id. It works fine so far but may certainly hit issues
> > >    in the future with extended uses.
> >
> > Okay I took a look at the latest autofdo now. It seems to be 
> > basically a LLVM project now that depends on LLVM to even build with 
> > all kinds of dependency hell on some old LLVM version and other packages.
> >
> > I guess gcc will really need a replacement that doesn't pull in all 
> > of LLVM if it wants to continue supporting autofdo.
> >
> > I'm myself unable to build now.
> >
> > I'm using the old version I had a git fork of and that was before 
> > all of this. I added a patch to make it work with the latest perf by 
> > ignoring increased perf_attr and unknown perf events.
> >
> > Honza please use
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%4
> > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af
> > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3
> > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D&a
> > mp;reserved=0 -b perf-future
> >
> > for testing.
> >
> > -Andi
> >

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

It looks like some version problem about protobuf-compiler and
libprotobuf-dev. Could you check what is the installed version on your
end for those two packages and see if they are consistent?

On my platform, they are both 3.12.4.

On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld
<Eugene.Rozenfeld@microsoft.com> wrote:
>
> That eliminates the previous error but there is a new one:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> [3/199] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
>          nrcpus = proto_uint32_metadata.data()[0];
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Monday, May 24, 2021 8:54 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
>
> Thanks,
> Wei.
>
> On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > That fixed the error I saw before but the build still fails. The
> > errors start with
> >
> >
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> >
> > [2/217] Building CXX object
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> > r/huge_page_deducer.cc.o
> >
> > FAILED:
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> > r/huge_page_deducer.cc.o
> >
> > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> >
> > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:14
> > 4:26: error: 'unordered_map' in namespace 'std' does not name a
> > template type
> >
> >    using container = std::unordered_map<key_t, value_t>;
> >
> >                           ^~~~~~~~~~~~~
> >
> >
> >
> >
> >
> >
> >
> > From: Wei Mi <wmi@google.com>
> > Sent: Monday, May 24, 2021 8:12 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> >
> >
> > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816277204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=hwJ%2BG64Yw%2BEGMAV7kzilOjAOkHoOv3TQpdqkzcHGO%2FM%3D&amp;reserved=0.
> >
> >
> >
> > Thanks,
> >
> > Wei.
> >
> >
> >
> > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] Building
> > CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> >
> > In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:114:0: note: this is the location of the previous
> > definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> >
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) /
> > sizeof(*x))
> >
> > In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:162:0: note: this is the location of the previous
> > definition  #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> >
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
> >           ^~~~~~~~~~~~~~~~
> > compilation terminated.
> > [6/241] Building CXX object
> > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > ninja: build stopped: subcommand failed.
> >
> > Thanks,
> >
> > Eugene
> >
> > From: Wei Mi <wmi@google.com>
> > Sent: Saturday, May 22, 2021 9:37 AM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PikMs8RtJHBsTUqdjCVWAKNSC5Rls%2B71pSEhq4z0XsM%3D&amp;reserved=0. Please try again.
> >
> > Thanks,
> > Wei.
> >
> > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp;reserved=0 got build failures:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] Building
> > CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > /usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil
> > -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:114:0: note: this is the location of the previous
> > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file
> > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > sizeof(*x)) In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:162:0: note: this is the location of the previous
> > definition #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> >           ^~~~~~~~~~~~~~~~
> > compilation terminated.
> >
> > What is supposed to generate perf_stat.pb.h?
> >
> > Thanks,
> >
> > Eugene
> >
> > From: Wei Mi <mailto:wmi@google.com>
> > Sent: Monday, May 10, 2021 4:47 PM
> > To: Andi Kleen <mailto:ak@linux.intel.com>
> > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li
> > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>;
> > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld
> > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He
> > <mailto:wenlei@fb.com>
> > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=a9gGf48MYY2jzbSucOAlhsmdLuYt%2BcM8V5hvGoA39Ms%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> >
> > Please look at "2.2 Build autofdo tool for gcc" in
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af
> > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8
> > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp;res
> > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen
> > <mailto:ak@linux.intel.com> wrote:
> > >
> > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > >    Andi, thanks for pointing out the perf script issues. Can you please
> > > >    elaborate a bit on the exact issue you have seen? We've been using
> > > >    specific output of perf script such as mmap, LBR and callstack events
> > > >    filtered by process id. It works fine so far but may certainly hit issues
> > > >    in the future with extended uses.
> > >
> > > Okay I took a look at the latest autofdo now. It seems to be
> > > basically a LLVM project now that depends on LLVM to even build with
> > > all kinds of dependency hell on some old LLVM version and other packages.
> > >
> > > I guess gcc will really need a replacement that doesn't pull in all
> > > of LLVM if it wants to continue supporting autofdo.
> > >
> > > I'm myself unable to build now.
> > >
> > > I'm using the old version I had a git fork of and that was before
> > > all of this. I added a patch to make it work with the latest perf by
> > > ignoring increased perf_attr and unknown perf events.
> > >
> > > Honza please use
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%4
> > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af
> > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3
> > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D&a
> > > mp;reserved=0 -b perf-future
> > >
> > > for testing.
> > >
> > > -Andi
> > >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Both are 3.0.0-9.1ubuntu1:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list protobuf-compiler
Listing... Done
protobuf-compiler/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list libprotobuf-dev
Listing... Done
libprotobuf-dev/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]

-----Original Message-----
From: Wei Mi <wmi@google.com> 
Sent: Tuesday, May 25, 2021 9:17 AM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent?

On my platform, they are both 3.12.4.

On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
>
> That eliminates the previous error but there is a new one:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] Building 
> CXX object 
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> r/perf_reader.cc.o
> FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
>          nrcpus = proto_uint32_metadata.data()[0];
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Monday, May 24, 2021 8:54 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang 
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; 
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
>
> Thanks,
> Wei.
>
> On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > That fixed the error I saw before but the build still fails. The 
> > errors start with
> >
> >
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> >
> > [2/217] Building CXX object
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > pe
> > r/huge_page_deducer.cc.o
> >
> > FAILED:
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > pe
> > r/huge_page_deducer.cc.o
> >
> > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> >
> > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > 14
> > 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> > template type
> >
> >    using container = std::unordered_map<key_t, value_t>;
> >
> >                           ^~~~~~~~~~~~~
> >
> >
> >
> >
> >
> >
> >
> > From: Wei Mi <wmi@google.com>
> > Sent: Monday, May 24, 2021 8:12 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> >
> >
> > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315397315%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=9G9uGhG%2BmK35bdV4QL9BOTVg6bFFQrVP46YdLOi3Ak4%3D&amp;reserved=0.
> >
> >
> >
> > Thanks,
> >
> > Wei.
> >
> >
> >
> > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] 
> > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> >
> > In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:114:0: note: this is the location of the previous 
> > definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> >
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) /
> > sizeof(*x))
> >
> > In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:162:0: note: this is the location of the previous 
> > definition  #define arraysize(array) 
> > (sizeof(ArraySizeHelper(array)))
> >
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
> >           ^~~~~~~~~~~~~~~~
> > compilation terminated.
> > [6/241] Building CXX object
> > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > ninja: build stopped: subcommand failed.
> >
> > Thanks,
> >
> > Eugene
> >
> > From: Wei Mi <wmi@google.com>
> > Sent: Saturday, May 22, 2021 9:37 AM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=SL5mLhelrkeB26Mg7tNaml0qjZ2OklYPwQR9btVhsqk%3D&amp;reserved=0. Please try again.
> >
> > Thanks,
> > Wei.
> >
> > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PYVm%2BHsRk%2Bfw7%2BaNgYNWn8zMnBsqcqqVSpyXQktd7sA%3D&amp;reserved=0 got build failures:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] 
> > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > /usr/bin/c++  -I../ -I../third_party/glog/src 
> > -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:114:0: note: this is the location of the previous 
> > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file 
> > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > sizeof(*x)) In file included from ../profile.h:14:0,
> >                  from ../profile.cc:5:
> > ../base/macros.h:162:0: note: this is the location of the previous 
> > definition #define arraysize(array) (sizeof(ArraySizeHelper(array))) 
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> >                  from ../sample_reader.h:18,
> >                  from ../profile.h:15,
> >                  from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> >           ^~~~~~~~~~~~~~~~
> > compilation terminated.
> >
> > What is supposed to generate perf_stat.pb.h?
> >
> > Thanks,
> >
> > Eugene
> >
> > From: Wei Mi <mailto:wmi@google.com>
> > Sent: Monday, May 10, 2021 4:47 PM
> > To: Andi Kleen <mailto:ak@linux.intel.com>
> > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li 
> > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>; 
> > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld 
> > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He 
> > <mailto:wenlei@fb.com>
> > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=uwQwWdjE%2FL%2F2r0P%2Bdn7r5ulE8kRmPPw8A2XOQuZnUhM%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> >
> > Please look at "2.2 Build autofdo tool for gcc" in 
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > th 
> > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfe
> > ld 
> > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141
> > af
> > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3
> > d8
> > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> > C1 
> > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp;r
> > es
> > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen 
> > <mailto:ak@linux.intel.com> wrote:
> > >
> > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > >    Andi, thanks for pointing out the perf script issues. Can you please
> > > >    elaborate a bit on the exact issue you have seen? We've been using
> > > >    specific output of perf script such as mmap, LBR and callstack events
> > > >    filtered by process id. It works fine so far but may certainly hit issues
> > > >    in the future with extended uses.
> > >
> > > Okay I took a look at the latest autofdo now. It seems to be 
> > > basically a LLVM project now that depends on LLVM to even build 
> > > with all kinds of dependency hell on some old LLVM version and other packages.
> > >
> > > I guess gcc will really need a replacement that doesn't pull in 
> > > all of LLVM if it wants to continue supporting autofdo.
> > >
> > > I'm myself unable to build now.
> > >
> > > I'm using the old version I had a git fork of and that was before 
> > > all of this. I added a patch to make it work with the latest perf 
> > > by ignoring increased perf_attr and unknown perf events.
> > >
> > > Honza please use
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > gi
> > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld
> > > %4 
> > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141
> > > af
> > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs
> > > b3 
> > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > > 3D 
> > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D
> > > &a
> > > mp;reserved=0 -b perf-future
> > >
> > > for testing.
> > >
> > > -Andi
> > >

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

I checked the source of protobuf 3.0.0 and it didn't contain the
operator[] in RepeatedField. Need to install a newer version of
protobuf.

Thanks,
Wei.

On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld
<Eugene.Rozenfeld@microsoft.com> wrote:
>
> Both are 3.0.0-9.1ubuntu1:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list protobuf-compiler
> Listing... Done
> protobuf-compiler/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list libprotobuf-dev
> Listing... Done
> libprotobuf-dev/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed]
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Tuesday, May 25, 2021 9:17 AM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent?
>
> On my platform, they are both 3.12.4.
>
> On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > That eliminates the previous error but there is a new one:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] Building
> > CXX object
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> > r/perf_reader.cc.o
> > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
> >          nrcpus = proto_uint32_metadata.data()[0];
> >
> > -----Original Message-----
> > From: Wei Mi <wmi@google.com>
> > Sent: Monday, May 24, 2021 8:54 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
> >
> > Thanks,
> > Wei.
> >
> > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > That fixed the error I saw before but the build still fails. The
> > > errors start with
> > >
> > >
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > >
> > > [2/217] Building CXX object
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > FAILED:
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > >
> > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > 14
> > > 4:26: error: 'unordered_map' in namespace 'std' does not name a
> > > template type
> > >
> > >    using container = std::unordered_map<key_t, value_t>;
> > >
> > >                           ^~~~~~~~~~~~~
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Monday, May 24, 2021 8:12 PM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > >
> > >
> > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315397315%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=9G9uGhG%2BmK35bdV4QL9BOTVg6bFFQrVP46YdLOi3Ak4%3D&amp;reserved=0.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Wei.
> > >
> > >
> > >
> > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241]
> > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > >
> > > In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:114:0: note: this is the location of the previous
> > > definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > >
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) /
> > > sizeof(*x))
> > >
> > > In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:162:0: note: this is the location of the previous
> > > definition  #define arraysize(array)
> > > (sizeof(ArraySizeHelper(array)))
> > >
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > > fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
> > >           ^~~~~~~~~~~~~~~~
> > > compilation terminated.
> > > [6/241] Building CXX object
> > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > > ninja: build stopped: subcommand failed.
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Saturday, May 22, 2021 9:37 AM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=SL5mLhelrkeB26Mg7tNaml0qjZ2OklYPwQR9btVhsqk%3D&amp;reserved=0. Please try again.
> > >
> > > Thanks,
> > > Wei.
> > >
> > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PYVm%2BHsRk%2Bfw7%2BaNgYNWn8zMnBsqcqqVSpyXQktd7sA%3D&amp;reserved=0 got build failures:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228]
> > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > /usr/bin/c++  -I../ -I../third_party/glog/src
> > > -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:114:0: note: this is the location of the previous
> > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file
> > > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > sizeof(*x)) In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:162:0: note: this is the location of the previous
> > > definition #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> > >           ^~~~~~~~~~~~~~~~
> > > compilation terminated.
> > >
> > > What is supposed to generate perf_stat.pb.h?
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> > > From: Wei Mi <mailto:wmi@google.com>
> > > Sent: Monday, May 10, 2021 4:47 PM
> > > To: Andi Kleen <mailto:ak@linux.intel.com>
> > > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li
> > > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>;
> > > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld
> > > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He
> > > <mailto:wenlei@fb.com>
> > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=uwQwWdjE%2FL%2F2r0P%2Bdn7r5ulE8kRmPPw8A2XOQuZnUhM%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> > >
> > > Please look at "2.2 Build autofdo tool for gcc" in
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfe
> > > ld
> > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141
> > > af
> > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3
> > > d8
> > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> > > C1
> > > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp;r
> > > es
> > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen
> > > <mailto:ak@linux.intel.com> wrote:
> > > >
> > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > >    Andi, thanks for pointing out the perf script issues. Can you please
> > > > >    elaborate a bit on the exact issue you have seen? We've been using
> > > > >    specific output of perf script such as mmap, LBR and callstack events
> > > > >    filtered by process id. It works fine so far but may certainly hit issues
> > > > >    in the future with extended uses.
> > > >
> > > > Okay I took a look at the latest autofdo now. It seems to be
> > > > basically a LLVM project now that depends on LLVM to even build
> > > > with all kinds of dependency hell on some old LLVM version and other packages.
> > > >
> > > > I guess gcc will really need a replacement that doesn't pull in
> > > > all of LLVM if it wants to continue supporting autofdo.
> > > >
> > > > I'm myself unable to build now.
> > > >
> > > > I'm using the old version I had a git fork of and that was before
> > > > all of this. I added a patch to make it work with the latest perf
> > > > by ignoring increased perf_attr and unknown perf events.
> > > >
> > > > Honza please use
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > > gi
> > > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld
> > > > %4
> > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141
> > > > af
> > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs
> > > > b3
> > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > > > 3D
> > > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D
> > > > &a
> > > > mp;reserved=0 -b perf-future
> > > >
> > > > for testing.
> > > >
> > > > -Andi
> > > >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

3.0.0 is the latest supported version on Ubuntu 18.04. I verified that the build works on Ubuntu 20.04 with  protobuf-compiler and libprotobuf-dev version 3.6.1.3.

Eugene

-----Original Message-----
From: Wei Mi <wmi@google.com> 
Sent: Tuesday, May 25, 2021 8:07 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

I checked the source of protobuf 3.0.0 and it didn't contain the operator[] in RepeatedField. Need to install a newer version of protobuf.

Thanks,
Wei.

On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
>
> Both are 3.0.0-9.1ubuntu1:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> protobuf-compiler Listing... Done protobuf-compiler/bionic,now 
> 3.0.0-9.1ubuntu1 amd64 [installed] 
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now 
> 3.0.0-9.1ubuntu1 amd64 [installed]
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Tuesday, May 25, 2021 9:17 AM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang 
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; 
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent?
>
> On my platform, they are both 3.12.4.
>
> On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > That eliminates the previous error but there is a new one:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] 
> > Building CXX object 
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > pe
> > r/perf_reader.cc.o
> > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
> >          nrcpus = proto_uint32_metadata.data()[0];
> >
> > -----Original Message-----
> > From: Wei Mi <wmi@google.com>
> > Sent: Monday, May 24, 2021 8:54 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
> >
> > Thanks,
> > Wei.
> >
> > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > That fixed the error I saw before but the build still fails. The 
> > > errors start with
> > >
> > >
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > >
> > > [2/217] Building CXX object
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > ip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > FAILED:
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > ip
> > > pe
> > > r/huge_page_deducer.cc.o
> > >
> > > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > >
> > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > 14
> > > 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> > > template type
> > >
> > >    using container = std::unordered_map<key_t, value_t>;
> > >
> > >                           ^~~~~~~~~~~~~
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Monday, May 24, 2021 8:12 PM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > >
> > >
> > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275806806%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=JnEc%2FRvRNehbDEh2NhqjONnPJJDEaoFEqPCet%2BhByXU%3D&amp;reserved=0.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Wei.
> > >
> > >
> > >
> > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] 
> > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > >
> > > In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:114:0: note: this is the location of the previous 
> > > definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > >
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) /
> > > sizeof(*x))
> > >
> > > In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:162:0: note: this is the location of the previous 
> > > definition  #define arraysize(array)
> > > (sizeof(ArraySizeHelper(array)))
> > >
> > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > > fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
> > >           ^~~~~~~~~~~~~~~~
> > > compilation terminated.
> > > [6/241] Building CXX object
> > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > > ninja: build stopped: subcommand failed.
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Saturday, May 22, 2021 9:37 AM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275806806%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=g3hPIoq%2BbklBPFZOFdtXhcAtAtky7W4KyjOdL6S%2F7bI%3D&amp;reserved=0. Please try again.
> > >
> > > Thanks,
> > > Wei.
> > >
> > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275816803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=mkBZF667WnlvaktNmI9VInqfLLKbIu1B41g%2BaqUi03w%3D&amp;reserved=0 got build failures:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] 
> > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > /usr/bin/c++  -I../ -I../third_party/glog/src 
> > > -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:114:0: note: this is the location of the previous 
> > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file 
> > > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > sizeof(*x)) In file included from ../profile.h:14:0,
> > >                  from ../profile.cc:5:
> > > ../base/macros.h:162:0: note: this is the location of the previous 
> > > definition #define arraysize(array) 
> > > (sizeof(ArraySizeHelper(array))) In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > >                  from ../sample_reader.h:18,
> > >                  from ../profile.h:15,
> > >                  from ../profile.cc:5:
> > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> > >           ^~~~~~~~~~~~~~~~
> > > compilation terminated.
> > >
> > > What is supposed to generate perf_stat.pb.h?
> > >
> > > Thanks,
> > >
> > > Eugene
> > >
> > > From: Wei Mi <mailto:wmi@google.com>
> > > Sent: Monday, May 10, 2021 4:47 PM
> > > To: Andi Kleen <mailto:ak@linux.intel.com>
> > > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li 
> > > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>; 
> > > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld 
> > > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He 
> > > <mailto:wenlei@fb.com>
> > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275816803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=THWF6m1ly3ZHjfqLCSDummsISA2bItUWVyHmdLIvxmU%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> > >
> > > Please look at "2.2 Build autofdo tool for gcc" in 
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > gi
> > > th
> > > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozen
> > > fe
> > > ld
> > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f1
> > > 41
> > > af
> > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs
> > > b3
> > > d8
> > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > > %7
> > > C1
> > > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp
> > > ;r
> > > es
> > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen 
> > > <mailto:ak@linux.intel.com> wrote:
> > > >
> > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > >    Andi, thanks for pointing out the perf script issues. Can you please
> > > > >    elaborate a bit on the exact issue you have seen? We've been using
> > > > >    specific output of perf script such as mmap, LBR and callstack events
> > > > >    filtered by process id. It works fine so far but may certainly hit issues
> > > > >    in the future with extended uses.
> > > >
> > > > Okay I took a look at the latest autofdo now. It seems to be 
> > > > basically a LLVM project now that depends on LLVM to even build 
> > > > with all kinds of dependency hell on some old LLVM version and other packages.
> > > >
> > > > I guess gcc will really need a replacement that doesn't pull in 
> > > > all of LLVM if it wants to continue supporting autofdo.
> > > >
> > > > I'm myself unable to build now.
> > > >
> > > > I'm using the old version I had a git fork of and that was 
> > > > before all of this. I added a patch to make it work with the 
> > > > latest perf by ignoring increased perf_attr and unknown perf events.
> > > >
> > > > Honza please use
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > 2F
> > > > gi
> > > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfe
> > > > ld
> > > > %4
> > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f1
> > > > 41
> > > > af
> > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbG
> > > > Zs
> > > > b3
> > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > > > 0%
> > > > 3D
> > > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%
> > > > 3D
> > > > &a
> > > > mp;reserved=0 -b perf-future
> > > >
> > > > for testing.
> > > >
> > > > -Andi
> > > >

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

Thanks. Good to know the build works with newer protobuf.

Wei.

On Wed, May 26, 2021 at 4:40 PM Eugene Rozenfeld
<Eugene.Rozenfeld@microsoft.com> wrote:
>
> 3.0.0 is the latest supported version on Ubuntu 18.04. I verified that the build works on Ubuntu 20.04 with  protobuf-compiler and libprotobuf-dev version 3.6.1.3.
>
> Eugene
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Tuesday, May 25, 2021 8:07 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> I checked the source of protobuf 3.0.0 and it didn't contain the operator[] in RepeatedField. Need to install a newer version of protobuf.
>
> Thanks,
> Wei.
>
> On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > Both are 3.0.0-9.1ubuntu1:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list
> > protobuf-compiler Listing... Done protobuf-compiler/bionic,now
> > 3.0.0-9.1ubuntu1 amd64 [installed]
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list
> > libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now
> > 3.0.0-9.1ubuntu1 amd64 [installed]
> >
> > -----Original Message-----
> > From: Wei Mi <wmi@google.com>
> > Sent: Tuesday, May 25, 2021 9:17 AM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent?
> >
> > On my platform, they are both 3.12.4.
> >
> > On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > That eliminates the previous error but there is a new one:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199]
> > > Building CXX object
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip
> > > pe
> > > r/perf_reader.cc.o
> > > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
> > >          nrcpus = proto_uint32_metadata.data()[0];
> > >
> > > -----Original Message-----
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Monday, May 24, 2021 8:54 PM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
> > >
> > > Thanks,
> > > Wei.
> > >
> > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > > >
> > > > That fixed the error I saw before but the build still fails. The
> > > > errors start with
> > > >
> > > >
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > > >
> > > > [2/217] Building CXX object
> > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > > ip
> > > > pe
> > > > r/huge_page_deducer.cc.o
> > > >
> > > > FAILED:
> > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > > ip
> > > > pe
> > > > r/huge_page_deducer.cc.o
> > > >
> > > > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > > >
> > > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > > 14
> > > > 4:26: error: 'unordered_map' in namespace 'std' does not name a
> > > > template type
> > > >
> > > >    using container = std::unordered_map<key_t, value_t>;
> > > >
> > > >                           ^~~~~~~~~~~~~
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Wei Mi <wmi@google.com>
> > > > Sent: Monday, May 24, 2021 8:12 PM
> > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > >
> > > >
> > > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275806806%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=JnEc%2FRvRNehbDEh2NhqjONnPJJDEaoFEqPCet%2BhByXU%3D&amp;reserved=0.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Wei.
> > > >
> > > >
> > > >
> > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > > >
> > > > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241]
> > > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > > /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > > >
> > > > In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:114:0: note: this is the location of the previous
> > > > definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > > >
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) /
> > > > sizeof(*x))
> > > >
> > > > In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:162:0: note: this is the location of the previous
> > > > definition  #define arraysize(array)
> > > > (sizeof(ArraySizeHelper(array)))
> > > >
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > > > fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
> > > >           ^~~~~~~~~~~~~~~~
> > > > compilation terminated.
> > > > [6/241] Building CXX object
> > > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > > > ninja: build stopped: subcommand failed.
> > > >
> > > > Thanks,
> > > >
> > > > Eugene
> > > >
> > > > From: Wei Mi <wmi@google.com>
> > > > Sent: Saturday, May 22, 2021 9:37 AM
> > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275806806%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=g3hPIoq%2BbklBPFZOFdtXhcAtAtky7W4KyjOdL6S%2F7bI%3D&amp;reserved=0. Please try again.
> > > >
> > > > Thanks,
> > > > Wei.
> > > >
> > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > > > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275816803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=mkBZF667WnlvaktNmI9VInqfLLKbIu1B41g%2BaqUi03w%3D&amp;reserved=0 got build failures:
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228]
> > > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > > /usr/bin/c++  -I../ -I../third_party/glog/src
> > > > -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:114:0: note: this is the location of the previous
> > > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file
> > > > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > > sizeof(*x)) In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:162:0: note: this is the location of the previous
> > > > definition #define arraysize(array)
> > > > (sizeof(ArraySizeHelper(array))) In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > > > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> > > >           ^~~~~~~~~~~~~~~~
> > > > compilation terminated.
> > > >
> > > > What is supposed to generate perf_stat.pb.h?
> > > >
> > > > Thanks,
> > > >
> > > > Eugene
> > > >
> > > > From: Wei Mi <mailto:wmi@google.com>
> > > > Sent: Monday, May 10, 2021 4:47 PM
> > > > To: Andi Kleen <mailto:ak@linux.intel.com>
> > > > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li
> > > > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>;
> > > > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld
> > > > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He
> > > > <mailto:wenlei@fb.com>
> > > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cc13414d95f7a4b50bd9108d91ff35765%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575952275816803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=THWF6m1ly3ZHjfqLCSDummsISA2bItUWVyHmdLIvxmU%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> > > >
> > > > Please look at "2.2 Build autofdo tool for gcc" in
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > > gi
> > > > th
> > > > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozen
> > > > fe
> > > > ld
> > > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f1
> > > > 41
> > > > af
> > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs
> > > > b3
> > > > d8
> > > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > > > %7
> > > > C1
> > > > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&amp
> > > > ;r
> > > > es
> > > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen
> > > > <mailto:ak@linux.intel.com> wrote:
> > > > >
> > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > > >    Andi, thanks for pointing out the perf script issues. Can you please
> > > > > >    elaborate a bit on the exact issue you have seen? We've been using
> > > > > >    specific output of perf script such as mmap, LBR and callstack events
> > > > > >    filtered by process id. It works fine so far but may certainly hit issues
> > > > > >    in the future with extended uses.
> > > > >
> > > > > Okay I took a look at the latest autofdo now. It seems to be
> > > > > basically a LLVM project now that depends on LLVM to even build
> > > > > with all kinds of dependency hell on some old LLVM version and other packages.
> > > > >
> > > > > I guess gcc will really need a replacement that doesn't pull in
> > > > > all of LLVM if it wants to continue supporting autofdo.
> > > > >
> > > > > I'm myself unable to build now.
> > > > >
> > > > > I'm using the old version I had a git fork of and that was
> > > > > before all of this. I added a patch to make it work with the
> > > > > latest perf by ignoring increased perf_attr and unknown perf events.
> > > > >
> > > > > Honza please use
> > > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > 2F
> > > > > gi
> > > > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfe
> > > > > ld
> > > > > %4
> > > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f1
> > > > > 41
> > > > > af
> > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbG
> > > > > Zs
> > > > > b3
> > > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > > > > 0%
> > > > > 3D
> > > > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%
> > > > > 3D
> > > > > &a
> > > > > mp;reserved=0 -b perf-future
> > > > >
> > > > > for testing.
> > > > >
> > > > > -Andi
> > > > >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

It appears that create_gcov doesn't support binaries with dwarf version 5 (which is the current default). 
I tried a trivial example and got reasonable gcov files for binaries with dwarf v2, v3, and v4 but the same example with dwarf v5 produced 
"File './sort' has mangled .debug_info section."
and a gcov file with 0 functions.

Does create_llvm_prof has the same limitation?

Eugene

-----Original Message-----
From: Wei Mi <wmi@google.com> 
Sent: Wednesday, May 26, 2021 7:52 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

Thanks. Good to know the build works with newer protobuf.

Wei.

On Wed, May 26, 2021 at 4:40 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
>
> 3.0.0 is the latest supported version on Ubuntu 18.04. I verified that the build works on Ubuntu 20.04 with  protobuf-compiler and libprotobuf-dev version 3.6.1.3.
>
> Eugene
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Tuesday, May 25, 2021 8:07 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang 
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; 
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> I checked the source of protobuf 3.0.0 and it didn't contain the operator[] in RepeatedField. Need to install a newer version of protobuf.
>
> Thanks,
> Wei.
>
> On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > Both are 3.0.0-9.1ubuntu1:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> > protobuf-compiler Listing... Done protobuf-compiler/bionic,now
> > 3.0.0-9.1ubuntu1 amd64 [installed]
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list 
> > libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now
> > 3.0.0-9.1ubuntu1 amd64 [installed]
> >
> > -----Original Message-----
> > From: Wei Mi <wmi@google.com>
> > Sent: Tuesday, May 25, 2021 9:17 AM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent?
> >
> > On my platform, they are both 3.12.4.
> >
> > On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > That eliminates the previous error but there is a new one:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] 
> > > Building CXX object 
> > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > ip
> > > pe
> > > r/perf_reader.cc.o
> > > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
> > >          nrcpus = proto_uint32_metadata.data()[0];
> > >
> > > -----Original Message-----
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Monday, May 24, 2021 8:54 PM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
> > >
> > > Thanks,
> > > Wei.
> > >
> > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > > >
> > > > That fixed the error I saw before but the build still fails. The 
> > > > errors start with
> > > >
> > > >
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > > >
> > > > [2/217] Building CXX object
> > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/
> > > > qu
> > > > ip
> > > > pe
> > > > r/huge_page_deducer.cc.o
> > > >
> > > > FAILED:
> > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/
> > > > qu
> > > > ip
> > > > pe
> > > > r/huge_page_deducer.cc.o
> > > >
> > > > /usr/bin/c++   -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > > >
> > > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > > 14
> > > > 4:26: error: 'unordered_map' in namespace 'std' does not name a 
> > > > template type
> > > >
> > > >    using container = std::unordered_map<key_t, value_t>;
> > > >
> > > >                           ^~~~~~~~~~~~~
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Wei Mi <wmi@google.com>
> > > > Sent: Monday, May 24, 2021 8:12 PM
> > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > >
> > > >
> > > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=826uc8So5JIW6bTbsxBMiEbLWexdiBiPjYrxexHOLmI%3D&amp;reserved=0.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Wei.
> > > >
> > > >
> > > >
> > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> > > >
> > > > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] 
> > > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > > /usr/bin/c++   -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > > >
> > > > In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:114:0: note: this is the location of the 
> > > > previous definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName) 
> > > > \
> > > >
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x) 
> > > > /
> > > > sizeof(*x))
> > > >
> > > > In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:162:0: note: this is the location of the 
> > > > previous definition  #define arraysize(array)
> > > > (sizeof(ArraySizeHelper(array)))
> > > >
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > > > fatal error: perf_data.pb.h: No such file or directory  #include "perf_data.pb.h"
> > > >           ^~~~~~~~~~~~~~~~
> > > > compilation terminated.
> > > > [6/241] Building CXX object
> > > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > > > ninja: build stopped: subcommand failed.
> > > >
> > > > Thanks,
> > > >
> > > > Eugene
> > > >
> > > > From: Wei Mi <wmi@google.com>
> > > > Sent: Saturday, May 22, 2021 9:37 AM
> > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; 
> > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka 
> > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2Br9qK1fc2SQu%2FT9rHOCsJcYqzpkXnDu7SNfpLB9NCqQ%3D&amp;reserved=0. Please try again.
> > > >
> > > > Thanks,
> > > > Wei.
> > > >
> > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > > > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=KpbQ1SBpwtPLdgJCpJQ6aW6iHj3mvisPN5kZVRiRQhM%3D&amp;reserved=0 got build failures:
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] 
> > > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > > /usr/bin/c++  -I../ -I../third_party/glog/src 
> > > > -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:114:0: note: this is the location of the 
> > > > previous definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ 
> > > > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > > sizeof(*x)) In file included from ../profile.h:14:0,
> > > >                  from ../profile.cc:5:
> > > > ../base/macros.h:162:0: note: this is the location of the 
> > > > previous definition #define arraysize(array)
> > > > (sizeof(ArraySizeHelper(array))) In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > >                  from ../sample_reader.h:18,
> > > >                  from ../profile.h:15,
> > > >                  from ../profile.cc:5:
> > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > > > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> > > >           ^~~~~~~~~~~~~~~~
> > > > compilation terminated.
> > > >
> > > > What is supposed to generate perf_stat.pb.h?
> > > >
> > > > Thanks,
> > > >
> > > > Eugene
> > > >
> > > > From: Wei Mi <mailto:wmi@google.com>
> > > > Sent: Monday, May 10, 2021 4:47 PM
> > > > To: Andi Kleen <mailto:ak@linux.intel.com>
> > > > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li 
> > > > <mailto:davidxl@google.com>; Jan Hubicka 
> > > > <mailto:hubicka@ucw.cz>; mailto:gcc@gcc.gnu.org; Eugene 
> > > > Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He 
> > > > <mailto:wenlei@fb.com>
> > > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258135446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=0qVyYYmYDZfHt3cdxg1LjUYvonck6G6PLAOFeYCNnTc%3D&amp;reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> > > >
> > > > Please look at "2.2 Build autofdo tool for gcc" in 
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > 2F
> > > > gi
> > > > th
> > > > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Roz
> > > > en
> > > > fe
> > > > ld
> > > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86
> > > > f1
> > > > 41
> > > > af
> > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbG
> > > > Zs
> > > > b3
> > > > d8
> > > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > > > 3D
> > > > %7
> > > > C1
> > > > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&a
> > > > mp
> > > > ;r
> > > > es
> > > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen 
> > > > <mailto:ak@linux.intel.com> wrote:
> > > > >
> > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > > >    Andi, thanks for pointing out the perf script issues. Can you please
> > > > > >    elaborate a bit on the exact issue you have seen? We've been using
> > > > > >    specific output of perf script such as mmap, LBR and callstack events
> > > > > >    filtered by process id. It works fine so far but may certainly hit issues
> > > > > >    in the future with extended uses.
> > > > >
> > > > > Okay I took a look at the latest autofdo now. It seems to be 
> > > > > basically a LLVM project now that depends on LLVM to even 
> > > > > build with all kinds of dependency hell on some old LLVM version and other packages.
> > > > >
> > > > > I guess gcc will really need a replacement that doesn't pull 
> > > > > in all of LLVM if it wants to continue supporting autofdo.
> > > > >
> > > > > I'm myself unable to build now.
> > > > >
> > > > > I'm using the old version I had a git fork of and that was 
> > > > > before all of this. I added a patch to make it work with the 
> > > > > latest perf by ignoring increased perf_attr and unknown perf events.
> > > > >
> > > > > Honza please use
> > > > >
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > F%
> > > > > 2F
> > > > > gi
> > > > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozen
> > > > > fe
> > > > > ld
> > > > > %4
> > > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86
> > > > > f1
> > > > > 41
> > > > > af
> > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFp
> > > > > bG
> > > > > Zs
> > > > > b3
> > > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > > > Mn
> > > > > 0%
> > > > > 3D
> > > > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8
> > > > > g%
> > > > > 3D
> > > > > &a
> > > > > mp;reserved=0 -b perf-future
> > > > >
> > > > > for testing.
> > > > >
> > > > > -Andi
> > > > >

Re: [EXTERNAL] Re: State of AutoFDO in GCC

[Wei Mi]

Yeah, create_llvm_prof depends on llvm_symbolizer which shouldn't have the
same limitation. To separate from llvm, create_gcov still depends on the
old set of addr2line library which likely doesn't support dwarf v5. That is
an issue indeed.

Thanks,
Wei.

On Fri, Jun 11, 2021 at 6:14 PM Eugene Rozenfeld <
Eugene.Rozenfeld@microsoft.com> wrote:

> It appears that create_gcov doesn't support binaries with dwarf version 5
> (which is the current default).
> I tried a trivial example and got reasonable gcov files for binaries with
> dwarf v2, v3, and v4 but the same example with dwarf v5 produced
> "File './sort' has mangled .debug_info section."
> and a gcov file with 0 functions.
>
> Does create_llvm_prof has the same limitation?
>
> Eugene
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Wednesday, May 26, 2021 7:52 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> Thanks. Good to know the build works with newer protobuf.
>
> Wei.
>
> On Wed, May 26, 2021 at 4:40 PM Eugene Rozenfeld <
> Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > 3.0.0 is the latest supported version on Ubuntu 18.04. I verified that
> the build works on Ubuntu 20.04 with  protobuf-compiler and libprotobuf-dev
> version 3.6.1.3.
> >
> > Eugene
> >
> > -----Original Message-----
> > From: Wei Mi <wmi@google.com>
> > Sent: Tuesday, May 25, 2021 8:07 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > I checked the source of protobuf 3.0.0 and it didn't contain the
> operator[] in RepeatedField. Need to install a newer version of protobuf.
> >
> > Thanks,
> > Wei.
> >
> > On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld <
> Eugene.Rozenfeld@microsoft.com> wrote:
> > >
> > > Both are 3.0.0-9.1ubuntu1:
> > >
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list
> > > protobuf-compiler Listing... Done protobuf-compiler/bionic,now
> > > 3.0.0-9.1ubuntu1 amd64 [installed]
> > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list
> > > libprotobuf-dev Listing... Done libprotobuf-dev/bionic,now
> > > 3.0.0-9.1ubuntu1 amd64 [installed]
> > >
> > > -----Original Message-----
> > > From: Wei Mi <wmi@google.com>
> > > Sent: Tuesday, May 25, 2021 9:17 AM
> > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > >
> > > It looks like some version problem about protobuf-compiler and
> libprotobuf-dev. Could you check what is the installed version on your end
> for those two packages and see if they are consistent?
> > >
> > > On my platform, they are both 3.12.4.
> > >
> > > On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld <
> Eugene.Rozenfeld@microsoft.com> wrote:
> > > >
> > > > That eliminates the previous error but there is a new one:
> > > >
> > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199]
> > > > Building CXX object
> > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/qu
> > > > ip
> > > > pe
> > > > r/perf_reader.cc.o
> > > > FAILED:
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> > > > /usr/bin/c++   -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../
> -I../third_party/glog/src -I../third_party/abseil -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> -MF
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d
> -o
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> > > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In
> member function 'bool
> quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> > > >
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46:
> error: no match for 'operator[]' (operand types are 'const
> google::protobuf::RepeatedField<unsigned int>' and 'int')
> > > >          nrcpus = proto_uint32_metadata.data()[0];
> > > >
> > > > -----Original Message-----
> > > > From: Wei Mi <wmi@google.com>
> > > > Sent: Monday, May 24, 2021 8:54 PM
> > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > >
> > > > It isn't exposed on my platform either. Looks like a bug in
> perf_data_converter (i.e., quipper). Could you try adding #include
> <unordered_map> in
> third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if
> it fixes the problem? If it works, I will need to file a bug against
> perf_data_converter.
> > > >
> > > > Thanks,
> > > > Wei.
> > > >
> > > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <
> Eugene.Rozenfeld@microsoft.com> wrote:
> > > > >
> > > > > That fixed the error I saw before but the build still fails. The
> > > > > errors start with
> > > > >
> > > > >
> > > > >
> > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> > > > >
> > > > > [2/217] Building CXX object
> > > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/
> > > > > qu
> > > > > ip
> > > > > pe
> > > > > r/huge_page_deducer.cc.o
> > > > >
> > > > > FAILED:
> > > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/
> > > > > qu
> > > > > ip
> > > > > pe
> > > > > r/huge_page_deducer.cc.o
> > > > >
> > > > > /usr/bin/c++   -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../
> -I../third_party/glog/src -I../third_party/abseil -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
> -MF
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d
> -o
> CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o
> -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> > > > >
> > > > >
> ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:
> > > > > 14
> > > > > 4:26: error: 'unordered_map' in namespace 'std' does not name a
> > > > > template type
> > > > >
> > > > >    using container = std::unordered_map<key_t, value_t>;
> > > > >
> > > > >                           ^~~~~~~~~~~~~
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Wei Mi <wmi@google.com>
> > > > > Sent: Monday, May 24, 2021 8:12 PM
> > > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > > >
> > > > >
> > > > >
> > > > > Sorry, I added dependency for create_gcov but missed it for
> dump_gcov. Fixed it at
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=826uc8So5JIW6bTbsxBMiEbLWexdiBiPjYrxexHOLmI%3D&amp;reserved=0
> .
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Wei.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <
> Eugene.Rozenfeld@microsoft.com> wrote:
> > > > >
> > > > > Thank you Wei. Looks like something is still missing. This time
> perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04
> with cmake 3.12.1):
> > > > >
> > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241]
> > > > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > > > > /usr/bin/c++   -I../ -I../third_party/glog/src
> -I../third_party/abseil -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o
> CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > > > > In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > > >                  from ../sample_reader.h:18,
> > > > >                  from ../profile.h:15,
> > > > >                  from ../profile.cc:5:
> > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined  #define
> > > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > > > >
> > > > > In file included from ../profile.h:14:0,
> > > > >                  from ../profile.cc:5:
> > > > > ../base/macros.h:114:0: note: this is the location of the
> > > > > previous definition  #define DISALLOW_COPY_AND_ASSIGN(TypeName)
> > > > > \
> > > > >
> > > > > In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > > >                  from ../sample_reader.h:18,
> > > > >                  from ../profile.h:15,
> > > > >                  from ../profile.cc:5:
> > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > > > warning: "arraysize" redefined  #define arraysize(x) (sizeof(x)
> > > > > /
> > > > > sizeof(*x))
> > > > >
> > > > > In file included from ../profile.h:14:0,
> > > > >                  from ../profile.cc:5:
> > > > > ../base/macros.h:162:0: note: this is the location of the
> > > > > previous definition  #define arraysize(array)
> > > > > (sizeof(ArraySizeHelper(array)))
> > > > >
> > > > > In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > > >                  from ../sample_reader.h:18,
> > > > >                  from ../profile.h:15,
> > > > >                  from ../profile.cc:5:
> > > > >
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > > > > fatal error: perf_data.pb.h: No such file or directory  #include
> "perf_data.pb.h"
> > > > >           ^~~~~~~~~~~~~~~~
> > > > > compilation terminated.
> > > > > [6/241] Building CXX object
> > > > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > > > > ninja: build stopped: subcommand failed.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Eugene
> > > > >
> > > > > From: Wei Mi <wmi@google.com>
> > > > > Sent: Saturday, May 22, 2021 9:37 AM
> > > > > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > > > > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>;
> > > > > Xinliang David Li <davidxl@google.com>; Jan Hubicka
> > > > > <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> > > > >
> > > > > It is a proto library build dependency issue which didn't expose
> on my platform. I fix it at
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2Br9qK1fc2SQu%2FT9rHOCsJcYqzpkXnDu7SNfpLB9NCqQ%3D&amp;reserved=0.
> Please try again.
> > > > >
> > > > > Thanks,
> > > > > Wei.
> > > > >
> > > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:
> Eugene.Rozenfeld@microsoft.com> wrote:
> > > > > I tried following the instructions in "2.2 Build autofdo tool for
> gcc" in
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258125451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=KpbQ1SBpwtPLdgJCpJQ6aW6iHj3mvisPN5kZVRiRQhM%3D&amp;reserved=0
> got build failures:
> > > > >
> > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228]
> > > > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > > > > /usr/bin/c++  -I../ -I../third_party/glog/src
> > > > > -I../third_party/abseil -I../third_party/perf_data_converter/src
> -I../third_party/perf_data_converter/src/quipper -I../util -I.
> -Ithird_party/glog -std=gnu++1z -MD -MT
> CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF
> CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o
> CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file
> included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > > >                  from ../sample_reader.h:18,
> > > > >                  from ../profile.h:15,
> > > > >                  from ../profile.cc:5:
> > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from
> ../profile.h:14:0,
> > > > >                  from ../profile.cc:5:
> > > > > ../base/macros.h:114:0: note: this is the location of the
> > > > > previous definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> > > > > In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > > > >                  from ../sample_reader.h:18,
> > > > >                  from ../profile.h:15,
> > > > >                  from ../profile.cc:5:
> > > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > > > > sizeof(*x)) In file included from ../profile.h:14:0,
> > > > >                  from ../profile.cc:5:
> > > > > ../base/macros.h:162:0: note: this is the location of the
> > > > > previous definition #define arraysize(array)
> > > > > (sizeof(ArraySizeHelper(array))) In file included from
> ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > > > >                  from ../sample_reader.h:18,
> > > > >                  from ../profile.h:15,
> > > > >                  from ../profile.cc:5:
> > > > >
> ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > > > > fatal error: perf_stat.pb.h: No such file or directory #include
> "perf_stat.pb.h"
> > > > >           ^~~~~~~~~~~~~~~~
> > > > > compilation terminated.
> > > > >
> > > > > What is supposed to generate perf_stat.pb.h?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Eugene
> > > > >
> > > > > From: Wei Mi <mailto:wmi@google.com>
> > > > > Sent: Monday, May 10, 2021 4:47 PM
> > > > > To: Andi Kleen <mailto:ak@linux.intel.com>
> > > > > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li
> > > > > <mailto:davidxl@google.com>; Jan Hubicka
> > > > > <mailto:hubicka@ucw.cz>; mailto:gcc@gcc.gnu.org; Eugene
> > > > > Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He
> > > > > <mailto:wenlei@fb.com>
> > > > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> > > > >
> > > > >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cf1e7a16701ba4295722c08d920ba6846%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637576807258135446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=0qVyYYmYDZfHt3cdxg1LjUYvonck6G6PLAOFeYCNnTc%3D&amp;reserved=0
> has been updated. Now create_gcov/dump_gcov are added back and can be built
> separately.
> > > > >
> > > > > Please look at "2.2 Build autofdo tool for gcc" in
> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > > > > 2F
> > > > > gi
> > > > > th
> > > > > ub.com%2Fgoogle%2Fautofdo%23readme&amp;data=04%7C01%7CEugene.Roz
> > > > > en
> > > > > fe
> > > > > ld
> > > > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86
> > > > > f1
> > > > > 41
> > > > > af
> > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbG
> > > > > Zs
> > > > > b3
> > > > > d8
> > > > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > > > > 3D
> > > > > %7
> > > > > C1
> > > > > 000&amp;sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&a
> > > > > mp
> > > > > ;r
> > > > > es
> > > > > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen
> > > > > <mailto:ak@linux.intel.com> wrote:
> > > > > >
> > > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > > > >    Andi, thanks for pointing out the perf script issues. Can
> you please
> > > > > > >    elaborate a bit on the exact issue you have seen? We've
> been using
> > > > > > >    specific output of perf script such as mmap, LBR and
> callstack events
> > > > > > >    filtered by process id. It works fine so far but may
> certainly hit issues
> > > > > > >    in the future with extended uses.
> > > > > >
> > > > > > Okay I took a look at the latest autofdo now. It seems to be
> > > > > > basically a LLVM project now that depends on LLVM to even
> > > > > > build with all kinds of dependency hell on some old LLVM version
> and other packages.
> > > > > >
> > > > > > I guess gcc will really need a replacement that doesn't pull
> > > > > > in all of LLVM if it wants to continue supporting autofdo.
> > > > > >
> > > > > > I'm myself unable to build now.
> > > > > >
> > > > > > I'm using the old version I had a git fork of and that was
> > > > > > before all of this. I added a patch to make it work with the
> > > > > > latest perf by ignoring increased perf_attr and unknown perf
> events.
> > > > > >
> > > > > > Honza please use
> > > > > >
> > > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > > > > > F%
> > > > > > 2F
> > > > > > gi
> > > > > > thub.com%2Fandikleen%2Fautofdo&amp;data=04%7C01%7CEugene.Rozen
> > > > > > fe
> > > > > > ld
> > > > > > %4
> > > > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86
> > > > > > f1
> > > > > > 41
> > > > > > af
> > > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFp
> > > > > > bG
> > > > > > Zs
> > > > > > b3
> > > > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > > > > Mn
> > > > > > 0%
> > > > > > 3D
> > > > > > %7C1000&amp;sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8
> > > > > > g%
> > > > > > 3D
> > > > > > &a
> > > > > > mp;reserved=0 -b perf-future
> > > > > >
> > > > > > for testing.
> > > > > >
> > > > > > -Andi
> > > > > >
>

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Hi Andy,

I'm trying to revive autofdo testing. One of the issues I'm running into with my setup is that PEBS doesn't work for with perf record even though PEBS is enabled.
I'm running Ubuntu 20.04 in a Hyper-V virtual machine; the processor is Icelake (GenuineIntel-6-7E).

I did the following:

1. Enabled pmu, lbr, and pebs in my Hyper-V virtual machine as described in https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/performance-monitoring-hardware
2. Verified that pmu, lbr, and pebs are enabled in the vm by running 
	erozen@erozen-Virtual-Machine:~/objdir/gcc$ dmesg | egrep -i 'pmu'
	[    0.266474] Performance Events: PEBS fmt4+, Icelake events, 32-deep LBR, full-width counters, Intel PMU driver.
3. Ran
	erozen@erozen-Virtual-Machine:~/objdir/gcc$ perf record -e cpu/event=0xc4,umask=0x20/pu -b -m8 true -v
	Error:
	The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (cpu/event=0xc4,umask=0x20/pu).
	/bin/dmesg | grep -i perf may provide additional information.

Omitting /p works fine:
	erozen@erozen-Virtual-Machine:~/objdir/gcc$ perf record -e cpu/event=0xc4,umask=0x20/u -b -m8 true -v
	[ perf record: Woken up 0 times to write data ]
	[ perf record: Captured and wrote 0.007 MB perf.data (11 samples) ]

Is there a way to get PEBS working with perf record in a vm? I would appreciate any pointers on how to investigate this.

The version of perf I'm using is 5.8.18.

Thanks,

Eugene

-----Original Message-----
From: Andi Kleen <ak@linux.intel.com> 
Sent: Friday, April 30, 2021 2:46 PM
To: Eugene Rozenfeld via Gcc <gcc@gcc.gnu.org>
Cc: Xinliang David Li <davidxl@google.com>; Richard Biener <richard.guenther@gmail.com>; Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>; Jan Hubicka <hubicka@ucw.cz>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC

Eugene Rozenfeld via Gcc <gcc@gcc.gnu.org> writes:

> Is the format produced by create_gcov and expected by GCC under 
> -fauto-rpofile documented somewhere? How is it different from .gcda 
> used in FDO, e.g., as described here:
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsrc.gnu-darwin.org%2Fsrc%2Fcontrib%2Fgcc%2Fgcov-io.h.html&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C6c14ea3d93c44364845008d90c214cb6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637554159427749575%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=q9ROiOTma41UeQ%2FQG%2BUktEOrHAWonTTpcPRPmx%2Fgw0g%3D&amp;reserved=0?

I believe it's very similar.

> I would prefer that AutoFDO is not removed from GCC and it would be 
> helpful if create_gcov were restored in google/autofdo. I checked out 
> a revision before the recent merge and tried it on a simple example 
> and it seems to work.
> I'm also interested in contributing improvements for AutoFDO so will 
> try to investigate 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D71672&amp;data=04%7C01%7CEuge
> ne.Rozenfeld%40microsoft.com%7C6c14ea3d93c44364845008d90c214cb6%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637554159427749575%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn0%3D%7C1000&amp;sdata=99Igueuxq7AoHU%2B20BZs4E4K5rgdPFCiR8eygKaJdK
> E%3D&amp;reserved=0 and 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D81379&amp;data=04%7C01%7CEuge
> ne.Rozenfeld%40microsoft.com%7C6c14ea3d93c44364845008d90c214cb6%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637554159427759566%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn0%3D%7C1000&amp;sdata=Es91Dtt5Wt6%2BJtPWxHhkHqdWBVwzCiF5PcuXoHjY%2
> Bzs%3D&amp;reserved=0

That would be great.

-Andi

Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Hello GCC community.

I started this thread on the state of AutoFDO in GCC more than a year ago. Here is the first message in the thread: https://gcc.gnu.org/pipermail/gcc/2021-April/235860.html

Since then I committed a number of patches to revive AutoFDO in GCC:

Fix a typo in an AutoFDO error string<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=23691ddd3aa3ffe55892b2bff54f9a15a89de2b4>
Update gen_autofdo_event.py and gcc-auto-profile.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=01d402c5e0ac1ddf5618bbe316b50067625fda46>
Fixes for AutoFDO tests<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=f9ad3d5339faaaed6e15a7b27d90fbc66eb72f37>
Fix indir-call-prof-2.c with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=0ed093c7c3f755bc1cd80e5186abeb2f5c50ee0c>
Fixes for AutoFDO testing<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=9265b378531391498ec1727f67a45da72a6c07e9>
Fix indirect call inlining with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0>
Improve AutoFDO count propagation algorithm<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=3d9e6767939e9658260e2506e81ec32b37cba041>
AutoFDO: don't set param_early_inliner_max_iterations to 10.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=c17975d81aaed49ff759c20c68b31304a6953d58>
AutoFDO: Don't try to promote indirect calls that result in recursive direct calls<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ba125745d9e9fe90a18a2af8701b3269c5fdd468>
Fix profile count maintenance in vectorizer peeling.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0>

I also made a number of fixes and improvements to create_gcov tool in https://github.com/google/autofdo .

AutoFDO in GCC is in a much better shape now.

I have a further set of patches that improve DWARF discriminator support in GCC and enable AutoFDO to use discriminators. It's based on commits in an old Google vendor branch as described in Andi's mail below
but uses a different approach for keeping track of per-instruction discriminators.

I submitted the first (and the biggest) of these patches almost 2 months ago on June 2: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0
but only got a review from Andi (https://gcc.gnu.org/pipermail/gcc-patches/2022-June/596549.html) who is not allowed to approve patches for commit. I pinged gcc-patches twice with no success.

I would appreciate help in getting a review on this patch so that I can get it committed and submit patches that depend on it.

Thank you,

Eugene

-----Original Message-----
From: Andi Kleen <ak@linux.intel.com>
Sent: Monday, May 10, 2021 10:21 AM
To: Joseph Myers <joseph@codesourcery.com>
Cc: Jan Hubicka <hubicka@ucw.cz>; gcc <gcc@gcc.gnu.org>; Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

On Mon, May 10, 2021 at 04:55:50PM +0000, Joseph Myers wrote:
> On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
>
> > It's difficult to find now because it was a branch in the old SVN
> > that wasn't converted. Sadly the great git conversion was quite lossy.
>
> All branches and tags, including deleted ones, were converted (under
> not-fetched-by-default refs in some cases); the git repository has
> everything that might plausibly be useful, omitting only a few things
> that would have been meaningless to convert, such as mistaken branch
> creations in the root of the repository and the SVN hooks directory.
> Use "git ls-remote git://gcc.gnu.org/git/gcc.git" to see the full list
> of over 5000 refs available in the repository (or do a clone with
> --mirror to fetch them all).

Okay thanks. I don't see them in any of the web interfaces, neither on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fgit%2Fgitweb.cgi%3Fp%3Dgcc.git&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ts53XULDtR3o7fevlntCJdtzRqTo9R85LrxJ0ZfOBnE%3D&amp;reserved=0
nor on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgcc-mirror%2Fgcc&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2FPGF3vy3hD1OwiXmWzkUnOt9%2BR3YArZw0kCVueOKYpc%3D&amp;reserved=0
but
git fetch origin vendors/google/heads/gcc-4_8 does the trick for fetching the commits, but not the symbolic branches.

Anyways with that it looks like the discriminator changes are:

commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
Date:   Thu Oct 10 14:39:31 2013 +0000

    Use only lineno+discriminator (remove the callee function name) as the key to represent callsite. Because each callsite will have its discriminator if in the same line.

    2013-10-10  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>

            * gcc/auto-profile.c (get_function_instance_by_decl): Remove
            callee_name from callsite.
            (read_function_instance): Likewise.

    From-SVN: r203379

commit 3987da76affbfbe7195c0a16b33beedc649ec14f
Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
Date:   Tue Aug 27 16:46:49 2013 +0000

    Refactor AutoFDO to:

    1. Now that we have discriminator for inlined callsite, we do not need special handling for callsite location any more.
    2. If a source line is mapped to multiple BBs, only the first BB will be annotated.
    3. Before actual annotation, mark everythin BB/edge as not annotated.

    2013-08-27  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>

            * gcc/auto-profile.c (location_set): New data structure.
            (get_count_info): Add new parameter.
            (get_combined_location): Remove unused parameter.
            (get_inline_stack): Remove unused parameter.
            (afdo_get_bb_count): Add new parameter.
            (afdo_annotate_cfg): Reset annotated flags.
commit 1e6c4a7a8fb8e20545bb9f9032d3854f3f794c18
Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
Date:   Thu Aug 22 17:20:29 2013 +0000

    Set discriminator for call stmts within a same basic block.

    2013-08-22  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>

            * gcc/tree-cfg.c (assign_discriminators): assign discriminator for
            call stmt in a same BB if it is mapped to a same line.

commit b0be0175f4cf18fdd77bb013b181eead3a0a4773
Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
Date:   Mon Aug 19 21:26:33 2013 +0000

    Fix the discriminator assignment bug during hashing.

    2013-08-19  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>

            * tree-cfg.c (next_discriminator_for_locus): Fix discriminator
            assignment bug.

    From-SVN: r201857
commit 9fa26998a63d4b22b637ed8702520819e408a694
Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
Date:   Mon Aug 19 20:16:47 2013 +0000

    Add discrminator for inlined callsites.

    2013-08-19  Dehao Chen  (dehao@google.com<mailto:dehao@google.com>)

            * include/dwarf2.def (DW_AT_GNU_discriminator): New attribute.
            * gcc/dwarf2out.c (add_call_src_coords_attributes): Emit discriminator
            attribute for inlined callsite.

    From-SVN: r201856

-Andi

Re: State of AutoFDO in GCC

[David Edelsohn]

On Tue, Jul 26, 2022 at 4:13 PM Eugene Rozenfeld via Gcc
<gcc@gcc.gnu.org> wrote:
>
> Hello GCC community.
>
> I started this thread on the state of AutoFDO in GCC more than a year ago. Here is the first message in the thread: https://gcc.gnu.org/pipermail/gcc/2021-April/235860.html
>
> Since then I committed a number of patches to revive AutoFDO in GCC:
>
> Fix a typo in an AutoFDO error string<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=23691ddd3aa3ffe55892b2bff54f9a15a89de2b4>
> Update gen_autofdo_event.py and gcc-auto-profile.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=01d402c5e0ac1ddf5618bbe316b50067625fda46>
> Fixes for AutoFDO tests<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=f9ad3d5339faaaed6e15a7b27d90fbc66eb72f37>
> Fix indir-call-prof-2.c with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=0ed093c7c3f755bc1cd80e5186abeb2f5c50ee0c>
> Fixes for AutoFDO testing<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=9265b378531391498ec1727f67a45da72a6c07e9>
> Fix indirect call inlining with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0>
> Improve AutoFDO count propagation algorithm<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=3d9e6767939e9658260e2506e81ec32b37cba041>
> AutoFDO: don't set param_early_inliner_max_iterations to 10.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=c17975d81aaed49ff759c20c68b31304a6953d58>
> AutoFDO: Don't try to promote indirect calls that result in recursive direct calls<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ba125745d9e9fe90a18a2af8701b3269c5fdd468>
> Fix profile count maintenance in vectorizer peeling.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0>
>
> I also made a number of fixes and improvements to create_gcov tool in https://github.com/google/autofdo .
>
> AutoFDO in GCC is in a much better shape now.
>
> I have a further set of patches that improve DWARF discriminator support in GCC and enable AutoFDO to use discriminators. It's based on commits in an old Google vendor branch as described in Andi's mail below
> but uses a different approach for keeping track of per-instruction discriminators.
>
> I submitted the first (and the biggest) of these patches almost 2 months ago on June 2: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0
> but only got a review from Andi (https://gcc.gnu.org/pipermail/gcc-patches/2022-June/596549.html) who is not allowed to approve patches for commit. I pinged gcc-patches twice with no success.
>
> I would appreciate help in getting a review on this patch so that I can get it committed and submit patches that depend on it.

Hi, Eugene

Thanks for your efforts to fix and improve AutoFDO in GCC.  I believe
that part of the difficulty with obtaining a review of the patches is
that the original authors have dispersed and no one in the GCC
community officially is the maintainer for the feature.  Because you
seem to be one of the primary users and developers, would you be
interested to take on the responsibility of maintaining the
AutoFDO-specific portions of the code, with guidance and mentorship
from other GCC maintainers, especially the ones responsible for gcov
and PDO?

Thanks, David

>
> Thank you,
>
> Eugene
>
> -----Original Message-----
> From: Andi Kleen <ak@linux.intel.com>
> Sent: Monday, May 10, 2021 10:21 AM
> To: Joseph Myers <joseph@codesourcery.com>
> Cc: Jan Hubicka <hubicka@ucw.cz>; gcc <gcc@gcc.gnu.org>; Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Subject: [EXTERNAL] Re: State of AutoFDO in GCC
>
> On Mon, May 10, 2021 at 04:55:50PM +0000, Joseph Myers wrote:
> > On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
> >
> > > It's difficult to find now because it was a branch in the old SVN
> > > that wasn't converted. Sadly the great git conversion was quite lossy.
> >
> > All branches and tags, including deleted ones, were converted (under
> > not-fetched-by-default refs in some cases); the git repository has
> > everything that might plausibly be useful, omitting only a few things
> > that would have been meaningless to convert, such as mistaken branch
> > creations in the root of the repository and the SVN hooks directory.
> > Use "git ls-remote git://gcc.gnu.org/git/gcc.git" to see the full list
> > of over 5000 refs available in the repository (or do a clone with
> > --mirror to fetch them all).
>
> Okay thanks. I don't see them in any of the web interfaces, neither on
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fgit%2Fgitweb.cgi%3Fp%3Dgcc.git&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ts53XULDtR3o7fevlntCJdtzRqTo9R85LrxJ0ZfOBnE%3D&amp;reserved=0
> nor on
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgcc-mirror%2Fgcc&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2FPGF3vy3hD1OwiXmWzkUnOt9%2BR3YArZw0kCVueOKYpc%3D&amp;reserved=0
> but
> git fetch origin vendors/google/heads/gcc-4_8 does the trick for fetching the commits, but not the symbolic branches.
>
> Anyways with that it looks like the discriminator changes are:
>
> commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Thu Oct 10 14:39:31 2013 +0000
>
>     Use only lineno+discriminator (remove the callee function name) as the key to represent callsite. Because each callsite will have its discriminator if in the same line.
>
>     2013-10-10  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
>
>             * gcc/auto-profile.c (get_function_instance_by_decl): Remove
>             callee_name from callsite.
>             (read_function_instance): Likewise.
>
>     From-SVN: r203379
>
> commit 3987da76affbfbe7195c0a16b33beedc649ec14f
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Tue Aug 27 16:46:49 2013 +0000
>
>     Refactor AutoFDO to:
>
>     1. Now that we have discriminator for inlined callsite, we do not need special handling for callsite location any more.
>     2. If a source line is mapped to multiple BBs, only the first BB will be annotated.
>     3. Before actual annotation, mark everythin BB/edge as not annotated.
>
>     2013-08-27  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
>
>             * gcc/auto-profile.c (location_set): New data structure.
>             (get_count_info): Add new parameter.
>             (get_combined_location): Remove unused parameter.
>             (get_inline_stack): Remove unused parameter.
>             (afdo_get_bb_count): Add new parameter.
>             (afdo_annotate_cfg): Reset annotated flags.
> commit 1e6c4a7a8fb8e20545bb9f9032d3854f3f794c18
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Thu Aug 22 17:20:29 2013 +0000
>
>     Set discriminator for call stmts within a same basic block.
>
>     2013-08-22  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
>
>             * gcc/tree-cfg.c (assign_discriminators): assign discriminator for
>             call stmt in a same BB if it is mapped to a same line.
>
> commit b0be0175f4cf18fdd77bb013b181eead3a0a4773
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Mon Aug 19 21:26:33 2013 +0000
>
>     Fix the discriminator assignment bug during hashing.
>
>     2013-08-19  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
>
>             * tree-cfg.c (next_discriminator_for_locus): Fix discriminator
>             assignment bug.
>
>     From-SVN: r201857
> commit 9fa26998a63d4b22b637ed8702520819e408a694
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Mon Aug 19 20:16:47 2013 +0000
>
>     Add discrminator for inlined callsites.
>
>     2013-08-19  Dehao Chen  (dehao@google.com<mailto:dehao@google.com>)
>
>             * include/dwarf2.def (DW_AT_GNU_discriminator): New attribute.
>             * gcc/dwarf2out.c (add_call_src_coords_attributes): Emit discriminator
>             attribute for inlined callsite.
>
>     From-SVN: r201856
>
> -Andi
>

Re: State of AutoFDO in GCC

[Xionghu Luo]

On 2022/7/27 04:12, Eugene Rozenfeld via Gcc wrote:
> Hello GCC community.
> 
> I started this thread on the state of AutoFDO in GCC more than a year ago. Here is the first message in the thread: https://gcc.gnu.org/pipermail/gcc/2021-April/235860.html
> 
> Since then I committed a number of patches to revive AutoFDO in GCC:
> 
> Fix a typo in an AutoFDO error string<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=23691ddd3aa3ffe55892b2bff54f9a15a89de2b4>
> Update gen_autofdo_event.py and gcc-auto-profile.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=01d402c5e0ac1ddf5618bbe316b50067625fda46>
> Fixes for AutoFDO tests<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=f9ad3d5339faaaed6e15a7b27d90fbc66eb72f37>
> Fix indir-call-prof-2.c with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=0ed093c7c3f755bc1cd80e5186abeb2f5c50ee0c>
> Fixes for AutoFDO testing<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=9265b378531391498ec1727f67a45da72a6c07e9>
> Fix indirect call inlining with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0>

This "indirect call inlining with AutoFDO" patch need to be backported 
to at least gcc-10 as it will caused stack overflow for some cases I've 
met...

Xionghu

Re: State of AutoFDO in GCC

[Xionghu Luo]

On 2022/7/27 09:31, Xionghu Luo wrote:
> 
> 
> On 2022/7/27 04:12, Eugene Rozenfeld via Gcc wrote:
>> Hello GCC community.
>>
>> I started this thread on the state of AutoFDO in GCC more than a year 
>> ago. Here is the first message in the thread: 
>> https://gcc.gnu.org/pipermail/gcc/2021-April/235860.html
>>
>> Since then I committed a number of patches to revive AutoFDO in GCC:
>>
>> Fix a typo in an AutoFDO error 
>> string<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=23691ddd3aa3ffe55892b2bff54f9a15a89de2b4> 
>>
>> Update gen_autofdo_event.py and 
>> gcc-auto-profile.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=01d402c5e0ac1ddf5618bbe316b50067625fda46> 
>>
>> Fixes for AutoFDO 
>> tests<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=f9ad3d5339faaaed6e15a7b27d90fbc66eb72f37> 
>>
>> Fix indir-call-prof-2.c with 
>> AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=0ed093c7c3f755bc1cd80e5186abeb2f5c50ee0c> 
>>
>> Fixes for AutoFDO 
>> testing<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=9265b378531391498ec1727f67a45da72a6c07e9> 
>>
>> Fix indirect call inlining with 
>> AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0> 
>>
> 
> This "indirect call inlining with AutoFDO" patch need to be backported 
> to at least gcc-10 as it will caused stack overflow for some cases I've 
> met...


Sorry, should be this:
  AutoFDO: Don't try to promote indirect calls that result in recursive 
direct 
calls<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ba125745d9e9fe90a18a2af8701b3269c5fdd468>,

Though the "indirect call inlining with AutoFDO" also need verify on old
branches.  indirect call format is updated with Martin's dynamic topn patch.

> 
> Xionghu

Re: State of AutoFDO in GCC

[Jan Hubicka]

> On Tue, Jul 26, 2022 at 4:13 PM Eugene Rozenfeld via Gcc
> <gcc@gcc.gnu.org> wrote:
> >
> > Hello GCC community.
> >
> > I started this thread on the state of AutoFDO in GCC more than a year ago. Here is the first message in the thread: https://gcc.gnu.org/pipermail/gcc/2021-April/235860.html
> >
> > Since then I committed a number of patches to revive AutoFDO in GCC:
> >
> > Fix a typo in an AutoFDO error string<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=23691ddd3aa3ffe55892b2bff54f9a15a89de2b4>
> > Update gen_autofdo_event.py and gcc-auto-profile.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=01d402c5e0ac1ddf5618bbe316b50067625fda46>
> > Fixes for AutoFDO tests<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=f9ad3d5339faaaed6e15a7b27d90fbc66eb72f37>
> > Fix indir-call-prof-2.c with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=0ed093c7c3f755bc1cd80e5186abeb2f5c50ee0c>
> > Fixes for AutoFDO testing<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=9265b378531391498ec1727f67a45da72a6c07e9>
> > Fix indirect call inlining with AutoFDO<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0>
> > Improve AutoFDO count propagation algorithm<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=3d9e6767939e9658260e2506e81ec32b37cba041>
> > AutoFDO: don't set param_early_inliner_max_iterations to 10.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=c17975d81aaed49ff759c20c68b31304a6953d58>
> > AutoFDO: Don't try to promote indirect calls that result in recursive direct calls<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ba125745d9e9fe90a18a2af8701b3269c5fdd468>
> > Fix profile count maintenance in vectorizer peeling.<https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0>
> >
> > I also made a number of fixes and improvements to create_gcov tool in https://github.com/google/autofdo .
> >
> > AutoFDO in GCC is in a much better shape now.
> >
> > I have a further set of patches that improve DWARF discriminator support in GCC and enable AutoFDO to use discriminators. It's based on commits in an old Google vendor branch as described in Andi's mail below
> > but uses a different approach for keeping track of per-instruction discriminators.
> >
> > I submitted the first (and the biggest) of these patches almost 2 months ago on June 2: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=5af22024f62f1f596a35d3c138d41d47d5697ca0
> > but only got a review from Andi (https://gcc.gnu.org/pipermail/gcc-patches/2022-June/596549.html) who is not allowed to approve patches for commit. I pinged gcc-patches twice with no success.
> >
> > I would appreciate help in getting a review on this patch so that I can get it committed and submit patches that depend on it.
> 
> Hi, Eugene
> 
> Thanks for your efforts to fix and improve AutoFDO in GCC.  I believe
> that part of the difficulty with obtaining a review of the patches is
> that the original authors have dispersed and no one in the GCC
> community officially is the maintainer for the feature.  Because you
> seem to be one of the primary users and developers, would you be
> interested to take on the responsibility of maintaining the
> AutoFDO-specific portions of the code, with guidance and mentorship
> from other GCC maintainers, especially the ones responsible for gcov
> and PDO?

I missed the patches (it would help to add me to CC :) and will review
the FDO/profile facing parts.  Since it also extends debug info
generation and front-ends I think we also need reviewer for that part.

Having auto-FDO co-maintainer would be welcome.

Honza
> 
> Thanks, David
> 
> >
> > Thank you,
> >
> > Eugene
> >
> > -----Original Message-----
> > From: Andi Kleen <ak@linux.intel.com>
> > Sent: Monday, May 10, 2021 10:21 AM
> > To: Joseph Myers <joseph@codesourcery.com>
> > Cc: Jan Hubicka <hubicka@ucw.cz>; gcc <gcc@gcc.gnu.org>; Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > On Mon, May 10, 2021 at 04:55:50PM +0000, Joseph Myers wrote:
> > > On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
> > >
> > > > It's difficult to find now because it was a branch in the old SVN
> > > > that wasn't converted. Sadly the great git conversion was quite lossy.
> > >
> > > All branches and tags, including deleted ones, were converted (under
> > > not-fetched-by-default refs in some cases); the git repository has
> > > everything that might plausibly be useful, omitting only a few things
> > > that would have been meaningless to convert, such as mistaken branch
> > > creations in the root of the repository and the SVN hooks directory.
> > > Use "git ls-remote git://gcc.gnu.org/git/gcc.git" to see the full list
> > > of over 5000 refs available in the repository (or do a clone with
> > > --mirror to fetch them all).
> >
> > Okay thanks. I don't see them in any of the web interfaces, neither on
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fgit%2Fgitweb.cgi%3Fp%3Dgcc.git&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ts53XULDtR3o7fevlntCJdtzRqTo9R85LrxJ0ZfOBnE%3D&amp;reserved=0
> > nor on
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgcc-mirror%2Fgcc&amp;data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C9d79b87018f24bcbf8cc08d913d80bd0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637562640903545786%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=%2FPGF3vy3hD1OwiXmWzkUnOt9%2BR3YArZw0kCVueOKYpc%3D&amp;reserved=0
> > but
> > git fetch origin vendors/google/heads/gcc-4_8 does the trick for fetching the commits, but not the symbolic branches.
> >
> > Anyways with that it looks like the discriminator changes are:
> >
> > commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Thu Oct 10 14:39:31 2013 +0000
> >
> >     Use only lineno+discriminator (remove the callee function name) as the key to represent callsite. Because each callsite will have its discriminator if in the same line.
> >
> >     2013-10-10  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * gcc/auto-profile.c (get_function_instance_by_decl): Remove
> >             callee_name from callsite.
> >             (read_function_instance): Likewise.
> >
> >     From-SVN: r203379
> >
> > commit 3987da76affbfbe7195c0a16b33beedc649ec14f
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Tue Aug 27 16:46:49 2013 +0000
> >
> >     Refactor AutoFDO to:
> >
> >     1. Now that we have discriminator for inlined callsite, we do not need special handling for callsite location any more.
> >     2. If a source line is mapped to multiple BBs, only the first BB will be annotated.
> >     3. Before actual annotation, mark everythin BB/edge as not annotated.
> >
> >     2013-08-27  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * gcc/auto-profile.c (location_set): New data structure.
> >             (get_count_info): Add new parameter.
> >             (get_combined_location): Remove unused parameter.
> >             (get_inline_stack): Remove unused parameter.
> >             (afdo_get_bb_count): Add new parameter.
> >             (afdo_annotate_cfg): Reset annotated flags.
> > commit 1e6c4a7a8fb8e20545bb9f9032d3854f3f794c18
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Thu Aug 22 17:20:29 2013 +0000
> >
> >     Set discriminator for call stmts within a same basic block.
> >
> >     2013-08-22  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * gcc/tree-cfg.c (assign_discriminators): assign discriminator for
> >             call stmt in a same BB if it is mapped to a same line.
> >
> > commit b0be0175f4cf18fdd77bb013b181eead3a0a4773
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Mon Aug 19 21:26:33 2013 +0000
> >
> >     Fix the discriminator assignment bug during hashing.
> >
> >     2013-08-19  Dehao Chen  <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * tree-cfg.c (next_discriminator_for_locus): Fix discriminator
> >             assignment bug.
> >
> >     From-SVN: r201857
> > commit 9fa26998a63d4b22b637ed8702520819e408a694
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Mon Aug 19 20:16:47 2013 +0000
> >
> >     Add discrminator for inlined callsites.
> >
> >     2013-08-19  Dehao Chen  (dehao@google.com<mailto:dehao@google.com>)
> >
> >             * include/dwarf2.def (DW_AT_GNU_discriminator): New attribute.
> >             * gcc/dwarf2out.c (add_call_src_coords_attributes): Emit discriminator
> >             attribute for inlined callsite.
> >
> >     From-SVN: r201856
> >
> > -Andi
> >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Hi David,

Thank you for your offer to take on the responsibility of maintaining the AutoFDO-specific portions of the code. I'll be happy to do that with Jan's and other GGC maintainers' help.
I hope more people will start using and contributing to AutoFDO.

Thanks,

Eugene

-----Original Message-----
From: David Edelsohn <dje.gcc@gmail.com> 
Sent: Tuesday, July 26, 2022 3:38 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>; Jan Hubicka <hubicka@ucw.cz>; Martin Liska <mliska@suse.cz>; Xinliang David Li <davidxl@google.com>
Cc: Andi Kleen <ak@linux.intel.com>; Joseph Myers <joseph@codesourcery.com>; gcc <gcc@gcc.gnu.org>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

[You don't often get email from dje.gcc@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

On Tue, Jul 26, 2022 at 4:13 PM Eugene Rozenfeld via Gcc <gcc@gcc.gnu.org> wrote:
>
> Hello GCC community.
>
> I started this thread on the state of AutoFDO in GCC more than a year 
> ago. Here is the first message in the thread: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fpipermail%2Fgcc%2F2021-April%2F235860.html&amp;data=05%7C01%
> 7CEugene.Rozenfeld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944718882043585%7CUnkn
> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwi
> LCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=5RoiLypTlkNRIZW8yAufYR4qiO573
> 0PADO%2BFsS6InIU%3D&amp;reserved=0
>
> Since then I committed a number of patches to revive AutoFDO in GCC:
>
> Fix a typo in an AutoFDO error 
> string<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D23691ddd3aa3f
> fe55892b2bff54f9a15a89de2b4&amp;data=05%7C01%7CEugene.Rozenfeld%40micr
> osoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> 7C%7C&amp;sdata=LRgW4qyh%2FAdrlKluUXnaDTFCNW8tnS1WX8bCs1WAT7s%3D&amp;r
> eserved=0> Update gen_autofdo_event.py and 
> gcc-auto-profile.<https://nam06.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D01
> d402c5e0ac1ddf5618bbe316b50067625fda46&amp;data=05%7C01%7CEugene.Rozen
> feld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f1
> 41af91ab2d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C&amp;sdata=oBJvTW6RrkY6uURmr0UK5gaNiVS1b8vFfIyOoqq8AkM
> %3D&amp;reserved=0> Fixes for AutoFDO 
> tests<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> 2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Df9ad3d5339faaa
> ed6e15a7b27d90fbc66eb72f37&amp;data=05%7C01%7CEugene.Rozenfeld%40micro
> soft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> C%7C&amp;sdata=6evlP10%2BQJyUuVZ3u%2Fv%2FP9nSIsASLjtWETyqeBfnQ2k%3D&am
> p;reserved=0> Fix indir-call-prof-2.c with 
> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D0ed093c7c3f7
> 55bc1cd80e5186abeb2f5c50ee0c&amp;data=05%7C01%7CEugene.Rozenfeld%40mic
> rosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=mtMd21By6r8DiIIXzk5ePHP0iU%2FHfnDQG8%2FXJbAr5qE%3D&am
> p;reserved=0> Fixes for AutoFDO 
> testing<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D9265b3785313
> 91498ec1727f67a45da72a6c07e9&amp;data=05%7C01%7CEugene.Rozenfeld%40mic
> rosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=QnKrxt7W9wfTYY3Drjm%2FSn7D4yHwxjInvI8dj9KEIs4%3D&amp;
> reserved=0> Fix indirect call inlining with 
> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D285aa6895d47
> 9bed8e72ad363290846645b6faa0&amp;data=05%7C01%7CEugene.Rozenfeld%40mic
> rosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=695ieVftlFRwICzxxMmmIb9%2F%2FDBlYBy9jJn%2ByfzkhP0%3D&
> amp;reserved=0> Improve AutoFDO count propagation 
> algorithm<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D3d9e676793
> 9e9658260e2506e81ec32b37cba041&amp;data=05%7C01%7CEugene.Rozenfeld%40m
> icrosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJW
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%
> 7C%7C%7C&amp;sdata=IXpOMFrH%2FSsrI7f1WbEJQHtl4Nt1ClVYzvM%2FCbR%2FDm0%3
> D&amp;reserved=0>
> AutoFDO: don't set param_early_inliner_max_iterations to 
> 10.<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> gcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Dc17975d81aaed49f
> f759c20c68b31304a6953d58&amp;data=05%7C01%7CEugene.Rozenfeld%40microso
> ft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
> 7C&amp;sdata=%2FM2iGyPxGCi8%2FnhqMQ4eUafsNyWy1IrNABa2TgVdBdQ%3D&amp;re
> served=0>
> AutoFDO: Don't try to promote indirect calls that result in recursive 
> direct 
> calls<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> 2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Dba125745d9e9fe
> 90a18a2af8701b3269c5fdd468&amp;data=05%7C01%7CEugene.Rozenfeld%40micro
> soft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> C%7C&amp;sdata=%2BMV7YxH617Nf6PjEF8Z3ImdEkvBptM9y47fp1ja4wVY%3D&amp;re
> served=0> Fix profile count maintenance in vectorizer 
> peeling.<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
> 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D5af22024f62
> f1f596a35d3c138d41d47d5697ca0&amp;data=05%7C01%7CEugene.Rozenfeld%40mi
> crosoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWI
> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
> C%7C%7C&amp;sdata=aiXW3IyLauhHOvpsPcqDXwg9VT3nn7o1Zh1KPKLxlDo%3D&amp;r
> eserved=0>
>
> I also made a number of fixes and improvements to create_gcov tool in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=05%7C01%7CEugene.Rozenfeld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=zGQgQdD%2F6a9tk8EcaFLcQ1DiX5nGgzIjWUrWqzEIjk4%3D&amp;reserved=0 .
>
> AutoFDO in GCC is in a much better shape now.
>
> I have a further set of patches that improve DWARF discriminator 
> support in GCC and enable AutoFDO to use discriminators. It's based on commits in an old Google vendor branch as described in Andi's mail below but uses a different approach for keeping track of per-instruction discriminators.
>
> I submitted the first (and the biggest) of these patches almost 2 
> months ago on June 2: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D5af22024f62f1f596a35
> d3c138d41d47d5697ca0&amp;data=05%7C01%7CEugene.Rozenfeld%40microsoft.c
> om%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7cd011db
> 47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&a
> mp;sdata=aiXW3IyLauhHOvpsPcqDXwg9VT3nn7o1Zh1KPKLxlDo%3D&amp;reserved=0
> but only got a review from Andi (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fpipermail%2Fgcc-patches%2F2022-June%2F596549.html&amp;data=05%7C01%7CEugene.Rozenfeld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=TC0cBmoLsDJ3rx9paqvfFFTR9mSVuW%2Fb%2Bu4Y8jFyf%2FE%3D&amp;reserved=0) who is not allowed to approve patches for commit. I pinged gcc-patches twice with no success.
>
> I would appreciate help in getting a review on this patch so that I can get it committed and submit patches that depend on it.

Hi, Eugene

Thanks for your efforts to fix and improve AutoFDO in GCC.  I believe that part of the difficulty with obtaining a review of the patches is that the original authors have dispersed and no one in the GCC community officially is the maintainer for the feature.  Because you seem to be one of the primary users and developers, would you be interested to take on the responsibility of maintaining the AutoFDO-specific portions of the code, with guidance and mentorship from other GCC maintainers, especially the ones responsible for gcov and PDO?

Thanks, David

>
> Thank you,
>
> Eugene
>
> -----Original Message-----
> From: Andi Kleen <ak@linux.intel.com>
> Sent: Monday, May 10, 2021 10:21 AM
> To: Joseph Myers <joseph@codesourcery.com>
> Cc: Jan Hubicka <hubicka@ucw.cz>; gcc <gcc@gcc.gnu.org>; Eugene 
> Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Subject: [EXTERNAL] Re: State of AutoFDO in GCC
>
> On Mon, May 10, 2021 at 04:55:50PM +0000, Joseph Myers wrote:
> > On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
> >
> > > It's difficult to find now because it was a branch in the old SVN 
> > > that wasn't converted. Sadly the great git conversion was quite lossy.
> >
> > All branches and tags, including deleted ones, were converted (under 
> > not-fetched-by-default refs in some cases); the git repository has 
> > everything that might plausibly be useful, omitting only a few 
> > things that would have been meaningless to convert, such as mistaken 
> > branch creations in the root of the repository and the SVN hooks directory.
> > Use "git ls-remote git://gcc.gnu.org/git/gcc.git" to see the full 
> > list of over 5000 refs available in the repository (or do a clone 
> > with --mirror to fetch them all).
>
> Okay thanks. I don't see them in any of the web interfaces, neither on
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.
> gnu.org%2Fgit%2Fgitweb.cgi%3Fp%3Dgcc.git&amp;data=05%7C01%7CEugene.Roz
> enfeld%40microsoft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86
> f141af91ab2d7cd011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbG
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C3000%7C%7C%7C&amp;sdata=spoRRd6x520PaarYjzVmTcmjhIXTtVZWw%2Bh0LJc
> jshE%3D&amp;reserved=0
> nor on
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fgcc-mirror%2Fgcc&amp;data=05%7C01%7CEugene.Rozenfeld%40micros
> oft.com%7Ccabaec504f234632a03a08da6f5780d2%7C72f988bf86f141af91ab2d7cd
> 011db47%7C1%7C0%7C637944718882199821%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
> C4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> %7C&amp;sdata=442RFALNd91XWlWDIo4482Q7YUuJBBlETZwE1WZDZP8%3D&amp;reser
> ved=0
> but
> git fetch origin vendors/google/heads/gcc-4_8 does the trick for fetching the commits, but not the symbolic branches.
>
> Anyways with that it looks like the discriminator changes are:
>
> commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Thu Oct 10 14:39:31 2013 +0000
>
>     Use only lineno+discriminator (remove the callee function name) as the key to represent callsite. Because each callsite will have its discriminator if in the same line.
>
>     2013-10-10  Dehao Chen  
> <dehao@google.com<mailto:dehao@google.com>>
>
>             * gcc/auto-profile.c (get_function_instance_by_decl): Remove
>             callee_name from callsite.
>             (read_function_instance): Likewise.
>
>     From-SVN: r203379
>
> commit 3987da76affbfbe7195c0a16b33beedc649ec14f
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Tue Aug 27 16:46:49 2013 +0000
>
>     Refactor AutoFDO to:
>
>     1. Now that we have discriminator for inlined callsite, we do not need special handling for callsite location any more.
>     2. If a source line is mapped to multiple BBs, only the first BB will be annotated.
>     3. Before actual annotation, mark everythin BB/edge as not annotated.
>
>     2013-08-27  Dehao Chen  
> <dehao@google.com<mailto:dehao@google.com>>
>
>             * gcc/auto-profile.c (location_set): New data structure.
>             (get_count_info): Add new parameter.
>             (get_combined_location): Remove unused parameter.
>             (get_inline_stack): Remove unused parameter.
>             (afdo_get_bb_count): Add new parameter.
>             (afdo_annotate_cfg): Reset annotated flags.
> commit 1e6c4a7a8fb8e20545bb9f9032d3854f3f794c18
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Thu Aug 22 17:20:29 2013 +0000
>
>     Set discriminator for call stmts within a same basic block.
>
>     2013-08-22  Dehao Chen  
> <dehao@google.com<mailto:dehao@google.com>>
>
>             * gcc/tree-cfg.c (assign_discriminators): assign discriminator for
>             call stmt in a same BB if it is mapped to a same line.
>
> commit b0be0175f4cf18fdd77bb013b181eead3a0a4773
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Mon Aug 19 21:26:33 2013 +0000
>
>     Fix the discriminator assignment bug during hashing.
>
>     2013-08-19  Dehao Chen  
> <dehao@google.com<mailto:dehao@google.com>>
>
>             * tree-cfg.c (next_discriminator_for_locus): Fix discriminator
>             assignment bug.
>
>     From-SVN: r201857
> commit 9fa26998a63d4b22b637ed8702520819e408a694
> Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> Date:   Mon Aug 19 20:16:47 2013 +0000
>
>     Add discrminator for inlined callsites.
>
>     2013-08-19  Dehao Chen  
> (dehao@google.com<mailto:dehao@google.com>)
>
>             * include/dwarf2.def (DW_AT_GNU_discriminator): New attribute.
>             * gcc/dwarf2out.c (add_call_src_coords_attributes): Emit discriminator
>             attribute for inlined callsite.
>
>     From-SVN: r201856
>
> -Andi
>

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Hi Jan,

Thank you for your reply. I did have you on the TO line in my latest patch:
https://gcc.gnu.org/pipermail/gcc-patches/2022-June/596065.html

That's the patch I need a review on.

I'm looking forward to co-maintaining AutoFDO with you.

Thanks,

Eugene

-----Original Message-----
From: Jan Hubicka <hubicka@ucw.cz> 
Sent: Wednesday, July 27, 2022 12:27 AM
To: David Edelsohn <dje.gcc@gmail.com>
Cc: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>; Martin Liska <mliska@suse.cz>; Xinliang David Li <davidxl@google.com>; gcc <gcc@gcc.gnu.org>; Andi Kleen <ak@linux.intel.com>; Joseph Myers <joseph@codesourcery.com>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

> On Tue, Jul 26, 2022 at 4:13 PM Eugene Rozenfeld via Gcc 
> <gcc@gcc.gnu.org> wrote:
> >
> > Hello GCC community.
> >
> > I started this thread on the state of AutoFDO in GCC more than a 
> > year ago. Here is the first message in the thread: 
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgc
> > c.gnu.org%2Fpipermail%2Fgcc%2F2021-April%2F235860.html&amp;data=05%7
> > C01%7CEugene.Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa
> > 1619e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63794503618637077
> > 0%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi
> > I6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=9%2Bnv2ShWxKh88K%
> > 2BsOeqPgQX3lOCJQ0lnF%2F7SUs4K4uI%3D&amp;reserved=0
> >
> > Since then I committed a number of patches to revive AutoFDO in GCC:
> >
> > Fix a typo in an AutoFDO error 
> > string<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
> > 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D23691ddd3
> > aa3ffe55892b2bff54f9a15a89de2b4&amp;data=05%7C01%7CEugene.Rozenfeld%
> > 40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141a
> > f91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZsb
> > 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> > D%7C3000%7C%7C%7C&amp;sdata=qkecfE9uH5gy91vILQQlCk9RpExqPZxO4q02wiN1
> > EFw%3D&amp;reserved=0> Update gen_autofdo_event.py and 
> > gcc-auto-profile.<https://nam06.safelinks.protection.outlook.com/?ur
> > l=https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%
> > 3D01d402c5e0ac1ddf5618bbe316b50067625fda46&amp;data=05%7C01%7CEugene
> > .Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f9
> > 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%
> > 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=E52qVFlfdfFGnW9yDsBNhh4k2ey8g
> > 3aJEGzH40MuSOc%3D&amp;reserved=0> Fixes for AutoFDO 
> > tests<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Df9ad3d5339
> > faaaed6e15a7b27d90fbc66eb72f37&amp;data=05%7C01%7CEugene.Rozenfeld%4
> > 0microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af
> > 91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZsb3
> > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C3000%7C%7C%7C&amp;sdata=XYlFoY3OTTXHp18O1v8BY47A17NyNPXvUWWYsVnbD
> > 0U%3D&amp;reserved=0> Fix indir-call-prof-2.c with 
> > AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> > %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D0ed093c7
> > c3f755bc1cd80e5186abeb2f5c50ee0c&amp;data=05%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141
> > af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > 3D%7C3000%7C%7C%7C&amp;sdata=sLftY6hjvzSuE9ZgkGmXZLDpRMjlDo%2FEAyDyP
> > CviY5Q%3D&amp;reserved=0> Fixes for AutoFDO 
> > testing<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> > %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D9265b378
> > 531391498ec1727f67a45da72a6c07e9&amp;data=05%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141
> > af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > 3D%7C3000%7C%7C%7C&amp;sdata=lXZ%2F%2FbcfYD%2BQyIiXMAaCxOujEAfDXSY1p
> > 78kUb2md7w%3D&amp;reserved=0> Fix indirect call inlining with 
> > AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A
> > %2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D285aa689
> > 5d479bed8e72ad363290846645b6faa0&amp;data=05%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141
> > af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > 3D%7C3000%7C%7C%7C&amp;sdata=ypoF%2BZnEe3eC3Gat6%2FjUJLV2XiltdjJHe68
> > pue64fSU%3D&amp;reserved=0> Improve AutoFDO count propagation 
> > algorithm<https://nam06.safelinks.protection.outlook.com/?url=https%
> > 3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D3d9e67
> > 67939e9658260e2506e81ec32b37cba041&amp;data=05%7C01%7CEugene.Rozenfe
> > ld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f1
> > 41af91ab2d7cd011db47%7C1%7C0%7C637945036186370770%7CUnknown%7CTWFpbG
> > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > 0%3D%7C3000%7C%7C%7C&amp;sdata=IyKR391AJksu3p85Z5q5xS%2FT7i0IC%2B9oX
> > Y3VKE49s58%3D&amp;reserved=0>
> > AutoFDO: don't set param_early_inliner_max_iterations to 
> > 10.<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> > 2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Dc17975d81aae
> > d49ff759c20c68b31304a6953d58&amp;data=05%7C01%7CEugene.Rozenfeld%40m
> > icrosoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af91
> > ab2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZsb3d8
> > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> > C3000%7C%7C%7C&amp;sdata=d%2Fj8IyRYVjCA1grtov7PNzhp4eRszLEflY21kKCN8
> > eo%3D&amp;reserved=0>
> > AutoFDO: Don't try to promote indirect calls that result in 
> > recursive direct 
> > calls<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> > F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Dba125745d9
> > e9fe90a18a2af8701b3269c5fdd468&amp;data=05%7C01%7CEugene.Rozenfeld%4
> > 0microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af
> > 91ab2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZsb3
> > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C3000%7C%7C%7C&amp;sdata=L4bypORscLpmmhxnaAupzTWepno447nbMcPUGiah2
> > b0%3D&amp;reserved=0> Fix profile count maintenance in vectorizer 
> > peeling.<https://nam06.safelinks.protection.outlook.com/?url=https%3
> > A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D5af2202
> > 4f62f1f596a35d3c138d41d47d5697ca0&amp;data=05%7C01%7CEugene.Rozenfel
> > d%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f14
> > 1af91ab2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZ
> > sb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> > %3D%7C3000%7C%7C%7C&amp;sdata=LTaHWhkpp2wH%2FaqoJZQ%2Bvcj5o%2FKZ4kQP
> > OstxpsLuOKI%3D&amp;reserved=0>
> >
> > I also made a number of fixes and improvements to create_gcov tool in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&amp;data=05%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=6i14y0lx0m%2FSaN2jw3DKykIun5XiKj17x5%2BBL%2BGkBd4%3D&amp;reserved=0 .
> >
> > AutoFDO in GCC is in a much better shape now.
> >
> > I have a further set of patches that improve DWARF discriminator 
> > support in GCC and enable AutoFDO to use discriminators. It's based on commits in an old Google vendor branch as described in Andi's mail below but uses a different approach for keeping track of per-instruction discriminators.
> >
> > I submitted the first (and the biggest) of these patches almost 2 
> > months ago on June 2: 
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgc
> > c.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D5af22024f62f1f59
> > 6a35d3c138d41d47d5697ca0&amp;data=05%7C01%7CEugene.Rozenfeld%40micro
> > soft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af91ab2d
> > 7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZsb3d8eyJW
> > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300
> > 0%7C%7C%7C&amp;sdata=LTaHWhkpp2wH%2FaqoJZQ%2Bvcj5o%2FKZ4kQPOstxpsLuO
> > KI%3D&amp;reserved=0 but only got a review from Andi 
> > (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fpipermail%2Fgcc-patches%2F2022-June%2F596549.html&amp;data=05%7C01%7CEugene.Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=G2578gq1F7h5s8ussOZVZtERXNKkgQD%2FJOvJ0KB7ct8%3D&amp;reserved=0) who is not allowed to approve patches for commit. I pinged gcc-patches twice with no success.
> >
> > I would appreciate help in getting a review on this patch so that I can get it committed and submit patches that depend on it.
> 
> Hi, Eugene
> 
> Thanks for your efforts to fix and improve AutoFDO in GCC.  I believe 
> that part of the difficulty with obtaining a review of the patches is 
> that the original authors have dispersed and no one in the GCC 
> community officially is the maintainer for the feature.  Because you 
> seem to be one of the primary users and developers, would you be 
> interested to take on the responsibility of maintaining the 
> AutoFDO-specific portions of the code, with guidance and mentorship 
> from other GCC maintainers, especially the ones responsible for gcov 
> and PDO?

I missed the patches (it would help to add me to CC :) and will review the FDO/profile facing parts.  Since it also extends debug info generation and front-ends I think we also need reviewer for that part.

Having auto-FDO co-maintainer would be welcome.

Honza
> 
> Thanks, David
> 
> >
> > Thank you,
> >
> > Eugene
> >
> > -----Original Message-----
> > From: Andi Kleen <ak@linux.intel.com>
> > Sent: Monday, May 10, 2021 10:21 AM
> > To: Joseph Myers <joseph@codesourcery.com>
> > Cc: Jan Hubicka <hubicka@ucw.cz>; gcc <gcc@gcc.gnu.org>; Eugene 
> > Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > On Mon, May 10, 2021 at 04:55:50PM +0000, Joseph Myers wrote:
> > > On Mon, 10 May 2021, Andi Kleen via Gcc wrote:
> > >
> > > > It's difficult to find now because it was a branch in the old 
> > > > SVN that wasn't converted. Sadly the great git conversion was quite lossy.
> > >
> > > All branches and tags, including deleted ones, were converted 
> > > (under not-fetched-by-default refs in some cases); the git 
> > > repository has everything that might plausibly be useful, omitting 
> > > only a few things that would have been meaningless to convert, 
> > > such as mistaken branch creations in the root of the repository and the SVN hooks directory.
> > > Use "git ls-remote git://gcc.gnu.org/git/gcc.git" to see the full 
> > > list of over 5000 refs available in the repository (or do a clone 
> > > with --mirror to fetch them all).
> >
> > Okay thanks. I don't see them in any of the web interfaces, neither 
> > on
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgc
> > c.gnu.org%2Fgit%2Fgitweb.cgi%3Fp%3Dgcc.git&amp;data=05%7C01%7CEugene
> > .Rozenfeld%40microsoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f9
> > 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%
> > 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> > JXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=BC%2BdUNgy2eSrnK7dWA%2B1hj1my
> > OhW7O94Mdcywbx1AI8%3D&amp;reserved=0
> > nor on
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > thub.com%2Fgcc-mirror%2Fgcc&amp;data=05%7C01%7CEugene.Rozenfeld%40mi
> > crosoft.com%7Cfe5184091a18487fd92d08da6fa1619e%7C72f988bf86f141af91a
> > b2d7cd011db47%7C1%7C0%7C637945036186683243%7CUnknown%7CTWFpbGZsb3d8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> > 3000%7C%7C%7C&amp;sdata=TcRqYfDpYTTRgh7y%2FxRmuAjLDrLDUMLwrnJsklD2mZ
> > I%3D&amp;reserved=0
> > but
> > git fetch origin vendors/google/heads/gcc-4_8 does the trick for fetching the commits, but not the symbolic branches.
> >
> > Anyways with that it looks like the discriminator changes are:
> >
> > commit fd9de90d750e3588b1e5a218b28102b6c8bb8434
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Thu Oct 10 14:39:31 2013 +0000
> >
> >     Use only lineno+discriminator (remove the callee function name) as the key to represent callsite. Because each callsite will have its discriminator if in the same line.
> >
> >     2013-10-10  Dehao Chen  
> > <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * gcc/auto-profile.c (get_function_instance_by_decl): Remove
> >             callee_name from callsite.
> >             (read_function_instance): Likewise.
> >
> >     From-SVN: r203379
> >
> > commit 3987da76affbfbe7195c0a16b33beedc649ec14f
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Tue Aug 27 16:46:49 2013 +0000
> >
> >     Refactor AutoFDO to:
> >
> >     1. Now that we have discriminator for inlined callsite, we do not need special handling for callsite location any more.
> >     2. If a source line is mapped to multiple BBs, only the first BB will be annotated.
> >     3. Before actual annotation, mark everythin BB/edge as not annotated.
> >
> >     2013-08-27  Dehao Chen  
> > <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * gcc/auto-profile.c (location_set): New data structure.
> >             (get_count_info): Add new parameter.
> >             (get_combined_location): Remove unused parameter.
> >             (get_inline_stack): Remove unused parameter.
> >             (afdo_get_bb_count): Add new parameter.
> >             (afdo_annotate_cfg): Reset annotated flags.
> > commit 1e6c4a7a8fb8e20545bb9f9032d3854f3f794c18
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Thu Aug 22 17:20:29 2013 +0000
> >
> >     Set discriminator for call stmts within a same basic block.
> >
> >     2013-08-22  Dehao Chen  
> > <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * gcc/tree-cfg.c (assign_discriminators): assign discriminator for
> >             call stmt in a same BB if it is mapped to a same line.
> >
> > commit b0be0175f4cf18fdd77bb013b181eead3a0a4773
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Mon Aug 19 21:26:33 2013 +0000
> >
> >     Fix the discriminator assignment bug during hashing.
> >
> >     2013-08-19  Dehao Chen  
> > <dehao@google.com<mailto:dehao@google.com>>
> >
> >             * tree-cfg.c (next_discriminator_for_locus): Fix discriminator
> >             assignment bug.
> >
> >     From-SVN: r201857
> > commit 9fa26998a63d4b22b637ed8702520819e408a694
> > Author: Dehao Chen <dehao@gcc.gnu.org<mailto:dehao@gcc.gnu.org>>
> > Date:   Mon Aug 19 20:16:47 2013 +0000
> >
> >     Add discrminator for inlined callsites.
> >
> >     2013-08-19  Dehao Chen  
> > (dehao@google.com<mailto:dehao@google.com>)
> >
> >             * include/dwarf2.def (DW_AT_GNU_discriminator): New attribute.
> >             * gcc/dwarf2out.c (add_call_src_coords_attributes): Emit discriminator
> >             attribute for inlined callsite.
> >
> >     From-SVN: r201856
> >
> > -Andi
> >

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld]

Hi Xionghu,

Yes, I'm pretty sure both of the fixes you mentioned are applicable to GCC 10. I'm not sure what the bar is for backporting fixes.
Jan, can you comment on that?
The fixes that Xionghu mentioned are:

	Fix indirect call inlining with AutoFDO
	https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=285aa6895d479bed8e72ad363290846645b6faa0
	AutoFDO: Don't try to promote indirect calls that result in recursive direct calls
	https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ba125745d9e9fe90a18a2af8701b3269c5fdd468

Thanks,

Eugene

-----Original Message-----
From: Xionghu Luo <yinyuefengyi@gmail.com> 
Sent: Tuesday, July 26, 2022 6:41 PM
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>; Andi Kleen <ak@linux.intel.com>; Joseph Myers <joseph@codesourcery.com>; Jan Hubicka <hubicka@ucw.cz>; gcc <gcc@gcc.gnu.org>
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

[You don't often get email from yinyuefengyi@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

On 2022/7/27 09:31, Xionghu Luo wrote:
>
>
> On 2022/7/27 04:12, Eugene Rozenfeld via Gcc wrote:
>> Hello GCC community.
>>
>> I started this thread on the state of AutoFDO in GCC more than a year 
>> ago. Here is the first message in the thread:
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc
>> .gnu.org%2Fpipermail%2Fgcc%2F2021-April%2F235860.html&amp;data=05%7C0
>> 1%7CEugene.Rozenfeld%40microsoft.com%7Ccda1f6374c584732950e08da6f7126
>> 83%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944829019804556%7C
>> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=t%2B94vOYKb8SDZ9i86Fnin
>> iYXt7LTGXqbrbftH1TVqEs%3D&amp;reserved=0
>>
>> Since then I committed a number of patches to revive AutoFDO in GCC:
>>
>> Fix a typo in an AutoFDO error
>> string<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
>> F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D23691ddd3aa
>> 3ffe55892b2bff54f9a15a89de2b4&amp;data=05%7C01%7CEugene.Rozenfeld%40m
>> icrosoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91a
>> b2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8ey
>> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C30
>> 00%7C%7C%7C&amp;sdata=4WD%2FgKxeXEFz7V6pcqDsYxvAbSOP3ldxXHb%2FoMxV5kc
>> %3D&amp;reserved=0>
>>
>> Update gen_autofdo_event.py and
>> gcc-auto-profile.<https://nam06.safelinks.protection.outlook.com/?url
>> =https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D
>> 01d402c5e0ac1ddf5618bbe316b50067625fda46&amp;data=05%7C01%7CEugene.Ro
>> zenfeld%40microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf
>> 86f141af91ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWF
>> pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
>> Mn0%3D%7C3000%7C%7C%7C&amp;sdata=TA%2FhV%2FKsRF3MUZmUL%2FpTWbESdx03VK
>> PIudXrrBjAMvQ%3D&amp;reserved=0>
>>
>> Fixes for AutoFDO
>> tests<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
>> %2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Df9ad3d5339fa
>> aaed6e15a7b27d90fbc66eb72f37&amp;data=05%7C01%7CEugene.Rozenfeld%40mi
>> crosoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91ab
>> 2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8eyJ
>> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300
>> 0%7C%7C%7C&amp;sdata=AvWCTytUxlcXq%2BpjkWJmhkmV0nH%2Fn0CzC4alU9XA9%2F
>> 4%3D&amp;reserved=0>
>>
>> Fix indir-call-prof-2.c with
>> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
>> 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D0ed093c7c3
>> f755bc1cd80e5186abeb2f5c50ee0c&amp;data=05%7C01%7CEugene.Rozenfeld%40
>> microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91
>> ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8e
>> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
>> 000%7C%7C%7C&amp;sdata=3dlTtTfe4XOOm6BEy5YLWG0l3WlQdfbCyFiXs3Q7W1I%3D
>> &amp;reserved=0>
>>
>> Fixes for AutoFDO
>> testing<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
>> 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D9265b37853
>> 1391498ec1727f67a45da72a6c07e9&amp;data=05%7C01%7CEugene.Rozenfeld%40
>> microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91
>> ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8e
>> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
>> 000%7C%7C%7C&amp;sdata=ZWSIQ0jb6t2DZQpDb%2F7e5FqKM6KKskM%2FAYzLpxbUkp
>> 4%3D&amp;reserved=0>
>>
>> Fix indirect call inlining with
>> AutoFDO<https://nam06.safelinks.protection.outlook.com/?url=https%3A%
>> 2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3D285aa6895d
>> 479bed8e72ad363290846645b6faa0&amp;data=05%7C01%7CEugene.Rozenfeld%40
>> microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91
>> ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8e
>> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
>> 000%7C%7C%7C&amp;sdata=W8kTuSjC188BYd3fkZvCK5UxWrYutosB%2FeZnWbmAbPI%
>> 3D&amp;reserved=0>
>>
>
> This "indirect call inlining with AutoFDO" patch need to be backported 
> to at least gcc-10 as it will caused stack overflow for some cases 
> I've met...


Sorry, should be this:
  AutoFDO: Don't try to promote indirect calls that result in recursive direct calls<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fgit%2F%3Fp%3Dgcc.git%3Ba%3Dcommit%3Bh%3Dba125745d9e9fe90a18a2af8701b3269c5fdd468&amp;data=05%7C01%7CEugene.Rozenfeld%40microsoft.com%7Ccda1f6374c584732950e08da6f712683%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637944829019804556%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=tNqkTMULm5eS4ZZTb15Pd1bOtRj%2BvM9cGRzsW4gGJZQ%3D&amp;reserved=0>,

Though the "indirect call inlining with AutoFDO" also need verify on old branches.  indirect call format is updated with Martin's dynamic topn patch.

>
> Xionghu