Re: AutoFDO tools for GCC

[Snehasish Kumar <snehasishk@google.com>]

[-- Attachment #1: Type: text/plain, Size: 4243 bytes --]

Thanks for initiating this discussion Eugene. For a little bit more context
on the motivation -- Meta has developed a new type of AutoFDO which is
committed upstream in LLVM and we want to unify our tooling with this
approach.

> I do wonder how much common code there is
> between the LLVM and the GCC tooling though and whether it makes sense
> to keep it common (and working with both frontends)?

The key components are the perf.data reader, profile construction data
structures and the profile writer. The reader parses perf.data as protobuf
[1] and suffers from a few drawbacks (as Andi pointed out). The
intermediate data structure which represents the profile is shared
(~1500LOC in [2]). Finally, LLVM and GCC have their own bespoke profile
writers [3]. So given the drawbacks of reader and limited sharing I think
it would be best to fork these tools into the GCC repo. Having perf
generate the profiles is an interesting idea and in addition to addressing
the issues Andi raised, would also simplify replicated symbolization logic.
In fact, the new implementation in LLVM parses perf script output [4] to
generate AutoFDO profiles. Finally, if AutoFDO is adopted by the kernel, a
dependence on another repository is undesirable.

> I think what makes sense to have from the code based are
> profile_diff/merge etc. which are needed for scalable collection.
> Or perhaps it would be best if gcov just gained those functionalities.

Yes, this should be straight-forward.

>> In tree would need convincing Google to assign the copyright.
>
> Would it?  Looks like it's under a free license (apache 2), not
> everything in the tree is copyright FSF or GPL3.

I can ask around more on my end if I get clarification on this.

Thanks,
Snehasish


[1] https://github.com/google/perf_data_converter
[2] https://github.com/google/autofdo/blob/master/symbol_map.cc
[3] https://github.com/google/autofdo/blob/master/profile_writer.cc
[4]
https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-profgen/PerfReader.cpp


On Wed, Mar 27, 2024 at 1:49 PM Jason Merrill <jason@redhat.com> wrote:

> On Tue, Mar 26, 2024 at 6:41 PM Andi Kleen via Gcc <gcc@gcc.gnu.org>
> wrote:
> >
> > On Tue, Mar 26, 2024 at 08:45:22AM +0100, Richard Biener wrote:
> > > On Mon, Mar 25, 2024 at 9:54 PM Eugene Rozenfeld via Gcc
> > > <gcc@gcc.gnu.org> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I've been the AutoFDO maintainer for the last 1.5 years. I've
> resurrected autoprofiledbootstrap build and made a number of other
> fixes/improvements (e.g., discriminator support).
> > > >
> > > > The tools for AutoFDO (create_gcov, etc.) currently live in
> https://github.com/google/AutoFDO  repo and GCC AutoFDO documentation
> points users to that repo. That repo also has tools for LLVM AutoFDO.
> > > > https://github.com/google/AutoFDO  has several submodules:
> https://github.com/google/autofdo/blob/master/.gitmodules
> > > >
> > > > I got a message from Snehasish (cc'd) that google intends to migrate
> the tools for LLVM to the LLVM repo and wants to archive
> https://github.com/google/AutoFDO. That will be a problem for AutoFDO in
> GCC. The idea to find a different home for GCC AutoFDO tools was discussed
> before on this alias but this becomes more urgent now. One idea was to
> build these tools from GCC repo and another was to produce gcov from perf
> tool directly. Andi (cc'd)  had some early unfinished prototype for latter.
> > > >
> > > > Please let me know if you have thoughts on how we should proceed.
> > >
> > > I think it makes sense for GCC specific parts to live in the GCC
> > > repository alongside gcov tools.  I do wonder how much common code
> > > there is
> > > between the LLVM and the GCC tooling though and whether it makes sense
> > > to keep it common (and working with both frontends)?  The
> > > pragmatic solution would have been to fork the repo on github to a
> > > place not within the google group ...
> >
> > In tree would need convincing Google to assign the copyright.
>
> Would it?  Looks like it's under a free license (apache 2), not
> everything in the tree is copyright FSF or GPL3.
>
> Jason
>
>