* AutoFDO tools for GCC @ 2024-03-25 20:52 Eugene Rozenfeld 2024-03-26 7:45 ` Richard Biener 0 siblings, 1 reply; 5+ messages in thread From: Eugene Rozenfeld @ 2024-03-25 20:52 UTC (permalink / raw) To: gcc; +Cc: Snehasish Kumar, Andi Kleen, Jan Hubicka [-- Attachment #1: Type: text/plain, Size: 1099 bytes --] Hello, I've been the AutoFDO maintainer for the last 1.5 years. I've resurrected autoprofiledbootstrap build and made a number of other fixes/improvements (e.g., discriminator support). The tools for AutoFDO (create_gcov, etc.) currently live in https://github.com/google/AutoFDO repo and GCC AutoFDO documentation points users to that repo. That repo also has tools for LLVM AutoFDO. https://github.com/google/AutoFDO has several submodules: https://github.com/google/autofdo/blob/master/.gitmodules I got a message from Snehasish (cc'd) that google intends to migrate the tools for LLVM to the LLVM repo and wants to archive https://github.com/google/AutoFDO. That will be a problem for AutoFDO in GCC. The idea to find a different home for GCC AutoFDO tools was discussed before on this alias but this becomes more urgent now. One idea was to build these tools from GCC repo and another was to produce gcov from perf tool directly. Andi (cc'd) had some early unfinished prototype for latter. Please let me know if you have thoughts on how we should proceed. Thanks, Eugene ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AutoFDO tools for GCC 2024-03-25 20:52 AutoFDO tools for GCC Eugene Rozenfeld @ 2024-03-26 7:45 ` Richard Biener 2024-03-26 22:39 ` Andi Kleen 0 siblings, 1 reply; 5+ messages in thread From: Richard Biener @ 2024-03-26 7:45 UTC (permalink / raw) To: Eugene Rozenfeld; +Cc: gcc, Snehasish Kumar, Andi Kleen, Jan Hubicka On Mon, Mar 25, 2024 at 9:54 PM Eugene Rozenfeld via Gcc <gcc@gcc.gnu.org> wrote: > > Hello, > > I've been the AutoFDO maintainer for the last 1.5 years. I've resurrected autoprofiledbootstrap build and made a number of other fixes/improvements (e.g., discriminator support). > > The tools for AutoFDO (create_gcov, etc.) currently live in https://github.com/google/AutoFDO repo and GCC AutoFDO documentation points users to that repo. That repo also has tools for LLVM AutoFDO. > https://github.com/google/AutoFDO has several submodules: https://github.com/google/autofdo/blob/master/.gitmodules > > I got a message from Snehasish (cc'd) that google intends to migrate the tools for LLVM to the LLVM repo and wants to archive https://github.com/google/AutoFDO. That will be a problem for AutoFDO in GCC. The idea to find a different home for GCC AutoFDO tools was discussed before on this alias but this becomes more urgent now. One idea was to build these tools from GCC repo and another was to produce gcov from perf tool directly. Andi (cc'd) had some early unfinished prototype for latter. > > Please let me know if you have thoughts on how we should proceed. I think it makes sense for GCC specific parts to live in the GCC repository alongside gcov tools. I do wonder how much common code there is between the LLVM and the GCC tooling though and whether it makes sense to keep it common (and working with both frontends)? The pragmatic solution would have been to fork the repo on github to a place not within the google group ... Richard. > Thanks, > > Eugene ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AutoFDO tools for GCC 2024-03-26 7:45 ` Richard Biener @ 2024-03-26 22:39 ` Andi Kleen 2024-03-27 20:49 ` Jason Merrill 0 siblings, 1 reply; 5+ messages in thread From: Andi Kleen @ 2024-03-26 22:39 UTC (permalink / raw) To: Richard Biener; +Cc: Eugene Rozenfeld, gcc, Snehasish Kumar, Jan Hubicka On Tue, Mar 26, 2024 at 08:45:22AM +0100, Richard Biener wrote: > On Mon, Mar 25, 2024 at 9:54 PM Eugene Rozenfeld via Gcc > <gcc@gcc.gnu.org> wrote: > > > > Hello, > > > > I've been the AutoFDO maintainer for the last 1.5 years. I've resurrected autoprofiledbootstrap build and made a number of other fixes/improvements (e.g., discriminator support). > > > > The tools for AutoFDO (create_gcov, etc.) currently live in https://github.com/google/AutoFDO repo and GCC AutoFDO documentation points users to that repo. That repo also has tools for LLVM AutoFDO. > > https://github.com/google/AutoFDO has several submodules: https://github.com/google/autofdo/blob/master/.gitmodules > > > > I got a message from Snehasish (cc'd) that google intends to migrate the tools for LLVM to the LLVM repo and wants to archive https://github.com/google/AutoFDO. That will be a problem for AutoFDO in GCC. The idea to find a different home for GCC AutoFDO tools was discussed before on this alias but this becomes more urgent now. One idea was to build these tools from GCC repo and another was to produce gcov from perf tool directly. Andi (cc'd) had some early unfinished prototype for latter. > > > > Please let me know if you have thoughts on how we should proceed. > > I think it makes sense for GCC specific parts to live in the GCC > repository alongside gcov tools. I do wonder how much common code > there is > between the LLVM and the GCC tooling though and whether it makes sense > to keep it common (and working with both frontends)? The > pragmatic solution would have been to fork the repo on github to a > place not within the google group ... In tree would need convincing Google to assign the copyright. Some of the code has problems however. The main problem is that it always loads everything into memory, so you cannot do longer recording session with larger data files. Even the gcc boot strap is already pushing it on a reasonably memory sized machine. The scheme of only processing one target at a time is also quite inefficient. For systems with shared libraries and multiple binaries it would be much more efficient to do it all in one pass and write multiple gcovs, and then just let Makefiles or gcc wrapper pick. The other big issue is that the perf format keeps adding new records and their perf parser is written in a way that it always asserts on anything it doesn't understand, so there is constant breakage with new perf versions. That is fixable however (I had a patch submitted but it wasn't merged) I think what makes sense to have from the code based are profile_diff/merge etc. which are needed for scalable collection. Or perhaps it would be best if gcov just gained those functionalities. -Andi ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AutoFDO tools for GCC 2024-03-26 22:39 ` Andi Kleen @ 2024-03-27 20:49 ` Jason Merrill 2024-04-02 22:30 ` Snehasish Kumar 0 siblings, 1 reply; 5+ messages in thread From: Jason Merrill @ 2024-03-27 20:49 UTC (permalink / raw) To: Andi Kleen Cc: Richard Biener, Eugene Rozenfeld, gcc, Snehasish Kumar, Jan Hubicka On Tue, Mar 26, 2024 at 6:41 PM Andi Kleen via Gcc <gcc@gcc.gnu.org> wrote: > > On Tue, Mar 26, 2024 at 08:45:22AM +0100, Richard Biener wrote: > > On Mon, Mar 25, 2024 at 9:54 PM Eugene Rozenfeld via Gcc > > <gcc@gcc.gnu.org> wrote: > > > > > > Hello, > > > > > > I've been the AutoFDO maintainer for the last 1.5 years. I've resurrected autoprofiledbootstrap build and made a number of other fixes/improvements (e.g., discriminator support). > > > > > > The tools for AutoFDO (create_gcov, etc.) currently live in https://github.com/google/AutoFDO repo and GCC AutoFDO documentation points users to that repo. That repo also has tools for LLVM AutoFDO. > > > https://github.com/google/AutoFDO has several submodules: https://github.com/google/autofdo/blob/master/.gitmodules > > > > > > I got a message from Snehasish (cc'd) that google intends to migrate the tools for LLVM to the LLVM repo and wants to archive https://github.com/google/AutoFDO. That will be a problem for AutoFDO in GCC. The idea to find a different home for GCC AutoFDO tools was discussed before on this alias but this becomes more urgent now. One idea was to build these tools from GCC repo and another was to produce gcov from perf tool directly. Andi (cc'd) had some early unfinished prototype for latter. > > > > > > Please let me know if you have thoughts on how we should proceed. > > > > I think it makes sense for GCC specific parts to live in the GCC > > repository alongside gcov tools. I do wonder how much common code > > there is > > between the LLVM and the GCC tooling though and whether it makes sense > > to keep it common (and working with both frontends)? The > > pragmatic solution would have been to fork the repo on github to a > > place not within the google group ... > > In tree would need convincing Google to assign the copyright. Would it? Looks like it's under a free license (apache 2), not everything in the tree is copyright FSF or GPL3. Jason ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AutoFDO tools for GCC 2024-03-27 20:49 ` Jason Merrill @ 2024-04-02 22:30 ` Snehasish Kumar 0 siblings, 0 replies; 5+ messages in thread From: Snehasish Kumar @ 2024-04-02 22:30 UTC (permalink / raw) To: Jason Merrill Cc: Andi Kleen, Richard Biener, Eugene Rozenfeld, gcc, Jan Hubicka [-- Attachment #1: Type: text/plain, Size: 4243 bytes --] Thanks for initiating this discussion Eugene. For a little bit more context on the motivation -- Meta has developed a new type of AutoFDO which is committed upstream in LLVM and we want to unify our tooling with this approach. > I do wonder how much common code there is > between the LLVM and the GCC tooling though and whether it makes sense > to keep it common (and working with both frontends)? The key components are the perf.data reader, profile construction data structures and the profile writer. The reader parses perf.data as protobuf [1] and suffers from a few drawbacks (as Andi pointed out). The intermediate data structure which represents the profile is shared (~1500LOC in [2]). Finally, LLVM and GCC have their own bespoke profile writers [3]. So given the drawbacks of reader and limited sharing I think it would be best to fork these tools into the GCC repo. Having perf generate the profiles is an interesting idea and in addition to addressing the issues Andi raised, would also simplify replicated symbolization logic. In fact, the new implementation in LLVM parses perf script output [4] to generate AutoFDO profiles. Finally, if AutoFDO is adopted by the kernel, a dependence on another repository is undesirable. > I think what makes sense to have from the code based are > profile_diff/merge etc. which are needed for scalable collection. > Or perhaps it would be best if gcov just gained those functionalities. Yes, this should be straight-forward. >> In tree would need convincing Google to assign the copyright. > > Would it? Looks like it's under a free license (apache 2), not > everything in the tree is copyright FSF or GPL3. I can ask around more on my end if I get clarification on this. Thanks, Snehasish [1] https://github.com/google/perf_data_converter [2] https://github.com/google/autofdo/blob/master/symbol_map.cc [3] https://github.com/google/autofdo/blob/master/profile_writer.cc [4] https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-profgen/PerfReader.cpp On Wed, Mar 27, 2024 at 1:49 PM Jason Merrill <jason@redhat.com> wrote: > On Tue, Mar 26, 2024 at 6:41 PM Andi Kleen via Gcc <gcc@gcc.gnu.org> > wrote: > > > > On Tue, Mar 26, 2024 at 08:45:22AM +0100, Richard Biener wrote: > > > On Mon, Mar 25, 2024 at 9:54 PM Eugene Rozenfeld via Gcc > > > <gcc@gcc.gnu.org> wrote: > > > > > > > > Hello, > > > > > > > > I've been the AutoFDO maintainer for the last 1.5 years. I've > resurrected autoprofiledbootstrap build and made a number of other > fixes/improvements (e.g., discriminator support). > > > > > > > > The tools for AutoFDO (create_gcov, etc.) currently live in > https://github.com/google/AutoFDO repo and GCC AutoFDO documentation > points users to that repo. That repo also has tools for LLVM AutoFDO. > > > > https://github.com/google/AutoFDO has several submodules: > https://github.com/google/autofdo/blob/master/.gitmodules > > > > > > > > I got a message from Snehasish (cc'd) that google intends to migrate > the tools for LLVM to the LLVM repo and wants to archive > https://github.com/google/AutoFDO. That will be a problem for AutoFDO in > GCC. The idea to find a different home for GCC AutoFDO tools was discussed > before on this alias but this becomes more urgent now. One idea was to > build these tools from GCC repo and another was to produce gcov from perf > tool directly. Andi (cc'd) had some early unfinished prototype for latter. > > > > > > > > Please let me know if you have thoughts on how we should proceed. > > > > > > I think it makes sense for GCC specific parts to live in the GCC > > > repository alongside gcov tools. I do wonder how much common code > > > there is > > > between the LLVM and the GCC tooling though and whether it makes sense > > > to keep it common (and working with both frontends)? The > > > pragmatic solution would have been to fork the repo on github to a > > > place not within the google group ... > > > > In tree would need convincing Google to assign the copyright. > > Would it? Looks like it's under a free license (apache 2), not > everything in the tree is copyright FSF or GPL3. > > Jason > > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-04-02 22:31 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-03-25 20:52 AutoFDO tools for GCC Eugene Rozenfeld 2024-03-26 7:45 ` Richard Biener 2024-03-26 22:39 ` Andi Kleen 2024-03-27 20:49 ` Jason Merrill 2024-04-02 22:30 ` Snehasish Kumar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).