From: Wei Mi <wmi@google.com>
To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
Cc: Andi Kleen <ak@linux.intel.com>, Hongtao Yu <hoy@fb.com>,
Xinliang David Li <davidxl@google.com>,
Jan Hubicka <hubicka@ucw.cz>, "gcc@gcc.gnu.org" <gcc@gcc.gnu.org>,
Wenlei He <wenlei@fb.com>
Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
Date: Tue, 25 May 2021 09:16:54 -0700 [thread overview]
Message-ID: <CA+4CFy44-dAcYaim3ztZpEh4=rH1VRzJ=MrOUmd-KgNR35JY+g@mail.gmail.com> (raw)
In-Reply-To: <MWHPR21MB0798525139634A6EA53606B991259@MWHPR21MB0798.namprd21.prod.outlook.com>
It looks like some version problem about protobuf-compiler and
libprotobuf-dev. Could you check what is the installed version on your
end for those two packages and see if they are consistent?
On my platform, they are both 3.12.4.
On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld
<Eugene.Rozenfeld@microsoft.com> wrote:
>
> That eliminates the previous error but there is a new one:
>
> eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> [3/199] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o
> /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_converter/src/quipper/perf_reader.cc
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
> ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: error: no match for 'operator[]' (operand types are 'const google::protobuf::RepeatedField<unsigned int>' and 'int')
> nrcpus = proto_uint32_metadata.data()[0];
>
> -----Original Message-----
> From: Wei Mi <wmi@google.com>
> Sent: Monday, May 24, 2021 8:54 PM
> To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>; gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
>
> It isn't exposed on my platform either. Looks like a bug in perf_data_converter (i.e., quipper). Could you try adding #include <unordered_map> in third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if it fixes the problem? If it works, I will need to file a bug against perf_data_converter.
>
> Thanks,
> Wei.
>
> On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > That fixed the error I saw before but the build still fails. The
> > errors start with
> >
> >
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
> >
> > [2/217] Building CXX object
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> > r/huge_page_deducer.cc.o
> >
> > FAILED:
> > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe
> > r/huge_page_deducer.cc.o
> >
> > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_party/abseil -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc
> >
> > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:14
> > 4:26: error: 'unordered_map' in namespace 'std' does not name a
> > template type
> >
> > using container = std::unordered_map<key_t, value_t>;
> >
> > ^~~~~~~~~~~~~
> >
> >
> >
> >
> >
> >
> >
> > From: Wei Mi <wmi@google.com>
> > Sent: Monday, May 24, 2021 8:12 PM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> >
> >
> > Sorry, I added dependency for create_gcov but missed it for dump_gcov. Fixed it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e27746ca4fc5bf6&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816277204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hwJ%2BG64Yw%2BEGMAV7kzilOjAOkHoOv3TQpdqkzcHGO%2FM%3D&reserved=0.
> >
> >
> >
> > Thanks,
> >
> > Wei.
> >
> >
> >
> > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com> wrote:
> >
> > Thank you Wei. Looks like something is still missing. This time perf_data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cmake 3.12.1):
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] Building
> > CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o
> > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > from ../sample_reader.h:18,
> > from ../profile.h:15,
> > from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > DISALLOW_COPY_AND_ASSIGN(TypeName) \
> >
> > In file included from ../profile.h:14:0,
> > from ../profile.cc:5:
> > ../base/macros.h:114:0: note: this is the location of the previous
> > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
> >
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > from ../sample_reader.h:18,
> > from ../profile.h:15,
> > from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > sizeof(*x))
> >
> > In file included from ../profile.h:14:0,
> > from ../profile.cc:5:
> > ../base/macros.h:162:0: note: this is the location of the previous
> > definition #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> >
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > from ../sample_reader.h:18,
> > from ../profile.h:15,
> > from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10:
> > fatal error: perf_data.pb.h: No such file or directory #include "perf_data.pb.h"
> > ^~~~~~~~~~~~~~~~
> > compilation terminated.
> > [6/241] Building CXX object
> > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
> > ninja: build stopped: subcommand failed.
> >
> > Thanks,
> >
> > Eugene
> >
> > From: Wei Mi <wmi@google.com>
> > Sent: Saturday, May 22, 2021 9:37 AM
> > To: Eugene Rozenfeld <Eugene.Rozenfeld@microsoft.com>
> > Cc: Andi Kleen <ak@linux.intel.com>; Hongtao Yu <hoy@fb.com>; Xinliang
> > David Li <davidxl@google.com>; Jan Hubicka <hubicka@ucw.cz>;
> > gcc@gcc.gnu.org; Wenlei He <wenlei@fb.com>
> > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > It is a proto library build dependency issue which didn't expose on my platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885cc5eb1bd917eb2d12731710&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PikMs8RtJHBsTUqdjCVWAKNSC5Rls%2B71pSEhq4z0XsM%3D&reserved=0. Please try again.
> >
> > Thanks,
> > Wei.
> >
> > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld <mailto:Eugene.Rozenfeld@microsoft.com> wrote:
> > I tried following the instructions in "2.2 Build autofdo tool for gcc" in https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&reserved=0 got build failures:
> >
> > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] Building
> > CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
> > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil
> > -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -std=gnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > from ../sample_reader.h:18,
> > from ../profile.h:15,
> > from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0:
> > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define
> > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h:14:0,
> > from ../profile.cc:5:
> > ../base/macros.h:114:0: note: this is the location of the previous
> > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file
> > included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
> > from ../sample_reader.h:18,
> > from ../profile.h:15,
> > from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0:
> > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) /
> > sizeof(*x)) In file included from ../profile.h:14:0,
> > from ../profile.cc:5:
> > ../base/macros.h:162:0: note: this is the location of the previous
> > definition #define arraysize(array) (sizeof(ArraySizeHelper(array)))
> > In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
> > from ../sample_reader.h:18,
> > from ../profile.h:15,
> > from ../profile.cc:5:
> > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10:
> > fatal error: perf_stat.pb.h: No such file or directory #include "perf_stat.pb.h"
> > ^~~~~~~~~~~~~~~~
> > compilation terminated.
> >
> > What is supposed to generate perf_stat.pb.h?
> >
> > Thanks,
> >
> > Eugene
> >
> > From: Wei Mi <mailto:wmi@google.com>
> > Sent: Monday, May 10, 2021 4:47 PM
> > To: Andi Kleen <mailto:ak@linux.intel.com>
> > Cc: Hongtao Yu <mailto:hoy@fb.com>; Xinliang David Li
> > <mailto:davidxl@google.com>; Jan Hubicka <mailto:hubicka@ucw.cz>;
> > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld
> > <mailto:Eugene.Rozenfeld@microsoft.com>; Wenlei He
> > <mailto:wenlei@fb.com>
> > Subject: [EXTERNAL] Re: State of AutoFDO in GCC
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=a9gGf48MYY2jzbSucOAlhsmdLuYt%2BcM8V5hvGoA39Ms%3D&reserved=0 has been updated. Now create_gcov/dump_gcov are added back and can be built separately.
> >
> > Please look at "2.2 Build autofdo tool for gcc" in
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Fgoogle%2Fautofdo%23readme&data=04%7C01%7CEugene.Rozenfeld
> > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af
> > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8
> > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> > 000&sdata=EwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&res
> > erved=0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen
> > <mailto:ak@linux.intel.com> wrote:
> > >
> > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote:
> > > > Andi, thanks for pointing out the perf script issues. Can you please
> > > > elaborate a bit on the exact issue you have seen? We've been using
> > > > specific output of perf script such as mmap, LBR and callstack events
> > > > filtered by process id. It works fine so far but may certainly hit issues
> > > > in the future with extended uses.
> > >
> > > Okay I took a look at the latest autofdo now. It seems to be
> > > basically a LLVM project now that depends on LLVM to even build with
> > > all kinds of dependency hell on some old LLVM version and other packages.
> > >
> > > I guess gcc will really need a replacement that doesn't pull in all
> > > of LLVM if it wants to continue supporting autofdo.
> > >
> > > I'm myself unable to build now.
> > >
> > > I'm using the old version I had a git fork of and that was before
> > > all of this. I added a patch to make it work with the latest perf by
> > > ignoring increased perf_attr and unknown perf events.
> > >
> > > Honza please use
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > thub.com%2Fandikleen%2Fautofdo&data=04%7C01%7CEugene.Rozenfeld%4
> > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af
> > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3
> > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > > %7C1000&sdata=Dilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D&a
> > > mp;reserved=0 -b perf-future
> > >
> > > for testing.
> > >
> > > -Andi
> > >
next prev parent reply other threads:[~2021-05-25 16:17 UTC|newest]
Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-22 19:58 Eugene Rozenfeld
2021-04-22 20:16 ` Martin Liška
2021-04-22 22:29 ` Jan Hubicka
2021-04-23 4:14 ` Xinliang David Li
2021-04-23 7:00 ` Richard Biener
2021-04-23 7:18 ` Martin Liška
2021-04-23 9:32 ` Richard Biener
2021-04-23 16:41 ` Xinliang David Li
2021-04-23 16:54 ` Jan Hubicka
2021-04-23 17:04 ` Xinliang David Li
2021-04-23 17:16 ` Jan Hubicka
2021-04-23 17:27 ` Xinliang David Li
2021-04-23 17:28 ` Xinliang David Li
2021-04-23 19:28 ` Jan Hubicka
2021-04-23 19:58 ` Xinliang David Li
2021-04-25 19:07 ` Jan Hubicka
2021-04-25 23:18 ` Xinliang David Li
2021-04-26 4:22 ` Wei Mi
2021-04-26 15:11 ` Andi Kleen
2021-04-26 16:57 ` Xinliang David Li
2021-04-26 18:00 ` Andi Kleen
2021-04-26 18:05 ` Xinliang David Li
2021-04-26 18:40 ` Hongtao Yu
2021-04-26 19:13 ` Andi Kleen
2021-04-29 5:40 ` Andi Kleen
2021-04-29 14:45 ` 172060045
2021-04-30 21:43 ` Andi Kleen
2021-05-08 11:25 ` 172060045
2021-05-09 16:28 ` Andi Kleen
2021-05-09 17:01 ` Jan Hubicka
2021-05-10 15:36 ` Andi Kleen
2021-05-10 16:55 ` Joseph Myers
2021-05-10 17:21 ` Andi Kleen
2022-07-26 20:12 ` Eugene Rozenfeld
2022-07-26 22:37 ` David Edelsohn
2022-07-27 7:26 ` Jan Hubicka
2022-07-27 18:30 ` [EXTERNAL] " Eugene Rozenfeld
2022-07-27 18:24 ` Eugene Rozenfeld
2022-07-27 1:31 ` Xionghu Luo
2022-07-27 1:41 ` Xionghu Luo
2022-07-27 18:38 ` [EXTERNAL] " Eugene Rozenfeld
2021-05-10 23:46 ` Wei Mi
2021-05-22 1:28 ` [EXTERNAL] " Eugene Rozenfeld
2021-05-22 16:36 ` Wei Mi
2021-05-25 1:39 ` Eugene Rozenfeld
2021-05-25 3:11 ` Wei Mi
2021-05-25 3:33 ` Eugene Rozenfeld
2021-05-25 3:54 ` Wei Mi
2021-05-25 7:01 ` Eugene Rozenfeld
2021-05-25 16:16 ` Wei Mi [this message]
2021-05-25 20:49 ` Eugene Rozenfeld
2021-05-26 3:06 ` Wei Mi
2021-05-26 23:39 ` Eugene Rozenfeld
2021-05-27 2:51 ` Wei Mi
2021-06-12 1:14 ` Eugene Rozenfeld
2021-06-14 17:00 ` Wei Mi
2021-04-23 17:20 ` Jan Hubicka
2021-04-23 16:36 ` Xinliang David Li
2021-04-30 18:48 ` [EXTERNAL] " Eugene Rozenfeld
2021-04-30 21:45 ` Andi Kleen
2021-06-24 21:45 ` Eugene Rozenfeld
2021-04-23 1:46 ` Bin.Cheng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CA+4CFy44-dAcYaim3ztZpEh4=rH1VRzJ=MrOUmd-KgNR35JY+g@mail.gmail.com' \
--to=wmi@google.com \
--cc=Eugene.Rozenfeld@microsoft.com \
--cc=ak@linux.intel.com \
--cc=davidxl@google.com \
--cc=gcc@gcc.gnu.org \
--cc=hoy@fb.com \
--cc=hubicka@ucw.cz \
--cc=wenlei@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).