From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) by sourceware.org (Postfix) with ESMTPS id 075993857400 for ; Wed, 26 May 2021 03:07:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 075993857400 Received: by mail-lj1-x22a.google.com with SMTP id 131so40830806ljj.3 for ; Tue, 25 May 2021 20:07:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=npGiOh5QQqymY7e/bUjfDD6QwThupEVG/ta9TrOKJ3o=; b=ufG5zkFJRZp91TTL8SwUjYsHKqI1ezrZaFtU1cIkF1Gikqti9aa7FbK5xeU71EB+hG cL1fuByALpQ9fSjmBOUOxYlAynRqNjrY3hbc4clPw1aRHvCd4tNFq2U3083I+eGGiMO/ N3NPlX9dOnT7/5oJWQt5+J+GtdYZtoNn7n4eWfz8bla/C5rBGf/mnIsIQ83kyXKSN7qu M9SIQTtfnY1/s+PLvzQfH14F91S7i5CC6t1r6jL/aVtHpVOdbVxQ4x9QyjpEQhLFNebK GiQeSZS5kwA46UFQOf4R6ISzVU6DUZerwtga68XFL6oZrmt/ceZXog2JAKM3NTyMYGMG aKoQ== X-Gm-Message-State: AOAM531kFzkYVIbngibp8myomC5+STzukM1amfi2tSiAwl40ceQLR0zM 0B+U+tCteH5JL1XT3u8g5MgSs2CP8ltXv8/vm6+VtA== X-Google-Smtp-Source: ABdhPJxFdn8q5pURU70XIlfxR2HjEMlZ45/rI+fwqc5Rn+wJ3xVOYD+yonlisigpKUbmBKCOv+ABpdgxzYvue7fuv4A= X-Received: by 2002:a2e:7011:: with SMTP id l17mr541256ljc.139.1621998423318; Tue, 25 May 2021 20:07:03 -0700 (PDT) MIME-Version: 1.0 References: <20210423192834.GA1949@kam.mff.cuni.cz> <20210425190754.GA88857@kam.mff.cuni.cz> <87a6plulkz.fsf@linux.intel.com> <20210426180011.GA1401198@tassilo.jf.intel.com> <20210429054025.GB4032392@tassilo.jf.intel.com> In-Reply-To: From: Wei Mi Date: Tue, 25 May 2021 20:06:51 -0700 Message-ID: Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC To: Eugene Rozenfeld Cc: Andi Kleen , Hongtao Yu , Xinliang David Li , Jan Hubicka , "gcc@gcc.gnu.org" , Wenlei He Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-17.0 required=5.0 tests=BAYES_00, DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH, KAM_LOTSOFHASH, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, USER_IN_DEF_DKIM_WL, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 May 2021 03:07:07 -0000 I checked the source of protobuf 3.0.0 and it didn't contain the operator[] in RepeatedField. Need to install a newer version of protobuf. Thanks, Wei. On Tue, May 25, 2021 at 1:49 PM Eugene Rozenfeld wrote: > > Both are 3.0.0-9.1ubuntu1: > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list protobuf-compile= r > Listing... Done > protobuf-compiler/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed] > eugene@eugene-Virtual-Machine:~/autofdo1/build$ apt list libprotobuf-dev > Listing... Done > libprotobuf-dev/bionic,now 3.0.0-9.1ubuntu1 amd64 [installed] > > -----Original Message----- > From: Wei Mi > Sent: Tuesday, May 25, 2021 9:17 AM > To: Eugene Rozenfeld > Cc: Andi Kleen ; Hongtao Yu ; Xinliang Da= vid Li ; Jan Hubicka ; gcc@gcc.gnu.org;= Wenlei He > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > It looks like some version problem about protobuf-compiler and libprotobu= f-dev. Could you check what is the installed version on your end for those = two packages and see if they are consistent? > > On my platform, they are both 3.12.4. > > On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld wrote: > > > > That eliminates the previous error but there is a new one: > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [3/199] Building > > CXX object > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe > > r/perf_reader.cc.o > > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src= /quipper/perf_reader.cc.o > > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_part= y/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../thir= d_party/abseil -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT CMak= eFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_re= ader.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/s= rc/quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/pe= rf_data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_= converter/src/quipper/perf_reader.cc > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In membe= r function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::Data= Reader*, size_t)': > > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: = error: no match for 'operator[]' (operand types are 'const google::protobuf= ::RepeatedField' and 'int') > > nrcpus =3D proto_uint32_metadata.data()[0]; > > > > -----Original Message----- > > From: Wei Mi > > Sent: Monday, May 24, 2021 8:54 PM > > To: Eugene Rozenfeld > > Cc: Andi Kleen ; Hongtao Yu ; Xinliang > > David Li ; Jan Hubicka ; > > gcc@gcc.gnu.org; Wenlei He > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > It isn't exposed on my platform either. Looks like a bug in perf_data_c= onverter (i.e., quipper). Could you try adding #include in = third_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if= it fixes the problem? If it works, I will need to file a bug against perf_= data_converter. > > > > Thanks, > > Wei. > > > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld wrote: > > > > > > That fixed the error I saw before but the build still fails. The > > > errors start with > > > > > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja > > > > > > [2/217] Building CXX object > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip > > > pe > > > r/huge_page_deducer.cc.o > > > > > > FAILED: > > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quip > > > pe > > > r/huge_page_deducer.cc.o > > > > > > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_pa= rty/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../th= ird_party/abseil -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT CM= akeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_= page_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_con= verter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/= third_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../th= ird_party/perf_data_converter/src/quipper/huge_page_deducer.cc > > > > > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc: > > > 14 > > > 4:26: error: 'unordered_map' in namespace 'std' does not name a > > > template type > > > > > > using container =3D std::unordered_map; > > > > > > ^~~~~~~~~~~~~ > > > > > > > > > > > > > > > > > > > > > > > > From: Wei Mi > > > Sent: Monday, May 24, 2021 8:12 PM > > > To: Eugene Rozenfeld > > > Cc: Andi Kleen ; Hongtao Yu ; > > > Xinliang David Li ; Jan Hubicka > > > ; gcc@gcc.gnu.org; Wenlei He > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > > > > > Sorry, I added dependency for create_gcov but missed it for dump_gcov= . Fixed it at https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3= A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e2= 7746ca4fc5bf6&data=3D04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C0cd91= 708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63= 7575562315397315%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz= IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3D9G9uGhG%2BmK35bdV4QL9BO= TVg6bFFQrVP46YdLOi3Ak4%3D&reserved=3D0. > > > > > > > > > > > > Thanks, > > > > > > Wei. > > > > > > > > > > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld wrote: > > > > > > Thank you Wei. Looks like something is still missing. This time perf_= data.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with c= make 3.12.1): > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] > > > Building CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/absei= l -I../third_party/perf_data_converter/src -I../third_party/perf_data_conve= rter/src/quipper -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT CM= akeFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/pr= ofile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc > > > In file included from ../third_party/perf_data_converter/src/quipper/= perf_parser.h:18:0, > > > from ../sample_reader.h:18, > > > from ../profile.h:15, > > > from ../profile.cc:5: > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > > > In file included from ../profile.h:14:0, > > > from ../profile.cc:5: > > > ../base/macros.h:114:0: note: this is the location of the previous > > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > > > In file included from ../third_party/perf_data_converter/src/quipper/= perf_parser.h:18:0, > > > from ../sample_reader.h:18, > > > from ../profile.h:15, > > > from ../profile.cc:5: > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > > sizeof(*x)) > > > > > > In file included from ../profile.h:14:0, > > > from ../profile.cc:5: > > > ../base/macros.h:162:0: note: this is the location of the previous > > > definition #define arraysize(array) > > > (sizeof(ArraySizeHelper(array))) > > > > > > In file included from ../third_party/perf_data_converter/src/quipper/= perf_parser.h:21:0, > > > from ../sample_reader.h:18, > > > from ../profile.h:15, > > > from ../profile.cc:5: > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: > > > fatal error: perf_data.pb.h: No such file or directory #include "per= f_data.pb.h" > > > ^~~~~~~~~~~~~~~~ > > > compilation terminated. > > > [6/241] Building CXX object > > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o > > > ninja: build stopped: subcommand failed. > > > > > > Thanks, > > > > > > Eugene > > > > > > From: Wei Mi > > > Sent: Saturday, May 22, 2021 9:37 AM > > > To: Eugene Rozenfeld > > > Cc: Andi Kleen ; Hongtao Yu ; > > > Xinliang David Li ; Jan Hubicka > > > ; gcc@gcc.gnu.org; Wenlei He > > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > It is a proto library build dependency issue which didn't expose on m= y platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url= =3Dhttps%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885= cc5eb1bd917eb2d12731710&data=3D04%7C01%7CEugene.Rozenfeld%40microsoft.c= om%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db47%7C= 1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ= IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DSL5mLhelrkeB2= 6Mg7tNaml0qjZ2OklYPwQR9btVhsqk%3D&reserved=3D0. Please try again. > > > > > > Thanks, > > > Wei. > > > > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld wrote: > > > I tried following the instructions in "2.2 Build autofdo tool for gcc= " in https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgi= thub.com%2Fgoogle%2Fautofdo%23readme&data=3D04%7C01%7CEugene.Rozenfeld%= 40microsoft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d= 7cd011db47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4= wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3D= PYVm%2BHsRk%2Bfw7%2BaNgYNWn8zMnBsqcqqVSpyXQktd7sA%3D&reserved=3D0 got b= uild failures: > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] > > > Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o > > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o > > > /usr/bin/c++ -I../ -I../third_party/glog/src > > > -I../third_party/abseil -I../third_party/perf_data_converter/src -I..= /third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/gl= og -std=3Dgnu++1z -MD -MT CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF C= MakeFiles/create_gcov_lib.dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.= dir/profile.cc.o -c ../profile.cc In file included from ../third_party/perf= _data_converter/src/quipper/perf_parser.h:18:0, > > > from ../sample_reader.h:18, > > > from ../profile.h:15, > > > from ../profile.cc:5: > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile= .h:14:0, > > > from ../profile.cc:5: > > > ../base/macros.h:114:0: note: this is the location of the previous > > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file > > > included from ../third_party/perf_data_converter/src/quipper/perf_par= ser.h:18:0, > > > from ../sample_reader.h:18, > > > from ../profile.h:15, > > > from ../profile.cc:5: > > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > > sizeof(*x)) In file included from ../profile.h:14:0, > > > from ../profile.cc:5: > > > ../base/macros.h:162:0: note: this is the location of the previous > > > definition #define arraysize(array) (sizeof(ArraySizeHelper(array))) > > > In file included from ../third_party/perf_data_converter/src/quipper/= perf_parser.h:21:0, > > > from ../sample_reader.h:18, > > > from ../profile.h:15, > > > from ../profile.cc:5: > > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: > > > fatal error: perf_stat.pb.h: No such file or directory #include "perf= _stat.pb.h" > > > ^~~~~~~~~~~~~~~~ > > > compilation terminated. > > > > > > What is supposed to generate perf_stat.pb.h? > > > > > > Thanks, > > > > > > Eugene > > > > > > From: Wei Mi > > > Sent: Monday, May 10, 2021 4:47 PM > > > To: Andi Kleen > > > Cc: Hongtao Yu ; Xinliang David Li > > > ; Jan Hubicka ; > > > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld > > > ; Wenlei He > > > > > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fg= ithub.com%2Fgoogle%2Fautofdo&data=3D04%7C01%7CEugene.Rozenfeld%40micros= oft.com%7C0cd91708868547a3f63608d91f988b90%7C72f988bf86f141af91ab2d7cd011db= 47%7C1%7C0%7C637575562315407308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA= iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DuwQwWdjE= %2FL%2F2r0P%2Bdn7r5ulE8kRmPPw8A2XOQuZnUhM%3D&reserved=3D0 has been upda= ted. Now create_gcov/dump_gcov are added back and can be built separately. > > > > > > Please look at "2.2 Build autofdo tool for gcc" in > > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fg= i > > > th > > > ub.com%2Fgoogle%2Fautofdo%23readme&data=3D04%7C01%7CEugene.Rozenf= e > > > ld > > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141 > > > af > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3 > > > d8 > > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7 > > > C1 > > > 000&sdata=3DEwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&= r > > > es > > > erved=3D0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen > > > wrote: > > > > > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote: > > > > > Andi, thanks for pointing out the perf script issues. Can you = please > > > > > elaborate a bit on the exact issue you have seen? We've been u= sing > > > > > specific output of perf script such as mmap, LBR and callstack= events > > > > > filtered by process id. It works fine so far but may certainly= hit issues > > > > > in the future with extended uses. > > > > > > > > Okay I took a look at the latest autofdo now. It seems to be > > > > basically a LLVM project now that depends on LLVM to even build > > > > with all kinds of dependency hell on some old LLVM version and othe= r packages. > > > > > > > > I guess gcc will really need a replacement that doesn't pull in > > > > all of LLVM if it wants to continue supporting autofdo. > > > > > > > > I'm myself unable to build now. > > > > > > > > I'm using the old version I had a git fork of and that was before > > > > all of this. I added a patch to make it work with the latest perf > > > > by ignoring increased perf_attr and unknown perf events. > > > > > > > > Honza please use > > > > > > > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2= F > > > > gi > > > > thub.com%2Fandikleen%2Fautofdo&data=3D04%7C01%7CEugene.Rozenfel= d > > > > %4 > > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141 > > > > af > > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZs > > > > b3 > > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% > > > > 3D > > > > %7C1000&sdata=3DDilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3= D > > > > &a > > > > mp;reserved=3D0 -b perf-future > > > > > > > > for testing. > > > > > > > > -Andi > > > >