From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) by sourceware.org (Postfix) with ESMTPS id 26E0F393BC08 for ; Tue, 25 May 2021 16:17:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 26E0F393BC08 Received: by mail-lj1-x22d.google.com with SMTP id t17so22186056ljd.9 for ; Tue, 25 May 2021 09:17:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XJNX1lqbD916a7iSFgZDkWUHHa6ydEEsAP4KwBK/a/M=; b=JQ76vlp7AYNC3BEihVzWLtTZ+E3OzM2exn6cdv49yurBuf1cjS4lBJ4vICeHWWHKza mu8+bHSxBUoKkxaFfsn//7LckzyTPDED93ccxW/veoLhZEHjgF66hHgjHYfZtiCw8s/Y l79Pyueygw6kSALz9+AeF3LC4vKbVADo5bDVKlFpictcuIfDXRLXCsvqMhfLst0qMGwC X2pkaA3wSAsHEVmVZEyHzfZ9V9aiYUysWgfTHeihGO3MIJNH04ktEi6xpiVT/7gnlHE4 5d8sT+01WUGjWAFLa2nYCjKGg4nne1ccXsGDRiUgSzn40iQXhYPLolMDQQVExBbDlfPN vQTQ== X-Gm-Message-State: AOAM531bG/cBc0KdEI1TlNM7RSzN6RSrEoHKOFJDMHrl2ti7MP8OLO1o f1kFuHLg/NKV9J3uSZOqGA9SPqQLwW9YCxCJHwyvdQ== X-Google-Smtp-Source: ABdhPJzBh42/Wz0UC3w+gphKmDy9i+TTHNSePLgP+8okcSNpDVDj1JA6w1avEyvll1AflzFacumB/apf4jtFZiCuQ/E= X-Received: by 2002:a05:651c:329:: with SMTP id b9mr21529843ljp.128.1621959426642; Tue, 25 May 2021 09:17:06 -0700 (PDT) MIME-Version: 1.0 References: <20210423192834.GA1949@kam.mff.cuni.cz> <20210425190754.GA88857@kam.mff.cuni.cz> <87a6plulkz.fsf@linux.intel.com> <20210426180011.GA1401198@tassilo.jf.intel.com> <20210429054025.GB4032392@tassilo.jf.intel.com> In-Reply-To: From: Wei Mi Date: Tue, 25 May 2021 09:16:54 -0700 Message-ID: Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC To: Eugene Rozenfeld Cc: Andi Kleen , Hongtao Yu , Xinliang David Li , Jan Hubicka , "gcc@gcc.gnu.org" , Wenlei He Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-17.3 required=5.0 tests=BAYES_00, DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH, KAM_LOTSOFHASH, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, USER_IN_DEF_DKIM_WL, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 May 2021 16:17:11 -0000 It looks like some version problem about protobuf-compiler and libprotobuf-dev. Could you check what is the installed version on your end for those two packages and see if they are consistent? On my platform, they are both 3.12.4. On Tue, May 25, 2021 at 12:01 AM Eugene Rozenfeld wrote: > > That eliminates the previous error but there is a new one: > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja > [3/199] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_= data_converter/src/quipper/perf_reader.cc.o > FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/q= uipper/perf_reader.cc.o > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_party/= perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../third_= party/abseil -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT CMakeF= iles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_read= er.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src= /quipper/perf_reader.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf= _data_converter/src/quipper/perf_reader.cc.o -c ../third_party/perf_data_co= nverter/src/quipper/perf_reader.cc > ../third_party/perf_data_converter/src/quipper/perf_reader.cc: In member = function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataRe= ader*, size_t)': > ../third_party/perf_data_converter/src/quipper/perf_reader.cc:1518:46: er= ror: no match for 'operator[]' (operand types are 'const google::protobuf::= RepeatedField' and 'int') > nrcpus =3D proto_uint32_metadata.data()[0]; > > -----Original Message----- > From: Wei Mi > Sent: Monday, May 24, 2021 8:54 PM > To: Eugene Rozenfeld > Cc: Andi Kleen ; Hongtao Yu ; Xinliang Da= vid Li ; Jan Hubicka ; gcc@gcc.gnu.org;= Wenlei He > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > It isn't exposed on my platform either. Looks like a bug in perf_data_con= verter (i.e., quipper). Could you try adding #include in th= ird_party/perf_data_converter/src/quipper/huge_page_deducer.cc and see if i= t fixes the problem? If it works, I will need to file a bug against perf_da= ta_converter. > > Thanks, > Wei. > > On Mon, May 24, 2021 at 8:33 PM Eugene Rozenfeld wrote: > > > > That fixed the error I saw before but the build still fails. The > > errors start with > > > > > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja > > > > [2/217] Building CXX object > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe > > r/huge_page_deducer.cc.o > > > > FAILED: > > CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quippe > > r/huge_page_deducer.cc.o > > > > /usr/bin/c++ -I../third_party/perf_data_converter/src -I../third_part= y/perf_data_converter/src/quipper -I../ -I../third_party/glog/src -I../thir= d_party/abseil -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT CMak= eFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/huge_pa= ge_deducer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_conve= rter/src/quipper/huge_page_deducer.cc.o.d -o CMakeFiles/quipper_perf.dir/th= ird_party/perf_data_converter/src/quipper/huge_page_deducer.cc.o -c ../thir= d_party/perf_data_converter/src/quipper/huge_page_deducer.cc > > > > ../third_party/perf_data_converter/src/quipper/huge_page_deducer.cc:14 > > 4:26: error: 'unordered_map' in namespace 'std' does not name a > > template type > > > > using container =3D std::unordered_map; > > > > ^~~~~~~~~~~~~ > > > > > > > > > > > > > > > > From: Wei Mi > > Sent: Monday, May 24, 2021 8:12 PM > > To: Eugene Rozenfeld > > Cc: Andi Kleen ; Hongtao Yu ; Xinliang > > David Li ; Jan Hubicka ; > > gcc@gcc.gnu.org; Wenlei He > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > > > > > Sorry, I added dependency for create_gcov but missed it for dump_gcov. = Fixed it at https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%= 2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F6ca36cdc30986f13583a3aef3e277= 46ca4fc5bf6&data=3D04%7C01%7CEugene.Rozenfeld%40microsoft.com%7C1bcb2fd= a4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6375= 75116816277204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi= LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DhwJ%2BG64Yw%2BEGMAV7kzilO= jAOkHoOv3TQpdqkzcHGO%2FM%3D&reserved=3D0. > > > > > > > > Thanks, > > > > Wei. > > > > > > > > On Mon, May 24, 2021 at 6:39 PM Eugene Rozenfeld wrote: > > > > Thank you Wei. Looks like something is still missing. This time perf_da= ta.pb.h is not found. I'm getting the error below (on Ubuntu 18.04 with cma= ke 3.12.1): > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/241] Building > > CXX object CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > FAILED: CMakeFiles/dump_gcov_lib.dir/profile.cc.o > > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil = -I../third_party/perf_data_converter/src -I../third_party/perf_data_convert= er/src/quipper -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT CMak= eFiles/dump_gcov_lib.dir/profile.cc.o -MF CMakeFiles/dump_gcov_lib.dir/prof= ile.cc.o.d -o CMakeFiles/dump_gcov_lib.dir/profile.cc.o -c ../profile.cc > > In file included from ../third_party/perf_data_converter/src/quipper/pe= rf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > In file included from ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:114:0: note: this is the location of the previous > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ > > > > In file included from ../third_party/perf_data_converter/src/quipper/pe= rf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > sizeof(*x)) > > > > In file included from ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:162:0: note: this is the location of the previous > > definition #define arraysize(array) (sizeof(ArraySizeHelper(array))) > > > > In file included from ../third_party/perf_data_converter/src/quipper/pe= rf_parser.h:21:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: > > fatal error: perf_data.pb.h: No such file or directory #include "perf_= data.pb.h" > > ^~~~~~~~~~~~~~~~ > > compilation terminated. > > [6/241] Building CXX object > > CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o > > ninja: build stopped: subcommand failed. > > > > Thanks, > > > > Eugene > > > > From: Wei Mi > > Sent: Saturday, May 22, 2021 9:37 AM > > To: Eugene Rozenfeld > > Cc: Andi Kleen ; Hongtao Yu ; Xinliang > > David Li ; Jan Hubicka ; > > gcc@gcc.gnu.org; Wenlei He > > Subject: Re: [EXTERNAL] Re: State of AutoFDO in GCC > > > > It is a proto library build dependency issue which didn't expose on my = platform. I fix it at https://nam06.safelinks.protection.outlook.com/?url= =3Dhttps%3A%2F%2Fgithub.com%2Fgoogle%2Fautofdo%2Fcommit%2F98269aee9674cc885= cc5eb1bd917eb2d12731710&data=3D04%7C01%7CEugene.Rozenfeld%40microsoft.c= om%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47%7C= 1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ= IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DPikMs8RtJHBsT= UqdjCVWAKNSC5Rls%2B71pSEhq4z0XsM%3D&reserved=3D0. Please try again. > > > > Thanks, > > Wei. > > > > On Fri, May 21, 2021 at 6:28 PM Eugene Rozenfeld wrote: > > I tried following the instructions in "2.2 Build autofdo tool for gcc" = in https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith= ub.com%2Fgoogle%2Fautofdo%23readme&data=3D04%7C01%7CEugene.Rozenfeld%40= microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7c= d011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL= jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DEw= HtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&reserved=3D0 got build f= ailures: > > > > eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja [1/228] Building > > CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o > > FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o > > /usr/bin/c++ -I../ -I../third_party/glog/src -I../third_party/abseil > > -I../third_party/perf_data_converter/src -I../third_party/perf_data_con= verter/src/quipper -I../util -I. -Ithird_party/glog -std=3Dgnu++1z -MD -MT = CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF CMakeFiles/create_gcov_lib.= dir/profile.cc.o.d -o CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../pro= file.cc In file included from ../third_party/perf_data_converter/src/quippe= r/perf_parser.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: > > warning: "DISALLOW_COPY_AND_ASSIGN" redefined #define > > DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file included from ../profile.h= :14:0, > > from ../profile.cc:5: > > ../base/macros.h:114:0: note: this is the location of the previous > > definition #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ In file > > included from ../third_party/perf_data_converter/src/quipper/perf_parse= r.h:18:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: > > warning: "arraysize" redefined #define arraysize(x) (sizeof(x) / > > sizeof(*x)) In file included from ../profile.h:14:0, > > from ../profile.cc:5: > > ../base/macros.h:162:0: note: this is the location of the previous > > definition #define arraysize(array) (sizeof(ArraySizeHelper(array))) > > In file included from ../third_party/perf_data_converter/src/quipper/pe= rf_parser.h:21:0, > > from ../sample_reader.h:18, > > from ../profile.h:15, > > from ../profile.cc:5: > > ../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: > > fatal error: perf_stat.pb.h: No such file or directory #include "perf_s= tat.pb.h" > > ^~~~~~~~~~~~~~~~ > > compilation terminated. > > > > What is supposed to generate perf_stat.pb.h? > > > > Thanks, > > > > Eugene > > > > From: Wei Mi > > Sent: Monday, May 10, 2021 4:47 PM > > To: Andi Kleen > > Cc: Hongtao Yu ; Xinliang David Li > > ; Jan Hubicka ; > > mailto:gcc@gcc.gnu.org; Eugene Rozenfeld > > ; Wenlei He > > > > Subject: [EXTERNAL] Re: State of AutoFDO in GCC > > > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgit= hub.com%2Fgoogle%2Fautofdo&data=3D04%7C01%7CEugene.Rozenfeld%40microsof= t.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af91ab2d7cd011db47= %7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL= CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3Da9gGf48MYY= 2jzbSucOAlhsmdLuYt%2BcM8V5hvGoA39Ms%3D&reserved=3D0 has been updated. N= ow create_gcov/dump_gcov are added back and can be built separately. > > > > Please look at "2.2 Build autofdo tool for gcc" in > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgit= h > > ub.com%2Fgoogle%2Fautofdo%23readme&data=3D04%7C01%7CEugene.Rozenfel= d > > %40microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3d8 > > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1 > > 000&sdata=3DEwHtN1OqRmBwL6qMgTc1lAU6cz5%2BJzQZxpd5nos11FM%3D&re= s > > erved=3D0 On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen > > wrote: > > > > > > On Mon, Apr 26, 2021 at 06:40:56PM +0000, Hongtao Yu wrote: > > > > Andi, thanks for pointing out the perf script issues. Can you pl= ease > > > > elaborate a bit on the exact issue you have seen? We've been usi= ng > > > > specific output of perf script such as mmap, LBR and callstack e= vents > > > > filtered by process id. It works fine so far but may certainly h= it issues > > > > in the future with extended uses. > > > > > > Okay I took a look at the latest autofdo now. It seems to be > > > basically a LLVM project now that depends on LLVM to even build with > > > all kinds of dependency hell on some old LLVM version and other packa= ges. > > > > > > I guess gcc will really need a replacement that doesn't pull in all > > > of LLVM if it wants to continue supporting autofdo. > > > > > > I'm myself unable to build now. > > > > > > I'm using the old version I had a git fork of and that was before > > > all of this. I added a patch to make it work with the latest perf by > > > ignoring increased perf_attr and unknown perf events. > > > > > > Honza please use > > > > > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fg= i > > > thub.com%2Fandikleen%2Fautofdo&data=3D04%7C01%7CEugene.Rozenfeld%= 4 > > > 0microsoft.com%7C1bcb2fda4fce4f173c1808d91f30d1fc%7C72f988bf86f141af > > > 91ab2d7cd011db47%7C1%7C0%7C637575116816287200%7CUnknown%7CTWFpbGZsb3 > > > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > > > %7C1000&sdata=3DDilu5PUjpAbA6e6d6GBd5idoKLAMMr7H5eCve%2FGdC8g%3D&= a > > > mp;reserved=3D0 -b perf-future > > > > > > for testing. > > > > > > -Andi > > >