* [PATCH V3] gprofng: a new GNU profiler [not found] <047143ae-be5e-9eec-237f-ad9ffe3795c0@oracle.com> @ 2021-09-01 18:45 ` Vladimir Mezentsev 2022-05-21 8:30 ` Sergei Trofimovich 0 siblings, 1 reply; 3+ messages in thread From: Vladimir Mezentsev @ 2021-09-01 18:45 UTC (permalink / raw) To: binutils Hi people! 09/01/2021: Changes from V2 =========================== - Renamed experiment.h to gp-experiment.h (Hannes Domani reported problem on Windows). - Fixed regression in build on ARM. - Improved gprofng/testsuite. Where to get gprofng ==================== Due to the size of the contribution, we thought it would be better to submit it in the form of a git branch, instead of a regular emailed patch series. Repository: https://www.github.com/oracle/binutils-gdb Branch: oracle/gprofng-v3 Build gprofng ============= For x86_64/i686, gprofng should be built twice to support profiling for both 32-bit and 64-bit applications: configure --enable-shared --libdir=PREFIX/lib64 && make install configure --enable-shared CC='gcc -m32' CXX='g++ -m32' --target=i686-pc-linux-gnu --disable-gprofng-tools && make install For ARM: configure --enable-shared && make install To support java profiling, configure build with -with-jdk=PATH_TO_JDK if javac is not in your PATH. o8/25/2021: Changes from V1 =========================== We addressed all of Joseph Myers's remarks about gprofng Makefiles and configure scripts: - Report an error when bison is absent, or a version of bison is too old. - Added preload_libdirs in ${prefix}/etc/gprofng.rc to configure the location of the gprofng libraries (this was hard-coded previously). We also took note of Jim Wilson’s comment about the naming of files in src/machinemodels. In particular, file m5.ermm and his suggestion to make it clear this is for a SPARC processor. For example sparc-m5.ermm. This is going to be part of a bigger overhaul of the naming convention for these files. The most obvious choice is to include the product name, but we definitely appreciate any suggestions this group may have. 08/11/2021: a new GNU profiler ============================== In this submission we are contributing a new profiler to the GNU binary utilities, called gprofng (for GNU profiler, next generation). Why a new profiler? =================== The GNU profiler, gprof, works well enough in many cases. However, it hasn't aged well and it is not that very well suited for profiling modern-world applications. Examples of its limitations are lack of support for profiling multithreaded programs, and shared objects. Both are ubiquitous nowadays. Main characteristics of gprofng =============================== gprofng supports profiling C, C++ and Java programs. Unlike the old gprof, it doesn't require to build annotated versions of the programs. Profiling "production" binaries should work just fine. Another distinguishing feature of gprofng is the support for various filters that allow the user to easily drill deeper into an area of interest. The profiler is commanded through a driver program called `gprofng'. This driver supports the following sub-commands: gpronfg collect app EXECUTABLE This runs EXECUTABLE and collects application performance data. gprofng display text EXPERIMENT This runs a client command-line interface that provides access to the collected performance data stored in the experiment directory. gprofng display html EXPERIMENT This generates an HTML report from the collected performance data. stored in the experiment directory. gprofng display src OBJECT-FILE This displays source (if available) or disassembly interleaved with the source code. gprofng archive EXPERIMENT Archive the associated application binaries, load objects and source files in an existing experiment directory to make it self contained. There is also an extensive graphical user interface (written in Java) that displays and analyzes gprofng collected data in a very sophisticated way. We plan to release this GUI as a separate project. While WIP, we would like to share some screenshots of the current development version. These show the following: pic1.png - a flame graph: https://jemarch.net/gprofng-pics/pic1.png pic2.png - color coded call stacks as a function of time ("the timeline"): https://jemarch.net/gprofng-pics/pic2.png pic3.png - zoom in on the timeline and adapt colors to identify details: https://jemarch.net/gprofng-pics/pic3.png pic4.png - compare two mulithreaded profiles: https://jemarch.net/gprofng-pics/pic4.png Some notes on the implementation ================================ - The gp-display-html tool is written in Perl. All other components are written in C/C++. - gprofng sources are mostly contained in a new top-level directory gprofng/ that in turn contains: + src/ contains the source code of the gp-* programs and libgprofng. + libcollector/ contains the sources of libcollector. + common/ contains a few source files that are used by both the gp-* utilities and libcollector. + doc/ contains the Texinfo sources for the gprofng manual. + testsuite/ contains the gprofng testsuite. Three installed header files are distributed in the top-level include/ directory. These are libcollector.h, libfcollector.h, and collectorAPI.h. - Currently gprofng supports profiling programs in GNU/Linux systems running on x86_64 and aarch64 hardware. It is possible to add support for additional architectures. - The tools come with a set of man pages. They are generated upon installation and can be found in the installation directory under share/man/man1. Platform support ================ The basic profiling features are supported on most processors from Intel. Regarding AMD we did not yet test on their recent EPYC processors, but do not expect serious issues. We also support the Arm processors as used in systems from Ampere. Hardware event counters, which are optional and used by gprofng in advanced profiling, are supported for many modern Intel and AMD processors. If a particular processor is not supported, a warning message will be issued when trying to run an event counter experiment. This code has been developed and tested on Oracle Linux 8 with the latest GNU toolchain from the sourceware git repos. Structure of the patch series ============================= The first patch is preparatory and makes the x86 disassembler in opcodes to be thread-safe. This is so it can be used by gprofng. The second patch is the implementation of gprofng proper. This includes source code for the libraries (libcollector, libgprofng) and the utilities (gp-collect, etc). In this patch there are also updates to the corresponding build machinery (e.g. configure.ac, Makefile.def, plus binutils/MAINTAINERS to cover gprofng) The third patch adds a testsuite in gprofng/testsuite. The fourth patch adds a Texinfo manual for gprofng. The manual is still WIP but already provides a tutorial-like introduction to the tools. Limitations =========== The gp-display-html tool is present, and can be executed, but it is not functional yet. Full support for this tool is expected to be delivered in a future patch. Requirements =========== In order to successfully build gprofng, the following versions of external components are required: - Bison 3.7.5, or higher - Texinfo 6.7, or higher - Java include files (--with-jdk=PATH) if java profiling should be enabled Maintenance =========== We are of course volunteering to maintain gprofng once it is incorporated into the main binutils distribution. We suggest having feedback and discussion in this mail thread. ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH V3] gprofng: a new GNU profiler 2021-09-01 18:45 ` [PATCH V3] gprofng: a new GNU profiler Vladimir Mezentsev @ 2022-05-21 8:30 ` Sergei Trofimovich 2022-05-24 3:09 ` Vladimir Mezentsev 0 siblings, 1 reply; 3+ messages in thread From: Sergei Trofimovich @ 2022-05-21 8:30 UTC (permalink / raw) To: Vladimir Mezentsev via Binutils On Wed, 1 Sep 2021 11:45:01 -0700 Vladimir Mezentsev via Binutils <binutils@sourceware.org> wrote: > Hi people! > > > 09/01/2021: Changes from V2 > =========================== > > - Renamed experiment.h to gp-experiment.h (Hannes Domani reported > problem on Windows). > - Fixed regression in build on ARM. > - Improved gprofng/testsuite. > > > Where to get gprofng > ==================== > > Due to the size of the contribution, we thought it would be better to > submit it in the form > of a git branch, instead of a regular emailed patch series. > > Repository: https://www.github.com/oracle/binutils-gdb > Branch: oracle/gprofng-v3 Hi Vladimir! [Months later] I noticed that gprofng is doing a bit of indirection in format strings. Many distributions (including NixOS) build projects with CFLAGS/CXXFLAGS=-Werror=format-security to avoid trivial format string vulnerabilities. Today's binutils-gdb master fails to build under such a setup. The rest of binutils is generally -Werror=format-security clean. Do you think it's something gprofng could support or workaround? Currently I see the following failures: ../../../gprofng/src/Print.cc: In function 'int print_one_visible(FILE*, char*, char*, char*, TValue*, int)': ../../../gprofng/src/Print.cc:547:35: error: format not a string literal and no format arguments [-Werror=format-security] 547 | nc = fprintf (out_file, fmt_real0); | ^ ../../../gprofng/src/Print.cc:574:35: error: format not a string literal and no format arguments [-Werror=format-security] 574 | nc = fprintf (out_file, fmt_real0); | ^ ../../../gprofng/src/Print.cc: In function 'int print_one_tvisible(FILE*, char*, char*, TValue*, int, int)': ../../../gprofng/src/Print.cc:596:38: error: format not a string literal and no format arguments [-Werror=format-security] 596 | nc = fprintf (out_file, fmt_real0); | ^ ../../../gprofng/src/Print.cc: In function 'void print_one(FILE*, Hist_data*, Hist_data::HistItem*, char**, char**, char**, MetricList*, Metric::HistMetric*, char*, Histable::NameFormat)': ../../../gprofng/src/Print.cc:692:31: error: format not a string literal and no format arguments [-Werror=format-security] 692 | nc += fprintf (out_file, stmp); | ^~~~ ../../../gprofng/src/Print.cc: In member function 'void er_print_histogram::dump_detail(int)': ../../../gprofng/src/Print.cc:1044:24: error: format not a string literal and no format arguments [-Werror=format-security] 1044 | fprintf (out_file, fmt_real1); | ^~~~~~~~~ ../../../gprofng/src/Print.cc:1051:24: error: format not a string literal and no format arguments [-Werror=format-security] 1051 | fprintf (out_file, fmt_int1); | ^~~~~~~~ ../../../gprofng/src/Print.cc:1062:23: error: format not a string literal and no format arguments [-Werror=format-security] 1062 | fprintf (out_file, fmt_real1); | ^~~~~~~~~ ../../../gprofng/src/Print.cc:1066:28: error: format not a string literal and no format arguments [-Werror=format-security] 1066 | fprintf (out_file, fmt_long1); | ^~~~~~~~~ ../../../gprofng/src/Print.cc:1105:24: error: format not a string literal and no format arguments [-Werror=format-security] 1105 | fprintf (out_file, fmt_real1); | ^~~~~~~~~ ../../../gprofng/src/Print.cc: In member function 'void er_print_experiment::overview_value(Value*, ValueTag, double)': ../../../gprofng/src/Print.cc:2616:26: error: format not a string literal and no format arguments [-Werror=format-security] 2616 | fprintf (out_file, fmt3); | ^~~~ cc1plus: some warnings being treated as errors make[5]: *** [Makefile:870: Print.lo] Error 1 depbase=`echo parse.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\ /nix/store/w61p7kw12h9ppxlfy5lsp2k32hcrwg4z-bash-5.1-p16/bin/bash ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I.. -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2 -MT parse.lo -MD -MP -MF $depbase.Tpo -c -o parse.lo ../../../gprofng/src/parse.cc &&\ mv -f $depbase.Tpo $depbase.Plo libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I.. -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2 -MT parse.lo -MD -MP -MF .deps/parse.Tpo -c ../../../gprofng/src/parse.cc -o parse.o ../../../gprofng/src/parse.cc: In member function 'virtual int Experiment::process_Linux_kernel_cmd(hrtime_t)': ../../../gprofng/src/parse.cc:909:40: error: format not a string literal and no format arguments [-Werror=format-security] 909 | sprintf (last_mod_name, mod_name_ptr); | ^ Thanks! -- Sergei ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH V3] gprofng: a new GNU profiler 2022-05-21 8:30 ` Sergei Trofimovich @ 2022-05-24 3:09 ` Vladimir Mezentsev 0 siblings, 0 replies; 3+ messages in thread From: Vladimir Mezentsev @ 2022-05-24 3:09 UTC (permalink / raw) To: Sergei Trofimovich, Vladimir Mezentsev via Binutils On 5/21/22 01:30, Sergei Trofimovich wrote: > On Wed, 1 Sep 2021 11:45:01 -0700 > Vladimir Mezentsev via Binutils <binutils@sourceware.org> wrote: > >> Hi people! >> >> >> 09/01/2021: Changes from V2 >> =========================== >> >> - Renamed experiment.h to gp-experiment.h (Hannes Domani reported >> problem on Windows). >> - Fixed regression in build on ARM. >> - Improved gprofng/testsuite. >> >> >> Where to get gprofng >> ==================== >> >> Due to the size of the contribution, we thought it would be better to >> submit it in the form >> of a git branch, instead of a regular emailed patch series. >> >> Repository: https://www.github.com/oracle/binutils-gdb >> Branch: oracle/gprofng-v3 > Hi Vladimir! [Months later] I noticed that gprofng is doing a bit of > indirection in format strings. > > Many distributions (including NixOS) build projects with > CFLAGS/CXXFLAGS=-Werror=format-security > to avoid trivial format string vulnerabilities. > > Today's binutils-gdb master fails to build under such a setup. > The rest of binutils is generally -Werror=format-security clean. > > Do you think it's something gprofng could support or workaround? Hi Sergei, H.J. Lu failed Bug 28968 - gprofng doesn't build with -Werror=format-security We plan to fix this soon. Thank you, -Vladimir > > Currently I see the following failures: > > ../../../gprofng/src/Print.cc: In function 'int print_one_visible(FILE*, char*, char*, char*, TValue*, int)': > ../../../gprofng/src/Print.cc:547:35: error: format not a string literal and no format arguments [-Werror=format-security] > 547 | nc = fprintf (out_file, fmt_real0); > | ^ > ../../../gprofng/src/Print.cc:574:35: error: format not a string literal and no format arguments [-Werror=format-security] > 574 | nc = fprintf (out_file, fmt_real0); > | ^ > ../../../gprofng/src/Print.cc: In function 'int print_one_tvisible(FILE*, char*, char*, TValue*, int, int)': > ../../../gprofng/src/Print.cc:596:38: error: format not a string literal and no format arguments [-Werror=format-security] > 596 | nc = fprintf (out_file, fmt_real0); > | ^ > ../../../gprofng/src/Print.cc: In function 'void print_one(FILE*, Hist_data*, Hist_data::HistItem*, char**, char**, char**, MetricList*, Metric::HistMetric*, char*, Histable::NameFormat)': > ../../../gprofng/src/Print.cc:692:31: error: format not a string literal and no format arguments [-Werror=format-security] > 692 | nc += fprintf (out_file, stmp); > | ^~~~ > ../../../gprofng/src/Print.cc: In member function 'void er_print_histogram::dump_detail(int)': > ../../../gprofng/src/Print.cc:1044:24: error: format not a string literal and no format arguments [-Werror=format-security] > 1044 | fprintf (out_file, fmt_real1); > | ^~~~~~~~~ > ../../../gprofng/src/Print.cc:1051:24: error: format not a string literal and no format arguments [-Werror=format-security] > 1051 | fprintf (out_file, fmt_int1); > | ^~~~~~~~ > ../../../gprofng/src/Print.cc:1062:23: error: format not a string literal and no format arguments [-Werror=format-security] > 1062 | fprintf (out_file, fmt_real1); > | ^~~~~~~~~ > ../../../gprofng/src/Print.cc:1066:28: error: format not a string literal and no format arguments [-Werror=format-security] > 1066 | fprintf (out_file, fmt_long1); > | ^~~~~~~~~ > ../../../gprofng/src/Print.cc:1105:24: error: format not a string literal and no format arguments [-Werror=format-security] > 1105 | fprintf (out_file, fmt_real1); > | ^~~~~~~~~ > ../../../gprofng/src/Print.cc: In member function 'void er_print_experiment::overview_value(Value*, ValueTag, double)': > ../../../gprofng/src/Print.cc:2616:26: error: format not a string literal and no format arguments [-Werror=format-security] > 2616 | fprintf (out_file, fmt3); > | ^~~~ > cc1plus: some warnings being treated as errors > make[5]: *** [Makefile:870: Print.lo] Error 1 > depbase=`echo parse.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\ > /nix/store/w61p7kw12h9ppxlfy5lsp2k32hcrwg4z-bash-5.1-p16/bin/bash ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I.. -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2 -MT parse.lo -MD -MP -MF $depbase.Tpo -c -o parse.lo ../../../gprofng/src/parse.cc &&\ > mv -f $depbase.Tpo $depbase.Plo > libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I.. -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2 -MT parse.lo -MD -MP -MF .deps/parse.Tpo -c ../../../gprofng/src/parse.cc -o parse.o > ../../../gprofng/src/parse.cc: In member function 'virtual int Experiment::process_Linux_kernel_cmd(hrtime_t)': > ../../../gprofng/src/parse.cc:909:40: error: format not a string literal and no format arguments [-Werror=format-security] > 909 | sprintf (last_mod_name, mod_name_ptr); > | ^ > > Thanks! > ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-05-24 3:09 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <047143ae-be5e-9eec-237f-ad9ffe3795c0@oracle.com> 2021-09-01 18:45 ` [PATCH V3] gprofng: a new GNU profiler Vladimir Mezentsev 2022-05-21 8:30 ` Sergei Trofimovich 2022-05-24 3:09 ` Vladimir Mezentsev
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).