[PATCH V3] gprofng: a new GNU profiler

[PATCH V3] gprofng: a new GNU profiler

[Vladimir Mezentsev]

Hi people!


09/01/2021: Changes from V2
===========================

   - Renamed experiment.h to gp-experiment.h (Hannes Domani reported 
problem on Windows).
   - Fixed regression in build on ARM.
   - Improved gprofng/testsuite.


Where to get gprofng
====================

Due to the size of the contribution, we thought it would be better to 
submit it in the form
of a git branch, instead of a regular emailed patch series.

Repository: https://www.github.com/oracle/binutils-gdb
Branch:     oracle/gprofng-v3


Build gprofng
=============

For x86_64/i686, gprofng should be built twice to support profiling for 
both 32-bit and 64-bit applications:
    configure --enable-shared --libdir=PREFIX/lib64 && make install
    configure --enable-shared CC='gcc -m32' CXX='g++ -m32' 
--target=i686-pc-linux-gnu --disable-gprofng-tools && make install
For ARM:
    configure --enable-shared && make install

To support java profiling, configure build with -with-jdk=PATH_TO_JDK if 
javac is not in your PATH.



o8/25/2021: Changes from V1
===========================

We addressed all of Joseph Myers's remarks about gprofng Makefiles and 
configure scripts:
   - Report an error when bison is absent, or a version of bison is too old.
   - Added preload_libdirs in ${prefix}/etc/gprofng.rc to configure the 
location of the
     gprofng libraries (this was hard-coded previously).

We also took note of Jim Wilson’s comment about the naming of files in 
src/machinemodels.
In particular, file m5.ermm and his suggestion to make it clear this is 
for a SPARC
processor. For example sparc-m5.ermm.

This is going to be part of a bigger overhaul of the naming convention 
for these files.
The most obvious choice is to include the product name, but we 
definitely appreciate any
suggestions this group may have.




08/11/2021: a new GNU profiler
==============================

In this submission we are contributing a new profiler to the GNU binary
utilities, called gprofng (for GNU profiler, next generation).

Why a new profiler?
===================

The GNU profiler, gprof, works well enough in many cases. However, it
hasn't aged well and it is not that very well suited for profiling
modern-world applications. Examples of its limitations are lack of support
for profiling multithreaded programs, and shared objects. Both are
ubiquitous nowadays.

Main characteristics of gprofng
===============================

gprofng supports profiling C, C++ and Java programs. Unlike the old
gprof, it doesn't require to build annotated versions of the programs.
Profiling "production" binaries should work just fine.

Another distinguishing feature of gprofng is the support for various filters
that allow the user to easily drill deeper into an area of interest.

The profiler is commanded through a driver program called `gprofng'.
This driver supports the following sub-commands:

gpronfg collect app EXECUTABLE

This runs EXECUTABLE and collects application performance data.

gprofng display text EXPERIMENT

This runs a client command-line interface that provides access to the
collected performance data stored in the experiment directory.

gprofng display html EXPERIMENT

This generates an HTML report from the collected performance data.
stored in the experiment directory.

gprofng display src OBJECT-FILE

This displays source (if available) or disassembly interleaved
with the source code.

gprofng archive EXPERIMENT

Archive the associated application binaries, load objects and source
files in an existing experiment directory to make it self contained.

There is also an extensive graphical user interface (written in Java)
that displays and analyzes gprofng collected data in a very sophisticated
way. We plan to release this GUI as a separate project.

While WIP, we would like to share some screenshots of the current
development version. These show the following:

pic1.png - a flame graph:
https://jemarch.net/gprofng-pics/pic1.png

pic2.png - color coded call stacks as a function of time ("the timeline"):
https://jemarch.net/gprofng-pics/pic2.png

pic3.png - zoom in on the timeline and adapt colors to identify details:
https://jemarch.net/gprofng-pics/pic3.png

pic4.png - compare two mulithreaded profiles:
https://jemarch.net/gprofng-pics/pic4.png

Some notes on the implementation
================================

- The gp-display-html tool is written in Perl. All other components are
written in C/C++.

- gprofng sources are mostly contained in a new top-level directory
gprofng/ that in turn contains:

+ src/ contains the source code of the gp-* programs and libgprofng.

+ libcollector/ contains the sources of libcollector.

+ common/ contains a few source files that are used by both the gp-*
utilities and libcollector.

+ doc/ contains the Texinfo sources for the gprofng manual.

+ testsuite/ contains the gprofng testsuite.

Three installed header files are distributed in the top-level include/
directory. These are libcollector.h, libfcollector.h, and
collectorAPI.h.

- Currently gprofng supports profiling programs in GNU/Linux systems
running on x86_64 and aarch64 hardware. It is possible to add support
for additional architectures.

- The tools come with a set of man pages. They are generated upon
installation and can be found in the installation directory under
share/man/man1.

Platform support
================

The basic profiling features are supported on most processors from
Intel. Regarding AMD we did not yet test on their recent EPYC
processors, but do not expect serious issues. We also support the Arm
processors as used in systems from Ampere.

Hardware event counters, which are optional and used by gprofng in
advanced profiling, are supported for many modern Intel and AMD
processors. If a particular processor is not supported, a warning
message will be issued when trying to run an event counter experiment.

This code has been developed and tested on Oracle Linux 8 with the
latest GNU toolchain from the sourceware git repos.

Structure of the patch series
=============================

The first patch is preparatory and makes the x86 disassembler in opcodes
to be thread-safe. This is so it can be used by gprofng.

The second patch is the implementation of gprofng proper. This includes
source code for the libraries (libcollector, libgprofng) and the
utilities (gp-collect, etc).

In this patch there are also updates to the corresponding build machinery
(e.g. configure.ac, Makefile.def, plus binutils/MAINTAINERS to cover 
gprofng)

The third patch adds a testsuite in gprofng/testsuite.

The fourth patch adds a Texinfo manual for gprofng. The manual is still
WIP but already provides a tutorial-like introduction to the tools.


Limitations
===========

The gp-display-html tool is present, and can be executed, but it is not
functional yet. Full support for this tool is expected to be delivered
in a future patch.

Requirements
===========

In order to successfully build gprofng, the following versions of
external components are required:

- Bison 3.7.5, or higher
- Texinfo 6.7, or higher
- Java include files (--with-jdk=PATH) if java profiling should be enabled

Maintenance
===========

We are of course volunteering to maintain gprofng once it is
incorporated into the main binutils distribution.
We suggest having feedback and discussion in this mail thread.

Re: [PATCH V3] gprofng: a new GNU profiler

[Sergei Trofimovich]

On Wed, 1 Sep 2021 11:45:01 -0700
Vladimir Mezentsev via Binutils <binutils@sourceware.org> wrote:

> Hi people!
> 
> 
> 09/01/2021: Changes from V2
> ===========================
> 
>    - Renamed experiment.h to gp-experiment.h (Hannes Domani reported 
> problem on Windows).
>    - Fixed regression in build on ARM.
>    - Improved gprofng/testsuite.
> 
> 
> Where to get gprofng
> ====================
> 
> Due to the size of the contribution, we thought it would be better to 
> submit it in the form
> of a git branch, instead of a regular emailed patch series.
> 
> Repository: https://www.github.com/oracle/binutils-gdb
> Branch:     oracle/gprofng-v3

Hi Vladimir! [Months later] I noticed that gprofng is doing a bit of
indirection in format strings.

Many distributions (including NixOS) build projects with
    CFLAGS/CXXFLAGS=-Werror=format-security
to avoid trivial format string vulnerabilities.

Today's binutils-gdb master fails to build under such a setup.
The rest of binutils is generally -Werror=format-security clean.

Do you think it's something gprofng could support or workaround?

Currently I see the following failures:

../../../gprofng/src/Print.cc: In function 'int print_one_visible(FILE*, char*, char*, char*, TValue*, int)':
../../../gprofng/src/Print.cc:547:35: error: format not a string literal and no format arguments [-Werror=format-security]
  547 |  nc = fprintf (out_file, fmt_real0);
      |                                   ^
../../../gprofng/src/Print.cc:574:35: error: format not a string literal and no format arguments [-Werror=format-security]
  574 |  nc = fprintf (out_file, fmt_real0);
      |                                   ^
../../../gprofng/src/Print.cc: In function 'int print_one_tvisible(FILE*, char*, char*, TValue*, int, int)':
../../../gprofng/src/Print.cc:596:38: error: format not a string literal and no format arguments [-Werror=format-security]
  596 |     nc = fprintf (out_file, fmt_real0);
      |                                      ^
../../../gprofng/src/Print.cc: In function 'void print_one(FILE*, Hist_data*, Hist_data::HistItem*, char**, char**, char**, MetricList*, Metric::HistMetric*, char*, Histable::NameFormat)':
../../../gprofng/src/Print.cc:692:31: error: format not a string literal and no format arguments [-Werror=format-security]
  692 |      nc += fprintf (out_file, stmp);
      |                               ^~~~
../../../gprofng/src/Print.cc: In member function 'void er_print_histogram::dump_detail(int)':
../../../gprofng/src/Print.cc:1044:24: error: format not a string literal and no format arguments [-Werror=format-security]
 1044 |     fprintf (out_file, fmt_real1);
      |                        ^~~~~~~~~
../../../gprofng/src/Print.cc:1051:24: error: format not a string literal and no format arguments [-Werror=format-security]
 1051 |     fprintf (out_file, fmt_int1);
      |                        ^~~~~~~~
../../../gprofng/src/Print.cc:1062:23: error: format not a string literal and no format arguments [-Werror=format-security]
 1062 |    fprintf (out_file, fmt_real1);
      |                       ^~~~~~~~~
../../../gprofng/src/Print.cc:1066:28: error: format not a string literal and no format arguments [-Werror=format-security]
 1066 |         fprintf (out_file, fmt_long1);
      |                            ^~~~~~~~~
../../../gprofng/src/Print.cc:1105:24: error: format not a string literal and no format arguments [-Werror=format-security]
 1105 |     fprintf (out_file, fmt_real1);
      |                        ^~~~~~~~~
../../../gprofng/src/Print.cc: In member function 'void er_print_experiment::overview_value(Value*, ValueTag, double)':
../../../gprofng/src/Print.cc:2616:26: error: format not a string literal and no format arguments [-Werror=format-security]
 2616 |       fprintf (out_file, fmt3);
      |                          ^~~~
cc1plus: some warnings being treated as errors
make[5]: *** [Makefile:870: Print.lo] Error 1
depbase=`echo parse.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/nix/store/w61p7kw12h9ppxlfy5lsp2k32hcrwg4z-bash-5.1-p16/bin/bash ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I..  -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd   -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2     -MT parse.lo -MD -MP -MF $depbase.Tpo -c -o parse.lo ../../../gprofng/src/parse.cc &&\
mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I.. -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2 -MT parse.lo -MD -MP -MF .deps/parse.Tpo -c ../../../gprofng/src/parse.cc -o parse.o
../../../gprofng/src/parse.cc: In member function 'virtual int Experiment::process_Linux_kernel_cmd(hrtime_t)':
../../../gprofng/src/parse.cc:909:40: error: format not a string literal and no format arguments [-Werror=format-security]
  909 |    sprintf (last_mod_name, mod_name_ptr);
      |                                        ^

Thanks!

-- 

  Sergei

Re: [PATCH V3] gprofng: a new GNU profiler

[Vladimir Mezentsev]

On 5/21/22 01:30, Sergei Trofimovich wrote:
> On Wed, 1 Sep 2021 11:45:01 -0700
> Vladimir Mezentsev via Binutils <binutils@sourceware.org> wrote:
>
>> Hi people!
>>
>>
>> 09/01/2021: Changes from V2
>> ===========================
>>
>>     - Renamed experiment.h to gp-experiment.h (Hannes Domani reported
>> problem on Windows).
>>     - Fixed regression in build on ARM.
>>     - Improved gprofng/testsuite.
>>
>>
>> Where to get gprofng
>> ====================
>>
>> Due to the size of the contribution, we thought it would be better to
>> submit it in the form
>> of a git branch, instead of a regular emailed patch series.
>>
>> Repository: https://www.github.com/oracle/binutils-gdb
>> Branch:     oracle/gprofng-v3
> Hi Vladimir! [Months later] I noticed that gprofng is doing a bit of
> indirection in format strings.
>
> Many distributions (including NixOS) build projects with
>      CFLAGS/CXXFLAGS=-Werror=format-security
> to avoid trivial format string vulnerabilities.
>
> Today's binutils-gdb master fails to build under such a setup.
> The rest of binutils is generally -Werror=format-security clean.
>
> Do you think it's something gprofng could support or workaround?


Hi Sergei,

H.J. Lu failed  Bug 28968 - gprofng doesn't build with 
-Werror=format-security

We plan to fix this soon.

Thank you,
-Vladimir


>
> Currently I see the following failures:
>
> ../../../gprofng/src/Print.cc: In function 'int print_one_visible(FILE*, char*, char*, char*, TValue*, int)':
> ../../../gprofng/src/Print.cc:547:35: error: format not a string literal and no format arguments [-Werror=format-security]
>    547 |  nc = fprintf (out_file, fmt_real0);
>        |                                   ^
> ../../../gprofng/src/Print.cc:574:35: error: format not a string literal and no format arguments [-Werror=format-security]
>    574 |  nc = fprintf (out_file, fmt_real0);
>        |                                   ^
> ../../../gprofng/src/Print.cc: In function 'int print_one_tvisible(FILE*, char*, char*, TValue*, int, int)':
> ../../../gprofng/src/Print.cc:596:38: error: format not a string literal and no format arguments [-Werror=format-security]
>    596 |     nc = fprintf (out_file, fmt_real0);
>        |                                      ^
> ../../../gprofng/src/Print.cc: In function 'void print_one(FILE*, Hist_data*, Hist_data::HistItem*, char**, char**, char**, MetricList*, Metric::HistMetric*, char*, Histable::NameFormat)':
> ../../../gprofng/src/Print.cc:692:31: error: format not a string literal and no format arguments [-Werror=format-security]
>    692 |      nc += fprintf (out_file, stmp);
>        |                               ^~~~
> ../../../gprofng/src/Print.cc: In member function 'void er_print_histogram::dump_detail(int)':
> ../../../gprofng/src/Print.cc:1044:24: error: format not a string literal and no format arguments [-Werror=format-security]
>   1044 |     fprintf (out_file, fmt_real1);
>        |                        ^~~~~~~~~
> ../../../gprofng/src/Print.cc:1051:24: error: format not a string literal and no format arguments [-Werror=format-security]
>   1051 |     fprintf (out_file, fmt_int1);
>        |                        ^~~~~~~~
> ../../../gprofng/src/Print.cc:1062:23: error: format not a string literal and no format arguments [-Werror=format-security]
>   1062 |    fprintf (out_file, fmt_real1);
>        |                       ^~~~~~~~~
> ../../../gprofng/src/Print.cc:1066:28: error: format not a string literal and no format arguments [-Werror=format-security]
>   1066 |         fprintf (out_file, fmt_long1);
>        |                            ^~~~~~~~~
> ../../../gprofng/src/Print.cc:1105:24: error: format not a string literal and no format arguments [-Werror=format-security]
>   1105 |     fprintf (out_file, fmt_real1);
>        |                        ^~~~~~~~~
> ../../../gprofng/src/Print.cc: In member function 'void er_print_experiment::overview_value(Value*, ValueTag, double)':
> ../../../gprofng/src/Print.cc:2616:26: error: format not a string literal and no format arguments [-Werror=format-security]
>   2616 |       fprintf (out_file, fmt3);
>        |                          ^~~~
> cc1plus: some warnings being treated as errors
> make[5]: *** [Makefile:870: Print.lo] Error 1
> depbase=`echo parse.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
> /nix/store/w61p7kw12h9ppxlfy5lsp2k32hcrwg4z-bash-5.1-p16/bin/bash ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I..  -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd   -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2     -MT parse.lo -MD -MP -MF $depbase.Tpo -c -o parse.lo ../../../gprofng/src/parse.cc &&\
> mv -f $depbase.Tpo $depbase.Plo
> libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../gprofng/src -I.. -U_ASM -DLOCALEDIR=\"/mnt/archive/big/git/binutils-gdb/outputs/out/share/locale\" -I.. -I../../../gprofng/src -I../../../gprofng/src/../common -I../../../gprofng/src/../../include -I../../../gprofng/src/../../opcodes -I../../bfd -I../../../gprofng/src/../../bfd -Wall -pthread -Wno-format-truncation -Wno-switch -g -O2 -MT parse.lo -MD -MP -MF .deps/parse.Tpo -c ../../../gprofng/src/parse.cc -o parse.o
> ../../../gprofng/src/parse.cc: In member function 'virtual int Experiment::process_Linux_kernel_cmd(hrtime_t)':
> ../../../gprofng/src/parse.cc:909:40: error: format not a string literal and no format arguments [-Werror=format-security]
>    909 |    sprintf (last_mod_name, mod_name_ptr);
>        |                                        ^
>
> Thanks!
>