[PATCH V4] gprofng: a new GNU profiler

[PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

Hi people!

Here is a slightly updated patch for the gprofng utility. We have fixed
some issues we have noticed while testing and added first support for
hardware event counter profiling on Arm.

We think gprofng will be a good addition to binutils and hope that
we can move forward. In particular we'd like to ask for the review
process to start.

This is a high level description of the various components in gprofng
with the purpose of helping reviewers:

gprofng tools:
     gprofng         - the driver program
     gp-collect-app  - create experiment with program performance data
     gp-display-text - print an ASCII report from one or more 
performance experiments
     gp-display-html - generate an HTML based directory structure from 
an experiment
     gp-archive      - archive the binaries and sources for experiment
     gp-display-src  - print a source or dissasembly listing for the 
load object

     lib/libgprofng.so - read experiments.
                         libgprofng.so is linked with each gprofng tools.

gprofng collector libraries:
     lib/libgp-collector.so - gprofng collector library
     lib/libgp-heap.so      - Heap tracing library
     lib/libgp-iotrace.so   - I/O tracing library
     lib/libgp-sync.so      - Synchronization tracing library


gprofng source tree:
     gprofng/doc/*          - gprofng documentation
     gprofng/libcollector/* - sources of libgp-collector.so, libgp-heap.so,
                              libgp-iotrace.so and libgp-sync.so
     gprofng/src/*          - sources of all gprofng tools and libgprofng.so
     gprofng/common/*       - these files are common components used
                              when building the gprofng libraries


Known problems
==============
- To convert cycles to time, we need the frequency the core is running at.
   On Arm we read the cntrfrq_e10 register, but this sometimes returns an
   incorrect value. For the time being, it may be safest to ignore the
   cycles derived timings on Arm.

- The support for hardware event counters on recent processors is limited.
   You can use the “gprofng collect app -h” command to check for support.
   In case a specific processor is not supported, a warning message will be
   printed.  Otherwise a list with the supported events is printed.


How to get gprofng
==================

Due to the size of the contribution, we thought it would be better to submit
it in the form of a git branch, instead of a regular emailed patch series.

Repository:https://www.github.com/oracle/binutils-gdb
Branch:     oracle/gprofng-v4


Build gprofng
=============

For x86_64/i686, gprofng should be built twice to support profiling for
both 32-bit and 64-bit applications:
   PREFIX=<YOUR_INSTALL_DIR>
   configure --enable-shared --prefix=$PREFIX --libdir=$PREFIX/lib64 && 
make && make install
   configure --enable-shared --prefix=$PREFIX CC='gcc -m32' CXX='g++ -m32' \
          --target=i686-pc-linux-gnu --disable-gprofng-tools && make && 
make install

For ARM:
   configure --enable-shared --prefix=$PREFIX && make install

To support java profiling, configure the build with -with-jdk=PATH_TO_JDK in
case javac is not in your PATH.


11/02/2021: Changes from V3 of the patch
============================================
- First support for hardware event counter profiling on Arm.
- Fixed a problem with dlopen() for libc on Arm.
- Fixed several bugs and implemented some small changes in the output


09/01/2021: Changes from V2 of the patch
========================================

  - Renamed experiment.h to gp-experiment.h to fixe a problem on Windows 
that Hannes
    Domani reported.
  - Fixed a regression in the build on Arm.
  - Improved the gprofng testsuite.


08/25/2021: Changes from V1 of the patch
========================================

Addressed all of Joseph Myers's remarks about gprofng Makefiles and 
configure scripts:
  - Report an error when bison is absent, or a version of bison is too old.
  - Added preload_libdirs in ${prefix}/etc/gprofng.rc to configure the 
location of the
    gprofng libraries (this was hard-coded previously).

We also took note of Jim Wilson's comment about the naming of files in 
src/machinemodels.
In particular, file m5.ermm and the suggestion to make it clear this is 
for a SPARC
processor. For example sparc-m5.ermm.

This is going to be part of a bigger overhaul of the naming convention 
for these files.
The most obvious choice is to include the product name, but we 
definitely appreciate any
suggestions this group may have.



A new GNU profiler
==============================

In this submission we are contributing a new profiler to the GNU binary
utilities, called gprofng (for GNU profiler, next generation).

Why a new profiler?
===================

The GNU profiler, gprof, works well enough in many cases. However, it
hasn't aged well and it is not that very well suited for profiling
modern-world applications. Examples of its limitations are lack of support
for profiling multithreaded programs, and shared objects. Both are
ubiquitous nowadays.

Main characteristics of gprofng
===============================

gprofng supports profiling C, C++ and Java programs. Unlike the old
gprof, it doesn't require to build annotated versions of the programs.
Profiling "production" binaries should work just fine.

Another distinguishing feature of gprofng is the support for various filters
that allow the user to easily drill deeper into an area of interest.

The profiler is commanded through a driver program called `gprofng'.
This driver supports the following sub-commands:

gprofng collect app EXECUTABLE

This runs EXECUTABLE and collects application performance data.

gprofng display text EXPERIMENT

This runs a client command-line interface that provides access to the
collected performance data stored in the experiment directory.

gprofng display html EXPERIMENT

This generates an HTML report from the collected performance data
stored in the experiment directory.

gprofng display src OBJECT-FILE

This displays source (if available) or disassembly interleaved
with the source code.

gprofng archive EXPERIMENT

Archive the associated application binaries, load objects and source
files in an existing experiment directory to make it self contained.

There is also an extensive graphical user interface (written in Java)
that displays and analyzes gprofng collected data in a very sophisticated
way. We plan to release this GUI as a separate project.

While WIP, we would like to share some screenshots of the current
development version. These show the following:

pic1.png - a flame graph:
https://jemarch.net/gprofng-pics/pic1.png

pic2.png - color coded call stacks as a function of time ("the timeline"):
https://jemarch.net/gprofng-pics/pic2.png

pic3.png - zoom in on the timeline and adapt colors to identify details:
https://jemarch.net/gprofng-pics/pic3.png

pic4.png - compare two mulithreaded profiles:
https://jemarch.net/gprofng-pics/pic4.png


Notes on the implementation
===========================

- The gp-display-html tool is written in Perl. All other components are
written in C/C++.

- gprofng sources are mostly contained in a new top-level directory
gprofng/.

- The libcollector.h, libfcollector.h, and collectorAPI.h header files
are header files are distributed in the top-level include/ directory.

- Currently gprofng supports profiling programs in GNU/Linux systems
running on x86_64 and aarch64 hardware. It is possible to add support
for additional architectures.

- The tools come with a set of man pages. They are generated upon
installation and can be found in the installation directory under
share/man/man1.

Platform support
================

The basic profiling features are supported on most processors from
Intel. Regarding AMD we did not yet test on their recent EPYC
processors, but do not expect serious issues for the callstack
sampling. We also support the Arm processors as used in systems
from Ampere.

Hardware event counters are very processor specific. While gprofng
supports using such counters, we currently lack support for recent
Intel and AMD processors, like the EPYC architecture in case of
the latter.
If a particular processor is not supported, but a hardware event
counter experiment is requested, a warning message will be issued
before gprofng terminates.

This code has been developed and tested on Oracle Linux 8 with the
latest GNU toolchain from the sourceware git repos.

Structure of the patch series
=============================

The first patch is preparatory and makes the x86 disassembler in opcodes
to be thread-safe. This is so it can be used by gprofng.

The second patch is the implementation of gprofng proper. This includes
source code for the libraries (libcollector, libgprofng) and the
utilities (gp-collect, etc).

In this patch there are also updates to the corresponding build machinery
(e.g. configure.ac, Makefile.def, plus binutils/MAINTAINERS to cover 
gprofng)

The third patch adds a testsuite in gprofng/testsuite.

The fourth patch adds a Texinfo manual for gprofng. The manual is still
WIP but already provides a tutorial-like introduction to the tools.


Limitations
===========

The gp-display-html tool is present, and can be executed, but it is not
functional yet. Full support for this tool is expected to be delivered
in a future patch.

Requirements
===========

In order to successfully build gprofng, the following versions of
external components are required:

- Bison 3.7.5, or higher
- Texinfo 6.7, or higher
- Java include files (--with-jdk=PATH) if java profiling should be enabled

Maintenance
===========

We are of course volunteering to maintain gprofng once it is
incorporated into the main binutils distribution.
We suggest having feedback and discussion in this mail thread.

Ping: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

Any suggestions, recomendations to make the gprofng sources public ?
We need a code review.
Thank you,
-Vladimir



-------- Forwarded Message --------
Subject: 	[PATCH V4] gprofng: a new GNU profiler
Date: 	Wed, 3 Nov 2021 11:05:16 -0700
From: 	Vladimir Mezentsev <vladimir.mezentsev@oracle.com>
To: 	binutils@sourceware.org




Hi people!

Here is a slightly updated patch for the gprofng utility. We have fixed
some issues we have noticed while testing and added first support for
hardware event counter profiling on Arm.

We think gprofng will be a good addition to binutils and hope that
we can move forward. In particular we'd like to ask for the review
process to start.

This is a high level description of the various components in gprofng
with the purpose of helping reviewers:

gprofng tools:
     gprofng         - the driver program
     gp-collect-app  - create experiment with program performance data
     gp-display-text - print an ASCII report from one or more 
performance experiments
     gp-display-html - generate an HTML based directory structure from 
an experiment
     gp-archive      - archive the binaries and sources for experiment
     gp-display-src  - print a source or dissasembly listing for the 
load object

     lib/libgprofng.so - read experiments.
                         libgprofng.so is linked with each gprofng tools.

gprofng collector libraries:
     lib/libgp-collector.so - gprofng collector library
     lib/libgp-heap.so      - Heap tracing library
     lib/libgp-iotrace.so   - I/O tracing library
     lib/libgp-sync.so      - Synchronization tracing library


gprofng source tree:
     gprofng/doc/*          - gprofng documentation
     gprofng/libcollector/* - sources of libgp-collector.so, libgp-heap.so,
                              libgp-iotrace.so and libgp-sync.so
     gprofng/src/*          - sources of all gprofng tools and libgprofng.so
     gprofng/common/*       - these files are common components used
                              when building the gprofng libraries


Known problems
==============
- To convert cycles to time, we need the frequency the core is running at.
   On Arm we read the cntrfrq_e10 register, but this sometimes returns an
   incorrect value. For the time being, it may be safest to ignore the
   cycles derived timings on Arm.

- The support for hardware event counters on recent processors is limited.
   You can use the “gprofng collect app -h” command to check for support.
   In case a specific processor is not supported, a warning message will be
   printed.  Otherwise a list with the supported events is printed.


How to get gprofng
==================

Due to the size of the contribution, we thought it would be better to submit
it in the form of a git branch, instead of a regular emailed patch series.

Repository:https://www.github.com/oracle/binutils-gdb
Branch:     oracle/gprofng-v4


Build gprofng
=============

For x86_64/i686, gprofng should be built twice to support profiling for
both 32-bit and 64-bit applications:
   PREFIX=<YOUR_INSTALL_DIR>
   configure --enable-shared --prefix=$PREFIX --libdir=$PREFIX/lib64 && 
make && make install
   configure --enable-shared --prefix=$PREFIX CC='gcc -m32' CXX='g++ -m32' \
          --target=i686-pc-linux-gnu --disable-gprofng-tools && make && 
make install

For ARM:
   configure --enable-shared --prefix=$PREFIX && make install

To support java profiling, configure the build with -with-jdk=PATH_TO_JDK in
case javac is not in your PATH.


11/02/2021: Changes from V3 of the patch
============================================
- First support for hardware event counter profiling on Arm.
- Fixed a problem with dlopen() for libc on Arm.
- Fixed several bugs and implemented some small changes in the output


09/01/2021: Changes from V2 of the patch
========================================

  - Renamed experiment.h to gp-experiment.h to fixe a problem on Windows 
that Hannes
    Domani reported.
  - Fixed a regression in the build on Arm.
  - Improved the gprofng testsuite.


08/25/2021: Changes from V1 of the patch
========================================

Addressed all of Joseph Myers's remarks about gprofng Makefiles and 
configure scripts:
  - Report an error when bison is absent, or a version of bison is too old.
  - Added preload_libdirs in ${prefix}/etc/gprofng.rc to configure the 
location of the
    gprofng libraries (this was hard-coded previously).

We also took note of Jim Wilson's comment about the naming of files in 
src/machinemodels.
In particular, file m5.ermm and the suggestion to make it clear this is 
for a SPARC
processor. For example sparc-m5.ermm.

This is going to be part of a bigger overhaul of the naming convention 
for these files.
The most obvious choice is to include the product name, but we 
definitely appreciate any
suggestions this group may have.



A new GNU profiler
==============================

In this submission we are contributing a new profiler to the GNU binary
utilities, called gprofng (for GNU profiler, next generation).

Why a new profiler?
===================

The GNU profiler, gprof, works well enough in many cases. However, it
hasn't aged well and it is not that very well suited for profiling
modern-world applications. Examples of its limitations are lack of support
for profiling multithreaded programs, and shared objects. Both are
ubiquitous nowadays.

Main characteristics of gprofng
===============================

gprofng supports profiling C, C++ and Java programs. Unlike the old
gprof, it doesn't require to build annotated versions of the programs.
Profiling "production" binaries should work just fine.

Another distinguishing feature of gprofng is the support for various filters
that allow the user to easily drill deeper into an area of interest.

The profiler is commanded through a driver program called `gprofng'.
This driver supports the following sub-commands:

gprofng collect app EXECUTABLE

This runs EXECUTABLE and collects application performance data.

gprofng display text EXPERIMENT

This runs a client command-line interface that provides access to the
collected performance data stored in the experiment directory.

gprofng display html EXPERIMENT

This generates an HTML report from the collected performance data
stored in the experiment directory.

gprofng display src OBJECT-FILE

This displays source (if available) or disassembly interleaved
with the source code.

gprofng archive EXPERIMENT

Archive the associated application binaries, load objects and source
files in an existing experiment directory to make it self contained.

There is also an extensive graphical user interface (written in Java)
that displays and analyzes gprofng collected data in a very sophisticated
way. We plan to release this GUI as a separate project.

While WIP, we would like to share some screenshots of the current
development version. These show the following:

pic1.png - a flame graph:
https://jemarch.net/gprofng-pics/pic1.png

pic2.png - color coded call stacks as a function of time ("the timeline"):
https://jemarch.net/gprofng-pics/pic2.png

pic3.png - zoom in on the timeline and adapt colors to identify details:
https://jemarch.net/gprofng-pics/pic3.png

pic4.png - compare two mulithreaded profiles:
https://jemarch.net/gprofng-pics/pic4.png


Notes on the implementation
===========================

- The gp-display-html tool is written in Perl. All other components are
written in C/C++.

- gprofng sources are mostly contained in a new top-level directory
gprofng/.

- The libcollector.h, libfcollector.h, and collectorAPI.h header files
are header files are distributed in the top-level include/ directory.

- Currently gprofng supports profiling programs in GNU/Linux systems
running on x86_64 and aarch64 hardware. It is possible to add support
for additional architectures.

- The tools come with a set of man pages. They are generated upon
installation and can be found in the installation directory under
share/man/man1.

Platform support
================

The basic profiling features are supported on most processors from
Intel. Regarding AMD we did not yet test on their recent EPYC
processors, but do not expect serious issues for the callstack
sampling. We also support the Arm processors as used in systems
from Ampere.

Hardware event counters are very processor specific. While gprofng
supports using such counters, we currently lack support for recent
Intel and AMD processors, like the EPYC architecture in case of
the latter.
If a particular processor is not supported, but a hardware event
counter experiment is requested, a warning message will be issued
before gprofng terminates.

This code has been developed and tested on Oracle Linux 8 with the
latest GNU toolchain from the sourceware git repos.

Structure of the patch series
=============================

The first patch is preparatory and makes the x86 disassembler in opcodes
to be thread-safe. This is so it can be used by gprofng.

The second patch is the implementation of gprofng proper. This includes
source code for the libraries (libcollector, libgprofng) and the
utilities (gp-collect, etc).

In this patch there are also updates to the corresponding build machinery
(e.g. configure.ac, Makefile.def, plus binutils/MAINTAINERS to cover 
gprofng)

The third patch adds a testsuite in gprofng/testsuite.

The fourth patch adds a Texinfo manual for gprofng. The manual is still
WIP but already provides a tutorial-like introduction to the tools.


Limitations
===========

The gp-display-html tool is present, and can be executed, but it is not
functional yet. Full support for this tool is expected to be delivered
in a future patch.

Requirements
===========

In order to successfully build gprofng, the following versions of
external components are required:

- Bison 3.7.5, or higher
- Texinfo 6.7, or higher
- Java include files (--with-jdk=PATH) if java profiling should be enabled

Maintenance
===========

We are of course volunteering to maintain gprofng once it is
incorporated into the main binutils distribution.
We suggest having feedback and discussion in this mail thread.

Ping: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

  Ping.


-------- Forwarded Message --------
Subject: 	Ping: [PATCH V4] gprofng: a new GNU profiler
Date: 	Mon, 15 Nov 2021 10:13:51 -0800
From: 	Vladimir Mezentsev <vladimir.mezentsev@oracle.com>
To: 	binutils@sourceware.org




Any suggestions, recomendations  to make the gprofng sources public ?
We need a code review.
Thank you,
-Vladimir



-------- Forwarded Message --------
Subject: 	[PATCH V4] gprofng: a new GNU profiler
Date: 	Wed, 3 Nov 2021 11:05:16 -0700
From: 	Vladimir Mezentsev <vladimir.mezentsev@oracle.com>
To: 	binutils@sourceware.org




Hi people!

Here is a slightly updated patch for the gprofng utility. We have fixed
some issues we have noticed while testing and added first support for
hardware event counter profiling on Arm.

We think gprofng will be a good addition to binutils and hope that
we can move forward. In particular we'd like to ask for the review
process to start.

This is a high level description of the various components in gprofng
with the purpose of helping reviewers:

gprofng tools:
     gprofng         - the driver program
     gp-collect-app  - create experiment with program performance data
     gp-display-text - print an ASCII report from one or more 
performance experiments
     gp-display-html - generate an HTML based directory structure from 
an experiment
     gp-archive      - archive the binaries and sources for experiment
     gp-display-src  - print a source or dissasembly listing for the 
load object

     lib/libgprofng.so - read experiments.
                         libgprofng.so is linked with each gprofng tools.

gprofng collector libraries:
     lib/libgp-collector.so - gprofng collector library
     lib/libgp-heap.so      - Heap tracing library
     lib/libgp-iotrace.so   - I/O tracing library
     lib/libgp-sync.so      - Synchronization tracing library


gprofng source tree:
     gprofng/doc/*          - gprofng documentation
     gprofng/libcollector/* - sources of libgp-collector.so, libgp-heap.so,
                              libgp-iotrace.so and libgp-sync.so
     gprofng/src/*          - sources of all gprofng tools and libgprofng.so
     gprofng/common/*       - these files are common components used
                              when building the gprofng libraries


Known problems
==============
- To convert cycles to time, we need the frequency the core is running at.
   On Arm we read the cntrfrq_e10 register, but this sometimes returns an
   incorrect value. For the time being, it may be safest to ignore the
   cycles derived timings on Arm.

- The support for hardware event counters on recent processors is limited.
   You can use the “gprofng collect app -h” command to check for support.
   In case a specific processor is not supported, a warning message will be
   printed.  Otherwise a list with the supported events is printed.


How to get gprofng
==================

Due to the size of the contribution, we thought it would be better to submit
it in the form of a git branch, instead of a regular emailed patch series.

Repository:https://www.github.com/oracle/binutils-gdb
Branch:     oracle/gprofng-v4


Build gprofng
=============

For x86_64/i686, gprofng should be built twice to support profiling for
both 32-bit and 64-bit applications:
   PREFIX=<YOUR_INSTALL_DIR>
   configure --enable-shared --prefix=$PREFIX --libdir=$PREFIX/lib64 && 
make && make install
   configure --enable-shared --prefix=$PREFIX CC='gcc -m32' CXX='g++ -m32' \
          --target=i686-pc-linux-gnu --disable-gprofng-tools && make && 
make install

For ARM:
   configure --enable-shared --prefix=$PREFIX && make install

To support java profiling, configure the build with -with-jdk=PATH_TO_JDK in
case javac is not in your PATH.


11/02/2021: Changes from V3 of the patch
============================================
- First support for hardware event counter profiling on Arm.
- Fixed a problem with dlopen() for libc on Arm.
- Fixed several bugs and implemented some small changes in the output


09/01/2021: Changes from V2 of the patch
========================================

  - Renamed experiment.h to gp-experiment.h to fixe a problem on Windows 
that Hannes
    Domani reported.
  - Fixed a regression in the build on Arm.
  - Improved the gprofng testsuite.


08/25/2021: Changes from V1 of the patch
========================================

Addressed all of Joseph Myers's remarks about gprofng Makefiles and 
configure scripts:
  - Report an error when bison is absent, or a version of bison is too old.
  - Added preload_libdirs in ${prefix}/etc/gprofng.rc to configure the 
location of the
    gprofng libraries (this was hard-coded previously).

We also took note of Jim Wilson's comment about the naming of files in 
src/machinemodels.
In particular, file m5.ermm and the suggestion to make it clear this is 
for a SPARC
processor. For example sparc-m5.ermm.

This is going to be part of a bigger overhaul of the naming convention 
for these files.
The most obvious choice is to include the product name, but we 
definitely appreciate any
suggestions this group may have.



A new GNU profiler
==============================

In this submission we are contributing a new profiler to the GNU binary
utilities, called gprofng (for GNU profiler, next generation).

Why a new profiler?
===================

The GNU profiler, gprof, works well enough in many cases. However, it
hasn't aged well and it is not that very well suited for profiling
modern-world applications. Examples of its limitations are lack of support
for profiling multithreaded programs, and shared objects. Both are
ubiquitous nowadays.

Main characteristics of gprofng
===============================

gprofng supports profiling C, C++ and Java programs. Unlike the old
gprof, it doesn't require to build annotated versions of the programs.
Profiling "production" binaries should work just fine.

Another distinguishing feature of gprofng is the support for various filters
that allow the user to easily drill deeper into an area of interest.

The profiler is commanded through a driver program called `gprofng'.
This driver supports the following sub-commands:

gprofng collect app EXECUTABLE

This runs EXECUTABLE and collects application performance data.

gprofng display text EXPERIMENT

This runs a client command-line interface that provides access to the
collected performance data stored in the experiment directory.

gprofng display html EXPERIMENT

This generates an HTML report from the collected performance data
stored in the experiment directory.

gprofng display src OBJECT-FILE

This displays source (if available) or disassembly interleaved
with the source code.

gprofng archive EXPERIMENT

Archive the associated application binaries, load objects and source
files in an existing experiment directory to make it self contained.

There is also an extensive graphical user interface (written in Java)
that displays and analyzes gprofng collected data in a very sophisticated
way. We plan to release this GUI as a separate project.

While WIP, we would like to share some screenshots of the current
development version. These show the following:

pic1.png - a flame graph:
https://jemarch.net/gprofng-pics/pic1.png

pic2.png - color coded call stacks as a function of time ("the timeline"):
https://jemarch.net/gprofng-pics/pic2.png

pic3.png - zoom in on the timeline and adapt colors to identify details:
https://jemarch.net/gprofng-pics/pic3.png

pic4.png - compare two mulithreaded profiles:
https://jemarch.net/gprofng-pics/pic4.png


Notes on the implementation
===========================

- The gp-display-html tool is written in Perl. All other components are
written in C/C++.

- gprofng sources are mostly contained in a new top-level directory
gprofng/.

- The libcollector.h, libfcollector.h, and collectorAPI.h header files
are header files are distributed in the top-level include/ directory.

- Currently gprofng supports profiling programs in GNU/Linux systems
running on x86_64 and aarch64 hardware. It is possible to add support
for additional architectures.

- The tools come with a set of man pages. They are generated upon
installation and can be found in the installation directory under
share/man/man1.

Platform support
================

The basic profiling features are supported on most processors from
Intel. Regarding AMD we did not yet test on their recent EPYC
processors, but do not expect serious issues for the callstack
sampling. We also support the Arm processors as used in systems
from Ampere.

Hardware event counters are very processor specific. While gprofng
supports using such counters, we currently lack support for recent
Intel and AMD processors, like the EPYC architecture in case of
the latter.
If a particular processor is not supported, but a hardware event
counter experiment is requested, a warning message will be issued
before gprofng terminates.

This code has been developed and tested on Oracle Linux 8 with the
latest GNU toolchain from the sourceware git repos.

Structure of the patch series
=============================

The first patch is preparatory and makes the x86 disassembler in opcodes
to be thread-safe. This is so it can be used by gprofng.

The second patch is the implementation of gprofng proper. This includes
source code for the libraries (libcollector, libgprofng) and the
utilities (gp-collect, etc).

In this patch there are also updates to the corresponding build machinery
(e.g. configure.ac, Makefile.def, plus binutils/MAINTAINERS to cover 
gprofng)

The third patch adds a testsuite in gprofng/testsuite.

The fourth patch adds a Texinfo manual for gprofng. The manual is still
WIP but already provides a tutorial-like introduction to the tools.


Limitations
===========

The gp-display-html tool is present, and can be executed, but it is not
functional yet. Full support for this tool is expected to be delivered
in a future patch.

Requirements
===========

In order to successfully build gprofng, the following versions of
external components are required:

- Bison 3.7.5, or higher
- Texinfo 6.7, or higher
- Java include files (--with-jdk=PATH) if java profiling should be enabled

Maintenance
===========

We are of course volunteering to maintain gprofng once it is
incorporated into the main binutils distribution.
We suggest having feedback and discussion in this mail thread.

Re: [PATCH V4] gprofng: a new GNU profiler

[Nick Clifton]

Hi Vladimir,

> For x86_64/i686, gprofng should be built twice to support profiling for
> both 32-bit and 64-bit applications:
>    PREFIX=<YOUR_INSTALL_DIR>
>    configure --enable-shared --prefix=$PREFIX --libdir=$PREFIX/lib64 && make && make install
>    configure --enable-shared --prefix=$PREFIX CC='gcc -m32' CXX='g++ -m32' \
>           --target=i686-pc-linux-gnu --disable-gprofng-tools && make && make install
> 
> For ARM:
>    configure --enable-shared --prefix=$PREFIX && make install

Am I correct in thinking that gprofng only supports x86/x86_64 and ARM ?

To me this is a concern as the intention of the binutils project is to
support as wide a range of architectures as possible.

Is it simple to add other architectures ?  What work needs to be done ?

Cheers
   Nick

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

Hi Nick,

On 12/10/21 01:38, Nick Clifton wrote:
> Hi Vladimir,
>
>> For x86_64/i686, gprofng should be built twice to support profiling for
>> both 32-bit and 64-bit applications:
>>    PREFIX=<YOUR_INSTALL_DIR>
>>    configure --enable-shared --prefix=$PREFIX --libdir=$PREFIX/lib64 
>> && make && make install
>>    configure --enable-shared --prefix=$PREFIX CC='gcc -m32' CXX='g++ 
>> -m32' \
>>           --target=i686-pc-linux-gnu --disable-gprofng-tools && make 
>> && make install
>>
>> For ARM:
>>    configure --enable-shared --prefix=$PREFIX && make install
>
> Am I correct in thinking that gprofng only supports x86/x86_64 and ARM ?

yes, it currently on x86/x86_64 and aarch64.


>
> To me this is a concern as the intention of the binutils project is to
> support as wide a range of architectures as possible.
>
> Is it simple to add other architectures ?  What work needs to be done ?

We utilize disassembler() from libopcodes and several functions from 
libbfd which are already cross platform.
The rest of porting is stack unwinding and hardware counter profiling 
(It is in binutils-gdb.git/gprofng/libcollector).
We will start writing a few page porting guide.

Thank you,
-Vladimir

>
> Cheers
>   Nick
>

Re: [PATCH V4] gprofng: a new GNU profiler

[Nick Clifton]

Hi Vladimir,

>> To me this is a concern as the intention of the binutils project is to
>> support as wide a range of architectures as possible.
>>
>> Is it simple to add other architectures ?  What work needs to be done ?
> 
> We utilize disassembler() from libopcodes and several functions from libbfd which are already cross platform.
> The rest of porting is stack unwinding and hardware counter profiling (It is in binutils-gdb.git/gprofng/libcollector).
> We will start writing a few page porting guide.

OK, well I do not want to hold up this work any longer, so the patch series is approved.  Please apply.

Cheers
   Nick

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

On 12/13/21 07:13, Nick Clifton wrote:
> Hi Vladimir,
>
>>> To me this is a concern as the intention of the binutils project is to
>>> support as wide a range of architectures as possible.
>>>
>>> Is it simple to add other architectures ?  What work needs to be done ?
>>
>> We utilize disassembler() from libopcodes and several functions from 
>> libbfd which are already cross platform.
>> The rest of porting is stack unwinding and hardware counter profiling 
>> (It is in binutils-gdb.git/gprofng/libcollector).
>> We will start writing a few page porting guide.
>
> OK, well I do not want to hold up this work any longer, so the patch 
> series is approved.  Please apply.

   Hi Nick,
thank you for the approval.
After `git rebase master` and rerun tests, I found a regression.
I made an additional fix in opcodes/i386-dis.c in swap_operand() and in 
couple similar places.

For example:
swap_operand (void)
{
   mnemonicendp[0] = '.';
   mnemonicendp[1] = 's';
*->>     mnemonicendp[2] = '\0'; <<<<- this line was absent*
   mnemonicendp += 2;
}


  After new fixes, I updated the several filters in 
gas/testsuite/gas/i386/ directory.

I will send my fixes in opcodes/i386-dis.c and ingas/testsuite for 
review today.


Thank you,
-Vladimir

>
> Cheers
>   Nick
>
>

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

On 12/13/21 07:13, Nick Clifton wrote:
> Hi Vladimir,
>
>>> To me this is a concern as the intention of the binutils project is to
>>> support as wide a range of architectures as possible.
>>>
>>> Is it simple to add other architectures ?  What work needs to be done ?
>>
>> We utilize disassembler() from libopcodes and several functions from 
>> libbfd which are already cross platform.
>> The rest of porting is stack unwinding and hardware counter profiling 
>> (It is in binutils-gdb.git/gprofng/libcollector).
>> We will start writing a few page porting guide.
>
> OK, well I do not want to hold up this work any longer, so the patch 
> series is approved.  Please apply.

   Hi Nick,

my fixes in opcodes/i386-dis.c are already in the master branch.
I created 4 patches to integrate gprofng into binutils-gdb.

% ls -al 000*
-rw-rw-r-- 1 vmezents wheel 5901152 Jan 11 12:39 
0001-gprofng-add-the-gprofng-tool.patch
-rw-rw-r-- 1 vmezents wheel  224725 Jan 11 12:39 
0002-gprofng-add-testsuite-for-gprofng.patch
-rw-rw-r-- 1 vmezents wheel  179400 Jan 11 12:39 
0003-gprofng-add-the-gprofng-documentation.patch
-rw-rw-r-- 1 vmezents wheel   26170 Jan 11 12:39 
0004-gprofng-add-the-gprofng-build.patch

The first 3 pathes add or change sources only in 
binutils-gdb.git/gprofng directory.
0004-gprofng-add-the-gprofng-build.patch has fixes in Makefile.def and 
configure.ac to configure the gprofng build.

0001-gprofng-add-the-gprofng-tool.patch is probably so big to send to 
binutils@sourceware.org.

I have no permissions to apply pathes.
What should I do to pushmy fixes for gprofng in binutils-gdb.git ?

Thank you,
-Vladimir

Re: [PATCH V4] gprofng: a new GNU profiler

[Nick Clifton]

Hi Vladimir,

> I created 4 patches to integrate gprofng into binutils-gdb.
> 
> % ls -al 000*
> -rw-rw-r-- 1 vmezents wheel 5901152 Jan 11 12:39 0001-gprofng-add-the-gprofng-tool.patch
> -rw-rw-r-- 1 vmezents wheel  224725 Jan 11 12:39 0002-gprofng-add-testsuite-for-gprofng.patch
> -rw-rw-r-- 1 vmezents wheel  179400 Jan 11 12:39 0003-gprofng-add-the-gprofng-documentation.patch
> -rw-rw-r-- 1 vmezents wheel   26170 Jan 11 12:39 0004-gprofng-add-the-gprofng-build.patch
> 
> The first 3 pathes add or change sources only in binutils-gdb.git/gprofng directory.
> 0004-gprofng-add-the-gprofng-build.patch has fixes in Makefile.def and configure.ac to configure the gprofng build.
> 
> 0001-gprofng-add-the-gprofng-tool.patch is probably so big to send to binutils@sourceware.org.
> 
> I have no permissions to apply pathes.
> What should I do to pushmy fixes for gprofng in binutils-gdb.git ?

Send them to me (or point me at them - they are probably in the email archive somewhere
but I am a little bit swamped at the moment, so anything that you can do to make my life
simpler is a benefit).  I strongly suggest that you compress the 0001 patch first though :-)
xz is your friend.  Or lzip.  Or well pretty much any compression tool...

Cheers
   Nick

Re: [PATCH V4] gprofng: a new GNU profiler

[Jose E. Marchesi]

> Hi Vladimir,
>
>> I created 4 patches to integrate gprofng into binutils-gdb.
>> % ls -al 000*
>> -rw-rw-r-- 1 vmezents wheel 5901152 Jan 11 12:39 0001-gprofng-add-the-gprofng-tool.patch
>> -rw-rw-r-- 1 vmezents wheel  224725 Jan 11 12:39 0002-gprofng-add-testsuite-for-gprofng.patch
>> -rw-rw-r-- 1 vmezents wheel  179400 Jan 11 12:39
>> 0003-gprofng-add-the-gprofng-documentation.patch
>> -rw-rw-r-- 1 vmezents wheel   26170 Jan 11 12:39 0004-gprofng-add-the-gprofng-build.patch
>> The first 3 pathes add or change sources only in
>> binutils-gdb.git/gprofng directory.
>> 0004-gprofng-add-the-gprofng-build.patch has fixes in Makefile.def
>> and configure.ac to configure the gprofng build.
>> 0001-gprofng-add-the-gprofng-tool.patch is probably so big to send
>> to binutils@sourceware.org.
>> I have no permissions to apply pathes.
>> What should I do to pushmy fixes for gprofng in binutils-gdb.git ?
>
> Send them to me (or point me at them - they are probably in the email archive somewhere
> but I am a little bit swamped at the moment, so anything that you can do to make my life
> simpler is a benefit).  I strongly suggest that you compress the 0001 patch first though :-)
> xz is your friend.  Or lzip.  Or well pretty much any compression tool...

I can do the push on Vlad's behalf.  Much easier.

Re: [PATCH V4] gprofng: a new GNU profiler

[Jose E. Marchesi]

>> Hi Vladimir,
>>
>>> I created 4 patches to integrate gprofng into binutils-gdb.
>>> % ls -al 000*
>>> -rw-rw-r-- 1 vmezents wheel 5901152 Jan 11 12:39 0001-gprofng-add-the-gprofng-tool.patch
>>> -rw-rw-r-- 1 vmezents wheel  224725 Jan 11 12:39 0002-gprofng-add-testsuite-for-gprofng.patch
>>> -rw-rw-r-- 1 vmezents wheel  179400 Jan 11 12:39
>>> 0003-gprofng-add-the-gprofng-documentation.patch
>>> -rw-rw-r-- 1 vmezents wheel   26170 Jan 11 12:39 0004-gprofng-add-the-gprofng-build.patch
>>> The first 3 pathes add or change sources only in
>>> binutils-gdb.git/gprofng directory.
>>> 0004-gprofng-add-the-gprofng-build.patch has fixes in Makefile.def
>>> and configure.ac to configure the gprofng build.
>>> 0001-gprofng-add-the-gprofng-tool.patch is probably so big to send
>>> to binutils@sourceware.org.
>>> I have no permissions to apply pathes.
>>> What should I do to pushmy fixes for gprofng in binutils-gdb.git ?
>>
>> Send them to me (or point me at them - they are probably in the email archive somewhere
>> but I am a little bit swamped at the moment, so anything that you can do to make my life
>> simpler is a benefit).  I strongly suggest that you compress the 0001 patch first though :-)
>> xz is your friend.  Or lzip.  Or well pretty much any compression tool...
>
> I can do the push on Vlad's behalf.  Much easier.

Hm, now that I think about it... this is a new subsystem being added
here.  We can expect issues that need to be resolved fast.  Vladimir
can't rely on third persons to do that.

Nick, OK for me sponsoring Vlad for an account in sourceware and write
access to binutils-gdb.git?

Re: [PATCH V4] gprofng: a new GNU profiler

[Nick Clifton]

Hi Vladimir,

> % tar tf mez.tar.gz
> 0001-gprofng-add-the-gprofng-tool.patch
> 0002-gprofng-add-testsuite-for-gprofng.patch
> 0003-gprofng-add-the-gprofng-documentation.patch
> 0004-gprofng-add-the-gprofng-build.patch

Right - I am now testing these patches locally.  I have however run
into a possible issue.  The patches apply cleanly and everything builds
(although the configure script did mention that I do not have the JDK
kit installed, which is true).  But... when I run "make check" in the
gprofng directory I get these results:

   % make check
   [...]
   Native configuration is x86_64-pc-linux-gnu

		=== gprofng tests ===

   Schedule of variations:
       unix

   Running target unix
   Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
   Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
   Using /work/sources/binutils/current/gprofng/testsuite/config/default.exp as tool-and-target-specific interface file.
   Running /work/sources/binutils/current/gprofng/testsuite/gprofng.display/display.exp ...
   ERROR: compilation of test program in jsynprog failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed

		=== gprofng Summary ===

   # of unresolved testcases	13

Is this expected ?

Is it something that I can ignore for now, or is there a test
harness issue that needs to be resolved ?

Cheers
   Nick

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

   Hi Nick,

On 1/26/22 04:04, Nick Clifton wrote:
> Hi Vladimir,
>
>> % tar tf mez.tar.gz
>> 0001-gprofng-add-the-gprofng-tool.patch
>> 0002-gprofng-add-testsuite-for-gprofng.patch
>> 0003-gprofng-add-the-gprofng-documentation.patch
>> 0004-gprofng-add-the-gprofng-build.patch
>
> Right - I am now testing these patches locally.  I have however run
> into a possible issue.  The patches apply cleanly and everything builds
> (although the configure script did mention that I do not have the JDK
> kit installed, which is true).  But... when I run "make check" in the
> gprofng directory I get these results:
>
>   % make check
>   [...]
>   Native configuration is x86_64-pc-linux-gnu
>
>         === gprofng tests ===
>
>   Schedule of variations:
>       unix
>
>   Running target unix
>   Using /usr/share/dejagnu/baseboards/unix.exp as board description 
> file for target.
>   Using /usr/share/dejagnu/config/unix.exp as generic interface file 
> for target.
>   Using 
> /work/sources/binutils/current/gprofng/testsuite/config/default.exp as 
> tool-and-target-specific interface file.
>   Running 
> /work/sources/binutils/current/gprofng/testsuite/gprofng.display/display.exp 
> ...
>   ERROR: compilation of test program in jsynprog failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>
>         === gprofng Summary ===
>
>   # of unresolved testcases    13
>
> Is this expected ?


It looks like you configured build  without --enable-shared.
If yes, this is expected.

-Vladimir


> Is it something that I can ignore for now, or is there a test
> harness issue that needs to be resolved ?
>
> Cheers
>   Nick
>

Re: [PATCH V4] gprofng: a new GNU profiler

[Jose E. Marchesi]

>   Hi Nick,
>
> On 1/26/22 04:04, Nick Clifton wrote:
>> Hi Vladimir,
>>
>>> % tar tf mez.tar.gz
>>> 0001-gprofng-add-the-gprofng-tool.patch
>>> 0002-gprofng-add-testsuite-for-gprofng.patch
>>> 0003-gprofng-add-the-gprofng-documentation.patch
>>> 0004-gprofng-add-the-gprofng-build.patch
>>
>> Right - I am now testing these patches locally.  I have however run
>> into a possible issue.  The patches apply cleanly and everything builds
>> (although the configure script did mention that I do not have the JDK
>> kit installed, which is true).  But... when I run "make check" in the
>> gprofng directory I get these results:
>>
>>   % make check
>>   [...]
>>   Native configuration is x86_64-pc-linux-gnu
>>
>>         === gprofng tests ===
>>
>>   Schedule of variations:
>>       unix
>>
>>   Running target unix
>>   Using /usr/share/dejagnu/baseboards/unix.exp as board description
>> file for target.
>>   Using /usr/share/dejagnu/config/unix.exp as generic interface file
>> for target.
>>   Using
>> /work/sources/binutils/current/gprofng/testsuite/config/default.exp
>> as tool-and-target-specific interface file.
>>   Running
>> /work/sources/binutils/current/gprofng/testsuite/gprofng.display/display.exp 
>> ...
>>   ERROR: compilation of test program in jsynprog failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in mttest failed
>>   ERROR: compilation of test program in synprog failed
>>   ERROR: compilation of test program in synprog failed
>>   ERROR: compilation of test program in synprog failed
>>   ERROR: compilation of test program in synprog failed
>>   ERROR: compilation of test program in synprog failed
>>
>>         === gprofng Summary ===
>>
>>   # of unresolved testcases    13
>>
>> Is this expected ?
>
>
> It looks like you configured build  without --enable-shared.
> If yes, this is expected.

Woudn't it be better to skip building gprofng in case no --enable-shared
is specified at `configure' time?

You can set `notsupp' accordingly in configure.ac and it will then be
notified here:

  echo "*** This configuration is not supported in the following subdirectories:" 1>&2
  echo "    ${notsupp}" 1>&2
  echo "    (Any other directories should still work fine.)" 1>&2


>
> -Vladimir
>
>
>> Is it something that I can ignore for now, or is there a test
>> harness issue that needs to be resolved ?
>>
>> Cheers
>>   Nick
>>

Re: [PATCH V4] gprofng: a new GNU profiler

[Maciej W. Rozycki]

On Thu, 27 Jan 2022, Jose E. Marchesi via Binutils wrote:

> > It looks like you configured build  without --enable-shared.
> > If yes, this is expected.
> 
> Woudn't it be better to skip building gprofng in case no --enable-shared
> is specified at `configure' time?

 Is it actually necessary to require `--enable-shared' in the first place 
for the program to work?

 The thing is there is (otherwise) little point in using the option with 
binutils, given that both libbfd and libopcodes are internal libraries not 
for general use and neither provides a stable ABI, so every revision would 
have to be separately installed if any external programs were to use them.

 Therefore having these libraries built as DSOs causes all the drawbacks 
of dynamic loading while no advantages of same are virtually used, and 
consequently I believe hardly anyone uses such a configuration.

  Maciej

Re: [PATCH V4] gprofng: a new GNU profiler

[Jose E. Marchesi]

> On Thu, 27 Jan 2022, Jose E. Marchesi via Binutils wrote:
>
>> > It looks like you configured build  without --enable-shared.
>> > If yes, this is expected.
>> 
>> Woudn't it be better to skip building gprofng in case no --enable-shared
>> is specified at `configure' time?
>
>  Is it actually necessary to require `--enable-shared' in the first place 
> for the program to work?

I think so.  AFAIK the gprofng collector works as an interposed shared
object.

>  The thing is there is (otherwise) little point in using the option with 
> binutils, given that both libbfd and libopcodes are internal libraries not 
> for general use and neither provides a stable ABI, so every revision would 
> have to be separately installed if any external programs were to use them.
>
>  Therefore having these libraries built as DSOs causes all the drawbacks 
> of dynamic loading while no advantages of same are virtually used, and 
> consequently I believe hardly anyone uses such a configuration.

There is also libctf, which is used by the linker but also intended for
external usage.

RHEL, for example, builds native binutils with --enable-shared, then
excludes both libbfd.so and libopcodes.so from being installed:

%if %{enable_shared}
%{_libdir}/lib*.so
%{_libdir}/libctf*
%exclude %{_libdir}/libbfd.so
%exclude %{_libdir}/libopcodes.so
%endif

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

On 2/2/22 08:30, Jose E. Marchesi via Binutils wrote:
>> On Thu, 27 Jan 2022, Jose E. Marchesi via Binutils wrote:
>>
>>>> It looks like you configured build  without --enable-shared.
>>>> If yes, this is expected.
>>> Woudn't it be better to skip building gprofng in case no --enable-shared
>>> is specified at `configure' time?
>>   Is it actually necessary to require `--enable-shared' in the first place
>> for the program to work?
> I think so.  AFAIK the gprofng collector works as an interposed shared
> object.

yes.
We use the gprofng library in LD_PRELOAD. This library should be a 
shared library.


I am trying to set  extra_configure_flags='--enable-shared' for gprofng 
build only.
But gprofng depens on libopcodes and this library should build with -fPIC.
I haven't yet found how to solve this build issue.

-Vladimir



>
>>   The thing is there is (otherwise) little point in using the option with
>> binutils, given that both libbfd and libopcodes are internal libraries not
>> for general use and neither provides a stable ABI, so every revision would
>> have to be separately installed if any external programs were to use them.
>>
>>   Therefore having these libraries built as DSOs causes all the drawbacks
>> of dynamic loading while no advantages of same are virtually used, and
>> consequently I believe hardly anyone uses such a configuration.
> There is also libctf, which is used by the linker but also intended for
> external usage.
>
> RHEL, for example, builds native binutils with --enable-shared, then
> excludes both libbfd.so and libopcodes.so from being installed:
>
> %if %{enable_shared}
> %{_libdir}/lib*.so
> %{_libdir}/libctf*
> %exclude %{_libdir}/libbfd.so
> %exclude %{_libdir}/libopcodes.so
> %endif
>

Re: [PATCH V4] gprofng: a new GNU profiler

[Maciej W. Rozycki]

On Wed, 2 Feb 2022, Vladimir Mezentsev wrote:

> > > > Woudn't it be better to skip building gprofng in case no --enable-shared
> > > > is specified at `configure' time?
> > >   Is it actually necessary to require `--enable-shared' in the first place
> > > for the program to work?
> > I think so.  AFAIK the gprofng collector works as an interposed shared
> > object.
> 
> yes.
> We use the gprofng library in LD_PRELOAD. This library should be a shared
> library.
> 
> 
> I am trying to set  extra_configure_flags='--enable-shared' for gprofng build
> only.
> But gprofng depens on libopcodes and this library should build with -fPIC.
> I haven't yet found how to solve this build issue.

 Forcing `--enable-shared' in gprofng/ unconditionally could be confusing 
to the user who might want to avoid shared libraries for a reason.

 I think perhaps the best solution would be using AC_ENABLE_SHARED in 
gprofng/ or the equivalent LT_INIT option (I think it's the default 
anyway) and bail out with a suitable message such as:

"Building in gprofng is unsupported with shared libraries disabled.
Please reconfigure with `--enable-shared=gprofng' or `--disable-gprofng'."

or suchlike.

 Then in libopcodes/ you probably want the `pic-only' option to LT_INIT 
(or use `--with-pic') unless `--enable-shared' has been also used for the 
library, though I think it would best be conditionalised on gprofng having 
been enabled if possible as there's a performance hit from using PIC code.  
As there's a `--without-pic' option too the user can give this all may 
have to be coordinated at the top level rather than within gprofng/.

 Perhaps the way how `extra_host_libiberty_configure_flags' are handled 
could serve as an example (though I think it should also use `--with-pic' 
rather than `--enable-shared' unless `--enable-shared' has been explicitly 
given already).

 I realise what I have outlined is not a complete solution, but I hope it 
can help regardless.

  Maciej

Re: [PATCH V4] gprofng: a new GNU profiler

[Jose E. Marchesi]

> On Wed, 2 Feb 2022, Vladimir Mezentsev wrote:
>
>> > > > Woudn't it be better to skip building gprofng in case no --enable-shared
>> > > > is specified at `configure' time?
>> > >   Is it actually necessary to require `--enable-shared' in the first place
>> > > for the program to work?
>> > I think so.  AFAIK the gprofng collector works as an interposed shared
>> > object.
>> 
>> yes.
>> We use the gprofng library in LD_PRELOAD. This library should be a shared
>> library.
>> 
>> 
>> I am trying to set  extra_configure_flags='--enable-shared' for gprofng build
>> only.
>> But gprofng depens on libopcodes and this library should build with -fPIC.
>> I haven't yet found how to solve this build issue.
>
>  Forcing `--enable-shared' in gprofng/ unconditionally could be confusing 
> to the user who might want to avoid shared libraries for a reason.
>
>  I think perhaps the best solution would be using AC_ENABLE_SHARED in 
> gprofng/ or the equivalent LT_INIT option (I think it's the default 
> anyway) and bail out with a suitable message such as:
>
> "Building in gprofng is unsupported with shared libraries disabled.
> Please reconfigure with `--enable-shared=gprofng' or `--disable-gprofng'."
>
> or suchlike.

I think this warning could be implemented in the top-level configure.ac,
setting `notsupp' for gprofng if `enable_shared' is "no".

Then, if the user builds binutils without --enable-shared, or in a host
or target for which gprofng is not supported, you get the familiar
message:

*** This configuration is not supported in the following subdirectories:
    gprofng
    (Any other directories should still work fine.)

>  Then in libopcodes/ you probably want the `pic-only' option to LT_INIT 
> (or use `--with-pic') unless `--enable-shared' has been also used for the 
> library, though I think it would best be conditionalised on gprofng having 
> been enabled if possible as there's a performance hit from using PIC code.  
> As there's a `--without-pic' option too the user can give this all may 
> have to be coordinated at the top level rather than within gprofng/.
>
>  Perhaps the way how `extra_host_libiberty_configure_flags' are handled 
> could serve as an example (though I think it should also use `--with-pic' 
> rather than `--enable-shared' unless `--enable-shared' has been explicitly 
> given already).
>
>  I realise what I have outlined is not a complete solution, but I hope it 
> can help regardless.
>
>   Maciej

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

Hi Nick,

We were approved but decided to defer committing the patch series until 
after the 2.38 release was completed.

In the meantime, we have made a few improvements:
     - added configure.ac for gprofng/libcollector to generate shared 
library using AC_ENABLE_SHARE
     - --enable-shared is no longer needed to configure the gprofng build
     - removed the build dependency on bison
     - lowered the build dependency on texinfo version to 6.5
     - fixed a bug reported in stack unwind()
     - updated the documentation
     - updated the sources for GNU coding standard


  We have pushed these changes here:
     url: https://github.com/oracle/binutils-gdb.git/
     branch: oracle/gprofng-v4-1

  We would like approval to commit this updated patch series.


Thank you,
-Vladimir




On 12/13/21 07:13, Nick Clifton wrote:
> Hi Vladimir,
>
>>> To me this is a concern as the intention of the binutils project is to
>>> support as wide a range of architectures as possible.
>>>
>>> Is it simple to add other architectures ?  What work needs to be done ?
>>
>> We utilize disassembler() from libopcodes and several functions from 
>> libbfd which are already cross platform.
>> The rest of porting is stack unwinding and hardware counter profiling 
>> (It is in binutils-gdb.git/gprofng/libcollector).
>> We will start writing a few page porting guide.
>
> OK, well I do not want to hold up this work any longer, so the patch 
> series is approved.  Please apply.
>
> Cheers
>   Nick
>
>

Re: [PATCH V4] gprofng: a new GNU profiler

[Nick Clifton]

Hi Vladimir,

>   We have pushed these changes here:
>      url: https://github.com/oracle/binutils-gdb.git/
>      branch: oracle/gprofng-v4-1

OK, so using those sources, I built a set of the binutils binaries
without any special configuration options, ie:

   configure -- quiet

and then I built gprofng and the other binutils tools.  This worked
OK, but when I tried to run the gprofng testsuite only a few tests
appeared to run:

   Running gprofng/testsuite/gprofng.display/display.exp ...
   ERROR: compilation of test program in jsynprog failed

		=== gprofng Summary ===

   # of expected passes		14
   # of unresolved testcases	1

Which does not look right.  But then I remembered that I needed to
enable shared builds, so I tried:

   configure --quiet --enable-shared

and this produced even worse test results:

   Running gprofng/testsuite/gprofng.display/display.exp ...
   ERROR: compilation of test program in jsynprog failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in mttest failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed
   ERROR: compilation of test program in synprog failed

		=== gprofng Summary ===

   # of unresolved testcases	15

The jsynprog and synprog failures is probably expected, since I do not
have the JDK installed, but the mttest compilation fails because I am
not testing an installed version of gprofng:

   /dev/shm/delme/gprofng/gprofng/tmpdir/root/usr/local/bin/gprofng: error while loading shared libraries: libgprofng.so.0: cannot open shared object file: No such file or 
directory


It would be really nice if I could test a built-but-not-installed
version of the sources....

Cheers
   Nick

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

  Hi Nick,

On 3/4/22 05:53, Nick Clifton wrote:
> Hi Vladimir,
>
>>   We have pushed these changes here:
>>      url: https://github.com/oracle/binutils-gdb.git/
>>      branch: oracle/gprofng-v4-1
>
> OK, so using those sources, I built a set of the binutils binaries
> without any special configuration options, ie:
>
>   configure -- quiet
>
> and then I built gprofng and the other binutils tools.  This worked
> OK, but when I tried to run the gprofng testsuite only a few tests
> appeared to run:
>
>   Running gprofng/testsuite/gprofng.display/display.exp ...
>   ERROR: compilation of test program in jsynprog failed
>
>         === gprofng Summary ===
>
>   # of expected passes        14
>   # of unresolved testcases    1

  It is expected.  JDK is not installed.


>
> Which does not look right.  But then I remembered that I needed to
> enable shared builds, so I tried:
>
>   configure --quiet --enable-shared
>
> and this produced even worse test results:
>
>   Running gprofng/testsuite/gprofng.display/display.exp ...
>   ERROR: compilation of test program in jsynprog failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in mttest failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>   ERROR: compilation of test program in synprog failed
>
>         === gprofng Summary ===
>
>   # of unresolved testcases    15
>
> The jsynprog and synprog failures is probably expected, since I do not
> have the JDK installed, but the mttest compilation fails because I am
> not testing an installed version of gprofng:
>
> /dev/shm/delme/gprofng/gprofng/tmpdir/root/usr/local/bin/gprofng: 
> error while loading shared libraries: libgprofng.so.0: cannot open 
> shared object file: No such file or directory
>
>
> It would be really nice if I could test a built-but-not-installed
> version of the sources....

  Done.

Please review:
   url:     https://github.com/oracle/binutils-gdb.git/
   branch:  oracle/gprofng-v4-2

Thank you,
-Vladimir


>
> Cheers
>   Nick
>

Re: [PATCH V4] gprofng: a new GNU profiler

[H.J. Lu]

On Mon, Mar 7, 2022 at 1:42 PM Vladimir Mezentsev via Binutils
<binutils@sourceware.org> wrote:
>
>   Hi Nick,
>
> On 3/4/22 05:53, Nick Clifton wrote:
> > Hi Vladimir,
> >
> >>   We have pushed these changes here:
> >>      url: https://github.com/oracle/binutils-gdb.git/
> >>      branch: oracle/gprofng-v4-1
> >
> > OK, so using those sources, I built a set of the binutils binaries
> > without any special configuration options, ie:
> >
> >   configure -- quiet
> >
> > and then I built gprofng and the other binutils tools.  This worked
> > OK, but when I tried to run the gprofng testsuite only a few tests
> > appeared to run:
> >
> >   Running gprofng/testsuite/gprofng.display/display.exp ...
> >   ERROR: compilation of test program in jsynprog failed
> >
> >         === gprofng Summary ===
> >
> >   # of expected passes        14
> >   # of unresolved testcases    1
>
>   It is expected.  JDK is not installed.

You should skip this test if JDK isn't installed.

>
> >
> > Which does not look right.  But then I remembered that I needed to
> > enable shared builds, so I tried:
> >
> >   configure --quiet --enable-shared
> >
> > and this produced even worse test results:
> >
> >   Running gprofng/testsuite/gprofng.display/display.exp ...
> >   ERROR: compilation of test program in jsynprog failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in mttest failed
> >   ERROR: compilation of test program in synprog failed
> >   ERROR: compilation of test program in synprog failed
> >   ERROR: compilation of test program in synprog failed
> >   ERROR: compilation of test program in synprog failed
> >   ERROR: compilation of test program in synprog failed
> >   ERROR: compilation of test program in synprog failed
> >   ERROR: compilation of test program in synprog failed
> >
> >         === gprofng Summary ===
> >
> >   # of unresolved testcases    15
> >
> > The jsynprog and synprog failures is probably expected, since I do not
> > have the JDK installed, but the mttest compilation fails because I am
> > not testing an installed version of gprofng:
> >
> > /dev/shm/delme/gprofng/gprofng/tmpdir/root/usr/local/bin/gprofng:
> > error while loading shared libraries: libgprofng.so.0: cannot open
> > shared object file: No such file or directory
> >
> >
> > It would be really nice if I could test a built-but-not-installed
> > version of the sources....
>
>   Done.
>
> Please review:
>    url:     https://github.com/oracle/binutils-gdb.git/
>    branch:  oracle/gprofng-v4-2
>
> Thank you,
> -Vladimir
>
>
> >
> > Cheers
> >   Nick
> >
>


-- 
H.J.

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

On 3/7/22 13:57, H.J. Lu wrote:
> On Mon, Mar 7, 2022 at 1:42 PM Vladimir Mezentsev via Binutils
> <binutils@sourceware.org> wrote:
>>    Hi Nick,
>>
>> On 3/4/22 05:53, Nick Clifton wrote:
>>> Hi Vladimir,
>>>
>>>>    We have pushed these changes here:
>>>>       url: https://github.com/oracle/binutils-gdb.git/
>>>>       branch: oracle/gprofng-v4-1
>>> OK, so using those sources, I built a set of the binutils binaries
>>> without any special configuration options, ie:
>>>
>>>    configure -- quiet
>>>
>>> and then I built gprofng and the other binutils tools.  This worked
>>> OK, but when I tried to run the gprofng testsuite only a few tests
>>> appeared to run:
>>>
>>>    Running gprofng/testsuite/gprofng.display/display.exp ...
>>>    ERROR: compilation of test program in jsynprog failed
>>>
>>>          === gprofng Summary ===
>>>
>>>    # of expected passes        14
>>>    # of unresolved testcases    1
>>    It is expected.  JDK is not installed.
> You should skip this test if JDK isn't installed.
>
>

  In this case, we will not see that gprofng was built without  java 
profiling support.

-Vladimir

Re: [PATCH V4] gprofng: a new GNU profiler

[Nick Clifton]

Hi Vladimir,

>> It would be really nice if I could test a built-but-not-installed
>> version of the sources....
> 
>   Done.
> 
> Please review:
>    url:     https://github.com/oracle/binutils-gdb.git/
>    branch:  oracle/gprofng-v4-2

Much better.  I think that this branch is ready for merging into
the mainline.

Not being a git expert, I am going to ask for your advice on this
matter.  Do you want to merge that branch into the sourceware
mainline in a way that preserves your commit history ?  If so,
is there a git command that can achieve this ?  Alternatively do
you have a set of patches that I can just apply to the mainline
sources ?  Or maybe patches for the generic code, plus a blanket
import of the gprofng/ directory from the gprofng-v4-2 branch ?

Cheers
   Nick

Re: [PATCH V4] gprofng: a new GNU profiler

[Vladimir Mezentsev]

  Hi Nick,
I sent you a tar file with patches.
Could you please apply to the mainline sources.

Thanks to everyone who worked on gprofng.
Thanks to everyone who helped make gprofng public.

-Vladimir



On 3/9/22 08:36, Nick Clifton wrote:
> Hi Vladimir,
>
>>> It would be really nice if I could test a built-but-not-installed
>>> version of the sources....
>>
>>   Done.
>>
>> Please review:
>>    url:     https://github.com/oracle/binutils-gdb.git/
>>    branch:  oracle/gprofng-v4-2
>
> Much better.  I think that this branch is ready for merging into
> the mainline.
>
> Not being a git expert, I am going to ask for your advice on this
> matter.  Do you want to merge that branch into the sourceware
> mainline in a way that preserves your commit history ?  If so,
> is there a git command that can achieve this ?  Alternatively do
> you have a set of patches that I can just apply to the mainline
> sources ?  Or maybe patches for the generic code, plus a blanket
> import of the gprofng/ directory from the gprofng-v4-2 branch ?
>
> Cheers
>   Nick
>
>

Re: [PATCH V4] gprofng: a new GNU profiler

[Martin Liška]

On 3/9/22 22:13, Vladimir Mezentsev via Binutils wrote:
>   Hi Nick,
> I sent you a tar file with patches.
> Could you please apply to the mainline sources.
> 
> Thanks to everyone who worked on gprofng.
> Thanks to everyone who helped make gprofng public.

Hello.

Can you please take a look at:
https://sourceware.org/bugzilla/show_bug.cgi?id=28962

Thanks,
Martin

> 
> -Vladimir
> 
> 
> 
> On 3/9/22 08:36, Nick Clifton wrote:
>> Hi Vladimir,
>>
>>>> It would be really nice if I could test a built-but-not-installed
>>>> version of the sources....
>>>
>>>   Done.
>>>
>>> Please review:
>>>    url:     https://github.com/oracle/binutils-gdb.git/
>>>    branch:  oracle/gprofng-v4-2
>>
>> Much better.  I think that this branch is ready for merging into
>> the mainline.
>>
>> Not being a git expert, I am going to ask for your advice on this
>> matter.  Do you want to merge that branch into the sourceware
>> mainline in a way that preserves your commit history ?  If so,
>> is there a git command that can achieve this ?  Alternatively do
>> you have a set of patches that I can just apply to the mainline
>> sources ?  Or maybe patches for the generic code, plus a blanket
>> import of the gprofng/ directory from the gprofng-v4-2 branch ?
>>
>> Cheers
>>   Nick
>>
>>
>