[Bug gcov-profile/47618] New: Collecting multiple profiles and using all for PGO

[Bug gcov-profile/47618] New: Collecting multiple profiles and using all for PGO

[roland at rschulz dot eu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

           Summary: Collecting multiple profiles and using all for PGO
           Product: gcc
           Version: 4.6.0
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: P3
         Component: gcov-profile
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: roland@rschulz.eu


Currently only the file from one profiling run can be used for PGO. Especially
for MPI programs it would be nice if several folders containing profiling files
could be merged or several directories could be used together for
-fprofile-use.

For saving the profiling files it would be great if the folder name could
contain an environment variable or could be set by an environment variable.

Thus I suggest that one could either say:
-fprofile-dir /some/path/%q{SOME_ENV}  #same syntax as valgrind
or
export GCC_PROFILE_DIR=/some/path/$SOME_ENV

This would be very useful because MPI implementation provide the MPI rank as a
environment variable. Thus with the suggestion one could store the profile of
each MPI rank in a different folder.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

Steven Bosscher <steven at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2012-07-21
                 CC|                            |steven at gcc dot gnu.org
     Ever Confirmed|0                           |1

--- Comment #1 from Steven Bosscher <steven at gcc dot gnu.org> 2012-07-21 23:57:18 UTC ---
A tool to merge multiple gcda files shoulnd't be very difficult to write. I
don't think this should be done by the compiler itself, that would greatly
complicate things. But a separate tool, gcov-merge say, would work, and this
isn't a big job to create using libgcov (and gcov-dump as an example). You'd
also be able to merge profile information from different directories.

Would something like the above work for you?

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> 2012-07-22 00:42:28 UTC ---
We have one internally at Cavium which is designed to run afterwards and merge
a few gcda file.  It is designed for how we run multi-core programs and write a
gcda file for each run.

And there one here:
http://gcc.gnu.org/ml/gcc-patches/2007-06/msg00423.html

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[xunxun1982 at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

xunxun <xunxun1982 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |xunxun1982 at gmail dot com

--- Comment #3 from xunxun <xunxun1982 at gmail dot com> 2012-07-22 06:56:11 UTC ---
(In reply to comment #1)
> A tool to merge multiple gcda files shoulnd't be very difficult to write. I
> don't think this should be done by the compiler itself, that would greatly
> complicate things. But a separate tool, gcov-merge say, would work, and this
> isn't a big job to create using libgcov (and gcov-dump as an example). You'd
> also be able to merge profile information from different directories.
> 
> Would something like the above work for you?

But for VC and Intel Compiler

they can auto merge all PGO information.

Will we make gcc to have the similar behavior?

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #4 from Andrew Pinski <pinskia at gcc dot gnu.org> 2012-07-22 07:06:47 UTC ---
-fprofile-dir= is already implemented.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #5 from Steven Bosscher <steven at gcc dot gnu.org> 2012-07-22 10:23:37 UTC ---
(In reply to comment #3)
> (In reply to comment #1)
> > A tool to merge multiple gcda files shoulnd't be very difficult to write. I
> > don't think this should be done by the compiler itself, that would greatly
> > complicate things. But a separate tool, gcov-merge say, would work, and this
> > isn't a big job to create using libgcov (and gcov-dump as an example). You'd
> > also be able to merge profile information from different directories.
> > 
> > Would something like the above work for you?
> 
> But for VC and Intel Compiler
> 
> they can auto merge all PGO information.
> 
> Will we make gcc to have the similar behavior?

xunxun,

GCC does merge profile information from different runs into one gcda file. It
works differently from ICC in that ICC produces one .dyn file per test run and
uses prof_merge to generate merge multiple .dyn files into a summary file. GCC
does this merging from multiple runs automatically.

What GCC does not do, is merge multiple gcda files (which would be the
equivalent of merging multiple pgopti.dpi files with ICC).

The issue in this problem report, is that with MPI there will be multiple
images of the same program running simultaneously. The different images can't
share the same set of gcda files (you'd have races) so each image generates its
own set of gcda files. For that, a new merge tool is necessary.

Ideally, this tool would also run transparently. One way to do this could be to
take multiple arguments for -fprofile-dir and merge profile info from each
directory.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #6 from Steven Bosscher <steven at gcc dot gnu.org> 2012-07-22 10:46:30 UTC ---
(In reply to comment #2)
> We have one internally at Cavium which is designed to run afterwards and merge
> a few gcda file.  It is designed for how we run multi-core programs and write a
> gcda file for each run.

And now, of course, you're going to contribute that? ;-)


> And there one here:
> http://gcc.gnu.org/ml/gcc-patches/2007-06/msg00423.html

This merges results for files without their own gcno file but mentioned more
than once in gcda files for multiple source files (e.g. for inline functions in
headers). You can't merge multiple gcda files for one source file, but the
patch does provide the infrastructure to support this.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #7 from Andrew Pinski <pinskia at gcc dot gnu.org> 2012-07-24 22:12:44 UTC ---
Created attachment 27869
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=27869
Patch for adding merge-gcda

here is the patch which adds merge-gcda .  I don't add any testcases as it is
currently designed only for how Cavium's Simple-exec works in that each core
writes out its own gcda file.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #8 from Andrew Pinski <pinskia at gcc dot gnu.org> 2012-07-24 23:20:14 UTC ---
(In reply to comment #7)
> Created attachment 27869 [details]
> Patch for adding merge-gcda

I am changing the copyright over to the FSF based on the fact Cavium (Networks)
has a blanket copyright assignment in place.  I just forgot to do it in the
patch itself.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[roland at rschulz dot eu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #9 from Roland Schulz <roland at rschulz dot eu> 2012-07-24 23:52:41 UTC ---
I think a tool to merge would be a good partial solution.

As far as I can see what would still be missing for user-friendly usage, is a
mechanism to guarantee that all pre-merged files are saved with different
names, so that different processes don't overwrite each others output files. In
the case of MPI one would want to have the mpi rank as part of the output
folder to guarantee unique file names. Thus my suggestion to support
-fprofile-dir /some/path/%q{SOME_ENV}, where SOME_ENV would be the environment
variable containing the mpi rank. Without being able to make the output path
depending on a environment variable one would be required to write some wrapper
scripts and that might not even be possible in all cases.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #10 from Andrew Pinski <pinskia at gcc dot gnu.org> 2012-07-25 00:05:40 UTC ---
> so that different processes don't overwrite each others output files. 

They don't overwrite each other, rather they are merged together at write out
time.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[roland at rschulz dot eu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #11 from Roland Schulz <roland at rschulz dot eu> 2012-07-25 00:50:30 UTC ---
Steven wrote that they are not merged but that race conditions occur. That is
also what I observed. To clarify: Message Passing Interface (MPI) is a
parallelization method which executes the same binary multiple times in
parallel (with support for messages for communication). Allowing to merge the
output into one file at runtime would require file-locking (often over network
file-systems) and would not scale because MPI applications are often used with
more than >10000 (or even >1M) parallel processes simultaneous.

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[steven at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

--- Comment #12 from Steven Bosscher <steven at gcc dot gnu.org> 2012-07-25 08:24:49 UTC ---
(In reply to comment #9)
> I think a tool to merge would be a good partial solution.

We will go with the tool solution. I'll take care of the tool before GCC 4.8,
if that's OK with apinski.

I think we shouldn't have a new tool, though. I'd prefer to teach the gcov
program to do it instead. What would you prefer?


> As far as I can see what would still be missing for user-friendly usage, is a
> mechanism to guarantee that all pre-merged files are saved with different
> names, so that different processes don't overwrite each others output files.

Deeply berried in the GCC manuals is this section:
http://gcc.gnu.org/onlinedocs/gcc-4.7.1/gcc/Cross_002dprofiling.html

With the right combination of GCOV_PREFIX_STRIP and GCOV_PREFIX, it should be
possible to send the gcda files to unique directories per MPI rank. But I think
that a more practical solution is necessary. (I also don't know how these
environment variables interact with -profile-dir. I doubt anyone looked into
this before now...)

I like the %q (and %p) variables from Valgrind, and I don't think it's very
hard to add support for them in libgcov.
(http://valgrind.org/docs/manual/manual-core.html)

[Bug gcov-profile/47618] Collecting multiple profiles and using all for PGO

[ygribov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

Yury Gribov <ygribov at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ygribov at gcc dot gnu.org

--- Comment #29 from Yury Gribov <ygribov at gcc dot gnu.org> ---
> > 1. it can only merge two directories at one time. So, for multiple
> > directories, for example "n", we have to invoke gcov-tool merge n-1 times in
> > order to merge all of them?
> 
> Yep. I guess one can write a simple bash script that does that.

I've added one at
https://github.com/yugr/maintainer-scripts/blob/master/gcov-tool-many