[Bug gcov-profile/95348] New: GCC records zero functions and modules in the profiling data file, ICC does NOT

[Bug gcov-profile/95348] New: GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

            Bug ID: 95348
           Summary: GCC records zero functions and modules in the
                    profiling data file, ICC does NOT
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: qinzhao at gcc dot gnu.org
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

when using GCC and ICC to build a big parallel application with profiling
feedback, the size of the profiling feedback data of GCC is over 20x times
larger than that of ICC. 
As we studied, one of the major reason for this size difference is:
GCC records all functions and modules that execute 0 times, but ICC does NOT. 
since I cannot expose the details of the real application that has the
profiling data size issue. I come up with a small testing case to show this
problem.

******testing case: 
[]$ cat lib.h 
void five (int); 
void ten (int); 
[]$ cat lib.c 
#include <stdio.h> 

void five(int n) { 
        if (n > 5) { 
           printf("%d is greater than five\n", n);     
        } else { 
           printf("%d is not greater than five\n", n);     
        } 
} 

void ten(int n) { 
        if (n > 10) { 
           printf("%d is greater than ten\n", n);     
        } else { 
           printf("%d is not greater than ten\n", n);     
        } 
} 
[]$ cat ten.c 
#include <stdlib.h> 
#include "lib.h" 

int main(int argc, char *argv[]) { 
        if (argc != 2) { 
                return 2; 
        } 

        int n = atoi(argv[1]); 
        ten(n); 
        return 0; 
} 
[]$ cat five.c 
#include "lib.h" 

void foo(int n) { 
 if (n > 0) 
   five(n); 
 return; 
} 

******ICC :
[]$ cat build_it_icc 
#!/bin/bash 
ICC=/ICC-install-dir/bin/icc 
opt="-O0 " 
opt_gen="-prof_gen" 
opt_gen="$opt_gen -prof_dir ./icc_prof_dir" 
tf1="five.c" 
tf2="ten.c" 
libf="lib.c" 

rm *.o ten 
rm -rf icc_prof_dir 
mkdir icc_prof_dir 

echo $ICC $opt $opt_gen -c $tf1 -o five.o 
$ICC $opt $opt_gen -c $tf1 -o five.o 

echo $ICC $opt $opt_gen -c $libf -o lib.o 
$ICC $opt $opt_gen -c $libf -o lib.o 

echo $ICC $opt $opt_gen -c $tf2 -o ten.o 
$ICC $opt $opt_gen -c $tf2 -o ten.o 

echo $ICC $opt $opt_gen ten.o five.o lib.o -o ten 
$ICC $opt $opt_gen ten.o five.o lib.o -o ten 

./ten 12 
echo "Done" 

[]$ sh build_it_icc
then we got the profiling data under 
./icc_prof_dir/5ec6e83f_78751.dyn 
using 
 /ICC-install-dir/bin/profmerge -dump 5ec6e83f_78751.dyn > data 

we can see, only two functions, "main" in ten.c, and "ten" in lib.c have 
records in this profiling data file. 

******GCC: with latest upstream gcc11: 
[]$ cat build_it_gcc
#!/bin/bash 
GCC=/GCC-install-dir/bin/gcc 
opt="-O0 " 
opt="$opt -fno-inline" 
opt_gen="-fprofile-generate" 
opt_gen="$opt_gen -fprofile-dir=gcc_prof_dir/%p" 
tf1="five.c" 
tf2="ten.c" 
libf="lib.c" 

rm -rf gcc_prof_dir 

echo $GCC $opt $opt_gen -c $libf 
$GCC $opt $opt_gen -c $libf -o lib.o 

echo $GCC $opt $opt_gen $tf1 
$GCC $opt $opt_gen -c $tf1 -o five.o 

echo $GCC $opt $opt_gen $tf2 
$GCC $opt $opt_gen -c $tf2 -o ten.o 
echo $GCC $opt $opt_gen ten.o five.o lib.o -o ten 
$GCC $opt $opt_gen ten.o five.o lib.o -o ten 

./ten 12 
echo "Done" 

[]$ build_it_gcc
then the profiling data are under 
./gcc_prof_dir/16856

under ~/Bugs/profile/small_gcc/gcc_prof_dir/16856 
[]$ ls -l 
total 12 
-rw-r--r-- 1 qinzhao qinzhao 100 May 26 19:18 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda 
-rw-r--r-- 1 qinzhao qinzhao 184 May 26 19:18 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda 
-rw-r--r-- 1 qinzhao qinzhao 100 May 26 19:18 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda 

[]$ /home/qinzhao/Install/latest/bin/gcov-dump *.gcda 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda:data:magic `gcda':version 
`B10e' 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda:stamp 1375590637 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda:  a1000000:   2:OBJECT_SUMMARY 
runs=1, sum_max=1 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda:  01000000:   3:FUNCTION 
ident=1636255671, lineno_checksum=0x13fda123, cfg_checksum=0xc7b3f828 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda:    01a10000:   6:COUNTERS 
arcs 3 counts 
#home#qinzhao#Bugs#profile#small_gcc#five.gcda:    01af0000:   2:COUNTERS 
time_profiler 1 counts 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:data:magic `gcda':version 
`B10e' 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:stamp 1375590591 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:  a1000000:   2:OBJECT_SUMMARY 
runs=1, sum_max=1 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:  01000000:   3:FUNCTION 
ident=1977925159, lineno_checksum=0x5bf41dc5, cfg_checksum=0xf9e50e8f 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:    01a10000:   8:COUNTERS arcs 
4 counts 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:    01af0000:   2:COUNTERS 
time_profiler 1 counts 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:  01000000:   3:FUNCTION 
ident=193180204, lineno_checksum=0x020d7b16, cfg_checksum=0xf9e50e8f 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:    01a10000:   8:COUNTERS arcs 
4 counts 
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:    01af0000:   2:COUNTERS 
time_profiler 1 counts 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda:data:magic `gcda':version 
`B10e' 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda:stamp 1375590675 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda:  a1000000:   2:OBJECT_SUMMARY 
runs=1, sum_max=1 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda:  01000000:   3:FUNCTION 
ident=108032747, lineno_checksum=0x2fbc5f5a, cfg_checksum=0x5018cc66 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda:    01a10000:   6:COUNTERS arcs 
3 counts 
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda:    01af0000:   2:COUNTERS 
time_profiler 1 counts 

from the above, there are records for all functions and modules that are 
instrumented, for example, the routine "five" in lib.c, and the whole module 
"five.c".  all the records for these are zero.

Is it possible for GCC to only record functions and modules that have non-zero
counts?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2020-05-27
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
Note there's a difference between zero recorded executions and no recorded data
from uninstrumented code.  That distinction has to be preserved somehow.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org
   Target Milestone|---                         |11.0
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org

--- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
Thank you for the report. It's a known limitation Honza noticed me about.
Is the size problematic from size perspective or speed perspective?
Have you tried compressing the gcda files?

Anyway, I'm going to work on that this stage1.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #3 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #2)
> Thank you for the report. It's a known limitation Honza noticed me about.
> Is the size problematic from size perspective or speed perspective?

I think both size and speed.

for the full execution of our application, the GCC's profiling data is too big
to out of the disk space.  
at the same time, since we have to merge all the profiling data for different
processes, the merging process take a long time due to so many profiling data
files. 

> Have you tried compressing the gcda files?
> 
> Anyway, I'm going to work on that this stage1.

thanks a lot. this is a high priority task for us.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to qinzhao from comment #3)
> (In reply to Martin Liška from comment #2)
> > Thank you for the report. It's a known limitation Honza noticed me about.
> > Is the size problematic from size perspective or speed perspective?
> 
> I think both size and speed.

That would be very similar to what Honza sees for Firefox.

> 
> for the full execution of our application, the GCC's profiling data is too
> big to out of the disk space.  
> at the same time, since we have to merge all the profiling data for
> different processes, the merging process take a long time due to so many
> profiling data files. 

Can you please share some statistics how big are the files and how many runs do
you merge?

Would it be possible to share 'gcov-dump -l' for all your .gcda files?
You can strip the leading filename to make the data completely anonymous.

> 
> > Have you tried compressing the gcda files?
> > 
> > Anyway, I'm going to work on that this stage1.
> 
> thanks a lot. this is a high priority task for us.

Sounds like good motivation for me to implement it.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #5 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #4)> 
> Can you please share some statistics how big are the files and how many runs do you merge?

  There were on the order of 10,000 processes. Source code coverage 
  approximately 20%. Size of the profiling data gathered in the vicinity of
1TB.

> Would it be possible to share 'gcov-dump -l' for all your .gcda files?

It is impossible since too many .gdca files, each process has one directory,
there are over 10,000 directories and under each directory, there are over
thousand .gdca files. 

the situation is similar as the small testing case I added in the first
comment. the functions and modules that do not execute have records in the
.gdca file with zero counts.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #6 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to qinzhao from comment #5)
> (In reply to Martin Liška from comment #4)> 
> > Can you please share some statistics how big are the files and how many runs do you merge?
> 
>   There were on the order of 10,000 processes. Source code coverage 
>   approximately 20%. Size of the profiling data gathered in the vicinity of
> 1TB.

Which means one run takes 100MB is size, right? As you mentioned, having 1000
.gcda files, it means that one takes 0.1MB?

> 
> > Would it be possible to share 'gcov-dump -l' for all your .gcda files?
> 
> It is impossible since too many .gdca files, each process has one directory,
> there are over 10,000 directories and under each directory, there are over
> thousand .gdca files. 
> 
> the situation is similar as the small testing case I added in the first
> comment. the functions and modules that do not execute have records in the
> .gdca file with zero counts.

Can you please provide dump of one directory? At least for portion of .gcda
files?
How is it common that an entire module is empty?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |WAITING

--- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
Ok, I spent some time thinking about your workload and I would recommend the
following steps:

1) You should not generate profile data for each process to a different folder,
but rather merge it.

GCC PGO bootstrap contains ~500 .gcda files where the process is executed
~2000x.
Note that .gcda file merging happens per-file and the file is locked. It should
be a reasonable small window that can delay parallel process execution.

2) I would like to know how long does one process run and what portion is spent
in merging (and dumping) of a profile.

3) You may consider shrinking training run, 10.000 executions seems like a
massive training run to me.

4) GCDA file format is not ideal and can be simply and rapidly shrank by e.g.
gzip. For GCC PGO, it shrinks 10x.

Please provide as much information about the workload so that we can find a
feasible solution.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #8 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #6
> Which means one run takes 100MB is size, right? As you mentioned, having
> 1000 .gcda files, it means that one takes 0.1MB?

around 14000 processes, they are not the same executable, so not all the run
take the same size. some bigger, some smaller. the bigger ones take over 100MB. 

for the bigger one, there are over 5000 .gcda files. 
> Can you please provide dump of one directory? At least for portion of .gcda files?
> How is it common that an entire module is empty?
I will try to get this info for you.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #9 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #7)
> 1) You should not generate profile data for each process to a different
> folder, but rather merge it.

not sure how to do this? can you provide more details on this approach?

> 2) I would like to know how long does one process run and what portion is
> spent in merging (and dumping) of a profile.
will try to get this info.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #10 from Martin Liška <marxin at gcc dot gnu.org> ---
> 
> around 14000 processes, they are not the same executable, so not all the run
>

Both I guess they share the majority of compiled object files, right?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #11 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to qinzhao from comment #9)
> (In reply to Martin Liška from comment #7)
> > 1) You should not generate profile data for each process to a different
> > folder, but rather merge it.
> 
> not sure how to do this? can you provide more details on this approach?

If you have an instrumented binary and you run it multiple times, then each
exit of the application merges profile to existing .gcda files on a disk.

The only exception is a cross-profiling:
https://gcc.gnu.org/onlinedocs/gcc/Cross-profiling.html

where one can use GCOV_PREFIX environment variable to save .gcda files to a
separate location.

Do you use it? Or do you use any of -fprofile-dir options?

> 
> > 2) I would like to know how long does one process run and what portion is
> > spent in merging (and dumping) of a profile.
> will try to get this info.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #12 from Martin Liška <marxin at gcc dot gnu.org> ---
> Do you use it? Or do you use any of -fprofile-dir options?

Ah, ok, you use it. Based on the report:

-fprofile-dir=gcc_prof_dir/%p"

So my recommendation would be not to use it and let GCOV run-time library merge
the profiles. Of course, I'll be interested in `perf report` of such a
instrumented run..

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #13 from Qing Zhao <qing.zhao at oracle dot com> ---
> The only exception is a cross-profiling:
> https://gcc.gnu.org/onlinedocs/gcc/Cross-profiling.html
> 
> where one can use GCOV_PREFIX environment variable to save .gcda files to a
> separate location.
> 
> Do you use it? Or do you use any of -fprofile-dir options?

Yes,  We use -fprofile-dir=pd%p when compile the modules.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #14 from Qing Zhao <qing.zhao at oracle dot com> ---
> 
> -fprofile-dir=gcc_prof_dir/%p"
> 
> So my recommendation would be not to use it and let GCOV run-time library merge
> the profiles. Of course, I'll be interested in `perf report` of such a
> instrumented run..

For our application, all processes generating profiling feedback data to a
single directory seems is not a choice. 
We chose -fprofile-dir=%p and then use gcov-merge to merge them.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #15 from qinzhao at gcc dot gnu.org ---
please refer to 
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47618

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #16 from Martin Liška <marxin at gcc dot gnu.org> ---
> For our application, all processes generating profiling feedback data to a
> single directory seems is not a choice. 

Why is it problem? You need to provide reasoning for that.

> We chose -fprofile-dir=%p and then use gcov-merge to merge them.

Sure, that's possible but you pay with a lot of disk space needed.
Btw. how long does it take to merge all the collected data with gcov-tool?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #17 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #6)

more details:

> 
> Which means one run takes 100MB is size, right? As you mentioned, having
> 1000 .gcda files, it means that one takes 0.1MB?
> 

Out of the 14,239 processes, the amount of gcda data saved is:

- Around 6500 processes in the 120-130M range.
- Around 1000 processes in the 16M-18M range
- Around   10 processes in the 736K - 764K range
- Around 6000 processes in the 8K-32K range

we are mostly interested in those 120-130M range, a typical process in the 130M
bucket has over 5000 gcd files per directory.

> 
> Can you please provide dump of one directory? At least for portion of .gcda
> files?
> How is it common that an entire module is empty?

I compared the GCC profiling data and ICC profiling data for the similar
process, the following are some interesting data:

for GCC, among 5144 modules, there are 4308 empty modules, i.e 83% modules in
GCC are empty;
on function level, among all 187338 functions, only 3524 functions executed,
i.e, 98% functions have zero counts. 

GCC records all the zero count functions and modules, But ICC ONLY records
functions and modules that have non-zero count.

So, GCC's profiling data is MUCH larger than ICC's. 

I believe that this is a big issue that need to be fixed from GCC.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #18 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #16)
> > For our application, all processes generating profiling feedback data to a
> > single directory seems is not a choice. 
> 
> Why is it problem? You need to provide reasoning for that.

this is a long time ago. If I remembered correctly, we tried the scheme that 
all the processes generate profiling feedback data to the single directory, 
but looks like a lot of info got lost, resulting bad performance effect. 
then we switched to this current approach. 

> 
> > We chose -fprofile-dir=%p and then use gcov-merge to merge them.
> 
> Sure, that's possible but you pay with a lot of disk space needed.
> Btw. how long does it take to merge all the collected data with gcov-tool?
I didn't get very accurate info on this, but it's over 24 hours for merging.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #19 from Qing Zhao <qing.zhao at oracle dot com> ---
Hi, Martin,

I attached 3 profiling data files with this email (among over 5000 files under
one typical directory), 
Hope this is helpful.

Thanks.

Qing

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #20 from Qing Zhao <qing.zhao at oracle dot com> ---
Created attachment 48653
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48653&action=edit
A.data

--- Comment #21 from Qing Zhao <qing.zhao at oracle dot com> ---
Created attachment 48654
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48654&action=edit
B.data

--- Comment #22 from Qing Zhao <qing.zhao at oracle dot com> ---
Created attachment 48655
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48655&action=edit
C.data

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #23 from Martin Liška <marxin at gcc dot gnu.org> ---
Created attachment 48660
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48660&action=edit
work-in-progress patch

There's patch that does not stream all zero counters for a function. The patch
only supports streaming and gcov-dump:

$ gcov-dump -l a-foo.gcda
a-foo.gcda:data:magic `gcda':version `B10e'
a-foo.gcda:stamp 1948224357
a-foo.gcda:  a1000000:   2:OBJECT_SUMMARY runs=1, sum_max=1
a-foo.gcda:  01000000:   3:FUNCTION ident=108032747,
lineno_checksum=0x2408bf36, cfg_checksum=0x7cb34af9
a-foo.gcda:    01a10000:   4:COUNTERS arcs 2 counts
a-foo.gcda:                   0: 1 1 
a-foo.gcda:    01af0000:   2:COUNTERS time_profiler 1 counts
a-foo.gcda:                   0: 1 
a-foo.gcda:  01000000:   3:FUNCTION ident=932535731,
lineno_checksum=0x74cbb94c, cfg_checksum=0xeb219516
a-foo.gcda:    01a10000:   4:COUNTERS arcs 2 counts (all zero)
a-foo.gcda:                   0: 0 0 
a-foo.gcda:    01af0000:   2:COUNTERS time_profiler 1 counts (all zero)
a-foo.gcda:                   0: 0 

Can you please test the patch and measure how much does it help for your
test-case?
Note that the FUNCTION section will be streamed (it contains 3 64-bit integers
for each fn:
ident=932535731, lineno_checksum=0x74cbb94c, cfg_checksum=0xeb219516).

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #24 from qinzhao at gcc dot gnu.org ---
with the patch added to gcc11, I tested it with the small testing case, and got
the following data:

******without the patch:
qinzhao@gcc14:~/Bugs/profile/small_gcc/gcc_prof_dir/13248$ ls -l
-rw-r--r-- 1 qinzhao qinzhao 100 Jun  2 19:02
#home#qinzhao#Bugs#profile#small_gcc#five.gcda
-rw-r--r-- 1 qinzhao qinzhao 184 Jun  2 19:02
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda
-rw-r--r-- 1 qinzhao qinzhao 100 Jun  2 19:02
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda

******with the patch:
qinzhao@gcc14:~/Bugs/profile/small_gcc/gcc_prof_dir/20668$ ls -l
-rw-r--r-- 1 qinzhao qinzhao  68 Jun  2 19:34
#home#qinzhao#Bugs#profile#small_gcc#five.gcda
-rw-r--r-- 1 qinzhao qinzhao 144 Jun  2 19:34
#home#qinzhao#Bugs#profile#small_gcc#lib.gcda
-rw-r--r-- 1 qinzhao qinzhao 100 Jun  2 19:34
#home#qinzhao#Bugs#profile#small_gcc#ten.gcda

from the above data, we can see:

1. there are size reduction for "five.c" and "lib.c" as expected.
2. However, we still keep the *.gcda file for five.c even though there is no
any meaningful data in this file. 

I will try to get more data on our real application. 

one question: why not just delete the entire records whose counter is zero and
delete the entire file whose counter is zero?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #25 from Martin Liška <marxin at gcc dot gnu.org> ---
> I will try to get more data on our real application. 
> 
> one question: why not just delete the entire records whose counter is zero
> and delete the entire file whose counter is zero?

Because we need to distinguish in between situations where a function was
really not executed (counter == 0) and the situation where we miss profile for
a function.

One another solution could be the usage of gzip that can shrink file sizes
~10x.
That will be beneficial in situation where you don't merge profiles.

How exactly do you merge profiles? Do you run parallel invocation which can
take log2(N)?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #26 from Qing Zhao <qing.zhao at oracle dot com> ---
> --- Comment #25 from Martin Liška <marxin at gcc dot gnu.org> ---
>> I will try to get more data on our real application. 
>> 
>> one question: why not just delete the entire records whose counter is zero
>> and delete the entire file whose counter is zero?
> 
> Because we need to distinguish in between situations where a function was
> really not executed (counter == 0) and the situation where we miss profile for
> a function.
Understood.  However, is it possible to just provide an option for the user to
choose
to just delete all the zero records and files in order to save more space?
> 
> How exactly do you merge profiles? Do you run parallel invocation which can
> take log2(N)?
We run serial merge adding one at a time right now. 
Is it possible for gcov-merge to add a new functionality to automatically merge
complete
Set of subdirectories?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #27 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to qinzhao from comment #24)
> with the patch added to gcc11, I tested it with the small testing case, and
> got the following data:
> 

I would be more interested in overall statistics for your training scenario.
How much can you get from ~1TB of data?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #28 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Qing Zhao from comment #26)
> > --- Comment #25 from Martin Liška <marxin at gcc dot gnu.org> ---
> >> I will try to get more data on our real application. 
> >> 
> >> one question: why not just delete the entire records whose counter is zero
> >> and delete the entire file whose counter is zero?
> > 
> > Because we need to distinguish in between situations where a function was
> > really not executed (counter == 0) and the situation where we miss profile for
> > a function.
> Understood.  However, is it possible to just provide an option for the user
> to choose
> to just delete all the zero records and files in order to save more space?

It can be possible to add option for that, yes. To be honest, I don't have much
time to work on that right now.

And you still haven't replied to my essential question: Why can't you merge
profiles into one directory during run? Or at least merge to a reasonable
number of folders that you'll merge later?

Note that you can use -fprofile-dir=path with:
%q{VAR}
value of environment variable VAR

which can be used to multiplex into multiple folders for concurrent processes.


> > 
> > How exactly do you merge profiles? Do you run parallel invocation which can
> > take log2(N)?
> We run serial merge adding one at a time right now. 
> Is it possible for gcov-merge to add a new functionality to automatically
> merge complete
> Set of subdirectories?

One would need to run it in multiple threads and I've made a Python script
prototype:
https://github.com/marxin/script-misc/blob/master/gcov-merge-parallel.py

So yes, current GCOV streaming is far from being optimal, but with a small
tweaks we can make it working for your scenario..

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #29 from Qing Zhao <qing.zhao at oracle dot com> ---
> 
> And you still haven't replied to my essential question: Why can't you merge
> profiles into one directory during run? Or at least merge to a reasonable
> number of folders that you'll merge later?
Comment #18 has my answer to the above question.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #30 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Qing Zhao from comment #29)
> > 
> > And you still haven't replied to my essential question: Why can't you merge
> > profiles into one directory during run? Or at least merge to a reasonable
> > number of folders that you'll merge later?
> Comment #18 has my answer to the above question.

The explanation is not sufficient.
You mentioned that merging takes 24 hours, so I would expect that directing
merging will same you quite some time.
Feel free to provide some 'perf report' and I can make some suggestions based
on that.

I'm still missing information like:
- how long does it take the training run?
- how many parallel runs do you have?
- what's the average duration of a process?

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #31 from Qing Zhao <qing.zhao at oracle dot com> ---
> The explanation is not sufficient.
You mean the following explanation: (in comment 18)

we tried the scheme that all the processes generate profiling feedback data to
the single directory, 
but looks like a lot of profiling info got lost, resulting bad performance
effect. 
We thought this might relate to the parallel running environment of our
application, 
then we switched to this current approach. 

Per the documentation of -fprofile-dir=path at:
https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
<https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html>

"When an executable is run in a massive parallel environment, it is recommended
to save profile to different folders.
 That can be done with variables in path that are exported during run-time:
%p
process ID.

%q{VAR}
value of environment variable VAR"


> You mentioned that merging takes 24 hours, so I would expect that directing
> merging will same you quite some time.
Yes, that will help.
But at the same time, if the profiling feedback size can be reduced, then the
time for merging will be reduced too. 

> I'm still missing information like:
> - how long does it take the training run?
I have asked the question too (I don’t have permission to run that real app, so
I need to collect such info from other engineer), but no answer so far.
> - how many parallel runs do you have?
Over 10000 processes are running at the same time. 
> - what's the average duration of a process?
I have asked the question too, no answer so far.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #32 from Qing Zhao <qing.zhao at oracle dot com> ---
> I would be more interested in overall statistics for your training scenario.
> How much can you get from ~1TB of data?

The profile directory generated by the new executable compiled with this patch
was 112G  in size, a lot smaller than previous 1TB. 
Though still bigger than what ICC generated.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #33 from Martin Liška <marxin at gcc dot gnu.org> ---
> 
> The profile directory generated by the new executable compiled with this
> patch was 112G  in size, a lot smaller than previous 1TB. 

That's quite a promising reduction.

> Though still bigger than what ICC generated.

Yep, but we should be only 2x bigger right now?

Can you please test the parallel merging script? I can merge 10GB gcov files
(5000 runs with 264 files each) in about 50s.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qing.zhao at oracle dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #34 from Qing Zhao <qing.zhao at oracle dot com> ---
> 
>> Though still bigger than what ICC generated.
> 
> Yep, but we should be only 2x bigger right now?
Yes, around 2-3 times bigger, much better now.
> 
> Can you please test the parallel merging script? I can merge 10GB gcov files
> (5000 runs with 264 files each) in about 50s.

I will make the request soon (I don’t have the permission to do this). Might
might take some time for others to do this.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #35 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Qing Zhao from comment #34)
> > 
> >> Though still bigger than what ICC generated.
> > 
> > Yep, but we should be only 2x bigger right now?
> Yes, around 2-3 times bigger, much better now.

Fine. I'm going to finalize the patch and send to it GCC mailing list.

> > 
> > Can you please test the parallel merging script? I can merge 10GB gcov files
> > (5000 runs with 264 files each) in about 50s.
> 
> I will make the request soon (I don’t have the permission to do this). Might
> might take some time for others to do this.

Thanks.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #36 from qinzhao at gcc dot gnu.org ---
I found a bug with this proposed patch:
when doing automatic merging, the following error message is emitted:
Merge mismatch for function 1.

the bug can be repeated with the small testing case I added in the comment #1
and the following script:

#!/bin/bash 
GCC=/GCC-install-dir/bin/gcc 
opt="-O0 " 
opt="$opt -fno-inline" 
opt_gen="-fprofile-generate" 
opt_gen="$opt_gen -fprofile-dir=gcc_prof_dir" 
tf1="five.c" 
tf2="ten.c" 
libf="lib.c" 

rm -rf gcc_prof_dir 

echo $GCC $opt $opt_gen -c $libf 
$GCC $opt $opt_gen -c $libf -o lib.o 

echo $GCC $opt $opt_gen $tf1 
$GCC $opt $opt_gen -c $tf1 -o five.o 

echo $GCC $opt $opt_gen $tf2 
$GCC $opt $opt_gen -c $tf2 -o ten.o 
echo $GCC $opt $opt_gen ten.o five.o lib.o -o ten 
$GCC $opt $opt_gen ten.o five.o lib.o -o ten 

./ten 12 
./ten 11
echo "Done" 

[qinzhao@localhost small_gcc]$ sh build_it_gcc
/home/qinzhao/Install/latest-821/bin/gcc -O0 -fno-inline -fprofile-generate
-fprofile-dir=gcc_prof_dir -c lib.c
/home/qinzhao/Install/latest-821/bin/gcc -O0 -fno-inline -fprofile-generate
-fprofile-dir=gcc_prof_dir five.c
/home/qinzhao/Install/latest-821/bin/gcc -O0 -fno-inline -fprofile-generate
-fprofile-dir=gcc_prof_dir ten.c
/home/qinzhao/Install/latest-821/bin/gcc -O0 -fno-inline -fprofile-generate
-fprofile-dir=gcc_prof_dir ten.o five.o lib.o -o ten
12 is greater than ten
11 is greater than ten
profiling:gcc_prof_dir/#home#qinzhao#Bugs#profile#small_gcc#lib.gcda:Merge
mismatch for function 1
profiling:gcc_prof_dir/#home#qinzhao#Bugs#profile#small_gcc#five.gcda:Merge
mismatch for function 0
Done

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #37 from qinzhao at gcc dot gnu.org ---
So, the previous prof data size for the real application might not be correct. 
After this bug is fixed, we might need to collect the new real code size
reduction.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #48660|0                           |1
        is obsolete|                            |

--- Comment #38 from Martin Liška <marxin at gcc dot gnu.org> ---
Created attachment 48738
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48738&action=edit
Patch candidate v2

There's a properly tested patch that supports all operations.
Please apply also pending patch:
https://gcc.gnu.org/pipermail/gcc-patches/2020-June/548206.html

and let me know what are numbers now.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[qinzhao at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #39 from qinzhao at gcc dot gnu.org ---
(In reply to Martin Liška from comment #38)
> Created attachment 48738 [details]
> Patch candidate v2
I have added this patch to my private gcc 8 with some change, works fine with
the small testing case, and fixed the bug I reported in comment #36. 
> 
> There's a properly tested patch that supports all operations.
> Please apply also pending patch:
> https://gcc.gnu.org/pipermail/gcc-patches/2020-June/548206.html

do I need to apply this as well in order to get the data from the real
application? (note I need to apply the patch to gcc 8, this patch is not easy
to be applied to gcc 8).

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |ASSIGNED

--- Comment #40 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to qinzhao from comment #39)
> (In reply to Martin Liška from comment #38)
> > Created attachment 48738 [details]
> > Patch candidate v2
> I have added this patch to my private gcc 8 with some change, works fine
> with the small testing case, and fixed the bug I reported in comment #36. 

Great! Please report collected statistics for the patch and I'm also curious
about a parallel gcov-tool merging?

> > 
> > There's a properly tested patch that supports all operations.
> > Please apply also pending patch:
> > https://gcc.gnu.org/pipermail/gcc-patches/2020-June/548206.html
> 
> do I need to apply this as well in order to get the data from the real
> application? (note I need to apply the patch to gcc 8, this patch is not
> easy to be applied to gcc 8).

This patch is already applied to master and it does not effect GCC 8 branch.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #41 from Martin Liška <marxin at gcc dot gnu.org> ---
The patch survives PGO bootstrap of GCC and it shrinks size of gcda
file from 17MB to 12MB.

And compression can achieve the following:

zstd: 3.3 MB

$ time zstd *
real    0m0.082s
user    0m0.068s
sys    0m0.013s

gzip: 3.3 MB

$ time gzip *
real    0m0.357s
user    0m0.328s
sys    0m0.029s

So using compression is a very promising approach.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

--- Comment #42 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Martin Liska <marxin@gcc.gnu.org>:

https://gcc.gnu.org/g:ece21ff6ea9d969d3b6aae82136622a7126eefc1

commit r11-1778-gece21ff6ea9d969d3b6aae82136622a7126eefc1
Author: Martin Liska <mliska@suse.cz>
Date:   Tue Jun 2 10:11:07 2020 +0200

    Do not stream all zeros for gcda files.

    gcc/ChangeLog:

            PR gcov-profile/95348
            * coverage.c (read_counts_file): Read only COUNTERS that are
            not all-zero.
            * gcov-dump.c (tag_function): Change signature from unsigned to
            signed integer.
            (tag_blocks): Likewise.
            (tag_arcs): Likewise.
            (tag_lines): Likewise.
            (tag_counters): Likewise.
            (tag_summary): Likewise.
            * gcov.c (read_count_file): Read all non-zero counters
            sensitively.

    libgcc/ChangeLog:

            PR gcov-profile/95348
            * libgcov-driver.c (merge_one_data): Merge only profiles
            that are not of non-zero type.
            (write_one_data): Write counters only if there's one non-zero
            value.
            * libgcov-util.c (tag_function): Change signature from unsigned
            to int.
            (tag_blocks): Likewise.
            (tag_arcs): Likewise.
            (tag_counters): Likewise.
            (tag_summary): Likewise.
            (tag_lines): Read only if COUNTERS is non-zero.
            (read_gcda_file): Handle negative length for COUNTERS type.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|marxin at gcc dot gnu.org          |unassigned at gcc dot gnu.org

--- Comment #43 from Martin Liška <marxin at gcc dot gnu.org> ---
Partially fixed, I'm resetting me from the assignee.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |NEW

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|11.0                        |11.2

--- Comment #44 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
GCC 11.1 has been released, retargeting bugs to GCC 11.2.

[Bug gcov-profile/95348] GCC records zero functions and modules in the profiling data file, ICC does NOT

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95348

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|11.2                        |---