[Bug gcov-profile/99105] New: profile streaming scales poorly to projects with many source files

[Bug gcov-profile/99105] New: profile streaming scales poorly to projects with many source files

[hubicka at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

            Bug ID: 99105
           Summary: profile streaming scales poorly to projects with many
                    source files
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: gcov-profile
          Assignee: unassigned at gcc dot gnu.org
          Reporter: hubicka at gcc dot gnu.org
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

Compared to clang we need significantly longer time to train Firefox (25
minutes compared to 7) and run clang 
make check-clang
which takes 12 hours compared to 27 minutes.

Most of time is spent by kernel by IO.  I suppose we really should consider
optionally producing per-binary rather then per-source file profile data dumps
and omit untrained parts of program.
This is perf top of running llvm testsuite (first with and second without
kernel symbols).  Seems merging of topn is high in profile now.


Overhead  Shared Object            Symbol                                       
   8.58%  libc-2.32.so             [.] read                                     
   7.24%  [kernel]                 [k] __x86_indirect_thunk_rax                 
   7.15%  [kernel]                 [k] entry_SYSCALL_64                         
   6.43%  [kernel]                 [k] __x64_sys_read                           
   5.70%  [kernel]                 [k] apparmor_file_permission                 
   5.49%  [kernel]                 [k] generic_file_buffered_read               
   4.45%  [kernel]                 [k] btrfs_file_read_iter                     
   4.10%  [kernel]                 [k] syscall_return_via_sysret                
   3.31%  [kernel]                 [k] new_sync_read                            
   3.07%  libc-2.32.so             [.] _IO_file_xsgetn                          
   2.77%  [kernel]                 [k] find_get_entry                           
   2.76%  libc-2.32.so             [.] _IO_fread                                
   2.60%  [kernel]                 [k] current_time                             
   2.33%  [kernel]                 [k] atime_needs_update                       
   2.18%  [kernel]                 [k] vfs_read                                 
   2.11%  clang-11                 [.] __gcov_merge_topn                        
   2.02%  [kernel]                 [k] pagecache_get_page                       
   1.97%  [kernel]                 [k] entry_SYSCALL_64_after_hwframe           
   1.89%  clang-11                 [.] gcov_read_words                          
   1.76%  [kernel]                 [k] __fsnotify_parent                        
   1.67%  [kernel]                 [k] syscall_exit_to_user_mode                
   1.60%  [kernel]                 [k] ksys_read                                
   1.40%  [kernel]                 [k] security_file_permission                 
   1.30%  [kernel]                 [k] aa_file_perm                             
   1.23%  [kernel]                 [k] syscall_enter_from_user_mode             
   1.11%  [kernel]                 [k] touch_atime                              
   1.02%  [kernel]                 [k] exit_to_user_mode_prepare                
   0.99%  [kernel]                 [k] xas_load                                 
   0.95%  [kernel]                 [k] xas_start                                
   0.74%  [kernel]                 [k] __fget_light                             
   0.71%  [kernel]                 [k] __fdget_pos                              
   0.69%  clang-11                 [.] __gcov_read_counter                      
   0.64%  [kernel]                 [k] do_syscall_64                            
   0.58%  [kernel]                 [k] ktime_get_coarse_real_ts64               
   0.55%  [kernel]                 [k] rw_verify_area                           
   0.50%  libc-2.32.so             [.] _IO_sgetn                                
   0.50%  [kernel]                 [k] PageHuge                                 
   0.45%  perf                     [.] rb_next                                  
   0.38%  [kernel]                 [k] iov_iter_init                            
For a higher level overview, try: perf top --sort comm,dso                      



Overhead  Shared Object                           Symbol                        
  43.43%  libc-2.32.so                            [.] read                      
  12.00%  libc-2.32.so                            [.] _IO_file_xsgetn           
  11.80%  libc-2.32.so                            [.] _IO_fread                 
   7.89%  clang-11                                [.] __gcov_merge_topn         
   7.28%  clang-11                                [.] gcov_read_words           
   2.32%  clang-11                                [.] __gcov_read_counter       
   2.28%  libc-2.32.so                            [.] _IO_sgetn                 
   2.08%  FileCheck                               [.] __gcov_merge_topn         
   1.46%  FileCheck                               [.] gcov_read_words           
   1.23%  perf                                    [.] rb_next                   
   1.08%  perf                                    [.] __symbols__insert         
   0.87%  libc-2.32.so                            [.] _IO_file_read             
   0.72%  clang-11                                [.] gcov_do_dump              
   0.38%  FileCheck                               [.] __gcov_read_counter       
   0.28%  perf                                    [.] rust_demangle_callback    
   0.25%  libc-2.32.so                            [.] _int_malloc               
   0.19%  clang-11                                [.] gcov_write_words          
   0.18%  libc-2.32.so                            [.] __strchr_avx2             
   0.18%  clang-11                                [.] fread@plt                 
   0.18%  libc-2.32.so                            [.] __libc_calloc             
   0.17%  perf                                    [.] dso__load_sym             
   0.15%  perf                                    [.] symbol__new               
   0.14%  perf                                    [.] rb_insert_color           
   0.11%  libc-2.32.so                            [.] __strlen_avx2             
   0.10%  perf                                    [.] 0x000000000087755b        
   0.08%  libc-2.32.so                            [.]
__memmove_avx_unaligned_erms                                                    
   0.08%  perf                                    [.] evsel__parse_sample       
   0.07%  libc-2.32.so                            [.] sysmalloc                 
   0.07%  perf                                    [.] symbols__fixup_end        
   0.07%  perf                                    [.] eprintf                   
   0.07%  libc-2.32.so                            [.]
__memset_avx2_unaligned_erms                                                    
   0.07%  libc-2.32.so                            [.] cfree@GLIBC_2.2.5         
   0.06%  perf                                    [.] bfd_demangle              
   0.06%  perf                                    [.] rust_demangle             
   0.05%  perf                                    [.] cplus_demangle            
   0.05%  libc-2.32.so                            [.] _int_free                 
   0.05%  libpthread-2.32.so                      [.] __pthread_mutex_init      
   0.04%  perf                                    [.] cplus_demangle_v3         
   0.04%  FileCheck                               [.] fread@plt                 
For a higher level overview, try: perf top --sort comm,dso

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #1 from Jan Hubicka <hubicka at gcc dot gnu.org> ---
I use following:

cmake -G 'Unix Makefiles' /home/jan/llvm-project/llvm
-DCLANG_TABLEGEN=/home/jan/llmm-build2-lto-fdo/stage1/bin/clang-tblgen
-DCMAKE_AR=/home/jan/trunk-instal/bin/gcc-ar -DCMAKE_BUILD_TY
PE=Release -DCMAKE_CXX_COMPILER=/home/jan/trunk-install/bin/g++
-DCMAKE_C_COMPILER=/home/jan/trunk-install/bin/gcc
'-DCMAKE_INSTALL_PREFIX=~/llvm11-install-gcc-instrument-lto' -DCMAKE_RA
NLIB=/home/jan/trunk-install/bin/gcc-ranlib
-DLLVM_BINUTILS_INCDIR=/home/jan//binutils-gdb/include  -DLLVM_BUILD_RUNTIME=No
-DLLVM_TABLEGEN=/home/jan/llmm-build2-lto-fdo/stage1/bin/llvm-
tblgen -DCMAKE_AR=/home/jan/trunk-install/bin/gcc-ar -DCMAKE_C_FLAGS="-O3 -flto
-fprofile-generate -fno-semantic-interposition" -DCMAKE_CXX_FLAGS="-O3 -flto
-fprofile-generate -fno-seman
tic-interposition"
make -j16 clang lld LLVMgold
make -j16 check-llvm check-clang


which is extracted from llvm profile feedback building script.

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
Thank you for the bug report. It's really something we should improve for the
next GCC release.
A small improvement can be achieved by the removal of libgcov I/O buffering:
https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=5a17015c096012b9e43a8dd45768a8d5fb3a3aee

But the key thing is likely the ability to omit profile modifications
(read/modify/write) for parts of a binary that are not trained.

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2021-02-15
     Ever confirmed|0                           |1
   Target Milestone|---                         |12.0
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org

Re: [Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[Jan Hubicka]

> A small improvement can be achieved by the removal of libgcov I/O buffering:
> https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=5a17015c096012b9e43a8dd45768a8d5fb3a3aee

So it effectively replaces gcov's own buffered I/O by stdio.  First I am
not sure how safe it is (as we had a lot of fun about using malloc) and
also it adds dependency on stdio that is not necessarily good idea for
embedded targets. Not sure how often it is used there.

But why glibc stdio is more effective? Is it because our buffer size of
1k is way too small (as it seems juding from the profile that is
dominated by fread calls rather than open/lock/close)?
> 
> But the key thing is likely the ability to omit profile modifications
> (read/modify/write) for parts of a binary that are not trained.
Problem there are the per-program summaries that needs to be updated
even for files never visited.

It seems that producing one file with tar-like format that can be
expanded to gcda files by gcov-tool would be good idea. Even if we need
to lock whole file it is probably faster than a lot of small I/Os.
To avoid waiting for lock one can simply allow multiple profile files to
be created and teach libgcov to acquire unlocked file in pseudorandom
order.

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #3 from Jan Hubicka <hubicka at ucw dot cz> ---
> A small improvement can be achieved by the removal of libgcov I/O buffering:
> https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=5a17015c096012b9e43a8dd45768a8d5fb3a3aee

So it effectively replaces gcov's own buffered I/O by stdio.  First I am
not sure how safe it is (as we had a lot of fun about using malloc) and
also it adds dependency on stdio that is not necessarily good idea for
embedded targets. Not sure how often it is used there.

But why glibc stdio is more effective? Is it because our buffer size of
1k is way too small (as it seems juding from the profile that is
dominated by fread calls rather than open/lock/close)?
> 
> But the key thing is likely the ability to omit profile modifications
> (read/modify/write) for parts of a binary that are not trained.
Problem there are the per-program summaries that needs to be updated
even for files never visited.

It seems that producing one file with tar-like format that can be
expanded to gcda files by gcov-tool would be good idea. Even if we need
to lock whole file it is probably faster than a lot of small I/Os.
To avoid waiting for lock one can simply allow multiple profile files to
be created and teach libgcov to acquire unlocked file in pseudorandom
order.

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Jan Hubicka from comment #3)
> > A small improvement can be achieved by the removal of libgcov I/O buffering:
> > https://gcc.gnu.org/git/?p=gcc.git;a=patch;h=5a17015c096012b9e43a8dd45768a8d5fb3a3aee
> 
> So it effectively replaces gcov's own buffered I/O by stdio.  First I am
> not sure how safe it is (as we had a lot of fun about using malloc)

Why is not safe? We use filesystem locking for .gcda file.

> also it adds dependency on stdio that is not necessarily good idea for
> embedded targets. Not sure how often it is used there.

It was motivated by PR97834. Well, I think it's better to rely on a system C
library
as it provides a faster implementation of buffered I/O.

For embedded targets, I plan to implement hooks that can be used instead of
I/O:
https://gcc.gnu.org/pipermail/gcc-patches/2020-November/559342.html

> 
> But why glibc stdio is more effective? Is it because our buffer size of
> 1k is way too small (as it seems juding from the profile that is
> dominated by fread calls rather than open/lock/close)?

It behaved the same on my machine, but BSD impact was more significant.

> > 
> > But the key thing is likely the ability to omit profile modifications
> > (read/modify/write) for parts of a binary that are not trained.
> Problem there are the per-program summaries that needs to be updated
> even for files never visited.
> 
> It seems that producing one file with tar-like format that can be
> expanded to gcda files by gcov-tool would be good idea. Even if we need
> to lock whole file it is probably faster than a lot of small I/Os.

I'm planning to collect more detailed statistics about why is a lot of small
I/Os slower.

In the case of Clang, I would expect 100s (or even 1000s) of object files.
During profiling
run (using all cores), I would expect each run takes 100ms (or even seconds),
so waiting
for a file lock of an object file should not block it much.

> To avoid waiting for lock one can simply allow multiple profile files to
> be created and teach libgcov to acquire unlocked file in pseudorandom
> order.
> 
> Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #5 from Jan Hubicka <hubicka at ucw dot cz> ---
> > So it effectively replaces gcov's own buffered I/O by stdio.  First I am
> > not sure how safe it is (as we had a lot of fun about using malloc)
> 
> Why is not safe? We use filesystem locking for .gcda file.

Because user apps may do funny thins with stdio such as they do with
malloc.  Fewer library stuff we rely on, the less likely we will hit the
problems.  So I am not sure if simply fixing i/o isn't better approach,
but I do not know.
> 
> > also it adds dependency on stdio that is not necessarily good idea for
> > embedded targets. Not sure how often it is used there.
> 
> It was motivated by PR97834. Well, I think it's better to rely on a system C
> library
> as it provides a faster implementation of buffered I/O.
> 
> For embedded targets, I plan to implement hooks that can be used instead of
> I/O:
> https://gcc.gnu.org/pipermail/gcc-patches/2020-November/559342.html
> 
> > 
> > But why glibc stdio is more effective? Is it because our buffer size of
> > 1k is way too small (as it seems juding from the profile that is
> > dominated by fread calls rather than open/lock/close)?
> 
> It behaved the same on my machine, but BSD impact was more significant.

Clang training seems to be a good extreme testcase and not that hard to
set up. It is relatively large testsuite and streaming is clearly
dominating over everything.

Profile also seems quite clear that read dominates other syscall
overhead.
> 
> I'm planning to collect more detailed statistics about why is a lot of small
> I/Os slower.

>From the perf it seems that simply the syscall overhead plays important
role (about 20% at kernel side, plus 9% on glibc side) followed by some
stupidness of opensuse setup - apparmor and btrfs.
> 
> In the case of Clang, I would expect 100s (or even 1000s) of object files.
> During profiling
> run (using all cores), I would expect each run takes 100ms (or even seconds),
> so waiting
> for a file lock of an object file should not block it much.

2727 gcda files, 44MB overall, 4MB xz compressed tar file.
I am actually surprised that the file count is quite small. Firefox has
more...

Honza
> 
> > To avoid waiting for lock one can simply allow multiple profile files to
> > be created and teach libgcov to acquire unlocked file in pseudorandom
> > order.
> > 
> > Honza
> 
> -- 
> You are receiving this mail because:
> You reported the bug.

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #6 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Jan Hubicka from comment #5)
> > > So it effectively replaces gcov's own buffered I/O by stdio.  First I am
> > > not sure how safe it is (as we had a lot of fun about using malloc)
> > 
> > Why is not safe? We use filesystem locking for .gcda file.
> 
> Because user apps may do funny thins with stdio such as they do with
> malloc.  Fewer library stuff we rely on, the less likely we will hit the
> problems.  So I am not sure if simply fixing i/o isn't better approach,
> but I do not know.

Sure. With the patch, we don't rely on any glibc feature. We will just
use a default read/write IO (which uses a buffering internally).

> > 
> > > also it adds dependency on stdio that is not necessarily good idea for
> > > embedded targets. Not sure how often it is used there.
> > 
> > It was motivated by PR97834. Well, I think it's better to rely on a system C
> > library
> > as it provides a faster implementation of buffered I/O.
> > 
> > For embedded targets, I plan to implement hooks that can be used instead of
> > I/O:
> > https://gcc.gnu.org/pipermail/gcc-patches/2020-November/559342.html
> > 
> > > 
> > > But why glibc stdio is more effective? Is it because our buffer size of
> > > 1k is way too small (as it seems juding from the profile that is
> > > dominated by fread calls rather than open/lock/close)?
> > 
> > It behaved the same on my machine, but BSD impact was more significant.
> 
> Clang training seems to be a good extreme testcase and not that hard to
> set up. It is relatively large testsuite and streaming is clearly
> dominating over everything.

Sure, I'll set it up.

> 
> Profile also seems quite clear that read dominates other syscall
> overhead.
> > 
> > I'm planning to collect more detailed statistics about why is a lot of small
> > I/Os slower.
> 
> From the perf it seems that simply the syscall overhead plays important
> role (about 20% at kernel side, plus 9% on glibc side) followed by some
> stupidness of opensuse setup - apparmor and btrfs.

Yes, that's pretty obvious from the profile.

> > 
> > In the case of Clang, I would expect 100s (or even 1000s) of object files.
> > During profiling
> > run (using all cores), I would expect each run takes 100ms (or even seconds),
> > so waiting
> > for a file lock of an object file should not block it much.
> 
> 2727 gcda files, 44MB overall, 4MB xz compressed tar file.
> I am actually surprised that the file count is quite small. Firefox has
> more...

To be honest, it's very small file size. I would expect these files should
definitely live in a page cache.

What type of disk do you use?

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #7 from Jan Hubicka <hubicka at ucw dot cz> ---
> > Because user apps may do funny thins with stdio such as they do with
> > malloc.  Fewer library stuff we rely on, the less likely we will hit the
> > problems.  So I am not sure if simply fixing i/o isn't better approach,
> > but I do not know.
> 
> Sure. With the patch, we don't rely on any glibc feature. We will just
> use a default read/write IO (which uses a buffering internally).

Well, buffered i/o is library feature :)
> > 2727 gcda files, 44MB overall, 4MB xz compressed tar file.
> > I am actually surprised that the file count is quite small. Firefox has
> > more...
> 
> To be honest, it's very small file size. I would expect these files should
> definitely live in a page cache.

For Firefox it is 2500 gcda files, 75MB overall, 6MB compressed.
> 
> What type of disk do you use?

/dev/nvme0n1     7QG00HVT             Seagate FireCuda 520 SSD ZP1000GM30002  
1           1.00  TB /   1.00  TB    512   B +  0 B STNSC014

(it is our new zen3 machine)

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
This is what I see for GCC PGO in train stage. It's from perf top:

   4.33%  cc1plus                          [.] __gcov_indirect_call_profiler_v4
                                 ◆
   2.28%  cc1plus                          [.] __gcov_topn_values_profiler     
                                 ▒
   0.85%  cc1plus                          [.] ggc_internal_alloc              
                                 ▒
   0.83%  [kernel]                         [k] perf_event_task_tick            
                                 ▒
   0.72%  libc-2.32.so                     [.] _int_malloc                     
                                 ▒
   0.71%  cc1plus                          [.] ht_lookup_with_hash             
                                 ▒
   0.53%  cc1plus                          [.] grokdeclarator                  
                                 ▒
   0.48%  cc1plus                          [.] df_note_compute                 
                                 ▒
   0.47%  cc1plus                          [.] get_ref_base_and_extent         
                                 ▒
   0.45%  [kernel]                         [k] clear_page_rep                  
                                 ▒
   0.44%  cc1plus                          [.] _cpp_lex_direct                 
                                 ▒
   0.41%  cc1plus                          [.] walk_tree_1                     
                                 ▒
   0.41%  cc1plus                          [.] et_splay                        
                                 ▒
   0.41%  cc1plus                          [.] bitmap_set_bit                  
                                 ▒
   0.40%  libc-2.32.so                     [.] _int_free                       
                                 ▒
   0.39%  cc1plus                          [.] bitmap_list_find_element        
                                 ▒
   0.36%  libc-2.32.so                     [.] malloc                          
                                 ▒
   0.35%  cc1plus                          [.] operand_compare::operand_equal_p
                                 ▒
   0.35%  cc1plus                          [.] hash_table<named_decl_hash,
false, xcallocator>::find_slot_with_ha▒

In the case of GCC, we emit 500 .gcda files.

@Honza: Can you please test my patch that uses glibc buffered I/O if it helps?

Re: [Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[Jan Hubicka]

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105
> 
> --- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
> This is what I see for GCC PGO in train stage. It's from perf top:
> 
>    4.33%  cc1plus                          [.] __gcov_indirect_call_profiler_v4
>                                  ◆
>    2.28%  cc1plus                          [.] __gcov_topn_values_profiler     
>                                  ▒
>    0.85%  cc1plus                          [.] ggc_internal_alloc              

Yep, this is usual profile I see.  Perhaps you want to try profile "make check"
> 
> In the case of GCC, we emit 500 .gcda files.
> 
> @Honza: Can you please test my patch that uses glibc buffered I/O if it helps?

I can give it a try later this week (I would like to collect some data on performance first)

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #9 from Jan Hubicka <hubicka at ucw dot cz> ---
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105
> 
> --- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
> This is what I see for GCC PGO in train stage. It's from perf top:
> 
>    4.33%  cc1plus                          [.] __gcov_indirect_call_profiler_v4
>                                  ◆
>    2.28%  cc1plus                          [.] __gcov_topn_values_profiler     
>                                  ▒
>    0.85%  cc1plus                          [.] ggc_internal_alloc              

Yep, this is usual profile I see.  Perhaps you want to try profile "make check"
> 
> In the case of GCC, we emit 500 .gcda files.
> 
> @Honza: Can you please test my patch that uses glibc buffered I/O if it helps?

I can give it a try later this week (I would like to collect some data on
performance first)

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #10 from Martin Liška <marxin at gcc dot gnu.org> ---
> From the perf it seems that simply the syscall overhead plays important
> role (about 20% at kernel side, plus 9% on glibc side) followed by some
> stupidness of opensuse setup - apparmor and btrfs.

And please remeasure with the AppArmor disabled.
It may slow down each I/O-related syscall rapidly!

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #11 from Martin Liška <marxin at gcc dot gnu.org> ---
> Yep, this is usual profile I see.  Perhaps you want to try profile "make
> check"

Ah, yeah, that will make a big difference.
So clang is using 'make check', running a test-suite for a PGO build, right?

Re: [Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[Jan Hubicka]

> Ah, yeah, that will make a big difference.
> So clang is using 'make check', running a test-suite for a PGO build, right?
It uses 
make check-llvm
make check-clang
and then it rebuilds whole llvm with the instrumented compiler.

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #12 from Jan Hubicka <hubicka at ucw dot cz> ---
> Ah, yeah, that will make a big difference.
> So clang is using 'make check', running a test-suite for a PGO build, right?
It uses 
make check-llvm
make check-clang
and then it rebuilds whole llvm with the instrumented compiler.

Honza

[Bug gcov-profile/99105] profile streaming scales poorly to projects with many source files

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #13 from Jan Hubicka <hubicka at ucw dot cz> ---
> And please remeasure with the AppArmor disabled.
> It may slow down each I/O-related syscall rapidly!

I tried disabling apparmor and it does not make much difference..

[Bug gcov-profile/99105] [11 regression] profile streaming scales poorly to projects with many source files

[hubicka at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

Jan Hubicka <hubicka at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|profile streaming scales    |[11 regression] profile
                   |poorly to projects with     |streaming scales poorly to
                   |many source files           |projects with many source
                   |                            |files

--- Comment #14 from Jan Hubicka <hubicka at gcc dot gnu.org> ---
With GCC10 the testuiste seems to run reasonably fast (in about 20 minutes
too).  Profile is
  21.38%  clang-11                                         [.] gcov_do_dump     
   9.99%  clang-11                                         [.] gcov_read_words  
   5.42%  [kernel]                                         [k]
copy_user_enhanced_fast_string                                                  
   2.68%  FileCheck                                        [.] lstep            
   2.63%  [kernel]                                         [k] clear_page_erms  
   2.53%  clang-11                                         [.] __gcov_merge_add 
   2.23%  clang-11                                         [.] gcov_write_words 
   1.93%  clang-11                                         [.]
__gcov_read_counter                                                             
   1.50%  clang-11                                         [.]
__gcov_merge_topn                                                               
   1.29%  [kernel]                                         [k]
native_queued_spin_lock_slowpath                                                
   1.26%  [kernel]                                         [k] alloc_set_pte    
   1.20%  [kernel]                                         [k]
__x86_indirect_thunk_rax                                                        
   1.05%  FileCheck                                        [.] sstep            
   0.79%  [kernel]                                         [k] _raw_spin_lock   
   0.78%  [kernel]                                         [k] kmem_cache_free  
   0.78%  [kernel]                                         [k]
generic_file_buffered_read                                                      
   0.72%  [kernel]                                         [k] kmem_cache_alloc 
   0.69%  [kernel]                                         [k] kfree            
   0.64%  [kernel]                                         [k] page_remove_rmap 
   0.63%  [kernel]                                         [k]
filemap_map_pages                                                               
   0.60%  [kernel]                                         [k] __slab_free      
   0.55%  [kernel]                                         [k] handle_mm_fault

So I am marking this a regression at least until we analyze it better.

[Bug gcov-profile/99105] [11 regression] profile streaming scales poorly to projects with many source files

[hubicka at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #15 from Jan Hubicka <hubicka at gcc dot gnu.org> ---
GCC10 testing time is:
Testing Time: 656.80s
  Unsupported      :   104
  Passed           : 21273
  Expectedly Failed:    26

I will see tomorrow if I can get GCC11 testing to finish.

[Bug gcov-profile/99105] [11 regression] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #16 from Martin Liška <marxin at gcc dot gnu.org> ---
I've got it:

(gdb) bt
#0  __gcov_merge_topn (counters=<optimized out>, n_counters=<optimized out>)
    at ../../../libgcc/libgcov-merge.c:119
#1  0x0000000007d57150 in merge_one_data (summary=<synthetic pointer>,
gi_ptr=0xbb74ae0,
    filename=0xd78d360
"/home/marxin/Programming/llvm-project/objdir2/tools/clang/lib/Basic/CMakeFiles/obj.cl
angBasic.dir/Targets.cpp.gcda") at ../../../libgcc/libgcov-driver.c:314
#2  dump_one_gcov (run_max=116847, run_counted=0, gf=<synthetic pointer>,
gi_ptr=0xbb74ae0)
    at ../../../libgcc/libgcov-driver.c:492
#3  gcov_do_dump (list=<optimized out>, run_counted=0) at
../../../libgcc/libgcov-driver.c:555
#4  0x0000000007d578d4 in __gcov_dump_one (root=0xd733740 <__gcov_root>)
    at ../../../libgcc/libgcov-driver.c:578
#5  __gcov_dump_one (root=0xd733740 <__gcov_root>) at
../../../libgcc/libgcov-driver.c:573
#6  __gcov_exit () at ../../../libgcc/libgcov-driver.c:600
#7  0x0000000000a68579 in _sub_D_00100_2.2 ()
#8  0x00007ffff7fe1c73 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#9  0x00007ffff7a37347 in __run_exit_handlers () from /lib64/libc.so.6
#10 0x00007ffff7a374f0 in exit () from /lib64/libc.so.6
#11 0x00007ffff7a1f159 in __libc_start_main () from /lib64/libc.so.6
#12 0x0000000000af225e in _start () at ../sysdeps/x86_64/start.S:120

where we try to merge chain with:
(gdb) p n
$10 = 28114944

items.

[Bug gcov-profile/99105] [11 regression] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #17 from Martin Liška <marxin at gcc dot gnu.org> ---
I've got a patch candidate.
Using the patch, make -j128 check-clang takes 6 minutes.

[Bug gcov-profile/99105] [11 regression] profile streaming scales poorly to projects with many source files

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

--- Comment #18 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Martin Liska <marxin@gcc.gnu.org>:

https://gcc.gnu.org/g:6a8fc0c31a9ae759fe9bf59b5418abf2af938f91

commit r11-7513-g6a8fc0c31a9ae759fe9bf59b5418abf2af938f91
Author: Martin Liska <mliska@suse.cz>
Date:   Tue Feb 16 16:28:06 2021 +0100

    profiling: fix streaming of TOPN counters

    libgcc/ChangeLog:

            PR gcov-profile/99105
            * libgcov-driver.c (write_top_counters): Rename to ...
            (write_topn_counters): ... this.
            (write_one_data): Pre-allocate buffer for number of items
            in the corresponding linked lists.
            * libgcov.h (malloc_mmap): New function.
            (allocate_gcov_kvp): Use it.

    gcc/testsuite/ChangeLog:

            PR gcov-profile/99105
            * gcc.dg/tree-prof/indir-call-prof-malloc.c: Use profile
            correction as the wrapped malloc is called one more time
            from libgcov.
            * gcc.dg/tree-prof/pr97461.c: Likewise.

[Bug gcov-profile/99105] [11 regression] profile streaming scales poorly to projects with many source files

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99105

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #19 from Martin Liška <marxin at gcc dot gnu.org> ---
Fixed.