HELP: Questions on multiple PROGRAM_SUMMARY sections in a profiling data file

HELP: Questions on multiple PROGRAM_SUMMARY sections in a profiling data file

[Qing Zhao]

Hi, Jan,

I am studying one profiling feedback ICE bug with GCC8 recently. 
It’s an assertion failure inside the routine “compute_working_sets”of gcov-io.c:

gcov_nonruntime_assert (ws_ix == NUM_GCOV_WORKING_SETS);

After some debugging and study, I found that the corresponding .gcda file has two PROGRAM_SUMMARY sections:

foo.gcda:  a3000000: 202:PROGRAM_SUMMARY checksum=0x91f3e3ae 
foo.gcda:                counts=10831, runs=0, sum_all=478965, run_max=125615, sum_max=201126 
foo.gcda:                counter histogram: 
foo.gcda:                 0: num counts=10187, min counter=0, cum_counter=0 
… 
foo.gcda:                51: num counts=1, min counter=14524, cum_counter=14524 
foo.gcda:                63: num counts=1, min counter=125615, cum_counter=125615 
foo.gcda:  a3000000: 137:PROGRAM_SUMMARY checksum=0xcf9f0896 
foo.gcda:                counts=10502, runs=1, sum_all=48618, run_max=13999, sum_max=14046 
foo.gcda:                counter histogram: 
foo.gcda:                 0: num counts=9821, min counter=0, cum_counter=0 
… 
foo.gcda:                43: num counts=1, min counter=3830, cum_counter=3830 
foo.gcda:                50: num counts=1, min counter=13999, cum_counter=13999 

Looks like the 2nd PROGRAM_SUMMARY has some issue. If I manually change gcc/coverage.c 
to ignore the 2nd PROGRAM_SUMMARY section, the ICE disappears. 

I have several questions for the profiling feedback data file:

1. Under what situation, there will be multiple PROGRAM_SUMMARY sections for one module?
2. How to check whether one of the PROGRAM_SUMMARY has issue?

Thanks a lot for any help.

Qing

Re: HELP: Questions on multiple PROGRAM_SUMMARY sections in a profiling data file

[Jan Hubicka]

> Hi, Jan,
> 
> I am studying one profiling feedback ICE bug with GCC8 recently. 
> It’s an assertion failure inside the routine “compute_working_sets”of gcov-io.c:
> 
> gcov_nonruntime_assert (ws_ix == NUM_GCOV_WORKING_SETS);
> 
> After some debugging and study, I found that the corresponding .gcda file has two PROGRAM_SUMMARY sections:
> 
> foo.gcda:  a3000000: 202:PROGRAM_SUMMARY checksum=0x91f3e3ae 
> foo.gcda:                counts=10831, runs=0, sum_all=478965, run_max=125615, sum_max=201126 
> foo.gcda:                counter histogram: 
> foo.gcda:                 0: num counts=10187, min counter=0, cum_counter=0 
> … 
> foo.gcda:                51: num counts=1, min counter=14524, cum_counter=14524 
> foo.gcda:                63: num counts=1, min counter=125615, cum_counter=125615 
> foo.gcda:  a3000000: 137:PROGRAM_SUMMARY checksum=0xcf9f0896 
> foo.gcda:                counts=10502, runs=1, sum_all=48618, run_max=13999, sum_max=14046 
> foo.gcda:                counter histogram: 
> foo.gcda:                 0: num counts=9821, min counter=0, cum_counter=0 
> … 
> foo.gcda:                43: num counts=1, min counter=3830, cum_counter=3830 
> foo.gcda:                50: num counts=1, min counter=13999, cum_counter=13999 
> 
> Looks like the 2nd PROGRAM_SUMMARY has some issue. If I manually change gcc/coverage.c 
> to ignore the 2nd PROGRAM_SUMMARY section, the ICE disappears. 
> 
> I have several questions for the profiling feedback data file:
> 
> 1. Under what situation, there will be multiple PROGRAM_SUMMARY sections for one module?

This is itended to resolve situations where one object file is linked
into multiple final binaries (just like libbackend is linked into
multiple FEs).  The overall binary checksum is known to
the runtime and it creates entry for each checksum.
> 2. How to check whether one of the PROGRAM_SUMMARY has issue?

The histograms was added by google to get better threshold for hot/cold
partitioning. They never worked too well, because it is hard to merge
histograms from multipe runs meaningfully, so we dropped them some time
ago.  With LTO it is easier to produce the histogram after reading all
profiles together and without LTO we use easier heuristics that only
computes constant fraction of maximum number of iterations.

What confuses me is the first summary having runs=0.  I wonder how with
zero runs it even has an entry?

What is the assert that fials in gcov-io.c?

Honza

> 
> Thanks a lot for any help.
> 
> Qing

Re: HELP: Questions on multiple PROGRAM_SUMMARY sections in a profiling data file

[Qing Zhao]

Honza, 

Thanks a lot for your information.

> On Mar 8, 2023, at 8:19 AM, Jan Hubicka <hubicka@ucw.cz> wrote:
> 
>> Hi, Jan,
>> 
>> I am studying one profiling feedback ICE bug with GCC8 recently. 
>> It’s an assertion failure inside the routine “compute_working_sets”of gcov-io.c:
>> 
>> gcov_nonruntime_assert (ws_ix == NUM_GCOV_WORKING_SETS);
>> 
>> After some debugging and study, I found that the corresponding .gcda file has two PROGRAM_SUMMARY sections:
>> 
>> foo.gcda:  a3000000: 202:PROGRAM_SUMMARY checksum=0x91f3e3ae 
>> foo.gcda:                counts=10831, runs=0, sum_all=478965, run_max=125615, sum_max=201126 
>> foo.gcda:                counter histogram: 
>> foo.gcda:                 0: num counts=10187, min counter=0, cum_counter=0 
>> … 
>> foo.gcda:                51: num counts=1, min counter=14524, cum_counter=14524 
>> foo.gcda:                63: num counts=1, min counter=125615, cum_counter=125615 
>> foo.gcda:  a3000000: 137:PROGRAM_SUMMARY checksum=0xcf9f0896 
>> foo.gcda:                counts=10502, runs=1, sum_all=48618, run_max=13999, sum_max=14046 
>> foo.gcda:                counter histogram: 
>> foo.gcda:                 0: num counts=9821, min counter=0, cum_counter=0 
>> … 
>> foo.gcda:                43: num counts=1, min counter=3830, cum_counter=3830 
>> foo.gcda:                50: num counts=1, min counter=13999, cum_counter=13999 
>> 
>> Looks like the 2nd PROGRAM_SUMMARY has some issue. If I manually change gcc/coverage.c 
>> to ignore the 2nd PROGRAM_SUMMARY section, the ICE disappears. 
>> 
>> I have several questions for the profiling feedback data file:
>> 
>> 1. Under what situation, there will be multiple PROGRAM_SUMMARY sections for one module?
> 
> This is itended to resolve situations where one object file is linked
> into multiple final binaries (just like libbackend is linked into
> multiple FEs).

Okay, I see. (That was my guess too. -:)
>  The overall binary checksum is known to
> the runtime and it creates entry for each checksum.
Okay. 
>> 2. How to check whether one of the PROGRAM_SUMMARY has issue?
> 
> The histograms was added by google to get better threshold for hot/cold
> partitioning. They never worked too well, because it is hard to merge
> histograms from multipe runs meaningfully, so we dropped them some time
> ago.

Just checked, the PROGRAM_SUMMARY has been removed completely since GCC9 with the following commit:

commit 512cc0151207de4c7ff3a84f040f730fe0d52458
Author: Martin Liska <mliska@suse.cz>
Date:   Fri Sep 21 10:41:17 2018 +0200

    Remove arc profile histogram in non-LTO mode.
    
    2018-09-21  Martin Liska  <mliska@suse.cz>

>  With LTO it is easier to produce the histogram after reading all
> profiles together and without LTO we use easier heuristics that only
> computes constant fraction of maximum number of iterations.
Okay. 
> 
> What confuses me is the first summary having runs=0.  I wonder how with
> zero runs it even has an entry?
I was surprised by this too. Don’t know the reason yet.

The application that has this problem is huge. I don’t have the complete source codes and make files. 

The thing that is even more strange is, there are multiple files have the same ICE during profiling use phase, and as I checked today, all the corresponding .gcda files have
the exactly same PROGRAM_SUMMARY sections, i.e.

foo.gcda:

foo.gcda:  a3000000: 202:PROGRAM_SUMMARY checksum=0x91f3e3ae 
foo.gcda:                counts=10831, runs=0, sum_all=478965, run_max=125615, sum_max=201126 
foo.gcda:                counter histogram: 
foo.gcda:                 0: num counts=10187, min counter=0, cum_counter=0 
… 
foo.gcda:                51: num counts=1, min counter=14524, cum_counter=14524 
foo.gcda:                63: num counts=1, min counter=125615, cum_counter=125615 
foo.gcda:  a3000000: 137:PROGRAM_SUMMARY checksum=0xcf9f0896 
foo.gcda:                counts=10502, runs=1, sum_all=48618, run_max=13999, sum_max=14046 
foo.gcda:                counter histogram: 
foo.gcda:                 0: num counts=9821, min counter=0, cum_counter=0 
… 
foo.gcda:                43: num counts=1, min counter=3830, cum_counter=3830 
foo.gcda:                50: num counts=1, min counter=13999, cum_counter=13999 

foo_1.gcda:
foo_1.gcda:  a3000000: 202:PROGRAM_SUMMARY checksum=0x91f3e3ae 
foo_1.gcda:                counts=10831, runs=0, sum_all=478965, run_max=125615, sum_max=201126 
foo_1.gcda:                counter histogram: 
foo_1.gcda:                 0: num counts=10187, min counter=0, cum_counter=0 
… 
foo_1.gcda:                51: num counts=1, min counter=14524, cum_counter=14524 
foo_1.gcda:                63: num counts=1, min counter=125615, cum_counter=125615 
foo_1.gcda:  a3000000: 137:PROGRAM_SUMMARY checksum=0xcf9f0896 
foo_1.gcda:                counts=10502, runs=1, sum_all=48618, run_max=13999, sum_max=14046 
foo_1.gcda:                counter histogram: 
foo_1.gcda:                 0: num counts=9821, min counter=0, cum_counter=0 
… 
foo_1.gcda:                43: num counts=1, min counter=3830, cum_counter=3830 
foo_1.gcda:                50: num counts=1, min counter=13999, cum_counter=13999 

Exactly the same. Really don’t understand what’s going on.

> 
> What is the assert that fials in gcov-io.c?

during IPA pass: profile
foo.c: In function ‘foo_print’:
foo.c:509:1: internal compiler error: in compute_working_sets, at gcov-io.c:1066

(NOTE, I changed the names of the file and the function).

For GCC8, what should we do to work around this issue? Do you have any suggestions?

Thanks.

Qing

> 
> Honza
> 
>> 
>> Thanks a lot for any help.
>> 
>> Qing