[Bug bootstrap/55051] New: [4.8 Regression] profiledbootstrap failed

[Bug bootstrap/55051] New: [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

             Bug #: 55051
           Summary: [4.8 Regression] profiledbootstrap failed
    Classification: Unclassified
           Product: gcc
           Version: 4.8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: bootstrap
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: hjl.tools@gmail.com


On Linux/x86-64, revision 192538 gave

# make profiledbootstrap -j8
...
In file included from /export/gnu/import/git/gcc/gcc/gcov-dump.c:30:0:
/export/gnu/import/git/gcc/gcc/gcov-io.c:554:54: error: array subscript is
above array bounds [-Werror=array-bounds]
               cur_bitvector = histo_bitvector[bv_ix++];
                                                      ^
cc1plus: all warnings being treated as errors
make[4]: *** [gcov-dump.o] Error 1
make[4]: *** Waiting for unfinished jobs....
In file included from /export/gnu/import/git/gcc/gcc/gcov.c:47:0:
/export/gnu/import/git/gcc/gcc/gcov-io.c: In function ‘int
read_count_file(function_t*)’:
/export/gnu/import/git/gcc/gcc/gcov-io.c:554:54: error: array subscript is
above array bounds [-Werror=array-bounds]
               cur_bitvector = histo_bitvector[bv_ix++];
                                                      ^
cc1plus: all warnings being treated as errors
make[4]: *** [gcov.o] Error 1

Revision 192529 is OK.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.8.0

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org

--- Comment #1 from H.J. Lu <hjl.tools at gmail dot com> 2012-10-24 23:49:48 UTC ---
It is caused by revision 192538:

http://gcc.gnu.org/ml/gcc-cvs/2012-10/msg00661.html

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #2 from H.J. Lu <hjl.tools at gmail dot com> 2012-10-25 00:49:09 UTC ---
LRA generates

(gdb) call debug_rtx (insn)
(insn 6 20 7 2 (set (reg/f:SI 0 ax [orig:65 gomp_tls_data.ts.work_share ] [65])
        (mem/f/j/c:SI (plus:SI (reg:SI 2 cx [68])
                (reg:DI 0 ax [64])) [0 gomp_tls_data.ts.work_share+0 S4 A32]))
/tmp/x.i:23 65 {*movsi_internal}
     (expr_list:REG_DEAD (reg:SI 2 cx [68])
        (expr_list:REG_DEAD (reg:DI 0 ax [64])
            (nil))))


from

(insn 6 5 7 2 (set (reg/f:SI 65 [ gomp_tls_data.ts.work_share ])
        (mem/f/j/c:SI (plus:DI (plus:DI (zero_extend:DI (unspec:SI [
                                (const_int 0 [0])
                            ] UNSPEC_TP))
                    (reg:DI 64))
                (const_int 4 [0x4])) [0 gomp_tls_data.ts.work_share+0 S4 A32])) 
/tmp/x.i:23 65 {*movsi_internal}
     (expr_list:REG_DEAD (reg:DI 64)
        (nil)))

It fails to properly handle zero_extend.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[ubizjak at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #3 from Uros Bizjak <ubizjak at gmail dot com> 2012-10-25 06:51:45 UTC ---
(In reply to comment #2)
> LRA generates

This comment belongs to PR55049 (copied there).

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #4 from H.J. Lu <hjl.tools at gmail dot com> 2012-11-14 15:06:44 UTC ---
With revision 193500, we got

libdecnumber -I../../src-trunk/gcc/../libdecnumber/bid -I../libdecnumber
-I../../src-trunk/gcc/../libbacktrace    ../../src-trunk/gcc/gimple-iterator.c
-o gimple-iterator.o
../../src-trunk/gcc/fold-const.c:16732:1: internal compiler error: in
edge_badness, at ipa-inline.c:921
 }
 ^
0x11ddca7 edge_badness
        ../../src-trunk/gcc/ipa-inline.c:921
0x11ddf41 update_edge_key
        ../../src-trunk/gcc/ipa-inline.c:987
0x11df257 inline_small_functions
        ../../src-trunk/gcc/ipa-inline.c:1476
0x11dff6a ipa_inline
        ../../src-trunk/gcc/ipa-inline.c:1794
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[6]: *** [fold-const.o] Error 1

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #5 from H.J. Lu <hjl.tools at gmail dot com> 2012-11-14 15:08:53 UTC ---
There are

      badness = (relative_time_benefit (callee_info, edge, edge_time)
                 * (INT_MIN / 16 / RELATIVE_TIME_BENEFIT_RANGE));
      badness /= (growth * MAX (1, callee_info->growth));
      gcc_checking_assert (badness <=0 && badness >= INT_MIN / 16);

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[markus at trippelsdorf dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

Markus Trippelsdorf <markus at trippelsdorf dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |markus at trippelsdorf dot
                   |                            |de

--- Comment #6 from Markus Trippelsdorf <markus at trippelsdorf dot de> 2012-11-14 15:13:08 UTC ---
(In reply to comment #5)
> There are
> 
>       badness = (relative_time_benefit (callee_info, edge, edge_time)
>                  * (INT_MIN / 16 / RELATIVE_TIME_BENEFIT_RANGE));
>       badness /= (growth * MAX (1, callee_info->growth));
>       gcc_checking_assert (badness <=0 && badness >= INT_MIN / 16);

Yes, badness is 1 in this case...
I'm reducing the issue right now.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #7 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-14 15:35:26 UTC ---
> --- Comment #6 from Markus Trippelsdorf <markus at trippelsdorf dot de> 2012-11-14 15:13:08 UTC ---
> (In reply to comment #5)
> > There are
> > 
> >       badness = (relative_time_benefit (callee_info, edge, edge_time)
> >                  * (INT_MIN / 16 / RELATIVE_TIME_BENEFIT_RANGE));
> >       badness /= (growth * MAX (1, callee_info->growth));
> >       gcc_checking_assert (badness <=0 && badness >= INT_MIN / 16);
> 
> Yes, badness is 1 in this case...
> I'm reducing the issue right now.

Thanks, I was just about to start looking into FDO inlining re-tunning.

relative_time_benefit should be in range 0...INT_MAX / 64 so it should end up
in multipling -4 times that should be safe.

I am going to try to reproduce it now.
/bin/bash: ./:q: Permission denied

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[markus at trippelsdorf dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #8 from Markus Trippelsdorf <markus at trippelsdorf dot de> 2012-11-14 20:48:21 UTC ---
Created attachment 28688
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=28688
testcase

The testcase is only reduced to 97K, but it gets reduced very slowly
and I'm giving up.

 $ g++ -w -c -O2 -fprofile-generate test.ii
...
badness 0
badness 0
badness 1
test.ii:784:42: internal compiler error: in edge_badness, at ipa-inline.c:921

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #9 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-14 23:03:27 UTC ---
Author: hubicka
Date: Wed Nov 14 23:03:22 2012
New Revision: 193512

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=193512
Log:

    PR bootstrap/55051
    * ipa-inline.c (edge_badness): Improve dumping; fix overflow.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/ipa-inline.c

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

Jan Hubicka <hubicka at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tejohnson at google dot com

--- Comment #10 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 00:07:12 UTC ---
With make -j24 I also get tons of Invocation mismatch messages Theresa
introduced。 Theresa, can you look into it?

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #11 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 00:21:36 UTC ---
Sure, I will see if I can reproduce it.
Teresa


On Wed, Nov 14, 2012 at 4:07 PM, hubicka at gcc dot gnu.org <
gcc-bugzilla@gcc.gnu.org> wrote:

>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> Jan Hubicka <hubicka at gcc dot gnu.org> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>                  CC|                            |tejohnson at google dot
> com
>
> --- Comment #10 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15
> 00:07:12 UTC ---
> With make -j24 I also get tons of Invocation mismatch messages Theresa
> introduced。 Theresa, can you look into it?
>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #12 from H.J. Lu <hjl.tools at gmail dot com> 2012-11-15 00:30:51 UTC ---
Revision 193513 gave:

../../src-trunk/gcc/common/common-targhooks.c: In function
'default_target_handle_option(gcc_options*, gcc_options*, cl_decoded_option
const*, unsigned int)':
../../src-trunk/gcc/common/common-targhooks.c:85:4: note: file
/export/gnu/import/git/gcc-test-profile/bld/gcc/common/common-targhooks.gcda
not found, execution counts estimated
   };

In file included from ../../src-trunk/gcc/gcov.c:47:0:
../../src-trunk/gcc/gcov.c: In function 'read_count_file(function_info*)':
../../src-trunk/gcc/gcov-io.c:555:54: error: array subscript is above array
bounds [-Werror=array-bounds]
               cur_bitvector = histo_bitvector[bv_ix++];
                                                      ^
In file included from ../../src-trunk/gcc/gcov-dump.c:30:0:
../../src-trunk/gcc/gcov-io.c:555:54: error: array subscript is above array
bounds [-Werror=array-bounds]
               cur_bitvector = histo_bitvector[bv_ix++];
                                                      ^
cc1plus: all warnings being treated as errors
make[6]: *** [gcov-dump.o] Error 1
make[6]: *** Waiting for unfinished jobs....
cc1plus: all warnings being treated as errors
make[6]: *** [gcov.o] Error 1
rm gcj-dbtool.pod jcf-dump.pod jv-convert.pod grmic.pod gc

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #13 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 01:03:09 UTC ---
OK, the false positive is on quite sloppy code in gcov-io.c. I attached
testcase to PR55079 and will fix the gcc_assert specifying the loop bounds to
not allow out-of-bound read.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #14 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 01:07:04 UTC ---
Author: hubicka
Date: Thu Nov 15 01:07:01 2012
New Revision: 193522

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=193522
Log:
    PR bootstrap/55051
    * gcov-io.c (gcov_read_summary): Fix array bound check.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/gcov-io.c

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #15 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 01:10:29 UTC ---
Note that profiledbootstrap still dies for me on
config.status: creating tests/rand/Makefile
../../libiberty/cp-demangle.c: In function 'd_print_cast.isra.8':
../../libiberty/cp-demangle.c:5642:1: error: the control flow of function
'd_print_cast.isra.8' does not match its profile data (counter 'arcs')
[-Werror=coverage-mismatch]
 }
 ^
../../libiberty/cp-demangle.c:5642:1: note: use -Wno-error=coverage-mismatch to
tolerate the mismatch but performance may drop if the function is hot
../../libiberty/cp-demangle.c: In function 'd_print_function_type.isra.7':
../../libiberty/cp-demangle.c:5642:1: error: the control flow of function
'd_print_function_type.isra.7' does not match its profile data (counter 'arcs')
[-Werror=coverage-mismatch]
../../libiberty/cp-demangle.c:5642:1: note: use -Wno-error=coverage-mismatch to
tolerate the mismatch but performance may drop if the function is hot
../../libiberty/cp-demangle.c: In function 'd_print_array_type.isra.6':
../../libiberty/cp-demangle.c:5642:1: error: the control flow of function
'd_print_array_type.isra.6' does not match its profile data (counter 'arcs')
[-Werror=coverage-mismatch]
../../libiberty/cp-demangle.c:5642:1: note: use -Wno-error=coverage-mismatch to
tolerate the mismatch but performance may drop if the function is hot
../../libiberty/cp-demangle.c: In function 'd_lookup_template_argument.isra.5':
../../libiberty/cp-demangle.c:5642:1: error: the control flow of function
'd_lookup_template_argument.isra.5' does not match its profile data (counter
'arcs') [-Werror=coverage-mismatch]
../../libiberty/cp-demangle.c:5642:1: note: use -Wno-error=coverage-mismatch to
tolerate the mismatch but performance may drop if the function is hot
../../libiberty/cp-demangle.c: In function 'd_number.isra.0':


I am unsure if this is bug in profile code or different configuration of
libberty...

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #16 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 01:17:43 UTC ---
Theresa: I am using gcc10 from compilation farm, but I think it is fairly
universal problem.
Also I think that gcc_assert should not be assert, but an user readable error
about gcov file corruption.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #17 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 01:28:47 UTC ---
On Wed, Nov 14, 2012 at 5:17 PM, hubicka at gcc dot gnu.org <
gcc-bugzilla@gcc.gnu.org> wrote:

>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> --- Comment #16 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15
> 01:17:43 UTC ---
> Theresa: I am using gcc10 from compilation farm, but I think it is fairly
> universal problem.
>

Ok, I am doing a profiledbootstrap  -j24 with trunk on linux x86_64 to see
if I can reproduce it.


> Also I think that gcc_assert should not be assert, but an user readable
> error
> about gcov file corruption.
>

Are we talking about the coverage mismatch error, i.e.:

 ../../libiberty/cp-demangle.c: In function 'd_print_cast.isra.8':
../../libiberty/cp-demangle.c:5642:1: error: the control flow of function
'd_print_cast.isra.8' does not match its profile data (counter 'arcs')
[-Werror=coverage-mismatch]
 }
 ^
../../libiberty/cp-demangle.c:5642:1: note: use
-Wno-error=coverage-mismatch to
tolerate the mismatch but performance may drop if the function is hot

I don't think this was introduced by my changes. Or are we talking about a
different error?

Thanks,
Teresa

>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #18 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 01:33:43 UTC ---
On Wed, Nov 14, 2012 at 5:28 PM, Teresa Johnson <tejohnson@google.com> wrote:
>
>
>
>
> On Wed, Nov 14, 2012 at 5:17 PM, hubicka at gcc dot gnu.org <gcc-bugzilla@gcc.gnu.org> wrote:
>>
>>
>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>>
>> --- Comment #16 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 01:17:43 UTC ---
>> Theresa: I am using gcc10 from compilation farm, but I think it is fairly
>> universal problem.
>
>
> Ok, I am doing a profiledbootstrap  -j24 with trunk on linux x86_64 to see if I can reproduce it.
>
>>
>> Also I think that gcc_assert should not be assert, but an user readable error
>> about gcov file corruption.
>
>
> Are we talking about the coverage mismatch error, i.e.:
>
>  ../../libiberty/cp-demangle.c: In function 'd_print_cast.isra.8':
> ../../libiberty/cp-demangle.c:5642:1: error: the control flow of function
> 'd_print_cast.isra.8' does not match its profile data (counter 'arcs')
> [-Werror=coverage-mismatch]
>  }
>  ^
> ../../libiberty/cp-demangle.c:5642:1: note: use -Wno-error=coverage-mismatch to
> tolerate the mismatch but performance may drop if the function is hot
>
> I don't think this was introduced by my changes. Or are we talking about a different error?


Oh got it - it is this one, right?:

profiling:/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/files.gcda:Invocation
mismatch - some data files may have been removed

I think this one was there before, but I had to modify it after my
histogram change. I will take a look.

Teresa

>
>
> Thanks,
> Teresa
>>
>>
>> --
>> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
>> ------- You are receiving this mail because: -------
>> You are on the CC list for the bug.
>
>
>
>
> --
> Teresa Johnson | Software Engineer |  tejohnson@google.com |  408-460-2413
>



--
Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413

Re: [Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[Jan Hubicka]

> Oh got it - it is this one, right?:
> 
> profiling:/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/files.gcda:Invocation
> mismatch - some data files may have been removed

Yes, it is this one.
> 
> I think this one was there before, but I had to modify it after my
> histogram change. I will take a look.

Also could you please make a patch to make maybe_hot_count_p to use
hitogram driven cutoff? Otherwise the histograms would be completely
unused for 4.8 and it would be stupid to carry all the extra data
for no use.  I would like this to be done soon, since I plan to base
some of inliner re-tunning to be based on this.

The mismatch is independent problem, yes.

Honza

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #19 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15 01:42:55 UTC ---
> Oh got it - it is this one, right?:
> 
> profiling:/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/files.gcda:Invocation
> mismatch - some data files may have been removed

Yes, it is this one.
> 
> I think this one was there before, but I had to modify it after my
> histogram change. I will take a look.

Also could you please make a patch to make maybe_hot_count_p to use
hitogram driven cutoff? Otherwise the histograms would be completely
unused for 4.8 and it would be stupid to carry all the extra data
for no use.  I would like this to be done soon, since I plan to base
some of inliner re-tunning to be based on this.

The mismatch is independent problem, yes.

Honza

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #20 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 01:52:45 UTC ---
On Wed, Nov 14, 2012 at 5:42 PM, hubicka at ucw dot cz
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> --- Comment #19 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15 01:42:55 UTC ---
>> Oh got it - it is this one, right?:
>>
>> profiling:/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/files.gcda:Invocation
>> mismatch - some data files may have been removed
>
> Yes, it is this one.
>>
>> I think this one was there before, but I had to modify it after my
>> histogram change. I will take a look.
>
> Also could you please make a patch to make maybe_hot_count_p to use
> hitogram driven cutoff? Otherwise the histograms would be completely
> unused for 4.8 and it would be stupid to carry all the extra data
> for no use.  I would like this to be done soon, since I plan to base
> some of inliner re-tunning to be based on this.

Ok, I can do that. I had tried that but didn't see any gain yet (need
to take a look at my results again). I have been playing with teasing
apart the various uses of this cutoff (inlining vs instruction
selection vs etc) too, but can do that in a later release once the
appropriate individual cutoffs are tuned. I also have a loop unroller
patch on the google branch that uses this that needs to be ported to
trunk. It was in the original working set patch I sent for review,
that ended up being split out and revised heavily. Shall I resubmit
this part for 4.8 or is it too late?

Thanks,
Teresa

>
> The mismatch is independent problem, yes.
>
> Honza
>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.



--
Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #21 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15 02:01:01 UTC ---
> Ok, I can do that. I had tried that but didn't see any gain yet (need
> to take a look at my results again). I have been playing with teasing
> apart the various uses of this cutoff (inlining vs instruction
> selection vs etc) too, but can do that in a later release once the
> appropriate individual cutoffs are tuned. I also have a loop unroller
> patch on the google branch that uses this that needs to be ported to
> trunk. It was in the original working set patch I sent for review,
> that ended up being split out and revised heavily. Shall I resubmit
> this part for 4.8 or is it too late?

I can't review the unroller change, so you should ask one of the maintainers
of the area.  For the hot/cold change, I would really like to see it in.
(and yes, I also plan to use it for more aggressive inlining in known to
be hot areas).

Honza

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #22 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 02:46:20 UTC ---
Ok, will see if I can submit that one tomorrow then, after double
checking the performance.

Teresa

On Wed, Nov 14, 2012 at 6:01 PM, hubicka at ucw dot cz
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> --- Comment #21 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15 02:01:01 UTC ---
>> Ok, I can do that. I had tried that but didn't see any gain yet (need
>> to take a look at my results again). I have been playing with teasing
>> apart the various uses of this cutoff (inlining vs instruction
>> selection vs etc) too, but can do that in a later release once the
>> appropriate individual cutoffs are tuned. I also have a loop unroller
>> patch on the google branch that uses this that needs to be ported to
>> trunk. It was in the original working set patch I sent for review,
>> that ended up being split out and revised heavily. Shall I resubmit
>> this part for 4.8 or is it too late?
>
> I can't review the unroller change, so you should ask one of the maintainers
> of the area.  For the hot/cold change, I would really like to see it in.
> (and yes, I also plan to use it for more aggressive inlining in known to
> be hot areas).
>
> Honza
>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #23 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 06:44:00 UTC ---
On Wed, Nov 14, 2012 at 5:17 PM, hubicka at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> --- Comment #16 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-15 01:17:43 UTC ---
> Theresa: I am using gcc10 from compilation farm, but I think it is fairly
> universal problem.
> Also I think that gcc_assert should not be assert, but an user readable error
> about gcov file corruption.

I took a look at this some more this evening, printing out the values
being compared. Note that the modification I made to this code was to
ignore the histogram when comparing the gcov_ctr_summary, since the
order of update can affect the merged histogram values as they are
prorated. So essentially we are doing the same comparison here that we
did before the histogram was added.

I found that the gcov_ctr_summary fields being compared are identical
except for the sum_all, which is different, leading to the error
message. The number of counters, run_max and sum_max are the same. I'm
not sure how that can happen given that locking is on and the sum_all
should be the same regardless of the merging order.

Note though that this is not an assert. It just emits a message to
stderr. Do you think a better error message is appropriate? I'm not
sure the "some data files may have been removed" is an accurate
description of the issue. Perhaps something like "Profile data file
mismatch may indicate corrupt profile data"?

Teresa

>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.



--
Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413

Re: [Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[Jan Hubicka]

> Note though that this is not an assert. It just emits a message to
> stderr. Do you think a better error message is appropriate? I'm not
> sure the "some data files may have been removed" is an accurate
> description of the issue. Perhaps something like "Profile data file
> mismatch may indicate corrupt profile data"?

Well, we should figure out why sum_all starts to diverge.  If we had
problems mixing cc1 and cc1plus executions, we should get mismatches in
number of counters.  What happens after the miscompare?

Honza

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #24 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15 10:56:53 UTC ---
> Note though that this is not an assert. It just emits a message to
> stderr. Do you think a better error message is appropriate? I'm not
> sure the "some data files may have been removed" is an accurate
> description of the issue. Perhaps something like "Profile data file
> mismatch may indicate corrupt profile data"?

Well, we should figure out why sum_all starts to diverge.  If we had
problems mixing cc1 and cc1plus executions, we should get mismatches in
number of counters.  What happens after the miscompare?

Honza

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #25 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 14:34:10 UTC ---
On Thu, Nov 15, 2012 at 2:56 AM, hubicka at ucw dot cz <
gcc-bugzilla@gcc.gnu.org> wrote:

>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> --- Comment #24 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15
> 10:56:53 UTC ---
> > Note though that this is not an assert. It just emits a message to
> > stderr. Do you think a better error message is appropriate? I'm not
> > sure the "some data files may have been removed" is an accurate
> > description of the issue. Perhaps something like "Profile data file
> > mismatch may indicate corrupt profile data"?
>
> Well, we should figure out why sum_all starts to diverge.  If we had
> problems mixing cc1 and cc1plus executions, we should get mismatches in
> number of counters.


Right, it doesn't appear to be different executables since the number of
counters is identical. I'll instrument it and see if I can figure out why
they diverge.


> What happens after the miscompare?
>

A flag is set so that the error is emitted at most once per merge, and then
we continue on with the merge and ignore it. Basically what it is doing is
saving the first merged summary (for the first object file's gcda we merge
into), and then for each additional object file that gets its counters
merged the resulting program summary is compared against the saved program
summary. But only if the number of runs is the same as the saved summary.
This could happen if the gcda files are walked in a different order during
updates (i.e. the gcov_list is in a different order for different processes
of the same executable), but I am not sure if that can happen.

Teresa


> Honza
>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.
>

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #26 from Teresa Johnson <tejohnson at google dot com> 2012-11-15 22:42:12 UTC ---
On Thu, Nov 15, 2012 at 6:33 AM, Teresa Johnson <tejohnson@google.com> wrote:
>
>
>
> On Thu, Nov 15, 2012 at 2:56 AM, hubicka at ucw dot cz
> <gcc-bugzilla@gcc.gnu.org> wrote:
>>
>>
>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>>
>> --- Comment #24 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-15
>> 10:56:53 UTC ---
>> > Note though that this is not an assert. It just emits a message to
>> > stderr. Do you think a better error message is appropriate? I'm not
>> > sure the "some data files may have been removed" is an accurate
>> > description of the issue. Perhaps something like "Profile data file
>> > mismatch may indicate corrupt profile data"?
>>
>> Well, we should figure out why sum_all starts to diverge.  If we had
>> problems mixing cc1 and cc1plus executions, we should get mismatches in
>> number of counters.
>
>
> Right, it doesn't appear to be different executables since the number of
> counters is identical. I'll instrument it and see if I can figure out why
> they diverge.
>
>>
>> What happens after the miscompare?
>
>
> A flag is set so that the error is emitted at most once per merge, and then
> we continue on with the merge and ignore it. Basically what it is doing is
> saving the first merged summary (for the first object file's gcda we merge
> into), and then for each additional object file that gets its counters
> merged the resulting program summary is compared against the saved program
> summary. But only if the number of runs is the same as the saved summary.
> This could happen if the gcda files are walked in a different order during
> updates (i.e. the gcov_list is in a different order for different processes
> of the same executable), but I am not sure if that can happen.

It appears that this is what is happening, and I think it makes sense
that it can.

We're essentially doing this:

  /* Now merge each file.  */
  for (gi_ptr = gcov_list; gi_ptr; gi_ptr = gi_ptr->next)
    {
        // Open existing gcda file for gi_ptr
        // Find program summary corresponding to this executable -> save in prg
        // Merge execution counts for each function
        // Merge program summary
        //      - If this is the first merged file for this execution,
save merged summary in all_prg
        //      - Otherwise if #runs the same in prg and all_prg,
print error message if prg != all_prg.
        // Write merged gcda
    }

I found that in a couple cases, we printed the error message for
libcpp/directives.gcda, where the saved all_prg summary was from
gcc/gcc.gcda.

I then instrumented the code so that each time we merge into one of
these 2 gcda files I emit the pids, the number of runs, the number of
counters and the merged sum_all. Comparing the results from all the
merges to these two gcda files I see that most of the time the merges
proceed in the same order, but there are a few cases where the order
is different, resulting in a different sum_all with the same number of
runs, and then things go back to normal and the sum_all matches again.
E.g., here is one place where things get out of order briefly,
resulting in one of the error messages being printed:

...
pid 28432 ppid 28429 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/gcc/gcc.gcda with runs 254 num
13193 sum_all 17058327
pid 28437 ppid 28365 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/gcc/gcc.gcda with runs 255 num
13193 sum_all 17064832
pid 28439 ppid 28367 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/gcc/gcc.gcda with runs 256 num
13193 sum_all 17071340
pid 28440 ppid 28436 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/gcc/gcc.gcda with runs 257 num
13193 sum_all 17177525
...

vs

...
pid 28432 ppid 28429 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/directives.gcda with runs
254 num 13193 sum_all 17058327
pid 28439 ppid 28367 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/directives.gcda with runs
255 num 13193 sum_all 17064835
pid 28437 ppid 28365 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/directives.gcda with runs
256 num 13193 sum_all 17071340
pid 28440 ppid 28436 Merging summary for
/home/tejohnson/extra/gcc_trunk_3_obj/libcpp/directives.gcda with runs
257 num 13193 sum_all 17177525
...

Notice the middle two pids are flipped, resulting in the sum_all being
different after run 255, and back to the same after run 256.

I believe this could happen if pids 28437 and 28439 finished
near-simultaneously, waited for the lock for gcc.gcda, and 28437 won
first, but then by some luck of timing they subsequently both
attempted to open directives.gcda at around the same time and 28439
happened to win the lock in the fcntl loop first.

I believe it is also possible for object files to be in different
orders in the gcov_list in different processes, since they are added
to the head of that list in __gcov_init, which is invoked when running
an object file's global constructors, according to the header comment.
And for C++ at least, the order of initialization across translation
units is undefined. That could also cause the sum_all to go
temporarily out of sync between different object file gcda files.

Overall, I think it makes sense to remove this check altogether. Would
you agree? Testing the patch to remove this right now.

Teresa

>
> Teresa
>
>>
>> Honza
>>
>> --
>> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
>> ------- You are receiving this mail because: -------
>> You are on the CC list for the bug.
>
>
>
>
> --
> Teresa Johnson | Software Engineer |  tejohnson@google.com |  408-460-2413
>



--
Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #27 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-16 17:42:26 UTC ---
  /* Now merge each file.  */
  for (gi_ptr = gcov_list; gi_ptr; gi_ptr = gi_ptr->next)
    {
        // Open existing gcda file for gi_ptr
        // Find program summary corresponding to this executable -> save in prg
        // Merge execution counts for each function
        // Merge program summary
        //      - If this is the first merged file for this execution,
save merged summary in all_prg
        //      - Otherwise if #runs the same in prg and all_prg,
print error message if prg != all_prg.
        // Write merged gcda
    }

Hmm, yes, it seems wrong.  We can not expect all gcda files to have same number
of runs.  We really need to process sum_all & friends locally for each file. 
Only I suppose we can check if number of runs of the prg happens to match the
last merged file then sum_all should match. That would be nice consistency
check. If the bootstrap works with this change, consider the patch preapproved.

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at google dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #28 from Teresa Johnson <tejohnson at google dot com> 2012-11-16 18:03:08 UTC ---
On Fri, Nov 16, 2012 at 9:42 AM, hubicka at gcc dot gnu.org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051
>
> --- Comment #27 from Jan Hubicka <hubicka at gcc dot gnu.org> 2012-11-16 17:42:26 UTC ---
>   /* Now merge each file.  */
>   for (gi_ptr = gcov_list; gi_ptr; gi_ptr = gi_ptr->next)
>     {
>         // Open existing gcda file for gi_ptr
>         // Find program summary corresponding to this executable -> save in prg
>         // Merge execution counts for each function
>         // Merge program summary
>         //      - If this is the first merged file for this execution,
> save merged summary in all_prg
>         //      - Otherwise if #runs the same in prg and all_prg,
> print error message if prg != all_prg.
>         // Write merged gcda
>     }
>
> Hmm, yes, it seems wrong.  We can not expect all gcda files to have same number
> of runs.  We really need to process sum_all & friends locally for each file.
> Only I suppose we can check if number of runs of the prg happens to match the
> last merged file then sum_all should match.

I'm confused - that is essentially what it is doing today (although
comparing against the first merged file instead of the last merged
file). It isn't expecting all the gcda files to have the same number
of runs, but when they do it expects them to have the same sum_all.
See the trace of updates I show below this pseudo code in the same
email that shows why that is not working.

> That would be nice consistency
> check. If the bootstrap works with this change, consider the patch preapproved.

I had sent a patch last night to simply rip out this check based on
the analysis I had done, but I assume you are not talking about that.

Thanks,
Teresa

>
> --
> Configure bugmail: http://gcc.gnu.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug.



--
Teresa Johnson | Software Engineer | tejohnson@google.com | 408-460-2413

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #29 from Jan Hubicka <hubicka at ucw dot cz> 2012-11-16 18:57:41 UTC ---
> 
> I'm confused - that is essentially what it is doing today (although
> comparing against the first merged file instead of the last merged
> file). It isn't expecting all the gcda files to have the same number
> of runs, but when they do it expects them to have the same sum_all.
> See the trace of updates I show below this pseudo code in the same
> email that shows why that is not working.

Hmm, after some tought I guess what can happen is that number of run matches
but these are different run, because more than 2 streamings may be running...

Honza

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[tejohnson at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

--- Comment #30 from tejohnson at gcc dot gnu.org 2012-11-19 05:21:06 UTC ---
Author: tejohnson
Date: Mon Nov 19 05:20:59 2012
New Revision: 193612

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=193612
Log:
This patch addresses the bogus "Invocation mismatch" messages seen in parallel
profiledbootstrap builds of gcc. See PR bootstrap/55051 for a discussion of
why this is occurring and why this checking is inaccurate. Leave it in when
!GCOV_LOCKED, to warn about concurrent update issues requiring locking.

2012-11-18  Teresa Johnson  <tejohnson@google.com>

    PR bootstrap/55051
    * libgcov.c (gcov_exit): Remove merged program summary
    comparison unless !GCOV_LOCKED.

Modified:
    trunk/libgcc/ChangeLog
    trunk/libgcc/libgcov.c

[Bug bootstrap/55051] [4.8 Regression] profiledbootstrap failed

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55051

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |FIXED

--- Comment #31 from Richard Biener <rguenth at gcc dot gnu.org> 2012-12-07 12:04:08 UTC ---
Assuming fixed.  I don't see any non-LTO profiledbootstrap fails on
gcc-regression.