Re: Backporting streaming and enum changes

[Christophe Lyon <christophe.lyon@linaro.org>]

Hi,

On Thu, 6 Aug 2020 at 16:39, Richard Biener <rguenther@suse.de> wrote:
>
> On Thu, 6 Aug 2020, Jan Hubicka wrote:
>
> > Hello,
> > as discussed some time ago, I would like to discuss possibility to
> > backport the straming and enum improvements.  The motivation is that
> > this brings quite noticeable improvements to builds of very large
> > projects where we currently have nonlinearity problem with anonymous
> > namespaces (which is solved by first set of patches) and also there is
> > quite noticeable overhead of streaming of enums that I noticed too late
> > for gcc 10.1. This is the second combine dpatch.
> >
> > There is also noticeable reduction of .o files (especially before
> > compression as hit to WPA->ltrans streaming) and some memory use
> > benefits.
> >
> > This is an optional thing to do, but I believe it may be helpful for
> > distro builds and those using LTO for large projects.
> >
> > For firefox the reduction in global stream (that is slowest part of WPA)
> > is from 25678391 tree bodies to 20821629, 11160520 SCC hash collisions
> > to 6002. 392382523 overal section size to 287891470 (both is
> > compressed).
> >
> > For Firefox streaming is under control, but other projects like Chromium
> > hits bigger issues. The reason is that Firefox has "unified build" that
> > #includes multiple cpp sources to one, so it consists of only about 8k
> > source files, while chromium is over 25k and it was tested on project
> > with over 250k sources. More smaller sources one gets, the more
> > noticeable bottleneck streaming become.
> >
> > The patches are not completely trivial, but they affect code that is
> > heavily executed during streaming and was in mainline for several
> > months, so I hope they are safe.
>
> So we've built the core of openSUSE (~3000 packages) on x86_64
> and i586 with these backported and sofar found no issues.
>
> I'm fine with backporting but I'll give Jakub the chance to
> object.
>
> Honza - please make sure to bump the LTO stream version minor
> together with the streaming change (I think the enum change
> doesn't require bumping).
>

Since this was backported as r10-8623-g0d96c3424bbb5e5f994b78c8f65d8704d215be54,
I've noticed ICEs on arm and aarch64:
    gcc.dg/pr34457-1.c (internal compiler error)
    gcc.dg/torture/pr92088-1.c   -O2 -flto -fno-use-linker-plugin
-flto-partition=none  (internal compiler error)
    gcc.dg/torture/pr92088-1.c   -O2 -flto -fuse-linker-plugin
-fno-fat-lto-objects  (internal compiler error)

I can see:
Excess errors:
during IPA pass: cp
lto1: internal compiler error: in operator[], at vec.h:878
0xa0a5d7 vec<lto_encoder_entry, va_heap, vl_embed>::operator[](unsigned int)
        /gcc/vec.h:878
0xa0a5d7 vec<lto_encoder_entry, va_heap, vl_ptr>::operator[](unsigned int)
        /gcc/vec.h:1444
0xa194d3 vec<lto_encoder_entry, va_heap, vl_ptr>::operator[](unsigned int)
        /gcc/tree.h:3408
0xa194d3 lto_symtab_encoder_deref
        /gcc/lto-streamer.h:1173
0xa194d3 ipa_prop_read_section
        /gcc/ipa-prop.c:5060
0xa194d3 ipa_prop_read_jump_functions()
        /gcc/ipa-prop.c:5089
0xb6ba71 ipa_read_summaries_1
        /gcc/passes.c:2837
0x6bc4b5 read_cgraph_and_symbols(unsigned int, char const**)
        /gcc/lto/lto-common.c:2921
0x69deb2 lto_main()
        /gcc/lto/lto.c:625

The tests pass on trunk.

Christophe
> Thanks,
> Richard.
>
> > Honza
> >
>
> --
> Richard Biener <rguenther@suse.de>
> SUSE Software Solutions Germany GmbH, Maxfeldstrasse 5, 90409 Nuernberg,
> Germany; GF: Felix Imendörffer; HRB 36809 (AG Nuernberg)