Re: [RFC] zstd as a compression algorithm for LTO

[Andrew Pinski <pinskia@gmail.com>]

On Wed, Jun 19, 2019 at 11:55 AM Richard Biener
<richard.guenther@gmail.com> wrote:
>
> On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law <law@redhat.com> wrote:
> >On 6/19/19 3:21 AM, Martin Liška wrote:
> >> Hi.
> >>
> >> I've written a patch draft that replaces zlib with the zstd
> >compression algorithm ([1])
> >> in LTO. I'm also sending statistics that are collected for couple of
> >quite big C++ source
> >> files. Observation I did:
> >>
> >> - LTO stream compression takes 3-4% of LGEN compile time
> >> - zstd in default compression level (3) generated slighly smaller LTO
> >elf files
> >> - zstd compression is 4-8x faster
> >> - decompression is quite negligible, but for a bigger project (godot)
> >I can
> >>   reduction from 1.37 to 0.53 seconds
> >> - ZSTD API is much simpler to use
> >>
> >> Suggestion based on the observation:
> >> - I would suggest to make zstd optional (--enable-zstd) and one would
> >>   use #include <zstd> + -lzstd
> >> - I like the default level as we want to mainly speed up LTO
> >compilation
> >> - we can provide an option to control algorithm
> >(-flto-compression-algorithm),
> >>   similarly to -flto-compression-level
> >> - we can discuss possible compression of LTO bytecode that is
> >distributed between WPA
> >>   stage and individual LTRANS phases.
> >Presumably the reason we're not being more aggressive about switching
> >is
> >the build/run time dependency on zstd?  I wonder if we could default to
> >zstd and fallback to zlib when zstd isn't available?
>
> Is zstd too big to include into the repository? But yes, we can properly encode the compression format in the LTO section header and use dlopen to 'find' the default to use.

At least allow it to be built as part of the normal build like GMP,
etc. are done.
And include it in downloading using contrib/download_prerequisites
like the libraries are done.

Thanks,
Andrew Pinski

>
> Richard.
>
> >jeff
>