[RFC] zstd as a compression algorithm for LTO

[RFC] zstd as a compression algorithm for LTO

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

Hi.

I've written a patch draft that replaces zlib with the zstd compression algorithm ([1])
in LTO. I'm also sending statistics that are collected for couple of quite big C++ source
files. Observation I did:

- LTO stream compression takes 3-4% of LGEN compile time
- zstd in default compression level (3) generated slighly smaller LTO elf files
- zstd compression is 4-8x faster
- decompression is quite negligible, but for a bigger project (godot) I can
  reduction from 1.37 to 0.53 seconds
- ZSTD API is much simpler to use

Suggestion based on the observation:
- I would suggest to make zstd optional (--enable-zstd) and one would
  use #include <zstd> + -lzstd
- I like the default level as we want to mainly speed up LTO compilation
- we can provide an option to control algorithm (-flto-compression-algorithm),
  similarly to -flto-compression-level
- we can discuss possible compression of LTO bytecode that is distributed between WPA
  stage and individual LTRANS phases.

Thoughts?
Thanks,
Martin

[1] https://github.com/facebook/zstd

[-- Attachment #2: 0001-Replace-zlib-with-zstd.patch --]
[-- Type: text/x-patch, Size: 9130 bytes --]

From 4939e90b2a8051128b7b2b0214a5fad5183f3bca Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Wed, 19 Jun 2019 09:40:35 +0200
Subject: [PATCH] Replace zlib with zstd.

---
 gcc/Makefile.in    |   2 +-
 gcc/common.opt     |   2 +-
 gcc/lto-compress.c | 161 ++++++++-------------------------------------
 gcc/timevar.def    |   4 +-
 4 files changed, 33 insertions(+), 136 deletions(-)

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index d9e0885b96b..8aedcccb717 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -373,7 +373,7 @@ OUTPUT_OPTION = -o $@
 # This is where we get zlib from.  zlibdir is -L../zlib and zlibinc is
 # -I../zlib, unless we were configured with --with-system-zlib, in which
 # case both are empty.
-ZLIB = @zlibdir@ -lz
+ZLIB = @zlibdir@ -lzstd -lz
 ZLIBINC = @zlibinc@
 
 # How to find GMP
diff --git a/gcc/common.opt b/gcc/common.opt
index a1544d06824..f15e21914f3 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1888,7 +1888,7 @@ Specify the algorithm to partition symbols and vars at linktime.
 
 ; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h.
 flto-compression-level=
-Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 9)
+Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 1111)
 -flto-compression-level=<number>	Use zlib compression level <number> for IL.
 
 flto-odr-type-merging
diff --git a/gcc/lto-compress.c b/gcc/lto-compress.c
index 3287178f257..b24f30f956e 100644
--- a/gcc/lto-compress.c
+++ b/gcc/lto-compress.c
@@ -27,13 +27,9 @@ along with GCC; see the file COPYING3.  If not see
 #include "gimple.h"
 #include "cgraph.h"
 #include "lto-streamer.h"
-/* zlib.h includes other system headers.  Those headers may test feature
-   test macros.  config.h may define feature test macros.  For this reason,
-   zlib.h needs to be included after, rather than before, config.h and
-   system.h.  */
-#include <zlib.h>
 #include "lto-compress.h"
 #include "timevar.h"
+#include <zstd.h>
 
 /* Compression stream structure, holds the flush callback and opaque token,
    the buffered data, and a note of whether compressing or uncompressing.  */
@@ -48,45 +44,23 @@ struct lto_compression_stream
   bool is_compression;
 };
 
-/* Overall compression constants for zlib.  */
-
-static const size_t Z_BUFFER_LENGTH = 4096;
 static const size_t MIN_STREAM_ALLOCATION = 1024;
 
-/* For zlib, allocate SIZE count of ITEMS and return the address, OPAQUE
-   is unused.  */
-
-static void *
-lto_zalloc (void *opaque, unsigned items, unsigned size)
-{
-  gcc_assert (opaque == Z_NULL);
-  return xmalloc (items * size);
-}
-
-/* For zlib, free memory at ADDRESS, OPAQUE is unused.  */
-
-static void
-lto_zfree (void *opaque, void *address)
-{
-  gcc_assert (opaque == Z_NULL);
-  free (address);
-}
-
-/* Return a zlib compression level that zlib will not reject.  Normalizes
+/* Return a zstd compression level that zstd will not reject.  Normalizes
    the compression level from the command line flag, clamping non-default
    values to the appropriate end of their valid range.  */
 
 static int
-lto_normalized_zlib_level (void)
+lto_normalized_zstd_level (void)
 {
   int level = flag_lto_compression_level;
 
-  if (level != Z_DEFAULT_COMPRESSION)
+  if (level != ZSTD_CLEVEL_DEFAULT)
     {
-      if (level < Z_NO_COMPRESSION)
-	level = Z_NO_COMPRESSION;
-      else if (level > Z_BEST_COMPRESSION)
-	level = Z_BEST_COMPRESSION;
+      if (level < 1)
+	level = 1;
+      else if (level > ZSTD_maxCLevel ())
+	level = ZSTD_maxCLevel ();
     }
 
   return level;
@@ -169,57 +143,19 @@ void
 lto_end_compression (struct lto_compression_stream *stream)
 {
   unsigned char *cursor = (unsigned char *) stream->buffer;
-  size_t remaining = stream->bytes;
-  const size_t outbuf_length = Z_BUFFER_LENGTH;
-  unsigned char *outbuf = (unsigned char *) xmalloc (outbuf_length);
-  z_stream out_stream;
-  size_t compressed_bytes = 0;
-  int status;
-
-  gcc_assert (stream->is_compression);
+  size_t size = stream->bytes;
 
   timevar_push (TV_IPA_LTO_COMPRESS);
+  size_t const outbuf_length = ZSTD_compressBound (size);
+  char *outbuf = (char *) xmalloc (outbuf_length);
 
-  out_stream.next_out = outbuf;
-  out_stream.avail_out = outbuf_length;
-  out_stream.next_in = cursor;
-  out_stream.avail_in = remaining;
-  out_stream.zalloc = lto_zalloc;
-  out_stream.zfree = lto_zfree;
-  out_stream.opaque = Z_NULL;
+  size_t const csize = ZSTD_compress (outbuf, outbuf_length, cursor, size,
+				      lto_normalized_zstd_level ());
 
-  status = deflateInit (&out_stream, lto_normalized_zlib_level ());
-  if (status != Z_OK)
-    internal_error ("compressed stream: %s", zError (status));
+  if (ZSTD_isError (csize))
+    internal_error ("compressed stream: %s", ZSTD_getErrorName (csize));
 
-  do
-    {
-      size_t in_bytes, out_bytes;
-
-      status = deflate (&out_stream, Z_FINISH);
-      if (status != Z_OK && status != Z_STREAM_END)
-	internal_error ("compressed stream: %s", zError (status));
-
-      in_bytes = remaining - out_stream.avail_in;
-      out_bytes = outbuf_length - out_stream.avail_out;
-
-      stream->callback ((const char *) outbuf, out_bytes, stream->opaque);
-      lto_stats.num_compressed_il_bytes += out_bytes;
-      compressed_bytes += out_bytes;
-
-      cursor += in_bytes;
-      remaining -= in_bytes;
-
-      out_stream.next_out = outbuf;
-      out_stream.avail_out = outbuf_length;
-      out_stream.next_in = cursor;
-      out_stream.avail_in = remaining;
-    }
-  while (status != Z_STREAM_END);
-
-  status = deflateEnd (&out_stream);
-  if (status != Z_OK)
-    internal_error ("compressed stream: %s", zError (status));
+  stream->callback (outbuf, csize, NULL);
 
   lto_destroy_compression_stream (stream);
   free (outbuf);
@@ -258,61 +194,22 @@ void
 lto_end_uncompression (struct lto_compression_stream *stream)
 {
   unsigned char *cursor = (unsigned char *) stream->buffer;
-  size_t remaining = stream->bytes;
-  const size_t outbuf_length = Z_BUFFER_LENGTH;
-  unsigned char *outbuf = (unsigned char *) xmalloc (outbuf_length);
-  size_t uncompressed_bytes = 0;
+  size_t size = stream->bytes;
 
-  gcc_assert (!stream->is_compression);
   timevar_push (TV_IPA_LTO_DECOMPRESS);
+  unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size);
+  if (rsize == ZSTD_CONTENTSIZE_ERROR)
+    internal_error ("not compressed by zstd");
+  else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
+    internal_error ("original size unknown");
 
-  while (remaining > 0)
-    {
-      z_stream in_stream;
-      size_t out_bytes;
-      int status;
-
-      in_stream.next_out = outbuf;
-      in_stream.avail_out = outbuf_length;
-      in_stream.next_in = cursor;
-      in_stream.avail_in = remaining;
-      in_stream.zalloc = lto_zalloc;
-      in_stream.zfree = lto_zfree;
-      in_stream.opaque = Z_NULL;
-
-      status = inflateInit (&in_stream);
-      if (status != Z_OK)
-	internal_error ("compressed stream: %s", zError (status));
-
-      do
-	{
-	  size_t in_bytes;
-
-	  status = inflate (&in_stream, Z_SYNC_FLUSH);
-	  if (status != Z_OK && status != Z_STREAM_END)
-	    internal_error ("compressed stream: %s", zError (status));
-
-	  in_bytes = remaining - in_stream.avail_in;
-	  out_bytes = outbuf_length - in_stream.avail_out;
-
-	  stream->callback ((const char *) outbuf, out_bytes, stream->opaque);
-	  lto_stats.num_uncompressed_il_bytes += out_bytes;
-	  uncompressed_bytes += out_bytes;
-
-	  cursor += in_bytes;
-	  remaining -= in_bytes;
-
-	  in_stream.next_out = outbuf;
-	  in_stream.avail_out = outbuf_length;
-	  in_stream.next_in = cursor;
-	  in_stream.avail_in = remaining;
-	}
-      while (!(status == Z_STREAM_END && out_bytes == 0));
-
-      status = inflateEnd (&in_stream);
-      if (status != Z_OK)
-	internal_error ("compressed stream: %s", zError (status));
-    }
+  char *outbuf = (char *) xmalloc (rsize);
+  size_t const dsize = ZSTD_decompress (outbuf, rsize, cursor, size);
+
+  if (ZSTD_isError (dsize))
+    internal_error ("decompressed stream: %s", ZSTD_getErrorName (dsize));
+
+  stream->callback (outbuf, dsize, stream->opaque);
 
   lto_destroy_compression_stream (stream);
   free (outbuf);
diff --git a/gcc/timevar.def b/gcc/timevar.def
index 13cb470b688..626ce493b76 100644
--- a/gcc/timevar.def
+++ b/gcc/timevar.def
@@ -78,8 +78,8 @@ DEFTIMEVAR (TV_IPA_INLINING          , "ipa inlining heuristics")
 DEFTIMEVAR (TV_IPA_FNSPLIT           , "ipa function splitting")
 DEFTIMEVAR (TV_IPA_COMDATS	     , "ipa comdats")
 DEFTIMEVAR (TV_IPA_OPT		     , "ipa various optimizations")
-DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS    , "lto stream inflate")
-DEFTIMEVAR (TV_IPA_LTO_COMPRESS      , "lto stream deflate")
+DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS    , "lto stream decompression")
+DEFTIMEVAR (TV_IPA_LTO_COMPRESS      , "lto stream compression")
 DEFTIMEVAR (TV_IPA_LTO_OUTPUT        , "lto stream output")
 DEFTIMEVAR (TV_IPA_LTO_GIMPLE_IN     , "ipa lto gimple in")
 DEFTIMEVAR (TV_IPA_LTO_GIMPLE_OUT    , "ipa lto gimple out")
-- 
2.21.0


[-- Attachment #3: zstd-vs-zlib.ods --]
[-- Type: application/vnd.oasis.opendocument.spreadsheet, Size: 18640 bytes --]

Re: [RFC] zstd as a compression algorithm for LTO

[Jeff Law]

On 6/19/19 3:21 AM, Martin Liška wrote:
> Hi.
> 
> I've written a patch draft that replaces zlib with the zstd compression algorithm ([1])
> in LTO. I'm also sending statistics that are collected for couple of quite big C++ source
> files. Observation I did:
> 
> - LTO stream compression takes 3-4% of LGEN compile time
> - zstd in default compression level (3) generated slighly smaller LTO elf files
> - zstd compression is 4-8x faster
> - decompression is quite negligible, but for a bigger project (godot) I can
>   reduction from 1.37 to 0.53 seconds
> - ZSTD API is much simpler to use
> 
> Suggestion based on the observation:
> - I would suggest to make zstd optional (--enable-zstd) and one would
>   use #include <zstd> + -lzstd
> - I like the default level as we want to mainly speed up LTO compilation
> - we can provide an option to control algorithm (-flto-compression-algorithm),
>   similarly to -flto-compression-level
> - we can discuss possible compression of LTO bytecode that is distributed between WPA
>   stage and individual LTRANS phases.
Presumably the reason we're not being more aggressive about switching is
the build/run time dependency on zstd?  I wonder if we could default to
zstd and fallback to zlib when zstd isn't available?

jeff

Re: [RFC] zstd as a compression algorithm for LTO

[Richard Biener]

On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law <law@redhat.com> wrote:
>On 6/19/19 3:21 AM, Martin Liška wrote:
>> Hi.
>> 
>> I've written a patch draft that replaces zlib with the zstd
>compression algorithm ([1])
>> in LTO. I'm also sending statistics that are collected for couple of
>quite big C++ source
>> files. Observation I did:
>> 
>> - LTO stream compression takes 3-4% of LGEN compile time
>> - zstd in default compression level (3) generated slighly smaller LTO
>elf files
>> - zstd compression is 4-8x faster
>> - decompression is quite negligible, but for a bigger project (godot)
>I can
>>   reduction from 1.37 to 0.53 seconds
>> - ZSTD API is much simpler to use
>> 
>> Suggestion based on the observation:
>> - I would suggest to make zstd optional (--enable-zstd) and one would
>>   use #include <zstd> + -lzstd
>> - I like the default level as we want to mainly speed up LTO
>compilation
>> - we can provide an option to control algorithm
>(-flto-compression-algorithm),
>>   similarly to -flto-compression-level
>> - we can discuss possible compression of LTO bytecode that is
>distributed between WPA
>>   stage and individual LTRANS phases.
>Presumably the reason we're not being more aggressive about switching
>is
>the build/run time dependency on zstd?  I wonder if we could default to
>zstd and fallback to zlib when zstd isn't available?

Is zstd too big to include into the repository? But yes, we can properly encode the compression format in the LTO section header and use dlopen to 'find' the default to use. 

Richard. 

>jeff

Re: [RFC] zstd as a compression algorithm for LTO

[Andrew Pinski]

On Wed, Jun 19, 2019 at 11:55 AM Richard Biener
<richard.guenther@gmail.com> wrote:
>
> On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law <law@redhat.com> wrote:
> >On 6/19/19 3:21 AM, Martin Liška wrote:
> >> Hi.
> >>
> >> I've written a patch draft that replaces zlib with the zstd
> >compression algorithm ([1])
> >> in LTO. I'm also sending statistics that are collected for couple of
> >quite big C++ source
> >> files. Observation I did:
> >>
> >> - LTO stream compression takes 3-4% of LGEN compile time
> >> - zstd in default compression level (3) generated slighly smaller LTO
> >elf files
> >> - zstd compression is 4-8x faster
> >> - decompression is quite negligible, but for a bigger project (godot)
> >I can
> >>   reduction from 1.37 to 0.53 seconds
> >> - ZSTD API is much simpler to use
> >>
> >> Suggestion based on the observation:
> >> - I would suggest to make zstd optional (--enable-zstd) and one would
> >>   use #include <zstd> + -lzstd
> >> - I like the default level as we want to mainly speed up LTO
> >compilation
> >> - we can provide an option to control algorithm
> >(-flto-compression-algorithm),
> >>   similarly to -flto-compression-level
> >> - we can discuss possible compression of LTO bytecode that is
> >distributed between WPA
> >>   stage and individual LTRANS phases.
> >Presumably the reason we're not being more aggressive about switching
> >is
> >the build/run time dependency on zstd?  I wonder if we could default to
> >zstd and fallback to zlib when zstd isn't available?
>
> Is zstd too big to include into the repository? But yes, we can properly encode the compression format in the LTO section header and use dlopen to 'find' the default to use.

At least allow it to be built as part of the normal build like GMP,
etc. are done.
And include it in downloading using contrib/download_prerequisites
like the libraries are done.

Thanks,
Andrew Pinski

>
> Richard.
>
> >jeff
>

Re: [RFC] zstd as a compression algorithm for LTO

[Jan Hubicka]

> 
> At least allow it to be built as part of the normal build like GMP,
> etc. are done.
> And include it in downloading using contrib/download_prerequisites
> like the libraries are done.

Anoying detail is that zstd builds with cmake, not autotools

Honza
> 
> Thanks,
> Andrew Pinski
> 
> >
> > Richard.
> >
> > >jeff
> >

Re: [RFC] zstd as a compression algorithm for LTO

[Andrew Pinski]

On Wed, Jun 19, 2019 at 12:29 PM Jan Hubicka <hubicka@ucw.cz> wrote:
>
> >
> > At least allow it to be built as part of the normal build like GMP,
> > etc. are done.
> > And include it in downloading using contrib/download_prerequisites
> > like the libraries are done.
>
> Anoying detail is that zstd builds with cmake, not autotools

That makes doing candian crosses interesting and 1000x times harder
than it should be :).

Thanks,
Andrew Pinski

>
> Honza
> >
> > Thanks,
> > Andrew Pinski
> >
> > >
> > > Richard.
> > >
> > > >jeff
> > >

Re: [RFC] zstd as a compression algorithm for LTO

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

Hello.

As mentioned by Honza, it's using cmake and to be honest I prefer to use a shared
library than a statically build library. Moreover, it's an optional requirement and
so that we don't have to include that to contrib/download_prerequisites.

I like the idea of marking of compression algorithm in 'LTO_header'. However,
we do compress the header as well. Proper solution would be to make a new
section .gnu.lto_.header where we'll put:
struct lto_header
{
  int16_t major_version;
  int16_t minor_version;
};

I don't see a reason why why should have that information in each LTO ELF section?

In the time being, I've written the code so that I fallback in decompression to zlib
if ZSTD detects that LTO bytecode was compressed with zlib. On the contrary, decompression
of zstd with zlib will end with:
lto1: internal compiler error: compressed stream: data error

I'm sending updated version of the patch that can properly detect zstd.

Martin


[-- Attachment #2: 0002-Add-optional-support-for-zstd.patch --]
[-- Type: text/x-patch, Size: 7359 bytes --]

From 869b630139676fb740fb5296d68086a8ef7f03ae Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Wed, 19 Jun 2019 09:40:35 +0200
Subject: [PATCH 2/2] Add optional support for zstd.

---
 gcc/common.opt     |   4 +-
 gcc/lto-compress.c | 139 ++++++++++++++++++++++++++++++++++++++-------
 gcc/timevar.def    |   4 +-
 3 files changed, 122 insertions(+), 25 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index a1544d06824..3b71a36552b 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1888,8 +1888,8 @@ Specify the algorithm to partition symbols and vars at linktime.
 
 ; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h.
 flto-compression-level=
-Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 9)
--flto-compression-level=<number>	Use zlib compression level <number> for IL.
+Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 19)
+-flto-compression-level=<number>	Use zlib/zstd compression level <number> for IL.
 
 flto-odr-type-merging
 Common Ignore
diff --git a/gcc/lto-compress.c b/gcc/lto-compress.c
index 3287178f257..327ff9e07b7 100644
--- a/gcc/lto-compress.c
+++ b/gcc/lto-compress.c
@@ -35,6 +35,10 @@ along with GCC; see the file COPYING3.  If not see
 #include "lto-compress.h"
 #include "timevar.h"
 
+#ifdef HAVE_ZSTD_H
+#include <zstd.h>
+#endif
+
 /* Compression stream structure, holds the flush callback and opaque token,
    the buffered data, and a note of whether compressing or uncompressing.  */
 
@@ -92,6 +96,95 @@ lto_normalized_zlib_level (void)
   return level;
 }
 
+/* Free the buffer and memory associated with STREAM.  */
+
+static void
+lto_destroy_compression_stream (struct lto_compression_stream *stream)
+{
+  free (stream->buffer);
+  free (stream);
+}
+
+#ifdef HAVE_ZSTD_H
+/* Return a zstd compression level that zstd will not reject.  Normalizes
+   the compression level from the command line flag, clamping non-default
+   values to the appropriate end of their valid range.  */
+
+static int
+lto_normalized_zstd_level (void)
+{
+  int level = flag_lto_compression_level;
+
+  if (level != ZSTD_CLEVEL_DEFAULT)
+    {
+      if (level < 1)
+	level = 1;
+      else if (level > ZSTD_maxCLevel ())
+	level = ZSTD_maxCLevel ();
+    }
+
+  return level;
+}
+
+/* Compress STREAM using ZSTD algorithm.  */
+
+static void
+lto_compression_zstd (struct lto_compression_stream *stream)
+{
+  unsigned char *cursor = (unsigned char *) stream->buffer;
+  size_t size = stream->bytes;
+
+  timevar_push (TV_IPA_LTO_COMPRESS);
+  size_t const outbuf_length = ZSTD_compressBound (size);
+  char *outbuf = (char *) xmalloc (outbuf_length);
+
+  size_t const csize = ZSTD_compress (outbuf, outbuf_length, cursor, size,
+				      lto_normalized_zstd_level ());
+
+  if (ZSTD_isError (csize))
+    internal_error ("compressed stream: %s", ZSTD_getErrorName (csize));
+
+  stream->callback (outbuf, csize, NULL);
+
+  lto_destroy_compression_stream (stream);
+  free (outbuf);
+  timevar_pop (TV_IPA_LTO_COMPRESS);
+}
+
+/* Uncompress STREAM using ZSTD algorithm.  */
+
+static bool
+lto_uncompression_zstd (struct lto_compression_stream *stream)
+{
+  unsigned char *cursor = (unsigned char *) stream->buffer;
+  size_t size = stream->bytes;
+
+  timevar_push (TV_IPA_LTO_DECOMPRESS);
+  unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size);
+  if (rsize == ZSTD_CONTENTSIZE_ERROR)
+    {
+      /* The content is probably using zlib.  */
+      return false;
+    }
+  else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
+    internal_error ("original size unknown");
+
+  char *outbuf = (char *) xmalloc (rsize);
+  size_t const dsize = ZSTD_decompress (outbuf, rsize, cursor, size);
+
+  if (ZSTD_isError (dsize))
+    internal_error ("decompressed stream: %s", ZSTD_getErrorName (dsize));
+
+  stream->callback (outbuf, dsize, stream->opaque);
+
+  lto_destroy_compression_stream (stream);
+  free (outbuf);
+  timevar_pop (TV_IPA_LTO_DECOMPRESS);
+  return true;
+}
+
+#endif
+
 /* Create a new compression stream, with CALLBACK flush function passed
    OPAQUE token, IS_COMPRESSION indicates if compressing or uncompressing.  */
 
@@ -132,15 +225,6 @@ lto_append_to_compression_stream (struct lto_compression_stream *stream,
   stream->bytes += num_chars;
 }
 
-/* Free the buffer and memory associated with STREAM.  */
-
-static void
-lto_destroy_compression_stream (struct lto_compression_stream *stream)
-{
-  free (stream->buffer);
-  free (stream);
-}
-
 /* Return a new compression stream, with CALLBACK flush function passed
    OPAQUE token.  */
 
@@ -163,10 +247,8 @@ lto_compress_block (struct lto_compression_stream *stream,
   lto_stats.num_output_il_bytes += num_chars;
 }
 
-/* Finalize STREAM compression, and free stream allocations.  */
-
-void
-lto_end_compression (struct lto_compression_stream *stream)
+static void
+lto_compression_zlib (struct lto_compression_stream *stream)
 {
   unsigned char *cursor = (unsigned char *) stream->buffer;
   size_t remaining = stream->bytes;
@@ -226,6 +308,16 @@ lto_end_compression (struct lto_compression_stream *stream)
   timevar_pop (TV_IPA_LTO_COMPRESS);
 }
 
+void
+lto_end_compression (struct lto_compression_stream *stream)
+{
+#ifdef HAVE_ZSTD_H
+  lto_compression_zstd (stream);
+#else
+  lto_compression_zlib (stream);
+#endif
+}
+
 /* Return a new uncompression stream, with CALLBACK flush function passed
    OPAQUE token.  */
 
@@ -248,14 +340,8 @@ lto_uncompress_block (struct lto_compression_stream *stream,
   lto_stats.num_input_il_bytes += num_chars;
 }
 
-/* Finalize STREAM uncompression, and free stream allocations.
-
-   Because of the way LTO IL streams are compressed, there may be several
-   concatenated compressed segments in the accumulated data, so for this
-   function we iterate decompressions until no data remains.  */
-
-void
-lto_end_uncompression (struct lto_compression_stream *stream)
+static void
+lto_uncompression_zlib (struct lto_compression_stream *stream)
 {
   unsigned char *cursor = (unsigned char *) stream->buffer;
   size_t remaining = stream->bytes;
@@ -318,3 +404,14 @@ lto_end_uncompression (struct lto_compression_stream *stream)
   free (outbuf);
   timevar_pop (TV_IPA_LTO_DECOMPRESS);
 }
+
+void
+lto_end_uncompression (struct lto_compression_stream *stream)
+{
+#ifdef HAVE_ZSTD_H
+  bool done = lto_uncompression_zstd (stream);
+  if (done)
+    return;
+#endif
+  lto_uncompression_zlib (stream);
+}
diff --git a/gcc/timevar.def b/gcc/timevar.def
index 13cb470b688..626ce493b76 100644
--- a/gcc/timevar.def
+++ b/gcc/timevar.def
@@ -78,8 +78,8 @@ DEFTIMEVAR (TV_IPA_INLINING          , "ipa inlining heuristics")
 DEFTIMEVAR (TV_IPA_FNSPLIT           , "ipa function splitting")
 DEFTIMEVAR (TV_IPA_COMDATS	     , "ipa comdats")
 DEFTIMEVAR (TV_IPA_OPT		     , "ipa various optimizations")
-DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS    , "lto stream inflate")
-DEFTIMEVAR (TV_IPA_LTO_COMPRESS      , "lto stream deflate")
+DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS    , "lto stream decompression")
+DEFTIMEVAR (TV_IPA_LTO_COMPRESS      , "lto stream compression")
 DEFTIMEVAR (TV_IPA_LTO_OUTPUT        , "lto stream output")
 DEFTIMEVAR (TV_IPA_LTO_GIMPLE_IN     , "ipa lto gimple in")
 DEFTIMEVAR (TV_IPA_LTO_GIMPLE_OUT    , "ipa lto gimple out")
-- 
2.21.0


[-- Attachment #3: 0001-Configure-detection-for-zstd.patch --]
[-- Type: text/x-patch, Size: 5987 bytes --]

From 7294489d924dbe6c01ba5c66e83793757677465f Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Thu, 20 Jun 2019 10:08:17 +0200
Subject: [PATCH 1/2] Configure detection for zstd.

---
 gcc/Makefile.in  |   4 +-
 gcc/config.in    |   6 +++
 gcc/configure    | 101 ++++++++++++++++++++++++++++++++++++++++++++++-
 gcc/configure.ac |  21 ++++++++++
 4 files changed, 129 insertions(+), 3 deletions(-)

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index d9e0885b96b..597dc01328b 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1065,7 +1065,7 @@ BUILD_LIBDEPS= $(BUILD_LIBIBERTY)
 LIBS = @LIBS@ libcommon.a $(CPPLIB) $(LIBINTL) $(LIBICONV) $(LIBBACKTRACE) \
 	$(LIBIBERTY) $(LIBDECNUMBER) $(HOST_LIBS)
 BACKENDLIBS = $(ISLLIBS) $(GMPLIBS) $(PLUGINLIBS) $(HOST_LIBS) \
-	$(ZLIB)
+	$(ZLIB) $(ZSTD_LIB)
 # Any system libraries needed just for GNAT.
 SYSLIBS = @GNAT_LIBEXC@
 
@@ -1076,6 +1076,8 @@ GNATMAKE = @GNATMAKE@
 # Libs needed (at present) just for jcf-dump.
 LDEXP_LIB = @LDEXP_LIB@
 
+ZSTD_LIB = @ZSTD_LIB@
+
 # Likewise, for use in the tools that must run on this machine
 # even if we are cross-building GCC.
 BUILD_LIBS = $(BUILD_LIBIBERTY)
diff --git a/gcc/config.in b/gcc/config.in
index a718ceaf3da..13fd7959dd7 100644
--- a/gcc/config.in
+++ b/gcc/config.in
@@ -1926,6 +1926,12 @@
 #endif
 
 
+/* Define if you have a working <zstd.h> header file. */
+#ifndef USED_FOR_TARGET
+#undef HAVE_ZSTD_H
+#endif
+
+
 /* Define if isl is in use. */
 #ifndef USED_FOR_TARGET
 #undef HAVE_isl
diff --git a/gcc/configure b/gcc/configure
index 955e9ccc09b..9c3439c156b 100755
--- a/gcc/configure
+++ b/gcc/configure
@@ -782,6 +782,7 @@ manext
 LIBICONV_DEP
 LTLIBICONV
 LIBICONV
+ZSTD_LIB
 DL_LIB
 LDEXP_LIB
 EXTRA_GCC_LIBS
@@ -9798,6 +9799,70 @@ DL_LIB="$LIBS"
 LIBS="$save_LIBS"
 
 
+# LTO can use zstd compression algorithm
+save_LIBS="$LIBS"
+LIBS=
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing ZSTD_compress" >&5
+$as_echo_n "checking for library containing ZSTD_compress... " >&6; }
+if ${ac_cv_search_ZSTD_compress+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_func_search_save_LIBS=$LIBS
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+  ;
+  return 0;
+}
+_ACEOF
+for ac_lib in '' zstd; do
+  if test -z "$ac_lib"; then
+    ac_res="none required"
+  else
+    ac_res=-l$ac_lib
+    LIBS="-l$ac_lib  $ac_func_search_save_LIBS"
+  fi
+  if ac_fn_cxx_try_link "$LINENO"; then :
+  ac_cv_search_ZSTD_compress=$ac_res
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext
+  if ${ac_cv_search_ZSTD_compress+:} false; then :
+  break
+fi
+done
+if ${ac_cv_search_ZSTD_compress+:} false; then :
+
+else
+  ac_cv_search_ZSTD_compress=no
+fi
+rm conftest.$ac_ext
+LIBS=$ac_func_search_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_search_ZSTD_compress" >&5
+$as_echo "$ac_cv_search_ZSTD_compress" >&6; }
+ac_res=$ac_cv_search_ZSTD_compress
+if test "$ac_res" != no; then :
+  test "$ac_res" = "none required" || LIBS="$ac_res $LIBS"
+
+fi
+
+ZSTD_LIB="$LIBS"
+LIBS="$save_LIBS"
+
+
+
 # Use <inttypes.h> only if it exists,
 # doesn't clash with <sys/types.h>, declares intmax_t and defines
 # PRId64
@@ -9838,6 +9903,38 @@ $as_echo "#define HAVE_INTTYPES_H 1" >>confdefs.h
 
 fi
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for zstd.h" >&5
+$as_echo_n "checking for zstd.h... " >&6; }
+if ${gcc_cv_header_zstd_h+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+#include <zstd.h>
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_cxx_try_compile "$LINENO"; then :
+  gcc_cv_header_zstd_h=yes
+else
+  gcc_cv_header_zstd_h=no
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+fi
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $gcc_cv_header_zstd_h" >&5
+$as_echo "$gcc_cv_header_zstd_h" >&6; }
+if test $gcc_cv_header_zstd_h = yes; then
+
+$as_echo "#define HAVE_ZSTD_H 1" >>confdefs.h
+
+fi
+
 
 
 for ac_func in times clock kill getrlimit setrlimit atoq \
@@ -18655,7 +18752,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 18658 "configure"
+#line 18755 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -18761,7 +18858,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 18764 "configure"
+#line 18861 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
diff --git a/gcc/configure.ac b/gcc/configure.ac
index 066a6f4c958..3ee4fd2e2ee 100644
--- a/gcc/configure.ac
+++ b/gcc/configure.ac
@@ -1237,6 +1237,15 @@ DL_LIB="$LIBS"
 LIBS="$save_LIBS"
 AC_SUBST(DL_LIB)
 
+# LTO can use zstd compression algorithm
+save_LIBS="$LIBS"
+LIBS=
+AC_SEARCH_LIBS(ZSTD_compress, zstd)
+ZSTD_LIB="$LIBS"
+LIBS="$save_LIBS"
+AC_SUBST(ZSTD_LIB)
+
+
 # Use <inttypes.h> only if it exists,
 # doesn't clash with <sys/types.h>, declares intmax_t and defines
 # PRId64
@@ -1258,6 +1267,18 @@ if test $gcc_cv_header_inttypes_h = yes; then
 	[Define if you have a working <inttypes.h> header file.])
 fi
 
+AC_MSG_CHECKING(for zstd.h)
+AC_CACHE_VAL(gcc_cv_header_zstd_h,
+[AC_COMPILE_IFELSE([AC_LANG_PROGRAM(
+[[#include <zstd.h>]])],
+  [gcc_cv_header_zstd_h=yes],
+  [gcc_cv_header_zstd_h=no])])
+AC_MSG_RESULT($gcc_cv_header_zstd_h)
+if test $gcc_cv_header_zstd_h = yes; then
+  AC_DEFINE(HAVE_ZSTD_H, 1,
+	[Define if you have a working <zstd.h> header file.])
+fi
+
 dnl Disabled until we have a complete test for buggy enum bitfields.
 dnl gcc_AC_C_ENUM_BF_UNSIGNED
 
-- 
2.21.0

Re: [RFC] zstd as a compression algorithm for LTO

[Thomas Koenig]

Am 20.06.19 um 11:07 schrieb Martin Liška:
> On the contrary, decompression
> of zstd with zlib will end with:
> lto1: internal compiler error: compressed stream: data error

Sogenerating object files on one system and trying to read them
on another system which does not happen to have a particular
library installed would lead to failure?  If that's the case,
I am not sure that this is a good way of handling things.

Re: [RFC] zstd as a compression algorithm for LTO

[Martin Liška]

On 6/20/19 12:58 PM, Thomas Koenig wrote:
> Am 20.06.19 um 11:07 schrieb Martin Liška:
>> On the contrary, decompression
>> of zstd with zlib will end with:
>> lto1: internal compiler error: compressed stream: data error
> 
> Sogenerating object files on one system and trying to read them
> on another system which does not happen to have a particular
> library installed would lead to failure?  If that's the case,
> I am not sure that this is a good way of handling things.

Yes, but LTO bytecode is not supposed to be a distributable format.

Martin

Re: [RFC] zstd as a compression algorithm for LTO

[Jan Hubicka]

> On 6/20/19 12:58 PM, Thomas Koenig wrote:
> > Am 20.06.19 um 11:07 schrieb Martin Liška:
> >> On the contrary, decompression
> >> of zstd with zlib will end with:
> >> lto1: internal compiler error: compressed stream: data error
> > 
> > Sogenerating object files on one system and trying to read them
> > on another system which does not happen to have a particular
> > library installed would lead to failure?  If that's the case,
> > I am not sure that this is a good way of handling things.
> 
> Yes, but LTO bytecode is not supposed to be a distributable format.

In longer term it should be.  We ought to make it host independent and
stable at least within major releases.
I guess it is still OK to make zstd enabled host build require zstd
enabled gcc elsewhere. Just the error message should be more informative
which I think is not hard to do - both zstd and zlip should have
recognizable header.

Other option is to put this into some common place per file.

Honza
> 
> Martin

[PATCH] Add .gnu.lto_.meta section.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

Hi.

The patch is about a new ELF section that will contain information
about LTO version. And for the future, used compression will be stored
here. The patch removes streaming of the version into each section.

Disadvantage is a format change that will lead to following errors when
LTO bytecode is used from a different version:

$ gcc x.o
lto1: fatal error: bytecode stream in file ‘x.o’ generated with GCC compiler older than 10.0

$ gcc-9 main.o
lto1: fatal error: bytecode stream in file ‘main.o’ generated with LTO version 850.0 instead of the expected 8.0

I've been testing the patch.
Thoughts?
Martin

[-- Attachment #2: 0001-Add-.gnu.lto_.meta-section.patch --]
[-- Type: text/x-patch, Size: 7278 bytes --]

From a694d3244812bb8c0ae79d3ef5aecc6f5a2bd9ca Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Fri, 21 Jun 2019 12:14:04 +0200
Subject: [PATCH] Add .gnu.lto_.meta section.

gcc/ChangeLog:

2019-06-21  Martin Liska  <mliska@suse.cz>

	* lto-section-in.c (lto_get_section_data): Do not
	decompress LTO_section_meta section.
	* lto-section-out.c (lto_destroy_simple_output_block):
	Do not set major and minor version.
	* lto-streamer-out.c (produce_asm): Likewise.
	(lto_output_toplevel_asms): Likewise here.
	(produce_lto_meta): New.
	(lto_output): Produce LTO meta section.
	(lto_write_mode_table):
	Do not set major and minor version.
	(produce_asm_for_decls): Likewise.
	* lto-streamer.h (enum lto_section_type): Add
	LTO_section_meta.
	(struct lto_simple_header): Do not inherit
	from lto_simple_header.

gcc/lto/ChangeLog:

2019-06-21  Martin Liska  <mliska@suse.cz>

	* lto-common.c: Check for LTO bytecode version.
---
 gcc/lto-section-in.c   |  9 +++------
 gcc/lto-section-out.c  |  2 --
 gcc/lto-streamer-out.c | 36 +++++++++++++++++++++---------------
 gcc/lto-streamer.h     |  3 ++-
 gcc/lto/lto-common.c   | 14 ++++++++++++++
 5 files changed, 40 insertions(+), 24 deletions(-)

diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c
index 4cfc0cad4be..96962f2f2c4 100644
--- a/gcc/lto-section-in.c
+++ b/gcc/lto-section-in.c
@@ -52,10 +52,10 @@ const char *lto_section_name[LTO_N_SECTION_TYPES] =
   "icf",
   "offload_table",
   "mode_table",
-  "hsa"
+  "hsa",
+  "meta"
 };
 
-
 /* Hooks so that the ipa passes can call into the lto front end to get
    sections.  */
 
@@ -146,7 +146,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
   /* WPA->ltrans streams are not compressed with exception of function bodies
      and variable initializers that has been verbatim copied from earlier
      compilations.  */
-  if (!flag_ltrans || decompress)
+  if ((!flag_ltrans || decompress) && section_type != LTO_section_meta)
     {
       /* Create a mapping header containing the underlying data and length,
 	 and prepend this to the uncompression buffer.  The uncompressed data
@@ -167,9 +167,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
       data = buffer.data + header_length;
     }
 
-  lto_check_version (((const lto_header *)data)->major_version,
-		     ((const lto_header *)data)->minor_version,
-		     file_data->file_name);
   return data;
 }
 
diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c
index c91e58f0465..7ae102164ef 100644
--- a/gcc/lto-section-out.c
+++ b/gcc/lto-section-out.c
@@ -285,8 +285,6 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob)
   /* Write the header which says how to decode the pieces of the
      t.  */
   memset (&header, 0, sizeof (struct lto_simple_header));
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
   header.main_size = ob->main_stream->total_size;
   lto_write_data (&header, sizeof header);
 
diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c
index b1084f4d3c5..bd259097dc4 100644
--- a/gcc/lto-streamer-out.c
+++ b/gcc/lto-streamer-out.c
@@ -1972,10 +1972,6 @@ produce_asm (struct output_block *ob, tree fn)
   /* The entire header is stream computed here.  */
   memset (&header, 0, sizeof (struct lto_function_header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   if (section_type == LTO_section_function_body)
     header.cfg_size = ob->cfg_stream->total_size;
   header.main_size = ob->main_stream->total_size;
@@ -2268,10 +2264,6 @@ lto_output_toplevel_asms (void)
   /* The entire header stream is computed here.  */
   memset (&header, 0, sizeof (header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
   lto_write_data (&header, sizeof header);
@@ -2388,6 +2380,25 @@ prune_offload_funcs (void)
     DECL_PRESERVE_P (fn_decl) = 1;
 }
 
+/* Produce LTO meta section that contains global information
+   about LTO bytecode.  */
+
+static void
+produce_lto_meta ()
+{
+  /* Stream LTO meta section.  */
+  output_block *ob = create_output_block (LTO_section_meta);
+
+  char * section_name = lto_get_section_name (LTO_section_meta, NULL, NULL);
+  lto_begin_section (section_name, false);
+  free (section_name);
+
+  lto_header header = { LTO_major_version, LTO_minor_version };
+  lto_write_data (&header, sizeof header);
+  lto_end_section ();
+  destroy_output_block (ob);
+}
+
 /* Main entry point from the pass manager.  */
 
 void
@@ -2406,6 +2417,8 @@ lto_output (void)
   /* Initialize the streamer.  */
   lto_streamer_init ();
 
+  produce_lto_meta ();
+
   n_nodes = lto_symtab_encoder_size (encoder);
   /* Process only the functions with bodies.  */
   for (i = 0; i < n_nodes; i++)
@@ -2825,10 +2838,6 @@ lto_write_mode_table (void)
   struct lto_simple_header_with_strings header;
   memset (&header, 0, sizeof (header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
   lto_write_data (&header, sizeof header);
@@ -2899,9 +2908,6 @@ produce_asm_for_decls (void)
       lto_output_decl_state_streams (ob, fn_out_state);
     }
 
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   /* Currently not used.  This field would allow us to preallocate
      the globals vector, so that it need not be resized as it is extended.  */
   header.num_nodes = -1;
diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h
index d087cba9bf0..1b20161d193 100644
--- a/gcc/lto-streamer.h
+++ b/gcc/lto-streamer.h
@@ -234,6 +234,7 @@ enum lto_section_type
   LTO_section_offload_table,
   LTO_section_mode_table,
   LTO_section_ipa_hsa,
+  LTO_section_meta,
   LTO_N_SECTION_TYPES		/* Must be last.  */
 };
 
@@ -389,7 +390,7 @@ struct lto_header
 
 /* The is the first part of the record in an LTO file for many of the
    IPA passes.  */
-struct lto_simple_header : lto_header
+struct lto_simple_header
 {
   /* Size of main gimple body of function.  */
   int32_t main_size;
diff --git a/gcc/lto/lto-common.c b/gcc/lto/lto-common.c
index 9d2f9c57780..6e64a5e4110 100644
--- a/gcc/lto/lto-common.c
+++ b/gcc/lto/lto-common.c
@@ -2058,6 +2058,20 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file)
 #else
   file_data->mode_table = lto_mode_identity_table;
 #endif
+
+  /* Read and verify LTO meta header.  */
+  data = lto_get_section_data (file_data, LTO_section_meta, NULL, &len, false);
+  if (data == NULL)
+    {
+      fatal_error (input_location, "bytecode stream in file %qs generated "
+		   "with GCC compiler older than 10.0", file_data->file_name);
+      return;
+    }
+
+  lto_check_version (((const lto_header *)data)->major_version,
+		     ((const lto_header *)data)->minor_version,
+		     file_data->file_name);
+
   data = lto_get_section_data (file_data, LTO_section_decls, NULL, &len);
   if (data == NULL)
     {
-- 
2.21.0

Re: [PATCH] Add .gnu.lto_.meta section.

[Richard Biener]

On Fri, Jun 21, 2019 at 12:20 PM Martin Liška <mliska@suse.cz> wrote:
>
> Hi.
>
> The patch is about a new ELF section that will contain information
> about LTO version. And for the future, used compression will be stored
> here. The patch removes streaming of the version into each section.

I'd like each section to have a header containing a compression method
(compressed or not, and now zlib vs. zstd).  We currently have a mix
and knowledge is hidden.

I also remember I had old patches to make the data streamer compress
the stream on-the-fly, not requiring the full uncompressed stream in
memory and then re-access it for compression (needing more temporary
memory).  But IIRC the speedup was marginal.

My thought is also that we should have exactly _one_ ELF section
for the LTO bytecode and our own container inside (much like
simple-object does for non-ELF).  So having another one is, well, ugly ;)

> Disadvantage is a format change that will lead to following errors when
> LTO bytecode is used from a different version:
>
> $ gcc x.o
> lto1: fatal error: bytecode stream in file ‘x.o’ generated with GCC compiler older than 10.0
>
> $ gcc-9 main.o
> lto1: fatal error: bytecode stream in file ‘main.o’ generated with LTO version 850.0 instead of the expected 8.0
>
> I've been testing the patch.
> Thoughts?
> Martin

Re: [PATCH] Add .gnu.lto_.meta section.

[Martin Liška]

On 6/21/19 2:34 PM, Richard Biener wrote:
> On Fri, Jun 21, 2019 at 12:20 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> Hi.
>>
>> The patch is about a new ELF section that will contain information
>> about LTO version. And for the future, used compression will be stored
>> here. The patch removes streaming of the version into each section.
> 
> I'd like each section to have a header containing a compression method
> (compressed or not, and now zlib vs. zstd).  We currently have a mix
> and knowledge is hidden.

That would be possible, good idea.

> 
> I also remember I had old patches to make the data streamer compress
> the stream on-the-fly, not requiring the full uncompressed stream in
> memory and then re-access it for compression (needing more temporary
> memory).  But IIRC the speedup was marginal.
> 
> My thought is also that we should have exactly _one_ ELF section
> for the LTO bytecode and our own container inside (much like
> simple-object does for non-ELF).  So having another one is, well, ugly ;)

Having N sections for all symbols (functions and variables) is handy because
we read some of the during WPA (by IPA ICF). So having all in a single section
would make decompression more complicated.

I'm going to prepare a patch that will pull out a LTO section header
from compressed stream.

Martin

> 
>> Disadvantage is a format change that will lead to following errors when
>> LTO bytecode is used from a different version:
>>
>> $ gcc x.o
>> lto1: fatal error: bytecode stream in file ‘x.o’ generated with GCC compiler older than 10.0
>>
>> $ gcc-9 main.o
>> lto1: fatal error: bytecode stream in file ‘main.o’ generated with LTO version 850.0 instead of the expected 8.0
>>
>> I've been testing the patch.
>> Thoughts?
>> Martin

Re: [PATCH] Add .gnu.lto_.meta section.

[Jan Hubicka]

> On 6/21/19 2:34 PM, Richard Biener wrote:
> > On Fri, Jun 21, 2019 at 12:20 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> Hi.
> >>
> >> The patch is about a new ELF section that will contain information
> >> about LTO version. And for the future, used compression will be stored
> >> here. The patch removes streaming of the version into each section.
> > 
> > I'd like each section to have a header containing a compression method
> > (compressed or not, and now zlib vs. zstd).  We currently have a mix
> > and knowledge is hidden.
> 
> That would be possible, good idea.
> 
> > 
> > I also remember I had old patches to make the data streamer compress
> > the stream on-the-fly, not requiring the full uncompressed stream in
> > memory and then re-access it for compression (needing more temporary
> > memory).  But IIRC the speedup was marginal.

It may be more interesting as memory optimization now we do parallel WPA
streaming BTW.
> > 
> > My thought is also that we should have exactly _one_ ELF section
> > for the LTO bytecode and our own container inside (much like
> > simple-object does for non-ELF).  So having another one is, well, ugly ;)
> 
> Having N sections for all symbols (functions and variables) is handy because
> we read some of the during WPA (by IPA ICF). So having all in a single section
> would make decompression more complicated.

There is nothing preventing us from implementing our own subsections in
that section where each is compressed independently and can be copied to
output file and still save size of headers needed for that.

For example all sections are keyed by the lto section enum + possibly a
decl that can be both streamed as integers rather than quite lenghtly
strings.  (the decl to name is streamed into the global decl stream and
available at all time).

I guess one can go for one global LTO major/minor in the section header
and then  section type/is compressed/size + optional decl ID for each
subsection reducing headers to about 1+8+optional 8 bytes per header.
This can be all placed at the begining of the file followed by RAW data
of individual sections.  I bet this is a lot smaller than ELF overhead :)
> 
> I'm going to prepare a patch that will pull out a LTO section header
> from compressed stream.
This looks like good step (and please stream it in host independent
way). I suppose all these issues can be done one-by-one.

Honza
> 
> Martin
> 
> > 
> >> Disadvantage is a format change that will lead to following errors when
> >> LTO bytecode is used from a different version:
> >>
> >> $ gcc x.o
> >> lto1: fatal error: bytecode stream in file ‘x.o’ generated with GCC compiler older than 10.0
> >>
> >> $ gcc-9 main.o
> >> lto1: fatal error: bytecode stream in file ‘main.o’ generated with LTO version 850.0 instead of the expected 8.0
> >>
> >> I've been testing the patch.
> >> Thoughts?
> >> Martin
> 

Re: [PATCH] Add .gnu.lto_.meta section.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On 6/21/19 2:57 PM, Jan Hubicka wrote:
> This looks like good step (and please stream it in host independent
> way). I suppose all these issues can be done one-by-one.

So there's a working patch for that. However one will see following errors
when using an older compiler or older LTO bytecode:

$ gcc main9.o -flto
lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0

$ gcc main.o
lto1: internal compiler error: compressed stream: data error

To be honest, I would prefer the new .gnu.lto_.meta section.
Richi why is that so ugly?

Martin

[-- Attachment #2: 0001-Come-up-with-ELF-LTO-section-header.patch --]
[-- Type: text/x-patch, Size: 5453 bytes --]

From cbba4d8209bfeed11856baeaa2f82e44d93f4cb7 Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Fri, 21 Jun 2019 15:57:08 +0200
Subject: [PATCH] Come up with ELF LTO section header.

---
 gcc/lto-section-in.c   | 14 +++++++++++---
 gcc/lto-section-out.c  | 16 ++++++++++++++--
 gcc/lto-streamer-out.c | 12 ++++--------
 gcc/lto-streamer.h     | 14 +++++++++++---
 4 files changed, 40 insertions(+), 16 deletions(-)

diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c
index 4cfc0cad4be..28dfb412b67 100644
--- a/gcc/lto-section-in.c
+++ b/gcc/lto-section-in.c
@@ -143,6 +143,17 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
   if (data == NULL)
     return NULL;
 
+  /* First check LTO section header.  */
+  if (*len < sizeof(lto_section_header))
+    return NULL;
+
+  const lto_section_header *sh = (const lto_section_header *)data;
+  lto_check_version (sh->major_version, sh->minor_version,
+		     file_data->file_name);
+
+  data += sizeof (lto_section_header);
+  *len -= sizeof (lto_section_header);
+
   /* WPA->ltrans streams are not compressed with exception of function bodies
      and variable initializers that has been verbatim copied from earlier
      compilations.  */
@@ -167,9 +178,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
       data = buffer.data + header_length;
     }
 
-  lto_check_version (((const lto_header *)data)->major_version,
-		     ((const lto_header *)data)->minor_version,
-		     file_data->file_name);
   return data;
 }
 
diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c
index c91e58f0465..9417bc9c174 100644
--- a/gcc/lto-section-out.c
+++ b/gcc/lto-section-out.c
@@ -81,6 +81,19 @@ lto_begin_section (const char *name, bool compress)
     compression_stream = lto_start_compression (lto_append_data, NULL);
 }
 
+/* Emit section header with LTO bytecode version and information about used
+   compression.  */
+
+void lto_emit_section_header (void)
+{
+  lto_compression compression = RAW;
+  if (compression_stream != NULL)
+    compression = ZLIB;
+
+  lto_section_header header
+    = { LTO_major_version, LTO_minor_version, compression };
+  lto_write_raw_data (&header, sizeof header);
+}
 
 /* End the current output section.  */
 
@@ -285,8 +298,7 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob)
   /* Write the header which says how to decode the pieces of the
      t.  */
   memset (&header, 0, sizeof (struct lto_simple_header));
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
+  lto_emit_section_header ();
   header.main_size = ob->main_stream->total_size;
   lto_write_data (&header, sizeof header);
 
diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c
index b1084f4d3c5..42f03ac504e 100644
--- a/gcc/lto-streamer-out.c
+++ b/gcc/lto-streamer-out.c
@@ -1973,8 +1973,7 @@ produce_asm (struct output_block *ob, tree fn)
   memset (&header, 0, sizeof (struct lto_function_header));
 
   /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
+  lto_emit_section_header ();
 
   if (section_type == LTO_section_function_body)
     header.cfg_size = ob->cfg_stream->total_size;
@@ -2269,8 +2268,7 @@ lto_output_toplevel_asms (void)
   memset (&header, 0, sizeof (header));
 
   /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
+  lto_emit_section_header ();
 
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
@@ -2826,8 +2824,7 @@ lto_write_mode_table (void)
   memset (&header, 0, sizeof (header));
 
   /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
+  lto_emit_section_header ();
 
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
@@ -2899,8 +2896,7 @@ produce_asm_for_decls (void)
       lto_output_decl_state_streams (ob, fn_out_state);
     }
 
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
+  lto_emit_section_header ();
 
   /* Currently not used.  This field would allow us to preallocate
      the globals vector, so that it need not be resized as it is extended.  */
diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h
index d087cba9bf0..8fc1ac01356 100644
--- a/gcc/lto-streamer.h
+++ b/gcc/lto-streamer.h
@@ -378,18 +378,25 @@ public:
   unsigned int len;
 };
 
+enum lto_compression
+{
+  RAW,
+  ZLIB,
+  ZSTD
+};
 
 /* The is the first part of the record for a function or constructor
    in the .o file.  */
-struct lto_header
+struct lto_section_header
 {
   int16_t major_version;
-  int16_t minor_version;
+  int16_t minor_version : 14;
+  lto_compression compression : 2;
 };
 
 /* The is the first part of the record in an LTO file for many of the
    IPA passes.  */
-struct lto_simple_header : lto_header
+struct lto_simple_header
 {
   /* Size of main gimple body of function.  */
   int32_t main_size;
@@ -784,6 +791,7 @@ extern void lto_value_range_error (const char *,
 
 /* In lto-section-out.c  */
 extern void lto_begin_section (const char *, bool);
+extern void lto_emit_section_header (void);
 extern void lto_end_section (void);
 extern void lto_write_data (const void *, unsigned int);
 extern void lto_write_raw_data (const void *, unsigned int);
-- 
2.21.0

Re: [PATCH] Add .gnu.lto_.meta section.

[Richard Biener]

On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/21/19 2:57 PM, Jan Hubicka wrote:
> > This looks like good step (and please stream it in host independent
> > way). I suppose all these issues can be done one-by-one.
>
> So there's a working patch for that. However one will see following errors
> when using an older compiler or older LTO bytecode:
>
> $ gcc main9.o -flto
> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
>
> $ gcc main.o
> lto1: internal compiler error: compressed stream: data error

This is because of your change to bitfields or because with the old
scheme the header with the
version is compressed (is it?).  I'd simply avoid any layout changes
in the version check range.

> To be honest, I would prefer the new .gnu.lto_.meta section.
> Richi why is that so ugly?

Because it's a change in the wrong direction and doesn't solve the
issue we already
have (cannot determine if a section is compressed or not).  ELF section overhead
is quite big if you have lots of small functions.

Richard.

>
> Martin

Re: [PATCH] Add .gnu.lto_.meta section.

[Martin Liška]

On 6/24/19 2:02 PM, Richard Biener wrote:
> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/21/19 2:57 PM, Jan Hubicka wrote:
>>> This looks like good step (and please stream it in host independent
>>> way). I suppose all these issues can be done one-by-one.
>>
>> So there's a working patch for that. However one will see following errors
>> when using an older compiler or older LTO bytecode:
>>
>> $ gcc main9.o -flto
>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
>>
>> $ gcc main.o
>> lto1: internal compiler error: compressed stream: data error
> 
> This is because of your change to bitfields or because with the old
> scheme the header with the
> version is compressed (is it?).

Because currently also the header is compressed.

> I'd simply avoid any layout changes
> in the version check range.

Well, then we have to find out how to distinguish between compression algorithms.

> 
>> To be honest, I would prefer the new .gnu.lto_.meta section.
>> Richi why is that so ugly?
> 
> Because it's a change in the wrong direction and doesn't solve the
> issue we already
> have (cannot determine if a section is compressed or not).

That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
also backport changes to older compiler that can read it and print a proper error
message about LTO bytecode version mismatch. 

> ELF section overhead
> is quite big if you have lots of small functions.

My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
and remove the section header from all other LTO sections. That will save space
for all function sections.

Martin

> 
> Richard.
> 
>>
>> Martin

Re: [PATCH] Add .gnu.lto_.meta section.

[Richard Biener]

On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/24/19 2:02 PM, Richard Biener wrote:
> > On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/21/19 2:57 PM, Jan Hubicka wrote:
> >>> This looks like good step (and please stream it in host independent
> >>> way). I suppose all these issues can be done one-by-one.
> >>
> >> So there's a working patch for that. However one will see following errors
> >> when using an older compiler or older LTO bytecode:
> >>
> >> $ gcc main9.o -flto
> >> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
> >>
> >> $ gcc main.o
> >> lto1: internal compiler error: compressed stream: data error
> >
> > This is because of your change to bitfields or because with the old
> > scheme the header with the
> > version is compressed (is it?).
>
> Because currently also the header is compressed.

That was it, yeah :/  Stupid decisions in the past.

I guess we have to bite the bullet and do this kind of incompatible
change, accepting
the odd error message above.

> > I'd simply avoid any layout changes
> > in the version check range.
>
> Well, then we have to find out how to distinguish between compression algorithms.
>
> >
> >> To be honest, I would prefer the new .gnu.lto_.meta section.
> >> Richi why is that so ugly?
> >
> > Because it's a change in the wrong direction and doesn't solve the
> > issue we already
> > have (cannot determine if a section is compressed or not).
>
> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
> also backport changes to older compiler that can read it and print a proper error
> message about LTO bytecode version mismatch.

We can always backport changes, yes, but I don't see why we have to.

> > ELF section overhead
> > is quite big if you have lots of small functions.
>
> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
> and remove the section header from all other LTO sections. That will save space
> for all function sections.

But we want the header there to at least say if the section is
compressed or not.
The fact that we have so many ELF section means we have the redundant version
info everywhere.

We should have a single .gnu.lto_ section (and also get rid of those
__gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for
existence of a symbol is more expensive compared to existence
of a section).

Richard.

> Martin
>
> >
> > Richard.
> >
> >>
> >> Martin
>

Re: [PATCH] Add .gnu.lto_.meta section.

[Martin Liška]

On 6/24/19 2:44 PM, Richard Biener wrote:
> On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/24/19 2:02 PM, Richard Biener wrote:
>>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote:
>>>>> This looks like good step (and please stream it in host independent
>>>>> way). I suppose all these issues can be done one-by-one.
>>>>
>>>> So there's a working patch for that. However one will see following errors
>>>> when using an older compiler or older LTO bytecode:
>>>>
>>>> $ gcc main9.o -flto
>>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
>>>>
>>>> $ gcc main.o
>>>> lto1: internal compiler error: compressed stream: data error
>>>
>>> This is because of your change to bitfields or because with the old
>>> scheme the header with the
>>> version is compressed (is it?).
>>
>> Because currently also the header is compressed.
> 
> That was it, yeah :/  Stupid decisions in the past.
> 
> I guess we have to bite the bullet and do this kind of incompatible
> change, accepting
> the odd error message above.
> 
>>> I'd simply avoid any layout changes
>>> in the version check range.
>>
>> Well, then we have to find out how to distinguish between compression algorithms.
>>
>>>
>>>> To be honest, I would prefer the new .gnu.lto_.meta section.
>>>> Richi why is that so ugly?
>>>
>>> Because it's a change in the wrong direction and doesn't solve the
>>> issue we already
>>> have (cannot determine if a section is compressed or not).
>>
>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
>> also backport changes to older compiler that can read it and print a proper error
>> message about LTO bytecode version mismatch.
> 
> We can always backport changes, yes, but I don't see why we have to.

I'm fine with the backward compatibility break. But we should also consider lto-plugin.c
that is parsing following 2 sections:

    91  #define LTO_SECTION_PREFIX      ".gnu.lto_.symtab"
    92  #define LTO_SECTION_PREFIX_LEN  (sizeof (LTO_SECTION_PREFIX) - 1)
    93  #define OFFLOAD_SECTION         ".gnu.offload_lto_.opts"
    94  #define OFFLOAD_SECTION_LEN     (sizeof (OFFLOAD_SECTION) - 1)

> 
>>> ELF section overhead
>>> is quite big if you have lots of small functions.
>>
>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
>> and remove the section header from all other LTO sections. That will save space
>> for all function sections.
> 
> But we want the header there to at least say if the section is
> compressed or not.
> The fact that we have so many ELF section means we have the redundant version
> info everywhere.
> 
> We should have a single .gnu.lto_ section (and also get rid of those
> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for
> existence of a symbol is more expensive compared to existence
> of a section).

I like removal of the 2 aforementioned sections. To be honest I would recommend to
add a new .gnu.lto_.meta section. We can use it instead of __gnu_lto_v1 and we can
have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all

  LTO_section_function_body,
  LTO_section_static_initializer

sections into a single one. That will require an index that will have to be created. I can discuss
that with Honza as he suggested using something smarter than function names.

Thoughts?
Martin

> 
> Richard.
> 
>> Martin
>>
>>>
>>> Richard.
>>>
>>>>
>>>> Martin
>>

Re: [PATCH] Add .gnu.lto_.meta section.

[Iain Sandoe]

> On 24 Jun 2019, at 14:31, Martin Liška <mliska@suse.cz> wrote:
> 
> On 6/24/19 2:44 PM, Richard Biener wrote:
>> On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote:
>>> 
>>> On 6/24/19 2:02 PM, Richard Biener wrote:
>>>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
>>>>> 
>>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote:
>>>>>> This looks like good step (and please stream it in host independent
>>>>>> way). I suppose all these issues can be done one-by-one.
>>>>> 
>>>>> So there's a working patch for that. However one will see following errors
>>>>> when using an older compiler or older LTO bytecode:
>>>>> 
>>>>> $ gcc main9.o -flto
>>>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
>>>>> 
>>>>> $ gcc main.o
>>>>> lto1: internal compiler error: compressed stream: data error
>>>> 
>>>> This is because of your change to bitfields or because with the old
>>>> scheme the header with the
>>>> version is compressed (is it?).
>>> 
>>> Because currently also the header is compressed.
>> 
>> That was it, yeah :/  Stupid decisions in the past.
>> 
>> I guess we have to bite the bullet and do this kind of incompatible
>> change, accepting
>> the odd error message above.
>> 
>>>> I'd simply avoid any layout changes
>>>> in the version check range.
>>> 
>>> Well, then we have to find out how to distinguish between compression algorithms.
>>> 
>>>> 
>>>>> To be honest, I would prefer the new .gnu.lto_.meta section.
>>>>> Richi why is that so ugly?
>>>> 
>>>> Because it's a change in the wrong direction and doesn't solve the
>>>> issue we already
>>>> have (cannot determine if a section is compressed or not).
>>> 
>>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
>>> also backport changes to older compiler that can read it and print a proper error
>>> message about LTO bytecode version mismatch.
>> 
>> We can always backport changes, yes, but I don't see why we have to.
> 
> I'm fine with the backward compatibility break. But we should also consider lto-plugin.c
> that is parsing following 2 sections:
> 
>    91  #define LTO_SECTION_PREFIX      ".gnu.lto_.symtab"
>    92  #define LTO_SECTION_PREFIX_LEN  (sizeof (LTO_SECTION_PREFIX) - 1)
>    93  #define OFFLOAD_SECTION         ".gnu.offload_lto_.opts"
>    94  #define OFFLOAD_SECTION_LEN     (sizeof (OFFLOAD_SECTION) - 1)
> 
>> 
>>>> ELF section overhead
>>>> is quite big if you have lots of small functions.
>>> 
>>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
>>> and remove the section header from all other LTO sections. That will save space
>>> for all function sections.
>> 
>> But we want the header there to at least say if the section is
>> compressed or not.
>> The fact that we have so many ELF section means we have the redundant version
>> info everywhere.
>> 
>> We should have a single .gnu.lto_ section (and also get rid of those
>> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for
>> existence of a symbol is more expensive compared to existence
>> of a section).
> 
> I like removal of the 2 aforementioned sections. To be honest I would recommend to
> add a new .gnu.lto_.meta section. We can use it instead of __gnu_lto_v1 and we can
> have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all
> 
>  LTO_section_function_body,
>  LTO_section_static_initializer
> 
> sections into a single one. That will require an index that will have to be created. I can discuss
> that with Honza as he suggested using something smarter than function names.

I already implemented a scheme (using three sections: INDEX, NAMES, PAYLOAD)  for Mach-O -
 since it doesn’t have  unlimited section count - it works - and hardly rocket science ;) 

- if one were to import the tabular portion of that at the start of a section and then the variable portion
 as a trailer … it could all be a single section.

iain

> Martin
> 
>> 
>> Richard.
>> 
>>> Martin
>>> 
>>>> 
>>>> Richard.
>>>> 
>>>>> 
>>>>> Martin

Re: [PATCH] Add .gnu.lto_.meta section.

[Richard Biener]

On Mon, Jun 24, 2019 at 3:31 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/24/19 2:44 PM, Richard Biener wrote:
> > On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/24/19 2:02 PM, Richard Biener wrote:
> >>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>
> >>>> On 6/21/19 2:57 PM, Jan Hubicka wrote:
> >>>>> This looks like good step (and please stream it in host independent
> >>>>> way). I suppose all these issues can be done one-by-one.
> >>>>
> >>>> So there's a working patch for that. However one will see following errors
> >>>> when using an older compiler or older LTO bytecode:
> >>>>
> >>>> $ gcc main9.o -flto
> >>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
> >>>>
> >>>> $ gcc main.o
> >>>> lto1: internal compiler error: compressed stream: data error
> >>>
> >>> This is because of your change to bitfields or because with the old
> >>> scheme the header with the
> >>> version is compressed (is it?).
> >>
> >> Because currently also the header is compressed.
> >
> > That was it, yeah :/  Stupid decisions in the past.
> >
> > I guess we have to bite the bullet and do this kind of incompatible
> > change, accepting
> > the odd error message above.
> >
> >>> I'd simply avoid any layout changes
> >>> in the version check range.
> >>
> >> Well, then we have to find out how to distinguish between compression algorithms.
> >>
> >>>
> >>>> To be honest, I would prefer the new .gnu.lto_.meta section.
> >>>> Richi why is that so ugly?
> >>>
> >>> Because it's a change in the wrong direction and doesn't solve the
> >>> issue we already
> >>> have (cannot determine if a section is compressed or not).
> >>
> >> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
> >> also backport changes to older compiler that can read it and print a proper error
> >> message about LTO bytecode version mismatch.
> >
> > We can always backport changes, yes, but I don't see why we have to.
>
> I'm fine with the backward compatibility break. But we should also consider lto-plugin.c
> that is parsing following 2 sections:
>
>     91  #define LTO_SECTION_PREFIX      ".gnu.lto_.symtab"
>     92  #define LTO_SECTION_PREFIX_LEN  (sizeof (LTO_SECTION_PREFIX) - 1)
>     93  #define OFFLOAD_SECTION         ".gnu.offload_lto_.opts"
>     94  #define OFFLOAD_SECTION_LEN     (sizeof (OFFLOAD_SECTION) - 1)

Yeah, I know.  And BFD and gold hard-coded those __gnu_lto_{v1,slim} symbols...

> >
> >>> ELF section overhead
> >>> is quite big if you have lots of small functions.
> >>
> >> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
> >> and remove the section header from all other LTO sections. That will save space
> >> for all function sections.
> >
> > But we want the header there to at least say if the section is
> > compressed or not.
> > The fact that we have so many ELF section means we have the redundant version
> > info everywhere.
> >
> > We should have a single .gnu.lto_ section (and also get rid of those
> > __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for
> > existence of a symbol is more expensive compared to existence
> > of a section).
>
> I like removal of the 2 aforementioned sections. To be honest I would recommend to
> add a new .gnu.lto_.meta section.

Why .meta?  Why not just .gnu.lto_?

> We can use it instead of __gnu_lto_v1 and we can
> have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all
>
>   LTO_section_function_body,
>   LTO_section_static_initializer
>
> sections into a single one. That will require an index that will have to be created. I can discuss
> that with Honza as he suggested using something smarter than function names.

I think the index belongs to symtab?

Let's properly do it if we want to change it.  Removing of
__gnu_lto_v1/slim is going to be
the most intrusive change btw. and orthogonal to the section changes.

Richard.

>
> Thoughts?
> Martin
>
> >
> > Richard.
> >
> >> Martin
> >>
> >>>
> >>> Richard.
> >>>
> >>>>
> >>>> Martin
> >>
>

Re: [PATCH] Add .gnu.lto_.meta section.

[Martin Liška]

On 6/24/19 8:05 PM, Richard Biener wrote:
> On Mon, Jun 24, 2019 at 3:31 PM Martin Liška <mliska@suse.cz> wrote:
>>
>> On 6/24/19 2:44 PM, Richard Biener wrote:
>>> On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote:
>>>>
>>>> On 6/24/19 2:02 PM, Richard Biener wrote:
>>>>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>>
>>>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote:
>>>>>>> This looks like good step (and please stream it in host independent
>>>>>>> way). I suppose all these issues can be done one-by-one.
>>>>>>
>>>>>> So there's a working patch for that. However one will see following errors
>>>>>> when using an older compiler or older LTO bytecode:
>>>>>>
>>>>>> $ gcc main9.o -flto
>>>>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
>>>>>>
>>>>>> $ gcc main.o
>>>>>> lto1: internal compiler error: compressed stream: data error
>>>>>
>>>>> This is because of your change to bitfields or because with the old
>>>>> scheme the header with the
>>>>> version is compressed (is it?).
>>>>
>>>> Because currently also the header is compressed.
>>>
>>> That was it, yeah :/  Stupid decisions in the past.
>>>
>>> I guess we have to bite the bullet and do this kind of incompatible
>>> change, accepting
>>> the odd error message above.
>>>
>>>>> I'd simply avoid any layout changes
>>>>> in the version check range.
>>>>
>>>> Well, then we have to find out how to distinguish between compression algorithms.
>>>>
>>>>>
>>>>>> To be honest, I would prefer the new .gnu.lto_.meta section.
>>>>>> Richi why is that so ugly?
>>>>>
>>>>> Because it's a change in the wrong direction and doesn't solve the
>>>>> issue we already
>>>>> have (cannot determine if a section is compressed or not).
>>>>
>>>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
>>>> also backport changes to older compiler that can read it and print a proper error
>>>> message about LTO bytecode version mismatch.
>>>
>>> We can always backport changes, yes, but I don't see why we have to.
>>
>> I'm fine with the backward compatibility break. But we should also consider lto-plugin.c
>> that is parsing following 2 sections:
>>
>>     91  #define LTO_SECTION_PREFIX      ".gnu.lto_.symtab"
>>     92  #define LTO_SECTION_PREFIX_LEN  (sizeof (LTO_SECTION_PREFIX) - 1)
>>     93  #define OFFLOAD_SECTION         ".gnu.offload_lto_.opts"
>>     94  #define OFFLOAD_SECTION_LEN     (sizeof (OFFLOAD_SECTION) - 1)
> 
> Yeah, I know.  And BFD and gold hard-coded those __gnu_lto_{v1,slim} symbols...

Yep, they do, 'nm' is also using that.

> 
>>>
>>>>> ELF section overhead
>>>>> is quite big if you have lots of small functions.
>>>>
>>>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
>>>> and remove the section header from all other LTO sections. That will save space
>>>> for all function sections.
>>>
>>> But we want the header there to at least say if the section is
>>> compressed or not.
>>> The fact that we have so many ELF section means we have the redundant version
>>> info everywhere.
>>>
>>> We should have a single .gnu.lto_ section (and also get rid of those
>>> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for
>>> existence of a symbol is more expensive compared to existence
>>> of a section).
>>
>> I like removal of the 2 aforementioned sections. To be honest I would recommend to
>> add a new .gnu.lto_.meta section.
> 
> Why .meta?  Why not just .gnu.lto_?

Works for me.

> 
>> We can use it instead of __gnu_lto_v1 and we can
>> have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all
>>
>>   LTO_section_function_body,
>>   LTO_section_static_initializer
>>
>> sections into a single one. That will require an index that will have to be created. I can discuss
>> that with Honza as he suggested using something smarter than function names.
> 
> I think the index belongs to symtab?
> 
> Let's properly do it if we want to change it.  Removing of
> __gnu_lto_v1/slim is going to be
> the most intrusive change btw. and orthogonal to the section changes.

I'm fine with a proper change. So do I understand that correctly that:
- we'll come up with .gnu.lto_ section that will be used by bfd, gold and nm
  to detect LTO objects
- for some time, we'll keep __gnu_lto_v1 and __gnu_lto_slim for backward
  compatibility with older binutils tool
- in couple of year, the legacy support will be removed

?

Martin

> 
> Richard.
> 
>>
>> Thoughts?
>> Martin
>>
>>>
>>> Richard.
>>>
>>>> Martin
>>>>
>>>>>
>>>>> Richard.
>>>>>
>>>>>>
>>>>>> Martin
>>>>
>>

Re: [PATCH] Add .gnu.lto_.meta section.

[Richard Biener]

On Tue, Jun 25, 2019 at 10:14 AM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/24/19 8:05 PM, Richard Biener wrote:
> > On Mon, Jun 24, 2019 at 3:31 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> On 6/24/19 2:44 PM, Richard Biener wrote:
> >>> On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>
> >>>> On 6/24/19 2:02 PM, Richard Biener wrote:
> >>>>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>>
> >>>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote:
> >>>>>>> This looks like good step (and please stream it in host independent
> >>>>>>> way). I suppose all these issues can be done one-by-one.
> >>>>>>
> >>>>>> So there's a working patch for that. However one will see following errors
> >>>>>> when using an older compiler or older LTO bytecode:
> >>>>>>
> >>>>>> $ gcc main9.o -flto
> >>>>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0
> >>>>>>
> >>>>>> $ gcc main.o
> >>>>>> lto1: internal compiler error: compressed stream: data error
> >>>>>
> >>>>> This is because of your change to bitfields or because with the old
> >>>>> scheme the header with the
> >>>>> version is compressed (is it?).
> >>>>
> >>>> Because currently also the header is compressed.
> >>>
> >>> That was it, yeah :/  Stupid decisions in the past.
> >>>
> >>> I guess we have to bite the bullet and do this kind of incompatible
> >>> change, accepting
> >>> the odd error message above.
> >>>
> >>>>> I'd simply avoid any layout changes
> >>>>> in the version check range.
> >>>>
> >>>> Well, then we have to find out how to distinguish between compression algorithms.
> >>>>
> >>>>>
> >>>>>> To be honest, I would prefer the new .gnu.lto_.meta section.
> >>>>>> Richi why is that so ugly?
> >>>>>
> >>>>> Because it's a change in the wrong direction and doesn't solve the
> >>>>> issue we already
> >>>>> have (cannot determine if a section is compressed or not).
> >>>>
> >>>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can
> >>>> also backport changes to older compiler that can read it and print a proper error
> >>>> message about LTO bytecode version mismatch.
> >>>
> >>> We can always backport changes, yes, but I don't see why we have to.
> >>
> >> I'm fine with the backward compatibility break. But we should also consider lto-plugin.c
> >> that is parsing following 2 sections:
> >>
> >>     91  #define LTO_SECTION_PREFIX      ".gnu.lto_.symtab"
> >>     92  #define LTO_SECTION_PREFIX_LEN  (sizeof (LTO_SECTION_PREFIX) - 1)
> >>     93  #define OFFLOAD_SECTION         ".gnu.offload_lto_.opts"
> >>     94  #define OFFLOAD_SECTION_LEN     (sizeof (OFFLOAD_SECTION) - 1)
> >
> > Yeah, I know.  And BFD and gold hard-coded those __gnu_lto_{v1,slim} symbols...
>
> Yep, they do, 'nm' is also using that.
>
> >
> >>>
> >>>>> ELF section overhead
> >>>>> is quite big if you have lots of small functions.
> >>>>
> >>>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section
> >>>> and remove the section header from all other LTO sections. That will save space
> >>>> for all function sections.
> >>>
> >>> But we want the header there to at least say if the section is
> >>> compressed or not.
> >>> The fact that we have so many ELF section means we have the redundant version
> >>> info everywhere.
> >>>
> >>> We should have a single .gnu.lto_ section (and also get rid of those
> >>> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for
> >>> existence of a symbol is more expensive compared to existence
> >>> of a section).
> >>
> >> I like removal of the 2 aforementioned sections. To be honest I would recommend to
> >> add a new .gnu.lto_.meta section.
> >
> > Why .meta?  Why not just .gnu.lto_?
>
> Works for me.
>
> >
> >> We can use it instead of __gnu_lto_v1 and we can
> >> have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all
> >>
> >>   LTO_section_function_body,
> >>   LTO_section_static_initializer
> >>
> >> sections into a single one. That will require an index that will have to be created. I can discuss
> >> that with Honza as he suggested using something smarter than function names.
> >
> > I think the index belongs to symtab?
> >
> > Let's properly do it if we want to change it.  Removing of
> > __gnu_lto_v1/slim is going to be
> > the most intrusive change btw. and orthogonal to the section changes.
>
> I'm fine with a proper change. So do I understand that correctly that:
> - we'll come up with .gnu.lto_ section that will be used by bfd, gold and nm
>   to detect LTO objects
> - for some time, we'll keep __gnu_lto_v1 and __gnu_lto_slim for backward
>   compatibility with older binutils tool
> - in couple of year, the legacy support will be removed

Yep.

Richard.

> ?
>
> Martin
>
> >
> > Richard.
> >
> >>
> >> Thoughts?
> >> Martin
> >>
> >>>
> >>> Richard.
> >>>
> >>>> Martin
> >>>>
> >>>>>
> >>>>> Richard.
> >>>>>
> >>>>>>
> >>>>>> Martin
> >>>>
> >>
>

[PATCH] Add .gnu.lto_.lto section.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

Hi.

So this is a patch candidate for the .gnu.lto_.lto ELF section.
As mentioned, the section contains information about bytecode version,
compression algorithm used and slim LTO object flag.

One minor issues is that I need to append random suffix to the section:
  [ 6] .gnu.lto_.lto.36e74acfb145b04b PROGBITS        0000000000000000 000086 000008 00   E  0   0  1

That's because of in WPA we load more object files and the hash is used for mapping.

Is the idea of the patch right?
Martin

[-- Attachment #2: 0001-Add-.gnu.lto_.lto-section.patch --]
[-- Type: text/x-patch, Size: 7558 bytes --]

From ec550463df26855982b7b70393766cc8e7c6cc8e Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Fri, 21 Jun 2019 12:14:04 +0200
Subject: [PATCH] Add .gnu.lto_.lto section.

---
 gcc/lto-section-in.c   |  9 +++------
 gcc/lto-section-out.c  |  2 --
 gcc/lto-streamer-out.c | 40 +++++++++++++++++++++++++---------------
 gcc/lto-streamer.h     | 25 +++++++++++++++++++++----
 gcc/lto/lto-common.c   | 15 +++++++++++++++
 5 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c
index 4cfc0cad4be..4e7d1181f23 100644
--- a/gcc/lto-section-in.c
+++ b/gcc/lto-section-in.c
@@ -52,10 +52,10 @@ const char *lto_section_name[LTO_N_SECTION_TYPES] =
   "icf",
   "offload_table",
   "mode_table",
-  "hsa"
+  "hsa",
+  "lto"
 };
 
-
 /* Hooks so that the ipa passes can call into the lto front end to get
    sections.  */
 
@@ -146,7 +146,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
   /* WPA->ltrans streams are not compressed with exception of function bodies
      and variable initializers that has been verbatim copied from earlier
      compilations.  */
-  if (!flag_ltrans || decompress)
+  if ((!flag_ltrans || decompress) && section_type != LTO_section_lto)
     {
       /* Create a mapping header containing the underlying data and length,
 	 and prepend this to the uncompression buffer.  The uncompressed data
@@ -167,9 +167,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
       data = buffer.data + header_length;
     }
 
-  lto_check_version (((const lto_header *)data)->major_version,
-		     ((const lto_header *)data)->minor_version,
-		     file_data->file_name);
   return data;
 }
 
diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c
index c91e58f0465..7ae102164ef 100644
--- a/gcc/lto-section-out.c
+++ b/gcc/lto-section-out.c
@@ -285,8 +285,6 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob)
   /* Write the header which says how to decode the pieces of the
      t.  */
   memset (&header, 0, sizeof (struct lto_simple_header));
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
   header.main_size = ob->main_stream->total_size;
   lto_write_data (&header, sizeof header);
 
diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c
index dc68429303c..7dee770aa11 100644
--- a/gcc/lto-streamer-out.c
+++ b/gcc/lto-streamer-out.c
@@ -1974,10 +1974,6 @@ produce_asm (struct output_block *ob, tree fn)
   /* The entire header is stream computed here.  */
   memset (&header, 0, sizeof (struct lto_function_header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   if (section_type == LTO_section_function_body)
     header.cfg_size = ob->cfg_stream->total_size;
   header.main_size = ob->main_stream->total_size;
@@ -2270,10 +2266,6 @@ lto_output_toplevel_asms (void)
   /* The entire header stream is computed here.  */
   memset (&header, 0, sizeof (header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
   lto_write_data (&header, sizeof header);
@@ -2390,6 +2382,29 @@ prune_offload_funcs (void)
     DECL_PRESERVE_P (fn_decl) = 1;
 }
 
+/* Produce LTO section that contains global information
+   about LTO bytecode.  */
+
+static void
+produce_lto_section ()
+{
+  /* Stream LTO meta section.  */
+  output_block *ob = create_output_block (LTO_section_lto);
+
+  char * section_name = lto_get_section_name (LTO_section_lto, NULL, NULL);
+  lto_begin_section (section_name, false);
+  free (section_name);
+
+  lto_compression compression = ZLIB;
+
+  bool slim_object = flag_generate_lto && !flag_fat_lto_objects;
+  lto_section s
+    = { LTO_major_version, LTO_minor_version, slim_object, compression, 0 };
+  lto_write_data (&s, sizeof s);
+  lto_end_section ();
+  destroy_output_block (ob);
+}
+
 /* Main entry point from the pass manager.  */
 
 void
@@ -2412,6 +2427,8 @@ lto_output (void)
   /* Initialize the streamer.  */
   lto_streamer_init ();
 
+  produce_lto_section ();
+
   n_nodes = lto_symtab_encoder_size (encoder);
   /* Process only the functions with bodies.  */
   for (i = 0; i < n_nodes; i++)
@@ -2827,10 +2844,6 @@ lto_write_mode_table (void)
   struct lto_simple_header_with_strings header;
   memset (&header, 0, sizeof (header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
   lto_write_data (&header, sizeof header);
@@ -2901,9 +2914,6 @@ produce_asm_for_decls (void)
       lto_output_decl_state_streams (ob, fn_out_state);
     }
 
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   /* Currently not used.  This field would allow us to preallocate
      the globals vector, so that it need not be resized as it is extended.  */
   header.num_nodes = -1;
diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h
index 3b1e07b4bfc..2f481aebeb2 100644
--- a/gcc/lto-streamer.h
+++ b/gcc/lto-streamer.h
@@ -234,6 +234,7 @@ enum lto_section_type
   LTO_section_offload_table,
   LTO_section_mode_table,
   LTO_section_ipa_hsa,
+  LTO_section_lto,
   LTO_N_SECTION_TYPES		/* Must be last.  */
 };
 
@@ -378,18 +379,31 @@ public:
   unsigned int len;
 };
 
+/* Compression algorithm used for compression of LTO bytecode.  */
 
-/* The is the first part of the record for a function or constructor
-   in the .o file.  */
-struct lto_header
+enum lto_compression
+{
+  ZLIB,
+  ZSTD
+};
+
+/* Structure that represents LTO ELF section with information
+   about the format.  */
+
+struct lto_section
 {
   int16_t major_version;
   int16_t minor_version;
+  unsigned char slim_object: 1;
+  lto_compression compression: 4;
+  int32_t reserved0: 27;
 };
 
+STATIC_ASSERT (sizeof (lto_section) == 8);
+
 /* The is the first part of the record in an LTO file for many of the
    IPA passes.  */
-struct lto_simple_header : lto_header
+struct lto_simple_header
 {
   /* Size of main gimple body of function.  */
   int32_t main_size;
@@ -589,6 +603,9 @@ struct GTY(()) lto_file_decl_data
 
   /* Mode translation table.  */
   const unsigned char *mode_table;
+
+  /* Read LTO section. */
+  lto_section lto_section_header;
 };
 
 typedef struct lto_file_decl_data *lto_file_decl_data_ptr;
diff --git a/gcc/lto/lto-common.c b/gcc/lto/lto-common.c
index 0d38ee6b57b..1cfbda0f554 100644
--- a/gcc/lto/lto-common.c
+++ b/gcc/lto/lto-common.c
@@ -2060,6 +2060,21 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file)
 #else
   file_data->mode_table = lto_mode_identity_table;
 #endif
+
+  /* Read and verify LTO section.  */
+  data = lto_get_section_data (file_data, LTO_section_lto, NULL, &len, false);
+  if (data == NULL)
+    {
+      fatal_error (input_location, "bytecode stream in file %qs generated "
+		   "with GCC compiler older than 10.0", file_data->file_name);
+      return;
+    }
+
+  file_data->lto_section_header = *(const lto_section *)data;
+  lto_check_version (file_data->lto_section_header.major_version,
+		     file_data->lto_section_header.minor_version,
+		     file_data->file_name);
+
   data = lto_get_section_data (file_data, LTO_section_decls, NULL, &len);
   if (data == NULL)
     {
-- 
2.21.0

Re: [PATCH] Add .gnu.lto_.lto section.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 122 bytes --]

Hi.

Ok, so there's a version with added ChangeLog that survives regression tests.

Ready to be installed?
Thanks,
Martin

[-- Attachment #2: 0001-Add-.gnu.lto_.lto-section.patch --]
[-- Type: text/x-patch, Size: 8463 bytes --]

From e6745583dc4b7f5543878c0a25498e818531f73e Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Fri, 21 Jun 2019 12:14:04 +0200
Subject: [PATCH 1/2] Add .gnu.lto_.lto section.

gcc/ChangeLog:

2019-07-01  Martin Liska  <mliska@suse.cz>

	* lto-section-in.c (lto_get_section_data): Add "lto" section.
	* lto-section-out.c (lto_destroy_simple_output_block): Never
	compress LTO_section_lto section.
	* lto-streamer-out.c (produce_asm): Do not set major_version
	and minor_version.
	(lto_output_toplevel_asms): Likewise.
	(produce_lto_section): New function.
	(lto_output): Call produce_lto_section.
	(lto_write_mode_table): Do not set major_version and
	minor_version.
	(produce_asm_for_decls): Likewise.
	* lto-streamer.h (enum lto_section_type): Add LTO_section_lto
	type.
	(struct lto_header): Remove.
	(struct lto_section): New struct.
	(struct lto_simple_header): Do not inherit from lto_header.
	(struct lto_file_decl_data): Add lto_section_header field.

gcc/lto/ChangeLog:

2019-07-01  Martin Liska  <mliska@suse.cz>

	* lto-common.c: Read LTO section and verify header.
---
 gcc/lto-section-in.c   |  9 +++------
 gcc/lto-section-out.c  |  2 --
 gcc/lto-streamer-out.c | 40 +++++++++++++++++++++++++---------------
 gcc/lto-streamer.h     | 25 +++++++++++++++++++++----
 gcc/lto/lto-common.c   | 15 +++++++++++++++
 5 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c
index 4cfc0cad4be..4e7d1181f23 100644
--- a/gcc/lto-section-in.c
+++ b/gcc/lto-section-in.c
@@ -52,10 +52,10 @@ const char *lto_section_name[LTO_N_SECTION_TYPES] =
   "icf",
   "offload_table",
   "mode_table",
-  "hsa"
+  "hsa",
+  "lto"
 };
 
-
 /* Hooks so that the ipa passes can call into the lto front end to get
    sections.  */
 
@@ -146,7 +146,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
   /* WPA->ltrans streams are not compressed with exception of function bodies
      and variable initializers that has been verbatim copied from earlier
      compilations.  */
-  if (!flag_ltrans || decompress)
+  if ((!flag_ltrans || decompress) && section_type != LTO_section_lto)
     {
       /* Create a mapping header containing the underlying data and length,
 	 and prepend this to the uncompression buffer.  The uncompressed data
@@ -167,9 +167,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
       data = buffer.data + header_length;
     }
 
-  lto_check_version (((const lto_header *)data)->major_version,
-		     ((const lto_header *)data)->minor_version,
-		     file_data->file_name);
   return data;
 }
 
diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c
index c91e58f0465..7ae102164ef 100644
--- a/gcc/lto-section-out.c
+++ b/gcc/lto-section-out.c
@@ -285,8 +285,6 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob)
   /* Write the header which says how to decode the pieces of the
      t.  */
   memset (&header, 0, sizeof (struct lto_simple_header));
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
   header.main_size = ob->main_stream->total_size;
   lto_write_data (&header, sizeof header);
 
diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c
index dc68429303c..7dee770aa11 100644
--- a/gcc/lto-streamer-out.c
+++ b/gcc/lto-streamer-out.c
@@ -1974,10 +1974,6 @@ produce_asm (struct output_block *ob, tree fn)
   /* The entire header is stream computed here.  */
   memset (&header, 0, sizeof (struct lto_function_header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   if (section_type == LTO_section_function_body)
     header.cfg_size = ob->cfg_stream->total_size;
   header.main_size = ob->main_stream->total_size;
@@ -2270,10 +2266,6 @@ lto_output_toplevel_asms (void)
   /* The entire header stream is computed here.  */
   memset (&header, 0, sizeof (header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
   lto_write_data (&header, sizeof header);
@@ -2390,6 +2382,29 @@ prune_offload_funcs (void)
     DECL_PRESERVE_P (fn_decl) = 1;
 }
 
+/* Produce LTO section that contains global information
+   about LTO bytecode.  */
+
+static void
+produce_lto_section ()
+{
+  /* Stream LTO meta section.  */
+  output_block *ob = create_output_block (LTO_section_lto);
+
+  char * section_name = lto_get_section_name (LTO_section_lto, NULL, NULL);
+  lto_begin_section (section_name, false);
+  free (section_name);
+
+  lto_compression compression = ZLIB;
+
+  bool slim_object = flag_generate_lto && !flag_fat_lto_objects;
+  lto_section s
+    = { LTO_major_version, LTO_minor_version, slim_object, compression, 0 };
+  lto_write_data (&s, sizeof s);
+  lto_end_section ();
+  destroy_output_block (ob);
+}
+
 /* Main entry point from the pass manager.  */
 
 void
@@ -2412,6 +2427,8 @@ lto_output (void)
   /* Initialize the streamer.  */
   lto_streamer_init ();
 
+  produce_lto_section ();
+
   n_nodes = lto_symtab_encoder_size (encoder);
   /* Process only the functions with bodies.  */
   for (i = 0; i < n_nodes; i++)
@@ -2827,10 +2844,6 @@ lto_write_mode_table (void)
   struct lto_simple_header_with_strings header;
   memset (&header, 0, sizeof (header));
 
-  /* Write the header.  */
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   header.main_size = ob->main_stream->total_size;
   header.string_size = ob->string_stream->total_size;
   lto_write_data (&header, sizeof header);
@@ -2901,9 +2914,6 @@ produce_asm_for_decls (void)
       lto_output_decl_state_streams (ob, fn_out_state);
     }
 
-  header.major_version = LTO_major_version;
-  header.minor_version = LTO_minor_version;
-
   /* Currently not used.  This field would allow us to preallocate
      the globals vector, so that it need not be resized as it is extended.  */
   header.num_nodes = -1;
diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h
index 3b1e07b4bfc..f1593d64c71 100644
--- a/gcc/lto-streamer.h
+++ b/gcc/lto-streamer.h
@@ -234,6 +234,7 @@ enum lto_section_type
   LTO_section_offload_table,
   LTO_section_mode_table,
   LTO_section_ipa_hsa,
+  LTO_section_lto,
   LTO_N_SECTION_TYPES		/* Must be last.  */
 };
 
@@ -378,18 +379,31 @@ public:
   unsigned int len;
 };
 
+/* Compression algorithm used for compression of LTO bytecode.  */
 
-/* The is the first part of the record for a function or constructor
-   in the .o file.  */
-struct lto_header
+enum lto_compression
+{
+  ZLIB,
+  ZSTD
+};
+
+/* Structure that represents LTO ELF section with information
+   about the format.  */
+
+struct lto_section
 {
   int16_t major_version;
   int16_t minor_version;
+  unsigned char slim_object: 1;
+  lto_compression compression: 4;
+  int32_t reserved0: 27;
 };
 
+STATIC_ASSERT (sizeof (lto_section) == 8);
+
 /* The is the first part of the record in an LTO file for many of the
    IPA passes.  */
-struct lto_simple_header : lto_header
+struct lto_simple_header
 {
   /* Size of main gimple body of function.  */
   int32_t main_size;
@@ -589,6 +603,9 @@ struct GTY(()) lto_file_decl_data
 
   /* Mode translation table.  */
   const unsigned char *mode_table;
+
+  /* Read LTO section.  */
+  lto_section lto_section_header;
 };
 
 typedef struct lto_file_decl_data *lto_file_decl_data_ptr;
diff --git a/gcc/lto/lto-common.c b/gcc/lto/lto-common.c
index 0d38ee6b57b..1cfbda0f554 100644
--- a/gcc/lto/lto-common.c
+++ b/gcc/lto/lto-common.c
@@ -2060,6 +2060,21 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file)
 #else
   file_data->mode_table = lto_mode_identity_table;
 #endif
+
+  /* Read and verify LTO section.  */
+  data = lto_get_section_data (file_data, LTO_section_lto, NULL, &len, false);
+  if (data == NULL)
+    {
+      fatal_error (input_location, "bytecode stream in file %qs generated "
+		   "with GCC compiler older than 10.0", file_data->file_name);
+      return;
+    }
+
+  file_data->lto_section_header = *(const lto_section *)data;
+  lto_check_version (file_data->lto_section_header.major_version,
+		     file_data->lto_section_header.minor_version,
+		     file_data->file_name);
+
   data = lto_get_section_data (file_data, LTO_section_decls, NULL, &len);
   if (data == NULL)
     {
-- 
2.22.0

[PATCH 2/2] Add zstd support for LTO bytecode compression.

[Martin Liška]

[-- Attachment #1: Type: text/plain, Size: 206 bytes --]

Hi.

This is updated version of the zstd patch that should handle all what Joseph
pointed out.

Patch can bootstrap on x86_64-linux-gnu and survives regression tests.

Ready to be installed?
Thanks,
Martin

[-- Attachment #2: 0002-Add-zstd-support-for-LTO-bytecode-compression.patch --]
[-- Type: text/x-patch, Size: 20134 bytes --]

From 5d2006c9c4d481f4083d5a591327ee64847b0bf7 Mon Sep 17 00:00:00 2001
From: Martin Liska <mliska@suse.cz>
Date: Thu, 20 Jun 2019 10:08:17 +0200
Subject: [PATCH 2/2] Add zstd support for LTO bytecode compression.

gcc/ChangeLog:

2019-07-01  Martin Liska  <mliska@suse.cz>

	* Makefile.in: Define ZSTD_LIB.
	* common.opt: Adjust compression level
	to support also zstd levels.
	* config.in: Regenerate.
	* configure: Likewise.
	* configure.ac: Add --with-zstd and --with-zstd-include options
	and detect ZSTD.
	* doc/install.texi: Mention zstd dependency.
	* gcc.c: Print supported LTO compression algorithms.
	* lto-compress.c (lto_normalized_zstd_level): Likewise.
	(lto_compression_zstd): Likewise.
	(lto_uncompression_zstd): Likewise.
	(lto_end_compression): Dispatch in between zlib and zstd.
	(lto_compression_zlib): Mark with ATTRIBUTE_UNUSED.
	(lto_uncompression_zlib): Make it static.
	* lto-compress.h (lto_end_uncompression): Fix GNU coding style.
	* lto-section-in.c (lto_get_section_data): Pass info
	about used compression.
	* lto-streamer-out.c: By default use zstd when possible.
	* timevar.def (TV_IPA_LTO_DECOMPRESS): Rename to decompression
	(TV_IPA_LTO_COMPRESS): Likewise for compression.
---
 gcc/Makefile.in        |   4 +-
 gcc/common.opt         |   4 +-
 gcc/config.in          |   6 ++
 gcc/configure          | 163 ++++++++++++++++++++++++++++++++++++++++-
 gcc/configure.ac       |  66 +++++++++++++++++
 gcc/doc/install.texi   |   6 ++
 gcc/gcc.c              |   5 ++
 gcc/lto-compress.c     | 141 +++++++++++++++++++++++++++++------
 gcc/lto-compress.h     |   3 +-
 gcc/lto-section-in.c   |   2 +-
 gcc/lto-streamer-out.c |   4 +
 gcc/timevar.def        |   4 +-
 12 files changed, 378 insertions(+), 30 deletions(-)

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index d9e0885b96b..597dc01328b 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1065,7 +1065,7 @@ BUILD_LIBDEPS= $(BUILD_LIBIBERTY)
 LIBS = @LIBS@ libcommon.a $(CPPLIB) $(LIBINTL) $(LIBICONV) $(LIBBACKTRACE) \
 	$(LIBIBERTY) $(LIBDECNUMBER) $(HOST_LIBS)
 BACKENDLIBS = $(ISLLIBS) $(GMPLIBS) $(PLUGINLIBS) $(HOST_LIBS) \
-	$(ZLIB)
+	$(ZLIB) $(ZSTD_LIB)
 # Any system libraries needed just for GNAT.
 SYSLIBS = @GNAT_LIBEXC@
 
@@ -1076,6 +1076,8 @@ GNATMAKE = @GNATMAKE@
 # Libs needed (at present) just for jcf-dump.
 LDEXP_LIB = @LDEXP_LIB@
 
+ZSTD_LIB = @ZSTD_LIB@
+
 # Likewise, for use in the tools that must run on this machine
 # even if we are cross-building GCC.
 BUILD_LIBS = $(BUILD_LIBIBERTY)
diff --git a/gcc/common.opt b/gcc/common.opt
index a1544d06824..3b71a36552b 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1888,8 +1888,8 @@ Specify the algorithm to partition symbols and vars at linktime.
 
 ; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h.
 flto-compression-level=
-Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 9)
--flto-compression-level=<number>	Use zlib compression level <number> for IL.
+Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 19)
+-flto-compression-level=<number>	Use zlib/zstd compression level <number> for IL.
 
 flto-odr-type-merging
 Common Ignore
diff --git a/gcc/config.in b/gcc/config.in
index a718ceaf3da..13fd7959dd7 100644
--- a/gcc/config.in
+++ b/gcc/config.in
@@ -1926,6 +1926,12 @@
 #endif
 
 
+/* Define if you have a working <zstd.h> header file. */
+#ifndef USED_FOR_TARGET
+#undef HAVE_ZSTD_H
+#endif
+
+
 /* Define if isl is in use. */
 #ifndef USED_FOR_TARGET
 #undef HAVE_isl
diff --git a/gcc/configure b/gcc/configure
index 955e9ccc09b..8c9f7742ac7 100755
--- a/gcc/configure
+++ b/gcc/configure
@@ -782,6 +782,8 @@ manext
 LIBICONV_DEP
 LTLIBICONV
 LIBICONV
+ZSTD_LIB
+ZSTD_INCLUDE
 DL_LIB
 LDEXP_LIB
 EXTRA_GCC_LIBS
@@ -959,6 +961,9 @@ with_pkgversion
 with_bugurl
 enable_languages
 with_multilib_list
+with_zstd
+with_zstd_include
+with_zstd_lib
 enable_rpath
 with_libiconv_prefix
 enable_sjlj_exceptions
@@ -1783,6 +1788,12 @@ Optional Packages:
   --with-pkgversion=PKG   Use PKG in the version string in place of "GCC"
   --with-bugurl=URL       Direct users to URL to report a bug
   --with-multilib-list    select multilibs (AArch64, SH and x86-64 only)
+  --with-zstd=PATH        specify prefix directory for installed zstd library.
+                          Equivalent to --with-zstd-include=PATH/include plus
+                          --with-zstd-lib=PATH/lib
+  --with-zstd-include=PATH
+                          specify directory for installed zstd include files
+  --with-zstd-lib=PATH    specify directory for the installed zstd library
   --with-gnu-ld           assume the C compiler uses GNU ld default=no
   --with-libiconv-prefix[=DIR]  search for libiconv in DIR/include and DIR/lib
   --without-libiconv-prefix     don't search for libiconv in includedir and libdir
@@ -9838,6 +9849,154 @@ $as_echo "#define HAVE_INTTYPES_H 1" >>confdefs.h
 
 fi
 
+# Look for the ZSTD package.
+ZSTD_INCLUDE=
+ZSTD_LIB=
+
+
+ZSTD_CPPFLAGS=
+ZSTD_LDFLAGS=
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+  withval=$with_zstd;
+fi
+
+
+# Check whether --with-zstd-include was given.
+if test "${with_zstd_include+set}" = set; then :
+  withval=$with_zstd_include;
+fi
+
+
+# Check whether --with-zstd-lib was given.
+if test "${with_zstd_lib+set}" = set; then :
+  withval=$with_zstd_lib;
+fi
+
+case "x$with_zstd" in
+  x) ;;
+  xno)
+    ZSTD_INCLUDE=no
+    ZSTD_LIB=no
+    ;;
+  *) ZSTD_INCLUDE=$with_zstd/include
+     ZSTD_LIB=$with_zstd/lib
+     ;;
+esac
+if test "x$with_zstd_include" != x; then
+  ZSTD_INCLUDE=$with_zstd_include
+fi
+if test "x$with_zstd_lib" != x; then
+  ZSTD_LIB=$with_zstd_lib
+fi
+if test "x$ZSTD_INCLUDE" != x \
+   && test "x$ZSTD_INCLUDE" != xno; then
+  ZSTD_CPPFLAGS=-I$ZSTD_INCLUDE
+fi
+if test "x$ZSTD_LIB" != x \
+   && test "x$ZSTD_LIB" != xno; then
+  ZSTD_LDFLAGS=-L$ZSTD_LIB
+fi
+
+CXXFLAGS="$CXXFLAGS $ZSTD_CPPFLAGS"
+LDFLAGS="$LDFLAGS $ZSTD_LDFLAGS"
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for zstd.h" >&5
+$as_echo_n "checking for zstd.h... " >&6; }
+if ${gcc_cv_header_zstd_h+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+#include <zstd.h>
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_cxx_try_compile "$LINENO"; then :
+  gcc_cv_header_zstd_h=yes
+else
+  gcc_cv_header_zstd_h=no
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+fi
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $gcc_cv_header_zstd_h" >&5
+$as_echo "$gcc_cv_header_zstd_h" >&6; }
+if test $gcc_cv_header_zstd_h = yes; then
+
+$as_echo "#define HAVE_ZSTD_H 1" >>confdefs.h
+
+fi
+
+# LTO can use zstd compression algorithm
+save_LIBS="$LIBS"
+LIBS=
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing ZSTD_compress" >&5
+$as_echo_n "checking for library containing ZSTD_compress... " >&6; }
+if ${ac_cv_search_ZSTD_compress+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_func_search_save_LIBS=$LIBS
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+  ;
+  return 0;
+}
+_ACEOF
+for ac_lib in '' zstd; do
+  if test -z "$ac_lib"; then
+    ac_res="none required"
+  else
+    ac_res=-l$ac_lib
+    LIBS="-l$ac_lib  $ac_func_search_save_LIBS"
+  fi
+  if ac_fn_cxx_try_link "$LINENO"; then :
+  ac_cv_search_ZSTD_compress=$ac_res
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext
+  if ${ac_cv_search_ZSTD_compress+:} false; then :
+  break
+fi
+done
+if ${ac_cv_search_ZSTD_compress+:} false; then :
+
+else
+  ac_cv_search_ZSTD_compress=no
+fi
+rm conftest.$ac_ext
+LIBS=$ac_func_search_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_search_ZSTD_compress" >&5
+$as_echo "$ac_cv_search_ZSTD_compress" >&6; }
+ac_res=$ac_cv_search_ZSTD_compress
+if test "$ac_res" != no; then :
+  test "$ac_res" = "none required" || LIBS="$ac_res $LIBS"
+
+fi
+
+ZSTD_LIB="$LIBS"
+LIBS="$save_LIBS"
+
+
 
 
 for ac_func in times clock kill getrlimit setrlimit atoq \
@@ -18655,7 +18814,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 18658 "configure"
+#line 18817 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -18761,7 +18920,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 18764 "configure"
+#line 18923 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
diff --git a/gcc/configure.ac b/gcc/configure.ac
index 066a6f4c958..137d5b469c3 100644
--- a/gcc/configure.ac
+++ b/gcc/configure.ac
@@ -1258,6 +1258,72 @@ if test $gcc_cv_header_inttypes_h = yes; then
 	[Define if you have a working <inttypes.h> header file.])
 fi
 
+# Look for the ZSTD package.
+ZSTD_INCLUDE=
+ZSTD_LIB=
+AC_SUBST(ZSTD_INCLUDE)
+AC_SUBST(ZSTD_LIB)
+ZSTD_CPPFLAGS=
+ZSTD_LDFLAGS=
+AC_ARG_WITH(zstd,
+	[AS_HELP_STRING([--with-zstd=PATH],
+		[specify prefix directory for installed zstd library.
+		 Equivalent to --with-zstd-include=PATH/include
+		 plus --with-zstd-lib=PATH/lib])])
+AC_ARG_WITH(zstd-include,
+	[AS_HELP_STRING([--with-zstd-include=PATH],
+		[specify directory for installed zstd include files])])
+AC_ARG_WITH(zstd-lib,
+	[AS_HELP_STRING([--with-zstd-lib=PATH],
+		[specify directory for the installed zstd library])])
+case "x$with_zstd" in
+  x) ;;
+  xno)
+    ZSTD_INCLUDE=no
+    ZSTD_LIB=no
+    ;;
+  *) ZSTD_INCLUDE=$with_zstd/include
+     ZSTD_LIB=$with_zstd/lib
+     ;;
+esac
+if test "x$with_zstd_include" != x; then
+  ZSTD_INCLUDE=$with_zstd_include
+fi
+if test "x$with_zstd_lib" != x; then
+  ZSTD_LIB=$with_zstd_lib
+fi
+if test "x$ZSTD_INCLUDE" != x \
+   && test "x$ZSTD_INCLUDE" != xno; then
+  ZSTD_CPPFLAGS=-I$ZSTD_INCLUDE
+fi
+if test "x$ZSTD_LIB" != x \
+   && test "x$ZSTD_LIB" != xno; then
+  ZSTD_LDFLAGS=-L$ZSTD_LIB
+fi
+
+CXXFLAGS="$CXXFLAGS $ZSTD_CPPFLAGS"
+LDFLAGS="$LDFLAGS $ZSTD_LDFLAGS"
+
+AC_MSG_CHECKING(for zstd.h)
+AC_CACHE_VAL(gcc_cv_header_zstd_h,
+[AC_COMPILE_IFELSE([AC_LANG_PROGRAM(
+[[#include <zstd.h>]])],
+  [gcc_cv_header_zstd_h=yes],
+  [gcc_cv_header_zstd_h=no])])
+AC_MSG_RESULT($gcc_cv_header_zstd_h)
+if test $gcc_cv_header_zstd_h = yes; then
+  AC_DEFINE(HAVE_ZSTD_H, 1,
+	[Define if you have a working <zstd.h> header file.])
+fi
+
+# LTO can use zstd compression algorithm
+save_LIBS="$LIBS"
+LIBS=
+AC_SEARCH_LIBS(ZSTD_compress, zstd)
+ZSTD_LIB="$LIBS"
+LIBS="$save_LIBS"
+AC_SUBST(ZSTD_LIB)
+
 dnl Disabled until we have a complete test for buggy enum bitfields.
 dnl gcc_AC_C_ENUM_BF_UNSIGNED
 
diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index 29d047004fd..12372a8c1f5 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -396,6 +396,12 @@ built together with GCC.  Alternatively, the @option{--with-isl} configure
 option should be used if isl is not installed in your default library
 search path.
 
+@item zstd Library.
+
+Necessary to build GCC with zstd compression used for LTO bytecode.
+The library is searched in your default library patch search.
+Alternatively, the @option{--with-zstd} configure option should be used.
+
 @end table
 
 @heading Tools/packages necessary for modifying GCC
diff --git a/gcc/gcc.c b/gcc/gcc.c
index 9bd65508b00..0c0a686eb9c 100644
--- a/gcc/gcc.c
+++ b/gcc/gcc.c
@@ -6791,6 +6791,11 @@ print_configuration (FILE *file)
 #endif
 
   fnotice (file, "Thread model: %s\n", thrmod);
+  fnotice (file, "Supported LTO compression algorithms: zlib");
+#ifdef HAVE_ZSTD_H
+  fnotice (file, " zstd");
+#endif
+  fnotice (file, "\n");
 
   /* compiler_version is truncated at the first space when initialized
   from version string, so truncate version_string at the first space
diff --git a/gcc/lto-compress.c b/gcc/lto-compress.c
index 3287178f257..b925363ac71 100644
--- a/gcc/lto-compress.c
+++ b/gcc/lto-compress.c
@@ -35,6 +35,10 @@ along with GCC; see the file COPYING3.  If not see
 #include "lto-compress.h"
 #include "timevar.h"
 
+#ifdef HAVE_ZSTD_H
+#include <zstd.h>
+#endif
+
 /* Compression stream structure, holds the flush callback and opaque token,
    the buffered data, and a note of whether compressing or uncompressing.  */
 
@@ -92,6 +96,91 @@ lto_normalized_zlib_level (void)
   return level;
 }
 
+/* Free the buffer and memory associated with STREAM.  */
+
+static void
+lto_destroy_compression_stream (struct lto_compression_stream *stream)
+{
+  free (stream->buffer);
+  free (stream);
+}
+
+#ifdef HAVE_ZSTD_H
+/* Return a zstd compression level that zstd will not reject.  Normalizes
+   the compression level from the command line flag, clamping non-default
+   values to the appropriate end of their valid range.  */
+
+static int
+lto_normalized_zstd_level (void)
+{
+  int level = flag_lto_compression_level;
+
+  if (level != ZSTD_CLEVEL_DEFAULT)
+    {
+      if (level < 1)
+	level = 1;
+      else if (level > ZSTD_maxCLevel ())
+	level = ZSTD_maxCLevel ();
+    }
+
+  return level;
+}
+
+/* Compress STREAM using ZSTD algorithm.  */
+
+static void
+lto_compression_zstd (struct lto_compression_stream *stream)
+{
+  unsigned char *cursor = (unsigned char *) stream->buffer;
+  size_t size = stream->bytes;
+
+  timevar_push (TV_IPA_LTO_COMPRESS);
+  size_t const outbuf_length = ZSTD_compressBound (size);
+  char *outbuf = (char *) xmalloc (outbuf_length);
+
+  size_t const csize = ZSTD_compress (outbuf, outbuf_length, cursor, size,
+				      lto_normalized_zstd_level ());
+
+  if (ZSTD_isError (csize))
+    internal_error ("compressed stream: %s", ZSTD_getErrorName (csize));
+
+  stream->callback (outbuf, csize, NULL);
+
+  lto_destroy_compression_stream (stream);
+  free (outbuf);
+  timevar_pop (TV_IPA_LTO_COMPRESS);
+}
+
+/* Uncompress STREAM using ZSTD algorithm.  */
+
+static void
+lto_uncompression_zstd (struct lto_compression_stream *stream)
+{
+  unsigned char *cursor = (unsigned char *) stream->buffer;
+  size_t size = stream->bytes;
+
+  timevar_push (TV_IPA_LTO_DECOMPRESS);
+  unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size);
+  if (rsize == ZSTD_CONTENTSIZE_ERROR)
+    internal_error ("original not compressed with zstd");
+  else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
+    internal_error ("original size unknown");
+
+  char *outbuf = (char *) xmalloc (rsize);
+  size_t const dsize = ZSTD_decompress (outbuf, rsize, cursor, size);
+
+  if (ZSTD_isError (dsize))
+    internal_error ("decompressed stream: %s", ZSTD_getErrorName (dsize));
+
+  stream->callback (outbuf, dsize, stream->opaque);
+
+  lto_destroy_compression_stream (stream);
+  free (outbuf);
+  timevar_pop (TV_IPA_LTO_DECOMPRESS);
+}
+
+#endif
+
 /* Create a new compression stream, with CALLBACK flush function passed
    OPAQUE token, IS_COMPRESSION indicates if compressing or uncompressing.  */
 
@@ -132,15 +221,6 @@ lto_append_to_compression_stream (struct lto_compression_stream *stream,
   stream->bytes += num_chars;
 }
 
-/* Free the buffer and memory associated with STREAM.  */
-
-static void
-lto_destroy_compression_stream (struct lto_compression_stream *stream)
-{
-  free (stream->buffer);
-  free (stream);
-}
-
 /* Return a new compression stream, with CALLBACK flush function passed
    OPAQUE token.  */
 
@@ -163,10 +243,8 @@ lto_compress_block (struct lto_compression_stream *stream,
   lto_stats.num_output_il_bytes += num_chars;
 }
 
-/* Finalize STREAM compression, and free stream allocations.  */
-
-void
-lto_end_compression (struct lto_compression_stream *stream)
+static void ATTRIBUTE_UNUSED
+lto_compression_zlib (struct lto_compression_stream *stream)
 {
   unsigned char *cursor = (unsigned char *) stream->buffer;
   size_t remaining = stream->bytes;
@@ -226,6 +304,16 @@ lto_end_compression (struct lto_compression_stream *stream)
   timevar_pop (TV_IPA_LTO_COMPRESS);
 }
 
+void
+lto_end_compression (struct lto_compression_stream *stream)
+{
+#ifdef HAVE_ZSTD_H
+  lto_compression_zstd (stream);
+#else
+  lto_compression_zlib (stream);
+#endif
+}
+
 /* Return a new uncompression stream, with CALLBACK flush function passed
    OPAQUE token.  */
 
@@ -248,14 +336,8 @@ lto_uncompress_block (struct lto_compression_stream *stream,
   lto_stats.num_input_il_bytes += num_chars;
 }
 
-/* Finalize STREAM uncompression, and free stream allocations.
-
-   Because of the way LTO IL streams are compressed, there may be several
-   concatenated compressed segments in the accumulated data, so for this
-   function we iterate decompressions until no data remains.  */
-
-void
-lto_end_uncompression (struct lto_compression_stream *stream)
+static void
+lto_uncompression_zlib (struct lto_compression_stream *stream)
 {
   unsigned char *cursor = (unsigned char *) stream->buffer;
   size_t remaining = stream->bytes;
@@ -318,3 +400,20 @@ lto_end_uncompression (struct lto_compression_stream *stream)
   free (outbuf);
   timevar_pop (TV_IPA_LTO_DECOMPRESS);
 }
+
+void
+lto_end_uncompression (struct lto_compression_stream *stream,
+		       lto_compression compression)
+{
+#ifdef HAVE_ZSTD_H
+  if (compression == ZSTD)
+    {
+      lto_uncompression_zstd (stream);
+      return;
+    }
+#endif
+  if (compression == ZSTD)
+    internal_error ("compiler does not support ZSTD LTO compression");
+
+  lto_uncompression_zlib (stream);
+}
diff --git a/gcc/lto-compress.h b/gcc/lto-compress.h
index 87f53f7cd65..564b17e9f98 100644
--- a/gcc/lto-compress.h
+++ b/gcc/lto-compress.h
@@ -37,6 +37,7 @@ extern struct lto_compression_stream
 			    void *opaque);
 extern void lto_uncompress_block (struct lto_compression_stream *stream,
 				  const char *base, size_t num_chars);
-extern void lto_end_uncompression (struct lto_compression_stream *stream);
+extern void lto_end_uncompression (struct lto_compression_stream *stream,
+				  lto_compression compression);
 
 #endif /* GCC_LTO_COMPRESS_H  */
diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c
index 4e7d1181f23..80fdb03f4d4 100644
--- a/gcc/lto-section-in.c
+++ b/gcc/lto-section-in.c
@@ -161,7 +161,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data,
 
       stream = lto_start_uncompression (lto_append_data, &buffer);
       lto_uncompress_block (stream, data, *len);
-      lto_end_uncompression (stream);
+      lto_end_uncompression (stream, file_data->lto_section_header.compression);
 
       *len = buffer.length - header_length;
       data = buffer.data + header_length;
diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c
index 7dee770aa11..49ca5cecad1 100644
--- a/gcc/lto-streamer-out.c
+++ b/gcc/lto-streamer-out.c
@@ -2395,7 +2395,11 @@ produce_lto_section ()
   lto_begin_section (section_name, false);
   free (section_name);
 
+#ifdef HAVE_ZSTD_H
+  lto_compression compression = ZSTD;
+#else
   lto_compression compression = ZLIB;
+#endif
 
   bool slim_object = flag_generate_lto && !flag_fat_lto_objects;
   lto_section s
diff --git a/gcc/timevar.def b/gcc/timevar.def
index 13cb470b688..626ce493b76 100644
--- a/gcc/timevar.def
+++ b/gcc/timevar.def
@@ -78,8 +78,8 @@ DEFTIMEVAR (TV_IPA_INLINING          , "ipa inlining heuristics")
 DEFTIMEVAR (TV_IPA_FNSPLIT           , "ipa function splitting")
 DEFTIMEVAR (TV_IPA_COMDATS	     , "ipa comdats")
 DEFTIMEVAR (TV_IPA_OPT		     , "ipa various optimizations")
-DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS    , "lto stream inflate")
-DEFTIMEVAR (TV_IPA_LTO_COMPRESS      , "lto stream deflate")
+DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS    , "lto stream decompression")
+DEFTIMEVAR (TV_IPA_LTO_COMPRESS      , "lto stream compression")
 DEFTIMEVAR (TV_IPA_LTO_OUTPUT        , "lto stream output")
 DEFTIMEVAR (TV_IPA_LTO_GIMPLE_IN     , "ipa lto gimple in")
 DEFTIMEVAR (TV_IPA_LTO_GIMPLE_OUT    , "ipa lto gimple out")
-- 
2.22.0

Re: [PATCH 2/2] Add zstd support for LTO bytecode compression.

[Jeff Law]

On 7/1/19 5:00 AM, Martin Liška wrote:
> Hi.
> 
> This is updated version of the zstd patch that should handle all what Joseph
> pointed out.
> 
> Patch can bootstrap on x86_64-linux-gnu and survives regression tests.
> 
> Ready to be installed?
> Thanks,
> Martin
> 
> 
> 0002-Add-zstd-support-for-LTO-bytecode-compression.patch
> 
> From 5d2006c9c4d481f4083d5a591327ee64847b0bf7 Mon Sep 17 00:00:00 2001
> From: Martin Liska <mliska@suse.cz>
> Date: Thu, 20 Jun 2019 10:08:17 +0200
> Subject: [PATCH 2/2] Add zstd support for LTO bytecode compression.
> 
> gcc/ChangeLog:
> 
> 2019-07-01  Martin Liska  <mliska@suse.cz>
> 
> 	* Makefile.in: Define ZSTD_LIB.
> 	* common.opt: Adjust compression level
> 	to support also zstd levels.
> 	* config.in: Regenerate.
> 	* configure: Likewise.
> 	* configure.ac: Add --with-zstd and --with-zstd-include options
> 	and detect ZSTD.
> 	* doc/install.texi: Mention zstd dependency.
> 	* gcc.c: Print supported LTO compression algorithms.
> 	* lto-compress.c (lto_normalized_zstd_level): Likewise.
> 	(lto_compression_zstd): Likewise.
> 	(lto_uncompression_zstd): Likewise.
> 	(lto_end_compression): Dispatch in between zlib and zstd.
> 	(lto_compression_zlib): Mark with ATTRIBUTE_UNUSED.
> 	(lto_uncompression_zlib): Make it static.
> 	* lto-compress.h (lto_end_uncompression): Fix GNU coding style.
> 	* lto-section-in.c (lto_get_section_data): Pass info
> 	about used compression.
> 	* lto-streamer-out.c: By default use zstd when possible.
> 	* timevar.def (TV_IPA_LTO_DECOMPRESS): Rename to decompression
> 	(TV_IPA_LTO_COMPRESS): Likewise for compression.
OK
jeff

Re: [PATCH] Add .gnu.lto_.lto section.

[Jeff Law]

On 7/1/19 4:59 AM, Martin Liška wrote:
> Hi.
> 
> Ok, so there's a version with added ChangeLog that survives regression tests.
> 
> Ready to be installed?
> Thanks,
> Martin
> 
> 
> 0001-Add-.gnu.lto_.lto-section.patch
> 
> From e6745583dc4b7f5543878c0a25498e818531f73e Mon Sep 17 00:00:00 2001
> From: Martin Liska <mliska@suse.cz>
> Date: Fri, 21 Jun 2019 12:14:04 +0200
> Subject: [PATCH 1/2] Add .gnu.lto_.lto section.
> 
> gcc/ChangeLog:
> 
> 2019-07-01  Martin Liska  <mliska@suse.cz>
> 
> 	* lto-section-in.c (lto_get_section_data): Add "lto" section.
> 	* lto-section-out.c (lto_destroy_simple_output_block): Never
> 	compress LTO_section_lto section.
> 	* lto-streamer-out.c (produce_asm): Do not set major_version
> 	and minor_version.
> 	(lto_output_toplevel_asms): Likewise.
> 	(produce_lto_section): New function.
> 	(lto_output): Call produce_lto_section.
> 	(lto_write_mode_table): Do not set major_version and
> 	minor_version.
> 	(produce_asm_for_decls): Likewise.
> 	* lto-streamer.h (enum lto_section_type): Add LTO_section_lto
> 	type.
> 	(struct lto_header): Remove.
> 	(struct lto_section): New struct.
> 	(struct lto_simple_header): Do not inherit from lto_header.
> 	(struct lto_file_decl_data): Add lto_section_header field.
OK
jeff

Re: [RFC] zstd as a compression algorithm for LTO

[Thomas Koenig]

Hi Martin,

> LTO bytecode is not supposed to be a distributable format.

One of my dreams is to make libgfortran LTO-clean.  There is
a lot of performance to be gained both in I/O (where a huge
number of special cases could be shortcut by LTO, because
hardly any program uses them all) and in array intrinsics,
where seeing through the array descriptors can also lead
to large benefits.  This is PR 77278.

Once this is achieved, it would make sense to distribute
libgfortran.a as a library of fat object files.

Regards

	Thomas

Re: [RFC] zstd as a compression algorithm for LTO

[Joseph Myers]

Any use of a host library should come with associated configure options to 
specify header and library paths for that library (and documentation for 
those options).  (See existing --with-gmp*, --with-isl* etc. options.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [RFC] zstd as a compression algorithm for LTO

[Segher Boessenkool]

On Wed, Jun 19, 2019 at 09:29:54PM +0200, Jan Hubicka wrote:
> > 
> > At least allow it to be built as part of the normal build like GMP,
> > etc. are done.
> > And include it in downloading using contrib/download_prerequisites
> > like the libraries are done.
> 
> Anoying detail is that zstd builds with cmake, not autotools

It looks like it would be pretty trivial to integrate into a real build
system though.

OTOH, how well has this been tested, then?  On different platforms, etc.?


Segher

Re: [RFC] zstd as a compression algorithm for LTO

[Martin Liška]

On 6/20/19 12:46 PM, Segher Boessenkool wrote:
> OTOH, how well has this been tested, then?  On different platforms, etc.?

Dunno.

The algorithm is quite new (4 years), but as it's becoming more popular.
So I would expect coverage would be quite good.

Maybe Nathan can comment on this?
Martin