* [RFC] zstd as a compression algorithm for LTO @ 2019-06-19 9:21 Martin Liška 2019-06-19 16:03 ` Jeff Law 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-19 9:21 UTC (permalink / raw) To: GCC Development; +Cc: Jan Hubicka [-- Attachment #1: Type: text/plain, Size: 1047 bytes --] Hi. I've written a patch draft that replaces zlib with the zstd compression algorithm ([1]) in LTO. I'm also sending statistics that are collected for couple of quite big C++ source files. Observation I did: - LTO stream compression takes 3-4% of LGEN compile time - zstd in default compression level (3) generated slighly smaller LTO elf files - zstd compression is 4-8x faster - decompression is quite negligible, but for a bigger project (godot) I can reduction from 1.37 to 0.53 seconds - ZSTD API is much simpler to use Suggestion based on the observation: - I would suggest to make zstd optional (--enable-zstd) and one would use #include <zstd> + -lzstd - I like the default level as we want to mainly speed up LTO compilation - we can provide an option to control algorithm (-flto-compression-algorithm), similarly to -flto-compression-level - we can discuss possible compression of LTO bytecode that is distributed between WPA stage and individual LTRANS phases. Thoughts? Thanks, Martin [1] https://github.com/facebook/zstd [-- Attachment #2: 0001-Replace-zlib-with-zstd.patch --] [-- Type: text/x-patch, Size: 9130 bytes --] From 4939e90b2a8051128b7b2b0214a5fad5183f3bca Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Wed, 19 Jun 2019 09:40:35 +0200 Subject: [PATCH] Replace zlib with zstd. --- gcc/Makefile.in | 2 +- gcc/common.opt | 2 +- gcc/lto-compress.c | 161 ++++++++------------------------------------- gcc/timevar.def | 4 +- 4 files changed, 33 insertions(+), 136 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index d9e0885b96b..8aedcccb717 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -373,7 +373,7 @@ OUTPUT_OPTION = -o $@ # This is where we get zlib from. zlibdir is -L../zlib and zlibinc is # -I../zlib, unless we were configured with --with-system-zlib, in which # case both are empty. -ZLIB = @zlibdir@ -lz +ZLIB = @zlibdir@ -lzstd -lz ZLIBINC = @zlibinc@ # How to find GMP diff --git a/gcc/common.opt b/gcc/common.opt index a1544d06824..f15e21914f3 100644 --- a/gcc/common.opt +++ b/gcc/common.opt @@ -1888,7 +1888,7 @@ Specify the algorithm to partition symbols and vars at linktime. ; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h. flto-compression-level= -Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 9) +Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 1111) -flto-compression-level=<number> Use zlib compression level <number> for IL. flto-odr-type-merging diff --git a/gcc/lto-compress.c b/gcc/lto-compress.c index 3287178f257..b24f30f956e 100644 --- a/gcc/lto-compress.c +++ b/gcc/lto-compress.c @@ -27,13 +27,9 @@ along with GCC; see the file COPYING3. If not see #include "gimple.h" #include "cgraph.h" #include "lto-streamer.h" -/* zlib.h includes other system headers. Those headers may test feature - test macros. config.h may define feature test macros. For this reason, - zlib.h needs to be included after, rather than before, config.h and - system.h. */ -#include <zlib.h> #include "lto-compress.h" #include "timevar.h" +#include <zstd.h> /* Compression stream structure, holds the flush callback and opaque token, the buffered data, and a note of whether compressing or uncompressing. */ @@ -48,45 +44,23 @@ struct lto_compression_stream bool is_compression; }; -/* Overall compression constants for zlib. */ - -static const size_t Z_BUFFER_LENGTH = 4096; static const size_t MIN_STREAM_ALLOCATION = 1024; -/* For zlib, allocate SIZE count of ITEMS and return the address, OPAQUE - is unused. */ - -static void * -lto_zalloc (void *opaque, unsigned items, unsigned size) -{ - gcc_assert (opaque == Z_NULL); - return xmalloc (items * size); -} - -/* For zlib, free memory at ADDRESS, OPAQUE is unused. */ - -static void -lto_zfree (void *opaque, void *address) -{ - gcc_assert (opaque == Z_NULL); - free (address); -} - -/* Return a zlib compression level that zlib will not reject. Normalizes +/* Return a zstd compression level that zstd will not reject. Normalizes the compression level from the command line flag, clamping non-default values to the appropriate end of their valid range. */ static int -lto_normalized_zlib_level (void) +lto_normalized_zstd_level (void) { int level = flag_lto_compression_level; - if (level != Z_DEFAULT_COMPRESSION) + if (level != ZSTD_CLEVEL_DEFAULT) { - if (level < Z_NO_COMPRESSION) - level = Z_NO_COMPRESSION; - else if (level > Z_BEST_COMPRESSION) - level = Z_BEST_COMPRESSION; + if (level < 1) + level = 1; + else if (level > ZSTD_maxCLevel ()) + level = ZSTD_maxCLevel (); } return level; @@ -169,57 +143,19 @@ void lto_end_compression (struct lto_compression_stream *stream) { unsigned char *cursor = (unsigned char *) stream->buffer; - size_t remaining = stream->bytes; - const size_t outbuf_length = Z_BUFFER_LENGTH; - unsigned char *outbuf = (unsigned char *) xmalloc (outbuf_length); - z_stream out_stream; - size_t compressed_bytes = 0; - int status; - - gcc_assert (stream->is_compression); + size_t size = stream->bytes; timevar_push (TV_IPA_LTO_COMPRESS); + size_t const outbuf_length = ZSTD_compressBound (size); + char *outbuf = (char *) xmalloc (outbuf_length); - out_stream.next_out = outbuf; - out_stream.avail_out = outbuf_length; - out_stream.next_in = cursor; - out_stream.avail_in = remaining; - out_stream.zalloc = lto_zalloc; - out_stream.zfree = lto_zfree; - out_stream.opaque = Z_NULL; + size_t const csize = ZSTD_compress (outbuf, outbuf_length, cursor, size, + lto_normalized_zstd_level ()); - status = deflateInit (&out_stream, lto_normalized_zlib_level ()); - if (status != Z_OK) - internal_error ("compressed stream: %s", zError (status)); + if (ZSTD_isError (csize)) + internal_error ("compressed stream: %s", ZSTD_getErrorName (csize)); - do - { - size_t in_bytes, out_bytes; - - status = deflate (&out_stream, Z_FINISH); - if (status != Z_OK && status != Z_STREAM_END) - internal_error ("compressed stream: %s", zError (status)); - - in_bytes = remaining - out_stream.avail_in; - out_bytes = outbuf_length - out_stream.avail_out; - - stream->callback ((const char *) outbuf, out_bytes, stream->opaque); - lto_stats.num_compressed_il_bytes += out_bytes; - compressed_bytes += out_bytes; - - cursor += in_bytes; - remaining -= in_bytes; - - out_stream.next_out = outbuf; - out_stream.avail_out = outbuf_length; - out_stream.next_in = cursor; - out_stream.avail_in = remaining; - } - while (status != Z_STREAM_END); - - status = deflateEnd (&out_stream); - if (status != Z_OK) - internal_error ("compressed stream: %s", zError (status)); + stream->callback (outbuf, csize, NULL); lto_destroy_compression_stream (stream); free (outbuf); @@ -258,61 +194,22 @@ void lto_end_uncompression (struct lto_compression_stream *stream) { unsigned char *cursor = (unsigned char *) stream->buffer; - size_t remaining = stream->bytes; - const size_t outbuf_length = Z_BUFFER_LENGTH; - unsigned char *outbuf = (unsigned char *) xmalloc (outbuf_length); - size_t uncompressed_bytes = 0; + size_t size = stream->bytes; - gcc_assert (!stream->is_compression); timevar_push (TV_IPA_LTO_DECOMPRESS); + unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size); + if (rsize == ZSTD_CONTENTSIZE_ERROR) + internal_error ("not compressed by zstd"); + else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN) + internal_error ("original size unknown"); - while (remaining > 0) - { - z_stream in_stream; - size_t out_bytes; - int status; - - in_stream.next_out = outbuf; - in_stream.avail_out = outbuf_length; - in_stream.next_in = cursor; - in_stream.avail_in = remaining; - in_stream.zalloc = lto_zalloc; - in_stream.zfree = lto_zfree; - in_stream.opaque = Z_NULL; - - status = inflateInit (&in_stream); - if (status != Z_OK) - internal_error ("compressed stream: %s", zError (status)); - - do - { - size_t in_bytes; - - status = inflate (&in_stream, Z_SYNC_FLUSH); - if (status != Z_OK && status != Z_STREAM_END) - internal_error ("compressed stream: %s", zError (status)); - - in_bytes = remaining - in_stream.avail_in; - out_bytes = outbuf_length - in_stream.avail_out; - - stream->callback ((const char *) outbuf, out_bytes, stream->opaque); - lto_stats.num_uncompressed_il_bytes += out_bytes; - uncompressed_bytes += out_bytes; - - cursor += in_bytes; - remaining -= in_bytes; - - in_stream.next_out = outbuf; - in_stream.avail_out = outbuf_length; - in_stream.next_in = cursor; - in_stream.avail_in = remaining; - } - while (!(status == Z_STREAM_END && out_bytes == 0)); - - status = inflateEnd (&in_stream); - if (status != Z_OK) - internal_error ("compressed stream: %s", zError (status)); - } + char *outbuf = (char *) xmalloc (rsize); + size_t const dsize = ZSTD_decompress (outbuf, rsize, cursor, size); + + if (ZSTD_isError (dsize)) + internal_error ("decompressed stream: %s", ZSTD_getErrorName (dsize)); + + stream->callback (outbuf, dsize, stream->opaque); lto_destroy_compression_stream (stream); free (outbuf); diff --git a/gcc/timevar.def b/gcc/timevar.def index 13cb470b688..626ce493b76 100644 --- a/gcc/timevar.def +++ b/gcc/timevar.def @@ -78,8 +78,8 @@ DEFTIMEVAR (TV_IPA_INLINING , "ipa inlining heuristics") DEFTIMEVAR (TV_IPA_FNSPLIT , "ipa function splitting") DEFTIMEVAR (TV_IPA_COMDATS , "ipa comdats") DEFTIMEVAR (TV_IPA_OPT , "ipa various optimizations") -DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS , "lto stream inflate") -DEFTIMEVAR (TV_IPA_LTO_COMPRESS , "lto stream deflate") +DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS , "lto stream decompression") +DEFTIMEVAR (TV_IPA_LTO_COMPRESS , "lto stream compression") DEFTIMEVAR (TV_IPA_LTO_OUTPUT , "lto stream output") DEFTIMEVAR (TV_IPA_LTO_GIMPLE_IN , "ipa lto gimple in") DEFTIMEVAR (TV_IPA_LTO_GIMPLE_OUT , "ipa lto gimple out") -- 2.21.0 [-- Attachment #3: zstd-vs-zlib.ods --] [-- Type: application/vnd.oasis.opendocument.spreadsheet, Size: 18640 bytes --] ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 9:21 [RFC] zstd as a compression algorithm for LTO Martin Liška @ 2019-06-19 16:03 ` Jeff Law 2019-06-19 18:55 ` Richard Biener 0 siblings, 1 reply; 32+ messages in thread From: Jeff Law @ 2019-06-19 16:03 UTC (permalink / raw) To: Martin Liška, GCC Development; +Cc: Jan Hubicka On 6/19/19 3:21 AM, Martin LiÅ¡ka wrote: > Hi. > > I've written a patch draft that replaces zlib with the zstd compression algorithm ([1]) > in LTO. I'm also sending statistics that are collected for couple of quite big C++ source > files. Observation I did: > > - LTO stream compression takes 3-4% of LGEN compile time > - zstd in default compression level (3) generated slighly smaller LTO elf files > - zstd compression is 4-8x faster > - decompression is quite negligible, but for a bigger project (godot) I can > reduction from 1.37 to 0.53 seconds > - ZSTD API is much simpler to use > > Suggestion based on the observation: > - I would suggest to make zstd optional (--enable-zstd) and one would > use #include <zstd> + -lzstd > - I like the default level as we want to mainly speed up LTO compilation > - we can provide an option to control algorithm (-flto-compression-algorithm), > similarly to -flto-compression-level > - we can discuss possible compression of LTO bytecode that is distributed between WPA > stage and individual LTRANS phases. Presumably the reason we're not being more aggressive about switching is the build/run time dependency on zstd? I wonder if we could default to zstd and fallback to zlib when zstd isn't available? jeff ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 16:03 ` Jeff Law @ 2019-06-19 18:55 ` Richard Biener 2019-06-19 19:26 ` Andrew Pinski 0 siblings, 1 reply; 32+ messages in thread From: Richard Biener @ 2019-06-19 18:55 UTC (permalink / raw) To: gcc, Jeff Law, Martin Liška, GCC Development; +Cc: Jan Hubicka On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law <law@redhat.com> wrote: >On 6/19/19 3:21 AM, Martin Liška wrote: >> Hi. >> >> I've written a patch draft that replaces zlib with the zstd >compression algorithm ([1]) >> in LTO. I'm also sending statistics that are collected for couple of >quite big C++ source >> files. Observation I did: >> >> - LTO stream compression takes 3-4% of LGEN compile time >> - zstd in default compression level (3) generated slighly smaller LTO >elf files >> - zstd compression is 4-8x faster >> - decompression is quite negligible, but for a bigger project (godot) >I can >> reduction from 1.37 to 0.53 seconds >> - ZSTD API is much simpler to use >> >> Suggestion based on the observation: >> - I would suggest to make zstd optional (--enable-zstd) and one would >> use #include <zstd> + -lzstd >> - I like the default level as we want to mainly speed up LTO >compilation >> - we can provide an option to control algorithm >(-flto-compression-algorithm), >> similarly to -flto-compression-level >> - we can discuss possible compression of LTO bytecode that is >distributed between WPA >> stage and individual LTRANS phases. >Presumably the reason we're not being more aggressive about switching >is >the build/run time dependency on zstd? I wonder if we could default to >zstd and fallback to zlib when zstd isn't available? Is zstd too big to include into the repository? But yes, we can properly encode the compression format in the LTO section header and use dlopen to 'find' the default to use. Richard. >jeff ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 18:55 ` Richard Biener @ 2019-06-19 19:26 ` Andrew Pinski 2019-06-19 19:29 ` Jan Hubicka 0 siblings, 1 reply; 32+ messages in thread From: Andrew Pinski @ 2019-06-19 19:26 UTC (permalink / raw) To: Richard Biener; +Cc: Jeff Law, Martin Liška, GCC Development, Jan Hubicka On Wed, Jun 19, 2019 at 11:55 AM Richard Biener <richard.guenther@gmail.com> wrote: > > On June 19, 2019 6:03:21 PM GMT+02:00, Jeff Law <law@redhat.com> wrote: > >On 6/19/19 3:21 AM, Martin Liška wrote: > >> Hi. > >> > >> I've written a patch draft that replaces zlib with the zstd > >compression algorithm ([1]) > >> in LTO. I'm also sending statistics that are collected for couple of > >quite big C++ source > >> files. Observation I did: > >> > >> - LTO stream compression takes 3-4% of LGEN compile time > >> - zstd in default compression level (3) generated slighly smaller LTO > >elf files > >> - zstd compression is 4-8x faster > >> - decompression is quite negligible, but for a bigger project (godot) > >I can > >> reduction from 1.37 to 0.53 seconds > >> - ZSTD API is much simpler to use > >> > >> Suggestion based on the observation: > >> - I would suggest to make zstd optional (--enable-zstd) and one would > >> use #include <zstd> + -lzstd > >> - I like the default level as we want to mainly speed up LTO > >compilation > >> - we can provide an option to control algorithm > >(-flto-compression-algorithm), > >> similarly to -flto-compression-level > >> - we can discuss possible compression of LTO bytecode that is > >distributed between WPA > >> stage and individual LTRANS phases. > >Presumably the reason we're not being more aggressive about switching > >is > >the build/run time dependency on zstd? I wonder if we could default to > >zstd and fallback to zlib when zstd isn't available? > > Is zstd too big to include into the repository? But yes, we can properly encode the compression format in the LTO section header and use dlopen to 'find' the default to use. At least allow it to be built as part of the normal build like GMP, etc. are done. And include it in downloading using contrib/download_prerequisites like the libraries are done. Thanks, Andrew Pinski > > Richard. > > >jeff > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 19:26 ` Andrew Pinski @ 2019-06-19 19:29 ` Jan Hubicka 2019-06-19 19:34 ` Andrew Pinski 2019-06-20 10:46 ` Segher Boessenkool 0 siblings, 2 replies; 32+ messages in thread From: Jan Hubicka @ 2019-06-19 19:29 UTC (permalink / raw) To: Andrew Pinski Cc: Richard Biener, Jeff Law, Martin Liška, GCC Development > > At least allow it to be built as part of the normal build like GMP, > etc. are done. > And include it in downloading using contrib/download_prerequisites > like the libraries are done. Anoying detail is that zstd builds with cmake, not autotools Honza > > Thanks, > Andrew Pinski > > > > > Richard. > > > > >jeff > > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 19:29 ` Jan Hubicka @ 2019-06-19 19:34 ` Andrew Pinski 2019-06-20 9:08 ` Martin Liška 2019-06-20 10:46 ` Segher Boessenkool 1 sibling, 1 reply; 32+ messages in thread From: Andrew Pinski @ 2019-06-19 19:34 UTC (permalink / raw) To: Jan Hubicka; +Cc: Richard Biener, Jeff Law, Martin Liška, GCC Development On Wed, Jun 19, 2019 at 12:29 PM Jan Hubicka <hubicka@ucw.cz> wrote: > > > > > At least allow it to be built as part of the normal build like GMP, > > etc. are done. > > And include it in downloading using contrib/download_prerequisites > > like the libraries are done. > > Anoying detail is that zstd builds with cmake, not autotools That makes doing candian crosses interesting and 1000x times harder than it should be :). Thanks, Andrew Pinski > > Honza > > > > Thanks, > > Andrew Pinski > > > > > > > > Richard. > > > > > > >jeff > > > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 19:34 ` Andrew Pinski @ 2019-06-20 9:08 ` Martin Liška 2019-06-20 10:59 ` Thomas Koenig 2019-06-20 17:02 ` Joseph Myers 0 siblings, 2 replies; 32+ messages in thread From: Martin Liška @ 2019-06-20 9:08 UTC (permalink / raw) To: Andrew Pinski, Jan Hubicka Cc: Richard Biener, Jeff Law, GCC Development, GCC Patches [-- Attachment #1: Type: text/plain, Size: 952 bytes --] Hello. As mentioned by Honza, it's using cmake and to be honest I prefer to use a shared library than a statically build library. Moreover, it's an optional requirement and so that we don't have to include that to contrib/download_prerequisites. I like the idea of marking of compression algorithm in 'LTO_header'. However, we do compress the header as well. Proper solution would be to make a new section .gnu.lto_.header where we'll put: struct lto_header { int16_t major_version; int16_t minor_version; }; I don't see a reason why why should have that information in each LTO ELF section? In the time being, I've written the code so that I fallback in decompression to zlib if ZSTD detects that LTO bytecode was compressed with zlib. On the contrary, decompression of zstd with zlib will end with: lto1: internal compiler error: compressed stream: data error I'm sending updated version of the patch that can properly detect zstd. Martin [-- Attachment #2: 0002-Add-optional-support-for-zstd.patch --] [-- Type: text/x-patch, Size: 7359 bytes --] From 869b630139676fb740fb5296d68086a8ef7f03ae Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Wed, 19 Jun 2019 09:40:35 +0200 Subject: [PATCH 2/2] Add optional support for zstd. --- gcc/common.opt | 4 +- gcc/lto-compress.c | 139 ++++++++++++++++++++++++++++++++++++++------- gcc/timevar.def | 4 +- 3 files changed, 122 insertions(+), 25 deletions(-) diff --git a/gcc/common.opt b/gcc/common.opt index a1544d06824..3b71a36552b 100644 --- a/gcc/common.opt +++ b/gcc/common.opt @@ -1888,8 +1888,8 @@ Specify the algorithm to partition symbols and vars at linktime. ; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h. flto-compression-level= -Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 9) --flto-compression-level=<number> Use zlib compression level <number> for IL. +Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 19) +-flto-compression-level=<number> Use zlib/zstd compression level <number> for IL. flto-odr-type-merging Common Ignore diff --git a/gcc/lto-compress.c b/gcc/lto-compress.c index 3287178f257..327ff9e07b7 100644 --- a/gcc/lto-compress.c +++ b/gcc/lto-compress.c @@ -35,6 +35,10 @@ along with GCC; see the file COPYING3. If not see #include "lto-compress.h" #include "timevar.h" +#ifdef HAVE_ZSTD_H +#include <zstd.h> +#endif + /* Compression stream structure, holds the flush callback and opaque token, the buffered data, and a note of whether compressing or uncompressing. */ @@ -92,6 +96,95 @@ lto_normalized_zlib_level (void) return level; } +/* Free the buffer and memory associated with STREAM. */ + +static void +lto_destroy_compression_stream (struct lto_compression_stream *stream) +{ + free (stream->buffer); + free (stream); +} + +#ifdef HAVE_ZSTD_H +/* Return a zstd compression level that zstd will not reject. Normalizes + the compression level from the command line flag, clamping non-default + values to the appropriate end of their valid range. */ + +static int +lto_normalized_zstd_level (void) +{ + int level = flag_lto_compression_level; + + if (level != ZSTD_CLEVEL_DEFAULT) + { + if (level < 1) + level = 1; + else if (level > ZSTD_maxCLevel ()) + level = ZSTD_maxCLevel (); + } + + return level; +} + +/* Compress STREAM using ZSTD algorithm. */ + +static void +lto_compression_zstd (struct lto_compression_stream *stream) +{ + unsigned char *cursor = (unsigned char *) stream->buffer; + size_t size = stream->bytes; + + timevar_push (TV_IPA_LTO_COMPRESS); + size_t const outbuf_length = ZSTD_compressBound (size); + char *outbuf = (char *) xmalloc (outbuf_length); + + size_t const csize = ZSTD_compress (outbuf, outbuf_length, cursor, size, + lto_normalized_zstd_level ()); + + if (ZSTD_isError (csize)) + internal_error ("compressed stream: %s", ZSTD_getErrorName (csize)); + + stream->callback (outbuf, csize, NULL); + + lto_destroy_compression_stream (stream); + free (outbuf); + timevar_pop (TV_IPA_LTO_COMPRESS); +} + +/* Uncompress STREAM using ZSTD algorithm. */ + +static bool +lto_uncompression_zstd (struct lto_compression_stream *stream) +{ + unsigned char *cursor = (unsigned char *) stream->buffer; + size_t size = stream->bytes; + + timevar_push (TV_IPA_LTO_DECOMPRESS); + unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size); + if (rsize == ZSTD_CONTENTSIZE_ERROR) + { + /* The content is probably using zlib. */ + return false; + } + else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN) + internal_error ("original size unknown"); + + char *outbuf = (char *) xmalloc (rsize); + size_t const dsize = ZSTD_decompress (outbuf, rsize, cursor, size); + + if (ZSTD_isError (dsize)) + internal_error ("decompressed stream: %s", ZSTD_getErrorName (dsize)); + + stream->callback (outbuf, dsize, stream->opaque); + + lto_destroy_compression_stream (stream); + free (outbuf); + timevar_pop (TV_IPA_LTO_DECOMPRESS); + return true; +} + +#endif + /* Create a new compression stream, with CALLBACK flush function passed OPAQUE token, IS_COMPRESSION indicates if compressing or uncompressing. */ @@ -132,15 +225,6 @@ lto_append_to_compression_stream (struct lto_compression_stream *stream, stream->bytes += num_chars; } -/* Free the buffer and memory associated with STREAM. */ - -static void -lto_destroy_compression_stream (struct lto_compression_stream *stream) -{ - free (stream->buffer); - free (stream); -} - /* Return a new compression stream, with CALLBACK flush function passed OPAQUE token. */ @@ -163,10 +247,8 @@ lto_compress_block (struct lto_compression_stream *stream, lto_stats.num_output_il_bytes += num_chars; } -/* Finalize STREAM compression, and free stream allocations. */ - -void -lto_end_compression (struct lto_compression_stream *stream) +static void +lto_compression_zlib (struct lto_compression_stream *stream) { unsigned char *cursor = (unsigned char *) stream->buffer; size_t remaining = stream->bytes; @@ -226,6 +308,16 @@ lto_end_compression (struct lto_compression_stream *stream) timevar_pop (TV_IPA_LTO_COMPRESS); } +void +lto_end_compression (struct lto_compression_stream *stream) +{ +#ifdef HAVE_ZSTD_H + lto_compression_zstd (stream); +#else + lto_compression_zlib (stream); +#endif +} + /* Return a new uncompression stream, with CALLBACK flush function passed OPAQUE token. */ @@ -248,14 +340,8 @@ lto_uncompress_block (struct lto_compression_stream *stream, lto_stats.num_input_il_bytes += num_chars; } -/* Finalize STREAM uncompression, and free stream allocations. - - Because of the way LTO IL streams are compressed, there may be several - concatenated compressed segments in the accumulated data, so for this - function we iterate decompressions until no data remains. */ - -void -lto_end_uncompression (struct lto_compression_stream *stream) +static void +lto_uncompression_zlib (struct lto_compression_stream *stream) { unsigned char *cursor = (unsigned char *) stream->buffer; size_t remaining = stream->bytes; @@ -318,3 +404,14 @@ lto_end_uncompression (struct lto_compression_stream *stream) free (outbuf); timevar_pop (TV_IPA_LTO_DECOMPRESS); } + +void +lto_end_uncompression (struct lto_compression_stream *stream) +{ +#ifdef HAVE_ZSTD_H + bool done = lto_uncompression_zstd (stream); + if (done) + return; +#endif + lto_uncompression_zlib (stream); +} diff --git a/gcc/timevar.def b/gcc/timevar.def index 13cb470b688..626ce493b76 100644 --- a/gcc/timevar.def +++ b/gcc/timevar.def @@ -78,8 +78,8 @@ DEFTIMEVAR (TV_IPA_INLINING , "ipa inlining heuristics") DEFTIMEVAR (TV_IPA_FNSPLIT , "ipa function splitting") DEFTIMEVAR (TV_IPA_COMDATS , "ipa comdats") DEFTIMEVAR (TV_IPA_OPT , "ipa various optimizations") -DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS , "lto stream inflate") -DEFTIMEVAR (TV_IPA_LTO_COMPRESS , "lto stream deflate") +DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS , "lto stream decompression") +DEFTIMEVAR (TV_IPA_LTO_COMPRESS , "lto stream compression") DEFTIMEVAR (TV_IPA_LTO_OUTPUT , "lto stream output") DEFTIMEVAR (TV_IPA_LTO_GIMPLE_IN , "ipa lto gimple in") DEFTIMEVAR (TV_IPA_LTO_GIMPLE_OUT , "ipa lto gimple out") -- 2.21.0 [-- Attachment #3: 0001-Configure-detection-for-zstd.patch --] [-- Type: text/x-patch, Size: 5987 bytes --] From 7294489d924dbe6c01ba5c66e83793757677465f Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Thu, 20 Jun 2019 10:08:17 +0200 Subject: [PATCH 1/2] Configure detection for zstd. --- gcc/Makefile.in | 4 +- gcc/config.in | 6 +++ gcc/configure | 101 ++++++++++++++++++++++++++++++++++++++++++++++- gcc/configure.ac | 21 ++++++++++ 4 files changed, 129 insertions(+), 3 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index d9e0885b96b..597dc01328b 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -1065,7 +1065,7 @@ BUILD_LIBDEPS= $(BUILD_LIBIBERTY) LIBS = @LIBS@ libcommon.a $(CPPLIB) $(LIBINTL) $(LIBICONV) $(LIBBACKTRACE) \ $(LIBIBERTY) $(LIBDECNUMBER) $(HOST_LIBS) BACKENDLIBS = $(ISLLIBS) $(GMPLIBS) $(PLUGINLIBS) $(HOST_LIBS) \ - $(ZLIB) + $(ZLIB) $(ZSTD_LIB) # Any system libraries needed just for GNAT. SYSLIBS = @GNAT_LIBEXC@ @@ -1076,6 +1076,8 @@ GNATMAKE = @GNATMAKE@ # Libs needed (at present) just for jcf-dump. LDEXP_LIB = @LDEXP_LIB@ +ZSTD_LIB = @ZSTD_LIB@ + # Likewise, for use in the tools that must run on this machine # even if we are cross-building GCC. BUILD_LIBS = $(BUILD_LIBIBERTY) diff --git a/gcc/config.in b/gcc/config.in index a718ceaf3da..13fd7959dd7 100644 --- a/gcc/config.in +++ b/gcc/config.in @@ -1926,6 +1926,12 @@ #endif +/* Define if you have a working <zstd.h> header file. */ +#ifndef USED_FOR_TARGET +#undef HAVE_ZSTD_H +#endif + + /* Define if isl is in use. */ #ifndef USED_FOR_TARGET #undef HAVE_isl diff --git a/gcc/configure b/gcc/configure index 955e9ccc09b..9c3439c156b 100755 --- a/gcc/configure +++ b/gcc/configure @@ -782,6 +782,7 @@ manext LIBICONV_DEP LTLIBICONV LIBICONV +ZSTD_LIB DL_LIB LDEXP_LIB EXTRA_GCC_LIBS @@ -9798,6 +9799,70 @@ DL_LIB="$LIBS" LIBS="$save_LIBS" +# LTO can use zstd compression algorithm +save_LIBS="$LIBS" +LIBS= +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing ZSTD_compress" >&5 +$as_echo_n "checking for library containing ZSTD_compress... " >&6; } +if ${ac_cv_search_ZSTD_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_func_search_save_LIBS=$LIBS +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char ZSTD_compress (); +int +main () +{ +return ZSTD_compress (); + ; + return 0; +} +_ACEOF +for ac_lib in '' zstd; do + if test -z "$ac_lib"; then + ac_res="none required" + else + ac_res=-l$ac_lib + LIBS="-l$ac_lib $ac_func_search_save_LIBS" + fi + if ac_fn_cxx_try_link "$LINENO"; then : + ac_cv_search_ZSTD_compress=$ac_res +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext + if ${ac_cv_search_ZSTD_compress+:} false; then : + break +fi +done +if ${ac_cv_search_ZSTD_compress+:} false; then : + +else + ac_cv_search_ZSTD_compress=no +fi +rm conftest.$ac_ext +LIBS=$ac_func_search_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_search_ZSTD_compress" >&5 +$as_echo "$ac_cv_search_ZSTD_compress" >&6; } +ac_res=$ac_cv_search_ZSTD_compress +if test "$ac_res" != no; then : + test "$ac_res" = "none required" || LIBS="$ac_res $LIBS" + +fi + +ZSTD_LIB="$LIBS" +LIBS="$save_LIBS" + + + # Use <inttypes.h> only if it exists, # doesn't clash with <sys/types.h>, declares intmax_t and defines # PRId64 @@ -9838,6 +9903,38 @@ $as_echo "#define HAVE_INTTYPES_H 1" >>confdefs.h fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for zstd.h" >&5 +$as_echo_n "checking for zstd.h... " >&6; } +if ${gcc_cv_header_zstd_h+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <zstd.h> +int +main () +{ + + ; + return 0; +} +_ACEOF +if ac_fn_cxx_try_compile "$LINENO"; then : + gcc_cv_header_zstd_h=yes +else + gcc_cv_header_zstd_h=no +fi +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $gcc_cv_header_zstd_h" >&5 +$as_echo "$gcc_cv_header_zstd_h" >&6; } +if test $gcc_cv_header_zstd_h = yes; then + +$as_echo "#define HAVE_ZSTD_H 1" >>confdefs.h + +fi + for ac_func in times clock kill getrlimit setrlimit atoq \ @@ -18655,7 +18752,7 @@ else lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2 lt_status=$lt_dlunknown cat > conftest.$ac_ext <<_LT_EOF -#line 18658 "configure" +#line 18755 "configure" #include "confdefs.h" #if HAVE_DLFCN_H @@ -18761,7 +18858,7 @@ else lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2 lt_status=$lt_dlunknown cat > conftest.$ac_ext <<_LT_EOF -#line 18764 "configure" +#line 18861 "configure" #include "confdefs.h" #if HAVE_DLFCN_H diff --git a/gcc/configure.ac b/gcc/configure.ac index 066a6f4c958..3ee4fd2e2ee 100644 --- a/gcc/configure.ac +++ b/gcc/configure.ac @@ -1237,6 +1237,15 @@ DL_LIB="$LIBS" LIBS="$save_LIBS" AC_SUBST(DL_LIB) +# LTO can use zstd compression algorithm +save_LIBS="$LIBS" +LIBS= +AC_SEARCH_LIBS(ZSTD_compress, zstd) +ZSTD_LIB="$LIBS" +LIBS="$save_LIBS" +AC_SUBST(ZSTD_LIB) + + # Use <inttypes.h> only if it exists, # doesn't clash with <sys/types.h>, declares intmax_t and defines # PRId64 @@ -1258,6 +1267,18 @@ if test $gcc_cv_header_inttypes_h = yes; then [Define if you have a working <inttypes.h> header file.]) fi +AC_MSG_CHECKING(for zstd.h) +AC_CACHE_VAL(gcc_cv_header_zstd_h, +[AC_COMPILE_IFELSE([AC_LANG_PROGRAM( +[[#include <zstd.h>]])], + [gcc_cv_header_zstd_h=yes], + [gcc_cv_header_zstd_h=no])]) +AC_MSG_RESULT($gcc_cv_header_zstd_h) +if test $gcc_cv_header_zstd_h = yes; then + AC_DEFINE(HAVE_ZSTD_H, 1, + [Define if you have a working <zstd.h> header file.]) +fi + dnl Disabled until we have a complete test for buggy enum bitfields. dnl gcc_AC_C_ENUM_BF_UNSIGNED -- 2.21.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-20 9:08 ` Martin Liška @ 2019-06-20 10:59 ` Thomas Koenig 2019-06-20 11:42 ` Martin Liška 2019-06-20 17:02 ` Joseph Myers 1 sibling, 1 reply; 32+ messages in thread From: Thomas Koenig @ 2019-06-20 10:59 UTC (permalink / raw) To: Martin Liška, Andrew Pinski, Jan Hubicka Cc: Richard Biener, Jeff Law, GCC Development, GCC Patches Am 20.06.19 um 11:07 schrieb Martin LiÅ¡ka: > On the contrary, decompression > of zstd with zlib will end with: > lto1: internal compiler error: compressed stream: data error Sogenerating object files on one system and trying to read them on another system which does not happen to have a particular library installed would lead to failure? If that's the case, I am not sure that this is a good way of handling things. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-20 10:59 ` Thomas Koenig @ 2019-06-20 11:42 ` Martin Liška 2019-06-20 12:02 ` Jan Hubicka 2019-06-20 12:12 ` [RFC] zstd as a compression algorithm for LTO Thomas Koenig 0 siblings, 2 replies; 32+ messages in thread From: Martin Liška @ 2019-06-20 11:42 UTC (permalink / raw) To: Thomas Koenig, Andrew Pinski, Jan Hubicka Cc: Richard Biener, Jeff Law, GCC Development, GCC Patches On 6/20/19 12:58 PM, Thomas Koenig wrote: > Am 20.06.19 um 11:07 schrieb Martin LiÅ¡ka: >> On the contrary, decompression >> of zstd with zlib will end with: >> lto1: internal compiler error: compressed stream: data error > > Sogenerating object files on one system and trying to read them > on another system which does not happen to have a particular > library installed would lead to failure? If that's the case, > I am not sure that this is a good way of handling things. Yes, but LTO bytecode is not supposed to be a distributable format. Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-20 11:42 ` Martin Liška @ 2019-06-20 12:02 ` Jan Hubicka 2019-06-21 10:20 ` [PATCH] Add .gnu.lto_.meta section Martin Liška 2019-06-20 12:12 ` [RFC] zstd as a compression algorithm for LTO Thomas Koenig 1 sibling, 1 reply; 32+ messages in thread From: Jan Hubicka @ 2019-06-20 12:02 UTC (permalink / raw) To: Martin Liška Cc: Thomas Koenig, Andrew Pinski, Richard Biener, Jeff Law, GCC Development, GCC Patches > On 6/20/19 12:58 PM, Thomas Koenig wrote: > > Am 20.06.19 um 11:07 schrieb Martin Liška: > >> On the contrary, decompression > >> of zstd with zlib will end with: > >> lto1: internal compiler error: compressed stream: data error > > > > Sogenerating object files on one system and trying to read them > > on another system which does not happen to have a particular > > library installed would lead to failure? If that's the case, > > I am not sure that this is a good way of handling things. > > Yes, but LTO bytecode is not supposed to be a distributable format. In longer term it should be. We ought to make it host independent and stable at least within major releases. I guess it is still OK to make zstd enabled host build require zstd enabled gcc elsewhere. Just the error message should be more informative which I think is not hard to do - both zstd and zlip should have recognizable header. Other option is to put this into some common place per file. Honza > > Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH] Add .gnu.lto_.meta section. 2019-06-20 12:02 ` Jan Hubicka @ 2019-06-21 10:20 ` Martin Liška 2019-06-21 12:34 ` Richard Biener 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-21 10:20 UTC (permalink / raw) To: Jan Hubicka Cc: Thomas Koenig, Andrew Pinski, Richard Biener, Jeff Law, GCC Development, GCC Patches [-- Attachment #1: Type: text/plain, Size: 631 bytes --] Hi. The patch is about a new ELF section that will contain information about LTO version. And for the future, used compression will be stored here. The patch removes streaming of the version into each section. Disadvantage is a format change that will lead to following errors when LTO bytecode is used from a different version: $ gcc x.o lto1: fatal error: bytecode stream in file âx.oâ generated with GCC compiler older than 10.0 $ gcc-9 main.o lto1: fatal error: bytecode stream in file âmain.oâ generated with LTO version 850.0 instead of the expected 8.0 I've been testing the patch. Thoughts? Martin [-- Attachment #2: 0001-Add-.gnu.lto_.meta-section.patch --] [-- Type: text/x-patch, Size: 7278 bytes --] From a694d3244812bb8c0ae79d3ef5aecc6f5a2bd9ca Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Fri, 21 Jun 2019 12:14:04 +0200 Subject: [PATCH] Add .gnu.lto_.meta section. gcc/ChangeLog: 2019-06-21 Martin Liska <mliska@suse.cz> * lto-section-in.c (lto_get_section_data): Do not decompress LTO_section_meta section. * lto-section-out.c (lto_destroy_simple_output_block): Do not set major and minor version. * lto-streamer-out.c (produce_asm): Likewise. (lto_output_toplevel_asms): Likewise here. (produce_lto_meta): New. (lto_output): Produce LTO meta section. (lto_write_mode_table): Do not set major and minor version. (produce_asm_for_decls): Likewise. * lto-streamer.h (enum lto_section_type): Add LTO_section_meta. (struct lto_simple_header): Do not inherit from lto_simple_header. gcc/lto/ChangeLog: 2019-06-21 Martin Liska <mliska@suse.cz> * lto-common.c: Check for LTO bytecode version. --- gcc/lto-section-in.c | 9 +++------ gcc/lto-section-out.c | 2 -- gcc/lto-streamer-out.c | 36 +++++++++++++++++++++--------------- gcc/lto-streamer.h | 3 ++- gcc/lto/lto-common.c | 14 ++++++++++++++ 5 files changed, 40 insertions(+), 24 deletions(-) diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c index 4cfc0cad4be..96962f2f2c4 100644 --- a/gcc/lto-section-in.c +++ b/gcc/lto-section-in.c @@ -52,10 +52,10 @@ const char *lto_section_name[LTO_N_SECTION_TYPES] = "icf", "offload_table", "mode_table", - "hsa" + "hsa", + "meta" }; - /* Hooks so that the ipa passes can call into the lto front end to get sections. */ @@ -146,7 +146,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data, /* WPA->ltrans streams are not compressed with exception of function bodies and variable initializers that has been verbatim copied from earlier compilations. */ - if (!flag_ltrans || decompress) + if ((!flag_ltrans || decompress) && section_type != LTO_section_meta) { /* Create a mapping header containing the underlying data and length, and prepend this to the uncompression buffer. The uncompressed data @@ -167,9 +167,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data, data = buffer.data + header_length; } - lto_check_version (((const lto_header *)data)->major_version, - ((const lto_header *)data)->minor_version, - file_data->file_name); return data; } diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c index c91e58f0465..7ae102164ef 100644 --- a/gcc/lto-section-out.c +++ b/gcc/lto-section-out.c @@ -285,8 +285,6 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob) /* Write the header which says how to decode the pieces of the t. */ memset (&header, 0, sizeof (struct lto_simple_header)); - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; header.main_size = ob->main_stream->total_size; lto_write_data (&header, sizeof header); diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c index b1084f4d3c5..bd259097dc4 100644 --- a/gcc/lto-streamer-out.c +++ b/gcc/lto-streamer-out.c @@ -1972,10 +1972,6 @@ produce_asm (struct output_block *ob, tree fn) /* The entire header is stream computed here. */ memset (&header, 0, sizeof (struct lto_function_header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - if (section_type == LTO_section_function_body) header.cfg_size = ob->cfg_stream->total_size; header.main_size = ob->main_stream->total_size; @@ -2268,10 +2264,6 @@ lto_output_toplevel_asms (void) /* The entire header stream is computed here. */ memset (&header, 0, sizeof (header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; lto_write_data (&header, sizeof header); @@ -2388,6 +2380,25 @@ prune_offload_funcs (void) DECL_PRESERVE_P (fn_decl) = 1; } +/* Produce LTO meta section that contains global information + about LTO bytecode. */ + +static void +produce_lto_meta () +{ + /* Stream LTO meta section. */ + output_block *ob = create_output_block (LTO_section_meta); + + char * section_name = lto_get_section_name (LTO_section_meta, NULL, NULL); + lto_begin_section (section_name, false); + free (section_name); + + lto_header header = { LTO_major_version, LTO_minor_version }; + lto_write_data (&header, sizeof header); + lto_end_section (); + destroy_output_block (ob); +} + /* Main entry point from the pass manager. */ void @@ -2406,6 +2417,8 @@ lto_output (void) /* Initialize the streamer. */ lto_streamer_init (); + produce_lto_meta (); + n_nodes = lto_symtab_encoder_size (encoder); /* Process only the functions with bodies. */ for (i = 0; i < n_nodes; i++) @@ -2825,10 +2838,6 @@ lto_write_mode_table (void) struct lto_simple_header_with_strings header; memset (&header, 0, sizeof (header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; lto_write_data (&header, sizeof header); @@ -2899,9 +2908,6 @@ produce_asm_for_decls (void) lto_output_decl_state_streams (ob, fn_out_state); } - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - /* Currently not used. This field would allow us to preallocate the globals vector, so that it need not be resized as it is extended. */ header.num_nodes = -1; diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h index d087cba9bf0..1b20161d193 100644 --- a/gcc/lto-streamer.h +++ b/gcc/lto-streamer.h @@ -234,6 +234,7 @@ enum lto_section_type LTO_section_offload_table, LTO_section_mode_table, LTO_section_ipa_hsa, + LTO_section_meta, LTO_N_SECTION_TYPES /* Must be last. */ }; @@ -389,7 +390,7 @@ struct lto_header /* The is the first part of the record in an LTO file for many of the IPA passes. */ -struct lto_simple_header : lto_header +struct lto_simple_header { /* Size of main gimple body of function. */ int32_t main_size; diff --git a/gcc/lto/lto-common.c b/gcc/lto/lto-common.c index 9d2f9c57780..6e64a5e4110 100644 --- a/gcc/lto/lto-common.c +++ b/gcc/lto/lto-common.c @@ -2058,6 +2058,20 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file) #else file_data->mode_table = lto_mode_identity_table; #endif + + /* Read and verify LTO meta header. */ + data = lto_get_section_data (file_data, LTO_section_meta, NULL, &len, false); + if (data == NULL) + { + fatal_error (input_location, "bytecode stream in file %qs generated " + "with GCC compiler older than 10.0", file_data->file_name); + return; + } + + lto_check_version (((const lto_header *)data)->major_version, + ((const lto_header *)data)->minor_version, + file_data->file_name); + data = lto_get_section_data (file_data, LTO_section_decls, NULL, &len); if (data == NULL) { -- 2.21.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-21 10:20 ` [PATCH] Add .gnu.lto_.meta section Martin Liška @ 2019-06-21 12:34 ` Richard Biener 2019-06-21 12:49 ` Martin Liška 0 siblings, 1 reply; 32+ messages in thread From: Richard Biener @ 2019-06-21 12:34 UTC (permalink / raw) To: Martin Liška Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On Fri, Jun 21, 2019 at 12:20 PM Martin Liška <mliska@suse.cz> wrote: > > Hi. > > The patch is about a new ELF section that will contain information > about LTO version. And for the future, used compression will be stored > here. The patch removes streaming of the version into each section. I'd like each section to have a header containing a compression method (compressed or not, and now zlib vs. zstd). We currently have a mix and knowledge is hidden. I also remember I had old patches to make the data streamer compress the stream on-the-fly, not requiring the full uncompressed stream in memory and then re-access it for compression (needing more temporary memory). But IIRC the speedup was marginal. My thought is also that we should have exactly _one_ ELF section for the LTO bytecode and our own container inside (much like simple-object does for non-ELF). So having another one is, well, ugly ;) > Disadvantage is a format change that will lead to following errors when > LTO bytecode is used from a different version: > > $ gcc x.o > lto1: fatal error: bytecode stream in file ‘x.o’ generated with GCC compiler older than 10.0 > > $ gcc-9 main.o > lto1: fatal error: bytecode stream in file ‘main.o’ generated with LTO version 850.0 instead of the expected 8.0 > > I've been testing the patch. > Thoughts? > Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-21 12:34 ` Richard Biener @ 2019-06-21 12:49 ` Martin Liška 2019-06-21 12:57 ` Jan Hubicka 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-21 12:49 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On 6/21/19 2:34 PM, Richard Biener wrote: > On Fri, Jun 21, 2019 at 12:20 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> Hi. >> >> The patch is about a new ELF section that will contain information >> about LTO version. And for the future, used compression will be stored >> here. The patch removes streaming of the version into each section. > > I'd like each section to have a header containing a compression method > (compressed or not, and now zlib vs. zstd). We currently have a mix > and knowledge is hidden. That would be possible, good idea. > > I also remember I had old patches to make the data streamer compress > the stream on-the-fly, not requiring the full uncompressed stream in > memory and then re-access it for compression (needing more temporary > memory). But IIRC the speedup was marginal. > > My thought is also that we should have exactly _one_ ELF section > for the LTO bytecode and our own container inside (much like > simple-object does for non-ELF). So having another one is, well, ugly ;) Having N sections for all symbols (functions and variables) is handy because we read some of the during WPA (by IPA ICF). So having all in a single section would make decompression more complicated. I'm going to prepare a patch that will pull out a LTO section header from compressed stream. Martin > >> Disadvantage is a format change that will lead to following errors when >> LTO bytecode is used from a different version: >> >> $ gcc x.o >> lto1: fatal error: bytecode stream in file âx.oâ generated with GCC compiler older than 10.0 >> >> $ gcc-9 main.o >> lto1: fatal error: bytecode stream in file âmain.oâ generated with LTO version 850.0 instead of the expected 8.0 >> >> I've been testing the patch. >> Thoughts? >> Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-21 12:49 ` Martin Liška @ 2019-06-21 12:57 ` Jan Hubicka 2019-06-21 14:01 ` Martin Liška 0 siblings, 1 reply; 32+ messages in thread From: Jan Hubicka @ 2019-06-21 12:57 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches > On 6/21/19 2:34 PM, Richard Biener wrote: > > On Fri, Jun 21, 2019 at 12:20 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: > >> > >> Hi. > >> > >> The patch is about a new ELF section that will contain information > >> about LTO version. And for the future, used compression will be stored > >> here. The patch removes streaming of the version into each section. > > > > I'd like each section to have a header containing a compression method > > (compressed or not, and now zlib vs. zstd). We currently have a mix > > and knowledge is hidden. > > That would be possible, good idea. > > > > > I also remember I had old patches to make the data streamer compress > > the stream on-the-fly, not requiring the full uncompressed stream in > > memory and then re-access it for compression (needing more temporary > > memory). But IIRC the speedup was marginal. It may be more interesting as memory optimization now we do parallel WPA streaming BTW. > > > > My thought is also that we should have exactly _one_ ELF section > > for the LTO bytecode and our own container inside (much like > > simple-object does for non-ELF). So having another one is, well, ugly ;) > > Having N sections for all symbols (functions and variables) is handy because > we read some of the during WPA (by IPA ICF). So having all in a single section > would make decompression more complicated. There is nothing preventing us from implementing our own subsections in that section where each is compressed independently and can be copied to output file and still save size of headers needed for that. For example all sections are keyed by the lto section enum + possibly a decl that can be both streamed as integers rather than quite lenghtly strings. (the decl to name is streamed into the global decl stream and available at all time). I guess one can go for one global LTO major/minor in the section header and then section type/is compressed/size + optional decl ID for each subsection reducing headers to about 1+8+optional 8 bytes per header. This can be all placed at the begining of the file followed by RAW data of individual sections. I bet this is a lot smaller than ELF overhead :) > > I'm going to prepare a patch that will pull out a LTO section header > from compressed stream. This looks like good step (and please stream it in host independent way). I suppose all these issues can be done one-by-one. Honza > > Martin > > > > >> Disadvantage is a format change that will lead to following errors when > >> LTO bytecode is used from a different version: > >> > >> $ gcc x.o > >> lto1: fatal error: bytecode stream in file âx.oâ generated with GCC compiler older than 10.0 > >> > >> $ gcc-9 main.o > >> lto1: fatal error: bytecode stream in file âmain.oâ generated with LTO version 850.0 instead of the expected 8.0 > >> > >> I've been testing the patch. > >> Thoughts? > >> Martin > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-21 12:57 ` Jan Hubicka @ 2019-06-21 14:01 ` Martin Liška 2019-06-24 12:02 ` Richard Biener 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-21 14:01 UTC (permalink / raw) To: Jan Hubicka Cc: Richard Biener, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches [-- Attachment #1: Type: text/plain, Size: 619 bytes --] On 6/21/19 2:57 PM, Jan Hubicka wrote: > This looks like good step (and please stream it in host independent > way). I suppose all these issues can be done one-by-one. So there's a working patch for that. However one will see following errors when using an older compiler or older LTO bytecode: $ gcc main9.o -flto lto1: fatal error: bytecode stream in file âmain9.oâ generated with LTO version -25480.4493 instead of the expected 9.0 $ gcc main.o lto1: internal compiler error: compressed stream: data error To be honest, I would prefer the new .gnu.lto_.meta section. Richi why is that so ugly? Martin [-- Attachment #2: 0001-Come-up-with-ELF-LTO-section-header.patch --] [-- Type: text/x-patch, Size: 5453 bytes --] From cbba4d8209bfeed11856baeaa2f82e44d93f4cb7 Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Fri, 21 Jun 2019 15:57:08 +0200 Subject: [PATCH] Come up with ELF LTO section header. --- gcc/lto-section-in.c | 14 +++++++++++--- gcc/lto-section-out.c | 16 ++++++++++++++-- gcc/lto-streamer-out.c | 12 ++++-------- gcc/lto-streamer.h | 14 +++++++++++--- 4 files changed, 40 insertions(+), 16 deletions(-) diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c index 4cfc0cad4be..28dfb412b67 100644 --- a/gcc/lto-section-in.c +++ b/gcc/lto-section-in.c @@ -143,6 +143,17 @@ lto_get_section_data (struct lto_file_decl_data *file_data, if (data == NULL) return NULL; + /* First check LTO section header. */ + if (*len < sizeof(lto_section_header)) + return NULL; + + const lto_section_header *sh = (const lto_section_header *)data; + lto_check_version (sh->major_version, sh->minor_version, + file_data->file_name); + + data += sizeof (lto_section_header); + *len -= sizeof (lto_section_header); + /* WPA->ltrans streams are not compressed with exception of function bodies and variable initializers that has been verbatim copied from earlier compilations. */ @@ -167,9 +178,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data, data = buffer.data + header_length; } - lto_check_version (((const lto_header *)data)->major_version, - ((const lto_header *)data)->minor_version, - file_data->file_name); return data; } diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c index c91e58f0465..9417bc9c174 100644 --- a/gcc/lto-section-out.c +++ b/gcc/lto-section-out.c @@ -81,6 +81,19 @@ lto_begin_section (const char *name, bool compress) compression_stream = lto_start_compression (lto_append_data, NULL); } +/* Emit section header with LTO bytecode version and information about used + compression. */ + +void lto_emit_section_header (void) +{ + lto_compression compression = RAW; + if (compression_stream != NULL) + compression = ZLIB; + + lto_section_header header + = { LTO_major_version, LTO_minor_version, compression }; + lto_write_raw_data (&header, sizeof header); +} /* End the current output section. */ @@ -285,8 +298,7 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob) /* Write the header which says how to decode the pieces of the t. */ memset (&header, 0, sizeof (struct lto_simple_header)); - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; + lto_emit_section_header (); header.main_size = ob->main_stream->total_size; lto_write_data (&header, sizeof header); diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c index b1084f4d3c5..42f03ac504e 100644 --- a/gcc/lto-streamer-out.c +++ b/gcc/lto-streamer-out.c @@ -1973,8 +1973,7 @@ produce_asm (struct output_block *ob, tree fn) memset (&header, 0, sizeof (struct lto_function_header)); /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; + lto_emit_section_header (); if (section_type == LTO_section_function_body) header.cfg_size = ob->cfg_stream->total_size; @@ -2269,8 +2268,7 @@ lto_output_toplevel_asms (void) memset (&header, 0, sizeof (header)); /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; + lto_emit_section_header (); header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; @@ -2826,8 +2824,7 @@ lto_write_mode_table (void) memset (&header, 0, sizeof (header)); /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; + lto_emit_section_header (); header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; @@ -2899,8 +2896,7 @@ produce_asm_for_decls (void) lto_output_decl_state_streams (ob, fn_out_state); } - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; + lto_emit_section_header (); /* Currently not used. This field would allow us to preallocate the globals vector, so that it need not be resized as it is extended. */ diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h index d087cba9bf0..8fc1ac01356 100644 --- a/gcc/lto-streamer.h +++ b/gcc/lto-streamer.h @@ -378,18 +378,25 @@ public: unsigned int len; }; +enum lto_compression +{ + RAW, + ZLIB, + ZSTD +}; /* The is the first part of the record for a function or constructor in the .o file. */ -struct lto_header +struct lto_section_header { int16_t major_version; - int16_t minor_version; + int16_t minor_version : 14; + lto_compression compression : 2; }; /* The is the first part of the record in an LTO file for many of the IPA passes. */ -struct lto_simple_header : lto_header +struct lto_simple_header { /* Size of main gimple body of function. */ int32_t main_size; @@ -784,6 +791,7 @@ extern void lto_value_range_error (const char *, /* In lto-section-out.c */ extern void lto_begin_section (const char *, bool); +extern void lto_emit_section_header (void); extern void lto_end_section (void); extern void lto_write_data (const void *, unsigned int); extern void lto_write_raw_data (const void *, unsigned int); -- 2.21.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-21 14:01 ` Martin Liška @ 2019-06-24 12:02 ` Richard Biener 2019-06-24 12:12 ` Martin Liška 0 siblings, 1 reply; 32+ messages in thread From: Richard Biener @ 2019-06-24 12:02 UTC (permalink / raw) To: Martin Liška Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/21/19 2:57 PM, Jan Hubicka wrote: > > This looks like good step (and please stream it in host independent > > way). I suppose all these issues can be done one-by-one. > > So there's a working patch for that. However one will see following errors > when using an older compiler or older LTO bytecode: > > $ gcc main9.o -flto > lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0 > > $ gcc main.o > lto1: internal compiler error: compressed stream: data error This is because of your change to bitfields or because with the old scheme the header with the version is compressed (is it?). I'd simply avoid any layout changes in the version check range. > To be honest, I would prefer the new .gnu.lto_.meta section. > Richi why is that so ugly? Because it's a change in the wrong direction and doesn't solve the issue we already have (cannot determine if a section is compressed or not). ELF section overhead is quite big if you have lots of small functions. Richard. > > Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-24 12:02 ` Richard Biener @ 2019-06-24 12:12 ` Martin Liška 2019-06-24 12:44 ` Richard Biener 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-24 12:12 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On 6/24/19 2:02 PM, Richard Biener wrote: > On Fri, Jun 21, 2019 at 4:01 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/21/19 2:57 PM, Jan Hubicka wrote: >>> This looks like good step (and please stream it in host independent >>> way). I suppose all these issues can be done one-by-one. >> >> So there's a working patch for that. However one will see following errors >> when using an older compiler or older LTO bytecode: >> >> $ gcc main9.o -flto >> lto1: fatal error: bytecode stream in file âmain9.oâ generated with LTO version -25480.4493 instead of the expected 9.0 >> >> $ gcc main.o >> lto1: internal compiler error: compressed stream: data error > > This is because of your change to bitfields or because with the old > scheme the header with the > version is compressed (is it?). Because currently also the header is compressed. > I'd simply avoid any layout changes > in the version check range. Well, then we have to find out how to distinguish between compression algorithms. > >> To be honest, I would prefer the new .gnu.lto_.meta section. >> Richi why is that so ugly? > > Because it's a change in the wrong direction and doesn't solve the > issue we already > have (cannot determine if a section is compressed or not). That's not true, the .gnu.lto_.meta section will be always uncompressed and we can also backport changes to older compiler that can read it and print a proper error message about LTO bytecode version mismatch. > ELF section overhead > is quite big if you have lots of small functions. My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section and remove the section header from all other LTO sections. That will save space for all function sections. Martin > > Richard. > >> >> Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-24 12:12 ` Martin Liška @ 2019-06-24 12:44 ` Richard Biener 2019-06-24 13:31 ` Martin Liška 0 siblings, 1 reply; 32+ messages in thread From: Richard Biener @ 2019-06-24 12:44 UTC (permalink / raw) To: Martin Liška Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/24/19 2:02 PM, Richard Biener wrote: > > On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/21/19 2:57 PM, Jan Hubicka wrote: > >>> This looks like good step (and please stream it in host independent > >>> way). I suppose all these issues can be done one-by-one. > >> > >> So there's a working patch for that. However one will see following errors > >> when using an older compiler or older LTO bytecode: > >> > >> $ gcc main9.o -flto > >> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0 > >> > >> $ gcc main.o > >> lto1: internal compiler error: compressed stream: data error > > > > This is because of your change to bitfields or because with the old > > scheme the header with the > > version is compressed (is it?). > > Because currently also the header is compressed. That was it, yeah :/ Stupid decisions in the past. I guess we have to bite the bullet and do this kind of incompatible change, accepting the odd error message above. > > I'd simply avoid any layout changes > > in the version check range. > > Well, then we have to find out how to distinguish between compression algorithms. > > > > >> To be honest, I would prefer the new .gnu.lto_.meta section. > >> Richi why is that so ugly? > > > > Because it's a change in the wrong direction and doesn't solve the > > issue we already > > have (cannot determine if a section is compressed or not). > > That's not true, the .gnu.lto_.meta section will be always uncompressed and we can > also backport changes to older compiler that can read it and print a proper error > message about LTO bytecode version mismatch. We can always backport changes, yes, but I don't see why we have to. > > ELF section overhead > > is quite big if you have lots of small functions. > > My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section > and remove the section header from all other LTO sections. That will save space > for all function sections. But we want the header there to at least say if the section is compressed or not. The fact that we have so many ELF section means we have the redundant version info everywhere. We should have a single .gnu.lto_ section (and also get rid of those __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for existence of a symbol is more expensive compared to existence of a section). Richard. > Martin > > > > > Richard. > > > >> > >> Martin > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-24 12:44 ` Richard Biener @ 2019-06-24 13:31 ` Martin Liška 2019-06-24 14:25 ` Iain Sandoe 2019-06-24 18:05 ` Richard Biener 0 siblings, 2 replies; 32+ messages in thread From: Martin Liška @ 2019-06-24 13:31 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On 6/24/19 2:44 PM, Richard Biener wrote: > On Mon, Jun 24, 2019 at 2:12 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/24/19 2:02 PM, Richard Biener wrote: >>> On Fri, Jun 21, 2019 at 4:01 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> >>>> On 6/21/19 2:57 PM, Jan Hubicka wrote: >>>>> This looks like good step (and please stream it in host independent >>>>> way). I suppose all these issues can be done one-by-one. >>>> >>>> So there's a working patch for that. However one will see following errors >>>> when using an older compiler or older LTO bytecode: >>>> >>>> $ gcc main9.o -flto >>>> lto1: fatal error: bytecode stream in file âmain9.oâ generated with LTO version -25480.4493 instead of the expected 9.0 >>>> >>>> $ gcc main.o >>>> lto1: internal compiler error: compressed stream: data error >>> >>> This is because of your change to bitfields or because with the old >>> scheme the header with the >>> version is compressed (is it?). >> >> Because currently also the header is compressed. > > That was it, yeah :/ Stupid decisions in the past. > > I guess we have to bite the bullet and do this kind of incompatible > change, accepting > the odd error message above. > >>> I'd simply avoid any layout changes >>> in the version check range. >> >> Well, then we have to find out how to distinguish between compression algorithms. >> >>> >>>> To be honest, I would prefer the new .gnu.lto_.meta section. >>>> Richi why is that so ugly? >>> >>> Because it's a change in the wrong direction and doesn't solve the >>> issue we already >>> have (cannot determine if a section is compressed or not). >> >> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can >> also backport changes to older compiler that can read it and print a proper error >> message about LTO bytecode version mismatch. > > We can always backport changes, yes, but I don't see why we have to. I'm fine with the backward compatibility break. But we should also consider lto-plugin.c that is parsing following 2 sections: 91 #define LTO_SECTION_PREFIX ".gnu.lto_.symtab" 92 #define LTO_SECTION_PREFIX_LEN (sizeof (LTO_SECTION_PREFIX) - 1) 93 #define OFFLOAD_SECTION ".gnu.offload_lto_.opts" 94 #define OFFLOAD_SECTION_LEN (sizeof (OFFLOAD_SECTION) - 1) > >>> ELF section overhead >>> is quite big if you have lots of small functions. >> >> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section >> and remove the section header from all other LTO sections. That will save space >> for all function sections. > > But we want the header there to at least say if the section is > compressed or not. > The fact that we have so many ELF section means we have the redundant version > info everywhere. > > We should have a single .gnu.lto_ section (and also get rid of those > __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for > existence of a symbol is more expensive compared to existence > of a section). I like removal of the 2 aforementioned sections. To be honest I would recommend to add a new .gnu.lto_.meta section. We can use it instead of __gnu_lto_v1 and we can have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all LTO_section_function_body, LTO_section_static_initializer sections into a single one. That will require an index that will have to be created. I can discuss that with Honza as he suggested using something smarter than function names. Thoughts? Martin > > Richard. > >> Martin >> >>> >>> Richard. >>> >>>> >>>> Martin >> ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-24 13:31 ` Martin Liška @ 2019-06-24 14:25 ` Iain Sandoe 2019-06-24 18:05 ` Richard Biener 1 sibling, 0 replies; 32+ messages in thread From: Iain Sandoe @ 2019-06-24 14:25 UTC (permalink / raw) To: Martin Liška Cc: Richard Biener, Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches > On 24 Jun 2019, at 14:31, Martin Liška <mliska@suse.cz> wrote: > > On 6/24/19 2:44 PM, Richard Biener wrote: >> On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote: >>> >>> On 6/24/19 2:02 PM, Richard Biener wrote: >>>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote: >>>>> >>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote: >>>>>> This looks like good step (and please stream it in host independent >>>>>> way). I suppose all these issues can be done one-by-one. >>>>> >>>>> So there's a working patch for that. However one will see following errors >>>>> when using an older compiler or older LTO bytecode: >>>>> >>>>> $ gcc main9.o -flto >>>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0 >>>>> >>>>> $ gcc main.o >>>>> lto1: internal compiler error: compressed stream: data error >>>> >>>> This is because of your change to bitfields or because with the old >>>> scheme the header with the >>>> version is compressed (is it?). >>> >>> Because currently also the header is compressed. >> >> That was it, yeah :/ Stupid decisions in the past. >> >> I guess we have to bite the bullet and do this kind of incompatible >> change, accepting >> the odd error message above. >> >>>> I'd simply avoid any layout changes >>>> in the version check range. >>> >>> Well, then we have to find out how to distinguish between compression algorithms. >>> >>>> >>>>> To be honest, I would prefer the new .gnu.lto_.meta section. >>>>> Richi why is that so ugly? >>>> >>>> Because it's a change in the wrong direction and doesn't solve the >>>> issue we already >>>> have (cannot determine if a section is compressed or not). >>> >>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can >>> also backport changes to older compiler that can read it and print a proper error >>> message about LTO bytecode version mismatch. >> >> We can always backport changes, yes, but I don't see why we have to. > > I'm fine with the backward compatibility break. But we should also consider lto-plugin.c > that is parsing following 2 sections: > > 91 #define LTO_SECTION_PREFIX ".gnu.lto_.symtab" > 92 #define LTO_SECTION_PREFIX_LEN (sizeof (LTO_SECTION_PREFIX) - 1) > 93 #define OFFLOAD_SECTION ".gnu.offload_lto_.opts" > 94 #define OFFLOAD_SECTION_LEN (sizeof (OFFLOAD_SECTION) - 1) > >> >>>> ELF section overhead >>>> is quite big if you have lots of small functions. >>> >>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section >>> and remove the section header from all other LTO sections. That will save space >>> for all function sections. >> >> But we want the header there to at least say if the section is >> compressed or not. >> The fact that we have so many ELF section means we have the redundant version >> info everywhere. >> >> We should have a single .gnu.lto_ section (and also get rid of those >> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for >> existence of a symbol is more expensive compared to existence >> of a section). > > I like removal of the 2 aforementioned sections. To be honest I would recommend to > add a new .gnu.lto_.meta section. We can use it instead of __gnu_lto_v1 and we can > have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all > > LTO_section_function_body, > LTO_section_static_initializer > > sections into a single one. That will require an index that will have to be created. I can discuss > that with Honza as he suggested using something smarter than function names. I already implemented a scheme (using three sections: INDEX, NAMES, PAYLOAD) for Mach-O - since it doesn’t have unlimited section count - it works - and hardly rocket science ;) - if one were to import the tabular portion of that at the start of a section and then the variable portion as a trailer … it could all be a single section. iain > Martin > >> >> Richard. >> >>> Martin >>> >>>> >>>> Richard. >>>> >>>>> >>>>> Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-24 13:31 ` Martin Liška 2019-06-24 14:25 ` Iain Sandoe @ 2019-06-24 18:05 ` Richard Biener 2019-06-25 8:14 ` Martin Liška 1 sibling, 1 reply; 32+ messages in thread From: Richard Biener @ 2019-06-24 18:05 UTC (permalink / raw) To: Martin Liška Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On Mon, Jun 24, 2019 at 3:31 PM Martin Liška <mliska@suse.cz> wrote: > > On 6/24/19 2:44 PM, Richard Biener wrote: > > On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/24/19 2:02 PM, Richard Biener wrote: > >>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote: > >>>> > >>>> On 6/21/19 2:57 PM, Jan Hubicka wrote: > >>>>> This looks like good step (and please stream it in host independent > >>>>> way). I suppose all these issues can be done one-by-one. > >>>> > >>>> So there's a working patch for that. However one will see following errors > >>>> when using an older compiler or older LTO bytecode: > >>>> > >>>> $ gcc main9.o -flto > >>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0 > >>>> > >>>> $ gcc main.o > >>>> lto1: internal compiler error: compressed stream: data error > >>> > >>> This is because of your change to bitfields or because with the old > >>> scheme the header with the > >>> version is compressed (is it?). > >> > >> Because currently also the header is compressed. > > > > That was it, yeah :/ Stupid decisions in the past. > > > > I guess we have to bite the bullet and do this kind of incompatible > > change, accepting > > the odd error message above. > > > >>> I'd simply avoid any layout changes > >>> in the version check range. > >> > >> Well, then we have to find out how to distinguish between compression algorithms. > >> > >>> > >>>> To be honest, I would prefer the new .gnu.lto_.meta section. > >>>> Richi why is that so ugly? > >>> > >>> Because it's a change in the wrong direction and doesn't solve the > >>> issue we already > >>> have (cannot determine if a section is compressed or not). > >> > >> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can > >> also backport changes to older compiler that can read it and print a proper error > >> message about LTO bytecode version mismatch. > > > > We can always backport changes, yes, but I don't see why we have to. > > I'm fine with the backward compatibility break. But we should also consider lto-plugin.c > that is parsing following 2 sections: > > 91 #define LTO_SECTION_PREFIX ".gnu.lto_.symtab" > 92 #define LTO_SECTION_PREFIX_LEN (sizeof (LTO_SECTION_PREFIX) - 1) > 93 #define OFFLOAD_SECTION ".gnu.offload_lto_.opts" > 94 #define OFFLOAD_SECTION_LEN (sizeof (OFFLOAD_SECTION) - 1) Yeah, I know. And BFD and gold hard-coded those __gnu_lto_{v1,slim} symbols... > > > >>> ELF section overhead > >>> is quite big if you have lots of small functions. > >> > >> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section > >> and remove the section header from all other LTO sections. That will save space > >> for all function sections. > > > > But we want the header there to at least say if the section is > > compressed or not. > > The fact that we have so many ELF section means we have the redundant version > > info everywhere. > > > > We should have a single .gnu.lto_ section (and also get rid of those > > __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for > > existence of a symbol is more expensive compared to existence > > of a section). > > I like removal of the 2 aforementioned sections. To be honest I would recommend to > add a new .gnu.lto_.meta section. Why .meta? Why not just .gnu.lto_? > We can use it instead of __gnu_lto_v1 and we can > have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all > > LTO_section_function_body, > LTO_section_static_initializer > > sections into a single one. That will require an index that will have to be created. I can discuss > that with Honza as he suggested using something smarter than function names. I think the index belongs to symtab? Let's properly do it if we want to change it. Removing of __gnu_lto_v1/slim is going to be the most intrusive change btw. and orthogonal to the section changes. Richard. > > Thoughts? > Martin > > > > > Richard. > > > >> Martin > >> > >>> > >>> Richard. > >>> > >>>> > >>>> Martin > >> > ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-24 18:05 ` Richard Biener @ 2019-06-25 8:14 ` Martin Liška 2019-06-25 14:15 ` Richard Biener 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-25 8:14 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On 6/24/19 8:05 PM, Richard Biener wrote: > On Mon, Jun 24, 2019 at 3:31 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >> >> On 6/24/19 2:44 PM, Richard Biener wrote: >>> On Mon, Jun 24, 2019 at 2:12 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>> >>>> On 6/24/19 2:02 PM, Richard Biener wrote: >>>>> On Fri, Jun 21, 2019 at 4:01 PM Martin LiÅ¡ka <mliska@suse.cz> wrote: >>>>>> >>>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote: >>>>>>> This looks like good step (and please stream it in host independent >>>>>>> way). I suppose all these issues can be done one-by-one. >>>>>> >>>>>> So there's a working patch for that. However one will see following errors >>>>>> when using an older compiler or older LTO bytecode: >>>>>> >>>>>> $ gcc main9.o -flto >>>>>> lto1: fatal error: bytecode stream in file âmain9.oâ generated with LTO version -25480.4493 instead of the expected 9.0 >>>>>> >>>>>> $ gcc main.o >>>>>> lto1: internal compiler error: compressed stream: data error >>>>> >>>>> This is because of your change to bitfields or because with the old >>>>> scheme the header with the >>>>> version is compressed (is it?). >>>> >>>> Because currently also the header is compressed. >>> >>> That was it, yeah :/ Stupid decisions in the past. >>> >>> I guess we have to bite the bullet and do this kind of incompatible >>> change, accepting >>> the odd error message above. >>> >>>>> I'd simply avoid any layout changes >>>>> in the version check range. >>>> >>>> Well, then we have to find out how to distinguish between compression algorithms. >>>> >>>>> >>>>>> To be honest, I would prefer the new .gnu.lto_.meta section. >>>>>> Richi why is that so ugly? >>>>> >>>>> Because it's a change in the wrong direction and doesn't solve the >>>>> issue we already >>>>> have (cannot determine if a section is compressed or not). >>>> >>>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can >>>> also backport changes to older compiler that can read it and print a proper error >>>> message about LTO bytecode version mismatch. >>> >>> We can always backport changes, yes, but I don't see why we have to. >> >> I'm fine with the backward compatibility break. But we should also consider lto-plugin.c >> that is parsing following 2 sections: >> >> 91 #define LTO_SECTION_PREFIX ".gnu.lto_.symtab" >> 92 #define LTO_SECTION_PREFIX_LEN (sizeof (LTO_SECTION_PREFIX) - 1) >> 93 #define OFFLOAD_SECTION ".gnu.offload_lto_.opts" >> 94 #define OFFLOAD_SECTION_LEN (sizeof (OFFLOAD_SECTION) - 1) > > Yeah, I know. And BFD and gold hard-coded those __gnu_lto_{v1,slim} symbols... Yep, they do, 'nm' is also using that. > >>> >>>>> ELF section overhead >>>>> is quite big if you have lots of small functions. >>>> >>>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section >>>> and remove the section header from all other LTO sections. That will save space >>>> for all function sections. >>> >>> But we want the header there to at least say if the section is >>> compressed or not. >>> The fact that we have so many ELF section means we have the redundant version >>> info everywhere. >>> >>> We should have a single .gnu.lto_ section (and also get rid of those >>> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for >>> existence of a symbol is more expensive compared to existence >>> of a section). >> >> I like removal of the 2 aforementioned sections. To be honest I would recommend to >> add a new .gnu.lto_.meta section. > > Why .meta? Why not just .gnu.lto_? Works for me. > >> We can use it instead of __gnu_lto_v1 and we can >> have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all >> >> LTO_section_function_body, >> LTO_section_static_initializer >> >> sections into a single one. That will require an index that will have to be created. I can discuss >> that with Honza as he suggested using something smarter than function names. > > I think the index belongs to symtab? > > Let's properly do it if we want to change it. Removing of > __gnu_lto_v1/slim is going to be > the most intrusive change btw. and orthogonal to the section changes. I'm fine with a proper change. So do I understand that correctly that: - we'll come up with .gnu.lto_ section that will be used by bfd, gold and nm to detect LTO objects - for some time, we'll keep __gnu_lto_v1 and __gnu_lto_slim for backward compatibility with older binutils tool - in couple of year, the legacy support will be removed ? Martin > > Richard. > >> >> Thoughts? >> Martin >> >>> >>> Richard. >>> >>>> Martin >>>> >>>>> >>>>> Richard. >>>>> >>>>>> >>>>>> Martin >>>> >> ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.meta section. 2019-06-25 8:14 ` Martin Liška @ 2019-06-25 14:15 ` Richard Biener 2019-06-27 12:28 ` [PATCH] Add .gnu.lto_.lto section Martin Liška 0 siblings, 1 reply; 32+ messages in thread From: Richard Biener @ 2019-06-25 14:15 UTC (permalink / raw) To: Martin Liška Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches On Tue, Jun 25, 2019 at 10:14 AM Martin Liška <mliska@suse.cz> wrote: > > On 6/24/19 8:05 PM, Richard Biener wrote: > > On Mon, Jun 24, 2019 at 3:31 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 6/24/19 2:44 PM, Richard Biener wrote: > >>> On Mon, Jun 24, 2019 at 2:12 PM Martin Liška <mliska@suse.cz> wrote: > >>>> > >>>> On 6/24/19 2:02 PM, Richard Biener wrote: > >>>>> On Fri, Jun 21, 2019 at 4:01 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>> > >>>>>> On 6/21/19 2:57 PM, Jan Hubicka wrote: > >>>>>>> This looks like good step (and please stream it in host independent > >>>>>>> way). I suppose all these issues can be done one-by-one. > >>>>>> > >>>>>> So there's a working patch for that. However one will see following errors > >>>>>> when using an older compiler or older LTO bytecode: > >>>>>> > >>>>>> $ gcc main9.o -flto > >>>>>> lto1: fatal error: bytecode stream in file ‘main9.o’ generated with LTO version -25480.4493 instead of the expected 9.0 > >>>>>> > >>>>>> $ gcc main.o > >>>>>> lto1: internal compiler error: compressed stream: data error > >>>>> > >>>>> This is because of your change to bitfields or because with the old > >>>>> scheme the header with the > >>>>> version is compressed (is it?). > >>>> > >>>> Because currently also the header is compressed. > >>> > >>> That was it, yeah :/ Stupid decisions in the past. > >>> > >>> I guess we have to bite the bullet and do this kind of incompatible > >>> change, accepting > >>> the odd error message above. > >>> > >>>>> I'd simply avoid any layout changes > >>>>> in the version check range. > >>>> > >>>> Well, then we have to find out how to distinguish between compression algorithms. > >>>> > >>>>> > >>>>>> To be honest, I would prefer the new .gnu.lto_.meta section. > >>>>>> Richi why is that so ugly? > >>>>> > >>>>> Because it's a change in the wrong direction and doesn't solve the > >>>>> issue we already > >>>>> have (cannot determine if a section is compressed or not). > >>>> > >>>> That's not true, the .gnu.lto_.meta section will be always uncompressed and we can > >>>> also backport changes to older compiler that can read it and print a proper error > >>>> message about LTO bytecode version mismatch. > >>> > >>> We can always backport changes, yes, but I don't see why we have to. > >> > >> I'm fine with the backward compatibility break. But we should also consider lto-plugin.c > >> that is parsing following 2 sections: > >> > >> 91 #define LTO_SECTION_PREFIX ".gnu.lto_.symtab" > >> 92 #define LTO_SECTION_PREFIX_LEN (sizeof (LTO_SECTION_PREFIX) - 1) > >> 93 #define OFFLOAD_SECTION ".gnu.offload_lto_.opts" > >> 94 #define OFFLOAD_SECTION_LEN (sizeof (OFFLOAD_SECTION) - 1) > > > > Yeah, I know. And BFD and gold hard-coded those __gnu_lto_{v1,slim} symbols... > > Yep, they do, 'nm' is also using that. > > > > >>> > >>>>> ELF section overhead > >>>>> is quite big if you have lots of small functions. > >>>> > >>>> My patch is actually shrinking space as I'm suggesting to add _one_ extra ELF section > >>>> and remove the section header from all other LTO sections. That will save space > >>>> for all function sections. > >>> > >>> But we want the header there to at least say if the section is > >>> compressed or not. > >>> The fact that we have so many ELF section means we have the redundant version > >>> info everywhere. > >>> > >>> We should have a single .gnu.lto_ section (and also get rid of those > >>> __gnu_lto_v1 and __gnu_lto_slim COMMON symbols - checking for > >>> existence of a symbol is more expensive compared to existence > >>> of a section). > >> > >> I like removal of the 2 aforementioned sections. To be honest I would recommend to > >> add a new .gnu.lto_.meta section. > > > > Why .meta? Why not just .gnu.lto_? > > Works for me. > > > > >> We can use it instead of __gnu_lto_v1 and we can > >> have a flag there instead of __gnu_lto_slim. As a second step, I'm willing to concatenate all > >> > >> LTO_section_function_body, > >> LTO_section_static_initializer > >> > >> sections into a single one. That will require an index that will have to be created. I can discuss > >> that with Honza as he suggested using something smarter than function names. > > > > I think the index belongs to symtab? > > > > Let's properly do it if we want to change it. Removing of > > __gnu_lto_v1/slim is going to be > > the most intrusive change btw. and orthogonal to the section changes. > > I'm fine with a proper change. So do I understand that correctly that: > - we'll come up with .gnu.lto_ section that will be used by bfd, gold and nm > to detect LTO objects > - for some time, we'll keep __gnu_lto_v1 and __gnu_lto_slim for backward > compatibility with older binutils tool > - in couple of year, the legacy support will be removed Yep. Richard. > ? > > Martin > > > > > Richard. > > > >> > >> Thoughts? > >> Martin > >> > >>> > >>> Richard. > >>> > >>>> Martin > >>>> > >>>>> > >>>>> Richard. > >>>>> > >>>>>> > >>>>>> Martin > >>>> > >> > ^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH] Add .gnu.lto_.lto section. 2019-06-25 14:15 ` Richard Biener @ 2019-06-27 12:28 ` Martin Liška 2019-07-01 10:59 ` Martin Liška 0 siblings, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-06-27 12:28 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches [-- Attachment #1: Type: text/plain, Size: 494 bytes --] Hi. So this is a patch candidate for the .gnu.lto_.lto ELF section. As mentioned, the section contains information about bytecode version, compression algorithm used and slim LTO object flag. One minor issues is that I need to append random suffix to the section: [ 6] .gnu.lto_.lto.36e74acfb145b04b PROGBITS 0000000000000000 000086 000008 00 E 0 0 1 That's because of in WPA we load more object files and the hash is used for mapping. Is the idea of the patch right? Martin [-- Attachment #2: 0001-Add-.gnu.lto_.lto-section.patch --] [-- Type: text/x-patch, Size: 7558 bytes --] From ec550463df26855982b7b70393766cc8e7c6cc8e Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Fri, 21 Jun 2019 12:14:04 +0200 Subject: [PATCH] Add .gnu.lto_.lto section. --- gcc/lto-section-in.c | 9 +++------ gcc/lto-section-out.c | 2 -- gcc/lto-streamer-out.c | 40 +++++++++++++++++++++++++--------------- gcc/lto-streamer.h | 25 +++++++++++++++++++++---- gcc/lto/lto-common.c | 15 +++++++++++++++ 5 files changed, 64 insertions(+), 27 deletions(-) diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c index 4cfc0cad4be..4e7d1181f23 100644 --- a/gcc/lto-section-in.c +++ b/gcc/lto-section-in.c @@ -52,10 +52,10 @@ const char *lto_section_name[LTO_N_SECTION_TYPES] = "icf", "offload_table", "mode_table", - "hsa" + "hsa", + "lto" }; - /* Hooks so that the ipa passes can call into the lto front end to get sections. */ @@ -146,7 +146,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data, /* WPA->ltrans streams are not compressed with exception of function bodies and variable initializers that has been verbatim copied from earlier compilations. */ - if (!flag_ltrans || decompress) + if ((!flag_ltrans || decompress) && section_type != LTO_section_lto) { /* Create a mapping header containing the underlying data and length, and prepend this to the uncompression buffer. The uncompressed data @@ -167,9 +167,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data, data = buffer.data + header_length; } - lto_check_version (((const lto_header *)data)->major_version, - ((const lto_header *)data)->minor_version, - file_data->file_name); return data; } diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c index c91e58f0465..7ae102164ef 100644 --- a/gcc/lto-section-out.c +++ b/gcc/lto-section-out.c @@ -285,8 +285,6 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob) /* Write the header which says how to decode the pieces of the t. */ memset (&header, 0, sizeof (struct lto_simple_header)); - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; header.main_size = ob->main_stream->total_size; lto_write_data (&header, sizeof header); diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c index dc68429303c..7dee770aa11 100644 --- a/gcc/lto-streamer-out.c +++ b/gcc/lto-streamer-out.c @@ -1974,10 +1974,6 @@ produce_asm (struct output_block *ob, tree fn) /* The entire header is stream computed here. */ memset (&header, 0, sizeof (struct lto_function_header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - if (section_type == LTO_section_function_body) header.cfg_size = ob->cfg_stream->total_size; header.main_size = ob->main_stream->total_size; @@ -2270,10 +2266,6 @@ lto_output_toplevel_asms (void) /* The entire header stream is computed here. */ memset (&header, 0, sizeof (header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; lto_write_data (&header, sizeof header); @@ -2390,6 +2382,29 @@ prune_offload_funcs (void) DECL_PRESERVE_P (fn_decl) = 1; } +/* Produce LTO section that contains global information + about LTO bytecode. */ + +static void +produce_lto_section () +{ + /* Stream LTO meta section. */ + output_block *ob = create_output_block (LTO_section_lto); + + char * section_name = lto_get_section_name (LTO_section_lto, NULL, NULL); + lto_begin_section (section_name, false); + free (section_name); + + lto_compression compression = ZLIB; + + bool slim_object = flag_generate_lto && !flag_fat_lto_objects; + lto_section s + = { LTO_major_version, LTO_minor_version, slim_object, compression, 0 }; + lto_write_data (&s, sizeof s); + lto_end_section (); + destroy_output_block (ob); +} + /* Main entry point from the pass manager. */ void @@ -2412,6 +2427,8 @@ lto_output (void) /* Initialize the streamer. */ lto_streamer_init (); + produce_lto_section (); + n_nodes = lto_symtab_encoder_size (encoder); /* Process only the functions with bodies. */ for (i = 0; i < n_nodes; i++) @@ -2827,10 +2844,6 @@ lto_write_mode_table (void) struct lto_simple_header_with_strings header; memset (&header, 0, sizeof (header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; lto_write_data (&header, sizeof header); @@ -2901,9 +2914,6 @@ produce_asm_for_decls (void) lto_output_decl_state_streams (ob, fn_out_state); } - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - /* Currently not used. This field would allow us to preallocate the globals vector, so that it need not be resized as it is extended. */ header.num_nodes = -1; diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h index 3b1e07b4bfc..2f481aebeb2 100644 --- a/gcc/lto-streamer.h +++ b/gcc/lto-streamer.h @@ -234,6 +234,7 @@ enum lto_section_type LTO_section_offload_table, LTO_section_mode_table, LTO_section_ipa_hsa, + LTO_section_lto, LTO_N_SECTION_TYPES /* Must be last. */ }; @@ -378,18 +379,31 @@ public: unsigned int len; }; +/* Compression algorithm used for compression of LTO bytecode. */ -/* The is the first part of the record for a function or constructor - in the .o file. */ -struct lto_header +enum lto_compression +{ + ZLIB, + ZSTD +}; + +/* Structure that represents LTO ELF section with information + about the format. */ + +struct lto_section { int16_t major_version; int16_t minor_version; + unsigned char slim_object: 1; + lto_compression compression: 4; + int32_t reserved0: 27; }; +STATIC_ASSERT (sizeof (lto_section) == 8); + /* The is the first part of the record in an LTO file for many of the IPA passes. */ -struct lto_simple_header : lto_header +struct lto_simple_header { /* Size of main gimple body of function. */ int32_t main_size; @@ -589,6 +603,9 @@ struct GTY(()) lto_file_decl_data /* Mode translation table. */ const unsigned char *mode_table; + + /* Read LTO section. */ + lto_section lto_section_header; }; typedef struct lto_file_decl_data *lto_file_decl_data_ptr; diff --git a/gcc/lto/lto-common.c b/gcc/lto/lto-common.c index 0d38ee6b57b..1cfbda0f554 100644 --- a/gcc/lto/lto-common.c +++ b/gcc/lto/lto-common.c @@ -2060,6 +2060,21 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file) #else file_data->mode_table = lto_mode_identity_table; #endif + + /* Read and verify LTO section. */ + data = lto_get_section_data (file_data, LTO_section_lto, NULL, &len, false); + if (data == NULL) + { + fatal_error (input_location, "bytecode stream in file %qs generated " + "with GCC compiler older than 10.0", file_data->file_name); + return; + } + + file_data->lto_section_header = *(const lto_section *)data; + lto_check_version (file_data->lto_section_header.major_version, + file_data->lto_section_header.minor_version, + file_data->file_name); + data = lto_get_section_data (file_data, LTO_section_decls, NULL, &len); if (data == NULL) { -- 2.21.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.lto section. 2019-06-27 12:28 ` [PATCH] Add .gnu.lto_.lto section Martin Liška @ 2019-07-01 10:59 ` Martin Liška 2019-07-01 11:01 ` [PATCH 2/2] Add zstd support for LTO bytecode compression Martin Liška 2019-07-02 20:49 ` [PATCH] Add .gnu.lto_.lto section Jeff Law 0 siblings, 2 replies; 32+ messages in thread From: Martin Liška @ 2019-07-01 10:59 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches [-- Attachment #1: Type: text/plain, Size: 122 bytes --] Hi. Ok, so there's a version with added ChangeLog that survives regression tests. Ready to be installed? Thanks, Martin [-- Attachment #2: 0001-Add-.gnu.lto_.lto-section.patch --] [-- Type: text/x-patch, Size: 8463 bytes --] From e6745583dc4b7f5543878c0a25498e818531f73e Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Fri, 21 Jun 2019 12:14:04 +0200 Subject: [PATCH 1/2] Add .gnu.lto_.lto section. gcc/ChangeLog: 2019-07-01 Martin Liska <mliska@suse.cz> * lto-section-in.c (lto_get_section_data): Add "lto" section. * lto-section-out.c (lto_destroy_simple_output_block): Never compress LTO_section_lto section. * lto-streamer-out.c (produce_asm): Do not set major_version and minor_version. (lto_output_toplevel_asms): Likewise. (produce_lto_section): New function. (lto_output): Call produce_lto_section. (lto_write_mode_table): Do not set major_version and minor_version. (produce_asm_for_decls): Likewise. * lto-streamer.h (enum lto_section_type): Add LTO_section_lto type. (struct lto_header): Remove. (struct lto_section): New struct. (struct lto_simple_header): Do not inherit from lto_header. (struct lto_file_decl_data): Add lto_section_header field. gcc/lto/ChangeLog: 2019-07-01 Martin Liska <mliska@suse.cz> * lto-common.c: Read LTO section and verify header. --- gcc/lto-section-in.c | 9 +++------ gcc/lto-section-out.c | 2 -- gcc/lto-streamer-out.c | 40 +++++++++++++++++++++++++--------------- gcc/lto-streamer.h | 25 +++++++++++++++++++++---- gcc/lto/lto-common.c | 15 +++++++++++++++ 5 files changed, 64 insertions(+), 27 deletions(-) diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c index 4cfc0cad4be..4e7d1181f23 100644 --- a/gcc/lto-section-in.c +++ b/gcc/lto-section-in.c @@ -52,10 +52,10 @@ const char *lto_section_name[LTO_N_SECTION_TYPES] = "icf", "offload_table", "mode_table", - "hsa" + "hsa", + "lto" }; - /* Hooks so that the ipa passes can call into the lto front end to get sections. */ @@ -146,7 +146,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data, /* WPA->ltrans streams are not compressed with exception of function bodies and variable initializers that has been verbatim copied from earlier compilations. */ - if (!flag_ltrans || decompress) + if ((!flag_ltrans || decompress) && section_type != LTO_section_lto) { /* Create a mapping header containing the underlying data and length, and prepend this to the uncompression buffer. The uncompressed data @@ -167,9 +167,6 @@ lto_get_section_data (struct lto_file_decl_data *file_data, data = buffer.data + header_length; } - lto_check_version (((const lto_header *)data)->major_version, - ((const lto_header *)data)->minor_version, - file_data->file_name); return data; } diff --git a/gcc/lto-section-out.c b/gcc/lto-section-out.c index c91e58f0465..7ae102164ef 100644 --- a/gcc/lto-section-out.c +++ b/gcc/lto-section-out.c @@ -285,8 +285,6 @@ lto_destroy_simple_output_block (struct lto_simple_output_block *ob) /* Write the header which says how to decode the pieces of the t. */ memset (&header, 0, sizeof (struct lto_simple_header)); - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; header.main_size = ob->main_stream->total_size; lto_write_data (&header, sizeof header); diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c index dc68429303c..7dee770aa11 100644 --- a/gcc/lto-streamer-out.c +++ b/gcc/lto-streamer-out.c @@ -1974,10 +1974,6 @@ produce_asm (struct output_block *ob, tree fn) /* The entire header is stream computed here. */ memset (&header, 0, sizeof (struct lto_function_header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - if (section_type == LTO_section_function_body) header.cfg_size = ob->cfg_stream->total_size; header.main_size = ob->main_stream->total_size; @@ -2270,10 +2266,6 @@ lto_output_toplevel_asms (void) /* The entire header stream is computed here. */ memset (&header, 0, sizeof (header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; lto_write_data (&header, sizeof header); @@ -2390,6 +2382,29 @@ prune_offload_funcs (void) DECL_PRESERVE_P (fn_decl) = 1; } +/* Produce LTO section that contains global information + about LTO bytecode. */ + +static void +produce_lto_section () +{ + /* Stream LTO meta section. */ + output_block *ob = create_output_block (LTO_section_lto); + + char * section_name = lto_get_section_name (LTO_section_lto, NULL, NULL); + lto_begin_section (section_name, false); + free (section_name); + + lto_compression compression = ZLIB; + + bool slim_object = flag_generate_lto && !flag_fat_lto_objects; + lto_section s + = { LTO_major_version, LTO_minor_version, slim_object, compression, 0 }; + lto_write_data (&s, sizeof s); + lto_end_section (); + destroy_output_block (ob); +} + /* Main entry point from the pass manager. */ void @@ -2412,6 +2427,8 @@ lto_output (void) /* Initialize the streamer. */ lto_streamer_init (); + produce_lto_section (); + n_nodes = lto_symtab_encoder_size (encoder); /* Process only the functions with bodies. */ for (i = 0; i < n_nodes; i++) @@ -2827,10 +2844,6 @@ lto_write_mode_table (void) struct lto_simple_header_with_strings header; memset (&header, 0, sizeof (header)); - /* Write the header. */ - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - header.main_size = ob->main_stream->total_size; header.string_size = ob->string_stream->total_size; lto_write_data (&header, sizeof header); @@ -2901,9 +2914,6 @@ produce_asm_for_decls (void) lto_output_decl_state_streams (ob, fn_out_state); } - header.major_version = LTO_major_version; - header.minor_version = LTO_minor_version; - /* Currently not used. This field would allow us to preallocate the globals vector, so that it need not be resized as it is extended. */ header.num_nodes = -1; diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h index 3b1e07b4bfc..f1593d64c71 100644 --- a/gcc/lto-streamer.h +++ b/gcc/lto-streamer.h @@ -234,6 +234,7 @@ enum lto_section_type LTO_section_offload_table, LTO_section_mode_table, LTO_section_ipa_hsa, + LTO_section_lto, LTO_N_SECTION_TYPES /* Must be last. */ }; @@ -378,18 +379,31 @@ public: unsigned int len; }; +/* Compression algorithm used for compression of LTO bytecode. */ -/* The is the first part of the record for a function or constructor - in the .o file. */ -struct lto_header +enum lto_compression +{ + ZLIB, + ZSTD +}; + +/* Structure that represents LTO ELF section with information + about the format. */ + +struct lto_section { int16_t major_version; int16_t minor_version; + unsigned char slim_object: 1; + lto_compression compression: 4; + int32_t reserved0: 27; }; +STATIC_ASSERT (sizeof (lto_section) == 8); + /* The is the first part of the record in an LTO file for many of the IPA passes. */ -struct lto_simple_header : lto_header +struct lto_simple_header { /* Size of main gimple body of function. */ int32_t main_size; @@ -589,6 +603,9 @@ struct GTY(()) lto_file_decl_data /* Mode translation table. */ const unsigned char *mode_table; + + /* Read LTO section. */ + lto_section lto_section_header; }; typedef struct lto_file_decl_data *lto_file_decl_data_ptr; diff --git a/gcc/lto/lto-common.c b/gcc/lto/lto-common.c index 0d38ee6b57b..1cfbda0f554 100644 --- a/gcc/lto/lto-common.c +++ b/gcc/lto/lto-common.c @@ -2060,6 +2060,21 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file) #else file_data->mode_table = lto_mode_identity_table; #endif + + /* Read and verify LTO section. */ + data = lto_get_section_data (file_data, LTO_section_lto, NULL, &len, false); + if (data == NULL) + { + fatal_error (input_location, "bytecode stream in file %qs generated " + "with GCC compiler older than 10.0", file_data->file_name); + return; + } + + file_data->lto_section_header = *(const lto_section *)data; + lto_check_version (file_data->lto_section_header.major_version, + file_data->lto_section_header.minor_version, + file_data->file_name); + data = lto_get_section_data (file_data, LTO_section_decls, NULL, &len); if (data == NULL) { -- 2.22.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 2/2] Add zstd support for LTO bytecode compression. 2019-07-01 10:59 ` Martin Liška @ 2019-07-01 11:01 ` Martin Liška 2019-07-02 20:50 ` Jeff Law 2019-07-02 20:49 ` [PATCH] Add .gnu.lto_.lto section Jeff Law 1 sibling, 1 reply; 32+ messages in thread From: Martin Liška @ 2019-07-01 11:01 UTC (permalink / raw) To: Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, Jeff Law, GCC Development, GCC Patches, Joseph Myers [-- Attachment #1: Type: text/plain, Size: 206 bytes --] Hi. This is updated version of the zstd patch that should handle all what Joseph pointed out. Patch can bootstrap on x86_64-linux-gnu and survives regression tests. Ready to be installed? Thanks, Martin [-- Attachment #2: 0002-Add-zstd-support-for-LTO-bytecode-compression.patch --] [-- Type: text/x-patch, Size: 20134 bytes --] From 5d2006c9c4d481f4083d5a591327ee64847b0bf7 Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Thu, 20 Jun 2019 10:08:17 +0200 Subject: [PATCH 2/2] Add zstd support for LTO bytecode compression. gcc/ChangeLog: 2019-07-01 Martin Liska <mliska@suse.cz> * Makefile.in: Define ZSTD_LIB. * common.opt: Adjust compression level to support also zstd levels. * config.in: Regenerate. * configure: Likewise. * configure.ac: Add --with-zstd and --with-zstd-include options and detect ZSTD. * doc/install.texi: Mention zstd dependency. * gcc.c: Print supported LTO compression algorithms. * lto-compress.c (lto_normalized_zstd_level): Likewise. (lto_compression_zstd): Likewise. (lto_uncompression_zstd): Likewise. (lto_end_compression): Dispatch in between zlib and zstd. (lto_compression_zlib): Mark with ATTRIBUTE_UNUSED. (lto_uncompression_zlib): Make it static. * lto-compress.h (lto_end_uncompression): Fix GNU coding style. * lto-section-in.c (lto_get_section_data): Pass info about used compression. * lto-streamer-out.c: By default use zstd when possible. * timevar.def (TV_IPA_LTO_DECOMPRESS): Rename to decompression (TV_IPA_LTO_COMPRESS): Likewise for compression. --- gcc/Makefile.in | 4 +- gcc/common.opt | 4 +- gcc/config.in | 6 ++ gcc/configure | 163 ++++++++++++++++++++++++++++++++++++++++- gcc/configure.ac | 66 +++++++++++++++++ gcc/doc/install.texi | 6 ++ gcc/gcc.c | 5 ++ gcc/lto-compress.c | 141 +++++++++++++++++++++++++++++------ gcc/lto-compress.h | 3 +- gcc/lto-section-in.c | 2 +- gcc/lto-streamer-out.c | 4 + gcc/timevar.def | 4 +- 12 files changed, 378 insertions(+), 30 deletions(-) diff --git a/gcc/Makefile.in b/gcc/Makefile.in index d9e0885b96b..597dc01328b 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -1065,7 +1065,7 @@ BUILD_LIBDEPS= $(BUILD_LIBIBERTY) LIBS = @LIBS@ libcommon.a $(CPPLIB) $(LIBINTL) $(LIBICONV) $(LIBBACKTRACE) \ $(LIBIBERTY) $(LIBDECNUMBER) $(HOST_LIBS) BACKENDLIBS = $(ISLLIBS) $(GMPLIBS) $(PLUGINLIBS) $(HOST_LIBS) \ - $(ZLIB) + $(ZLIB) $(ZSTD_LIB) # Any system libraries needed just for GNAT. SYSLIBS = @GNAT_LIBEXC@ @@ -1076,6 +1076,8 @@ GNATMAKE = @GNATMAKE@ # Libs needed (at present) just for jcf-dump. LDEXP_LIB = @LDEXP_LIB@ +ZSTD_LIB = @ZSTD_LIB@ + # Likewise, for use in the tools that must run on this machine # even if we are cross-building GCC. BUILD_LIBS = $(BUILD_LIBIBERTY) diff --git a/gcc/common.opt b/gcc/common.opt index a1544d06824..3b71a36552b 100644 --- a/gcc/common.opt +++ b/gcc/common.opt @@ -1888,8 +1888,8 @@ Specify the algorithm to partition symbols and vars at linktime. ; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h. flto-compression-level= -Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 9) --flto-compression-level=<number> Use zlib compression level <number> for IL. +Common Joined RejectNegative UInteger Var(flag_lto_compression_level) Init(-1) IntegerRange(0, 19) +-flto-compression-level=<number> Use zlib/zstd compression level <number> for IL. flto-odr-type-merging Common Ignore diff --git a/gcc/config.in b/gcc/config.in index a718ceaf3da..13fd7959dd7 100644 --- a/gcc/config.in +++ b/gcc/config.in @@ -1926,6 +1926,12 @@ #endif +/* Define if you have a working <zstd.h> header file. */ +#ifndef USED_FOR_TARGET +#undef HAVE_ZSTD_H +#endif + + /* Define if isl is in use. */ #ifndef USED_FOR_TARGET #undef HAVE_isl diff --git a/gcc/configure b/gcc/configure index 955e9ccc09b..8c9f7742ac7 100755 --- a/gcc/configure +++ b/gcc/configure @@ -782,6 +782,8 @@ manext LIBICONV_DEP LTLIBICONV LIBICONV +ZSTD_LIB +ZSTD_INCLUDE DL_LIB LDEXP_LIB EXTRA_GCC_LIBS @@ -959,6 +961,9 @@ with_pkgversion with_bugurl enable_languages with_multilib_list +with_zstd +with_zstd_include +with_zstd_lib enable_rpath with_libiconv_prefix enable_sjlj_exceptions @@ -1783,6 +1788,12 @@ Optional Packages: --with-pkgversion=PKG Use PKG in the version string in place of "GCC" --with-bugurl=URL Direct users to URL to report a bug --with-multilib-list select multilibs (AArch64, SH and x86-64 only) + --with-zstd=PATH specify prefix directory for installed zstd library. + Equivalent to --with-zstd-include=PATH/include plus + --with-zstd-lib=PATH/lib + --with-zstd-include=PATH + specify directory for installed zstd include files + --with-zstd-lib=PATH specify directory for the installed zstd library --with-gnu-ld assume the C compiler uses GNU ld default=no --with-libiconv-prefix[=DIR] search for libiconv in DIR/include and DIR/lib --without-libiconv-prefix don't search for libiconv in includedir and libdir @@ -9838,6 +9849,154 @@ $as_echo "#define HAVE_INTTYPES_H 1" >>confdefs.h fi +# Look for the ZSTD package. +ZSTD_INCLUDE= +ZSTD_LIB= + + +ZSTD_CPPFLAGS= +ZSTD_LDFLAGS= + +# Check whether --with-zstd was given. +if test "${with_zstd+set}" = set; then : + withval=$with_zstd; +fi + + +# Check whether --with-zstd-include was given. +if test "${with_zstd_include+set}" = set; then : + withval=$with_zstd_include; +fi + + +# Check whether --with-zstd-lib was given. +if test "${with_zstd_lib+set}" = set; then : + withval=$with_zstd_lib; +fi + +case "x$with_zstd" in + x) ;; + xno) + ZSTD_INCLUDE=no + ZSTD_LIB=no + ;; + *) ZSTD_INCLUDE=$with_zstd/include + ZSTD_LIB=$with_zstd/lib + ;; +esac +if test "x$with_zstd_include" != x; then + ZSTD_INCLUDE=$with_zstd_include +fi +if test "x$with_zstd_lib" != x; then + ZSTD_LIB=$with_zstd_lib +fi +if test "x$ZSTD_INCLUDE" != x \ + && test "x$ZSTD_INCLUDE" != xno; then + ZSTD_CPPFLAGS=-I$ZSTD_INCLUDE +fi +if test "x$ZSTD_LIB" != x \ + && test "x$ZSTD_LIB" != xno; then + ZSTD_LDFLAGS=-L$ZSTD_LIB +fi + +CXXFLAGS="$CXXFLAGS $ZSTD_CPPFLAGS" +LDFLAGS="$LDFLAGS $ZSTD_LDFLAGS" + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for zstd.h" >&5 +$as_echo_n "checking for zstd.h... " >&6; } +if ${gcc_cv_header_zstd_h+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <zstd.h> +int +main () +{ + + ; + return 0; +} +_ACEOF +if ac_fn_cxx_try_compile "$LINENO"; then : + gcc_cv_header_zstd_h=yes +else + gcc_cv_header_zstd_h=no +fi +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $gcc_cv_header_zstd_h" >&5 +$as_echo "$gcc_cv_header_zstd_h" >&6; } +if test $gcc_cv_header_zstd_h = yes; then + +$as_echo "#define HAVE_ZSTD_H 1" >>confdefs.h + +fi + +# LTO can use zstd compression algorithm +save_LIBS="$LIBS" +LIBS= +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing ZSTD_compress" >&5 +$as_echo_n "checking for library containing ZSTD_compress... " >&6; } +if ${ac_cv_search_ZSTD_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_func_search_save_LIBS=$LIBS +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char ZSTD_compress (); +int +main () +{ +return ZSTD_compress (); + ; + return 0; +} +_ACEOF +for ac_lib in '' zstd; do + if test -z "$ac_lib"; then + ac_res="none required" + else + ac_res=-l$ac_lib + LIBS="-l$ac_lib $ac_func_search_save_LIBS" + fi + if ac_fn_cxx_try_link "$LINENO"; then : + ac_cv_search_ZSTD_compress=$ac_res +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext + if ${ac_cv_search_ZSTD_compress+:} false; then : + break +fi +done +if ${ac_cv_search_ZSTD_compress+:} false; then : + +else + ac_cv_search_ZSTD_compress=no +fi +rm conftest.$ac_ext +LIBS=$ac_func_search_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_search_ZSTD_compress" >&5 +$as_echo "$ac_cv_search_ZSTD_compress" >&6; } +ac_res=$ac_cv_search_ZSTD_compress +if test "$ac_res" != no; then : + test "$ac_res" = "none required" || LIBS="$ac_res $LIBS" + +fi + +ZSTD_LIB="$LIBS" +LIBS="$save_LIBS" + + for ac_func in times clock kill getrlimit setrlimit atoq \ @@ -18655,7 +18814,7 @@ else lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2 lt_status=$lt_dlunknown cat > conftest.$ac_ext <<_LT_EOF -#line 18658 "configure" +#line 18817 "configure" #include "confdefs.h" #if HAVE_DLFCN_H @@ -18761,7 +18920,7 @@ else lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2 lt_status=$lt_dlunknown cat > conftest.$ac_ext <<_LT_EOF -#line 18764 "configure" +#line 18923 "configure" #include "confdefs.h" #if HAVE_DLFCN_H diff --git a/gcc/configure.ac b/gcc/configure.ac index 066a6f4c958..137d5b469c3 100644 --- a/gcc/configure.ac +++ b/gcc/configure.ac @@ -1258,6 +1258,72 @@ if test $gcc_cv_header_inttypes_h = yes; then [Define if you have a working <inttypes.h> header file.]) fi +# Look for the ZSTD package. +ZSTD_INCLUDE= +ZSTD_LIB= +AC_SUBST(ZSTD_INCLUDE) +AC_SUBST(ZSTD_LIB) +ZSTD_CPPFLAGS= +ZSTD_LDFLAGS= +AC_ARG_WITH(zstd, + [AS_HELP_STRING([--with-zstd=PATH], + [specify prefix directory for installed zstd library. + Equivalent to --with-zstd-include=PATH/include + plus --with-zstd-lib=PATH/lib])]) +AC_ARG_WITH(zstd-include, + [AS_HELP_STRING([--with-zstd-include=PATH], + [specify directory for installed zstd include files])]) +AC_ARG_WITH(zstd-lib, + [AS_HELP_STRING([--with-zstd-lib=PATH], + [specify directory for the installed zstd library])]) +case "x$with_zstd" in + x) ;; + xno) + ZSTD_INCLUDE=no + ZSTD_LIB=no + ;; + *) ZSTD_INCLUDE=$with_zstd/include + ZSTD_LIB=$with_zstd/lib + ;; +esac +if test "x$with_zstd_include" != x; then + ZSTD_INCLUDE=$with_zstd_include +fi +if test "x$with_zstd_lib" != x; then + ZSTD_LIB=$with_zstd_lib +fi +if test "x$ZSTD_INCLUDE" != x \ + && test "x$ZSTD_INCLUDE" != xno; then + ZSTD_CPPFLAGS=-I$ZSTD_INCLUDE +fi +if test "x$ZSTD_LIB" != x \ + && test "x$ZSTD_LIB" != xno; then + ZSTD_LDFLAGS=-L$ZSTD_LIB +fi + +CXXFLAGS="$CXXFLAGS $ZSTD_CPPFLAGS" +LDFLAGS="$LDFLAGS $ZSTD_LDFLAGS" + +AC_MSG_CHECKING(for zstd.h) +AC_CACHE_VAL(gcc_cv_header_zstd_h, +[AC_COMPILE_IFELSE([AC_LANG_PROGRAM( +[[#include <zstd.h>]])], + [gcc_cv_header_zstd_h=yes], + [gcc_cv_header_zstd_h=no])]) +AC_MSG_RESULT($gcc_cv_header_zstd_h) +if test $gcc_cv_header_zstd_h = yes; then + AC_DEFINE(HAVE_ZSTD_H, 1, + [Define if you have a working <zstd.h> header file.]) +fi + +# LTO can use zstd compression algorithm +save_LIBS="$LIBS" +LIBS= +AC_SEARCH_LIBS(ZSTD_compress, zstd) +ZSTD_LIB="$LIBS" +LIBS="$save_LIBS" +AC_SUBST(ZSTD_LIB) + dnl Disabled until we have a complete test for buggy enum bitfields. dnl gcc_AC_C_ENUM_BF_UNSIGNED diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi index 29d047004fd..12372a8c1f5 100644 --- a/gcc/doc/install.texi +++ b/gcc/doc/install.texi @@ -396,6 +396,12 @@ built together with GCC. Alternatively, the @option{--with-isl} configure option should be used if isl is not installed in your default library search path. +@item zstd Library. + +Necessary to build GCC with zstd compression used for LTO bytecode. +The library is searched in your default library patch search. +Alternatively, the @option{--with-zstd} configure option should be used. + @end table @heading Tools/packages necessary for modifying GCC diff --git a/gcc/gcc.c b/gcc/gcc.c index 9bd65508b00..0c0a686eb9c 100644 --- a/gcc/gcc.c +++ b/gcc/gcc.c @@ -6791,6 +6791,11 @@ print_configuration (FILE *file) #endif fnotice (file, "Thread model: %s\n", thrmod); + fnotice (file, "Supported LTO compression algorithms: zlib"); +#ifdef HAVE_ZSTD_H + fnotice (file, " zstd"); +#endif + fnotice (file, "\n"); /* compiler_version is truncated at the first space when initialized from version string, so truncate version_string at the first space diff --git a/gcc/lto-compress.c b/gcc/lto-compress.c index 3287178f257..b925363ac71 100644 --- a/gcc/lto-compress.c +++ b/gcc/lto-compress.c @@ -35,6 +35,10 @@ along with GCC; see the file COPYING3. If not see #include "lto-compress.h" #include "timevar.h" +#ifdef HAVE_ZSTD_H +#include <zstd.h> +#endif + /* Compression stream structure, holds the flush callback and opaque token, the buffered data, and a note of whether compressing or uncompressing. */ @@ -92,6 +96,91 @@ lto_normalized_zlib_level (void) return level; } +/* Free the buffer and memory associated with STREAM. */ + +static void +lto_destroy_compression_stream (struct lto_compression_stream *stream) +{ + free (stream->buffer); + free (stream); +} + +#ifdef HAVE_ZSTD_H +/* Return a zstd compression level that zstd will not reject. Normalizes + the compression level from the command line flag, clamping non-default + values to the appropriate end of their valid range. */ + +static int +lto_normalized_zstd_level (void) +{ + int level = flag_lto_compression_level; + + if (level != ZSTD_CLEVEL_DEFAULT) + { + if (level < 1) + level = 1; + else if (level > ZSTD_maxCLevel ()) + level = ZSTD_maxCLevel (); + } + + return level; +} + +/* Compress STREAM using ZSTD algorithm. */ + +static void +lto_compression_zstd (struct lto_compression_stream *stream) +{ + unsigned char *cursor = (unsigned char *) stream->buffer; + size_t size = stream->bytes; + + timevar_push (TV_IPA_LTO_COMPRESS); + size_t const outbuf_length = ZSTD_compressBound (size); + char *outbuf = (char *) xmalloc (outbuf_length); + + size_t const csize = ZSTD_compress (outbuf, outbuf_length, cursor, size, + lto_normalized_zstd_level ()); + + if (ZSTD_isError (csize)) + internal_error ("compressed stream: %s", ZSTD_getErrorName (csize)); + + stream->callback (outbuf, csize, NULL); + + lto_destroy_compression_stream (stream); + free (outbuf); + timevar_pop (TV_IPA_LTO_COMPRESS); +} + +/* Uncompress STREAM using ZSTD algorithm. */ + +static void +lto_uncompression_zstd (struct lto_compression_stream *stream) +{ + unsigned char *cursor = (unsigned char *) stream->buffer; + size_t size = stream->bytes; + + timevar_push (TV_IPA_LTO_DECOMPRESS); + unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size); + if (rsize == ZSTD_CONTENTSIZE_ERROR) + internal_error ("original not compressed with zstd"); + else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN) + internal_error ("original size unknown"); + + char *outbuf = (char *) xmalloc (rsize); + size_t const dsize = ZSTD_decompress (outbuf, rsize, cursor, size); + + if (ZSTD_isError (dsize)) + internal_error ("decompressed stream: %s", ZSTD_getErrorName (dsize)); + + stream->callback (outbuf, dsize, stream->opaque); + + lto_destroy_compression_stream (stream); + free (outbuf); + timevar_pop (TV_IPA_LTO_DECOMPRESS); +} + +#endif + /* Create a new compression stream, with CALLBACK flush function passed OPAQUE token, IS_COMPRESSION indicates if compressing or uncompressing. */ @@ -132,15 +221,6 @@ lto_append_to_compression_stream (struct lto_compression_stream *stream, stream->bytes += num_chars; } -/* Free the buffer and memory associated with STREAM. */ - -static void -lto_destroy_compression_stream (struct lto_compression_stream *stream) -{ - free (stream->buffer); - free (stream); -} - /* Return a new compression stream, with CALLBACK flush function passed OPAQUE token. */ @@ -163,10 +243,8 @@ lto_compress_block (struct lto_compression_stream *stream, lto_stats.num_output_il_bytes += num_chars; } -/* Finalize STREAM compression, and free stream allocations. */ - -void -lto_end_compression (struct lto_compression_stream *stream) +static void ATTRIBUTE_UNUSED +lto_compression_zlib (struct lto_compression_stream *stream) { unsigned char *cursor = (unsigned char *) stream->buffer; size_t remaining = stream->bytes; @@ -226,6 +304,16 @@ lto_end_compression (struct lto_compression_stream *stream) timevar_pop (TV_IPA_LTO_COMPRESS); } +void +lto_end_compression (struct lto_compression_stream *stream) +{ +#ifdef HAVE_ZSTD_H + lto_compression_zstd (stream); +#else + lto_compression_zlib (stream); +#endif +} + /* Return a new uncompression stream, with CALLBACK flush function passed OPAQUE token. */ @@ -248,14 +336,8 @@ lto_uncompress_block (struct lto_compression_stream *stream, lto_stats.num_input_il_bytes += num_chars; } -/* Finalize STREAM uncompression, and free stream allocations. - - Because of the way LTO IL streams are compressed, there may be several - concatenated compressed segments in the accumulated data, so for this - function we iterate decompressions until no data remains. */ - -void -lto_end_uncompression (struct lto_compression_stream *stream) +static void +lto_uncompression_zlib (struct lto_compression_stream *stream) { unsigned char *cursor = (unsigned char *) stream->buffer; size_t remaining = stream->bytes; @@ -318,3 +400,20 @@ lto_end_uncompression (struct lto_compression_stream *stream) free (outbuf); timevar_pop (TV_IPA_LTO_DECOMPRESS); } + +void +lto_end_uncompression (struct lto_compression_stream *stream, + lto_compression compression) +{ +#ifdef HAVE_ZSTD_H + if (compression == ZSTD) + { + lto_uncompression_zstd (stream); + return; + } +#endif + if (compression == ZSTD) + internal_error ("compiler does not support ZSTD LTO compression"); + + lto_uncompression_zlib (stream); +} diff --git a/gcc/lto-compress.h b/gcc/lto-compress.h index 87f53f7cd65..564b17e9f98 100644 --- a/gcc/lto-compress.h +++ b/gcc/lto-compress.h @@ -37,6 +37,7 @@ extern struct lto_compression_stream void *opaque); extern void lto_uncompress_block (struct lto_compression_stream *stream, const char *base, size_t num_chars); -extern void lto_end_uncompression (struct lto_compression_stream *stream); +extern void lto_end_uncompression (struct lto_compression_stream *stream, + lto_compression compression); #endif /* GCC_LTO_COMPRESS_H */ diff --git a/gcc/lto-section-in.c b/gcc/lto-section-in.c index 4e7d1181f23..80fdb03f4d4 100644 --- a/gcc/lto-section-in.c +++ b/gcc/lto-section-in.c @@ -161,7 +161,7 @@ lto_get_section_data (struct lto_file_decl_data *file_data, stream = lto_start_uncompression (lto_append_data, &buffer); lto_uncompress_block (stream, data, *len); - lto_end_uncompression (stream); + lto_end_uncompression (stream, file_data->lto_section_header.compression); *len = buffer.length - header_length; data = buffer.data + header_length; diff --git a/gcc/lto-streamer-out.c b/gcc/lto-streamer-out.c index 7dee770aa11..49ca5cecad1 100644 --- a/gcc/lto-streamer-out.c +++ b/gcc/lto-streamer-out.c @@ -2395,7 +2395,11 @@ produce_lto_section () lto_begin_section (section_name, false); free (section_name); +#ifdef HAVE_ZSTD_H + lto_compression compression = ZSTD; +#else lto_compression compression = ZLIB; +#endif bool slim_object = flag_generate_lto && !flag_fat_lto_objects; lto_section s diff --git a/gcc/timevar.def b/gcc/timevar.def index 13cb470b688..626ce493b76 100644 --- a/gcc/timevar.def +++ b/gcc/timevar.def @@ -78,8 +78,8 @@ DEFTIMEVAR (TV_IPA_INLINING , "ipa inlining heuristics") DEFTIMEVAR (TV_IPA_FNSPLIT , "ipa function splitting") DEFTIMEVAR (TV_IPA_COMDATS , "ipa comdats") DEFTIMEVAR (TV_IPA_OPT , "ipa various optimizations") -DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS , "lto stream inflate") -DEFTIMEVAR (TV_IPA_LTO_COMPRESS , "lto stream deflate") +DEFTIMEVAR (TV_IPA_LTO_DECOMPRESS , "lto stream decompression") +DEFTIMEVAR (TV_IPA_LTO_COMPRESS , "lto stream compression") DEFTIMEVAR (TV_IPA_LTO_OUTPUT , "lto stream output") DEFTIMEVAR (TV_IPA_LTO_GIMPLE_IN , "ipa lto gimple in") DEFTIMEVAR (TV_IPA_LTO_GIMPLE_OUT , "ipa lto gimple out") -- 2.22.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 2/2] Add zstd support for LTO bytecode compression. 2019-07-01 11:01 ` [PATCH 2/2] Add zstd support for LTO bytecode compression Martin Liška @ 2019-07-02 20:50 ` Jeff Law 0 siblings, 0 replies; 32+ messages in thread From: Jeff Law @ 2019-07-02 20:50 UTC (permalink / raw) To: Martin Liška, Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, GCC Development, GCC Patches, Joseph Myers On 7/1/19 5:00 AM, Martin LiÅ¡ka wrote: > Hi. > > This is updated version of the zstd patch that should handle all what Joseph > pointed out. > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > Ready to be installed? > Thanks, > Martin > > > 0002-Add-zstd-support-for-LTO-bytecode-compression.patch > > From 5d2006c9c4d481f4083d5a591327ee64847b0bf7 Mon Sep 17 00:00:00 2001 > From: Martin Liska <mliska@suse.cz> > Date: Thu, 20 Jun 2019 10:08:17 +0200 > Subject: [PATCH 2/2] Add zstd support for LTO bytecode compression. > > gcc/ChangeLog: > > 2019-07-01 Martin Liska <mliska@suse.cz> > > * Makefile.in: Define ZSTD_LIB. > * common.opt: Adjust compression level > to support also zstd levels. > * config.in: Regenerate. > * configure: Likewise. > * configure.ac: Add --with-zstd and --with-zstd-include options > and detect ZSTD. > * doc/install.texi: Mention zstd dependency. > * gcc.c: Print supported LTO compression algorithms. > * lto-compress.c (lto_normalized_zstd_level): Likewise. > (lto_compression_zstd): Likewise. > (lto_uncompression_zstd): Likewise. > (lto_end_compression): Dispatch in between zlib and zstd. > (lto_compression_zlib): Mark with ATTRIBUTE_UNUSED. > (lto_uncompression_zlib): Make it static. > * lto-compress.h (lto_end_uncompression): Fix GNU coding style. > * lto-section-in.c (lto_get_section_data): Pass info > about used compression. > * lto-streamer-out.c: By default use zstd when possible. > * timevar.def (TV_IPA_LTO_DECOMPRESS): Rename to decompression > (TV_IPA_LTO_COMPRESS): Likewise for compression. OK jeff ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH] Add .gnu.lto_.lto section. 2019-07-01 10:59 ` Martin Liška 2019-07-01 11:01 ` [PATCH 2/2] Add zstd support for LTO bytecode compression Martin Liška @ 2019-07-02 20:49 ` Jeff Law 1 sibling, 0 replies; 32+ messages in thread From: Jeff Law @ 2019-07-02 20:49 UTC (permalink / raw) To: Martin Liška, Richard Biener Cc: Jan Hubicka, Thomas Koenig, Andrew Pinski, GCC Development, GCC Patches On 7/1/19 4:59 AM, Martin LiÅ¡ka wrote: > Hi. > > Ok, so there's a version with added ChangeLog that survives regression tests. > > Ready to be installed? > Thanks, > Martin > > > 0001-Add-.gnu.lto_.lto-section.patch > > From e6745583dc4b7f5543878c0a25498e818531f73e Mon Sep 17 00:00:00 2001 > From: Martin Liska <mliska@suse.cz> > Date: Fri, 21 Jun 2019 12:14:04 +0200 > Subject: [PATCH 1/2] Add .gnu.lto_.lto section. > > gcc/ChangeLog: > > 2019-07-01 Martin Liska <mliska@suse.cz> > > * lto-section-in.c (lto_get_section_data): Add "lto" section. > * lto-section-out.c (lto_destroy_simple_output_block): Never > compress LTO_section_lto section. > * lto-streamer-out.c (produce_asm): Do not set major_version > and minor_version. > (lto_output_toplevel_asms): Likewise. > (produce_lto_section): New function. > (lto_output): Call produce_lto_section. > (lto_write_mode_table): Do not set major_version and > minor_version. > (produce_asm_for_decls): Likewise. > * lto-streamer.h (enum lto_section_type): Add LTO_section_lto > type. > (struct lto_header): Remove. > (struct lto_section): New struct. > (struct lto_simple_header): Do not inherit from lto_header. > (struct lto_file_decl_data): Add lto_section_header field. OK jeff ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-20 11:42 ` Martin Liška 2019-06-20 12:02 ` Jan Hubicka @ 2019-06-20 12:12 ` Thomas Koenig 1 sibling, 0 replies; 32+ messages in thread From: Thomas Koenig @ 2019-06-20 12:12 UTC (permalink / raw) To: Martin Liška, Andrew Pinski, Jan Hubicka Cc: Richard Biener, Jeff Law, GCC Development, GCC Patches, fortran Hi Martin, > LTO bytecode is not supposed to be a distributable format. One of my dreams is to make libgfortran LTO-clean. There is a lot of performance to be gained both in I/O (where a huge number of special cases could be shortcut by LTO, because hardly any program uses them all) and in array intrinsics, where seeing through the array descriptors can also lead to large benefits. This is PR 77278. Once this is achieved, it would make sense to distribute libgfortran.a as a library of fat object files. Regards Thomas ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-20 9:08 ` Martin Liška 2019-06-20 10:59 ` Thomas Koenig @ 2019-06-20 17:02 ` Joseph Myers 1 sibling, 0 replies; 32+ messages in thread From: Joseph Myers @ 2019-06-20 17:02 UTC (permalink / raw) To: Martin Liška Cc: Andrew Pinski, Jan Hubicka, Richard Biener, Jeff Law, GCC Development, GCC Patches Any use of a host library should come with associated configure options to specify header and library paths for that library (and documentation for those options). (See existing --with-gmp*, --with-isl* etc. options.) -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-19 19:29 ` Jan Hubicka 2019-06-19 19:34 ` Andrew Pinski @ 2019-06-20 10:46 ` Segher Boessenkool 2019-06-20 11:44 ` Martin Liška 1 sibling, 1 reply; 32+ messages in thread From: Segher Boessenkool @ 2019-06-20 10:46 UTC (permalink / raw) To: Jan Hubicka Cc: Andrew Pinski, Richard Biener, Jeff Law, Martin Liška, GCC Development On Wed, Jun 19, 2019 at 09:29:54PM +0200, Jan Hubicka wrote: > > > > At least allow it to be built as part of the normal build like GMP, > > etc. are done. > > And include it in downloading using contrib/download_prerequisites > > like the libraries are done. > > Anoying detail is that zstd builds with cmake, not autotools It looks like it would be pretty trivial to integrate into a real build system though. OTOH, how well has this been tested, then? On different platforms, etc.? Segher ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC] zstd as a compression algorithm for LTO 2019-06-20 10:46 ` Segher Boessenkool @ 2019-06-20 11:44 ` Martin Liška 0 siblings, 0 replies; 32+ messages in thread From: Martin Liška @ 2019-06-20 11:44 UTC (permalink / raw) To: Segher Boessenkool, Jan Hubicka Cc: Andrew Pinski, Richard Biener, Jeff Law, GCC Development, Nathan Sidwell On 6/20/19 12:46 PM, Segher Boessenkool wrote: > OTOH, how well has this been tested, then? On different platforms, etc.? Dunno. The algorithm is quite new (4 years), but as it's becoming more popular. So I would expect coverage would be quite good. Maybe Nathan can comment on this? Martin ^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2019-07-02 20:50 UTC | newest] Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-06-19 9:21 [RFC] zstd as a compression algorithm for LTO Martin Liška 2019-06-19 16:03 ` Jeff Law 2019-06-19 18:55 ` Richard Biener 2019-06-19 19:26 ` Andrew Pinski 2019-06-19 19:29 ` Jan Hubicka 2019-06-19 19:34 ` Andrew Pinski 2019-06-20 9:08 ` Martin Liška 2019-06-20 10:59 ` Thomas Koenig 2019-06-20 11:42 ` Martin Liška 2019-06-20 12:02 ` Jan Hubicka 2019-06-21 10:20 ` [PATCH] Add .gnu.lto_.meta section Martin Liška 2019-06-21 12:34 ` Richard Biener 2019-06-21 12:49 ` Martin Liška 2019-06-21 12:57 ` Jan Hubicka 2019-06-21 14:01 ` Martin Liška 2019-06-24 12:02 ` Richard Biener 2019-06-24 12:12 ` Martin Liška 2019-06-24 12:44 ` Richard Biener 2019-06-24 13:31 ` Martin Liška 2019-06-24 14:25 ` Iain Sandoe 2019-06-24 18:05 ` Richard Biener 2019-06-25 8:14 ` Martin Liška 2019-06-25 14:15 ` Richard Biener 2019-06-27 12:28 ` [PATCH] Add .gnu.lto_.lto section Martin Liška 2019-07-01 10:59 ` Martin Liška 2019-07-01 11:01 ` [PATCH 2/2] Add zstd support for LTO bytecode compression Martin Liška 2019-07-02 20:50 ` Jeff Law 2019-07-02 20:49 ` [PATCH] Add .gnu.lto_.lto section Jeff Law 2019-06-20 12:12 ` [RFC] zstd as a compression algorithm for LTO Thomas Koenig 2019-06-20 17:02 ` Joseph Myers 2019-06-20 10:46 ` Segher Boessenkool 2019-06-20 11:44 ` Martin Liška
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).