[Bug lto/114505] New: static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[Bug lto/114505] New: static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[eschwartz93 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

            Bug ID: 114505
           Summary: static archives built with gcc --without-zstd cause
                    ICE when read by gcc with zstd support
           Product: gcc
           Version: 13.2.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: eschwartz93 at gmail dot com
  Target Milestone: ---

Created attachment 57824
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57824&action=edit
build log

FAILED: glib/libglib-2.0.so.0.7800.4 
x86_64-pc-linux-gnu-gcc -m32 -mfpmath=sse  -o glib/libglib-2.0.so.0.7800.4
glib/libglib-2.0.so.0.7800.4.p/deprecated_gallocator.c.o
glib/libglib-2.0.so.0.7800.4.p/deprecated_gcache.c.o glib/libglib-2.0.so.0.7800
.4.p/deprecated_gcompletion.c.o
glib/libglib-2.0.so.0.7800.4.p/deprecated_grel.c.o
glib/libglib-2.0.so.0.7800.4.p/deprecated_gthread-deprecated.c.o
glib/libglib-2.0.so.0.7800.4.p/garcbox.c.o glib/libglib-2.0.so.
0.7800.4.p/garray.c.o glib/libglib-2.0.so.0.7800.4.p/gasyncqueue.c.o
glib/libglib-2.0.so.0.7800.4.p/gatomic.c.o
glib/libglib-2.0.so.0.7800.4.p/gbacktrace.c.o
glib/libglib-2.0.so.0.7800.4.p/gbase64.c.o glib/libgl
ib-2.0.so.0.7800.4.p/gbitlock.c.o
glib/libglib-2.0.so.0.7800.4.p/gbookmarkfile.c.o
glib/libglib-2.0.so.0.7800.4.p/gbytes.c.o
glib/libglib-2.0.so.0.7800.4.p/gcharset.c.o
glib/libglib-2.0.so.0.7800.4.p/gchecksum.c
.o glib/libglib-2.0.so.0.7800.4.p/gconvert.c.o
glib/libglib-2.0.so.0.7800.4.p/gdataset.c.o
glib/libglib-2.0.so.0.7800.4.p/gdate.c.o
glib/libglib-2.0.so.0.7800.4.p/gdatetime.c.o glib/libglib-2.0.so.0.7800.4.p/gdi
r.c.o glib/libglib-2.0.so.0.7800.4.p/genviron.c.o
glib/libglib-2.0.so.0.7800.4.p/gerror.c.o
glib/libglib-2.0.so.0.7800.4.p/gfileutils.c.o
glib/libglib-2.0.so.0.7800.4.p/ggettext.c.o glib/libglib-2.0.so.0.7800.4.
p/ghash.c.o glib/libglib-2.0.so.0.7800.4.p/ghmac.c.o
glib/libglib-2.0.so.0.7800.4.p/ghook.c.o
glib/libglib-2.0.so.0.7800.4.p/ghostutils.c.o
glib/libglib-2.0.so.0.7800.4.p/giochannel.c.o glib/libglib-2.0.so.0.780
0.4.p/gkeyfile.c.o glib/libglib-2.0.so.0.7800.4.p/glib-init.c.o
glib/libglib-2.0.so.0.7800.4.p/glib-private.c.o
glib/libglib-2.0.so.0.7800.4.p/glist.c.o
glib/libglib-2.0.so.0.7800.4.p/gmain.c.o glib/libglib-2.0.
so.0.7800.4.p/gmappedfile.c.o glib/libglib-2.0.so.0.7800.4.p/gmarkup.c.o
glib/libglib-2.0.so.0.7800.4.p/gmem.c.o
glib/libglib-2.0.so.0.7800.4.p/gmessages.c.o
glib/libglib-2.0.so.0.7800.4.p/gnode.c.o glib/libglib
-2.0.so.0.7800.4.p/goption.c.o glib/libglib-2.0.so.0.7800.4.p/gpathbuf.c.o
glib/libglib-2.0.so.0.7800.4.p/gpattern.c.o
glib/libglib-2.0.so.0.7800.4.p/gpoll.c.o
glib/libglib-2.0.so.0.7800.4.p/gprimes.c.o glib/lib
glib-2.0.so.0.7800.4.p/gqsort.c.o glib/libglib-2.0.so.0.7800.4.p/gquark.c.o
glib/libglib-2.0.so.0.7800.4.p/gqueue.c.o
glib/libglib-2.0.so.0.7800.4.p/grand.c.o
glib/libglib-2.0.so.0.7800.4.p/grcbox.c.o glib/libgl
ib-2.0.so.0.7800.4.p/grefcount.c.o
glib/libglib-2.0.so.0.7800.4.p/grefstring.c.o
glib/libglib-2.0.so.0.7800.4.p/gregex.c.o
glib/libglib-2.0.so.0.7800.4.p/gscanner.c.o
glib/libglib-2.0.so.0.7800.4.p/gsequence.c.o
 glib/libglib-2.0.so.0.7800.4.p/gshell.c.o
glib/libglib-2.0.so.0.7800.4.p/gslice.c.o
glib/libglib-2.0.so.0.7800.4.p/gslist.c.o
glib/libglib-2.0.so.0.7800.4.p/gstdio.c.o
glib/libglib-2.0.so.0.7800.4.p/gstrfuncs.c
.o glib/libglib-2.0.so.0.7800.4.p/gstring.c.o
glib/libglib-2.0.so.0.7800.4.p/gstringchunk.c.o
glib/libglib-2.0.so.0.7800.4.p/gstrvbuilder.c.o
glib/libglib-2.0.so.0.7800.4.p/gtestutils.c.o glib/libglib-2.0.so.0.7
800.4.p/gthread.c.o glib/libglib-2.0.so.0.7800.4.p/gthreadpool.c.o
glib/libglib-2.0.so.0.7800.4.p/gtimer.c.o
glib/libglib-2.0.so.0.7800.4.p/gtimezone.c.o
glib/libglib-2.0.so.0.7800.4.p/gtrace.c.o glib/libglib-2.
0.so.0.7800.4.p/gtranslit.c.o glib/libglib-2.0.so.0.7800.4.p/gtrashstack.c.o
glib/libglib-2.0.so.0.7800.4.p/gtree.c.o
glib/libglib-2.0.so.0.7800.4.p/guniprop.c.o
glib/libglib-2.0.so.0.7800.4.p/gutf8.c.o glib/lib
glib-2.0.so.0.7800.4.p/gunibreak.c.o
glib/libglib-2.0.so.0.7800.4.p/gunicollate.c.o
glib/libglib-2.0.so.0.7800.4.p/gunidecomp.c.o
glib/libglib-2.0.so.0.7800.4.p/guri.c.o
glib/libglib-2.0.so.0.7800.4.p/gutils.c.o
 glib/libglib-2.0.so.0.7800.4.p/guuid.c.o
glib/libglib-2.0.so.0.7800.4.p/gvariant.c.o
glib/libglib-2.0.so.0.7800.4.p/gvariant-core.c.o
glib/libglib-2.0.so.0.7800.4.p/gvariant-parser.c.o glib/libglib-2.0.so.0.780
0.4.p/gvariant-serialiser.c.o
glib/libglib-2.0.so.0.7800.4.p/gvarianttypeinfo.c.o
glib/libglib-2.0.so.0.7800.4.p/gvarianttype.c.o
glib/libglib-2.0.so.0.7800.4.p/gversion.c.o
glib/libglib-2.0.so.0.7800.4.p/gwakeu
p.c.o glib/libglib-2.0.so.0.7800.4.p/gprintf.c.o
glib/libglib-2.0.so.0.7800.4.p/glib-unix.c.o
glib/libglib-2.0.so.0.7800.4.p/gspawn.c.o
glib/libglib-2.0.so.0.7800.4.p/giounix.c.o glib/libglib-2.0.so.0.7800.4.p/g
journal-private.c.o glib/libglib-2.0.so.0.7800.4.p/gthread-posix.c.o -flto
-Wl,--as-needed -Wl,--no-undefined -shared -fPIC -Wl,--start-group
-Wl,-soname,libglib-2.0.so.0 -march=x86-64-v3 -O2 -pipe -Werror=odr -
Werror=lto-type-mismatch -Werror=strict-aliasing -Wl,-O1 -Wl,--as-needed
-Wl,-z,pack-relative-relocs glib/libcharset/libcharset.a -Wl,-z,nodelete
-Wl,-Bsymbolic-functions -lm /usr/lib/libsysprof-capture-4.a -pth
read /usr/lib/libpcre2-8.so -Wl,--end-group
lto1: internal compiler error: original not compressed with zstd
0x1abe950 internal_error(char const*, ...)
        ???:0
0xa07c44 lto_end_uncompression(lto_compression_stream*, lto_compression)
        ???:0
0xa021ea lto_get_section_data(lto_file_decl_data*, lto_section_type, char
const*, int, unsigned long*, bool)
        ???:0
0x7301ad cgraph_node::get_untransformed_body()
        ???:0
0x73ed9a cgraph_node::expand()
        ???:0
0x6ab38a lto_main()
        ???:0
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
make: *** [/var/tmp/portage/dev-libs/glib-2.78.4-r1/temp/ccd4PAIc.mk:41:
/var/tmp/portage/dev-libs/glib-2.78.4-r1/temp/ccQNyd3y.ltrans13.ltrans.o] Error
1
make: *** Waiting for unfinished jobs....
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld:
error: lto-wrapper failed
collect2: error: ld returned 1 exit status



This is not ideal. If the original was not compressed with zstd then the
compiler should decompress it with zlib instead. This issue showed up on Gentoo
when compiling the system with LTO, then enabling the zstd option and
rebuilding gcc, then attempting to build more programs.

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[xry111 at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

Xi Ruoyao <xry111 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2024-03-28
                 CC|                            |xry111 at gcc dot gnu.org
     Ever confirmed|0                           |1
           Keywords|                            |lto

--- Comment #1 from Xi Ruoyao <xry111 at gcc dot gnu.org> ---
Confirmed as I've seen it before.  At least we should turn it into a "normal"
error message instead of ICE.

But note that even "downgrading the compression algorithm" is supported, it
won't really help rolling-release distros.  When GCC itself is upgraded with a
LTO API version change, all previous LTO static archives will be unusable
anyway.  Generally the distros should not ship LTO static archives at all.

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|1                           |0
             Status|NEW                         |UNCONFIRMED
   Last reconfirmed|2024-03-28 00:00:00         |

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---


On the output side:
#ifdef HAVE_ZSTD_H
  lto_compression compression = ZSTD;
#else
  lto_compression compression = ZLIB;
#endif

  bool slim_object = flag_generate_lto && !flag_fat_lto_objects;
  lto_section s
    = { LTO_major_version, LTO_minor_version, slim_object, 0, 0 };
  s.set_compression (compression);

(flags of lto_section is just set to compression)


On the input side:
Wait this should not happen.

      lto_end_uncompression (stream,
                             file_data->lto_section_header.get_compression ());

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2024-03-28
     Ever confirmed|0                           |1

--- Comment #3 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
THis should not case should not being ICE and should be handled correctly. I am
thinking something else is wrong.

Now the opposite direction is definitely should be a fatal_error instead of an
internal_error.

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #4 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
For the opposite direction, this is the fix:
```
diff --git a/gcc/lto-compress.cc b/gcc/lto-compress.cc
index c167ac967aa..1475674e7ac 100644
--- a/gcc/lto-compress.cc
+++ b/gcc/lto-compress.cc
@@ -408,7 +408,7 @@ lto_end_uncompression (struct lto_compression_stream
*stream,
     }
 #endif
   if (compression == ZSTD)
-    internal_error ("compiler does not support ZSTD LTO compression");
+    fatal_error ("compiler does not support ZSTD LTO compression");

   lto_uncompression_zlib (stream);
 }
```

But this direction should work.

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[xry111 at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

Xi Ruoyao <xry111 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |needs-reduction

--- Comment #5 from Xi Ruoyao <xry111 at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #3)
> THis should not case should not being ICE and should be handled correctly. I
> am thinking something else is wrong.
> 
> Now the opposite direction is definitely should be a fatal_error instead of
> an internal_error.

I'm pretty sure I've seen the ICE before (when I forgot to strip the LTO
sections from a fat LTO archive built by a "GCC w/o zstd" to produce a non-LTO
archive).

Just need to find a test case...

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #6 from Sam James <sjames at gcc dot gnu.org> ---
I'm confident I've seen this happen before but I haven't got a quick testcase
(it should be easy to make one, but I don't have a gcc w/ zstd right this
second.)

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #7 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Something else must be going wrong here. Maybe a mistmatched version of ZSTD
which are incompatible somehow. Maybe ZSTD being miscompiled in one case.

The original ICE listed in this bug report should not happen unless there is
something wrong with ZSTD.

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #8 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
OE (Yocto) ran into this:
https://www.mail-archive.com/openembedded-core@lists.openembedded.org/msg158289.html

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://github.com/openwrt/
                   |                            |openwrt/issues/10832

--- Comment #9 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
openwrt ran into this too.

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #10 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
https://www.reddit.com/r/Gentoo/comments/twtxhc/problems_compiling_gjs_1712_on_amd64/

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #11 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
https://wiki.gentoo.org/wiki/QEMU#lto1:_internal_compiler_error:_original_not_compressed_with_zstd

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[eschwartz93 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #12 from Eli Schwartz <eschwartz93 at gmail dot com> ---
(In reply to Xi Ruoyao from comment #1)
> Confirmed as I've seen it before.  At least we should turn it into a
> "normal" error message instead of ICE.
> 
> But note that even "downgrading the compression algorithm" is supported, it
> won't really help rolling-release distros.  When GCC itself is upgraded with
> a LTO API version change, all previous LTO static archives will be unusable
> anyway.  Generally the distros should not ship LTO static archives at all.

Agreed, figuring out how to solve that is an open ticket too:
https://bugs.gentoo.org/926120

Still would like to solve the zstd case :D especially since I am sure given the
flexibility of a from-source distro there will be users who insist they'd
rather rebuild the entire OS on a GCC upgrade, so reducing the compression
algorithm edge case will come in use.


(In reply to Andrew Pinski from comment #2)
> On the input side:
> Wait this should not happen.
> 
>       lto_end_uncompression (stream,
>                              file_data->lto_section_header.get_compression
> ());


I cannot swear this was the case but the error very specifically occurred as
part of a Gentoo profile change that caused zstd to be enabled.

The version of zstd should be the same, at least...

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #13 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
I understand that this might be the reason why others have enabled zstd always
but reading the code this case should be supported.

That is LTO generated from a GCC without ZSTD support should be supported with
one compiled with it.

Plus:
/* Compression algorithm used for compression of LTO bytecode.  */

enum lto_compression
{
  ZLIB,
  ZSTD
};


ZSTD is 1 so it has to be something corrupting the field ...

[Bug lto/114505] static archives built with gcc --without-zstd cause ICE when read by gcc with zstd support

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114505

--- Comment #14 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #4)
> For the opposite direction, this is the fix:
> ```
> diff --git a/gcc/lto-compress.cc b/gcc/lto-compress.cc
> index c167ac967aa..1475674e7ac 100644
> --- a/gcc/lto-compress.cc
> +++ b/gcc/lto-compress.cc
> @@ -408,7 +408,7 @@ lto_end_uncompression (struct lto_compression_stream
> *stream,
>      }
>  #endif
>    if (compression == ZSTD)
> -    internal_error ("compiler does not support ZSTD LTO compression");
> +    fatal_error ("compiler does not support ZSTD LTO compression");
> 
>    lto_uncompression_zlib (stream);
>  }
> ```
> 
> But this direction should work.

This looks obvious to push, btw.  And yes, this was supposed to be "supported"