[Bug lto/101473] New: LTO makes debug info depend on toolchain path

[Bug lto/101473] New: LTO makes debug info depend on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

            Bug ID: 101473
           Summary: LTO makes debug info depend on toolchain path
           Product: gcc
           Version: 10.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: tonyb at cybernetics dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

When compiling with -flto -g, the resulting debug info varies depending on the
install path of the compiler toolchain.  This is a problem for reproducible
builds (https://reproducible-builds.org/) when the compiler toolchain is
installed in the user home directory.  For example, when compiling a file
lto_test.c with the same toolchain copied into two different paths, readelf -a
shows the following:

toolchain path 1:
23: 00000000000005f5     0 NOTYPE  WEAK   HIDDEN    28 lto_test.c.79b8ff9a

toolchain path 2:
42: 00000000000005f5     0 NOTYPE  WEAK   HIDDEN    28 lto_test.c.bba5e59b

The -frandom-seed flag does not help in this case.  The output is deterministic
(same output across multiple runs).

With -flto disabled, the problem goes away.  Non-LTO binaries are identical
regardless of the compiler toolchain path.

Stripping the debug info from the LTO binaries makes them identical, but using
"objcopy --add-gnu-debuglink" makes them different again because the embedded
checksum for the debug info is different.

I encountered this problem trying to make reproducible builds with -flto under
Yocto.  Here is the Yocto bug report, including test code for Yocto:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14481

Regarding the "LTO reproducible build test" attachment in the Yocto bug report:
Yocto sets up the compiler toolchain in a "recipe-sysroot-native" directory and
the system lib/, include/, and such in a "recipe-sysroot" directory, which are
prepared by running "bitbake lto-test".  Then the run-lto-test script makes two
copies of the compiler toolchain under two differnet names using "cp -rl",
compiles the simple test program using both toolchains, and compares them.

Comment 6 on this bug may be related:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66305#c6

[Bug lto/101473] LTO makes debug info depend on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rguenth at gcc dot gnu.org

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
The symbol name is computed by appending the DWARF CU chceksum to the base
filename of the main TU.  That means the actual DWARF is different.  Note
the symbol(s) could be removed after the link editing step but there's no easy
way to do that.  Still the actual DWARF would be different.

I'll see if I can reproduce the issue.

[Bug lto/101473] LTO makes debug info depend on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2021-07-19
     Ever confirmed|0                           |1

--- Comment #2 from Richard Biener <rguenth at gcc dot gnu.org> ---
I can't reproduce it with install locations /tmp/install_one and
/tmp/install_alternate_two and doing compilation in /tmp with

> cat t.c
int main()
{
  __builtin_puts ("Hello World\n");
  return 0;
}
> ./install_one/usr/local/bin/gcc t.c -g -flto -o t
> cp t t1
> ./install_alternate_two/usr/local/bin/gcc t.c -g -flto -o t
> cp t t2

readelf -w and the symbol name are identical.

Can you share steps how to reproduce?  I only tried with current master for
now.

[Bug lto/101473] LTO makes debug info depend on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #3 from tonyb at cybernetics dot com ---
The output is reproducible for me with that test program too.  Try this program
instead:

#include <stdlib.h>
#include <stdio.h>
int main(int argc, char **argv)
{
   if (freopen("/dev/null", "w", stdout) == NULL) return 1;
   return 0;
}

[Bug lto/101473] LTO makes debug info depend on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #4 from tonyb at cybernetics dot com ---
By "reproducible" I meant that with your test program, the binary output was
the same regardless of the toolchain path as in
https://reproducible-builds.org/, so the "problem" was not reproduced for me
with your test program.

[Bug lto/101473] LTO makes debug info depend on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW

--- Comment #5 from Richard Biener <rguenth at gcc dot gnu.org> ---
Confirmed.  The difference is

--- t2.g        2021-07-19 15:17:18.856725031 +0200
+++ t1.g        2021-07-19 15:17:20.632748927 +0200
@@ -1387,20 +1387,20 @@
  The Directory Table (offset 0x167, lines 5, columns 1):
   Entry        Name
   0    (indirect line string, offset: 0x0): /tmp
-  1    (indirect line string, offset: 0x16):
/tmp/install_alternate_two/usr/local/lib64/gcc/x86_64-pc-linux-gnu/12.0.0/include
-  2    (indirect line string, offset: 0x82): /usr/include/bits/types
-  3    (indirect line string, offset: 0x70): /usr/include/bits
-  4    (indirect line string, offset: 0x9a): /usr/include
+  1    (indirect line string, offset: 0x55):
/tmp/install_one/usr/local/lib64/gcc/x86_64-pc-linux-gnu/12.0.0/include
+  2    (indirect line string, offset: 0x3d): /usr/include/bits/types
+  3    (indirect line string, offset: 0x2b): /usr/include/bits
+  4    (indirect line string, offset: 0x16): /usr/include

where stddef.h is included from.  I wonder how we can avoid this difference
without making the DWARF "invalid" though.  stddef.h is provided by the
compiler.  Btw, this doesn't neeed -flto to trigger it's visible with
plain -g as well, it just doesn't end up in differences in the symbol
table without -flto but the debug info still differs.

[Bug lto/101473] LTO makes debug info depend on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #6 from tonyb at cybernetics dot com ---
The only thing I know is that roughly similar issues were handled with the
-fdebug-prefix-map=old=new switch, but I am not the expert.

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|LTO makes debug info depend |debug_line info depends on
                   |on toolchain path           |toolchain path
                 CC|                            |jakub at gcc dot gnu.org
          Component|lto                         |debug

--- Comment #7 from Richard Biener <rguenth at gcc dot gnu.org> ---
Easily pulled in via

typedef __SIZE_TYPE__ size_t;

in said include.  Maybe expanding the list of basetypes could help here...

 <1><2e>: Abbrev Number: 4 (DW_TAG_typedef)
    <2f>   DW_AT_name        : (indirect string, offset: 0x2f): size_t
    <33>   DW_AT_decl_file   : 2
    <34>   DW_AT_decl_line   : 209
    <35>   DW_AT_decl_column : 23
    <36>   DW_AT_type        : <0x3a>
 <1><3a>: Abbrev Number: 2 (DW_TAG_base_type)
    <3b>   DW_AT_byte_size   : 8
    <3c>   DW_AT_encoding    : 7        (unsigned)
    <3d>   DW_AT_name        : (indirect string, offset: 0x1ab): long unsigned
int

note there are other headers like omp.h which are also in install path
dependent directories.

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #8 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to tonyb from comment #6)
> The only thing I know is that roughly similar issues were handled with the
> -fdebug-prefix-map=old=new switch, but I am not the expert.

Yes, that works to strip the install location as well.  So not a bug?

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #9 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Richard Biener from comment #8)
> (In reply to tonyb from comment #6)
> > The only thing I know is that roughly similar issues were handled with the
> > -fdebug-prefix-map=old=new switch, but I am not the expert.
> 
> Yes, that works to strip the install location as well.  So not a bug?

But then -fdebug-prefix-map is applied after computing the CU checksum
and thus after producing the symbol that shows the difference with -flto.

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #10 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Richard Biener from comment #9)
> (In reply to Richard Biener from comment #8)
> > (In reply to tonyb from comment #6)
> > > The only thing I know is that roughly similar issues were handled with the
> > > -fdebug-prefix-map=old=new switch, but I am not the expert.
> > 
> > Yes, that works to strip the install location as well.  So not a bug?
> 
> But then -fdebug-prefix-map is applied after computing the CU checksum
> and thus after producing the symbol that shows the difference with -flto.

diff --git a/gcc/dwarf2out.c b/gcc/dwarf2out.c
index 82783c4968b..07ea7c5af0c 100644
--- a/gcc/dwarf2out.c
+++ b/gcc/dwarf2out.c
@@ -7064,7 +7064,7 @@ attr_checksum (dw_attr_node *at, struct md5_ctx *ctx, int
*mark)

     case dw_val_class_file:
     case dw_val_class_file_implicit:
-      CHECKSUM_STRING (AT_file (at)->filename);
+      CHECKSUM_STRING (remap_debug_filename (AT_file (at)->filename));
       break;

     case dw_val_class_data8:

might fix it (I wonder if we shouldn't simply remap once when we generate
the locations rather than in dozen places...).

[Bug debug/101473] debug_line info depends on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #11 from tonyb at cybernetics dot com ---
That change fixed the simple test program.  Next I will try a more complex
program, but it will take a while for everything to compile.

[Bug debug/101473] debug_line info depends on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #12 from tonyb at cybernetics dot com ---
The patch fixed my own programs, so I rebuilt all of Yocto with -flto in two
different directories.  I found that most shared libraries in /lib and /usr/lib
still have the problem (i.e. are still not binary reproducible), as well as
binaries from pciutils, e2fsprogs, dropbear, sysvinit, hdparm, and a few other
packages (they are all binary reproducible without -flto).  However, the patch
certainly did improve things, since some binaries that were different
previously are now the same.

[Bug debug/101473] debug_line info depends on toolchain path

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #13 from rguenther at suse dot de <rguenther at suse dot de> ---
On Mon, 19 Jul 2021, tonyb at cybernetics dot com wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473
> 
> --- Comment #12 from tonyb at cybernetics dot com ---
> The patch fixed my own programs, so I rebuilt all of Yocto with -flto in two
> different directories.  I found that most shared libraries in /lib and /usr/lib
> still have the problem (i.e. are still not binary reproducible), as well as
> binaries from pciutils, e2fsprogs, dropbear, sysvinit, hdparm, and a few other
> packages (they are all binary reproducible without -flto).  However, the patch
> certainly did improve things, since some binaries that were different
> previously are now the same.

OK, I will try to produce a patch that should catch a few more cases
and would be actually mergeable.  If issues remain with that some
testcases would be nice.  You should be able to see the difference
in the .o files symbol table already if it's the same underlying issue
(different DWARF checksum).

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |rguenth at gcc dot gnu.org

--- Comment #14 from Richard Biener <rguenth at gcc dot gnu.org> ---
Created attachment 51173
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=51173&action=edit
patch tested

I have successfully tested this patch.

[Bug debug/101473] debug_line info depends on toolchain path

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #15 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Richard Biener <rguenth@gcc.gnu.org>:

https://gcc.gnu.org/g:7cc2df084b7977653a9b59cbc34a9ad500ae619c

commit r12-2412-g7cc2df084b7977653a9b59cbc34a9ad500ae619c
Author: Richard Biener <rguenther@suse.de>
Date:   Tue Jul 20 11:00:33 2021 +0200

    debug/101473 - apply debug prefix maps before checksumming DIEs

    The following makes sure to apply the debug prefix maps to filenames
    before checksumming DIEs to create the global symbol for the CU DIE
    used by LTO to link the late debug to the early debug.  This avoids
    binary differences (in said symbol) when compiling with toolchains
    installed under a different path and that compensated with appropriate
    -fdebug-prefix-map options.

    The easiest and most scalable way is to record both the unmapped
    and the remapped filename in the dwarf_file_data so the remapping
    process takes place at a single point and only once (otherwise it
    creates GC garbage at each point doing that).

    2021-07-20  Richard Biener  <rguenther@suse.de>

            PR debug/101473
            * dwarf2out.h (dwarf_file_data): Add key member.
            * dwarf2out.c (dwarf_file_hasher::equal): Compare key.
            (dwarf_file_hasher::hash): Hash key.
            (lookup_filename): Remap the filename and store it in the
            filename member of dwarf_file_data when creating a new
            dwarf_file_data.
            (file_name_acquire): Do not remap the filename again.
            (maybe_emit_file): Likewise.

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #16 from Richard Biener <rguenth at gcc dot gnu.org> ---
I'm interested if remaining differences are due to the same underlying issue or
not, so I'm for now leaving this PR open (it's too late to backport for 11.2
but could be backported for 11.3 though it isn't technically a regression).

[Bug debug/101473] debug_line info depends on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #17 from tonyb at cybernetics dot com ---
(In reply to Richard Biener from comment #16)
> I'm interested if remaining differences are due to the same underlying issue
> or not

I tested your updated patch with gcc 10.2 (I had to make a small change due to
the definition of struct dwarf_file_data being in dwarf2out.c rather than
dwarf2out.h in gcc 10.2).  I got the same results as before, with the same list
of files being non-reproducible.  I decided to debug the problem with pciutils.
 For pciutils, the problem was caused by the following:

Yocto CFLAGS contains -ffile-prefix-map but LDFLAGS doesn't.
The pciutils Makefile compiles *.c files to *.o files with CFLAGS and links
them with LDFLAGS but not CFLAGS.
With LTO disabled, this results in a reproducible binary, but with LTO enabled,
this results in a non-reproducible binary.

So I can workaround by adding -ffile-prefix-map to LDFLAGS.

So is this considered a bug?  If the prefix map translation can be handled
completely before generating the *.o file, then it could be fixed in gcc, but I
am not sure how it all works.  One solution that is apparently off the table is
embedding the prefix map in the *.o file, since that would make the *.o file
non-reproducible (see e.g. bug 68848; apparently the prefix map used to be
present in the *.o file but was removed in some recent version of gcc).

If this issue is not considered a bug, then I will pass on to the Yocto people
that they should use -ffile-prefix-map in LDFLAGS in addition to CFLAGS, and
you can close this PR.

I haven't looked into the other packages that were not reproducible, but I
suspect the same issue.

I will attach a test script that shows the issue if you want to debug it
further.

[Bug debug/101473] debug_line info depends on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #18 from tonyb at cybernetics dot com ---
Created attachment 51181
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=51181&action=edit
LTO reproducible build test: -ffile-prefix-map in LDFLAGS

tar xjf lto-test.tar.bz2
cd lto-test
./run-lto-test
FAIL - LTO build is not reproducible
edit run-lto-test and change ENABLE_FIX to 1
./run-lto-test
PASS - LTO build is reproducible

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #19 from Richard Biener <rguenth at gcc dot gnu.org> ---
I think it is a bug - -fdebug-prefix-map should be fully reflected in the LTO
IL and thus we should not end up creating un-remapped references to files (we
should not create many references to files late anyway).

In particular we're remapping all files when streaming locations to LTO.  But
yes, we're explicitely _not_ streaming the -f*-prefix-map arguments since
we'd have a hard time using them when they do not agree between TUs.

Your lto-test shows the difference is in the LTRANS DWARF CU header:

@@ -215,7 +215,7 @@
     <9b>   DW_AT_producer    : (indirect string, offset: 0x1c7): GNU GIMPLE
10.3.0 -mtune=generic -march=x86-64 -g -O2 -fno-openmp -fno-openacc -fno-pie
-fcf-protection=none -ffat-lto-objects -fuse-linker-plugin -fltrans
     <9f>   DW_AT_language    : 12      (ANSI C99)
     <a0>   DW_AT_name        : (indirect string, offset: 0x1b2): <artificial>
-    <a4>   DW_AT_comp_dir    : (indirect string, offset: 0x263):
/tmp/lto-test/src0
+    <a4>   DW_AT_comp_dir    : (indirect string, offset: 0x263):
/tmp/lto-test/src1
     <a8>   DW_AT_ranges      : 0x40
     <ac>   DW_AT_low_pc      : 0x0
     <b4>   DW_AT_stmt_list   : 0xe8

and with DWARF5 in addition in the directory table:

  The Directory Table (offset 0x10a, lines 1, columns 1):
   Entry        Name
-  0    (indirect line string, offset: 0xd): /tmp/lto-test/src0
+  0    (indirect line string, offset: 0xd): /tmp/lto-test/src1

so indeed the prefix remapping is needed at link-time - the link time
CWD aka DW_AT_comp_dir might also not agree with the CWD at compile time.

Thus my suggestion would be to indeed add -f*-prefix-map to LDFLAGS,
at least I can't think of a good solution to the above issue.  Eventually
picking a random TU and copying the CWD from its compile-time to the
link-time CUs would work.  Or somehow not specifying a compilation-dir
at all for those CUs?

[Bug debug/101473] debug_line info depends on toolchain path

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #20 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
I think in DWARF for compilation units DW_AT_comp_dir is not optional.

[Bug debug/101473] debug_line info depends on toolchain path

[tonyb at cybernetics dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

--- Comment #21 from tonyb at cybernetics dot com ---
I tried adding -f*-prefix-map to LDFLAGS in Yocto, and that makes everything I
tested binary reproducible, except for some shared libraries in /lib and
/usr/lib because libtool filters out the -f*-prefix-map flags when linking *.so
libraries, but that is not a gcc issue.  Since that seems like a viable
solution, I will continue the discussion with the Yocto people.

Thanks for your help!

[Bug debug/101473] debug_line info depends on toolchain path

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #22 from Richard Biener <rguenth at gcc dot gnu.org> ---
So fixed for GCC 12.