[Bug go/104832] New: gccgo / libgo Reproducibility Problem

[Bug go/104832] New: gccgo / libgo Reproducibility Problem

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

            Bug ID: 104832
           Summary: gccgo / libgo Reproducibility Problem
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: go
          Assignee: ian at airs dot com
          Reporter: toolybird at tuta dot io
                CC: cmang at google dot com
  Target Milestone: ---

TL;DR -> Different binaries are produced depending on whether gccgo is already
installed on the host or not.

Background: I'm looking into broad issues of reproducibility in the context of
bootstrapping whole toolchains.

I first noticed this problem with gcc-11.2.0 but it's still present on GCC
trunk.

To reproduce, you will need to have the same version of gccgo installed on your
host in /usr.

For example, with current GCC trunk:

$ cat gcc/BASE-VER
$ 12.0.1

Step 1:
bootstrap GCC with `--enable-languages=c,c++,go --prefix=/usr
--libexecdir=/usr/lib' then install into a temp DESTDIR.

Step 2:
Uninstall gccgo from the host, or you can fudge it with:

$ sudo mv /usr/lib/go /usr/lib/go.XX

Step 3:
bootstrap GCC again with `--enable-languages=c,c++,go --prefix=/usr
--libexecdir=/usr/lib' then install into a 2nd temp DESTDIR.

Step 4: diff / compare the binaries in the DESTDIRs and you will see
differences.

Delving deeper and diffing the gcc-build dirs, there are a number of object
files affected. A list sorted with `ls -altr' is at the end of this report.

Here is a demonstration with "libgo/fmt.o" (although I suspect the real problem
is further up the chain):

$ cd gcc-build/x86_64-pc-linux-gnu/libgo

$ /build/gcc/src/gcc-build/./gcc/gccgo -B/build/gcc/src/gcc-build/./gcc/
-minline-all-stringops -O2 -g -I . -c -fgo-pkgpath=fmt
../../../gcc/libgo/go/fmt/doc.go ../../../gcc/libgo/go/fmt/errors.go
../../../gcc/libgo/go/fmt/format.go ../../../gcc/libgo/go/fmt/print.go
../../../gcc/libgo/go/fmt/scan.go -o fmt-test1.o

$ sudo mv /usr/lib/go /usr/lib/go.XX

$ /build/gcc/src/gcc-build/./gcc/gccgo -B/build/gcc/src/gcc-build/./gcc/
-minline-all-stringops -O2 -g -I . -c -fgo-pkgpath=fmt
../../../gcc/libgo/go/fmt/doc.go ../../../gcc/libgo/go/fmt/errors.go
../../../gcc/libgo/go/fmt/format.go ../../../gcc/libgo/go/fmt/print.go
../../../gcc/libgo/go/fmt/scan.go -o fmt-test2.o

$ diff fmt-test1.o fmt-test2.o
Binary files fmt-test1.o and fmt-test2.o differ

The differences are in the ".go_export" section of the ELF file.

Using "strings" helps to visualize the differences:

$ strings fmt-test1.o > 1.txt
$ strings fmt-test2.o > 2.txt
$ diff -u 1.txt 2.txt
--- 1.txt       2022-03-08 11:07:01.577545105 +1100
+++ 2.txt       2022-03-08 11:07:06.364263769 +1100
@@ -860,7 +860,7 @@
 import utf8 unicode/utf8 "unicode/utf8"
 init fmt fmt..import errors errors..import cpu internal_1cpu..import oserror
internal_1oserror..import poll internal_1poll..import reflectlite
internal_1reflectlite..import testlog internal_1testlog..import io io..import
fs io_1fs..import math math..import os os..import path path..import reflect
reflect..import runtime runtime..import sort sort..import strconv
strconv..import sync sync..import syscall syscall..import time time..import
unicode unicode..import abi ~internal_1abi bytealg ~internal_1bytealg fmtsort
~internal_1fmtsort goarch ~internal_1goarch goexperiment
~internal_1goexperiment goos ~internal_1goos itoa ~internal_1itoa race
~internal_1race execenv ~internal_1syscall_1execenv unix
~internal_1syscall_1unix unsafeheader ~internal_1unsafeheader bits ~math_1bits
atomic ~runtime_1internal_1atomic math ~runtime_1internal_1math sys
~runtime_1internal_1sys atomic ~sync_1atomic utf8 ~unicode_1utf8
 init_graph 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 0 14 0 15 0
16 0 17 0 18 0 19 1 2 1 5 1 13 3 1 3 2 3 5 3 13 4 1 4 2 4 3 4 5 4 7 4 13 4 16 4
17 4 18 5 2 5 13 6 2 6 13 6 16 7 1 7 2 7 5 7 13 7 16 8 1 8 2 8 3 8 5 8 7 8 11 8
13 8 14 8 16 8 17 8 18 9 2 10 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 11 10 13
10 14 10 16 10 17 10 18 11 1 11 2 11 5 11 13 12 1 12 2 12 5 12 9 12 13 12 15 12
16 12 19 13 2 14 2 14 5 14 13 15 1 15 2 15 5 15 9 15 13 16 2 16 13 17 1 17 2 17
3 17 5 17 13 17 16 18 1 18 2 18 3 18 5 18 13 18 16 18 17
-types 41 7 29 30 29 27 25 28 1067 3172 30 335 26 30 34 35 30 88 92 47 88 47 34
34 18 18 19 115 171 21 22 195 60 46 83 61 73 295 73 44 74 22
+types 41 7 29 30 29 27 25 28 1067 3172 30 335 26 30 34 35 30 88 92 47 88 47 38
34 18 18 19 115 171 21 22 195 60 46 83 61 73 295 73 44 74 22
 type 1 "Formatter" <type 31>
 type 2 "GoStringer" <type 32>
 type 3 "ScanState" <type 36>
@@ -972,7 +972,7 @@
 type 18 (? <type 29>, ? <type -21>) <type -11>
 type 19 (? <type 29>, ? <type -16>, ? <type -20>, ? <type -15>, ? <type -15>)
<type 29>
 type 20 (? <type -10>, ? <type -3>) <type -10>
-type 21 (? <type -21>) <type -15>
+type 21 (? <type -21>) (? <type -15>)
 type 22 (? <type -16>) <type -19>
 type 23 *<type 7>
 type 24 *<type 8>
@@ -1067,7 +1067,7 @@
   $ret10 = $convert(<type -15>, $false) //577
   return //577
  } //0

-checksum 22D2C8F1A933DC5F8B8E981917B8ADA04DEB2FC1
+checksum A916C151E089AA7EC237CBEBBE2B293A9595478C
 ScanState's Read should not be called. Use ReadRune
 fmt: scanning called UnreadRune with no rune available
 0123456789ABCDEFX


strace shows the host dirs being accessed:

First case:
newfstatat(AT_FDCWD, "/build/gcc/src/gcc-build/./gcc/go/12.0.1",
0x7ffc48250da0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/../lib/go/12.0.1", {st_mode=S_IFDIR|0755,
st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/lib/../lib/go/12.0.1/x86_64-pc-linux-gnu",
{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib/../lib/go/12.0.1", {st_mode=S_IFDIR|0755,
st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib/../lib/go/12.0.1/x86_64-pc-linux-gnu",
{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0

Second case:
newfstatat(AT_FDCWD, "/build/gcc/src/gcc-build/./gcc/go/12.0.1",
0x7ffee9c2a960, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/../lib/go/12.0.1", 0x7ffee9c2a960, 0) = -1 ENOENT
(No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/../lib/go/12.0.1", 0x7ffee9c2a960, 0) = -1
ENOENT (No such file or directory)

Any thoughts?
Thanks


gcc-build/x86_64-pc-linux-gnu/libgo/.libs/runtime.o
gcc-build/x86_64-pc-linux-gnu/libgo/.libs/syscall.o
gcc-build/x86_64-pc-linux-gnu/libgo/syscall.o
gcc-build/x86_64-pc-linux-gnu/libgo/index/suffixarray.o
gcc-build/x86_64-pc-linux-gnu/libgo/.libs/os.o
gcc-build/x86_64-pc-linux-gnu/libgo/os.o
gcc-build/x86_64-pc-linux-gnu/libgo/.libs/fmt.o
gcc-build/x86_64-pc-linux-gnu/libgo/fmt.o
gcc-build/x86_64-pc-linux-gnu/libgo/image/.libs/gif.o
gcc-build/x86_64-pc-linux-gnu/libgo/image/gif.o
gcc-build/x86_64-pc-linux-gnu/libgo/archive/.libs/zip.o
gcc-build/x86_64-pc-linux-gnu/libgo/archive/zip.o
gcc-build/x86_64-pc-linux-gnu/libgo/debug/.libs/macho.o
gcc-build/x86_64-pc-linux-gnu/libgo/text/.libs/template.o
gcc-build/x86_64-pc-linux-gnu/libgo/testing.o
gcc-build/x86_64-pc-linux-gnu/libgo/text/template.o
gcc-build/x86_64-pc-linux-gnu/libgo/go/ast.o
gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/mail.o
gcc-build/x86_64-pc-linux-gnu/libgo/net/mail.o
gcc-build/x86_64-pc-linux-gnu/libgo/html/.libs/template.o
gcc-build/x86_64-pc-linux-gnu/libgo/html/template.o
gcc-build/x86_64-pc-linux-gnu/libgo/crypto/.libs/x509.o
gcc-build/x86_64-pc-linux-gnu/libgo/crypto/tls.o
gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/http.o
gcc-build/x86_64-pc-linux-gnu/libgo/net/http.o
gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/rpc.o
gcc-build/x86_64-pc-linux-gnu/libgo/net/rpc.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modfetch.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/help.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/workcmd.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/work.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/clean.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/get.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/envcmd.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/bug.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modget.o
gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/test.o

[Bug go/104832] gccgo / libgo Reproducibility Problem

[schwab@linux-m68k.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #1 from Andreas Schwab <schwab@linux-m68k.org> ---
I think the basic problem is the default GOROOT of /usr (aka $prefix).  This is
related to bug 89171.

[Bug go/104832] gccgo / libgo Reproducibility Problem

[ian at airs dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #2 from Ian Lance Taylor <ian at airs dot com> ---
I just committed 2858e2afcb0a6553a222e724d8426451364ee755 which should fix the
specific problem in fmt.gox.

Let me know if you still see problems here.  Thanks.

[Bug go/104832] gccgo / libgo Reproducibility Problem

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #3 from Toolybird <toolybird at tuta dot io> ---
Thanks for the quick patch! Things are a bit different now but unfortunately
the problem persists.

It appears there is an element of randomness to this problem as the list of
affected object files has changed. Some have disappeared, some new news ones
have appeared. Here's a diff of the list between my first run and now with your
patch:

 gcc-build/x86_64-pc-linux-gnu/libgo/archive/.libs/zip.o
 gcc-build/x86_64-pc-linux-gnu/libgo/archive/zip.o
-gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/bug.o
-gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/clean.o
+gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/cache.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/envcmd.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/get.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/help.o
+gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/list.o
+gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/load.o
+gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modcmd.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modfetch.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modget.o
-gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/test.o
+gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modload.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/workcmd.o
 gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/work.o
-gcc-build/x86_64-pc-linux-gnu/libgo/crypto/.libs/x509.o
-gcc-build/x86_64-pc-linux-gnu/libgo/crypto/tls.o
-gcc-build/x86_64-pc-linux-gnu/libgo/debug/.libs/macho.o
-gcc-build/x86_64-pc-linux-gnu/libgo/fmt.o
-gcc-build/x86_64-pc-linux-gnu/libgo/go/ast.o
+gcc-build/x86_64-pc-linux-gnu/libgo/golang.org/x/mod/modfile.o
+gcc-build/x86_64-pc-linux-gnu/libgo/golang.org/x/tools/go/analysis/passes/printf.o
+gcc-build/x86_64-pc-linux-gnu/libgo/go/.libs/build.o
 gcc-build/x86_64-pc-linux-gnu/libgo/html/.libs/template.o
 gcc-build/x86_64-pc-linux-gnu/libgo/html/template.o
 gcc-build/x86_64-pc-linux-gnu/libgo/image/gif.o
 gcc-build/x86_64-pc-linux-gnu/libgo/image/.libs/gif.o
+gcc-build/x86_64-pc-linux-gnu/libgo/image/png.o
 gcc-build/x86_64-pc-linux-gnu/libgo/index/suffixarray.o
-gcc-build/x86_64-pc-linux-gnu/libgo/.libs/fmt.o
-gcc-build/x86_64-pc-linux-gnu/libgo/.libs/os.o
 gcc-build/x86_64-pc-linux-gnu/libgo/.libs/runtime.o
-gcc-build/x86_64-pc-linux-gnu/libgo/.libs/syscall.o
 gcc-build/x86_64-pc-linux-gnu/libgo/net/http.o
 gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/http.o
 gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/mail.o
 gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/rpc.o
 gcc-build/x86_64-pc-linux-gnu/libgo/net/mail.o
 gcc-build/x86_64-pc-linux-gnu/libgo/net/rpc.o
-gcc-build/x86_64-pc-linux-gnu/libgo/os.o
-gcc-build/x86_64-pc-linux-gnu/libgo/syscall.o
-gcc-build/x86_64-pc-linux-gnu/libgo/testing.o
+gcc-build/x86_64-pc-linux-gnu/libgo/runtime/.libs/pprof.o
+gcc-build/x86_64-pc-linux-gnu/libgo/runtime.o
+gcc-build/x86_64-pc-linux-gnu/libgo/runtime/pprof.o
 gcc-build/x86_64-pc-linux-gnu/libgo/text/.libs/template.o
-gcc-build/x86_64-pc-linux-gnu/libgo/text/template.o


Is it possible that ASLR is a factor here? That was indeed the case for another
bug fixed by Jakub last year [1]

Thanks again.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

[Bug go/104832] gccgo / libgo Reproducibility Problem

[ian at airs dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #4 from Ian Lance Taylor <ian at airs dot com> ---
Can you show the differences in one or two of the files, as you did before? 
Thanks.

ASLR could be a factor, but only in the sense that it would make it more likely
to reveal a bug in the code.  The code should produce identical output
regardless of whether using ASLR or not.

[Bug go/104832] gccgo / libgo Reproducibility Problem

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #5 from Toolybird <toolybird at tuta dot io> ---
Created attachment 52587
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52587&action=edit
image/gif.o differences

No worries. I'm attaching 2 diffs pertaining to "image/gif.o" and
"archive/zip.o".

Thanks

[Bug go/104832] gccgo / libgo Reproducibility Problem

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #6 from Toolybird <toolybird at tuta dot io> ---
Created attachment 52588
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52588&action=edit
archive/zip.o differences

[Bug go/104832] gccgo / libgo Reproducibility Problem

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832

--- Comment #7 from Toolybird <toolybird at tuta dot io> ---
I just stumbled across this openSUSE bug report which may or may not be
related:

https://bugzilla.opensuse.org/show_bug.cgi?id=1188621

Interesting that it mentions filesystem readdir which I hadn't considered as a
possible factor. For the record, I'm building on ext4.