[Bug go/101407] New: non-determinism in -fdump-go-spec

[Bug go/101407] New: non-determinism in -fdump-go-spec

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

            Bug ID: 101407
           Summary: non-determinism in -fdump-go-spec
           Product: gcc
           Version: 11.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: go
          Assignee: ian at airs dot com
          Reporter: toolybird at tuta dot io
                CC: cmang at google dot com
  Target Milestone: ---

I'm seeing a reproducibility issue when building gcc-go in GCC 11. Every build
run produces a different binary.

I've narrowed down the test case to essentially this:

$ gcc -fdump-go-spec=tmp-1.go -S -o sysinfo.s sysinfo.c
$ gcc -fdump-go-spec=tmp-2.go -S -o sysinfo.s sysinfo.c

$ diff -u tmp-1.go tmp-2.go
--- tmp-1.go    2021-07-11 07:26:58.512916883 +1000
+++ tmp-2.go    2021-07-11 07:27:07.976340655 +1000
@@ -8519,10 +8519,10 @@
 const _PRIxFAST8 = "x"
 const ___POSIX_FADV_DONTNEED = 4
 const _IPPROTO_MTP = 92
-type ___dirstream struct {}
-type ___va_list_tag struct {}
 type _iface struct {}
 type ___locale_data struct {}
 type __IO_marker struct {}
 type __IO_codecvt struct {}
 type __IO_wide_data struct {}
+type ___dirstream struct {}
+type ___va_list_tag struct {}

The file in question is libgo/sysinfo.c

The output seems to differ each time. Occasionally it is the same.

This didn't happen with GCC 10. I suppose I should try the trunk but haven't
done so yet.

Any thoughts? Thanks.

[Bug go/101407] non-determinism in -fdump-go-spec

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #1 from Toolybird <toolybird at tuta dot io> ---
The bug is present on trunk. Will try to bisect...

[Bug go/101407] non-determinism in -fdump-go-spec

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #2 from Toolybird <toolybird at tuta dot io> ---
> Will try to bisect

Well, that was a complete waste of time. There seems to an element of
randomness to the problem. It turns out that GCC 10 is also affected as I was
able to trigger it all the way back to

basepoints/gcc-10

I give up for the moment. Hopefully someone who cares about binary
reproducibility sees this, is able to replicate and fix.

[Bug go/101407] non-determinism in -fdump-go-spec

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
           Assignee|ian at airs dot com                |jakub at gcc dot gnu.org
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-07-12

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 51140
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=51140&action=edit
gcc12-pr101407.patch

I can reproduce it.  The problem is that hash_set for pointers by default uses
ptr_hash, which hashes the pointer values rather than strings they point to,
and that hash_set is then traversed and type lines are emitted during that
traversal.  So, with address space randomization the strings hash differently
between different runs.  This patch changes it to hash the strings instead and
that should be reproduceable.

[Bug go/101407] non-determinism in -fdump-go-spec

[toolybird at tuta dot io]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #4 from Toolybird <toolybird at tuta dot io> ---
Your patch solves the problem for me. Thank you very much!

[Bug go/101407] non-determinism in -fdump-go-spec

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #5 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:3be762c2ed79e36b9c8faaea2be04725c967a34e

commit r12-2293-g3be762c2ed79e36b9c8faaea2be04725c967a34e
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Wed Jul 14 10:22:50 2021 +0200

    godump: Fix -fdump-go-spec= reproduceability issue [PR101407]

    pot_dummy_types is a hash_set from whose traversal the code prints some
type
    lines.  hash_set normally uses default_hash_traits which for pointer types
    (the hash set hashes const char *) uses pointer_hash which hashes the
    addresses of the pointers except of the least significant 3 bits.
    With address space randomization, that results in non-determinism in the
    -fdump-go-specs= generated file, each invocation can have different order
of
    the lines emitted from pot_dummy_types traversal.

    This patch fixes it by hashing the string contents instead to make the
    hashes reproduceable.

    2021-07-14  Jakub Jelinek  <jakub@redhat.com>

            PR go/101407
            * godump.c (godump_str_hash): New type.
            (godump_container::pot_dummy_types): Use string_hash instead of
            ptr_hash in the hash_set.

[Bug go/101407] non-determinism in -fdump-go-spec

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #6 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-11 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:31b76a815fc177dd579adc03b671ba9a8846ae6c

commit r11-8771-g31b76a815fc177dd579adc03b671ba9a8846ae6c
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Wed Jul 14 10:22:50 2021 +0200

    godump: Fix -fdump-go-spec= reproduceability issue [PR101407]

    pot_dummy_types is a hash_set from whose traversal the code prints some
type
    lines.  hash_set normally uses default_hash_traits which for pointer types
    (the hash set hashes const char *) uses pointer_hash which hashes the
    addresses of the pointers except of the least significant 3 bits.
    With address space randomization, that results in non-determinism in the
    -fdump-go-specs= generated file, each invocation can have different order
of
    the lines emitted from pot_dummy_types traversal.

    This patch fixes it by hashing the string contents instead to make the
    hashes reproduceable.

    2021-07-14  Jakub Jelinek  <jakub@redhat.com>

            PR go/101407
            * godump.c (godump_str_hash): New type.
            (godump_container::pot_dummy_types): Use string_hash instead of
            ptr_hash in the hash_set.

    (cherry picked from commit 3be762c2ed79e36b9c8faaea2be04725c967a34e)

[Bug go/101407] non-determinism in -fdump-go-spec

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #7 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-10 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:2c7087f46bb8c3f698cc475ece3786582bd34da0

commit r10-10631-g2c7087f46bb8c3f698cc475ece3786582bd34da0
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Wed Jul 14 10:22:50 2021 +0200

    godump: Fix -fdump-go-spec= reproduceability issue [PR101407]

    pot_dummy_types is a hash_set from whose traversal the code prints some
type
    lines.  hash_set normally uses default_hash_traits which for pointer types
    (the hash set hashes const char *) uses pointer_hash which hashes the
    addresses of the pointers except of the least significant 3 bits.
    With address space randomization, that results in non-determinism in the
    -fdump-go-specs= generated file, each invocation can have different order
of
    the lines emitted from pot_dummy_types traversal.

    This patch fixes it by hashing the string contents instead to make the
    hashes reproduceable.

    2021-07-14  Jakub Jelinek  <jakub@redhat.com>

            PR go/101407
            * godump.c (godump_str_hash): New type.
            (godump_container::pot_dummy_types): Use string_hash instead of
            ptr_hash in the hash_set.

    (cherry picked from commit 3be762c2ed79e36b9c8faaea2be04725c967a34e)

[Bug go/101407] non-determinism in -fdump-go-spec

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

--- Comment #8 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-9 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:eb253e4148ba1a79789c623a062d43a126ec4c31

commit r9-10088-geb253e4148ba1a79789c623a062d43a126ec4c31
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Wed Jul 14 10:22:50 2021 +0200

    godump: Fix -fdump-go-spec= reproduceability issue [PR101407]

    pot_dummy_types is a hash_set from whose traversal the code prints some
type
    lines.  hash_set normally uses default_hash_traits which for pointer types
    (the hash set hashes const char *) uses pointer_hash which hashes the
    addresses of the pointers except of the least significant 3 bits.
    With address space randomization, that results in non-determinism in the
    -fdump-go-specs= generated file, each invocation can have different order
of
    the lines emitted from pot_dummy_types traversal.

    This patch fixes it by hashing the string contents instead to make the
    hashes reproduceable.

    2021-07-14  Jakub Jelinek  <jakub@redhat.com>

            PR go/101407
            * godump.c (godump_str_hash): New type.
            (godump_container::pot_dummy_types): Use string_hash instead of
            ptr_hash in the hash_set.

    (cherry picked from commit 3be762c2ed79e36b9c8faaea2be04725c967a34e)

[Bug go/101407] non-determinism in -fdump-go-spec

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Fixed.