[Bug lto/97290] New: Segmentation fault in lto-wrapper

[Bug lto/97290] New: Segmentation fault in lto-wrapper

[markus.rothe at rite dot cc]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

            Bug ID: 97290
           Summary: Segmentation fault in lto-wrapper
           Product: gcc
           Version: 10.2.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: markus.rothe at rite dot cc
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

Hello, I get a segmentation fault when building RPMs for the AWS SDK for C++ on
Fedora 33. This version of Fedora enables LTO by default, previous versions did
not use LTO by default. So there are several layers that I had to peal apart in
order to come up with what I can show you below. I welcome any suggestions on
how to debug this problem further.

Within the cmake based build process, lto-wrapper is called and it segfaults.

To reproduce, I did the following on a Fedora 33 machine:

[ Warning, ub_EC2.cpp.o is about 700MB large. ]

$ gcc --version
gcc (GCC) 10.2.1 20200826 (Red Hat 10.2.1-3)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ mkdir -p reproduce-tmp/CMakeFiles/aws-cpp-sdk-ec2.dir/
$ cd reproduce-tmp/
$ curl -o CMakeFiles/aws-cpp-sdk-ec2.dir/
https://files.markus289.com/aifoo3/ub_EC2.cpp.o
$ curl -o -lm.res https://files.markus289.com/aifoo3/-lm.res
$ tree
.
├── -lm.res
└── CMakeFiles
    └── aws-cpp-sdk-ec2.dir
        └── ub_EC2.cpp.o

2 directories, 2 files
$ gdb
[...]
(gdb) set environment COLLECT_GCC=/usr/bin/g++
(gdb) set environment
COMPILER_PATH="/usr/libexec/gcc/x86_64-redhat-linux/10/:/usr/libexec/gcc/x86_64-redhat-linux/10/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/10/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/libexec/gcc/x86_64-redhat-linux/10/:/usr/libexec/gcc/x86_64-redhat-linux/10/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/10/:/usr/lib/gcc/x86_64-redhat-linux/"
(gdb) set environment
LIBRARY_PATH="/usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/../lib64/:/lib/../lib64/../lib64/:/usr/lib/../lib64/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/10/:/usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/10/:/usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/10/../../../:/lib/:/usr/lib/:/usr/lib/gcc/x86_64-redhat-linux/10/../../../:/lib/:/usr/lib/"
(gdb) set environment COLLECT_GCC_OPTIONS="-c -fno-openmp -fno-openacc -g -O2
-fPIC -O2 -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe
-Werror=format-security -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
-mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -shared -pthread -v -save-temps
-shared-libgcc -march=x86-64 -dumpdir ./ -dumpbase libaws-cpp-sdk-ec2.so.wpa
-fltrans-output-list=libaws-cpp-sdk-ec2.so.ltrans.out -fwpa=12
-fresolution=-lm.res -flinker-output=dyn -shared-libgcc"
(gdb) file /usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper
Reading symbols from /usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper...
Reading symbols from
/usr/lib/debug/usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper-10.2.1-3.fc33.x86_64.debug...
(gdb) set args -fresolution=-lm.res -flinker-output=dyn
CMakeFiles/aws-cpp-sdk-ec2.dir/ub_EC2.cpp.o
(gdb) run
Starting program: /usr/libexec/gcc/x86_64-redhat-linux/10/lto-wrapper
-fresolution=-lm.res -flinker-output=dyn
CMakeFiles/aws-cpp-sdk-ec2.dir/ub_EC2.cpp.o
[Detaching after vfork from child process 109]

Program received signal SIGSEGV, Segmentation fault.
0x000000000043b314 in simple_object_elf_copy_lto_debug_sections (sobj=0x52b740,
dobj=0x52b7d0, pfn=<optimized out>, err=0x7fffffffdbf8)
    at ../../libiberty/simple-object-elf.c:1515
1515                    ELF_SET_FIELD (type_functions, ei_class, Sym,
(gdb) bt
#0  0x000000000043b314 in simple_object_elf_copy_lto_debug_sections
(sobj=0x52b740, dobj=0x52b7d0, pfn=<optimized out>, err=0x7fffffffdbf8)
    at ../../libiberty/simple-object-elf.c:1515
#1  0x0000000000439865 in simple_object_copy_lto_debug_sections (sobj=0x52b740,
dest=0x52b770 "/tmp/ccYbFOK9.debug.temp.o", 
    err=0x7fffffffdbf8, rename=1) at ../../libiberty/simple-object.c:352
#2  0x00000000004562c7 in debug_objcopy (infile=<optimized out>,
rename=rename@entry=true) at ../../gcc/lto-wrapper.c:1127
#3  0x000000000045857f in run_gcc (argc=<optimized out>, argv=<optimized out>)
at ../../gcc/lto-wrapper.c:1736
#4  0x000000000043e577 in main (argc=<optimized out>, argv=<optimized out>) at
../../gcc/lto-wrapper.c:1998
(gdb)


The error happens in the following line of code in
libiberty/simple-object-elf.c:1515:

ELF_SET_FIELD (type_functions, ei_class, Sym, ent, st_shndx, Elf_Half,
sh_map[st_shndx]);

Again, in gdb:

(gdb) info locals [copied only relevant output]
type_functions = 0x4e3d60 <elf_little_64_functions>
ei_class = 2 '\002'
ent = 0x1c75c08 "E"
st_shndx = 1851862528
sh_map = 0x1be3c50

I don't know what 'Sym' and 'Elf_Half' refer to.

Could someone please continue debugging this? I would be glad to offer more
help.

If there is need to come up with the steps that produce ub_EC2.cpp.o, then I
would write something that one can follow.

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2020-10-05

--- Comment #1 from Martin Liška <marxin at gcc dot gnu.org> ---
> 
> Could someone please continue debugging this? I would be glad to offer more
> help.

Sure, I can help you with that.

> 
> If there is need to come up with the steps that produce ub_EC2.cpp.o, then I
> would write something that one can follow.

Yes, please provide steps how the .o file is produced.

[Bug lto/97290] Segmentation fault in lto-wrapper

[markus.rothe at rite dot cc]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

--- Comment #2 from Markus Rothe <markus.rothe at rite dot cc> ---
I am writing the instructions on how the .o file is produced now.

The instructions to fetch the files are wrong. This is correct:

$ mkdir -p reproduce-tmp/CMakeFiles/aws-cpp-sdk-ec2.dir/
$ cd reproduce-tmp/
$ curl -o CMakeFiles/aws-cpp-sdk-ec2.dir/ub_EC2.cpp.o
https://files.markus289.com/aifoo3/ub_EC2.cpp.o
$ curl -o lm.res https://files.markus289.com/aifoo3/lm.res
$ mv -- lm.res -lm.res
$ tree
.
├── CMakeFiles
│   └── aws-cpp-sdk-ec2.dir
│       └── ub_EC2.cpp.o
└── lm.res

2 directories, 2 files

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

--- Comment #3 from Martin Liška <marxin at gcc dot gnu.org> ---
Or you can link to a SPEC file and I can try to build it locally..

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

--- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
I was able to build the package in openSUSE with both DCMAKE_BUILD_TYPE=Release
and DCMAKE_BUILD_TYPE=Debug and debugging enabled (-g):

https://build.opensuse.org/package/show/Cloud:Tools/aws-sdk-cpp

[Bug lto/97290] Segmentation fault in lto-wrapper

[markus.rothe at rite dot cc]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

--- Comment #5 from Markus Rothe <markus.rothe at rite dot cc> ---
The spec file is available here: https://github.com/markus289/aws-sdk-cpp-rpms

But I was also able to reproduce the problem on a Fedora 32 machine without
rpm/mock/etc involved as follows:

[ Note, that this would install some files in ${HOME}/test-error/ when running
'make' as it downloads dependencies and installs them to the prefix (e.g.
aws-c-common). ]

curl -L https://github.com/aws/aws-sdk-cpp/archive/1.8.61.tar.gz|tar xz
cd aws-sdk-cpp-1.8.61 && mkdir build && cd build
CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches
-pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64
-mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection'
export CFLAGS
CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection'
export CXXFLAGS
FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches
-pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64
-mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-I/usr/lib64/gfortran/modules'
export FFLAGS
FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches
-pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64
-mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-I/usr/lib64/gfortran/modules'
export FCFLAGS
LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' 
export LDFLAGS
LT_SYS_LIBRARY_PATH=/usr/lib64:
export LT_SYS_LIBRARY_PATH
CC=gcc
export CC
CXX=g++
export CXX
/usr/bin/cmake ../ -DCMAKE_C_FLAGS_RELEASE:STRING=-DNDEBUG
-DCMAKE_CXX_FLAGS_RELEASE:STRING=-DNDEBUG
-DCMAKE_Fortran_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
-DCMAKE_INSTALL_PREFIX:PATH=${HOME}/test-error/usr
-DINCLUDE_INSTALL_DIR:PATH=${HOME}/test-error/usr/include
-DLIB_INSTALL_DIR:PATH=${HOME}/test-error/usr/lib64
-DSYSCONF_INSTALL_DIR:PATH=${HOME}/test-error/etc
-DSHARE_INSTALL_PREFIX:PATH=${HOME}/test-error/usr/share -DLIB_SUFFIX=64
-DBUILD_SHARED_LIBS:BOOL=ON -DBUILD_DEPS:BOOL=TRUE
-DAUTORUN_UNIT_TESTS:BOOL=FALSE -DCUSTOM_MEMORY_MANAGEMENT:BOOL=FALSE
-DBUILD_ONLY=ec2
make
[...]
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

Does this help? I know that the *FLAGS are quite specific to Fedora...

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org
             Status|WAITING                     |ASSIGNED

--- Comment #6 from Martin Liška <marxin at gcc dot gnu.org> ---
Good, I can reproduce that right now. I didn't use -ffat-lto-objects, that's
why I didn't see it before.

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

--- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
I've got a patch candidate for it.

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

--- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
Fixed on master with g:190c04ba36d9c6c3dce41f12012aa97c6d7f22f5

commit 190c04ba36d9c6c3dce41f12012aa97c6d7f22f5
Author: Martin Liska <mliska@suse.cz>
Date:   Mon Oct 5 18:03:08 2020 +0200

    lto: fix LTO debug sections copying.

    readelf -S prints:

    There are 81999 section headers, starting at offset 0x1f488060:

    Section Headers:
      [Nr] Name              Type            Address          Off    Size   ES
Flg Lk Inf Al
      [ 0]                   NULL            0000000000000000 000000 01404f 00 
   81998   0  0
      [ 1] .group            GROUP           0000000000000000 000040 000008 04 
   81995 105027  4
    ...
      [81995] .symtab           SYMTAB          0000000000000000 d5d9298 2db310
18     81997 105026  8
      [81996] .symtab_shndx     SYMTAB SECTION INDICES 0000000000000000 d8b45a8
079dd8 04     81995   0  4
      [81997] .strtab           STRTAB          0000000000000000 d92e380 80460c
00      0   0  1
    ...

    Looking at the documentation:
    Table 7–15 ELF sh_link and sh_info Interpretation

    sh_type - sh_link
    SHT_SYMTAB - The section header index of the associated string table.
    SHT_SYMTAB_SHNDX - The section header index of the associated symbol table.

    As seen, sh_link of a SHT_SYMTAB always points to a .strtab and readelf
    confirms that.

    So we need to use reverse mapping taken from
      [81996] .symtab_shndx     SYMTAB SECTION INDICES 0000000000000000 d8b45a8
079dd8 04     81995   0  4

    where sh_link points to 81995.

    libiberty/ChangeLog:

            PR lto/97290
            * simple-object-elf.c (simple_object_elf_copy_lto_debug_sections):
            Use sh_link of a .symtab_shndx section.

[Bug lto/97290] Segmentation fault in lto-wrapper

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97290

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
      Known to work|                            |10.2.1
             Status|ASSIGNED                    |RESOLVED

--- Comment #9 from Martin Liška <marxin at gcc dot gnu.org> ---
Fixed on GCC 10 branch with g:136256c32db63600168516e562441f73c26a187a.
Closing now.