[Bug c++/101448] New: Use GCC 9.3.0 to build Ceph crimson-osd, linker failed for "relocation truncated to fit" against symbol

[Bug c++/101448] New: Use GCC 9.3.0 to build Ceph crimson-osd, linker failed for "relocation truncated to fit" against symbol

[kevin.zhao at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

            Bug ID: 101448
           Summary: Use GCC 9.3.0 to build Ceph crimson-osd, linker failed
                    for "relocation truncated to fit" against symbol
           Product: gcc
           Version: 9.3.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: kevin.zhao at linaro dot org
  Target Milestone: ---

Use GCC 9.3.0 binutils 2.34 to build Ceph, I experienced relocation truncated
to fit: R_AARCH64_CALL26
This binary after Linking usually around 2G size, I'm not sure if this is
related with the failure.
Btw, build with Clang 10 works well.

This is the regression point:
https://github.com/ceph/ceph/commit/80961c27d1adaa969e9ad3647d2fb9944b77904a#diff-195c71729a932038ce1774c2539247ffb0189fb3a7d10162403b06b26f2fdb54R264

After taking into this line, the Linking failed. The corresponding function in
seastar is:
https://github.com/scylladb/seastar/blob/master/src/core/prometheus.cc#L599

Log:
[462/462] Linking CXX executable bin/crimson-osd
FAILED: bin/crimson-osd
: && /usr/bin/ccache /usr/bin/c++  -g  -rdynamic
src/crimson/osd/CMakeFiles/crimson-osd.dir/backfill_state.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/ec_backend.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/heartbeat.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/main.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_meta.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/pg.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/pg_backend.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/pg_meta.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/replicated_backend.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/shard_services.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/object_context.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/ops_executer.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operation.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/client_request.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/client_request_common.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/compound_peering_request.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/internal_client_request.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/peering_event.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/pg_advance_map.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/replicated_request.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/background_recovery.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osd_operations/recovery_subrequest.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/pg_recovery.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/recovery_backend.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/replicated_recovery_backend.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/scheduler/scheduler.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/scheduler/mclock_scheduler.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/osdmap_gate.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/pg_map.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/pg_interval_interrupt_condition.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/objclass.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/objclass/class_api.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/ClassHandler.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/osd_op_util.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/OSDCap.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/PeeringState.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/PGPeeringEvent.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/PGStateUtils.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/MissingLoc.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/PGLog.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/recovery_types.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/__/__/osd/osd_perf_counters.cc.o
src/crimson/osd/CMakeFiles/crimson-osd.dir/watch.cc.o  -o bin/crimson-osd 
-Wl,-rpath,:::::::::::::::::::  lib/libcrimson-admin.a  lib/libcrimson-common.a
 lib/libcrimson-os.a  lib/libcrimson.a  /usr/lib/aarch64-linux-gnu/libfmt.a 
lib/libdmclock.a  lib/libcrimson-cyanstore.a  lib/libcrimson-os.a 
lib/libcrimson-cyanstore.a  lib/libcrimson-alienstore.a  lib/libkv.a 
/usr/lib/aarch64-linux-gnu/libleveldb.so  src/rocksdb/librocksdb.a 
/usr/lib/aarch64-linux-gnu/libsnappy.so  /usr/lib/aarch64-linux-gnu/liblz4.so 
/usr/lib/aarch64-linux-gnu/libz.so  lib/libheap_profiler.a 
lib/libcrimson-alien-common.a  /usr/lib/aarch64-linux-gnu/libblkid.so 
/usr/lib/aarch64-linux-gnu/libudev.so  lib/libblk.a 
/usr/lib/aarch64-linux-gnu/libaio.so  ../src/spdk/build/lib/libspdk_lvol.a 
../src/spdk/build/lib/libspdk_env_dpdk.a  -Wl,--whole-archive
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_bus_pci.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_eal.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_kvargs.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_mbuf.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_mempool.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_mempool_ring.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_pci.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_ring.a
/home/ubuntu/ceph-gcc/ceph/build/src/dpdk/lib/librte_telemetry.a
-Wl,--no-whole-archive  -Wl,-lnuma -Wl,-lpthread,-ldl  -lrt 
../src/spdk/build/lib/libspdk_sock.a  ../src/spdk/build/lib/libspdk_nvmf.a 
../src/spdk/build/lib/libspdk_bdev.a  ../src/spdk/build/lib/libspdk_nvme.a 
../src/spdk/build/lib/libspdk_conf.a  ../src/spdk/build/lib/libspdk_thread.a 
../src/spdk/build/lib/libspdk_trace.a  ../src/spdk/build/lib/libspdk_notify.a 
../src/spdk/build/lib/libspdk_accel.a 
../src/spdk/build/lib/libspdk_event_accel.a 
../src/spdk/build/lib/libspdk_blob.a  ../src/spdk/build/lib/libspdk_vmd.a 
../src/spdk/build/lib/libspdk_event_vmd.a 
../src/spdk/build/lib/libspdk_event_bdev.a 
../src/spdk/build/lib/libspdk_sock_posix.a 
../src/spdk/build/lib/libspdk_event_sock.a 
../src/spdk/build/lib/libspdk_event.a  ../src/spdk/build/lib/libspdk_rpc.a 
../src/spdk/build/lib/libspdk_jsonrpc.a  ../src/spdk/build/lib/libspdk_json.a 
../src/spdk/build/lib/libspdk_util.a  /usr/lib/aarch64-linux-gnu/libuuid.so 
../src/spdk/build/lib/libspdk_log.a  src/liburing/src/liburing.a 
lib/libcrimson-seastore.a  lib/libcrimson.a  lib/libcrimson-common.a 
lib/libcrc32.a  lib/libarch.a  boost/lib/libboost_iostreams.a 
boost/lib/libboost_regex.a  boost/lib/libboost_random.a  lib/libjson_spirit.a 
lib/libcommon_utf8.a  /usr/lib/aarch64-linux-gnu/libz.so 
/usr/lib/aarch64-linux-gnu/libcrypto.so  lib/libseastar.a 
/usr/lib/aarch64-linux-gnu/libfmt.a  boost/lib/libboost_program_options.a 
boost/lib/libboost_thread.a  boost/lib/libboost_chrono.a 
boost/lib/libboost_system.a  boost/lib/libboost_date_time.a 
boost/lib/libboost_atomic.a  /usr/lib/aarch64-linux-gnu/libcares.so 
/usr/lib/aarch64-linux-gnu/libcryptopp.so  /usr/lib/aarch64-linux-gnu/liblz4.so
 -ldl  /usr/lib/aarch64-linux-gnu/libgnutls.so  -latomic 
/usr/lib/aarch64-linux-gnu/libsctp.so 
/usr/lib/aarch64-linux-gnu/libprotobuf.so  -lrt 
/usr/lib/aarch64-linux-gnu/libyaml-cpp.so  -fsanitize=address 
-fsanitize=undefined  -fno-sanitize=vptr  /usr/lib/aarch64-linux-gnu/libnuma.so
 -lpthread && :
boost/lib/libboost_program_options.a(cmdline.o): in function
`boost::program_options::invalid_syntax::get_template[abi:cxx11](boost::program_options::invalid_syntax::kind_t)':
cmdline.cpp:(.text+0x70): relocation truncated to fit: R_AARCH64_CALL26 against
symbol `strlen' defined in .text section in
/usr/lib/gcc/aarch64-linux-gnu/9/libasan.so
cmdline.cpp:(.text+0xa4): relocation truncated to fit: R_AARCH64_CALL26 against
symbol `memcpy' defined in .text section in
/usr/lib/gcc/aarch64-linux-gnu/9/libasan.so
cmdline.cpp:(.text+0x14c): relocation truncated to fit: R_AARCH64_CALL26
against symbol `__stack_chk_fail@@GLIBC_2.17' defined in .text section in
/lib/aarch64-linux-gnu/libc.so.6
boost/lib/libboost_program_options.a(cmdline.o): in function
`boost::program_options::detail::cmdline::style(int)':
cmdline.cpp:(.text+0x1b0): relocation truncated to fit: R_AARCH64_CALL26
against symbol `strlen' defined in .text section in
/usr/lib/gcc/aarch64-linux-gnu/9/libasan.so
cmdline.cpp:(.text+0x1e4): relocation truncated to fit: R_AARCH64_CALL26
against symbol `memcpy' defined in .text section in
/usr/lib/gcc/aarch64-linux-gnu/9/libasan.so
cmdline.cpp:(.text+0x200): relocation truncated to fit: R_AARCH64_CALL26
against symbol `std::logic_error::logic_error(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)@@GLIBCXX_3.4.21' defined
in .text section in /usr/lib/gcc/aarch64-linux-gnu/9/libstdc++.so
cmdline.cpp:(.text+0x214): relocation truncated to fit: R_AARCH64_CALL26
against symbol `__cxa_allocate_exception@@CXXABI_1.3' defined in .text section
in /usr/lib/gcc/aarch64-linux-gnu/9/libstdc++.so
cmdline.cpp:(.text+0x230): relocation truncated to fit: R_AARCH64_CALL26
against symbol `std::logic_error::logic_error(std::logic_error
const&)@@GLIBCXX_3.4.21' defined in .text section in
/usr/lib/gcc/aarch64-linux-gnu/9/libstdc++.so
cmdline.cpp:(.text+0x274): relocation truncated to fit: R_AARCH64_CALL26
against symbol `__cxa_throw' defined in .text section in
/usr/lib/gcc/aarch64-linux-gnu/9/libasan.so
cmdline.cpp:(.text+0x2d4): relocation truncated to fit: R_AARCH64_CALL26
against symbol `__stack_chk_fail@@GLIBC_2.17' defined in .text section in
/lib/aarch64-linux-gnu/libc.so.6
cmdline.cpp:(.text+0x2fc): additional relocation overflows omitted from the
output
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd, linker failed for "relocation truncated to fit" against symbol

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Target|                            |aarch64
          Component|c++                         |target

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
Can you try compiling with -mcmodel=large?  OTOH 2G size should be within the
4G limit of the standard small model.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[kevin.zhao at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

--- Comment #2 from Kevin Zhao <kevin.zhao at linaro dot org> ---
Hi Richard,

Thanks for your comments.
Since we have built this with some of the .so file with -fPIC, and
-mcmodel=large is incompatible with -fPIC, so rebuild with -mcmodel is hard to
achieve.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[rearnsha at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

--- Comment #3 from Richard Earnshaw <rearnsha at gcc dot gnu.org> ---
What optimization level are you building with?  The R_AARCH64_CALL26 relocation
has a branch range of +/-2^27 bytes, or 128MBytes, so that puts a limit on the
size of the code segment of a binary.
If you're not building with optimization turned on code can bloat very quickly.
 You could try just -Og if you want minimal optimization or -Os if you really
want to reduce code size as much as possible.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[xinliang.liu at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

--- Comment #4 from Xinliang <xinliang.liu at linaro dot org> ---
Looking into the relocation code[1] of ld.
I'm very curious why ld can't handle long call here.

[1]:
```
 2976 static enum elf_aarch64_stub_type
 2977 aarch64_type_of_stub (asection *input_sec,
 2978                       const Elf_Internal_Rela *rel,
 2979                       asection *sym_sec,
 2980                       unsigned char st_type,
 2981                       bfd_vma destination)
 2982 {
 2983   bfd_vma location;
 2984   bfd_signed_vma branch_offset;
 2985   unsigned int r_type;
 2986   enum elf_aarch64_stub_type stub_type = aarch64_stub_none;
 2987
 2988   if (st_type != STT_FUNC
 2989       && (sym_sec == input_sec))
 2990     return stub_type;
 2991
 2992   /* Determine where the call point is.  */
 2993   location = (input_sec->output_offset
 2994               + input_sec->output_section->vma + rel->r_offset);
 2995
 2996   branch_offset = (bfd_signed_vma) (destination - location);
 2997
 2998   r_type = ELFNN_R_TYPE (rel->r_info);
 2999
 3000   /* We don't want to redirect any old unconditional jump in this way,
 3001      only one which is being used for a sibcall, where it is
 3002      acceptable for the IP0 and IP1 registers to be clobbered.  */
 3003   if ((r_type == AARCH64_R (CALL26) || r_type == AARCH64_R (JUMP26))
 3004       && (branch_offset > AARCH64_MAX_FWD_BRANCH_OFFSET
 3005           || branch_offset < AARCH64_MAX_BWD_BRANCH_OFFSET))
 3006     {
 3007       stub_type = aarch64_stub_long_branch;
 3008     }
 3009
 3010   return stub_type;
 3011 }
```
https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=bfd/elfnn-aarch64.c;h=097a275990f1d350be8f68943093926a5c66157a;hb=07f9ddfeba5b572451471f905473f7ddbba1d472#l2316

FYI, there is a similar bug with the same error msg:
https://sourceware.org/bugzilla/show_bug.cgi?id=18668

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[kevin.zhao at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

--- Comment #5 from Kevin Zhao <kevin.zhao at linaro dot org> ---
Hi Richard,

Thanks for your info.
Previously it is built without any -O parameters, so it should be -O0 by
default.

I've tried to use the "-Og" to link, it is quite interesting that use -Og we
can link success

Also, use -O1, -O2 and -Os, Linking works quite well.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[rearnsha at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

--- Comment #6 from Richard Earnshaw <rearnsha at gcc dot gnu.org> ---
(In reply to Kevin Zhao from comment #5)
> Hi Richard,
> 
> Thanks for your info.
> Previously it is built without any -O parameters, so it should be -O0 by
> default.
> 
> I've tried to use the "-Og" to link, it is quite interesting that use -Og we
> can link success
> 
> Also, use -O1, -O2 and -Os, Linking works quite well.

-O0 generates *very* poor (and large) code.  I wouldn't recommend using at all
except for basic syntax checking of your application.

-Og is pretty minimal optimization and should still be suitable for debugging
an application if needed.

So I think this is simply a case of you exceeded the limits of the system.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[kevin.zhao at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

--- Comment #7 from Kevin Zhao <kevin.zhao at linaro dot org> ---
Hi Richard,

Thanks for your clarification. Really appreciated!
I think that should be the root cause.

[Bug target/101448] Use GCC 9.3.0 to build Ceph crimson-osd on Arm64, linker failed for relocation truncated to fit: R_AARCH64_CALL26 against symbol

[rearnsha at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

Richard Earnshaw <rearnsha at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #8 from Richard Earnshaw <rearnsha at gcc dot gnu.org> ---
Closing as this is not a compiler bug.  If there's a problem anywhere it is
that the linker needs to implement mitigation strategies for very large code
bases, but even that is an enhancement rather than a strict bug.