ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

This is a continuation of trying to build some software for the Nintendo 3DS,  
which is an ARM platform. After applying the patches in  
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100767, I no longer get an ICE,
but now I'm getting a lot of undefined references to libc symbols, like free
and snprintf and malloc. Thing is, if I build this application without LTO, it
builds fine, so there appears to be something strange going on in the LTO
process that makes it so ld can't figure out how to link libc with everything
else.

Some things I've tried and done:

- I've confirmed that the libc being linked has those symbols by checking the  
  output of nm against that libc.a.

- I spent a few hours trying to step through ld to figure out how it
  determines when a symbol is undefined, but I didn't make much headway. I did
  find a place (_bfd_generic_link_add_one_symbol) where it appears to be
  setting some data structures to undefined, but I can't figure out how it's  
  making that decision (it's some sort of table???).

- I replaced the libc.a (and honestly all of lib/ for the cross-compiler, I
  made a backup to restore it after the test) with the one from devKitARM
  (from the devKitPro group, they provide GCC toolchains for a bunch of  
  Nintendo platforms), and I was still getting undefined references, so it
  doesn't seem to be a problem of how I'm building my own newlib library (or both
  of these libraries are built wrong).

- I tried to reproduce the problem with a simpler example, but I wasn't able
  to (I probably just don't know what I'm doing).

I'm more than willing to help collect information to track down the source of
the problem, but I'm going to need some pointers in the right direction. The
application in question has a somewhat convoluted build process (what is it
with embedded development communities and super fancy Makefiles?) and it also
requires specific patches against newlib (adds a new system library), so it's a
little hard for me to have someone else try to build this and reproduce it (the
"official" 3DS GCC toolchain suffers from the LTO ICE bug, so I can't check if
their libc behaves properly, and their toolchain is even more annoying to
build).

In summary, any tips on what I should be looking for to track down the cause?
 Let me know if you need more information from me also.

Thanks,

Gabe Marcano

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

 > This is a continuation of trying to build some software for the Nintendo 3DS,
> which is an ARM platform. After applying the patches in
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100767, I no longer get an ICE,
> but now I'm getting a lot of undefined references to libc symbols, like free
> and snprintf and malloc. Thing is, if I build this application without LTO, it
> builds fine, so there appears to be something strange going on in the LTO
> process that makes it so ld can't figure out how to link libc with everything
> else.
>
> Some things I've tried and done:
>
> - I've confirmed that the libc being linked has those symbols by checking the
>   output of nm against that libc.a.
>
> - I spent a few hours trying to step through ld to figure out how it
>   determines when a symbol is undefined, but I didn't make much headway. I did
>   find a place (_bfd_generic_link_add_one_symbol) where it appears to be
>   setting some data structures to undefined, but I can't figure out how it's
>   making that decision (it's some sort of table???).
>
> - I replaced the libc.a (and honestly all of lib/ for the cross-compiler, I
>   made a backup to restore it after the test) with the one from devKitARM
>   (from the devKitPro group, they provide GCC toolchains for a bunch of
>   Nintendo platforms), and I was still getting undefined references, so it
>   doesn't seem to be a problem of how I'm building my own newlib library (or both
>   of these libraries are built wrong).
>
> - I tried to reproduce the problem with a simpler example, but I wasn't able
>   to (I probably just don't know what I'm doing).
>
> I'm more than willing to help collect information to track down the source of
> the problem, but I'm going to need some pointers in the right direction. The
> application in question has a somewhat convoluted build process (what is it
> with embedded development communities and super fancy Makefiles?) and it also
> requires specific patches against newlib (adds a new system library), so it's a
> little hard for me to have someone else try to build this and reproduce it (the
> "official" 3DS GCC toolchain suffers from the LTO ICE bug, so I can't check if
> their libc behaves properly, and their toolchain is even more annoying to
> build).
>
> In summary, any tips on what I should be looking for to track down the cause?
> Let me know if you need more information from me also.
>
> Thanks,
>
> Gabe Marcano
>

I was able to reproduce the problem with a small example:

main.c:
 #include <string.h>

 int f();

 int main() {
         return f();
 }

 void _exit(int v)
 {
         for(;;);
 }

t1.c:
 #include <stdlib.h>

 int f() { return (int)malloc(0x0);}


With commands:
arm-none-eabi-gcc -O -flto -g -c -o main.o main.c
arm-none-eabi-gcc -O0 -flto -g -c -o t1.o t1.c
arm-none-eabi-gcc -O -flto -g main.o t1.o -o main

I was able to reproduce this with my non-3ds arm cross compiler that doesn't
have the https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100767 patch, so it seems
unrelated to that issue. I had to pass -O0 to the t1.c compilation so it
wouldn't just optimize the malloc out.

Let me know if you need any more information, or if you think this is a
binutils bug and I should go and report an issue there.

Thanks,

Gabe Marcano

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

 >> This is a continuation of trying to build some software for the Nintendo 3DS,
>> which is an ARM platform. After applying the patches in
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100767, I no longer get an ICE,
>> but now I'm getting a lot of undefined references to libc symbols, like free
>> and snprintf and malloc. Thing is, if I build this application without LTO, it
>> builds fine, so there appears to be something strange going on in the LTO
>> process that makes it so ld can't figure out how to link libc with everything
>> else.
>>
>> Some things I've tried and done:
>>
>> - I've confirmed that the libc being linked has those symbols by checking the
>>   output of nm against that libc.a.
>>
>> - I spent a few hours trying to step through ld to figure out how it
>>   determines when a symbol is undefined, but I didn't make much headway. I did
>>   find a place (_bfd_generic_link_add_one_symbol) where it appears to be
>>   setting some data structures to undefined, but I can't figure out how it's
>>   making that decision (it's some sort of table???).
>>
>> - I replaced the libc.a (and honestly all of lib/ for the cross-compiler, I
>>   made a backup to restore it after the test) with the one from devKitARM
>>   (from the devKitPro group, they provide GCC toolchains for a bunch of
>>   Nintendo platforms), and I was still getting undefined references, so it
>>   doesn't seem to be a problem of how I'm building my own newlib library (or both
>>   of these libraries are built wrong).
>>
>> - I tried to reproduce the problem with a simpler example, but I wasn't able
>>   to (I probably just don't know what I'm doing).
>>
>> I'm more than willing to help collect information to track down the source of
>> the problem, but I'm going to need some pointers in the right direction. The
>> application in question has a somewhat convoluted build process (what is it
>> with embedded development communities and super fancy Makefiles?) and it also
>> requires specific patches against newlib (adds a new system library), so it's a
>> little hard for me to have someone else try to build this and reproduce it (the
>> "official" 3DS GCC toolchain suffers from the LTO ICE bug, so I can't check if
>> their libc behaves properly, and their toolchain is even more annoying to
>> build).
>>
>> In summary, any tips on what I should be looking for to track down the cause?
>> Let me know if you need more information from me also.
>>
>> Thanks,
>>
>> Gabe Marcano
>>
>
> I was able to reproduce the problem with a small example:
>
> main.c:
>   #include <string.h>
>
>   int f();
>
>   int main() {
>           return f();
>   }
>
>   void _exit(int v)
>   {
>           for(;;);
>   }
>
> t1.c:
>   #include <stdlib.h>
>
>   int f() { return (int)malloc(0x0);}
>
>
> With commands:
> arm-none-eabi-gcc -O -flto -g -c -o main.o main.c
> arm-none-eabi-gcc -O0 -flto -g -c -o t1.o t1.c
> arm-none-eabi-gcc -O -flto -g main.o t1.o -o main
>
> I was able to reproduce this with my non-3ds arm cross compiler that doesn't
> have the https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100767 patch, so it seems
> unrelated to that issue. I had to pass -O0 to the t1.c compilation so it
> wouldn't just optimize the malloc out.
>
> Let me know if you need any more information, or if you think this is a
> binutils bug and I should go and report an issue there.
>
> Thanks,
>
> Gabe Marcano

To be clear, the error I'm seeing with the example is this:

 /usr/libexec/gcc/arm-3ds-none-eabi/ld: warning: cannot find entry symbol _start; defaulting to 0000000000008018
 /usr/libexec/gcc/arm-3ds-none-eabi/ld: /tmp/ccPp7lyA.ltrans0.ltrans.o: in function `f':
 /home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
 collect2: error: ld returned 1 exit status

Without LTO, I just realized it doesn't build, but it's because I'm missing the
_sbrk newlib syscall definition, which is expected. Just adding the following
function to main.c lets the non-lto build complete:

  void *_sbrk(int incr) {
          return NULL;
  }

Gabe Marcano

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Alexander Monakov]

> To be clear, the error I'm seeing with the example is this:
> 
>  /usr/libexec/gcc/arm-3ds-none-eabi/ld: warning: cannot find entry symbol _start; defaulting to 0000000000008018
>  /usr/libexec/gcc/arm-3ds-none-eabi/ld: /tmp/ccPp7lyA.ltrans0.ltrans.o: in function `f':
>  /home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
>  collect2: error: ld returned 1 exit status

I would start by adding -Wl,-y,malloc to last gcc command line (the link step
with '-o main') to see how linker sees references to malloc. You can do that
both with and without -flto to compare successful/failing scenarios.

If you have both BFD and Gold linker, try both (-fuse-ld=gold or =bfd).

You can also pass -Wl,-v to see the linker command line and compare it with
non-LTO compilation (or even native compilation).

Does it work if you pass -fno-builtin-malloc to compilation of t.c?

It looks like without the option, GCC LTO plugin does not see malloc as an
undefined symbol in t.o, the linker does not unpack malloc from libc, and when
LTO plugin passes a new object file that references malloc, it's too late.

I believe linkers had some hacks to help this case (rescanning libc for new
references from LTO object files), but I'm not exactly sure, it's been a while
since I looked at a related area.

Alexander

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

> To be clear, the error I'm seeing with the example is this:
>
>  /usr/libexec/gcc/arm-3ds-none-eabi/ld: warning: cannot find entry symbol _start; defaulting to 0000000000008018
>  /usr/libexec/gcc/arm-3ds-none-eabi/ld: /tmp/ccPp7lyA.ltrans0.ltrans.o: in function `f':
>  /home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
>  collect2: error: ld returned 1 exit status
>
>
> I would start by adding -Wl,-y,malloc to last gcc command line (the link step
> with '-o main') to see how linker sees references to malloc. You can do that
> both with and without -flto to compare successful/failing scenarios.
>
> If you have both BFD and Gold linker, try both (-fuse-ld=gold or =bfd).
>
> You can also pass -Wl,-v to see the linker command line and compare it with
> non-LTO compilation (or even native compilation).
>
> Does it work if you pass -fno-builtin-malloc to compilation of t.c?
>
> It looks like without the option, GCC LTO plugin does not see malloc as an
> undefined symbol in t.o, the linker does not unpack malloc from libc, and when
> LTO plugin passes a new object file that references malloc, it's too late.
>
> I believe linkers had some hacks to help this case (rescanning libc for new
> references from LTO object files), but I'm not exactly sure, it's been a while
> since I looked at a related area.
>
> Alexander

So, on a native x86_64 build -Wm,-y,malloc returns:
/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/../../../../x86_64-pc-linux-gnu/bin/ld:
  /lib64/libc.so.6: definition of malloc
/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/../../../../x86_64-pc-linux-gnu/bin/ld:
  /tmp/ccJJ05Rd.ltrans0.ltrans.o: reference to malloc

With the arm cross compiler:
/usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccupVhtJ.ltrans0.ltrans.o: reference to malloc
/usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccupVhtJ.ltrans0.ltrans.o: in function `f':
/home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
collect2: error: ld returned 1 exit status

So it's just not finding the symbol?

With gold it looks like it's having problems loading the LTO plugin:
/tmp/ccUp7UC6.ltrans0.ltrans.o: reference to malloc
/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a(lib_a-malloc.o):
  plugin needed to handle lto object
/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a(lib_a-malloc.o):
  plugin needed to handle lto object
/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a(lib_a-malloc.o):
  plugin needed to handle lto object
/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a(lib_a-malloc.o):
  plugin needed to handle lto object
/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o(.text+0x11c):
  error: undefined reference to '__bss_start__'
/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o(.text+0x120):
  error: undefined reference to '__bss_end__'
t1.c:3: error: undefined reference to 'malloc'
collect2: error: ld returned 1 exit status

But I see the plugin specified in the ld.gold invocation with -Wl,-v as
-plugin /usr/libexec/gcc/arm-none-eabi/11.1.0/liblto_plugin.so

There are also some additional plugin-opts being passed on.

The gold linker works fine on the native x86_64 build.

I'm not seeing anything glaringly different between the ld invocation for the
native build and arm. I'll try to format the arm ld output:
/usr/libexec/gcc/arm-none-eabi/ld -plugin
 /usr/libexec/gcc/arm-none-eabi/11.1.0/liblto_plugin.so
 -plugin-opt=/usr/libexec/gcc/arm-none-eabi/11.1.0/lto-wrapper
 -plugin-opt=-fresolution=/tmp/ccxcs2NR.res -plugin-opt=-pass-through=-lgcc
 -plugin-opt=-pass-through=-lg -plugin-opt=-pass-through=-lc
 --sysroot=/usr/arm-none-eabi -X -o main
 /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
 -L/usr/lib/gcc/arm-none-eabi/11.1.0
 -L/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib
 -L/usr/arm-none-eabi/lib -L/usr/arm-none-eabi/usr/lib -y malloc -v main.o t1.o
 --start-group -lgcc -lg -lc --end-group
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o

And... yeah, with -fno-builtin-malloc, it appears to build!

So, it looks like a bug in the linker? Although, I'm somewhat surprised gold is
also having issues.

Gabe Marcano

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Alexander Monakov]

> With the arm cross compiler:
> /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccupVhtJ.ltrans0.ltrans.o: reference to malloc
> /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccupVhtJ.ltrans0.ltrans.o: in function `f':
> /home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
> collect2: error: ld returned 1 exit status
> 
> So it's just not finding the symbol?

It's how static linking works: the defining object is not unpacked from libc.a
unless the linker previously saw a reference to it. The linker would need to
(re)scan libc.a after being handed this ltrans unit from the LTO plugin.

Can you show output with '-Wl,-v,-t', with and without LTO?

Alexander

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

>> With the arm cross compiler:
>> /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccupVhtJ.ltrans0.ltrans.o: reference to malloc
>> /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccupVhtJ.ltrans0.ltrans.o: in function `f':
>> /home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
>> collect2: error: ld returned 1 exit status
>>
>> So it's just not finding the symbol?
>
> It's how static linking works: the defining object is not unpacked from libc.a
> unless the linker previously saw a reference to it. The linker would need to
> (re)scan libc.a after being handed this ltrans unit from the LTO plugin.
>
> Can you show output with '-Wl,-v,-t', with and without LTO?
>
> Alexander

Sure thing.

Here's the non-lto build:
 arm-none-eabi-gcc -O0 -g -Wl,-y,malloc,-v,-t   -c -o main.o main.c
 arm-none-eabi-gcc -O0 -g -Wl,-y,malloc,-v,-t   -c -o t1.o t1.c
 arm-none-eabi-gcc -O -g -Wl,-y,malloc,-v,-t  main.o t1.o   -o main
 collect2 version 11.1.0
 /usr/libexec/gcc/arm-none-eabi/ld -plugin
   /usr/libexec/gcc/arm-none-eabi/11.1.0/liblto_plugin.so
   -plugin-opt=/usr/libexec/gcc/arm-none-eabi/11.1.0/lto-wrapper
   -plugin-opt=-fresolution=/tmp/ccyt1CxF.res -plugin-opt=-pass-through=-lgcc
   -plugin-opt=-pass-through=-lg -plugin-opt=-pass-through=-lc
   --sysroot=/usr/arm-none-eabi -X -o main
   /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
   /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
   /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
   -L/usr/lib/gcc/arm-none-eabi/11.1.0
   -L/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib
   -L/usr/arm-none-eabi/lib -L/usr/arm-none-eabi/usr/lib -y malloc -v -t main.o
   t1.o --start-group -lgcc -lg -lc --end-group
   /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
   /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
 GNU ld (Gentoo 2.36.1 p3) 2.36.1
 /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
 main.o
 t1.o
 /usr/libexec/gcc/arm-none-eabi/ld: t1.o: reference to malloc
 /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
 /usr/libexec/gcc/arm-none-eabi/ld: lib_a-malloc.o (symbol from plugin): definition of malloc
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
 /tmp/ccHV3ALY.ltrans0.ltrans.o
 /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccHV3ALY.ltrans0.ltrans.o: definition of malloc
 /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a

And with LTO:
 $ CC=arm-none-eabi-gcc make
 arm-none-eabi-gcc -O0 -flto -g -Wl,-y,malloc,-v,-t   -c -o main.o main.c
 arm-none-eabi-gcc -O0 -flto -g -Wl,-y,malloc,-v,-t   -c -o t1.o t1.c
 arm-none-eabi-gcc -O -flto -g -Wl,-y,malloc,-v,-t  main.o t1.o   -o main
 collect2 version 11.1.0
 /usr/libexec/gcc/arm-none-eabi/ld -plugin
   /usr/libexec/gcc/arm-none-eabi/11.1.0/liblto_plugin.so
   -plugin-opt=/usr/libexec/gcc/arm-none-eabi/11.1.0/lto-wrapper
   -plugin-opt=-fresolution=/tmp/ccg9cQHe.res -plugin-opt=-pass-through=-lgcc
   -plugin-opt=-pass-through=-lg -plugin-opt=-pass-through=-lc
   --sysroot=/usr/arm-none-eabi -X -o main
   /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
   /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
   /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
   -L/usr/lib/gcc/arm-none-eabi/11.1.0
   -L/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib
   -L/usr/arm-none-eabi/lib -L/usr/arm-none-eabi/usr/lib -y malloc -v -t main.o
   t1.o --start-group -lgcc -lg -lc --end-group
   /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
   /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
 GNU ld (Gentoo 2.36.1 p3) 2.36.1
 /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
 main.o
 t1.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
 /tmp/cc8yOhlK.ltrans0.ltrans.o
 /usr/libexec/gcc/arm-none-eabi/ld: /tmp/cc8yOhlK.ltrans0.ltrans.o: reference to malloc
 /tmp/ccMj1Vhu.debug.temp.o
 /tmp/ccLX2EyT.debug.temp.o
 /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
 /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
 /usr/libexec/gcc/arm-none-eabi/ld: /tmp/cc8yOhlK.ltrans0.ltrans.o: in function `f':
 /home/gabriel/tmp/t1.c:3: undefined reference to `malloc'
 collect2: error: ld returned 1 exit status
 make: *** [<builtin>: main] Error 1

So, it doesn't find the symbol in libc.a? The non-lto output shows a line about
the definition of malloc right before the libc.a line it output, but that's
missing in the lto one.

Gabe Marcano

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Alexander Monakov]

On Sat, 29 May 2021, Gabriel Marcano via Gcc-help wrote:

> Here's the non-lto build:
>  arm-none-eabi-gcc -O0 -g -Wl,-y,malloc,-v,-t   -c -o main.o main.c
>  arm-none-eabi-gcc -O0 -g -Wl,-y,malloc,-v,-t   -c -o t1.o t1.c
>  arm-none-eabi-gcc -O -g -Wl,-y,malloc,-v,-t  main.o t1.o   -o main
>  collect2 version 11.1.0
>  /usr/libexec/gcc/arm-none-eabi/ld -plugin
>    /usr/libexec/gcc/arm-none-eabi/11.1.0/liblto_plugin.so
>    -plugin-opt=/usr/libexec/gcc/arm-none-eabi/11.1.0/lto-wrapper
>    -plugin-opt=-fresolution=/tmp/ccyt1CxF.res -plugin-opt=-pass-through=-lgcc
>    -plugin-opt=-pass-through=-lg -plugin-opt=-pass-through=-lc
>    --sysroot=/usr/arm-none-eabi -X -o main
>    /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
>    /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
>    /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
>    -L/usr/lib/gcc/arm-none-eabi/11.1.0
>    -L/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib
>    -L/usr/arm-none-eabi/lib -L/usr/arm-none-eabi/usr/lib -y malloc -v -t main.o
>    t1.o --start-group -lgcc -lg -lc --end-group
>    /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
>    /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
>  GNU ld (Gentoo 2.36.1 p3) 2.36.1
>  /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
>  /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
>  main.o
>  t1.o
>  /usr/libexec/gcc/arm-none-eabi/ld: t1.o: reference to malloc
>  /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
>  /usr/libexec/gcc/arm-none-eabi/ld: lib_a-malloc.o (symbol from plugin): definition of malloc

So this line ("symbol from plugin")

>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
>  /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
>  /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
>  /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
>  /tmp/ccHV3ALY.ltrans0.ltrans.o
>  /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccHV3ALY.ltrans0.ltrans.o: definition of malloc

and this line (synthesized ltrans unit) indicate that your libc.a has LTO
bytecode.


How did that happen, did you build Newlib with -flto?

Alexander

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

>> Here's the non-lto build:
>>  arm-none-eabi-gcc -O0 -g -Wl,-y,malloc,-v,-t   -c -o main.o main.c
>>  arm-none-eabi-gcc -O0 -g -Wl,-y,malloc,-v,-t   -c -o t1.o t1.c
>>  arm-none-eabi-gcc -O -g -Wl,-y,malloc,-v,-t  main.o t1.o   -o main
>>  collect2 version 11.1.0
>>  /usr/libexec/gcc/arm-none-eabi/ld -plugin
>>    /usr/libexec/gcc/arm-none-eabi/11.1.0/liblto_plugin.so
>>    -plugin-opt=/usr/libexec/gcc/arm-none-eabi/11.1.0/lto-wrapper
>>    -plugin-opt=-fresolution=/tmp/ccyt1CxF.res -plugin-opt=-pass-through=-lgcc
>>    -plugin-opt=-pass-through=-lg -plugin-opt=-pass-through=-lc
>>    --sysroot=/usr/arm-none-eabi -X -o main
>>    /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
>>    /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
>>    /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
>>    -L/usr/lib/gcc/arm-none-eabi/11.1.0
>>    -L/usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib
>>    -L/usr/arm-none-eabi/lib -L/usr/arm-none-eabi/usr/lib -y malloc -v -t main.o
>>    t1.o --start-group -lgcc -lg -lc --end-group
>>    /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
>>    /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
>>  GNU ld (Gentoo 2.36.1 p3) 2.36.1
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/crti.o
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/crtbegin.o
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/crt0.o
>>  main.o
>>  t1.o
>>  /usr/libexec/gcc/arm-none-eabi/ld: t1.o: reference to malloc
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
>>  /usr/libexec/gcc/arm-none-eabi/ld: lib_a-malloc.o (symbol from plugin): definition of malloc
>
> So this line ("symbol from plugin")
>
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/libgcc.a
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libg.a
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/../../../../arm-none-eabi/lib/libc.a
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/crtend.o
>>  /usr/lib/gcc/arm-none-eabi/11.1.0/crtn.o
>>  /tmp/ccHV3ALY.ltrans0.ltrans.o
>>  /usr/libexec/gcc/arm-none-eabi/ld: /tmp/ccHV3ALY.ltrans0.ltrans.o: definition of malloc
>
> and this line (synthesized ltrans unit) indicate that your libc.a has LTO
> bytecode.
>
>
> How did that happen, did you build Newlib with -flto?
>
>
> Alexander
>

Most likely, I'm leveraging Gentoo's crossdev functionality to build
GCC/binutils/newlib, and I do have a global -flto flag enabled on my system,
which I selectively disable for specific packages whenever I find problems. Can
newlib not be built with -flto? Does it need something like -ffat-lto-objects?
And why does it matter if it was built with lto (or is this behavior just due
to the special treatment of libc's in general when linking/building)?

Thanks,

Gabe

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Alexander Monakov]

> Most likely, I'm leveraging Gentoo's crossdev functionality to build
> GCC/binutils/newlib, and I do have a global -flto flag enabled on my system,
> which I selectively disable for specific packages whenever I find problems. Can
> newlib not be built with -flto? Does it need something like -ffat-lto-objects?
> And why does it matter if it was built with lto (or is this behavior just due
> to the special treatment of libc's in general when linking/building)?

It matters because the compiler recognizes several libc functions by name
(including 'malloc'); recognizing such so-called "builtins" is necessary
to optimize their uses. Unfortunately, builtins are different from common
functions in LTO bytecode, and sometimes behave in unexpected ways. You
can find examples searching GCC Bugzilla for stuff like "LTO built-in".

In your case the reference to built-in malloc is not visible to the linker when
it invokes the plugin while scanning input files, so it does not unpack malloc.o
from libc.a. This is a simple case of a more general problem where a compiler
creates new references to built-in functions after optimizations (e.g.
transforming 'printf("Hello\n")' to 'puts("Hello")'). Somehow it happens to work
when libc is not slim-LTO (I guess it is rescanned), but rescanning for LTO
code-generation is not implemented, as far as I know.

More importantly, building libc with LTO is sufficiently unusual, so you should
have prominently mentioned that in early emails; nobody expects that by default.
Also, checking if the minimal example works with non-LTO Newlib would be
appropriate.

LTO'd libc is considerably more tricky than usual LTO scenarios, so expect
surprises and be prepared.

Alexander

Re: ARM cross-compiler with newlib, LTO, undefined reference to libc symbols

[Gabriel Marcano]

 >> Most likely, I'm leveraging Gentoo's crossdev functionality to build
>> GCC/binutils/newlib, and I do have a global -flto flag enabled on my system,
>> which I selectively disable for specific packages whenever I find problems. Can
>> newlib not be built with -flto? Does it need something like -ffat-lto-objects?
>> And why does it matter if it was built with lto (or is this behavior just due
>> to the special treatment of libc's in general when linking/building)?
>
> It matters because the compiler recognizes several libc functions by name
> (including 'malloc'); recognizing such so-called "builtins" is necessary
> to optimize their uses. Unfortunately, builtins are different from common
> functions in LTO bytecode, and sometimes behave in unexpected ways. You
> can find examples searching GCC Bugzilla for stuff like "LTO built-in".
>  
> In your case the reference to built-in malloc is not visible to the linker when
> it invokes the plugin while scanning input files, so it does not unpack malloc.o
> from libc.a. This is a simple case of a more general problem where a compiler
> creates new references to built-in functions after optimizations (e.g.
> transforming 'printf("Hello\n")' to 'puts("Hello")'). Somehow it happens to work
> when libc is not slim-LTO (I guess it is rescanned), but rescanning for LTO
> code-generation is not implemented, as far as I know.
>  
> More importantly, building libc with LTO is sufficiently unusual, so you should
> have prominently mentioned that in early emails; nobody expects that by default.
> Also, checking if the minimal example works with non-LTO Newlib would be
> appropriate.
>  
> LTO'd libc is considerably more tricky than usual LTO scenarios, so expect
> surprises and be prepared.
>  
>  
> Alexander

Sorry, I've been building most of my regular system with LTO for a while now,
and it just had not crossed my mind as a possible source of problems for
newlib. As I don't normally work with deeper parts of GCC and binutils, I'm not
familiar with how they all interconnect (I had no idea builtins were a thing,
for example).

Out of curiosity, digging into my glibc, Gentoo's build configuration files do
make sure to strip *FLAGS of anything not whitelisted, of which -flto is not.
Oddly enough, I don't know why newlib doesn't get the same treatment, but this
is a Gentoo detail, maybe I'll raise a bug report against our scripts to prevent
others from meeting my fate unwittingly.

And yes, if I do not build newlib with -flto, I no longer get strange linking
errors, which makes sense if the issue is as you describe.

I don't know much about compiler internals, else I'd try to figure out what's
going on and to see if there's any way I can help. As I don't know enough, I'm
just going to disable LTO for newlib and move on.

Thanks for the help!

Gabe