Symbol resolution differs when building with LTO compared to building without

Symbol resolution differs when building with LTO compared to building without

[Alice Wang]

Hi,

I've come across some unexpected behavior and I'd appreciate your input on
the issue. I have an example project below that reproduces the issue. In
short, when building the project without LTO, a function will resolve to my
expected version, but when building with LTO, the symbol is resolved to a
different definition that I do not want.

Specifically, when building without LTO (which results in the behavior I
want), the final binary will pull in the definition of `_write` from my
custom static library that is listed first in the library list. With LTO
enabled (the bad case), the final binary will pull in the definition of
`_write` from libnosys, which comes later in the list of static libraries.

The output of my test project is pasted below. `make` builds without LTO
and dumping the resulting binary shows that `_write` is my stub
implementation. `LTO=1 make` builds with LTO enabled, and dumping the
resulting binary shows the implementation comes from libnosys.

How can I enable LTO in my build while providing custom definitions for
library calls (like _write)?

~/D/lto_test ❯❯❯ make

Final binary:
00008ae0 <_write>:
    8ae0: 4610       mov r0, r2
    8ae2: 4770       bx lr

~/D/lto_test ❯❯❯ make clean
~/D/lto_test ❯❯❯ LTO=1 make

Final binary:
00008b50 <_write>:
    8b50: 4b02       ldr r3, [pc, #8] ; (8b5c <_write+0xc>)
    8b52: 2258       movs r2, #88 ; 0x58
    8b54: 601a       str r2, [r3, #0]
    8b56: f04f 30ff mov.w r0, #4294967295 ; 0xffffffff
    8b5a: 4770       bx lr
    8b5c: 00018c84 .word 0x00018c84

------------------------------------------------

Toolchain version:
ARM 9-2020-q2-update (based on the 9 series)

Repro steps:
make
make clean
LTO=1 make

Repro project:

main.c
```
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[]) {
    printf("Hello.\n");
    return 0;
}
```

write.c
```
#include  <unistd.h>

ssize_t  _write (int      fd,
                 void    *p_buf,
                 size_t   cnt)
{
    return cnt;
}
```

Makefile
Note: You'll have to change the path to point to your local installation of
the ARM 2020 Q2 toolchain.
```
TOOLCHAIN=<YOUR_PATH_TO_TOOLCHAIN>
LTO_PLUGIN=$(TOOLCHAIN)/lib/gcc/arm-none-eabi/9.3.1/liblto_plugin.so

CC=$(TOOLCHAIN)/bin/arm-none-eabi-gcc
AR=$(TOOLCHAIN)/bin/arm-none-eabi-ar
DUMP=$(TOOLCHAIN)/bin/arm-none-eabi-objdump
NM=$(TOOLCHAIN)/bin/arm-none-eabi-gcc-nm

FLAGS=-g -Os -mthumb -mcpu=cortex-m4 -ffunction-sections -fdata-sections
--specs=nano.specs --specs=nosys.specs
ifeq ($(LTO),1)
FLAGS+=-flto -ffat-lto-objects
endif


.PHONY: dump
dump: main.elf _my_write.o
@echo "\n======================== Dump ========================"
@$(NM) -nS _my_write.o
@echo "\n==================== Final binary ===================="
@$(DUMP) -d $< | grep "<_write>:" -A 6
@echo "\n====================== _my_write.o ======================"
@$(DUMP) -d $(word 2,$^) | grep "<_write>:" -A 6
@echo "\n======================= libnosys ========================"
@$(DUMP) -d $(TOOLCHAIN)/arm-none-eabi/lib/thumb/v7e-m/nofp/libnosys.a |
grep "<_write>:" -A 6

main.elf: main.a _my_write.a
@$(CC) $(FLAGS)  -fuse-linker-plugin -fno-common -Wl,-Map=output.map
-Wl,--gc-sections -o $@ -Wl,--start-group $^ -lc -Wl,--end-group

%.a: %.o
@$(AR) --plugin $(LTO_PLUGIN) -rcs $@ $<

_my_write.o: ./write.c Makefile
@$(CC) $(FLAGS) -c $< -o $@

main.o: main.c Makefile
@$(CC) $(FLAGS) -c $< -o $@

.PHONY: clean
clean:
@rm -rf _write.a main.a main.elf *.res *.out *.o *.s *.map *.dump *.a
```

Once again, any insight would be greatly appreciated. My goal is to enable
LTO in my project while being able to provide custom definitions of
specific standard library calls.

Thanks in advance.

From,
AW

Re: Symbol resolution differs when building with LTO compared to building without

[Kewen.Lin]

on 2020/6/16 上午10:56, Alice Wang via Gcc-help wrote:
> Hi,
> 
> I've come across some unexpected behavior and I'd appreciate your input on
> the issue. I have an example project below that reproduces the issue. In
> short, when building the project without LTO, a function will resolve to my
> expected version, but when building with LTO, the symbol is resolved to a
> different definition that I do not want.
> 
> Specifically, when building without LTO (which results in the behavior I
> want), the final binary will pull in the definition of `_write` from my
> custom static library that is listed first in the library list. With LTO
> enabled (the bad case), the final binary will pull in the definition of
> `_write` from libnosys, which comes later in the list of static libraries.
> 
> The output of my test project is pasted below. `make` builds without LTO
> and dumping the resulting binary shows that `_write` is my stub
> implementation. `LTO=1 make` builds with LTO enabled, and dumping the
> resulting binary shows the implementation comes from libnosys.
> 
> How can I enable LTO in my build while providing custom definitions for
> library calls (like _write)?
> 
> ~/D/lto_test ❯❯❯ make
> 
> Final binary:
> 00008ae0 <_write>:
>     8ae0: 4610       mov r0, r2
>     8ae2: 4770       bx lr
> 
> ~/D/lto_test ❯❯❯ make clean
> ~/D/lto_test ❯❯❯ LTO=1 make
> 
> Final binary:
> 00008b50 <_write>:
>     8b50: 4b02       ldr r3, [pc, #8] ; (8b5c <_write+0xc>)
>     8b52: 2258       movs r2, #88 ; 0x58
>     8b54: 601a       str r2, [r3, #0]
>     8b56: f04f 30ff mov.w r0, #4294967295 ; 0xffffffff
>     8b5a: 4770       bx lr
>     8b5c: 00018c84 .word 0x00018c84
> 
> ------------------------------------------------
> 
> Toolchain version:
> ARM 9-2020-q2-update (based on the 9 series)
> 
> Repro steps:
> make
> make clean
> LTO=1 make
> 
> Repro project:
> 
> main.c
> ```
> #include <stdint.h>
> #include <stdlib.h>
> #include <stdio.h>
> #include <string.h>
> 
> int main(int argc, char *argv[]) {
>     printf("Hello.\n");
>     return 0;
> }
> ```
> 
> write.c
> ```
> #include  <unistd.h>
> 
> ssize_t  _write (int      fd,
>                  void    *p_buf,
>                  size_t   cnt)
> {
>     return cnt;
> }
> ```
> 
> Makefile
> Note: You'll have to change the path to point to your local installation of
> the ARM 2020 Q2 toolchain.
> ```
> TOOLCHAIN=<YOUR_PATH_TO_TOOLCHAIN>
> LTO_PLUGIN=$(TOOLCHAIN)/lib/gcc/arm-none-eabi/9.3.1/liblto_plugin.so
> 
> CC=$(TOOLCHAIN)/bin/arm-none-eabi-gcc
> AR=$(TOOLCHAIN)/bin/arm-none-eabi-ar
> DUMP=$(TOOLCHAIN)/bin/arm-none-eabi-objdump
> NM=$(TOOLCHAIN)/bin/arm-none-eabi-gcc-nm
> 
> FLAGS=-g -Os -mthumb -mcpu=cortex-m4 -ffunction-sections -fdata-sections
> --specs=nano.specs --specs=nosys.specs
> ifeq ($(LTO),1)
> FLAGS+=-flto -ffat-lto-objects
> endif
> 
> 
> .PHONY: dump
> dump: main.elf _my_write.o
> @echo "\n======================== Dump ========================"
> @$(NM) -nS _my_write.o
> @echo "\n==================== Final binary ===================="
> @$(DUMP) -d $< | grep "<_write>:" -A 6
> @echo "\n====================== _my_write.o ======================"
> @$(DUMP) -d $(word 2,$^) | grep "<_write>:" -A 6
> @echo "\n======================= libnosys ========================"
> @$(DUMP) -d $(TOOLCHAIN)/arm-none-eabi/lib/thumb/v7e-m/nofp/libnosys.a |
> grep "<_write>:" -A 6
> 
> main.elf: main.a _my_write.a
> @$(CC) $(FLAGS)  -fuse-linker-plugin -fno-common -Wl,-Map=output.map
> -Wl,--gc-sections -o $@ -Wl,--start-group $^ -lc -Wl,--end-group
> 
> %.a: %.o
> @$(AR) --plugin $(LTO_PLUGIN) -rcs $@ $<
> 
> _my_write.o: ./write.c Makefile
> @$(CC) $(FLAGS) -c $< -o $@
> 
> main.o: main.c Makefile
> @$(CC) $(FLAGS) -c $< -o $@
> 
> .PHONY: clean
> clean:
> @rm -rf _write.a main.a main.elf *.res *.out *.o *.s *.map *.dump *.a
> ```
> 
> Once again, any insight would be greatly appreciated. My goal is to enable
> LTO in my project while being able to provide custom definitions of
> specific standard library calls.
> 
> Thanks in advance.
> 
> From,
> AW
> 

Hi Alice,

This symptom made me recall one PR https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91287

Sorry that I don't have the environment for reproduction locally.

Do you mind to try each of below:
  1) Don't use linker-plugin, use -fno-use-linker-plugin explicitly.
  2) Use shared library instead of static library for _write library.
  3) -Wl,--whole-archive -l<your_static_write_library> -Wl,--no-whole-archive
  
HTH.

BR,
Kewen

Re: Symbol resolution differs when building with LTO compared to building without

[Alice Wang]

Hi Kewen,

The ticket you linked does sound similar. And thanks for your suggestions.
I tried them out and summarized the results below:

1) Don't use linker-plugin, use -fno-use-linker-plugin explicitly.

The build finishes and outputs the result I expect. My custom
implementation of _write is pulled into the final binary. But wouldn't
-fno-use-linker-plugin effectively disable LTO in the sense that my project
won't be able to take advantage of more aggressive
interprocedural optimizations?

2) Use shared library instead of static library for _write library.

I can't use a shared library in my case since I'm compiling for an embedded
target without any sort of dynamic linker.

3) -Wl,--whole-archive -l<your_static_write_library> -Wl,--no-whole-archive

I changed the build command to the following:
 @$(CC) $(FLAGS)  -fuse-linker-plugin -fno-common -Wl,-Map=output.map
-Wl,--gc-sections -o $@ -Wl,--start-group main.a -lc -Wl,--end-group
-Wl,--whole-archive _my_write.a -Wl,--no-whole-archive

Result:
*```*
> LTO=1 make
.cmake/install/GNUARM-9.2.1-Darwin/bin/../lib/gcc/arm-none-eabi/9.3.1/../../../../arm-none-eabi/bin/ld:
/Users/aw/.cmake/install/GNUARM-9.2.1-Darwin/bin/../lib/gcc/arm-none-eabi/9.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m/nofp/libc_nano.a(lib_a-writer.o):
in function `_write_r':
writer.c:(.text._write_r+0x10): undefined reference to `_write'
collect2: error: ld returned 1 exit status
make: *** [main.elf] Error 1
```
Can't seem to link when using whole archives.

If you have any more suggestions, it would be much appreciated!

From,
Aw

On Sun, Jun 28, 2020 at 4:54 AM Kewen.Lin <linkw@linux.ibm.com> wrote:

> on 2020/6/16 上午10:56, Alice Wang via Gcc-help wrote:
> > Hi,
> >
> > I've come across some unexpected behavior and I'd appreciate your input
> on
> > the issue. I have an example project below that reproduces the issue. In
> > short, when building the project without LTO, a function will resolve to
> my
> > expected version, but when building with LTO, the symbol is resolved to a
> > different definition that I do not want.
> >
> > Specifically, when building without LTO (which results in the behavior I
> > want), the final binary will pull in the definition of `_write` from my
> > custom static library that is listed first in the library list. With LTO
> > enabled (the bad case), the final binary will pull in the definition of
> > `_write` from libnosys, which comes later in the list of static
> libraries.
> >
> > The output of my test project is pasted below. `make` builds without LTO
> > and dumping the resulting binary shows that `_write` is my stub
> > implementation. `LTO=1 make` builds with LTO enabled, and dumping the
> > resulting binary shows the implementation comes from libnosys.
> >
> > How can I enable LTO in my build while providing custom definitions for
> > library calls (like _write)?
> >
> > ~/D/lto_test ❯❯❯ make
> >
> > Final binary:
> > 00008ae0 <_write>:
> >     8ae0: 4610       mov r0, r2
> >     8ae2: 4770       bx lr
> >
> > ~/D/lto_test ❯❯❯ make clean
> > ~/D/lto_test ❯❯❯ LTO=1 make
> >
> > Final binary:
> > 00008b50 <_write>:
> >     8b50: 4b02       ldr r3, [pc, #8] ; (8b5c <_write+0xc>)
> >     8b52: 2258       movs r2, #88 ; 0x58
> >     8b54: 601a       str r2, [r3, #0]
> >     8b56: f04f 30ff mov.w r0, #4294967295 ; 0xffffffff
> >     8b5a: 4770       bx lr
> >     8b5c: 00018c84 .word 0x00018c84
> >
> > ------------------------------------------------
> >
> > Toolchain version:
> > ARM 9-2020-q2-update (based on the 9 series)
> >
> > Repro steps:
> > make
> > make clean
> > LTO=1 make
> >
> > Repro project:
> >
> > main.c
> > ```
> > #include <stdint.h>
> > #include <stdlib.h>
> > #include <stdio.h>
> > #include <string.h>
> >
> > int main(int argc, char *argv[]) {
> >     printf("Hello.\n");
> >     return 0;
> > }
> > ```
> >
> > write.c
> > ```
> > #include  <unistd.h>
> >
> > ssize_t  _write (int      fd,
> >                  void    *p_buf,
> >                  size_t   cnt)
> > {
> >     return cnt;
> > }
> > ```
> >
> > Makefile
> > Note: You'll have to change the path to point to your local installation
> of
> > the ARM 2020 Q2 toolchain.
> > ```
> > TOOLCHAIN=<YOUR_PATH_TO_TOOLCHAIN>
> > LTO_PLUGIN=$(TOOLCHAIN)/lib/gcc/arm-none-eabi/9.3.1/liblto_plugin.so
> >
> > CC=$(TOOLCHAIN)/bin/arm-none-eabi-gcc
> > AR=$(TOOLCHAIN)/bin/arm-none-eabi-ar
> > DUMP=$(TOOLCHAIN)/bin/arm-none-eabi-objdump
> > NM=$(TOOLCHAIN)/bin/arm-none-eabi-gcc-nm
> >
> > FLAGS=-g -Os -mthumb -mcpu=cortex-m4 -ffunction-sections -fdata-sections
> > --specs=nano.specs --specs=nosys.specs
> > ifeq ($(LTO),1)
> > FLAGS+=-flto -ffat-lto-objects
> > endif
> >
> >
> > .PHONY: dump
> > dump: main.elf _my_write.o
> > @echo "\n======================== Dump ========================"
> > @$(NM) -nS _my_write.o
> > @echo "\n==================== Final binary ===================="
> > @$(DUMP) -d $< | grep "<_write>:" -A 6
> > @echo "\n====================== _my_write.o ======================"
> > @$(DUMP) -d $(word 2,$^) | grep "<_write>:" -A 6
> > @echo "\n======================= libnosys ========================"
> > @$(DUMP) -d $(TOOLCHAIN)/arm-none-eabi/lib/thumb/v7e-m/nofp/libnosys.a |
> > grep "<_write>:" -A 6
> >
> > main.elf: main.a _my_write.a
> > @$(CC) $(FLAGS)  -fuse-linker-plugin -fno-common -Wl,-Map=output.map
> > -Wl,--gc-sections -o $@ -Wl,--start-group $^ -lc -Wl,--end-group
> >
> > %.a: %.o
> > @$(AR) --plugin $(LTO_PLUGIN) -rcs $@ $<
> >
> > _my_write.o: ./write.c Makefile
> > @$(CC) $(FLAGS) -c $< -o $@
> >
> > main.o: main.c Makefile
> > @$(CC) $(FLAGS) -c $< -o $@
> >
> > .PHONY: clean
> > clean:
> > @rm -rf _write.a main.a main.elf *.res *.out *.o *.s *.map *.dump *.a
> > ```
> >
> > Once again, any insight would be greatly appreciated. My goal is to
> enable
> > LTO in my project while being able to provide custom definitions of
> > specific standard library calls.
> >
> > Thanks in advance.
> >
> > From,
> > AW
> >
>
> Hi Alice,
>
> This symptom made me recall one PR
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91287
>
> Sorry that I don't have the environment for reproduction locally.
>
> Do you mind to try each of below:
>   1) Don't use linker-plugin, use -fno-use-linker-plugin explicitly.
>   2) Use shared library instead of static library for _write library.
>   3) -Wl,--whole-archive -l<your_static_write_library>
> -Wl,--no-whole-archive
>
> HTH.
>
> BR,
> Kewen
>

Re: Symbol resolution differs when building with LTO compared to building without

[Kewen.Lin]

Hi Alice,

on 2020/7/10 上午4:00, Alice Wang wrote:
> Hi Kewen,
> 
> The ticket you linked does sound similar. And thanks for your suggestions. I tried them out and summarized the results below:
> 
> 1) Don't use linker-plugin, use -fno-use-linker-plugin explicitly.
> 
> The build finishes and outputs the result I expect. My custom implementation of _write is pulled into the final binary. But wouldn't -fno-use-linker-plugin effectively disable LTO in the sense that my project won't be able to take advantage of more aggressive interprocedural optimizations? 
> 

Yes, I just noticed your write.c has its own static library (archive), you would need plugin
to extract these files in archive.  Do you mind to try with option "-u _write" in the linking
command to explicitly claim symbol _write is required.  Hope it can pull your expected object
file from the archive.

> 2) Use shared library instead of static library for _write library.
> 
> I can't use a shared library in my case since I'm compiling for an embedded target without any sort of dynamic linker. 
> 

OK.

> 3) -Wl,--whole-archive -l<your_static_write_library> -Wl,--no-whole-archive
> 
> I changed the build command to the following:
>  @$(CC) $(FLAGS)  -fuse-linker-plugin -fno-common -Wl,-Map=output.map -Wl,--gc-sections -o $@ -Wl,--start-group main.a -lc -Wl,--end-group -Wl,--whole-archive _my_write.a -Wl,--no-whole-archive
> 

As you mentioned your project is for embedded env, then I guess the size is critical and
this option isn't acceptable for you since it could pull in some useless objects from
the archive.  So it looks you can just ignore it.  :)


BR,
Kewen