undefined reference to `__aarch64_cas4_sync' error on arm64 native build

undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

Hi,

I have a strong power arm64 box, and the linux distro is ubuntu 22.04,
the native gcc version is:

$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

It will abort the kernel build with the complaint by 'make Image':
ld: Unexpected GOT/PLT entries detected!
ld: Unexpected run-time procedure linkages detected!
ld: ID map text too big or misaligned
ld: drivers/net/nvidia_eth.o: in function `osi_lock_irq_enabled':
osi_hal.c:(.text+0x3cc): undefined reference to `__aarch64_cas4_sync'
...

But the cross-compile with aarch64-linux-gnu-gcc on the x86 box
doesn't show the above error message.
Any comments/suggestions? Thanks very much!

Richard

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

Ah, the driver is trying to use the native gcc built atomic ops like:
__sync_val_compare_and_swap, but it seems the native aarch64 doesn't
provide these builtin atomic primitives while they are in the cross
compile toolchain.
The issue can be resolved by replacing the
**__sync_val_compare_and_swap** with **atomic_cmpxchg**.
But don't know why the native aarch64 toolchain doesn't have those
builtin atomic functions...

On Tue, Jan 2, 2024 at 11:29 AM richard clark
<richard.xnu.clark@gmail.com> wrote:
>
> Hi,
>
> I have a strong power arm64 box, and the linux distro is ubuntu 22.04,
> the native gcc version is:
>
> $ gcc --version
> gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> Copyright (C) 2021 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> It will abort the kernel build with the complaint by 'make Image':
> ld: Unexpected GOT/PLT entries detected!
> ld: Unexpected run-time procedure linkages detected!
> ld: ID map text too big or misaligned
> ld: drivers/net/nvidia_eth.o: in function `osi_lock_irq_enabled':
> osi_hal.c:(.text+0x3cc): undefined reference to `__aarch64_cas4_sync'
> ...
>
> But the cross-compile with aarch64-linux-gnu-gcc on the x86 box
> doesn't show the above error message.
> Any comments/suggestions? Thanks very much!
>
> Richard

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Mark Rutland]

On Tue, Jan 02, 2024 at 04:53:53PM +0800, richard clark wrote:
> Ah, the driver is trying to use the native gcc built atomic ops like:
> __sync_val_compare_and_swap, but it seems the native aarch64 doesn't
> provide these builtin atomic primitives while they are in the cross
> compile toolchain.
> The issue can be resolved by replacing the
> **__sync_val_compare_and_swap** with **atomic_cmpxchg**.

Yup, that's the right thing to do; drivers *shouldn't* use the builtins
directly, and *should* use the kernel's native atomic*() API.

> But don't know why the native aarch64 toolchain doesn't have those
> builtin atomic functions...

I suspect this is down to your toolchain enabling -moutline-atomics by default;
that expands the builtins into calls to out-of-line functions. I suspect your
cross-compile toolchain doesn't enable that by default.

As above, since nothing should be using the builtins, we don't implement
out-of-line versions nor do we override the option.

Mark.

> On Tue, Jan 2, 2024 at 11:29 AM richard clark
> <richard.xnu.clark@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a strong power arm64 box, and the linux distro is ubuntu 22.04,
> > the native gcc version is:
> >
> > $ gcc --version
> > gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> > Copyright (C) 2021 Free Software Foundation, Inc.
> > This is free software; see the source for copying conditions.  There is NO
> > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> >
> > It will abort the kernel build with the complaint by 'make Image':
> > ld: Unexpected GOT/PLT entries detected!
> > ld: Unexpected run-time procedure linkages detected!
> > ld: ID map text too big or misaligned
> > ld: drivers/net/nvidia_eth.o: in function `osi_lock_irq_enabled':
> > osi_hal.c:(.text+0x3cc): undefined reference to `__aarch64_cas4_sync'
> > ...
> >
> > But the cross-compile with aarch64-linux-gnu-gcc on the x86 box
> > doesn't show the above error message.
> > Any comments/suggestions? Thanks very much!
> >
> > Richard
> 

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

Hi Mark,

On Fri, Jan 5, 2024 at 2:18 AM Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Tue, Jan 02, 2024 at 04:53:53PM +0800, richard clark wrote:
> > Ah, the driver is trying to use the native gcc built atomic ops like:
> > __sync_val_compare_and_swap, but it seems the native aarch64 doesn't
> > provide these builtin atomic primitives while they are in the cross
> > compile toolchain.
> > The issue can be resolved by replacing the
> > **__sync_val_compare_and_swap** with **atomic_cmpxchg**.
>
> Yup, that's the right thing to do; drivers *shouldn't* use the builtins
> directly, and *should* use the kernel's native atomic*() API.
>
Right, that's the way I did to fix that.
>
> > But don't know why the native aarch64 toolchain doesn't have those
> > builtin atomic functions...
>
> I suspect this is down to your toolchain enabling -moutline-atomics by default;
> that expands the builtins into calls to out-of-line functions. I suspect your
> cross-compile toolchain doesn't enable that by default.
>
> As above, since nothing should be using the builtins, we don't implement
> out-of-line versions nor do we override the option.
>
AFAIK, the native build for the kernel will not link to the libc.so
but the userland application does, the builtin atomic primitives are
implemented in the glibc:
target-host $ objdump -t /lib/aarch64-linux-gnu/libc.so.6 | grep __aarch64_cas4
0000000000130950 l     F .text 0000000000000034 __aarch64_cas4_relax
0000000000130a10 l     F .text 0000000000000034 __aarch64_cas4_rel
0000000000130990 l     F .text 0000000000000034 __aarch64_cas4_acq
seems the '__sync_val_compare_and_swap' used in the application will
be renamed to _aarch64_cas4_{relax, rel, acq}. so the kernel will
complain it will
link to an 'undefined reference'. But interesting, why the
cross-compile kernel will not generate the 'undefined reference', the
cross-compile/build kernel will link to the glibc?


>
> Mark.
>
> > On Tue, Jan 2, 2024 at 11:29 AM richard clark
> > <richard.xnu.clark@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I have a strong power arm64 box, and the linux distro is ubuntu 22.04,
> > > the native gcc version is:
> > >
> > > $ gcc --version
> > > gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> > > Copyright (C) 2021 Free Software Foundation, Inc.
> > > This is free software; see the source for copying conditions.  There is NO
> > > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> > >
> > > It will abort the kernel build with the complaint by 'make Image':
> > > ld: Unexpected GOT/PLT entries detected!
> > > ld: Unexpected run-time procedure linkages detected!
> > > ld: ID map text too big or misaligned
> > > ld: drivers/net/nvidia_eth.o: in function `osi_lock_irq_enabled':
> > > osi_hal.c:(.text+0x3cc): undefined reference to `__aarch64_cas4_sync'
> > > ...
> > >
> > > But the cross-compile with aarch64-linux-gnu-gcc on the x86 box
> > > doesn't show the above error message.
> > > Any comments/suggestions? Thanks very much!
> > >
> > > Richard
> >

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Mark Rutland]

On Mon, Jan 08, 2024 at 09:28:56AM +0800, richard clark wrote:
> On Fri, Jan 5, 2024 at 2:18 AM Mark Rutland <mark.rutland@arm.com> wrote:
> > On Tue, Jan 02, 2024 at 04:53:53PM +0800, richard clark wrote:
> > > But don't know why the native aarch64 toolchain doesn't have those
> > > builtin atomic functions...
> >
> > I suspect this is down to your toolchain enabling -moutline-atomics by default;
> > that expands the builtins into calls to out-of-line functions. I suspect your
> > cross-compile toolchain doesn't enable that by default.
> >
> > As above, since nothing should be using the builtins, we don't implement
> > out-of-line versions nor do we override the option.
> >
> AFAIK, the native build for the kernel will not link to the libc.so
> but the userland application does, the builtin atomic primitives are
> implemented in the glibc:
> target-host $ objdump -t /lib/aarch64-linux-gnu/libc.so.6 | grep __aarch64_cas4
> 0000000000130950 l     F .text 0000000000000034 __aarch64_cas4_relax
> 0000000000130a10 l     F .text 0000000000000034 __aarch64_cas4_rel
> 0000000000130990 l     F .text 0000000000000034 __aarch64_cas4_acq
> seems the '__sync_val_compare_and_swap' used in the application will
> be renamed to _aarch64_cas4_{relax, rel, acq}. so the kernel will
> complain it will
> link to an 'undefined reference'. But interesting, why the
> cross-compile kernel will not generate the 'undefined reference', the
> cross-compile/build kernel will link to the glibc?

This is due to a difference in default options between the two compilers; the
kernel isn't linked against libc in either case.

Your native compiler evidently has -moutline-atomics enabled in its default
options. With that enabled, the builtin atomics generate calls to out-of-line
functions which the kernel itself does not provide, and hence those result in a
link-time error.

Your cross-compiler evidently does not have -moutline-atomics enabled in its
default options. Without that enabled, the builtin atomics generate inline
atomic instructions rather than function calls. Since these don't depend on
external functions there's no link-time error.

If you pass 'mno-outline-atomics' to your native compiler, the problem should
disappear.

Mark.

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Xi Ruoyao]

On Mon, 2024-01-08 at 10:51 +0000, Mark Rutland via Gcc-help wrote:
> > AFAIK, the native build for the kernel will not link to the libc.so
> > but the userland application does, the builtin atomic primitives are
> > implemented in the glibc:
> > target-host $ objdump -t /lib/aarch64-linux-gnu/libc.so.6 | grep __aarch64_cas4
> > 0000000000130950 l     F .text 0000000000000034 __aarch64_cas4_relax
> > 0000000000130a10 l     F .text 0000000000000034 __aarch64_cas4_rel
> > 0000000000130990 l     F .text 0000000000000034 __aarch64_cas4_acq
> > seems the '__sync_val_compare_and_swap' used in the application will
> > be renamed to _aarch64_cas4_{relax, rel, acq}. so the kernel will
> > complain it will
> > link to an 'undefined reference'. But interesting, why the
> > cross-compile kernel will not generate the 'undefined reference', the
> > cross-compile/build kernel will link to the glibc?
> 
> This is due to a difference in default options between the two compilers; the
> kernel isn't linked against libc in either case.

And even if it's not the kernel but a normal application, it still
cannot use these functions from Glibc as the objdump output contains
"l", meaning these symbols are local symbols and they cannot referred
somewhere out of the libc.so.6 itself.

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

On Mon, Jan 8, 2024 at 6:56 PM Xi Ruoyao <xry111@xry111.site> wrote:
>
> On Mon, 2024-01-08 at 10:51 +0000, Mark Rutland via Gcc-help wrote:
> > > AFAIK, the native build for the kernel will not link to the libc.so
> > > but the userland application does, the builtin atomic primitives are
> > > implemented in the glibc:
> > > target-host $ objdump -t /lib/aarch64-linux-gnu/libc.so.6 | grep __aarch64_cas4
> > > 0000000000130950 l     F .text 0000000000000034 __aarch64_cas4_relax
> > > 0000000000130a10 l     F .text 0000000000000034 __aarch64_cas4_rel
> > > 0000000000130990 l     F .text 0000000000000034 __aarch64_cas4_acq
> > > seems the '__sync_val_compare_and_swap' used in the application will
> > > be renamed to _aarch64_cas4_{relax, rel, acq}. so the kernel will
> > > complain it will
> > > link to an 'undefined reference'. But interesting, why the
> > > cross-compile kernel will not generate the 'undefined reference', the
> > > cross-compile/build kernel will link to the glibc?
> >
> > This is due to a difference in default options between the two compilers; the
> > kernel isn't linked against libc in either case.
>
> And even if it's not the kernel but a normal application, it still
> cannot use these functions from Glibc as the objdump output contains
> "l", meaning these symbols are local symbols and they cannot referred
> somewhere out of the libc.so.6 itself.
Actually you can call those builtin atomic functions in you normal
application without link time error, even execute the output binary in
the target machine in case of cross-compile, only if the linked .so is
in your target environment.
>
> --
> Xi Ruoyao <xry111@xry111.site>
> School of Aerospace Science and Technology, Xidian University

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Xi Ruoyao]

On Tue, 2024-01-09 at 10:55 +0800, richard clark wrote:
> On Mon, Jan 8, 2024 at 6:56 PM Xi Ruoyao <xry111@xry111.site> wrote:
> > 
> > On Mon, 2024-01-08 at 10:51 +0000, Mark Rutland via Gcc-help wrote:
> > > > AFAIK, the native build for the kernel will not link to the libc.so
> > > > but the userland application does, the builtin atomic primitives are
> > > > implemented in the glibc:
> > > > target-host $ objdump -t /lib/aarch64-linux-gnu/libc.so.6 | grep __aarch64_cas4
> > > > 0000000000130950 l     F .text 0000000000000034 __aarch64_cas4_relax
> > > > 0000000000130a10 l     F .text 0000000000000034 __aarch64_cas4_rel
> > > > 0000000000130990 l     F .text 0000000000000034 __aarch64_cas4_acq
> > > > seems the '__sync_val_compare_and_swap' used in the application will
> > > > be renamed to _aarch64_cas4_{relax, rel, acq}. so the kernel will
> > > > complain it will
> > > > link to an 'undefined reference'. But interesting, why the
> > > > cross-compile kernel will not generate the 'undefined reference', the
> > > > cross-compile/build kernel will link to the glibc?
> > > 
> > > This is due to a difference in default options between the two compilers; the
> > > kernel isn't linked against libc in either case.
> > 
> > And even if it's not the kernel but a normal application, it still
> > cannot use these functions from Glibc as the objdump output contains
> > "l", meaning these symbols are local symbols and they cannot referred
> > somewhere out of the libc.so.6 itself.
> Actually you can call those builtin atomic functions in you normal
> application without link time error, even execute the output binary in
> the target machine in case of cross-compile, only if the linked .so is
> in your target environment.

Because these functions are provided by libgcc.a:

xry111@defiant:~$ objdump -t /usr/lib/gcc/aarch64-unknown-linux-gnu/13.2.0/libgcc.a | grep cas
cas_1_1.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas1_relax
cas_2_1.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas2_relax
cas_4_1.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas4_relax
cas_8_1.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas8_relax
cas_16_1.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	000000000000003c .hidden __aarch64_cas16_relax
cas_1_2.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas1_acq
cas_2_2.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas2_acq
cas_4_2.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas4_acq
cas_8_2.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas8_acq
cas_16_2.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	000000000000003c .hidden __aarch64_cas16_acq
cas_1_3.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas1_rel
cas_2_3.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas2_rel
cas_4_3.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas4_rel
cas_8_3.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas8_rel
cas_16_3.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	000000000000003c .hidden __aarch64_cas16_rel
cas_1_4.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas1_acq_rel
cas_2_4.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas2_acq_rel
cas_4_4.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas4_acq_rel
cas_8_4.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000034 .hidden __aarch64_cas8_acq_rel
cas_16_4.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	000000000000003c .hidden __aarch64_cas16_acq_rel
cas_1_5.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000038 .hidden __aarch64_cas1_sync
cas_2_5.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000038 .hidden __aarch64_cas2_sync
cas_4_5.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000038 .hidden __aarch64_cas4_sync
cas_8_5.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000038 .hidden __aarch64_cas8_sync
cas_16_5.o:     file format elf64-littleaarch64
0000000000000000 g     F .text	0000000000000040 .hidden __aarch64_cas16_sync

It seems libc.so.6 just get these functions from libgcc.a (a hidden
global symbol becomes local when you link it into a shared library). 
But the Linux kernel cannot use neither libc.so nor libgcc.a.  (I know
some non-Linux kernel developers are overusing libgcc.a for kernels, but
IMO this is just wrong and Linux developers also do not do this.  If the
Linux kernel needs a symbol from libgcc the developers just provide
their own implementation.)
-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

On Tue, Jan 9, 2024 at 11:06 AM Xi Ruoyao <xry111@xry111.site> wrote:
>
> On Tue, 2024-01-09 at 10:55 +0800, richard clark wrote:
> > On Mon, Jan 8, 2024 at 6:56 PM Xi Ruoyao <xry111@xry111.site> wrote:
> > >
> > > On Mon, 2024-01-08 at 10:51 +0000, Mark Rutland via Gcc-help wrote:
> > > > > AFAIK, the native build for the kernel will not link to the libc.so
> > > > > but the userland application does, the builtin atomic primitives are
> > > > > implemented in the glibc:
> > > > > target-host $ objdump -t /lib/aarch64-linux-gnu/libc.so.6 | grep __aarch64_cas4
> > > > > 0000000000130950 l     F .text 0000000000000034 __aarch64_cas4_relax
> > > > > 0000000000130a10 l     F .text 0000000000000034 __aarch64_cas4_rel
> > > > > 0000000000130990 l     F .text 0000000000000034 __aarch64_cas4_acq
> > > > > seems the '__sync_val_compare_and_swap' used in the application will
> > > > > be renamed to _aarch64_cas4_{relax, rel, acq}. so the kernel will
> > > > > complain it will
> > > > > link to an 'undefined reference'. But interesting, why the
> > > > > cross-compile kernel will not generate the 'undefined reference', the
> > > > > cross-compile/build kernel will link to the glibc?
> > > >
> > > > This is due to a difference in default options between the two compilers; the
> > > > kernel isn't linked against libc in either case.
> > >
> > > And even if it's not the kernel but a normal application, it still
> > > cannot use these functions from Glibc as the objdump output contains
> > > "l", meaning these symbols are local symbols and they cannot referred
> > > somewhere out of the libc.so.6 itself.
> > Actually you can call those builtin atomic functions in you normal
> > application without link time error, even execute the output binary in
> > the target machine in case of cross-compile, only if the linked .so is
> > in your target environment.
>
> Because these functions are provided by libgcc.a:
>
> xry111@defiant:~$ objdump -t /usr/lib/gcc/aarch64-unknown-linux-gnu/13.2.0/libgcc.a | grep cas
> cas_1_1.o:     file format elf64-littleaarch64
[...]
> 0000000000000000 g     F .text  0000000000000038 .hidden __aarch64_cas1_sync
> cas_2_5.o:     file format elf64-littleaarch64
> 0000000000000000 g     F .text  0000000000000038 .hidden __aarch64_cas2_sync
> cas_4_5.o:     file format elf64-littleaarch64
[...]
>
> It seems libc.so.6 just get these functions from libgcc.a (a hidden
> global symbol becomes local when you link it into a shared library).

I think libgcc.a should be built with '-fvisibility=hidden' option and
hidden symbols, then as you said it will become a local sym when
linked with the libc.so.6.

> But the Linux kernel cannot use neither libc.so nor libgcc.a.  (I know
> some non-Linux kernel developers are overusing libgcc.a for kernels, but
> IMO this is just wrong and Linux developers also do not do this.  If the
> Linux kernel needs a symbol from libgcc the developers just provide
> their own implementation.)
Right, kernel should use its own implementation while not the one from
libc.so...
> --
> Xi Ruoyao <xry111@xry111.site>
> School of Aerospace Science and Technology, Xidian University

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Segher Boessenkool]

On Tue, Jan 09, 2024 at 11:05:57AM +0800, Xi Ruoyao via Gcc-help wrote:
> But the Linux kernel cannot use neither libc.so nor libgcc.a.

I have built Linux using libgcc for many years.  It is as easy as

+LIBGCC := $(shell $(CC) $(KBUILD_CFLAGS) -print-libgcc-file-name)
+libs-y += $(LIBGCC)

> (I know
> some non-Linux kernel developers are overusing libgcc.a for kernels, but
> IMO this is just wrong and Linux developers also do not do this.  If the
> Linux kernel needs a symbol from libgcc the developers just provide
> their own implementation.)

Yes, and often they have fallen behind.  When they eventually catch up
they usually just copy the GCC code anyway.

Originally the only reasonable argument for not linking against libgcc
was so kernel code would not accidentally use double-length divisions.
There are other simple ways to have all uses of __divti3 and similar
create link errors, so that is not really a good argument.

libgcc is an essential part of the compiler.  For most targets, for most
code, GCC will not generate function calls, there usually are faster (or
smaller) things it can do, but it still is necessary to have libgcc for
more uncommon things.  Using a partial copy of it, behind the times,
and maybe even incompatible, is not a great idea.


Segher

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Xi Ruoyao]

On Tue, 2024-01-09 at 01:48 -0600, Segher Boessenkool wrote:
> On Tue, Jan 09, 2024 at 11:05:57AM +0800, Xi Ruoyao via Gcc-help wrote:
> > But the Linux kernel cannot use neither libc.so nor libgcc.a.
> 
> I have built Linux using libgcc for many years.  It is as easy as
> 
> +LIBGCC := $(shell $(CC) $(KBUILD_CFLAGS) -print-libgcc-file-name)
> +libs-y += $(LIBGCC)
> 
> > (I know
> > some non-Linux kernel developers are overusing libgcc.a for kernels, but
> > IMO this is just wrong and Linux developers also do not do this.  If the
> > Linux kernel needs a symbol from libgcc the developers just provide
> > their own implementation.)
> 
> Yes, and often they have fallen behind.  When they eventually catch up
> they usually just copy the GCC code anyway.
> 
> Originally the only reasonable argument for not linking against libgcc
> was so kernel code would not accidentally use double-length divisions.
> There are other simple ways to have all uses of __divti3 and similar
> create link errors, so that is not really a good argument.
> 
> libgcc is an essential part of the compiler.  For most targets, for most
> code, GCC will not generate function calls, there usually are faster (or
> smaller) things it can do, but it still is necessary to have libgcc for
> more uncommon things.  Using a partial copy of it, behind the times,
> and maybe even incompatible, is not a great idea.

But for many targets the kernel uses the soft-float ABI while the user
space uses the hard-float ABI and the linker generally refuses to link
them altogether.  You may argue that "hey, build all multilibs" but I'd
say it's stupid and it does not solve all issues:

If GCC is configured with things like --with-arch=something, libgcc can
contain (FP/vector) instructions unsafe to use in kernel w/o special
handling.  Currently this issue is making the process very nasty to
build systemd-boot on LoongArch desktop distros where the toolchain
configured with vector extensions enabled by default.

If libgcc is so vital GCC needs to provide a way to make it work for a
set of compiler switches incompatible with any pre-built multilib.  For
example, installing the source of libgcc into /usr/lib/gcc and provide a
tool to build a libgcc.a with the specified options.

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Segher Boessenkool]

On Tue, Jan 09, 2024 at 04:09:20PM +0800, Xi Ruoyao wrote:
> If GCC is configured with things like --with-arch=something, libgcc can
> contain (FP/vector) instructions unsafe to use in kernel w/o special
> handling.

No, it can not.  Please file a bug (<https://gcc.gnu.org/bugs.html>) if
this happens to you.  Thanks!

--with-arch= only sets the *default* options, the user can override this
always (with an -march= option, say).

> If libgcc is so vital GCC needs to provide a way to make it work for a
> set of compiler switches incompatible with any pre-built multilib.  For
> example, installing the source of libgcc into /usr/lib/gcc and provide a
> tool to build a libgcc.a with the specified options.

That is how things are supposed to work, already.  Either libgcc uses
only instructions that work everywhere, or libgcc is multilibbed, things
like that.  In all normal cases this is completely transparent to the
compiler user.


Segher

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

Hi Segher,

A ported driver in linux kernel calls '__sync_val_compare_and_swap',
the cross-compiler 'aarch64-linux-gnu-gcc' doesn't complain
'__aarch64_cas1_sync' undefined reference, but the native compiler
will complain. As Mark mentioned, I double check that both cross and
native compiler should have  ''-moutline-atomics' option enabled, do
you know the reason for that?

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Segher Boessenkool]

On Wed, Jan 10, 2024 at 01:59:53PM +0800, richard clark wrote:
> A ported driver in linux kernel calls '__sync_val_compare_and_swap',

That is a builtin function.  It does not necessarily expand to an actual
function call.  aarch64 will typically expand it to inline code.

> the cross-compiler 'aarch64-linux-gnu-gcc' doesn't complain
> '__aarch64_cas1_sync' undefined reference, but the native compiler
> will complain. As Mark mentioned, I double check that both cross and
> native compiler should have  ''-moutline-atomics' option enabled, do
> you know the reason for that?

Ah, so you are requesting external functions.

I am no aarch64 expert, but apparently there were some problems in
GCC 11, maybe that is what you saw?  Or with some distros, anyway.

It also matters if you have ARMv8.1-A enabled, the LSE instructions.

Open a GCC bug report if you have more details?  Thanks,


Segher

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

On Wed, Jan 10, 2024 at 10:12 PM Segher Boessenkool
<segher@kernel.crashing.org> wrote:
>
> On Wed, Jan 10, 2024 at 01:59:53PM +0800, richard clark wrote:
> > A ported driver in linux kernel calls '__sync_val_compare_and_swap',
>
> That is a builtin function.  It does not necessarily expand to an actual
> function call.  aarch64 will typically expand it to inline code.
>
native gcc version:
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
cross-compiler gcc version:
aarch64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Interesting, the same '__sync_val_compare_and_swap' in the .c file
will be 'U __aarch64_cas4_sync' in the .o file compiled by native,
will be 't __cmpxchg_case_mb_32' in the .o file compiled by
aarch64-linux-gnu-gcc... don't know what the reason is

>
> > the cross-compiler 'aarch64-linux-gnu-gcc' doesn't complain
> > '__aarch64_cas1_sync' undefined reference, but the native compiler
> > will complain. As Mark mentioned, I double check that both cross and
> > native compiler should have  ''-moutline-atomics' option enabled, do
> > you know the reason for that?
>
> Ah, so you are requesting external functions.
>
> I am no aarch64 expert, but apparently there were some problems in
> GCC 11, maybe that is what you saw?  Or with some distros, anyway.
>
> It also matters if you have ARMv8.1-A enabled, the LSE instructions.
>
> Open a GCC bug report if you have more details?  Thanks,
>
>
> Segher

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[Mark Rutland]

On Thu, Jan 11, 2024 at 09:42:40AM +0800, richard clark wrote:
> On Wed, Jan 10, 2024 at 10:12 PM Segher Boessenkool
> <segher@kernel.crashing.org> wrote:
> >
> > On Wed, Jan 10, 2024 at 01:59:53PM +0800, richard clark wrote:
> > > A ported driver in linux kernel calls '__sync_val_compare_and_swap',
> >
> > That is a builtin function.  It does not necessarily expand to an actual
> > function call.  aarch64 will typically expand it to inline code.
> >
> native gcc version:
> gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> cross-compiler gcc version:
> aarch64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> 
> Interesting, the same '__sync_val_compare_and_swap' in the .c file
> will be 'U __aarch64_cas4_sync' in the .o file compiled by native,
> will be 't __cmpxchg_case_mb_32' in the .o file compiled by
> aarch64-linux-gnu-gcc... don't know what the reason is

The __cmpxchg_case_mb_32() function is kernel code from
arch/arm64/include/asm/cmpxchg.h, so I do not believe that's being generated by
the compiler from __sync_val_compare_and_swap().

Are you certain that's being built from the exact same C file?

Are you able to share the code in question? Where has it come from in the first
place?

Thanks,
Mark.

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

Hi Mark,

On Thu, Jan 11, 2024 at 7:26 PM Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Thu, Jan 11, 2024 at 09:42:40AM +0800, richard clark wrote:
> > On Wed, Jan 10, 2024 at 10:12 PM Segher Boessenkool
> > <segher@kernel.crashing.org> wrote:
> > >
> > > On Wed, Jan 10, 2024 at 01:59:53PM +0800, richard clark wrote:
> > > > A ported driver in linux kernel calls '__sync_val_compare_and_swap',
> > >
> > > That is a builtin function.  It does not necessarily expand to an actual
> > > function call.  aarch64 will typically expand it to inline code.
> > >
> > native gcc version:
> > gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> > cross-compiler gcc version:
> > aarch64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> >
> > Interesting, the same '__sync_val_compare_and_swap' in the .c file
> > will be 'U __aarch64_cas4_sync' in the .o file compiled by native,
> > will be 't __cmpxchg_case_mb_32' in the .o file compiled by
> > aarch64-linux-gnu-gcc... don't know what the reason is
>
> The __cmpxchg_case_mb_32() function is kernel code from
> arch/arm64/include/asm/cmpxchg.h, so I do not believe that's being generated by
> the compiler from __sync_val_compare_and_swap().
>
> Are you certain that's being built from the exact same C file?
>
> Are you able to share the code in question? Where has it come from in the first
> place?
>

Ah, double check and the misleading info about the native and cross
compiler, the native gcc(Ubuntu 11.4.0-1ubuntu1~22.04) and the
cross-compiler - aarch64-linux-gnu-gcc(Ubuntu 11.4.0-1ubuntu1~22.04)
will generate the same link error, but the cross-compiler -
aarch64-buildroot-linux-gnu-gcc.
(aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2020.08) 9.3.0)
will not generate the link error, I believe it should be as you said,
the aarch64-buildroot-linux-gnu-gcc doesn't enable '-moutline-atomics'
by default.


>
> Thanks,
> Mark.

Re: undefined reference to `__aarch64_cas4_sync' error on arm64 native build

[richard clark]

On Mon, Jan 8, 2024 at 6:51 PM Mark Rutland <mark.rutland@arm.com> wrote:

>
> This is due to a difference in default options between the two compilers; the
> kernel isn't linked against libc in either case.
>
> Your native compiler evidently has -moutline-atomics enabled in its default
> options. With that enabled, the builtin atomics generate calls to out-of-line
> functions which the kernel itself does not provide, and hence those result in a
> link-time error.
>
> Your cross-compiler evidently does not have -moutline-atomics enabled in its
> default options. Without that enabled, the builtin atomics generate inline
> atomic instructions rather than function calls. Since these don't depend on
> external functions there's no link-time error.
>
> If you pass 'mno-outline-atomics' to your native compiler, the problem should
> disappear.
>
Right, this should be the root cause, just pass 'mno-outline-atomics'
to my native compiler, the kernel link time issue doesn't show. But, I
test a userland application with '__sync_val_compare_and_swap' used,
for the 'aarch64-linux-gnu-gcc test.c', the 'nm a.out' will output:
...
0000000000000960 t __aarch64_cas4_sync
0000000000011011 B __aarch64_have_lse_atomics
...
From the above output, seems the cross compiler has
'-moutline-atomics' enabled by default, also with
'aarch64-linux-gnu-gcc -mno-outline-atomics test.c', the 'nm a.out'
doesn't show the '_aarch64_cas4_sync'. The same result for the native
build for the same test.c, this result can't explain why the
cross-compile linux kernel will not generate the link time error,...

>
> Mark.