ifunc resolving

ifunc resolving

[Fangrui Song]

I have seen ifunc relocation activities on glibc and ld recently.
https://sourceware.org/glibc/wiki/GNU_IFUNC is under-documented, some aspects
have not been well-known, and there are a lot differences across architectures
supporting ifunc, so I am sending this email hoping that these aspects can be
clarified, toolchain developers can get on the same page, and documentation can
be improved (if developers get confused at times, how could regular users
comfortably use them? :) )


1. An ifunc defined in the executable is called by a (link-time DT_NEEDED or
    runtime) shared object.

 From ld https://sourceware.org/bugzilla/show_bug.cgi?id=23169 (x86 only) this looks desired.
My understanding (comment 8) is that

(1) The main executable is relocated the last.
(2) By converting the main executable STT_GNU_IFUNC symbol to STT_FUNC, when
processing relocations in a DSO, the ifunc resolver will not be called while the
main executable is unresolved.

ifunc calls from within the executable do not incur additional costs.
ifunc calls from DSOs go through the main exe PLT and are punished.

When processing an ifunc relocation in a DSO, if the ifunc resolver is defined
in another DSO, according to comment 9 it will be errored.

The adds an executable-vs-shared difference to non-preemptible ifunc, but so be it.


The above sounds reasonable. However, the top-of-tree ld does not make -no-pie
and -pie behaviors consistent (note: ld does not support -no-pie yet).


cat > ./a.s <<eof
resolver:
   nop

.globl ifunc, _start
.type ifunc, @gnu_indirect_function
.set ifunc, resolver

_start:
   movq ifunc@GOTPCREL(%rip), %rax
   call ifunc
   # bl ifunc
eof
echo 'call ifunc' > ./b.s
as a.s -o a.o
gcc -shared -fpic b.s -o b.so

~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o -o a && readelf -W -s a | grep ifunc
~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o ./b.so -o a && readelf -W -s a | grep ifunc
~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o -o a && readelf -W --dyn-syms a | grep ifunc
~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o ./b.so -o a && readelf -W --dyn-syms a | grep ifunc

~/Dev/binutils-gdb/Debug/ld/ld-new is a top-of-tree ld.

% ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o -o a && readelf -W -s a | grep ifunc
      7: 0000000000401008     0 IFUNC   GLOBAL DEFAULT    3 ifunc
% ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o ./b.so -o a && readelf -W -s a | grep ifunc
      5: 0000000000401010     0 FUNC    GLOBAL DEFAULT    7 ifunc
      8: 0000000000401010     0 FUNC    GLOBAL DEFAULT    7 ifunc

% ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o -o a && readelf -W --dyn-syms a | grep ifunc
      5: 0000000000001020     0 IFUNC   GLOBAL DEFAULT    8 ifunc
% ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o ./b.so -o a && readelf -W --dyn-syms a | grep ifunc
      5: 0000000000001020     0 IFUNC   GLOBAL DEFAULT    8 ifunc

In the four combinations, -no-pie a.o ./b.so does the conversion.

Once a resolution is agreed, it'd be good to make aarch64/ppc/x86/etc consistent.


2. When to convert STT_GNU_IFUNC to STT_FUNC?

(This is more a ld question.)

In LLD, for a non-GOT-generating-non-PLT-generating relocation referencing a
STT_GNU_IFUNC, a canonical PLT entry is created and the symbol type is changed
to STT_FUNC. (An absolute relocation with 0 addend in a SHF_WRITE section used
to not trigger a nonical PLT entry. https://reviews.llvm.org/D65995 dropped the
case.) References from other modules will resolve to the PLT entry.

This approach has pros and cons:

* With a canonical PLT entry, the resolver of a symbol is called only once.
* If the relocation appears in a non-SHF_WRITE section, a text relocation can be avoided.
* Relocation types which are not valid dynamic relocation types are supported. GNU ld may error relocation R_X86_64_PC32 against STT_GNU_IFUNC symbol `ifunc' isn't supported
* References will bind to the canonical PLT entry. A function call needs to jump to the PLT, loads the value from the GOT, then does an indirect call.

Last time I checked, the architectures of GNU ld behaved quite differently. This
is an area that arch consistency should be improved.


3. Prefer .rela.dyn over .rela.plt for R_*_IRELATIVE?

ld powerpc produces R_*_IRELATIVE in .rela.dyn.
glibc powerpc32/powerpc64 do not process R_*_IRELATIVE if they are not in
[DT_JMPREL, DT_JMPREL+DT_PLTRELSZ).

This may be a good practice because R_*_IRELATIVE is by nature eagerly resolved.
The potentially lazy .rela.plt is not suitable.

I think at least aarch64 and x86 are still using .rela.plt.

In LLD I followed .rela.dyn and it has been working well https://reviews.llvm.org/D65651 .


4. When to define __rela_iplt_start and __rela_iplt_end?

Static pie and static no-pie relocation processing is very different in glibc.

* Static no-pie uses special code to process a magic array delimitered by __rela_iplt_start/__rela_iplt_end.
* Static pie uses self-relocation to take care of R_*_IRELATIVE. The above magic array code is executed as well. If __rela_iplt_start/__rela_iplt_end are defined, we will get 0 < __rela_iplt_start < __rela_iplt_end in csu/libc-start.c. ARCH_SETUP_IREL will crash when resolving the first relocation which has been processed.

LLD defines __rela_iplt_start/__rela_iplt_end in -pie mode (GNU ld doesn't) so
static pie elf/ldconfig segfaults.  If we take the patch "Make
_dl_relocate_static_pie return an int indicating whether it applied relocs."
from https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/maskray/lld ,
LLD linked static-pie glibc programs will work well (with another cleanup from
an unrelated thing: https://sourceware.org/pipermail/libc-alpha/2020-December/121144.html).

My idea is that defining __rela_iplt_start/__rela_iplt_end in -pie is justified.

I do see that GNU ld may not want a change (probably in a couple of years)
because it does not want to gratuitously break older glibc, but taking the patch
(probably with description rewritten) is a clarification to glibc code to me.

glibc maintainers can follow up on "[PATCH 0/3] Make glibc build with LLD"
if you accept that patch.

In a few years, when the compatibility for older glibc can be dropped.
ld can define __rela_iplt_start in -pie mode to drop the unneeded difference
in diff -u =(ld.bfd --verbose) =(ld.bfd -pie --verbose) output.

Re: ifunc resolving

[H.J. Lu]

On Mon, Jan 18, 2021 at 2:04 PM Fangrui Song <i@maskray.me> wrote:
>
>
> I have seen ifunc relocation activities on glibc and ld recently.
> https://sourceware.org/glibc/wiki/GNU_IFUNC is under-documented, some aspects
> have not been well-known, and there are a lot differences across architectures
> supporting ifunc, so I am sending this email hoping that these aspects can be
> clarified, toolchain developers can get on the same page, and documentation can
> be improved (if developers get confused at times, how could regular users
> comfortably use them? :) )
>
>
> 1. An ifunc defined in the executable is called by a (link-time DT_NEEDED or
>     runtime) shared object.
>
>  From ld https://sourceware.org/bugzilla/show_bug.cgi?id=23169 (x86 only) this looks desired.
> My understanding (comment 8) is that
>
> (1) The main executable is relocated the last.
> (2) By converting the main executable STT_GNU_IFUNC symbol to STT_FUNC, when
> processing relocations in a DSO, the ifunc resolver will not be called while the
> main executable is unresolved.
>
> ifunc calls from within the executable do not incur additional costs.
> ifunc calls from DSOs go through the main exe PLT and are punished.
>
> When processing an ifunc relocation in a DSO, if the ifunc resolver is defined
> in another DSO, according to comment 9 it will be errored.
>
> The adds an executable-vs-shared difference to non-preemptible ifunc, but so be it.
>
>
> The above sounds reasonable. However, the top-of-tree ld does not make -no-pie
> and -pie behaviors consistent (note: ld does not support -no-pie yet).
>
>
> cat > ./a.s <<eof
> resolver:
>    nop
>
> .globl ifunc, _start
> .type ifunc, @gnu_indirect_function
> .set ifunc, resolver
>
> _start:
>    movq ifunc@GOTPCREL(%rip), %rax
>    call ifunc
>    # bl ifunc
> eof
> echo 'call ifunc' > ./b.s
> as a.s -o a.o
> gcc -shared -fpic b.s -o b.so
>
> ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o -o a && readelf -W -s a | grep ifunc
> ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o ./b.so -o a && readelf -W -s a | grep ifunc
> ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o -o a && readelf -W --dyn-syms a | grep ifunc
> ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o ./b.so -o a && readelf -W --dyn-syms a | grep ifunc
>
> ~/Dev/binutils-gdb/Debug/ld/ld-new is a top-of-tree ld.
>
> % ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o -o a && readelf -W -s a | grep ifunc
>       7: 0000000000401008     0 IFUNC   GLOBAL DEFAULT    3 ifunc

.symtab is unused by ld.so.

> % ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic a.o ./b.so -o a && readelf -W -s a | grep ifunc
>       5: 0000000000401010     0 FUNC    GLOBAL DEFAULT    7 ifunc
>       8: 0000000000401010     0 FUNC    GLOBAL DEFAULT    7 ifunc

The KEY is that the address of the PLT entry in PDE is known at
link-time.  No IRELATIVE is needed.

> % ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o -o a && readelf -W --dyn-syms a | grep ifunc
>       5: 0000000000001020     0 IFUNC   GLOBAL DEFAULT    8 ifunc
> % ~/Dev/binutils-gdb/Debug/ld/ld-new --export-dynamic -pie a.o ./b.so -o a && readelf -W --dyn-syms a | grep ifunc
>       5: 0000000000001020     0 IFUNC   GLOBAL DEFAULT    8 ifunc

The address of the PLT entry in PIE is unknown at link-time.

> In the four combinations, -no-pie a.o ./b.so does the conversion.
>
> Once a resolution is agreed, it'd be good to make aarch64/ppc/x86/etc consistent.
>
>
> 2. When to convert STT_GNU_IFUNC to STT_FUNC?

Only when the address of the PLT entry in executable is known at link-time.

> (This is more a ld question.)
>
> In LLD, for a non-GOT-generating-non-PLT-generating relocation referencing a
> STT_GNU_IFUNC, a canonical PLT entry is created and the symbol type is changed
> to STT_FUNC. (An absolute relocation with 0 addend in a SHF_WRITE section used
> to not trigger a nonical PLT entry. https://reviews.llvm.org/D65995 dropped the
> case.) References from other modules will resolve to the PLT entry.
>
> This approach has pros and cons:
>
> * With a canonical PLT entry, the resolver of a symbol is called only once.
> * If the relocation appears in a non-SHF_WRITE section, a text relocation can be avoided.
> * Relocation types which are not valid dynamic relocation types are supported. GNU ld may error relocation R_X86_64_PC32 against STT_GNU_IFUNC symbol `ifunc' isn't supported
> * References will bind to the canonical PLT entry. A function call needs to jump to the PLT, loads the value from the GOT, then does an indirect call.

This allows IFUNC in PDE with external references from DSO.

> Last time I checked, the architectures of GNU ld behaved quite differently. This
> is an area that arch consistency should be improved.

Not all targets support this.

>
> 3. Prefer .rela.dyn over .rela.plt for R_*_IRELATIVE?
>
> ld powerpc produces R_*_IRELATIVE in .rela.dyn.
> glibc powerpc32/powerpc64 do not process R_*_IRELATIVE if they are not in
> [DT_JMPREL, DT_JMPREL+DT_PLTRELSZ).
>
> This may be a good practice because R_*_IRELATIVE is by nature eagerly resolved.
> The potentially lazy .rela.plt is not suitable.
>
> I think at least aarch64 and x86 are still using .rela.plt.
>
> In LLD I followed .rela.dyn and it has been working well https://reviews.llvm.org/D65651 .
>
>
> 4. When to define __rela_iplt_start and __rela_iplt_end?

I invented these.  __rela_iplt_start and __rela_iplt_end should be
defined ONLY when
there are no dynamic tags.  Since all ET_DYN files have dynamic tags,
ET_DYN files
shouldn't define them.

> Static pie and static no-pie relocation processing is very different in glibc.
>
> * Static no-pie uses special code to process a magic array delimitered by __rela_iplt_start/__rela_iplt_end.
> * Static pie uses self-relocation to take care of R_*_IRELATIVE. The above magic array code is executed as well. If __rela_iplt_start/__rela_iplt_end are defined, we will get 0 < __rela_iplt_start < __rela_iplt_end in csu/libc-start.c. ARCH_SETUP_IREL will crash when resolving the first relocation which has been processed.
>
> LLD defines __rela_iplt_start/__rela_iplt_end in -pie mode (GNU ld doesn't) so

That is wrong.  ET_DYN file shouldn't define them.

> static pie elf/ldconfig segfaults.  If we take the patch "Make
> _dl_relocate_static_pie return an int indicating whether it applied relocs."
> from https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/maskray/lld ,
> LLD linked static-pie glibc programs will work well (with another cleanup from
> an unrelated thing: https://sourceware.org/pipermail/libc-alpha/2020-December/121144.html).
>
> My idea is that defining __rela_iplt_start/__rela_iplt_end in -pie is justified.

You are wrong.

> I do see that GNU ld may not want a change (probably in a couple of years)

Never.

> because it does not want to gratuitously break older glibc, but taking the patch
> (probably with description rewritten) is a clarification to glibc code to me.

I strongly object to such bogus change.

> glibc maintainers can follow up on "[PATCH 0/3] Make glibc build with LLD"
> if you accept that patch.
>
> In a few years, when the compatibility for older glibc can be dropped.
> ld can define __rela_iplt_start in -pie mode to drop the unneeded difference
> in diff -u =(ld.bfd --verbose) =(ld.bfd -pie --verbose) output.

Not going to happen.

-- 
H.J.

Re: ifunc resolving

[Florian Weimer]

* Fangrui Song:

> I have seen ifunc relocation activities on glibc and ld recently.
> https://sourceware.org/glibc/wiki/GNU_IFUNC is under-documented, some aspects
> have not been well-known, and there are a lot differences across architectures
> supporting ifunc, so I am sending this email hoping that these aspects can be
> clarified, toolchain developers can get on the same page, and documentation can
> be improved (if developers get confused at times, how could regular users
> comfortably use them? :) )

We have two positions that still need to be reconciled:

* IFUNC resolvers must not themselves have relocation dependencies
  because they can be called at any time during relocation.  This
  restricts the functionality available to an IFUNC resolver.

* IFUNC resolvers may have relocation dependencies, but they may only be
  called after the object that contains them has been relocated.  This
  restricts how IFUNC symbols can be used (interposition is limited,
  correct dependency ordering via DT_NEEDED is required).

I do not think we have ever achieved consensus which position is the
correct one.

We have removed several IFUNC resolvers with IFUNC dependencies from
glibc itself because we could not follow the interposition/dependency
rules for the second approach.

Due to the x86-specific checks we have, those architectures currently
land in the second category.  I do not know if other architecture
maintainers agree.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: ifunc resolving

[Alan Modra]

On Tue, Jan 19, 2021 at 03:59:54PM +0100, Florian Weimer via Binutils wrote:
> We have two positions that still need to be reconciled:
> 
> * IFUNC resolvers must not themselves have relocation dependencies
>   because they can be called at any time during relocation.  This
>   restricts the functionality available to an IFUNC resolver.

On many architectures this cannot be achieved without hand-crafted
assembly.  The reason is obvious.  Ifunc resolvers return an address.
Compilers load addresses from the GOT, particularly for PIC.  GOT
entries need relocation.

> * IFUNC resolvers may have relocation dependencies, but they may only be
>   called after the object that contains them has been relocated.  This
>   restricts how IFUNC symbols can be used (interposition is limited,
>   correct dependency ordering via DT_NEEDED is required).

No interposition in practice, because the object doing the interposing
is almost always relocated later.

> I do not think we have ever achieved consensus which position is the
> correct one.

There is a third possibility.  If ld.so defers all irelative and other
relocations using ifunc symbols until all non-ifunc relocations have
been performed, globally, then ifunc resolvers would only have the
restriction that they not call other ifuncs.

That idea was floated a very long time ago.  For some reason it is
too hard or too slow to do in ld.so.

> We have removed several IFUNC resolvers with IFUNC dependencies from
> glibc itself because we could not follow the interposition/dependency
> rules for the second approach.

Right, I'm inclined to say ifunc has a fatal design flaw without
deferred relocation processing.  If it's too hard for glibc itself,
it's too hard for anyone to use anywhere.

> Due to the x86-specific checks we have, those architectures currently
> land in the second category.  I do not know if other architecture
> maintainers agree.
> 
> Thanks,
> Florian
> -- 
> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

-- 
Alan Modra
Australia Development Lab, IBM

Re: ifunc resolving

[Florian Weimer]

* Alan Modra via Libc-alpha:

> On Tue, Jan 19, 2021 at 03:59:54PM +0100, Florian Weimer via Binutils wrote:
>> We have two positions that still need to be reconciled:
>> 
>> * IFUNC resolvers must not themselves have relocation dependencies
>>   because they can be called at any time during relocation.  This
>>   restricts the functionality available to an IFUNC resolver.
>
> On many architectures this cannot be achieved without hand-crafted
> assembly.  The reason is obvious.  Ifunc resolvers return an address.
> Compilers load addresses from the GOT, particularly for PIC.  GOT
> entries need relocation.

Yes, this works best for what glibc considers PI_STATIC_AND_HIDDEN
architectures.
>
>> * IFUNC resolvers may have relocation dependencies, but they may only be
>>   called after the object that contains them has been relocated.  This
>>   restricts how IFUNC symbols can be used (interposition is limited,
>>   correct dependency ordering via DT_NEEDED is required).
>
> No interposition in practice, because the object doing the interposing
> is almost always relocated later.

Right, and that was hard for glibc.

>> I do not think we have ever achieved consensus which position is the
>> correct one.
>
> There is a third possibility.  If ld.so defers all irelative and other
> relocations using ifunc symbols until all non-ifunc relocations have
> been performed, globally, then ifunc resolvers would only have the
> restriction that they not call other ifuncs.
>
> That idea was floated a very long time ago.  For some reason it is
> too hard or too slow to do in ld.so.

It's not too hard, I wrote a patch.  I didn't mention it because it was
rejected.  It seemed about the only thing for which we had consensus. 8-/

My patch did not find an appropriate order in all cases.  I think that's
more or less unavoidable if IFUNC resolvers depend on relocations
against other IFUNC resolvers.  It would have nicely covered all
internal glibc uses at the time.

> If it's too hard for glibc itself, it's too hard for anyone to use
> anywhere.

Agreed, it's difficult to argue against that.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: ifunc resolving

[Zack Weinberg]

On Wed, Jan 20, 2021 at 4:33 AM Florian Weimer via Libc-alpha
<libc-alpha@sourceware.org> wrote:
> * Alan Modra via Libc-alpha:
> > On Tue, Jan 19, 2021 at 03:59:54PM +0100, Florian Weimer via Binutils wrote:
> > There is a third possibility.  If ld.so defers all irelative and other
> > relocations using ifunc symbols until all non-ifunc relocations have
> > been performed, globally, then ifunc resolvers would only have the
> > restriction that they not call other ifuncs.
> >
> > That idea was floated a very long time ago.  For some reason it is
> > too hard or too slow to do in ld.so.
>
> It's not too hard, I wrote a patch.  I didn't mention it because it was
> rejected.  It seemed about the only thing for which we had consensus. 8-/
>
> My patch did not find an appropriate order in all cases.  I think that's
> more or less unavoidable if IFUNC resolvers depend on relocations
> against other IFUNC resolvers.  It would have nicely covered all
> internal glibc uses at the time.

Every time this discussion comes up, I wonder how much it would help
if we completely scrapped the idea of lazy relocations.  Do what
LD_BIND_NOW=t does all the time.  Distributions are moving in this
direction already because that lets them turn on -z relro...

zw

Re: ifunc resolving

[Florian Weimer]

* Zack Weinberg:

> On Wed, Jan 20, 2021 at 4:33 AM Florian Weimer via Libc-alpha
> <libc-alpha@sourceware.org> wrote:
>> * Alan Modra via Libc-alpha:
>> > On Tue, Jan 19, 2021 at 03:59:54PM +0100, Florian Weimer via Binutils wrote:
>> > There is a third possibility.  If ld.so defers all irelative and other
>> > relocations using ifunc symbols until all non-ifunc relocations have
>> > been performed, globally, then ifunc resolvers would only have the
>> > restriction that they not call other ifuncs.
>> >
>> > That idea was floated a very long time ago.  For some reason it is
>> > too hard or too slow to do in ld.so.
>>
>> It's not too hard, I wrote a patch.  I didn't mention it because it was
>> rejected.  It seemed about the only thing for which we had consensus. 8-/
>>
>> My patch did not find an appropriate order in all cases.  I think that's
>> more or less unavoidable if IFUNC resolvers depend on relocations
>> against other IFUNC resolvers.  It would have nicely covered all
>> internal glibc uses at the time.
>
> Every time this discussion comes up, I wonder how much it would help
> if we completely scrapped the idea of lazy relocations.  Do what
> LD_BIND_NOW=t does all the time.  Distributions are moving in this
> direction already because that lets them turn on -z relro...

Lazy relocations make this much easier (if they can in fact be used).
Some tcmalloc versions called getenv from an IFUNC resolver and it
actually worked.  BIND_NOW broke it.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill