Re: Document use of IFUNC support outside of libc.

["Carlos O'Donell" <carlos@redhat.com>]

On 03/04/2016 12:54 PM, Szabolcs Nagy wrote:
> On 03/03/16 21:10, Carlos O'Donell wrote:
>> I attempted to distill some of your notes here:
>> https://sourceware.org/glibc/wiki/GNU_IFUNC
>>
> 
> thanks, i was meaning to write something about it on the wiki,
> but it is a bit hard to separate the bugs from the features.

I think we should make this work sensibly for a sensible set
of use cases. In particular we are probably going to have to
explicitly what is and is not supported, and what functions
you can and can't call. I'm happy for IFUNC to exist for user
code if we impose limits like: only access local variables,
only call local functions, only use POD data types, only call
the following glibc functions, etc. etc.

> i identified some issues:
> 
> * the first point about bind now is not entirely correct,
> lazy binding does not change that much.

Clarified. I agree the ordering doesn't change, all I wanted to
do was provide some background about *why* on certain machines
this fails.

> the reloc processing order at load time is:
> 
> 1) DT_REL(A) relocs
> 2) DT_REL(A) relocs that call ifunc resolvers
> 3) DT_JMPREL relocs (may call ifunc resolvers or delay them)
> 4) DT_JMPREL relocs that call ifunc resolvers

This is the ordering per elf_dynamic_do_Rel right? Where
we force IRELATIVE to be resolved after in every given
group (but not across the groups e.g. 1) 3) 2) 4)).

> (for example 1) can be data access through GOT, 2) is ifunc
> resolved function address access through GOT, 3) is extern
> function call, 4) is ifunc resolved function call that binds
> locally e.g. static function with _IRELATIVE reloc.)
> 
> the only difference between lazy binding and bind now is at
> step 3): run time vs load time ifunc resolution.

Agreed.

> of course the ordering in 3) can break resolvers with bind
> now that work with lazy binding, but the real problem is 2):
> a resolver called there must only depend on relocs in 1).

I was thinking about this.

Would it be possible on ARM and PPC64 whose R_*_IRELATIVE
relocs are in DT_REL* to reorder the processing in the dynamic
loader? Resolve DT_JMPREL first then DT_REL*

That would give those machines feature parity with x86_64
without needing to rewrite the relocations in binutils to
handler this case?

> it is still possible to call extern functions from an ifunc
> resolver, but only if it is forced to use relocs in 1) (e.g.
> call through a volatile funcptr or -fno-plt).  i'm not sure
> if glibc wants to document this to work, because the user
> needs to know about relocations (which is compiler/linker
> internals).  the nasty part is that the compiler is free to
> add extern calls (into libc or compiler runtime) which can
> break the resolver so it cannot be written in c or c++ in
> principle :(

Correct.

On x86 with multiversioning the compiler emits multiple clones
of a function with different optimizations and selects based
on cpuid results. To get the cpuid results the ifunc resolver
emitted by the compiler calls into libgcc. As it is 
implemented this multiversioning only works on x86 because of
the relocation ordering.

> the dynamic linker could do the reloc ordering a bit better
> (so e.g. 2) happens after 3) in case of lazy binding), but
> i'm not sure how much that would help if potentially all
> functions may be ifunc resolved in a module.

Could you expand on this a bit more? What would be the problem
in having the dynamic loader do relocation processing in this
order: 1) 3) 2) 4).
 
> * an omission from that wiki page is static linking:
> ifunc resolvers run very early then (so memcpy etc work
> during libc initialization), and that breaks stack-protection
> etc instrumentation: the thread pointer is not yet set up.

I mentioned that?

"The resolver must not be compiled with -fstack-protector-all
or any similar protections e.g. asan, since they may require
early setup which has not yet completed."

I just didn't talk about static vs. dynamic, I just forbid it
in general.

> the vdso is not yet set up either and the vsyscall mechanism
> uses ifunc now, so vdso does not work with static linking at
> all (!) clock_gettime goes through a syscall (i think this is
> a bug that can result in surprising perf regression for users
> who expect speedup from static linking so i opened BZ 19767 ).

Agreed.

> i suspect there might be other limitations on resolvers
> because ptr mangling is not set up either..

Maybe.

> probably static linking can be fixed by having two sets of
> ifunc resolvers: one that only the libc uses and runs early
> and another set that runs after some c runtime init is done
> similar to the dynamic linked case.

Right.

> i actually would like to use vdso from ifunc resolvers
> to do the ifunc dispatch based on information that is only
> available in the kernel and cannot be easily communicated
> through other means (e.g. sysfs stuff).

Sure. Examples needed.
 
> * yet another issue is that the ifunc resolver type
> signature is different on different targets.

This is really lame.

> (and if the user defined resolver takes no argument, but the
> dynamic linker calls it with arguments that is not strictly
> correct in c even if it happens to work for most call abis:
> there were hardening proposals based on type signature checks
> for indirect calls which the dynamic linker would violate).

Agreed, we need to fix this.

>> That way I can point users at this.
>>
>> In gperftools tcmalloc added an IFUNC use [1] which
>> violates some of the requirements under -Wl,z,now,
>> so I have a need to document this support and discuss
>> with tcmalloc developers what we might do. Right now
>> they call way too much code for this to work.
>>
>> Cheers,
>> Carlos.
>>
>> [1] https://github.com/gperftools/gperftools/commit/6fdfc5a7f40ebcff3fdaada1a2994ff54be2f9c7
>>
> +static bool sized_delete_enabled(void) {
> +  if (tcmalloc_sized_delete_enabled != 0) {
> +    return !!tcmalloc_sized_delete_enabled();
> +  }
> 
> i think this call happens to work because the func address
> check for the weak ref forces the reloc to happen at step 1).

OK.

> +  const char *flag = TCMallocGetenvSafe("TCMALLOC_ENABLE_SIZED_DELETE");
> +  return tcmalloc::commandlineflags::StringToBool(flag, false);
> 
> i think this will crash if the address of delete is used
> (so ifunc resolver runs at step 2 while PLTGOT entries are
> uninitialized) independently of binding lazy vs now.
> with binding now it may crash without taking the address
> of delete.

Right.
 
> i'll try to update the wiki, but will wait for some
> feedbacks here for a while.

Thanks! Feel free to update the page!

Cheers,
Carlos.