Re: Maybe we should get rid of ifuncs

[Palmer Dabbelt <palmer@dabbelt.com>]

On Tue, 23 Apr 2024 11:39:21 PDT (-0700), enh@google.com wrote:
> On Tue, Apr 23, 2024 at 11:14 AM Zack Weinberg <zack@owlfolio.org> wrote:
>>
>> I've been thinking about the XZ exploit (two versions of the compression
>> library `liblzma` included Trojan horse code that injected a back
>> door into sshd; see https://research.swtch.com/xz-timeline) and what
>> it means for glibc, and what I've come to is we should reconsider the
>> entire idea of ifuncs.
>>
>> The SSH protocol does not use XZ compression.  liblzma.so was loaded
>> into sshd's address space because some Linux distributions patched
>> sshd to use libsystemd, and some libsystemd functions (having to do
>> with systemd's "journal" logging subsystem, IIUC) do use liblzma, but
>> by itself that wouldn't have been enough to give the exploit control,
>> because the patched sshd doesn't use any of those functions.  But
>> these same Linux distributions also compile libsshd with -z now
>> (ironically, as a hardening measure, together with -z relro) and that
>> means the resolvers for all the ifuncs in *all* the loaded shared
>> libraries will be invoked, early enough in process startup that the
>> PLT and GOT are still writable.  The XZ exploit used an ifunc resolver
>> to rewrite a whole bunch of PLT entries, intercepting both calls
>> within sshd proper, and calls from sshd to libcrypto.so
>> (i.e. OpenSSL's general-purpose cryptography library).
>>
>> Ifuncs were already a problem -- resolvers are arbitrary application
>> code that gets called from deep within the guts of the dynamic loader,
>> possibly while internal locks are held (I don't know for sure).
>> In -z now mode, they are called not just before the core C library is
>> fully initialized, but before symbol resolution is complete, meaning
>> that they can't necessarily make *any* function calls; we've had any
>> number of bug reports about this.
>
> (apparently FreeBSD uses two passes to avoid this, but bionic has the
> same issue as glibc, and iirc musl doesn't implement ifuncs.)
>
>> They seem to be less troublesome in
>> lazy binding mode, as far as I can tell, but I can still imagine them
>> causing trouble (e.g. due to recursive invocation of the lazy symbol
>> resolution machinery, or due to injecting non-async-signal-safe code
>> into a call, from a signal handler, to a function that's *supposed* to
>> be async-signal-safe).  The glibc wiki page for ifuncs
>> (https://sourceware.org/glibc/wiki/GNU_IFUNC) warns readers that ifunc
>> resolvers are subject to severe restrictions that aren't documented or
>> even agreed upon.
>>
>> As far as I know, the only legitimate (non-malicious) use case anyone
>> wants for ifuncs is to allow a library to select one of several
>> implementations of a single function, based on the characteristics of
>> the CPU -- such as how glibc itself selects the best available
>> implementation of `memcpy` for the CPU.  It seems to me that we ought
>> to be able to come up with a completely declarative mechanism for this
>> use case.  Perhaps a library could supply an array of candidate
>> implementations of a function, each paired with a bit vector that
>> declares all of the CPU capabilities that that implementation
>> requires, sorted from most to least stringent, and the dynamic loader
>> could run down the list and pick the first one that will work.  This
>> would avoid all the problems with calling application code from the
>> guts of the loader.  And, in -z relro -z now mode, it would mean that
>> no application code could run before the PLT and GOT are made
>> read-only, closing the path that the XZ trojan used to hook itself
>> into sshd.  We'd have to keep STT_GNU_IFUNC support around for at
>> least a few releases, but we could officially deprecate it and provide
>> a tunable and/or a build-time switch to disable it.
>>
>> To figure out if this is a workable idea, questions for you all:
>> (1) Are there other use cases for ifuncs that I don't know about?
>
> one thing i think is interesting (having been looking at ifuncs while
> adding riscv64 support to Android) is that afaict bionic [Android's
> libc] is basically the only current user. every library i'm aware of
> (and certainly every library that's part of the OS) does _not_ use
> ifuncs, if only because iOS/macOS has no equivalent (and Windows
> too?), and if you've got to have the manual function pointer
> manipulation implementation for them...
>
> that said, for llvm at least there's work on function multi-versioning
> where the compiler basically writes the ifunc resolver. but (a) that's
> not quite finished yet (?) and (b) i haven't seen anyone _use_ it yet
> and (c) is at least by definition of being machine-generated pretty
> regular.
>
>> (2) Are there existing ifuncs that perform CPU-capability-based
>> function selection that *could not* be replaced with an array of bit
>> vectors like what I sketched in the previous paragraph?
>
> arm32 was a horrific mess where SoCs and their kernels were pretty
> confused about what they did/didn't have. Android's libc ifunc
> resolvers basically had to end up with "look, just write down in this
> file what you want us to use, and we'll use those" so the _device_
> owners could override the mess the SoC vendors had made.
>
> arm64 was slightly awkward when hwcap2 appeared out of nowhere...
>
> ...and riscv64 is the pathological case of that where you don't have
> hwcap but do have __riscv_hwprobe(2) and an unbounded set of keys.

I don't really have an opinion on the global glibc side of things here, 
but I don't think RISC-V is a good argument for keeping anything around 
if people don't want it otherwise.  We're just not that important of 
a target right now, there's no competitive hardware and the 
fragmentation is so pathologically bad that we'll just end up adding 
some crazy constraints to whatever people are trying to do.  Hopefully 
that will change at some point, but for now it's hard to make much of an 
argument for adding global complexity to improve RISC-V performance.

That said, I think some sort of "list of required features -> 
implementation pairs" would match what we're doing with IFUNCs.  The 
"required feature" set would have to be very big as we have many 
ISA/vendor features in RISC-V, and the features are a bit more complex 
than just a bitmap (some are multiple bits, possibly even quantitative) 
otherwise that's just what we're doing with the IFUNCs I can think of.

Right now we're just using IFUNCs because we can write adhoc arbitrary C 
code to encode those rules, but I don't see any reason we couldn't have 
an interface that's more specific to the features and priority rules.  
We'll need to end up doing something along those lines when we do 
function multi-versioning in GCC, so maybe we just piggyback on that 
effort?

> but that's not a "no" :-)
>
> i'm curious if anyone has real-world examples of hand-written ifunc
> resolvers _not_ in a libc?
>
>> zw