glibc 2.34 and Yocto Project's "uninative"

glibc 2.34 and Yocto Project's "uninative"

[Richard Purdie]

Hi,

Yocto Project is a cross compiling build system used to build customised
Linux, RTOSs, firmware and more. It has some interesting technology. One piece
of it is "pseudo" (http://git.yoctoproject.org/cgit.cgi/pseudo/) which is an 
LD_PRELOAD which intercepts calls to libc and fakes root privileges. One way or
another we've managed to keep that working with multiple libc versions for the
last decade.

A second piece of technology is uninative. We build a lot binary artefacts, some
for the target, some as tools running natively on the build system. We have a
cache of these artefacts people can share and reuse. Uninative is our way of
allowing one binary to run on any host distro. We do that by shipping a ld+libc
binary shim and change the interpreter in the native binary to point at our own.
As long as the shim is the same version or later than system version, it works.

In glibc 2.34, the merge of libdl and libpthread into libc is causing a problem
for us. Basically, I've been able to make pseudo work and I can make uninative
work however I can't make them both work together with glibc 2.34.

Pseudo uses minimal dl calls from libdl (dlsym, dlvsym and dlerror) and has a
pthread mutex, therefore it links to -ldl and -lpthread.

The issue is that pseudo being an LD_PRELOAD, if linked against glibc 2.34
doesn't "see" symbols in libdl and just links to libc. I did try forcing older
versions of the symbols along the lines of:

__asm__(".symver dlerror,dlerror@GLIBC_" DLSYMVER);
__asm__(".symver dlvsym,dlvsym@GLIBC_" DLVSYMVER);
__asm__(".symver dlsym,dlsym@GLIBC_" DLSYMVER);

with some arch specific version number defines however even if I do this, and
put back a libdl.so symlink to the lib, the linker ignores the weak wildcard
reference and links back to libc.so itself. When we then use this preloaded lib
on a system with an older libc:

relocation error: [...]/libpseudo.so: symbol dlerror, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference

I did try ensuring libpseudo.so has a RUNPATH that includes our uninative libc
however the loader ignores that for loading libc. I did then add our uninative
libc path to LD_LIBRARY_PATH however that fails:

libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

since the ld being used doesn't match the libc.

Can anyone see a way we could make things work?

Cheers,

Richard

Re: glibc 2.34 and Yocto Project's "uninative"

[Adhemerval Zanella]

On 20/08/2021 11:37, Richard Purdie via Libc-help wrote:
> Hi,
> 
> Yocto Project is a cross compiling build system used to build customised
> Linux, RTOSs, firmware and more. It has some interesting technology. One piece
> of it is "pseudo" (http://git.yoctoproject.org/cgit.cgi/pseudo/) which is an 
> LD_PRELOAD which intercepts calls to libc and fakes root privileges. One way or
> another we've managed to keep that working with multiple libc versions for the
> last decade.
> 
> A second piece of technology is uninative. We build a lot binary artefacts, some
> for the target, some as tools running natively on the build system. We have a
> cache of these artefacts people can share and reuse. Uninative is our way of
> allowing one binary to run on any host distro. We do that by shipping a ld+libc
> binary shim and change the interpreter in the native binary to point at our own.
> As long as the shim is the same version or later than system version, it works.
> 
> In glibc 2.34, the merge of libdl and libpthread into libc is causing a problem
> for us. Basically, I've been able to make pseudo work and I can make uninative
> work however I can't make them both work together with glibc 2.34.
> 
> Pseudo uses minimal dl calls from libdl (dlsym, dlvsym and dlerror) and has a
> pthread mutex, therefore it links to -ldl and -lpthread.
> 
> The issue is that pseudo being an LD_PRELOAD, if linked against glibc 2.34
> doesn't "see" symbols in libdl and just links to libc. I did try forcing older
> versions of the symbols along the lines of:
> 
> __asm__(".symver dlerror,dlerror@GLIBC_" DLSYMVER);
> __asm__(".symver dlvsym,dlvsym@GLIBC_" DLVSYMVER);
> __asm__(".symver dlsym,dlsym@GLIBC_" DLSYMVER);
> 
> with some arch specific version number defines however even if I do this, and
> put back a libdl.so symlink to the lib, the linker ignores the weak wildcard
> reference and links back to libc.so itself. When we then use this preloaded lib
> on a system with an older libc:
> 
> relocation error: [...]/libpseudo.so: symbol dlerror, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference

Unfortunately this was never officially supported, since it is basically
trying to run a binary built in a newer glibc on an older one.  The
issue is not that linker put back libc.so, but rather that with GLIBC
2.34 there is a lot of more default symbols that are tied to GLIBC_2.34
version.

For instance:

$ cat dl.c 
#include <dlfcn.h>
#include <assert.h>

int main (int argc, char *argv[])
{
  void *h = dlopen (argv[1], RTLD_NOW);
  assert (h != 0);
  return 0;
}

asm (".symver dlopen,dlopen@GLIBC_2.2.5");

$ readelf -Ws dl | grep GLIBC_2.34
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34 (2)
    24: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34


The __libc_start_main has the additional issue that it is provided by crt1.o,
so the symver in the main TU won't solve it.  You will need to hack it with
the static linker, even by adding some wrap symbol to redirect to the 
__libc_start_main@GLIBC_2.2.5.


> 
> I did try ensuring libpseudo.so has a RUNPATH that includes our uninative libc
> however the loader ignores that for loading libc. I did then add our uninative
> libc path to LD_LIBRARY_PATH however that fails:

And I don't think this would work, it would force two libc with potentially
two different versions.

> 
> libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
> 
> since the ld being used doesn't match the libc.
> 
> Can anyone see a way we could make things work?
I don't have easy library based solution for the requisites you posed,
building a newer glibc and running on a older one.  I think what *might*
work is to provide a auditor library linked against the newer glibc and
it then intercepts and routes the required library calls.  You will need
to redistribute the libc which the auditor was linked against.

Re: glibc 2.34 and Yocto Project's "uninative"

[Richard Purdie]

On Wed, 2021-08-25 at 12:05 -0300, Adhemerval Zanella wrote:
> 
> On 20/08/2021 11:37, Richard Purdie via Libc-help wrote:
> > Hi,
> > 
> > Yocto Project is a cross compiling build system used to build customised
> > Linux, RTOSs, firmware and more. It has some interesting technology. One piece
> > of it is "pseudo" (http://git.yoctoproject.org/cgit.cgi/pseudo/) which is an 
> > LD_PRELOAD which intercepts calls to libc and fakes root privileges. One way or
> > another we've managed to keep that working with multiple libc versions for the
> > last decade.
> > 
> > A second piece of technology is uninative. We build a lot binary artefacts, some
> > for the target, some as tools running natively on the build system. We have a
> > cache of these artefacts people can share and reuse. Uninative is our way of
> > allowing one binary to run on any host distro. We do that by shipping a ld+libc
> > binary shim and change the interpreter in the native binary to point at our own.
> > As long as the shim is the same version or later than system version, it works.
> > 
> > In glibc 2.34, the merge of libdl and libpthread into libc is causing a problem
> > for us. Basically, I've been able to make pseudo work and I can make uninative
> > work however I can't make them both work together with glibc 2.34.
> > 
> > Pseudo uses minimal dl calls from libdl (dlsym, dlvsym and dlerror) and has a
> > pthread mutex, therefore it links to -ldl and -lpthread.
> > 
> > The issue is that pseudo being an LD_PRELOAD, if linked against glibc 2.34
> > doesn't "see" symbols in libdl and just links to libc. I did try forcing older
> > versions of the symbols along the lines of:
> > 
> > __asm__(".symver dlerror,dlerror@GLIBC_" DLSYMVER);
> > __asm__(".symver dlvsym,dlvsym@GLIBC_" DLVSYMVER);
> > __asm__(".symver dlsym,dlsym@GLIBC_" DLSYMVER);
> > 
> > with some arch specific version number defines however even if I do this, and
> > put back a libdl.so symlink to the lib, the linker ignores the weak wildcard
> > reference and links back to libc.so itself. When we then use this preloaded lib
> > on a system with an older libc:
> > 
> > relocation error: [...]/libpseudo.so: symbol dlerror, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference
> 
> Unfortunately this was never officially supported, since it is basically
> trying to run a binary built in a newer glibc on an older one.  The
> issue is not that linker put back libc.so, but rather that with GLIBC
> 2.34 there is a lot of more default symbols that are tied to GLIBC_2.34
> version.

Thanks for the reply. I do appreciate this isn't officially supported. I had
hoped that by using minimal calls we might have been able to figure something
out but I hadn't realised there were the other symbols being added. That
certainly complicates it and explains a few things.

> For instance:
> 
> $ cat dl.c 
> #include <dlfcn.h>
> #include <assert.h>
> 
> int main (int argc, char *argv[])
> {
>   void *h = dlopen (argv[1], RTLD_NOW);
>   assert (h != 0);
>   return 0;
> }
> 
> asm (".symver dlopen,dlopen@GLIBC_2.2.5");
> 
> $ readelf -Ws dl | grep GLIBC_2.34
>      1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34 (2)
>     24: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34
> 
> 
> The __libc_start_main has the additional issue that it is provided by crt1.o,
> so the symver in the main TU won't solve it.  You will need to hack it with
> the static linker, even by adding some wrap symbol to redirect to the 
> __libc_start_main@GLIBC_2.2.5.

Needing to hack these kinds of internals starts to make me worry a lot about
whether there are details that will trip things up at runtime.

> > I did try ensuring libpseudo.so has a RUNPATH that includes our uninative libc
> > however the loader ignores that for loading libc. I did then add our uninative
> > libc path to LD_LIBRARY_PATH however that fails:
> 
> And I don't think this would work, it would force two libc with potentially
> two different versions.

It was very clear that the ld and libc need to be matched from the kinds of
mismatches and I gave up on this route quickly.

> > 
> > libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
> > 
> > since the ld being used doesn't match the libc.
> > 
> > Can anyone see a way we could make things work?
> I don't have easy library based solution for the requisites you posed,
> building a newer glibc and running on a older one.  I think what *might*
> work is to provide a auditor library linked against the newer glibc and
> it then intercepts and routes the required library calls.  You will need
> to redistribute the libc which the auditor was linked against.

The other libc is already there so that bit is easy. I hadn't thought about
using an auditor library and I'll have to explore that.

I was going to go down the route of dummy libraries to link against however I
realised it was easier for testing just to pull down 2.33 binaries and use those
to link against. I did have to replace use of pthread_atfork with
__register_atfork which is ugly but probably okish for our use as it has been
there since 2.3.2.

That probably gets us out the immediate problem although it does highlight we're
doing something that isn't supported and could break again in ways we may
struggle to fix. We can probably replace the pthread linkage for the mutex with
direct code/syscall usage. We're going to need the libdl usage regardless though
so it may be worth us figuring out the dummy library to link against for that
piece.

Thanks for the info though, it is much appreciated and it helps a lot to
understand what was breaking and some other avenues like the auditor lib to
explore.

Cheers,

Richard

Re: glibc 2.34 and Yocto Project's "uninative"

[Adhemerval Zanella]

On 25/08/2021 17:11, Richard Purdie wrote:
> On Wed, 2021-08-25 at 12:05 -0300, Adhemerval Zanella wrote:
>>
>> On 20/08/2021 11:37, Richard Purdie via Libc-help wrote:
>>> Hi,
>>>
>>> Yocto Project is a cross compiling build system used to build customised
>>> Linux, RTOSs, firmware and more. It has some interesting technology. One piece
>>> of it is "pseudo" (http://git.yoctoproject.org/cgit.cgi/pseudo/) which is an 
>>> LD_PRELOAD which intercepts calls to libc and fakes root privileges. One way or
>>> another we've managed to keep that working with multiple libc versions for the
>>> last decade.
>>>
>>> A second piece of technology is uninative. We build a lot binary artefacts, some
>>> for the target, some as tools running natively on the build system. We have a
>>> cache of these artefacts people can share and reuse. Uninative is our way of
>>> allowing one binary to run on any host distro. We do that by shipping a ld+libc
>>> binary shim and change the interpreter in the native binary to point at our own.
>>> As long as the shim is the same version or later than system version, it works.
>>>
>>> In glibc 2.34, the merge of libdl and libpthread into libc is causing a problem
>>> for us. Basically, I've been able to make pseudo work and I can make uninative
>>> work however I can't make them both work together with glibc 2.34.
>>>
>>> Pseudo uses minimal dl calls from libdl (dlsym, dlvsym and dlerror) and has a
>>> pthread mutex, therefore it links to -ldl and -lpthread.
>>>
>>> The issue is that pseudo being an LD_PRELOAD, if linked against glibc 2.34
>>> doesn't "see" symbols in libdl and just links to libc. I did try forcing older
>>> versions of the symbols along the lines of:
>>>
>>> __asm__(".symver dlerror,dlerror@GLIBC_" DLSYMVER);
>>> __asm__(".symver dlvsym,dlvsym@GLIBC_" DLVSYMVER);
>>> __asm__(".symver dlsym,dlsym@GLIBC_" DLSYMVER);
>>>
>>> with some arch specific version number defines however even if I do this, and
>>> put back a libdl.so symlink to the lib, the linker ignores the weak wildcard
>>> reference and links back to libc.so itself. When we then use this preloaded lib
>>> on a system with an older libc:
>>>
>>> relocation error: [...]/libpseudo.so: symbol dlerror, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference
>>
>> Unfortunately this was never officially supported, since it is basically
>> trying to run a binary built in a newer glibc on an older one.  The
>> issue is not that linker put back libc.so, but rather that with GLIBC
>> 2.34 there is a lot of more default symbols that are tied to GLIBC_2.34
>> version.
> 
> Thanks for the reply. I do appreciate this isn't officially supported. I had
> hoped that by using minimal calls we might have been able to figure something
> out but I hadn't realised there were the other symbols being added. That
> certainly complicates it and explains a few things.
> 
>> For instance:
>>
>> $ cat dl.c 
>> #include <dlfcn.h>
>> #include <assert.h>
>>
>> int main (int argc, char *argv[])
>> {
>>   void *h = dlopen (argv[1], RTLD_NOW);
>>   assert (h != 0);
>>   return 0;
>> }
>>
>> asm (".symver dlopen,dlopen@GLIBC_2.2.5");
>>
>> $ readelf -Ws dl | grep GLIBC_2.34
>>      1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34 (2)
>>     24: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34
>>
>>
>> The __libc_start_main has the additional issue that it is provided by crt1.o,
>> so the symver in the main TU won't solve it.  You will need to hack it with
>> the static linker, even by adding some wrap symbol to redirect to the 
>> __libc_start_main@GLIBC_2.2.5.
> 
> Needing to hack these kinds of internals starts to make me worry a lot about
> whether there are details that will trip things up at runtime.
> 
>>> I did try ensuring libpseudo.so has a RUNPATH that includes our uninative libc
>>> however the loader ignores that for loading libc. I did then add our uninative
>>> libc path to LD_LIBRARY_PATH however that fails:
>>
>> And I don't think this would work, it would force two libc with potentially
>> two different versions.
> 
> It was very clear that the ld and libc need to be matched from the kinds of
> mismatches and I gave up on this route quickly.
> 
>>>
>>> libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
>>>
>>> since the ld being used doesn't match the libc.
>>>
>>> Can anyone see a way we could make things work?
>> I don't have easy library based solution for the requisites you posed,
>> building a newer glibc and running on a older one.  I think what *might*
>> work is to provide a auditor library linked against the newer glibc and
>> it then intercepts and routes the required library calls.  You will need
>> to redistribute the libc which the auditor was linked against.
> 
> The other libc is already there so that bit is easy. I hadn't thought about
> using an auditor library and I'll have to explore that.

The auditor interface work on some examples, but unfortunately we also found
that they are fully correct on some more specific cases.  We are aiming to fix
for 2.35 [1].

> 
> I was going to go down the route of dummy libraries to link against however I
> realised it was easier for testing just to pull down 2.33 binaries and use those
> to link against. I did have to replace use of pthread_atfork with
> __register_atfork which is ugly but probably okish for our use as it has been
> there since 2.3.2.

Yes, this is what I would suggest you (use an older glibc to link this against).

> 
> That probably gets us out the immediate problem although it does highlight we're
> doing something that isn't supported and could break again in ways we may
> struggle to fix. We can probably replace the pthread linkage for the mutex with
> direct code/syscall usage. We're going to need the libdl usage regardless though
> so it may be worth us figuring out the dummy library to link against for that
> piece.

Why do you need to build it on a recent glibc and potentially use it on an older
one?  

> 
> Thanks for the info though, it is much appreciated and it helps a lot to
> understand what was breaking and some other avenues like the auditor lib to
> explore.
> 
> Cheers,
> 
> Richard
> 
> 

[1] https://patchwork.sourceware.org/project/glibc/list/?series=2577

Re: glibc 2.34 and Yocto Project's "uninative"

[Richard Purdie]

On Thu, 2021-08-26 at 08:16 -0300, Adhemerval Zanella wrote:
> 
> On 25/08/2021 17:11, Richard Purdie wrote:
> > On Wed, 2021-08-25 at 12:05 -0300, Adhemerval Zanella wrote:
> > > 
> > > On 20/08/2021 11:37, Richard Purdie via Libc-help wrote:
> > > 
> > > > Can anyone see a way we could make things work?
> > > I don't have easy library based solution for the requisites you posed,
> > > building a newer glibc and running on a older one.  I think what *might*
> > > work is to provide a auditor library linked against the newer glibc and
> > > it then intercepts and routes the required library calls.  You will need
> > > to redistribute the libc which the auditor was linked against.
> > 
> > The other libc is already there so that bit is easy. I hadn't thought about
> > using an auditor library and I'll have to explore that.
> 
> The auditor interface work on some examples, but unfortunately we also found
> that they are fully correct on some more specific cases.  We are aiming to fix
> for 2.35 [1].
> 

I have to admit I've not used the auditor interface before so I've a little
learning to do there! Thanks for the pointer to the patchset, it is handy to
know there are known issues.

> > I was going to go down the route of dummy libraries to link against however I
> > realised it was easier for testing just to pull down 2.33 binaries and use those
> > to link against. I did have to replace use of pthread_atfork with
> > __register_atfork which is ugly but probably okish for our use as it has been
> > there since 2.3.2.
> 
> Yes, this is what I would suggest you (use an older glibc to link this against).
> 
> > 
> > That probably gets us out the immediate problem although it does highlight we're
> > doing something that isn't supported and could break again in ways we may
> > struggle to fix. We can probably replace the pthread linkage for the mutex with
> > direct code/syscall usage. We're going to need the libdl usage regardless though
> > so it may be worth us figuring out the dummy library to link against for that
> > piece.
> 
> Why do you need to build it on a recent glibc and potentially use it on an older
> one?  
> 

We build standalone (and relocatable at install) tarballs of tools that can be
used on older distros to support our builds where we need newer tools such as
binutils/gcc/tar. We build most of our binaries against a recent libc and then
include a libc in those tarballs. We also support sharing of our "native" tools
amongst ranges of host distros using our uninative mechanism which involves
changing the interpreter to our own and using a latest libc everywhere.

pseudo is our fakeroot intercept that works via an LD_PRELOAD. We build that
against our own libc as with everything else and it will likely be more recent
that the host distro it runs on. We can't change the interpreter of binaries
from the host distro but we do need to still intercept their libc calls and it
will end up in an environment where the libc versions can be mixed.

As such, the preload portion of pseudo needs to run against older libcs.

Cheers,

Richard

Re: glibc 2.34 and Yocto Project's "uninative"

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2995 bytes --]

On 20 Aug 2021 15:37, Richard Purdie via Libc-help wrote:
> Yocto Project is a cross compiling build system used to build customised
> Linux, RTOSs, firmware and more. It has some interesting technology. One piece
> of it is "pseudo" (http://git.yoctoproject.org/cgit.cgi/pseudo/) which is an 
> LD_PRELOAD which intercepts calls to libc and fakes root privileges. One way or
> another we've managed to keep that working with multiple libc versions for the
> last decade.

this sounds like Debian's fakeroot [1] & Gentoo's sandbox [2].  these have
always been a challenge to keep working, and to be complete.  they're always
"best effort" because they require:
(1) every program being dynamic.  any staticly linked programs won't work.
(2) every interesting call having an actual interception point by interposing
    known functions that the C library provides.  any calls the C library
    makes via PLT bypass cannot be captured, and any syscalls the C library
    or the app makes cannot be captured.
(3) a superset of every symbol (versioned & unversioned) that a C library
    might expose.  this in particular is messy.

[1] https://wiki.debian.org/FakeRoot
[2] https://wiki.gentoo.org/wiki/Sandbox_(Portage)

a tool using ptrace to intercept the syscall addresses all these, but at the
cost of being slower (since every syscall is notified).  if you can require
seccomp & Linux 3.5+, then things look a lot better here as you can setup a
seccomp filter for the process tree and only get notified for the exact set
of syscalls you're interested in.  it does mean you have to handle every
syscall wart across different architectures, but that's a bit easier to do
than all the variety of C library ABI warts.
(i'll ignore instability in the ptrace framework itself that we've observed
when using the interface at scale.)

i haven't reviewed the even newer seccomp notify [3] mechanism to see if
that's useful.  maybe that would offer even more relief.

[3] https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html
[3] https://brauner.github.io/2020/07/23/seccomp-notify.html

> A second piece of technology is uninative. We build a lot binary artefacts, some
> for the target, some as tools running natively on the build system. We have a
> cache of these artefacts people can share and reuse. Uninative is our way of
> allowing one binary to run on any host distro. We do that by shipping a ld+libc
> binary shim and change the interpreter in the native binary to point at our own.
> As long as the shim is the same version or later than system version, it works.

we do this in CrOS with our toolchains by largely using pax-utils's lddtree [4].
it will bundle the active C library (and any other libs), and then generate a
wrapper shell script for each program that is actually executed at runtime.
that does the ldso & ldpath dance so it's all relocatable.

[4] https://gitweb.gentoo.org/proj/pax-utils.git/tree/lddtree.py
-mike

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]