From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qv1-xf2b.google.com (mail-qv1-xf2b.google.com [IPv6:2607:f8b0:4864:20::f2b]) by sourceware.org (Postfix) with ESMTPS id 93FB13857004 for ; Wed, 25 Aug 2021 15:05:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 93FB13857004 Received: by mail-qv1-xf2b.google.com with SMTP id g11so42077qvd.2 for ; Wed, 25 Aug 2021 08:05:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=IOZsU51PEsH+dSSSmf+7hcAh9/MoG05epRdGiIGIAOM=; b=nBTvddiDXxrQOpZBagzy2Lb6aqt75J2Wnqu9eOFaDOVIqbDUzoEuoT+BzG8B/9oDfd l4DILy95nnfQQLmx+TLwdD7vZbV9pKw1lIwybzF4va+3Zm16K1dmVE9QjFmj05YxjIet S2yZmR2Am/BQFgU4cBdjuW0797czM0fuPveNPX+H85AIv3cw7Lll2Eg+lukIzYzT8Y4e QhoeCYXbKAh15+FqqnH59jXqbEf5Jp5N/Oqk0N+0nqlh4hYsmOB8JnnpuXsC2B8nuLCL GWCSgBAGfZ74bRihSZnyi+pdmQ9nptWFC2NEsxT/rTeBvIwYrBnV3Mf+e3JvEppey6Sc k8Ng== X-Gm-Message-State: AOAM530L7KhMGZ6UocNpzchzpqXSw93W1O+zpooL7gSvAmZ2ae+eh/h4 rXDTy8jdTbMvPLhERuPQi4PV9oBBXlXbzQ== X-Google-Smtp-Source: ABdhPJyI4epdLdrG6cZxSKaK7qcJXOrznt65hyr6lQblSqzwP8NEU4EG4+aB4Bv/vXoFOdYQDmgvAQ== X-Received: by 2002:ad4:5be4:: with SMTP id k4mr22184608qvc.46.1629903937952; Wed, 25 Aug 2021 08:05:37 -0700 (PDT) Received: from ?IPv6:2804:431:c7ca:1a68:e3de:10e0:28dd:40db? ([2804:431:c7ca:1a68:e3de:10e0:28dd:40db]) by smtp.gmail.com with ESMTPSA id h16sm321qtx.23.2021.08.25.08.05.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 25 Aug 2021 08:05:37 -0700 (PDT) Subject: Re: glibc 2.34 and Yocto Project's "uninative" To: Richard Purdie , Libc-help References: <978104bdbebcd09d159f1713499cf1315edf40f2.camel@linuxfoundation.org> From: Adhemerval Zanella Message-ID: <4224ac85-83b7-6fe8-6781-b8e8f514924a@linaro.org> Date: Wed, 25 Aug 2021 12:05:35 -0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <978104bdbebcd09d159f1713499cf1315edf40f2.camel@linuxfoundation.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-6.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_BLACK autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-help@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-help mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Aug 2021 15:05:49 -0000 On 20/08/2021 11:37, Richard Purdie via Libc-help wrote: > Hi, > > Yocto Project is a cross compiling build system used to build customised > Linux, RTOSs, firmware and more. It has some interesting technology. One piece > of it is "pseudo" (http://git.yoctoproject.org/cgit.cgi/pseudo/) which is an > LD_PRELOAD which intercepts calls to libc and fakes root privileges. One way or > another we've managed to keep that working with multiple libc versions for the > last decade. > > A second piece of technology is uninative. We build a lot binary artefacts, some > for the target, some as tools running natively on the build system. We have a > cache of these artefacts people can share and reuse. Uninative is our way of > allowing one binary to run on any host distro. We do that by shipping a ld+libc > binary shim and change the interpreter in the native binary to point at our own. > As long as the shim is the same version or later than system version, it works. > > In glibc 2.34, the merge of libdl and libpthread into libc is causing a problem > for us. Basically, I've been able to make pseudo work and I can make uninative > work however I can't make them both work together with glibc 2.34. > > Pseudo uses minimal dl calls from libdl (dlsym, dlvsym and dlerror) and has a > pthread mutex, therefore it links to -ldl and -lpthread. > > The issue is that pseudo being an LD_PRELOAD, if linked against glibc 2.34 > doesn't "see" symbols in libdl and just links to libc. I did try forcing older > versions of the symbols along the lines of: > > __asm__(".symver dlerror,dlerror@GLIBC_" DLSYMVER); > __asm__(".symver dlvsym,dlvsym@GLIBC_" DLVSYMVER); > __asm__(".symver dlsym,dlsym@GLIBC_" DLSYMVER); > > with some arch specific version number defines however even if I do this, and > put back a libdl.so symlink to the lib, the linker ignores the weak wildcard > reference and links back to libc.so itself. When we then use this preloaded lib > on a system with an older libc: > > relocation error: [...]/libpseudo.so: symbol dlerror, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference Unfortunately this was never officially supported, since it is basically trying to run a binary built in a newer glibc on an older one. The issue is not that linker put back libc.so, but rather that with GLIBC 2.34 there is a lot of more default symbols that are tied to GLIBC_2.34 version. For instance: $ cat dl.c #include #include int main (int argc, char *argv[]) { void *h = dlopen (argv[1], RTLD_NOW); assert (h != 0); return 0; } asm (".symver dlopen,dlopen@GLIBC_2.2.5"); $ readelf -Ws dl | grep GLIBC_2.34 1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_main@GLIBC_2.34 (2) 24: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_main@GLIBC_2.34 The __libc_start_main has the additional issue that it is provided by crt1.o, so the symver in the main TU won't solve it. You will need to hack it with the static linker, even by adding some wrap symbol to redirect to the __libc_start_main@GLIBC_2.2.5. > > I did try ensuring libpseudo.so has a RUNPATH that includes our uninative libc > however the loader ignores that for loading libc. I did then add our uninative > libc path to LD_LIBRARY_PATH however that fails: And I don't think this would work, it would force two libc with potentially two different versions. > > libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference > > since the ld being used doesn't match the libc. > > Can anyone see a way we could make things work? I don't have easy library based solution for the requisites you posed, building a newer glibc and running on a older one. I think what *might* work is to provide a auditor library linked against the newer glibc and it then intercepts and routes the required library calls. You will need to redistribute the libc which the auditor was linked against.