From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by sourceware.org (Postfix) with ESMTPS id 559933858415 for ; Tue, 16 May 2023 22:51:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 559933858415 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=dabbelt.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=dabbelt.com Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-52160f75920so10314214a12.2 for ; Tue, 16 May 2023 15:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dabbelt-com.20221208.gappssmtp.com; s=20221208; t=1684277511; x=1686869511; h=content-transfer-encoding:mime-version:message-id:to:from:cc :in-reply-to:subject:date:from:to:cc:subject:date:message-id :reply-to; bh=Dt6EKipFYHNPh4K3s1XGZEXD8aNKgGBljhWFVSDczro=; b=fV88fuUoC8j78IjU2w2jE2L3q+t8v1joMBOubVW2P9xK7/aSIlswgNQEg4AVwj5ilL rjkxYG16g6qpv7DuPtXIFaRkeE2mBzpnMVZiL/ZINbnQaBjdHXXeI5ZvKMeT6RT76b66 QzLvG4cObhbPjkidOmqCrxjLoEFv5UoWb1OgTdyqZIShMHEO0qb7G5KNztgscb3G/7ti fgQgiT3axJ6lON4dDsqVX/QjvpcDkihMHBoiVryDt06HKW2BKMkEKjFoRk4vtI1oOlLn k2eYFG8o1dLS5DMYYLcj6Z/MAoPzcWnYvb3nGrmJpVUTl9zPDY+EYUmNyiFXh7KMqyg9 yf6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684277511; x=1686869511; h=content-transfer-encoding:mime-version:message-id:to:from:cc :in-reply-to:subject:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Dt6EKipFYHNPh4K3s1XGZEXD8aNKgGBljhWFVSDczro=; b=GCV+car5+kUhqF4AhZ1aV+hSv4rbRs3S14P2r8mqFz5aC0QMA9hvUxpyEQfUAkHLr3 9F4IEIluZFkOikHRCrR/Usoh2Ck2xFjcbqpCnNjjiJJDqQqsSnQBeBpVBuzo3sC9Qu8n khGbVg3ByJfYGxFCTOP4MCSJ6IoKYUkEolxbqWz1iWScDIxGh5tkaC2tDePHYlZ3zCdS cP8ehSCI7sp7eP92yPq+NYFZzZcHJabo+qdxbcLPiaWKZr1IyauDY7An6ToNmJo0+yoG v6cbYb0WpwnLttG1LMXL5ttDqrnxAJRRjY0RonBc+/4hWtoPzSzLLHEiCn/wKGATNdE0 jQcA== X-Gm-Message-State: AC+VfDyNc9LyOq9sRy42i4zsXSo4P0d6eRYRY7XAzUC6LO78CrLCq+cl HFYWVK/oi0pyMsMNoUeviCzd1A== X-Google-Smtp-Source: ACHHUZ4/aZaPwd0FpRy1H53gtEplUqnIxLOMMqZg3idpZ7De+wvghKGgVdlW9Y+w0ComIPiFb0dKyg== X-Received: by 2002:a05:6a20:72a1:b0:100:6a95:c289 with SMTP id o33-20020a056a2072a100b001006a95c289mr43119334pzk.5.1684277510901; Tue, 16 May 2023 15:51:50 -0700 (PDT) Received: from localhost ([50.221.140.188]) by smtp.gmail.com with ESMTPSA id fv4-20020a17090b0e8400b0024e0141353dsm100848pjb.28.2023.05.16.15.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 May 2023 15:51:50 -0700 (PDT) Date: Tue, 16 May 2023 15:51:50 -0700 (PDT) X-Google-Original-Date: Tue, 16 May 2023 15:51:28 PDT (-0700) Subject: Re: global pointer gets overwritten with dlopen(3) on RISC-V In-Reply-To: <20230516035647.cyii3vhk2r3ax5ss@google.com> CC: fweimer@redhat.com, l.stelmach@samsung.com, libc-alpha@sourceware.org, schwab@suse.de, adhemerval.zanella@linaro.org, joseph@codesourcery.com, binutils@sourceware.org, m.pikula@partner.samsung.com, m.szyprowski@samsung.com, k.lewandowsk@samsung.com, jeffreyalaw@gmail.com From: Palmer Dabbelt To: maskray@google.com Message-ID: Mime-Version: 1.0 (MHng) Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,KAM_INFOUSMEBIZ,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Mon, 15 May 2023 20:56:47 PDT (-0700), maskray@google.com wrote: > On 2023-05-12, Palmer Dabbelt wrote: >>On Fri, 12 May 2023 17:05:02 PDT (-0700), maskray@google.com wrote: >>>On 2023-05-12, Palmer Dabbelt wrote: >>>>On Fri, 12 May 2023 15:34:21 PDT (-0700), maskray@google.com wrote: >>>>>On 2023-05-12, Palmer Dabbelt wrote: >>>>>>On Fri, 12 May 2023 14:09:08 PDT (-0700), maskray@google.com wrote: >>>>>>>On 2023-05-12, Palmer Dabbelt wrote: >>>>>>>>On Fri, 12 May 2023 13:11:43 PDT (-0700), fweimer@redhat.com wrote: >>>>>>>>>* Fangrui Song: >>>>>>>>> >>>>>>>>>>>1. Make -mno-relax the default for ld(1) (on Linux?). We have no >>>>>>>>>>>benchmarks whatsoever, but global variables aren't very popular in >>>>>>>>>>>application code these days and the gp register allows access to a >>>>>>>>>>>single memory page (4kB) only. No big deal really. >>>>>>>>>> >>>>>>>>>>I do agree that --no-relax-gp is a sensible default choice for GNU ld. >>>>>>>>>>https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation#global-pointer-relaxation >>>>>>>>>> >>>>>>>>>>Perhaps you can start a separate topic on binutils? :) >>>>>>>>>> >>>>>>>>>>According to a doc from SiFive about -static -mcpu=sifive-u74 builds, >>>>>>>>>>https://docs.google.com/spreadsheets/d/14V7cPbyc80AcGHzsMaw9hYb232dzRbGCmTApnxj-SpU/edit#gid=0 >>>>>>>>>>global pointer relaxation saves at best 0.5% size (I guess that refers >>>>>>>>>>to .text. If we count all allocable sections, the percentage is likely >>>>>>>>>>even smaller.) >>>>>>>>> >>>>>>>>>For a mature toolchain, 0.5% in code size reduction would be *a lot*, >>>>>>>>>so I wouldn't dismiss that. >>>>>>>> >>>>>>>>That's broadly speaking why it sticks around. We've got a bunch of >>>>>>>>headaches related to relaxation, GP or otherwise, but they improve >>>>>>>>performance and nobody's figured out how to replace that yet. >>>>>>>> >>>>>>>>>Do we have a reproducer? Is the issue actually about gp relaxation for >>>>>>>>>the main executable? >>>>>>>> >>>>>>>>In general we don't reference GP from shared libraries as we don't >>>>>>>>have a GP save/restore scheme. There may be a bug floating around >>>>>>>>here somewhere, in which case we should fix it, but the original post >>>>>>>>sounds like it wasn't a supported use case. >>>>>>>> >>>>>>>>>Thanks, >>>>>>>>>Florian >>>>>>> >>>>>>>Global pointer relaxation only applies to +-2KiB data relative to __global_pointer$ (= .sdata + 0x800). >>>>>>>The area that potentially benefits global pointer relaxation is very small. >>>>>>> >>>>>>>0.5% code size reduction (relative to .text?) is the best case. I >>>>>>>suspect the program somehow has a lot global variable accesses and >>>>>>>placing these variables in .sdata helps. >>>>>>> >>>>>>>I've got results from Yingwei Zheng at PLCT lab using many >>>>>>>configurations. The saving is like 0.1%. >>>>>>>https://docs.google.com/spreadsheets/d/1Gz0h-C4U0toa9qELFtRaEWT_CzauE5JD9xMsLR8RyK8/edit#gid=1721258109 >>>>>>> >>>>>>>On the binutils side, we occasionally see patches to fix global pointer >>>>>>>relaxation bugs, e.g. the patch just sent a few hours ago: >>>>>>>https://sourceware.org/pipermail/binutils/2023-May/127413.html >>>>>>> >>>>>>>I do not know the embedded toolchain well, but for Linux desktop/server, >>>>>>>disabling global pointer relaxation seems like a sensible choice. If we >>>>>>>discover a better way to utilize GP (x3) in the future, disabling global >>>>>>>pointer relaxation today will result in fewer compatibility issues. >>>>>> >>>>>>This comes up all the time, you're just pushing for a backdoor ABI >>>>>>break. I get the desire to remove GP, if we were to be able to redo >>>>>>things I'd also not have it, but it's in the ABI and we can't change >>>>>>the binaries that exist. >>>>>> >>>>>>If you want a GP-free ABI then you should just go write one up. Then >>>>>>it'll become a distro problem, and if it turns out that users also >>>>>>don't want in then the GP ABI will rot and we can eventually deprecate >>>>>>it. >>>>> >>>>>I am advocating for a change in GNU ld to make --no-relax-gp the default >>>>>option, but I am not sure it can be considered an ABI break. >>>>> >>>>>When using ld --no-relax-gp, the conversion of code sequences to gp is >>>>>disabled, thus eliminating an assumption related to global pointer relaxation. >>>>>If an executable is relinked without global pointer relaxation, it should still >>>>>function properly. >>>>> >>>>>To the best of my knowledge, there is no official documentation that designates >>>>>linker relaxation as a mandatory feature. Relaxation schemes, including global >>>>>pointer relaxation, are optional. Making an optional feature opt-in does not >>>>>constitute an ABI break. >>>>> >>>>>glibc continues to initialize gp to __global_pointer$ to accommodate users who >>>>>opt-in for global pointer relaxation. Removing the initialization of gp would >>>>>indeed be an ABI break, and I have never proposed such a change. >>>> >>>>The ABI break is using GP for something else, but that always ends up >>>>being part of the argument for disabling GP-based relaxation. Without >>>>that the argument ends up just being that the GP-relaxations are too >>>>complicated for the benefit they provide. >>> >>>This has happened. Some platforms are investigating or using GP for >>>software shadow call stack. I think they include Android and Fuchsia. I >>>am not affiliated with them. >> >>IIUC the Android shadow stack stuff isn't 100% decided, but it's the >>way they're headed. I'd argue that's a great reason to actually write >>this down in an ABI spec: we've got users who either are or will soon >>violate the spec, we should fix the spec rather than force those users >>to fork it. >> >>>>I certainly understand that argument, these (and for the rest of the >>>>relaxation stuff) is a huge amount of pain for a small amount of >>>>benefit (which seems to be mostly based on benchmark dragracing, which >>>>is never that strong of an argument). That said, last time we had >>>>this discussion was only a few months ago. I don't think anything has >>>>changed since then so I doubt things well go any differently. >>> >>>The rest of the relaxation mechanism does offer significant savings that >>>should not be overlooked. I have heard claims of up to a 10% .text >>>reduction in certain embedded systems. >>> >>>However, it is important to acknowledge the costs associated with >>>increased toolchain complexity and bloating of debug information, LSDA , >>>and custom metadata sections. I discussed these concerns in detail at >>>https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation >>>(Apologies if the title came across as derogatory. It was never my >>>intention, and I genuinely enjoyed delving into the intricacies of >>>this toolchain technology.) >> >>I'd argue we should make this new ABI just forbid all relaxation. >>It's a ton of headache for an unknown amount of benefit, we can't >>easily measure it because we haven't chased down all the missed >>optimizations due to relaxation. Modern Linux distros are all PIE >>anyway, so it's not like most of the relaxation does much. >> >>Pretty much everyone who looks at RISC-V says relaxation is bad, let's >>go let folks prove that out -- and if it's right we can get ride of a >>pile of complexity, which is great. > > It's too late to make linker relaxation opt-in. I'll do my part to fix > some LLVM assembler deficiency, though. > >>>>Just turning GP-based relaxation off by default doesn't get rid of the >>>>complexity, they're still in the ABI and people will still use them >>>>(even if just for benchmark dragracing) so they'll still need to work. >>> >>>Regarding the ABI story, shared objects do not utilize global pointer >>>relaxation. Therefore, if only an executable requires GP for a >>>platform-specific purpose, I believe there is no ABI risk involved. The >>>risk arises when shared objects rely on the executable's initialized GP >>>for certain purposes. >>> >>>ABI breaks related to global pointer usage occur when a platform >>>initially ships something using GP and then switches to a different GP >>>usage. >> >>Using GP for anything that's not the global pointer violates the >>psABI. > > I disagree. The psABI makes it clear that linker relaxation is optional > with wording like "With linker relaxation enabled" "the linker is > permitted to use ... relaxation". > >>It's just a spec violation and I agree it's possible to >>violate the specs while still producing working systems. At that >>point it's really up to the projects to ensure their spec-violating >>systems work, there's not much we can do on the toolchain side of >>things. I'm guilty of doing that too (just look at a buncho of the >>Linux port, for example), but that's always a complicated way to do >>things. > > "If a platform requires use of a dedicated general-purpose register for a > platform-specific purpose, it is recommended to use gp (x3). The > platform ABI specification must document the use of this register. For > such platforms, care must be taken to ensure all code (compiler > generated or otherwise) avoids using gp in a way incompatible with the > platform specific purpose, and that global > pointer relaxation is disabled in the toolchain." > >>IMO it's still better to have a spec written down so the toolchain and >>users can be on the same page. > > You can make a proposal that global pointer must not be used for other > purposes and someone will object to it:) https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/379 >>>As I mentioned earlier, many non-Linux/glibc platforms don't use global >>>pointer relaxation from the beginning. Even within Linux/glibc, a new >>>distribution may choose not to use global pointer relaxation initially. >>>For these platforms, there would be no ABI break. >>> >>>Changing the default behavior does indeed have an impact. >>>--no-relax-gp is a binutils 2.41 feature, which means that projects that >>>discover improved ways to utilize GP in the future need to avoid >>>--no-relax-gp to maintain compatibility with older GNU ld versions. >>>They could use --no-relax with GNU ld versions, but that's a big hammer. >> >>I would describe that as an ABI break. > > I provide evidence to back up my point. > Can you give reasons that this is an ABI break? And break what? > >>> >>>>> >>>>>>>Haiku >>>>>>>(https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/298#issuecomment-1344724796), Android, and Fuchsia >>>>>>>have mentioned that they don't use global pointer relaxation.