From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by sourceware.org (Postfix) with ESMTPS id B1A633858C41 for ; Tue, 16 May 2023 23:21:13 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B1A633858C41 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=dabbelt.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=dabbelt.com Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-643912bca6fso56294b3a.0 for ; Tue, 16 May 2023 16:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dabbelt-com.20221208.gappssmtp.com; s=20221208; t=1684279273; x=1686871273; h=content-transfer-encoding:mime-version:message-id:to:from:cc :in-reply-to:subject:date:from:to:cc:subject:date:message-id :reply-to; bh=tGwSksK2O9DcgSaJuZ6Fb8Gxix22aEIs/pPaHVJ+e48=; b=pyqL7RGjuGIHNrBWpEDAZFxN1G5BjQKH8KpDrtbhwCiKQn8CkFid15b0Ihaosd973e +EC150BTCR74WPQDfFstoYroMKTy6Ka65CH3mXYmAeu8StBEbvFfWgDxCwNLDcEViA04 96XSnRrney4kd1iGvELagOI3Ly7DDwdKGDNdPESuDjctCCZ8wc3LFEspVHpvdD1NuDn2 mO6fq41lvgYJdXFOLNlAbgCVy57NFH9/SDwH8lTkr7VneNvRSxiA7NAoXGADdurA5gzt FvbUmkvReJMu0c5buJN8ebtrunVBmJDwbw0kc4ujej5hSQ1MdYmu23H8rQC3acNVdZ83 LCFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684279273; x=1686871273; h=content-transfer-encoding:mime-version:message-id:to:from:cc :in-reply-to:subject:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=tGwSksK2O9DcgSaJuZ6Fb8Gxix22aEIs/pPaHVJ+e48=; b=LM1YGlsRhH7cLFi79ziPRQ7xKIKiG1XqgJAUsqH00WdE2BGPsbJVPViUKlulx2/sEv rNmAaFU9VF3y38n0s+rZqL+XR/ZzFSt0XrTLX+w1jWFcRSf7gUu/Ls3A4SSot0CPB7CD m4XRxg5thXcwwrTBRiys2fM6KGyqA8tWADYKVmt1+hIFWYaCA6sWz+H6dg3lLq2jiREa XrWXL8v0RZlJWoTROUlJh0cW9KSvzLcKKVEk+0zTaGrPzmll8OCFMTTgktw4txtGt+ZR hrUz+6Xti4QIdBHt7PqTt437WNZGk5tVPSY46JiXMslwvruzDVEKrTz75Kma3I/h+DOQ nmSw== X-Gm-Message-State: AC+VfDxrU9UxYBZks5G88cXkQQB+P5RAScgra8AIrmDp3U4IvoAfyCgR 238OkOYwppghcTfMNq0ouoWSXA== X-Google-Smtp-Source: ACHHUZ5J6h8fid3lm+mDwkhXV6ovITu2GhVNfo7XZUstdj17Wk9Rp2pWEQAWB4OED5ACJzt6hWXw6g== X-Received: by 2002:a05:6a20:841b:b0:107:17f6:940b with SMTP id c27-20020a056a20841b00b0010717f6940bmr3670728pzd.41.1684279272481; Tue, 16 May 2023 16:21:12 -0700 (PDT) Received: from localhost ([50.221.140.188]) by smtp.gmail.com with ESMTPSA id 16-20020a630d50000000b0050fa6546a45sm14115279pgn.6.2023.05.16.16.21.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 May 2023 16:21:11 -0700 (PDT) Date: Tue, 16 May 2023 16:21:11 -0700 (PDT) X-Google-Original-Date: Tue, 16 May 2023 16:20:49 PDT (-0700) Subject: Re: global pointer gets overwritten with dlopen(3) on RISC-V In-Reply-To: CC: fweimer@redhat.com, l.stelmach@samsung.com, libc-alpha@sourceware.org, schwab@suse.de, adhemerval.zanella@linaro.org, joseph@codesourcery.com, binutils@sourceware.org, m.pikula@partner.samsung.com, m.szyprowski@samsung.com, k.lewandowsk@samsung.com, jeffreyalaw@gmail.com From: Palmer Dabbelt To: maskray@google.com Message-ID: Mime-Version: 1.0 (MHng) Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,KAM_INFOUSMEBIZ,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Tue, 16 May 2023 15:51:50 PDT (-0700), Palmer Dabbelt wrote: > On Mon, 15 May 2023 20:56:47 PDT (-0700), maskray@google.com wrote: >> On 2023-05-12, Palmer Dabbelt wrote: >>>On Fri, 12 May 2023 17:05:02 PDT (-0700), maskray@google.com wrote: >>>>On 2023-05-12, Palmer Dabbelt wrote: >>>>>On Fri, 12 May 2023 15:34:21 PDT (-0700), maskray@google.com wrote: >>>>>>On 2023-05-12, Palmer Dabbelt wrote: >>>>>>>On Fri, 12 May 2023 14:09:08 PDT (-0700), maskray@google.com wrote: >>>>>>>>On 2023-05-12, Palmer Dabbelt wrote: >>>>>>>>>On Fri, 12 May 2023 13:11:43 PDT (-0700), fweimer@redhat.com wrote: >>>>>>>>>>* Fangrui Song: >>>>>>>>>> >>>>>>>>>>>>1. Make -mno-relax the default for ld(1) (on Linux?). We have no >>>>>>>>>>>>benchmarks whatsoever, but global variables aren't very popular in >>>>>>>>>>>>application code these days and the gp register allows access to a >>>>>>>>>>>>single memory page (4kB) only. No big deal really. >>>>>>>>>>> >>>>>>>>>>>I do agree that --no-relax-gp is a sensible default choice for GNU ld. >>>>>>>>>>>https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation#global-pointer-relaxation >>>>>>>>>>> >>>>>>>>>>>Perhaps you can start a separate topic on binutils? :) >>>>>>>>>>> >>>>>>>>>>>According to a doc from SiFive about -static -mcpu=sifive-u74 builds, >>>>>>>>>>>https://docs.google.com/spreadsheets/d/14V7cPbyc80AcGHzsMaw9hYb232dzRbGCmTApnxj-SpU/edit#gid=0 >>>>>>>>>>>global pointer relaxation saves at best 0.5% size (I guess that refers >>>>>>>>>>>to .text. If we count all allocable sections, the percentage is likely >>>>>>>>>>>even smaller.) >>>>>>>>>> >>>>>>>>>>For a mature toolchain, 0.5% in code size reduction would be *a lot*, >>>>>>>>>>so I wouldn't dismiss that. >>>>>>>>> >>>>>>>>>That's broadly speaking why it sticks around. We've got a bunch of >>>>>>>>>headaches related to relaxation, GP or otherwise, but they improve >>>>>>>>>performance and nobody's figured out how to replace that yet. >>>>>>>>> >>>>>>>>>>Do we have a reproducer? Is the issue actually about gp relaxation for >>>>>>>>>>the main executable? >>>>>>>>> >>>>>>>>>In general we don't reference GP from shared libraries as we don't >>>>>>>>>have a GP save/restore scheme. There may be a bug floating around >>>>>>>>>here somewhere, in which case we should fix it, but the original post >>>>>>>>>sounds like it wasn't a supported use case. >>>>>>>>> >>>>>>>>>>Thanks, >>>>>>>>>>Florian >>>>>>>> >>>>>>>>Global pointer relaxation only applies to +-2KiB data relative to __global_pointer$ (= .sdata + 0x800). >>>>>>>>The area that potentially benefits global pointer relaxation is very small. >>>>>>>> >>>>>>>>0.5% code size reduction (relative to .text?) is the best case. I >>>>>>>>suspect the program somehow has a lot global variable accesses and >>>>>>>>placing these variables in .sdata helps. >>>>>>>> >>>>>>>>I've got results from Yingwei Zheng at PLCT lab using many >>>>>>>>configurations. The saving is like 0.1%. >>>>>>>>https://docs.google.com/spreadsheets/d/1Gz0h-C4U0toa9qELFtRaEWT_CzauE5JD9xMsLR8RyK8/edit#gid=1721258109 >>>>>>>> >>>>>>>>On the binutils side, we occasionally see patches to fix global pointer >>>>>>>>relaxation bugs, e.g. the patch just sent a few hours ago: >>>>>>>>https://sourceware.org/pipermail/binutils/2023-May/127413.html >>>>>>>> >>>>>>>>I do not know the embedded toolchain well, but for Linux desktop/server, >>>>>>>>disabling global pointer relaxation seems like a sensible choice. If we >>>>>>>>discover a better way to utilize GP (x3) in the future, disabling global >>>>>>>>pointer relaxation today will result in fewer compatibility issues. >>>>>>> >>>>>>>This comes up all the time, you're just pushing for a backdoor ABI >>>>>>>break. I get the desire to remove GP, if we were to be able to redo >>>>>>>things I'd also not have it, but it's in the ABI and we can't change >>>>>>>the binaries that exist. >>>>>>> >>>>>>>If you want a GP-free ABI then you should just go write one up. Then >>>>>>>it'll become a distro problem, and if it turns out that users also >>>>>>>don't want in then the GP ABI will rot and we can eventually deprecate >>>>>>>it. >>>>>> >>>>>>I am advocating for a change in GNU ld to make --no-relax-gp the default >>>>>>option, but I am not sure it can be considered an ABI break. >>>>>> >>>>>>When using ld --no-relax-gp, the conversion of code sequences to gp is >>>>>>disabled, thus eliminating an assumption related to global pointer relaxation. >>>>>>If an executable is relinked without global pointer relaxation, it should still >>>>>>function properly. >>>>>> >>>>>>To the best of my knowledge, there is no official documentation that designates >>>>>>linker relaxation as a mandatory feature. Relaxation schemes, including global >>>>>>pointer relaxation, are optional. Making an optional feature opt-in does not >>>>>>constitute an ABI break. >>>>>> >>>>>>glibc continues to initialize gp to __global_pointer$ to accommodate users who >>>>>>opt-in for global pointer relaxation. Removing the initialization of gp would >>>>>>indeed be an ABI break, and I have never proposed such a change. >>>>> >>>>>The ABI break is using GP for something else, but that always ends up >>>>>being part of the argument for disabling GP-based relaxation. Without >>>>>that the argument ends up just being that the GP-relaxations are too >>>>>complicated for the benefit they provide. >>>> >>>>This has happened. Some platforms are investigating or using GP for >>>>software shadow call stack. I think they include Android and Fuchsia. I >>>>am not affiliated with them. >>> >>>IIUC the Android shadow stack stuff isn't 100% decided, but it's the >>>way they're headed. I'd argue that's a great reason to actually write >>>this down in an ABI spec: we've got users who either are or will soon >>>violate the spec, we should fix the spec rather than force those users >>>to fork it. >>> >>>>>I certainly understand that argument, these (and for the rest of the >>>>>relaxation stuff) is a huge amount of pain for a small amount of >>>>>benefit (which seems to be mostly based on benchmark dragracing, which >>>>>is never that strong of an argument). That said, last time we had >>>>>this discussion was only a few months ago. I don't think anything has >>>>>changed since then so I doubt things well go any differently. >>>> >>>>The rest of the relaxation mechanism does offer significant savings that >>>>should not be overlooked. I have heard claims of up to a 10% .text >>>>reduction in certain embedded systems. >>>> >>>>However, it is important to acknowledge the costs associated with >>>>increased toolchain complexity and bloating of debug information, LSDA , >>>>and custom metadata sections. I discussed these concerns in detail at >>>>https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation >>>>(Apologies if the title came across as derogatory. It was never my >>>>intention, and I genuinely enjoyed delving into the intricacies of >>>>this toolchain technology.) >>> >>>I'd argue we should make this new ABI just forbid all relaxation. >>>It's a ton of headache for an unknown amount of benefit, we can't >>>easily measure it because we haven't chased down all the missed >>>optimizations due to relaxation. Modern Linux distros are all PIE >>>anyway, so it's not like most of the relaxation does much. >>> >>>Pretty much everyone who looks at RISC-V says relaxation is bad, let's >>>go let folks prove that out -- and if it's right we can get ride of a >>>pile of complexity, which is great. >> >> It's too late to make linker relaxation opt-in. I'll do my part to fix >> some LLVM assembler deficiency, though. >> >>>>>Just turning GP-based relaxation off by default doesn't get rid of the >>>>>complexity, they're still in the ABI and people will still use them >>>>>(even if just for benchmark dragracing) so they'll still need to work. >>>> >>>>Regarding the ABI story, shared objects do not utilize global pointer >>>>relaxation. Therefore, if only an executable requires GP for a >>>>platform-specific purpose, I believe there is no ABI risk involved. The >>>>risk arises when shared objects rely on the executable's initialized GP >>>>for certain purposes. >>>> >>>>ABI breaks related to global pointer usage occur when a platform >>>>initially ships something using GP and then switches to a different GP >>>>usage. >>> >>>Using GP for anything that's not the global pointer violates the >>>psABI. >> >> I disagree. The psABI makes it clear that linker relaxation is optional >> with wording like "With linker relaxation enabled" "the linker is >> permitted to use ... relaxation". >> >>>It's just a spec violation and I agree it's possible to >>>violate the specs while still producing working systems. At that >>>point it's really up to the projects to ensure their spec-violating >>>systems work, there's not much we can do on the toolchain side of >>>things. I'm guilty of doing that too (just look at a buncho of the >>>Linux port, for example), but that's always a complicated way to do >>>things. >> >> "If a platform requires use of a dedicated general-purpose register for a >> platform-specific purpose, it is recommended to use gp (x3). The >> platform ABI specification must document the use of this register. For >> such platforms, care must be taken to ensure all code (compiler >> generated or otherwise) avoids using gp in a way incompatible with the >> platform specific purpose, and that global >> pointer relaxation is disabled in the toolchain." >> >>>IMO it's still better to have a spec written down so the toolchain and >>>users can be on the same page. >> >> You can make a proposal that global pointer must not be used for other >> purposes and someone will object to it:) > > https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/379 and unsurprisingly, it's not wanted in the psABI ;) >>>>As I mentioned earlier, many non-Linux/glibc platforms don't use global >>>>pointer relaxation from the beginning. Even within Linux/glibc, a new >>>>distribution may choose not to use global pointer relaxation initially. >>>>For these platforms, there would be no ABI break. >>>> >>>>Changing the default behavior does indeed have an impact. >>>>--no-relax-gp is a binutils 2.41 feature, which means that projects that >>>>discover improved ways to utilize GP in the future need to avoid >>>>--no-relax-gp to maintain compatibility with older GNU ld versions. >>>>They could use --no-relax with GNU ld versions, but that's a big hammer. >>> >>>I would describe that as an ABI break. >> >> I provide evidence to back up my point. >> Can you give reasons that this is an ABI break? And break what? I'm not really sure how else to describe it: the spec you're quoting says it must be documented, and the scenerio you're describing is exactly the definition of an ABI break (changing what binaries the toolchain emits and then removing support for the old binaries). Like I said above, you (or anyone else) is free to violate the spec and then go make sure the systems you're building based on that unspecified behavior continue to work well enough for the users. >> >>>> >>>>>> >>>>>>>>Haiku >>>>>>>>(https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/298#issuecomment-1344724796), Android, and Fuchsia >>>>>>>>have mentioned that they don't use global pointer relaxation.