From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by sourceware.org (Postfix) with ESMTPS id 09F1E3858C5F for ; Thu, 5 Oct 2023 12:05:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 09F1E3858C5F Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-40651a72807so8186145e9.1 for ; Thu, 05 Oct 2023 05:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696507510; x=1697112310; darn=gcc.gnu.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=RS/kJcCbc8DFrj1Fh0FyNG26AoNVHBQHWoXJQZk0QWc=; b=Ew8C85eRcH39X8pCPoeqmCwfwfXQudKXPJyex5nQL1YcW3aRj+C3iy1Uyn0ufGk/bS kkAxcuzPBE88IZd2O0JgS27fs2qhdOkqOjusQNFDICzWOHbw3DLcPhImuRboqNWACiMd 9hBUY0SJqvni+3jUclinECvNsvqVfNytFtbp4WEGeZFmux0ePiQvHyqKOdrFm7HVwpU0 WUEzy0pcF1EDl3VfUCTrqDTMohFUd56lDUizXtl60ShPZ+ZqtYLzkfzW/oSIFNZQFIgs iXw6+rH1RXUGbK7kJnZ+z3eE+85xNMaI40VO93w3ry5GjfBtlxlSH+YWG6zKS8Pb4lQI U0sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696507510; x=1697112310; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=RS/kJcCbc8DFrj1Fh0FyNG26AoNVHBQHWoXJQZk0QWc=; b=bhaKgznC6k0kHm294PCtohC+cTHQXdhr2cEWEydCjyIkG9rGGFMtcr4VB7awJXS3BO 6hnqhv8ekqYA/q/9MixJRPCPjDDZQMTT12fZAsDh54M1TxxrC9r5hO4NJSrp5wkHCIPP TekznK4wIR+Q7CW/wwgqcthw2oD0gRkdB02mwiKCdZvn1ZAj+EoEDwa2036MR+bqci9L 8xv3eJHbtBuLF2o4eYiFb+t+/12hhRdhgDCzf9p049klgn6ADO7e1z+amEx1JxxfXud6 3DkXSym6rgB2gyDpBJHZKR96Upq12l6h66W7b38RgrKWXNY6jGXX6Ei1cr88UVvBXqul Dl7g== X-Gm-Message-State: AOJu0Yzuv6VhZ8O70zEMHjfkxVcY1ULz2ZUitYXmfZHDtJkrjsonPhvt bkQG7q5bFHOASuCFesLJeRBPaOh376g= X-Google-Smtp-Source: AGHT+IHS7Glcckl4BgcZRilzFplt4D+0I602pEKkRnC9TOo373ZBkRddrKgKU0Vg1ZEHRA6fatcuzg== X-Received: by 2002:a05:600c:c9:b0:406:4d8f:9fee with SMTP id u9-20020a05600c00c900b004064d8f9feemr4953603wmm.24.1696507510105; Thu, 05 Oct 2023 05:05:10 -0700 (PDT) Received: from nz.home ([2a00:23c8:a613:101:b681:80c1:a81d:5318]) by smtp.gmail.com with ESMTPSA id m19-20020a7bcb93000000b003fe61c33df5sm3625377wmi.3.2023.10.05.05.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 05:05:09 -0700 (PDT) Received: by nz.home (Postfix, from userid 1000) id 446BE114351C8F; Thu, 5 Oct 2023 13:05:09 +0100 (BST) Date: Thu, 5 Oct 2023 13:05:09 +0100 From: Sergei Trofimovich To: Ben Boeckel Cc: gcc@gcc.gnu.org Subject: Re: Scaling -fmacro-prefix-map= to thousands entries Message-ID: References: <20231004221932.06980e3f@nz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,KAM_SHORT,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Thu, Oct 05, 2023 at 07:20:35AM -0400, Ben Boeckel wrote: > On Wed, Oct 04, 2023 at 22:19:32 +0100, Sergei Trofimovich via Gcc wrote: > > The prototype that creates equivalent of the following commands does > > work for smaller packages: > > > > -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.37-8-dev > > -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev > > -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nlohmann_json-3.11.2 > > ... > > > > The above works for small amount of options (like, 100). But around 1000 > > options we start hitting linux limits on the single environment variable > > or real-world packages like `qemu` with a ton of input depends. > > Are you trying to pass this through via `CFLAGS` and friends? Roughly via `CFLAGS`. `nixpkgs` uses it's own private `NIX_CFLAGS_COMPILE` variable which gets extended in `gcc-wrapper` shel wrapper as expcit list of arguments to real `gcc-binary`. It's almost like `CFLAGS` but is expected to be transaparent to most build systems. > > The command-line limitations are in various places: > > - `gcc` limitation of lifting all command line options into a single > > environment variable: https://gcc.gnu.org/PR111527 > > - `linux` limitation of constraining single environ variable to a value > > way below than full available environment space: > > https://lkml.org/lkml/2023/9/24/381 > > > > `linux` fix would buy us 50x more budged (A Lot) but it will not help > > much other operating systems like `Darwin` where absolute environment > > limit is a lot lower than `linux`. > > > > I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192 > > (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against > > `master` as a proof of concept). > > > > What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS > > needs for `gcc`? I would like to implement something sensible I could > > upstream. > > How about `CFLAGS=@macro_prefix_map.args` and writing that file in the > same codepath where you generate the flags today. It'll work with just > about every compiler and tools like `ccache` will understand that it is > an input that affects the build and properly take the file's contents > into account. That was my initial attempt. I'll duplicate my response from https://gcc.gnu.org/pipermail/gcc/2023-October/242639.html here as is: """ Yeah, in theory response files would extend the limit. In practice `gcc` always extends response files internally into a single `COLLECT_GCC_OPTIONS` option and hits the environment variable limit very early: https://gcc.gnu.org/PR111527 Example reproducer: $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n" 200 1 200 2; done > a.rsp $ touch a.c; gcc @a.rsp -c a.c gcc: fatal error: cannot execute 'cc1': execv: Argument list too long compilation terminated. And if you want to look at the gory details: $ strace -f -etrace=execve -s 1000000 -v -v -v gcc @a.rsp -c a.c ... [pid 78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase", "a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64", "-fmacro-prefix-map=...=...", "-fmacro-prefix-map=...=...", ...], [..., "COLLECT_GCC=gcc", "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...' '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) = -1 E2BIG (Argument list too long) Note how `gcc` not only expands response file into an argument list (that is not too bad) but also duplicates the whole list as a single `COLLECT_GCC_OPTIONS=...` environment variable with added quoting on top. Would be nice if `gcc` just passed response files around as is :) """ -- Sergei