Re: Scaling -fmacro-prefix-map= to thousands entries

[Richard Biener <richard.guenther@gmail.com>]

On Wed, Oct 4, 2023 at 11:20 PM Sergei Trofimovich via Gcc
<gcc@gcc.gnu.org> wrote:
>
> Hi gcc developers!
>
> Tl;DR:
>
> I would like to implement a scalable way to pass `-fmacro-prefix-map=`
> for `NixOS` distribution to avoid leaking build-time paths generated by
> `__FILE__` macros used by various libraries.
>
> I need some guidance what path to take to be acceptable for `gcc`
> upstream.
>
> I have a few possible solutions and wonder what I should try to upstream
> to GCC. The options I see:
>
> 1. Hardcode NixOS-specific way to mangle paths.
>
>    Pros: simplest to implement, can be easily configured away if needed
>    Cons: inflexible, `clang` might or might not accept the same hack
>
> 2. Extend `-fmacro-prefix-map=` (or add a new `-fmacro-prefix-map-file=`)
>    to allow passing a file
>
>    Pros: still not too hard to implement, generic enough to be used in
>          other contexts.
>    Cons: Will require client to construct the map file.
>
> 3. Have more flexible `-fmacro-prefix-map-regex=` option that allows
>    patterns. Something like:
>
>       -fmacro-prefix-map-regex=/nix/store/[a-z0-9]{32}-=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-
>
>   Pros: at least for NixOS one option will be enough to cover all
>         packages as they all share above template.
>   Cons: pulls some form of regex with it's can of worms including escape
>         delimiters, might not be flexible enough for other use cases.
>
> 4. Something else?
>
> Which one(s) should I take to implement?
>
> More words:
>
> `NixOS` (and `nixpkgs` repository) install every software package into
> an individual directory with unique prefix. Some examples:
>
>     /nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev
>     /nix/store/rb3q4kcyfg77cmkiwywx2aqdd3x5ch93-libmpc-1.3.1
>     /nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2
>     ...
>
> It's a fundamental design decision to allow parallel package installs.
>
> From dependency tracking standpoint it's highly undesirable to have
> these absolute paths to be hardcoded into final executable binaries if
> they are not used at runtime.
>
> Example redundant path we would like not to have in final binaries:
>
>     $ strings result/bin/nix | grep phjcmy025rd1ankw5y1b21xsdii83cyk
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/json.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/output/serializer.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/conversions/to_chars.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/lexer.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iter_impl.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/json_sax.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iteration_proxy.hpp
>     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/parser.hpp
>
> Those paths are inserted via glibc's assert() uses of `__FILE__`
> directive and thus hardcode header file paths from various packages
> (like lttng-ust or nlohmann/json) into compiled binaries. Sometimes
> `__FILE__` usage is mire creating than assert().
>
> I would like to get rid of references to header files. I think
> `-fmacro-prefix-map=` are ideal for this particular use case.
>
> The prototype that creates equivalent of the following commands does
> work for smaller packages:
>
>     -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.37-8-dev
>     -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev
>     -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nlohmann_json-3.11.2
>     ...
>
> The above works for small amount of options (like, 100). But around 1000
> options we start hitting linux limits on the single environment variable
> or real-world packages like `qemu` with a ton of input depends.
>
> The command-line limitations are in various places:
> - `gcc` limitation of lifting all command line options into a single
>   environment variable: https://gcc.gnu.org/PR111527
> - `linux` limitation of constraining single environ variable to a value
>   way below than full available environment space:
>   https://lkml.org/lkml/2023/9/24/381
>
> `linux` fix would buy us 50x more budged (A Lot) but it will not help
> much other operating systems like `Darwin` where absolute environment
> limit is a lot lower than `linux`.
>
> I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192
> (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against
> `master` as a proof of concept).
>
> What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS
> needs for `gcc`? I would like to implement something sensible I could
> upstream.
>
> What do you think?

Go for (2) which I think is the only way to truly solve the command-line
limitation issue (with less regular paths even regex wouldn't cut it).

Btw, I thought we have response files to deal with command-line limits,
why doesn't that work here?  I see the driver expands response files
but IIRC it also builds those when the command-line gets too large
and uses it for the environment and the cc1 invocation?  If it doesn't
do the latter why not fix it that way?

Richard.

> Thanks!
>
> --
>
>   Sergei