Re: Scaling -fmacro-prefix-map= to thousands entries

[Sergei Trofimovich <slyich@gmail.com>]

On Thu, Oct 05, 2023 at 09:19:15AM +0200, Richard Biener wrote:
> On Wed, Oct 4, 2023 at 11:20 PM Sergei Trofimovich via Gcc
> <gcc@gcc.gnu.org> wrote:
> >
> > Hi gcc developers!
> >
> > Tl;DR:
> >
> > I would like to implement a scalable way to pass `-fmacro-prefix-map=`
> > for `NixOS` distribution to avoid leaking build-time paths generated by
> > `__FILE__` macros used by various libraries.
> >
> > I need some guidance what path to take to be acceptable for `gcc`
> > upstream.
> >
> > I have a few possible solutions and wonder what I should try to upstream
> > to GCC. The options I see:
> >
> > 1. Hardcode NixOS-specific way to mangle paths.
> >
> >    Pros: simplest to implement, can be easily configured away if needed
> >    Cons: inflexible, `clang` might or might not accept the same hack
> >
> > 2. Extend `-fmacro-prefix-map=` (or add a new `-fmacro-prefix-map-file=`)
> >    to allow passing a file
> >
> >    Pros: still not too hard to implement, generic enough to be used in
> >          other contexts.
> >    Cons: Will require client to construct the map file.
> >
> > 3. Have more flexible `-fmacro-prefix-map-regex=` option that allows
> >    patterns. Something like:
> >
> >       -fmacro-prefix-map-regex=/nix/store/[a-z0-9]{32}-=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-
> >
> >   Pros: at least for NixOS one option will be enough to cover all
> >         packages as they all share above template.
> >   Cons: pulls some form of regex with it's can of worms including escape
> >         delimiters, might not be flexible enough for other use cases.
> >
> > 4. Something else?
> >
> > Which one(s) should I take to implement?
> >
> > More words:
> >
> > `NixOS` (and `nixpkgs` repository) install every software package into
> > an individual directory with unique prefix. Some examples:
> >
> >     /nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev
> >     /nix/store/rb3q4kcyfg77cmkiwywx2aqdd3x5ch93-libmpc-1.3.1
> >     /nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2
> >     ...
> >
> > It's a fundamental design decision to allow parallel package installs.
> >
> > From dependency tracking standpoint it's highly undesirable to have
> > these absolute paths to be hardcoded into final executable binaries if
> > they are not used at runtime.
> >
> > Example redundant path we would like not to have in final binaries:
> >
> >     $ strings result/bin/nix | grep phjcmy025rd1ankw5y1b21xsdii83cyk
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/json.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/output/serializer.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/conversions/to_chars.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/lexer.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iter_impl.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/json_sax.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iteration_proxy.hpp
> >     /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/parser.hpp
> >
> > Those paths are inserted via glibc's assert() uses of `__FILE__`
> > directive and thus hardcode header file paths from various packages
> > (like lttng-ust or nlohmann/json) into compiled binaries. Sometimes
> > `__FILE__` usage is mire creating than assert().
> >
> > I would like to get rid of references to header files. I think
> > `-fmacro-prefix-map=` are ideal for this particular use case.
> >
> > The prototype that creates equivalent of the following commands does
> > work for smaller packages:
> >
> >     -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.37-8-dev
> >     -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gmp-with-cxx-6.3.0-dev
> >     -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nlohmann_json-3.11.2
> >     ...
> >
> > The above works for small amount of options (like, 100). But around 1000
> > options we start hitting linux limits on the single environment variable
> > or real-world packages like `qemu` with a ton of input depends.
> >
> > The command-line limitations are in various places:
> > - `gcc` limitation of lifting all command line options into a single
> >   environment variable: https://gcc.gnu.org/PR111527
> > - `linux` limitation of constraining single environ variable to a value
> >   way below than full available environment space:
> >   https://lkml.org/lkml/2023/9/24/381
> >
> > `linux` fix would buy us 50x more budged (A Lot) but it will not help
> > much other operating systems like `Darwin` where absolute environment
> > limit is a lot lower than `linux`.
> >
> > I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192
> > (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against
> > `master` as a proof of concept).
> >
> > What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS
> > needs for `gcc`? I would like to implement something sensible I could
> > upstream.
> >
> > What do you think?
> 
> Go for (2) which I think is the only way to truly solve the command-line
> limitation issue (with less regular paths even regex wouldn't cut it).

Sounds good. Do you have any preference over specific syntax? My
suggestions would be:

1. `-fmacro-prefix-map=file-name`: if `file-name` there is not in `key=val`
   format then treat it as file
2. `-fmacro-prefix-map=@file-name`: use @ as a signal to use file
3. `fmacro-prefix-map-file=file-name`: use a new option

> Btw, I thought we have response files to deal with command-line limits,
> why doesn't that work here?  I see the driver expands response files
> but IIRC it also builds those when the command-line gets too large
> and uses it for the environment and the cc1 invocation?  If it doesn't
> do the latter why not fix it that way?

Yeah, in theory response files would extend the limit. In practice `gcc`
always extends response files internally into a single
`COLLECT_GCC_OPTIONS` option and hits the environment variable limit
very early:

    https://gcc.gnu.org/PR111527

Example reproducer:

    $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n" 200 1 200 2; done > a.rsp
    $ touch a.c; gcc @a.rsp -c a.c
    gcc: fatal error: cannot execute 'cc1': execv: Argument list too long
    compilation terminated.

And if you want to look at the gory details:

    $ strace -f -etrace=execve -s 1000000 -v -v -v  gcc @a.rsp -c a.c
    ...
    [pid    78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase", "a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64",
    "-fmacro-prefix-map=...=...",
    "-fmacro-prefix-map=...=...",
    ...],
    [...,
     "COLLECT_GCC=gcc",
     "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...' '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) = -1 E2BIG (Argument list too long)

Note how `gcc` not only expands response file into an argument list
(that is not too bad) but also duplicates the whole list as a single
`COLLECT_GCC_OPTIONS=...` environment variable with added quoting on
top.

Would be nice if `gcc` just passed response files around as is :)

-- 

  Sergei