Fwd: [PATCH 0/4] Darwin: Replace environment runpath with embedded [PR88590].

[Iain Sandoe <iains.gcc@gmail.com>]

It seems git send-mail drops CC: headers when there’s a —cc= on the command line,
which is a bit sad when one wants to CC different folks for different patches in a series.

> Begin forwarded message:
> 
> From: Iain Sandoe <iains.gcc@gmail.com>
> Subject: [PATCH 0/4] Darwin: Replace environment runpath with embedded [PR88590].
> Date: 17 November 2021 at 21:03:15 GMT
> To: gcc-patches@gcc.gnu.org
> Cc: joseph@codesourcery.com
> Reply-To: iain@sandoe.co.uk
> 
> This is a fairly long explanation of the problems being addressed by
> the patch set.  Most of the changes are Darwin-specific - a change to
> the libtool component allowing for this @rpath and some minor additions
> to makefiles where libtool is not in use.  At present, this seems pretty
> specific to the GCC build; since we depend on accessing newly-built
> components during the bootstrap.
> 
> There are additional details relevant to each patch in its own commit
> message.
> 
> =====
> 
> Darwin builds shared libraries with information on the runpath as part
> of the library name.  For example, /installation/path/for/libfoo.dylib.
> 
> That is regarded as two components; the 'runpath' /installation/path/for/
> and the library name libfoo.dylib.
> 
> This means that (at runtime) two libraries with the same name can be
> disambiguated by their runpaths, and potentially used by the same exe.
> 
> ===== Problem #1
> 
> That is fine and works well; until we disturb the assumptions by
> overriding the library runpath with an environment one: DYLD_LIBRARY_PATH.
> 
> Now the library runpath(s) can be discarded and the libraries are first
> searched on the basis of that provided by the environment; two libraries
> with the same name are no longer distinct (if a library with that name is
> found in the environment path).
> 
> This causes problems in configuring, building and testing GCC because we
> set the runpath environment at a very high level so that it applies to
> stage1+ target configures and stage2+ host configures.  This is needed so
> that executables built during those configures get the newly-built libgcc_s
> when the target defaults to using a shared libgcc.
> 
> However, it also means that every tool that is used during the configure
> has its libgcc_s (or any of the newly-built bootstrapped libs) overriden
> to use the new one(s) - which might be buggy.
> 
> In the testsuite it is more serious - since more target libs come into
> play - especially libstdc++.  Several system tools on Darwin use(d) libc++
> and that has caused wrong or crashed test output.  In principle,
> LD_LIBRARY_PATH on Linux has the same issue - although perhaps there is
> less tendency to default to use of shared dependent libs.
> 
> Ideally, one would have several environment paths, and some way to use
> the appropriate one at the appropriate time.  I experimented with this
> as a solution to both this and the following problem, but it proved
> unrealistic - since the process would have to be applied to all relevant
> OSS projects using auto-tools to be safe - and mostly the uninstalled
> use of libraries is a GCC build-time issue.
> 
> ===== Problem #2
> 
> A change in security policy for Darwin means that DYLD_LIBRARY_PATH is
> now removed from the environment for all system tools (e.g. /usr/sh, env
> etc).  This means that for all realistic build steps that use any system
> utility (like sh) will no longer see the the environment runpath and the
> only ones available will be those in the libraries.
> 
> This breaks GCC's configuration since the steps mentioned above are now
> not seeing the newly-built shared libraries, but actually much olders ones
> installed on the system.  It means that for all Darwin15+ we misconfigure
> libstdc++.
> 
> /bin/sh is hardwired into autoconf, one cannot use CONFIG_SHELL to work
> around this - because /bin/sh is invoked first, and then passes control to
> CONFIG_SHELL.
> 
> A second problem is that we cannot bump the SO name for libgcc_s (which
> I need to do to solve an EH problem) - since the new SO name is not
> available on the system, and therefore none of the stage1+ target configures
> will succeed.  This is because the eventual install path is correctly
> encoded into the built library, but it is not present at the install
> position (and, in general, cannot be installed - since that might not even
> be a suitable path on the build system).
> 
> This has also meant that we could not do in-tree testing without first
> installing the target libraries (which is mostly inconvenient rather than
> a show-stopper, but still).
> 
> ===== Tested solution.
> 
> Darwin has the ability to make the runpaths install-position independent.
> 
> One sets the library runpath to @rpath/ which essentially means "use the
> runpath available at the time we want to load this".
> 
> One can then add (potentially multiple) runpaths to the executable, the
> built library can be put anywhere convenient - providing we can put that
> path into the exe.
> 
> For GCC's build, test and install process this means that we need at each
> stage to build exes with the runpaths that are relevant to the positions
> of the dependent libraries.
> 
> To do this, we add an rpath for each of the startfile paths.  While we are
> building/testing GCC these correspond to (for example gcc/  or 
> <target>/libstdc++/src/.libs etc) and then, after the compiler is installed
> at its intended install path - these become /compiler/installation/path/lib
> etc.
> 
> I have tested this widely on i686, powerpc, x86_64 and aarch64 Darwin over
> more than a year.
> 
> So patch 1 : provides a spec that expands to -rpath xxx for each xxx in the
> startfiles (which means that this works for anything added as -B and thus
> the testsuite also works without installing the target libs again).
> 
> The remaining patches implement the Darwin changes to build the target
> libraries with @rpath runpaths - this is the largest part of the patch
> in bytes - because the libtool change expands.
> 
> I would like to get this in early in stage 3 so that it has time to be
> kicked around before we branch.
> 
> OK for master?
> (ideally, we'd also backport to open branches because the breakage is
> quite significant)
> 
> thanks
> Iain
> 
> 
> Iain Sandoe (4):
>  Driver : Provide a spec to insert rpaths for compiler lib dirs.
>  Darwin : Handle rpaths given on the command line.
>  Darwin : Allow for configuring Darwin to use embedded runpath.
>  Darwin, Ada : Add loader path as a default rpath.
> 
> configure                         |   5 +
> configure.ac                      |   5 +
> fixincludes/config.h.in           | 204 ---------------------------
> fixincludes/configure             |   2 +-
> gcc/aclocal.m4                    |  50 +++++++
> gcc/ada/gcc-interface/Makefile.in |   2 +
> gcc/config/darwin-driver.c        |  18 +++
> gcc/config/darwin.h               |  11 +-
> gcc/config/darwin.opt             |   4 +
> gcc/configure                     | 157 +++++++++++++++++++--
> gcc/gcc.c                         |  18 +++
> libatomic/Makefile.am             |   6 +-
> libatomic/Makefile.in             |   5 +-
> libatomic/configure               |  82 ++++++++++-
> libatomic/testsuite/Makefile.in   |   1 +
> libbacktrace/configure            |  80 ++++++++++-
> libcc1/Makefile.am                |   9 ++
> libcc1/Makefile.in                |  12 +-
> libcc1/configure                  | 157 +++++++++++++++++++--
> libffi/Makefile.am                |   7 +-
> libffi/Makefile.in                |   6 +-
> libffi/configure                  | 157 +++++++++++++++++++--
> libgcc/config.host                |  22 ++-
> libgcc/config/t-darwin-rpath      |   5 +
> libgcc/config/t-slibgcc-darwin    |  13 +-
> libgfortran/Makefile.am           |   3 +
> libgfortran/Makefile.in           |  30 ++--
> libgfortran/configure             | 155 ++++++++++++++++++--
> libgfortran/configure.ac          |   4 +-
> libgomp/Makefile.am               |   6 +-
> libgomp/Makefile.in               |   3 +-
> libgomp/configure                 | 151 ++++++++++++++++++--
> libitm/Makefile.am                |   5 +-
> libitm/Makefile.in                |   3 +-
> libitm/configure                  | 171 ++++++++++++++++++++---
> libobjc/configure                 |  96 +++++++++++--
> libobjc/configure.ac              |  14 +-
> liboffloadmic/configure           | 199 +++++++++++++++++++++-----
> liboffloadmic/plugin/Makefile.in  |   2 +-
> liboffloadmic/plugin/aclocal.m4   |   2 +-
> liboffloadmic/plugin/configure    | 199 +++++++++++++++++++++-----
> libphobos/configure               | 151 ++++++++++++++++++--
> libphobos/libdruntime/Makefile.am |   5 +-
> libphobos/libdruntime/Makefile.in |   3 +-
> libphobos/src/Makefile.am         |   5 +-
> libphobos/src/Makefile.in         |   3 +-
> libquadmath/Makefile.am           |   6 +-
> libquadmath/Makefile.in           |   4 +-
> libquadmath/configure             | 225 +++++++++++++++++++++++++++++-
> libquadmath/configure.ac          |   2 +
> libsanitizer/asan/Makefile.am     |   6 +-
> libsanitizer/asan/Makefile.in     |   5 +-
> libsanitizer/configure            | 157 +++++++++++++++++++--
> libsanitizer/hwasan/Makefile.am   |   6 +-
> libsanitizer/hwasan/Makefile.in   |   5 +-
> libsanitizer/lsan/Makefile.am     |   7 +-
> libsanitizer/lsan/Makefile.in     |   6 +-
> libsanitizer/tsan/Makefile.am     |   6 +-
> libsanitizer/tsan/Makefile.in     |   5 +-
> libsanitizer/ubsan/Makefile.am    |   6 +-
> libsanitizer/ubsan/Makefile.in    |   5 +-
> libssp/Makefile.am                |   5 +-
> libssp/Makefile.in                |   3 +-
> libssp/configure                  |  82 ++++++++++-
> libstdc++-v3/configure            | 169 +++++++++++++++++++---
> libstdc++-v3/src/Makefile.am      |   6 +-
> libstdc++-v3/src/Makefile.in      |   3 +-
> libtool.m4                        |  64 ++++++++-
> libvtv/configure                  | 171 ++++++++++++++++++++---
> lto-plugin/Makefile.in            |   1 -
> lto-plugin/configure              |  80 ++++++++++-
> zlib/configure                    |  82 ++++++++++-
> 72 files changed, 2832 insertions(+), 533 deletions(-)
> create mode 100644 libgcc/config/t-darwin-rpath
> 
> -- 
> 2.24.3 (Apple Git-128)
>