Re: [PATCH] pch: Add support for PCH for relocatable executables

[Iain Sandoe <iain@sandoe.co.uk>]

[-- Attachment #1: Type: text/plain, Size: 2307 bytes --]

Hi Folks,

IMO both this series
 - which restores the ability to work with PIE exes but requires a known address for the PCH 
and the series I posted
 - which allows a configuration to opt out of PCH anyway

could be useful - for Darwin I prefer this series.

of course, it would be very nice to have a relocatable impl (or the tree streamer) .. I fear
that relying on finding a fixed hole in the VM addresses is probably fragile w.r.t OS updates.

> On 10 Nov 2021, at 20:24, Iain Sandoe <iain@sandoe.co.uk> wrote:

>> On 10 Nov 2021, at 08:14, Iain Sandoe <iain@sandoe.co.uk> wrote:
> 
>>> On 9 Nov 2021, at 12:18, Jakub Jelinek via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>>> 
>>> On Tue, Nov 09, 2021 at 11:40:08AM +0000, Iain Sandoe wrote:
>>>> There were two issues, of which one remains and probably affects all targets.
> 
>>>> 2. This problem remains.
> 
> This problem is also present on master without making any changes to the PCH
> implementation - if one fixes up the read-in to simulate a corrupted file, cc1 hangs
> 
> (which means it’s no barrier to the revised PCH implementation)


>> That seems reasonable for the case that we call fatal_error from ggc-common, but
>> I don’t think it will work if fancy_abort is called (for e.g. a segv) - we might need to 
>> make a local fancy_abort() as well for that specific file, perhaps.
>> 
>> Or in some way defer overwriting the data until we’ve succeeded in reading/relocating
>> the whole file (not sure what the largest PCH is we might encounter).

> 
> (answering my own question) around 150Mb for largest libstdc++ and similar for an 
> Objective-C include of Foundation + AppKit etc.
> 
> The underlying reason here is that diagnostics have become much more sophisticated,
> and they do all sorts of context checking and include the libcpp stuff directly which is a lot
> of GTY(()) stuff.
> 
> I cannot immediately see any small set of state that we can save / restore around the
> PCH read in,

I was wrong about that… patch posted that fixes most of this issue.


===

To add to Jakub's two patches that do the heavy lifting - two configure changes (I have also
darwin-local changes which are under test at the moment with the intention to apply them
anyway).




[-- Attachment #2: 0001-configure-gcc-Add-enable-pie-tools.patch --]
[-- Type: application/octet-stream, Size: 3994 bytes --]

From 697f235d8efbc53e2c5e63859cded0205f337969 Mon Sep 17 00:00:00 2001
From: Iain Sandoe <iain@sandoe.co.uk>
Date: Fri, 12 Nov 2021 17:01:50 +0000
Subject: [PATCH 1/2] configure, gcc: Add --enable-pie-tools.

This adds a configure option to allow a configurer to choose
to build the compilers and other supporting tools as PIE.

This is distinct from whether the compiler defaults to
producing PIE code or not (i.e. one could have a PIE compiler
producing non-PIE code by default, or vice-versa).

Note that this implementation expects that the option will
have been validated at the top-level before being passed to
the gcc/ directory configure.

Signed-off-by: Iain Sandoe <iain@sandoe.co.uk>

config/ChangeLog:

	* mh-darwin: Note that shared (PIC) code is also
	needed to support PIE.

gcc/ChangeLog:

	* Makefile.in: When pie-tools are enabled, do not add
	-no-PIE options to the compile and link lines.
	* config.in: Regenerate.
	* configure: Regenerate.
	* configure.ac: Add --enable-pie-tools confingure option.
---
 config/mh-darwin |  3 ++-
 gcc/Makefile.in  |  3 +++
 gcc/config.in    |  7 +++++++
 gcc/configure    | 30 ++++++++++++++++++++++++------
 gcc/configure.ac | 13 +++++++++++++
 5 files changed, 49 insertions(+), 7 deletions(-)

diff --git a/config/mh-darwin b/config/mh-darwin
index b72835ae953..bb4112773c9 100644
--- a/config/mh-darwin
+++ b/config/mh-darwin
@@ -11,7 +11,8 @@
 # non-bootstrapped compiler), later stages will be built by GCC which supports
 # the required flags.
 
-# We cannot use mdynamic-no-pic when building shared host resources.
+# We cannot use mdynamic-no-pic when building shared host resources, or for PIE
+# tool executables, which also enables host-shared.
 
 ifeq (${host_shared},no)
 BOOTSTRAP_TOOL_CAN_USE_MDYNAMIC_NO_PIC := $(shell \
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 571e9c28e29..878fada6862 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -270,11 +270,13 @@ COMPILER += $(CET_HOST_FLAGS)
 NO_PIE_CFLAGS = @NO_PIE_CFLAGS@
 NO_PIE_FLAG = @NO_PIE_FLAG@
 
+ifneq (@enable_pie_tools@,yes)
 # We don't want to compile the compilers with -fPIE, it make PCH fail.
 COMPILER += $(NO_PIE_CFLAGS)
 
 # Link with -no-pie since we compile the compiler with -fno-PIE.
 LINKER += $(NO_PIE_FLAG)
+endif
 
 # Like LINKER, but use a mutex for serializing front end links.
 ifeq (@DO_LINK_MUTEX@,true)
@@ -407,6 +409,7 @@ ifeq ($(enable_plugin),yes)
 endif
 
 enable_host_shared = @enable_host_shared@
+enable_default_pie = @enable_default_pie@
 
 enable_as_accelerator = @enable_as_accelerator@
 
diff --git a/gcc/config.in b/gcc/config.in
index b5bec3971dc..8bbe6492594 100644
--- a/gcc/config.in
+++ b/gcc/config.in
@@ -224,6 +224,13 @@
 #endif
 
 
+/* Define if you build Position Independent Executables for the compilers and
+   other tools. */
+#ifndef USED_FOR_TARGET
+#undef ENABLE_PIE_TOOLS
+#endif
+
+
 /* Define to enable plugin support. */
 #ifndef USED_FOR_TARGET
 #undef ENABLE_PLUGIN

diff --git a/gcc/configure.ac b/gcc/configure.ac
index 065080a4b39..4940cdcc16f 100644
--- a/gcc/configure.ac
+++ b/gcc/configure.ac
@@ -7542,6 +7542,19 @@ if test x$enable_default_pie = xyes ; then
 fi
 AC_SUBST([enable_default_pie])
 
+# Check whether --enable-pie-tools was given; this is passed automatically
+# from the top level where it has already been validated.
+AC_ARG_ENABLE(pie-tools,
+[AS_HELP_STRING([--enable-pie-tools],
+  [build Position Independent Executables for the compilers and other tools])],
+[enable_pie_tools=$enableval],
+[enable_pie_tools=no])
+if test x$enable_pie_tools = xyes ; then
+  AC_DEFINE(ENABLE_PIE_TOOLS, 1,
+      [Define if you build Position Independent Executables for the compilers and other tools.])
+fi
+AC_SUBST([enable_pie_tools])
+
 # Check if -fno-PIE works.
 AC_CACHE_CHECK([for -fno-PIE option],
   [gcc_cv_c_no_fpie],
-- 
2.24.3 (Apple Git-128)


[-- Attachment #3: 0002-configure-Add-top-level-configure-support-for-enable.patch --]
[-- Type: application/octet-stream, Size: 4127 bytes --]

From 1c65df304d264f1472384d1cf76c3de490b5578f Mon Sep 17 00:00:00 2001
From: Iain Sandoe <iain@sandoe.co.uk>
Date: Fri, 12 Nov 2021 17:06:25 +0000
Subject: [PATCH 2/2] configure: Add top-level configure support for
 --enable-pie-tools.

This recognises and validates the --enable-pie-tools configure
option.  It is done here because the choice can affect the code-
gen decisions for dependent libraries (e.g. in the case of Dariwn
we need to switch PIC support on and, therefore, for 32bit tool-
chains disable mdynamic-no-pic).

The validated option is passed to sub-configures that use it
(currently just gcc/).

Signed-off-by: Iain Sandoe <iain@sandoe.co.uk>

ChangeLog:

	* Makefile.def: Pass enable-pie-tools configure option
	to gcc/ configure.
	* Makefile.in: Regenerated.
	* Makefile.tpl: Export enable_pie_tools value.
	* configure: Regenerate.
	* configure.ac: Add --enable-pie-tools option and
	validate it for Darwin.
---
 Makefile.def |  3 ++-
 Makefile.in  | 32 ++++++++++++++++++++++----------
 Makefile.tpl |  3 +++
 configure    | 42 ++++++++++++++++++++++++++++++++++++++++++
 configure.ac | 34 ++++++++++++++++++++++++++++++++++
 5 files changed, 103 insertions(+), 11 deletions(-)

diff --git a/Makefile.def b/Makefile.def
index a504192e6d7..b9f86938ab3 100644
--- a/Makefile.def
+++ b/Makefile.def
@@ -47,7 +47,8 @@ host_modules= { module= fixincludes; bootstrap=true;
 host_modules= { module= flex; no_check_cross= true; };
 host_modules= { module= gas; bootstrap=true; };
 host_modules= { module= gcc; bootstrap=true; 
-		extra_make_flags="$(EXTRA_GCC_FLAGS)"; };
+		extra_make_flags="$(EXTRA_GCC_FLAGS)";
+		extra_configure_flags='--enable-pie-tools=@enable_pie_tools@'; };
 host_modules= { module= gmp; lib_path=.libs; bootstrap=true;
 		// Work around in-tree gmp configure bug with missing flex.
 		extra_configure_flags='--disable-shared LEX="touch lex.yy.c"';

diff --git a/Makefile.tpl b/Makefile.tpl
index 213052f8226..4c402759d36 100644
--- a/Makefile.tpl
+++ b/Makefile.tpl
@@ -115,6 +115,9 @@ GCC_SHLIB_SUBDIR = @GCC_SHLIB_SUBDIR@
 # If the build should make suitable code for shared host resources.
 host_shared = @host_shared@
 
+# If we should build compilers and supporting tools as PIE.
+enable_pie_tools = @enable_pie_tools@
+
 # Build programs are put under this directory.
 BUILD_SUBDIR = @build_subdir@
 # This is set by the configure script to the arguments to use when configuring

diff --git a/configure.ac b/configure.ac
index 550e6993b59..6859d8f950d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1860,7 +1860,41 @@ AC_ARG_ENABLE(host-shared,
 [AS_HELP_STRING([--enable-host-shared],
 		[build host code as shared libraries])],
 [host_shared=$enableval], [host_shared=no])
+
+# Check whether --enable-pie-tools was given.
+# Checked early because it can affect host make fragments.
+AC_ARG_ENABLE(pie-tools,
+[AS_HELP_STRING([--enable-pie-tools],
+  [build Position Independent Executables for the compilers and other tools])],
+[enable_pie_tools=$enableval
+ case $target in
+   aarch64-*-darwin1[[1-9]]*)
+     if test x$enable_pie_tools != xyes ; then
+       echo configure.ac: warning: aarch64-darwin must use PIE, pie-tools setting ignored. 1>&2
+       enable_pie_tools=yes
+       host_shared=yes
+     fi ;;
+    *) ;;
+ esac],
+[case $target in
+  # PIE is the default for macOS 10.7+ so reflect that in the configure.
+  # However, we build 32b toolchains mdynamic-no-pic by default which is
+  # not compatible with PIE.
+  x86_64-*-darwin1[[1-9]]* | *-*-darwin2*) enable_pie_tools=yes ;;
+  *) enable_pie_tools=no ;;
+ esac])
+
+case $target in
+  *-*-darwin*)
+    if test x$enable_pie_tools = xyes && test x$host_shared != xyes ; then
+      echo configure.ac: warning: for Darwin PIE requires PIC code, switching host-shared on 1>&2
+      host_shared=yes
+    fi ;;
+  *) ;;
+esac
+
 AC_SUBST(host_shared)
+AC_SUBST([enable_pie_tools])
 
 # By default, C and C++ are the only stage 1 languages.
 stage1_languages=,c,
-- 
2.24.3 (Apple Git-128)