Add LTO binutils wrappers

Add LTO binutils wrappers

[Andi Kleen]

This came out of the discussion last time I submitted -print-lto-plugin.
For a slim LTO build tools like ar or ranlib need to be LTO 
aware. This patch first readds the -print-lto-plugin option
and then adds three wrapper scripts for ar, ranlib that use.
I also added a ld wrapper that calls the gcc driver.

Passes bootstrap, full test on x86-64. Ok to commit?

-andi

[PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

Earlier review resulted in a request for having gcc-... wrappers
to handle the LTO symbol tables for ranlib/ar. This is needed for slim
lto files, because these tools cannot read the symbol table
otherwise. Essentially they just call binutils with
the LTO plugin specified.

Other compilers with LTO support tend to have similar tools.

This patch implements those wrappers. I also added a wrapper for ld
because I needed one for a large existing Makefile with a lot
of complicated ld calls. It turns an ld command line into
a gcc command line suitable for lto.

The wrappers are also needed for a LTO slim of gcc. Right now
they have to be manually specified for that.

The wrappers require uptodate binutils (the upcoming release)
with plugin support enabled.  There is currently no autoconf
support to detect that. The wrappers query the gcc driver
in an attempt to handle cross compilation and so naming everywhere.

gcc/lto/

2010-10-26  Andi Kleen  <ak@linux.intel.com>

	* Make-lang.in (AR_INSTALL_NAME, RANLIB_INSTALL_NAME, LD_INSTALL_NAME,
	LTO_WRAPPERS): Add.
	(lto.all.cross, lto.start.cross): Add dependency to LTO_WRAPPERS.
	(lto.install.common): Install wrappers.
	(lto.mostlyclean): Clean wrappers.
	(gcc-ar, gcc-ranlib, gcc-ld): Add.
	* config-lang.in (outputs): Add.
	* gcc-ar.in, gcc-ld.in, gcc-ranlib.in: Add.

gcc/

2010-10-26  Andi Kleen  <ak@linux.intel.com>

	* doc/invoke.texi (gcc-ar, gcc-ld, gcc-ranlib): Document.
---
 gcc/doc/invoke.texi    |    5 ++++
 gcc/lto/Make-lang.in   |   31 +++++++++++++++++++++++++++--
 gcc/lto/config-lang.in |    2 +
 gcc/lto/gcc-ar.in      |    8 +++++++
 gcc/lto/gcc-ld.in      |   49 ++++++++++++++++++++++++++++++++++++++++++++++++
 gcc/lto/gcc-ranlib.in  |    8 +++++++
 6 files changed, 100 insertions(+), 3 deletions(-)
 create mode 100644 gcc/lto/gcc-ar.in
 create mode 100644 gcc/lto/gcc-ld.in
 create mode 100644 gcc/lto/gcc-ranlib.in

diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index d6784ad..0810e77 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -7565,6 +7565,11 @@ be exported, it is possible to combine @option{-flto} and
 interprocedural optimizers to use more aggressive assumptions which
 may lead to improved optimization opportunities.
 
+In some cases you may need to use the @command{gcc-ar} and 
+@command{gcc-ranlib} commands to manage ar files of LTO objects. 
+@command{gcc-ld} can be also used to turn a @command{ld} command
+line into one suitable for LTO.
+
 Regarding portability: the current implementation of LTO makes no
 attempt at generating bytecode that can be ported between different
 types of hosts.  The bytecode files are versioned and there is a
diff --git a/gcc/lto/Make-lang.in b/gcc/lto/Make-lang.in
index 2dc6409..55fb83a 100644
--- a/gcc/lto/Make-lang.in
+++ b/gcc/lto/Make-lang.in
@@ -28,16 +28,31 @@ LTO_H = lto/lto.h $(HASHTAB_H)
 LINKER_PLUGIN_API_H = $(srcdir)/../include/plugin-api.h
 LTO_TREE_H = lto/lto-tree.h $(LINKER_PLUGIN_API_H)
 
+AR_INSTALL_NAME := $(shell echo gcc-ar|sed '$(program_transform_name)')
+RANLIB_INSTALL_NAME := $(shell echo gcc-ranlib|sed '$(program_transform_name)')
+LD_INSTALL_NAME := $(shell echo gcc-ld|sed '$(program_transform_name)')
+
+ifeq ($(enable_plugin),yes)
+LTO_WRAPPERS = lto/gcc-ar lto/gcc-ld lto/gcc-ranlib
+else
+LTO_WRAPPERS =
+endif
 
 # Rules
 
 # These hooks are used by the main GCC Makefile.  Consult that
 # Makefile for documentation.
-lto.all.cross: $(LTO_EXE)
-lto.start.encap: $(LTO_EXE)
+lto.all.cross: $(LTO_EXE) $(LTO_WRAPPERS)
+lto.start.encap: $(LTO_EXE) $(LTO_WRAPPERS)
 lto.rest.encap:
 lto.tags:
 lto.install-common:
+ifeq ($(enable_plugin),yes)
+	$(INSTALL_SCRIPT) lto/gcc-ar $(DESTDIR)$(bindir)/$(AR_INSTALL_NAME)
+	$(INSTALL_SCRIPT) lto/gcc-ld $(DESTDIR)$(bindir)/$(LD_INSTALL_NAME)
+	$(INSTALL_SCRIPT) lto/gcc-ranlib $(DESTDIR)$(bindir)/$(RANLIB_INSTALL_NAME)
+endif
+
 lto.install-man:
 lto.install-info:
 lto.dvi:
@@ -54,7 +69,7 @@ lto.srcinfo:
 lto.install-plugin:
 
 lto.mostlyclean:
-	rm -f $(LTO_OBJS) $(LTO_EXE)
+	rm -f $(LTO_OBJS) $(LTO_EXE) lto/gcc-ar lto/gcc-ranlib lto/gcc-ld
 
 lto.clean:
 lto.distclean:
@@ -75,6 +90,16 @@ $(LTO_EXE): $(LTO_OBJS) $(BACKEND) $(LIBDEPS)
 	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 		$(LTO_OBJS) $(BACKEND) $(BACKENDLIBS) $(LIBS) $(LTO_USE_LIBELF)
 
+# generate plugin calling toolchain wrappers
+lto/gcc-ar: lto/gcc-ar.in
+	CONFIG_FILES=lto/gcc-ar CONFIG_HEADERS= ./config.status
+
+lto/gcc-ranlib: lto/gcc-ranlib.in
+	CONFIG_FILES=lto/gcc-ranlib CONFIG_HEADERS= ./config.status
+
+lto/gcc-ld: lto/gcc-ld.in
+	CONFIG_FILES=lto/gcc-ld CONFIG_HEADERS= ./config.status
+
 # Dependencies
 lto/lto-lang.o: lto/lto-lang.c $(CONFIG_H) coretypes.h debug.h \
 	flags.h $(GGC_H) langhooks.h $(LANGHOOKS_DEF_H) $(SYSTEM_H) \
diff --git a/gcc/lto/config-lang.in b/gcc/lto/config-lang.in
index 72ed2dd..c125408 100644
--- a/gcc/lto/config-lang.in
+++ b/gcc/lto/config-lang.in
@@ -33,3 +33,5 @@ build_by_default=no
 
 # Add LTO to boot language if it is enabled.
 boot_language=$enable_lto
+
+outputs="lto/gcc-ar lto/gcc-ld lto/gcc-ranlib"
diff --git a/gcc/lto/gcc-ar.in b/gcc/lto/gcc-ar.in
new file mode 100644
index 0000000..6246446
--- /dev/null
+++ b/gcc/lto/gcc-ar.in
@@ -0,0 +1,8 @@
+#!/bin/sh
+# wrapper for ar with GCC LTO support
+prefix=@prefix@
+exec_prefix=@exec_prefix@
+GCC_BINDIR=${GCC_LIBEXEC:-@bindir@}
+AR=${AR:-`$GCC_BINDIR/gcc -print-prog-name=ar`}
+
+exec $AR --plugin `$GCC_BINDIR/gcc -print-lto-plugin` "$@"
diff --git a/gcc/lto/gcc-ld.in b/gcc/lto/gcc-ld.in
new file mode 100644
index 0000000..f204764
--- /dev/null
+++ b/gcc/lto/gcc-ld.in
@@ -0,0 +1,49 @@
+#!/bin/sh
+# run gcc with ld options
+# used as a wrapper to execute link time optimizations for
+# makefiles that want to call the linker directly
+# this runs ld or gold as needed
+
+ARGS=""
+R=""
+
+prefix=@prefix@
+exec_prefix=@exec_prefix@
+GCC=${GCCBINDIR:-@bindir@}/gcc
+
+while test "x$1" != "x" ; do
+	case "$1" in
+	-r) 	R=1
+		N="$1" 
+		;;
+	-[olvO]|-f*|-nostdlib) 
+		N="$1" 
+		;;
+	--end-group|--start-group)
+		N="-Wl,$1" 
+		;;
+	-[RTFGhIezcbmyYu]*|\
+--script|--defsym|-init|-Map|--oformat|-rpath|\
+-rpath-link|--sort-section|--section-start|-Tbss|-Tdata|-Ttext|\
+--version-script|--dynamic-list|--version-exports-symbol|--wrap)
+		A="$1" 
+		shift
+		N="-Wl,$A,$1" 
+		;;
+	-*)     N="-Wl,$1" 
+		;;
+	*)  	N="$1" 
+		;;
+	esac
+	ARGS="$ARGS $N"
+	shift
+done
+
+if test "x$R" != "x" ; then
+	# work around gold bug, -r and -fuse-linker-plugin
+	# do not interact well. It's not needed anyways.
+	# the gold bug has been fixed, but still keep this.
+	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`
+fi
+
+exec $GCC $ARGS
diff --git a/gcc/lto/gcc-ranlib.in b/gcc/lto/gcc-ranlib.in
new file mode 100644
index 0000000..4222ae9
--- /dev/null
+++ b/gcc/lto/gcc-ranlib.in
@@ -0,0 +1,8 @@
+#!/bin/sh
+# wrapper for ranlib with GCC LTO support
+prefix=@prefix@
+exec_prefix=@exec_prefix@
+GCC_BINDIR=${GCC_BINDIR:-@bindir@}
+AR=${AR:-`$GCC_BINDIR/gcc -print-prog-name=ar`}
+
+exec $AR -s --plugin `$GCC_BINDIR/gcc -print-lto-plugin` "$@"
-- 
1.7.1

[PATCH 1/2] Add -print-lto-plugin

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

binutils have LTO linker plugin support now, but it requires hardcoding
the path to gcc's libexec dir in the Makefile. Add an option to the
gcc driver instead to print the full file name to avoid this.

This is especially useful for ar or nm.

gcc/

2010-10-02  Andi Kleen <ak@linux.intel.com>

	* common.opt (-print-lto-plugin): Add.
	* doc/invoke.texi (-print-lto-plugin): Document.
	* gcc.c (display_help): Describe -print-lto-plugin.
	(main.c): Handle print_lto_plugin.
---
 gcc/common.opt      |    6 ++++++
 gcc/doc/invoke.texi |    9 ++++++++-
 gcc/gcc.c           |   13 +++++++++++++
 3 files changed, 27 insertions(+), 1 deletions(-)

diff --git a/gcc/common.opt b/gcc/common.opt
index a74d60d..2b6c73b 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -183,6 +183,9 @@ Driver JoinedOrMissing Alias(print-file-name=)
 -print-libgcc-file-name
 Driver Alias(print-libgcc-file-name)
 
+-print-lto-plugin
+Driver Alias(print-lto-plugin)
+
 -print-multi-directory
 Driver Alias(print-multi-directory)
 
@@ -1932,6 +1935,9 @@ Driver JoinedOrMissing Var(print_file_name)
 print-libgcc-file-name
 Driver
 
+print-lto-plugin
+Driver Var(print_lto_plugin)
+
 print-multi-directory
 Driver Var(print_multi_directory)
 
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 7ea042f..d6784ad 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -322,7 +322,7 @@ Objective-C and Objective-C++ Dialects}.
 -fdebug-prefix-map=@var{old}=@var{new} @gol
 -femit-struct-debug-baseonly -femit-struct-debug-reduced @gol
 -femit-struct-debug-detailed@r{[}=@var{spec-list}@r{]} @gol
--p  -pg  -print-file-name=@var{library}  -print-libgcc-file-name @gol
+-p  -pg  -print-file-name=@var{library}  -print-libgcc-file-name  -print-lto-plugin @gol
 -print-multi-directory  -print-multi-lib  -print-multi-os-directory @gol
 -print-prog-name=@var{program}  -print-search-dirs  -Q @gol
 -print-sysroot -print-sysroot-headers-suffix @gol
@@ -5725,6 +5725,13 @@ Same as @option{-print-file-name=libgcc.a}.
 This is useful when you use @option{-nostdlib} or @option{-nodefaultlibs}
 but you do want to link with @file{libgcc.a}.  You can do
 
+@item -print-lto-plugin
+@opindex print-lto-plugin
+Print the path name of the LTO linker plugin used by the compiler.
+
+This is useful to pass to ar, gold, ld or other binutils with linker 
+plugin support. 
+
 @smallexample
 gcc -nostdlib @var{files}@dots{} `gcc -print-libgcc-file-name`
 @end smallexample
diff --git a/gcc/gcc.c b/gcc/gcc.c
index 13635d5..e6fc6d5 100644
--- a/gcc/gcc.c
+++ b/gcc/gcc.c
@@ -2928,6 +2928,7 @@ display_help (void)
   fputs (_("  -dumpmachine             Display the compiler's target processor\n"), stdout);
   fputs (_("  -print-search-dirs       Display the directories in the compiler's search path\n"), stdout);
   fputs (_("  -print-libgcc-file-name  Display the name of the compiler's companion library\n"), stdout);
+  fputs (_("  -print-lto-plugin        Display the name of the compiler's LTO plugin\n"), stdout);
   fputs (_("  -print-file-name=<lib>   Display the full path to library <lib>\n"), stdout);
   fputs (_("  -print-prog-name=<prog>  Display the full path to compiler component <prog>\n"), stdout);
   fputs (_("  -print-multi-directory   Display the root directory for versions of libgcc\n"), stdout);
@@ -3222,6 +3223,7 @@ driver_handle_option (struct gcc_options *opts,
     case OPT_print_sysroot:
     case OPT_print_multi_os_directory:
     case OPT_print_sysroot_headers_suffix:
+    case OPT_print_lto_plugin:
     case OPT_time:
     case OPT_wrapper:
       /* These options set the variables specified in common.opt
@@ -6536,6 +6538,17 @@ main (int argc, char **argv)
 	fatal_error ("not configured with sysroot headers suffix");
     }
 
+  if (print_lto_plugin)
+    {
+      char *plugin;
+
+      plugin = find_a_file (&exec_prefixes, "liblto_plugin.so", R_OK, false);
+      if (!plugin)
+        fatal_error ("-print-lto-plugin specified, but no plugin found");    
+      printf ("%s\n", plugin);
+      return (0);
+    }
+
   if (print_help_list)
     {
       display_help ();
-- 
1.7.1

Re: Add LTO binutils wrappers

[Diego Novillo]

On Fri, Oct 29, 2010 at 07:46, Andi Kleen <andi@firstfloor.org> wrote:
> This came out of the discussion last time I submitted -print-lto-plugin.
> For a slim LTO build tools like ar or ranlib need to be LTO
> aware. This patch first readds the -print-lto-plugin option
> and then adds three wrapper scripts for ar, ranlib that use.

Patch is missing.


Diego.

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Joseph S. Myers]

On Fri, 29 Oct 2010, Andi Kleen wrote:

> 	* gcc-ar.in, gcc-ld.in, gcc-ranlib.in: Add.

Installed scripts are a bad idea because GCC is portable to hosts that 
cannot run shell scripts (the *build system* can run shell scripts, but 
the *host* may be e.g. MinGW and not able to run them).  The portable way 
to wrap subprocesses is to use C programs using the pexecute interface, as 
in collect2 or lto-wrapper.

> +ARGS=""

By building up a string like this you'll have lots of problems when 
arguments contain spaces.  A C wrapper naturally avoids that.

> +prefix=@prefix@
> +exec_prefix=@exec_prefix@

No.  GCC installations are meant to be relocatable; if the wrapper is 
executed from somewhere other than the $bindir in which it is installed, 
it must find other programs relative to the directory to which it has been 
moved, rather than reusing configure-time absolute paths.  In a C wrapper 
you can use make_relative_prefix.

> +while test "x$1" != "x" ; do
> +	case "$1" in
> +	-r) 	R=1
> +		N="$1" 
> +		;;
> +	-[olvO]|-f*|-nostdlib) 
> +		N="$1" 
> +		;;

-o and -l take arguments, which you don't seem to be allowing for here.

> +	--end-group|--start-group)
> +		N="-Wl,$1" 
> +		;;
> +	-[RTFGhIezcbmyYu]*|\
> +--script|--defsym|-init|-Map|--oformat|-rpath|\
> +-rpath-link|--sort-section|--section-start|-Tbss|-Tdata|-Ttext|\
> +--version-script|--dynamic-list|--version-exports-symbol|--wrap)
> +		A="$1" 
> +		shift

I've been eliminating duplicate information in the compiler about which 
*GCC* options take arguments.  Now as I understand this code you actually 
want to know about which *GNU ld* (or gold?) options take arguments, 
rather than GCC options, so I suppose at present there's no good way to 
avoid duplication of information, though better integration in these areas 
between toolchain components would certainly be nice.

> +if test "x$R" != "x" ; then
> +	# work around gold bug, -r and -fuse-linker-plugin
> +	# do not interact well. It's not needed anyways.
> +	# the gold bug has been fixed, but still keep this.
> +	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`

Apart from the issues with spaces, echo will process some options on some 
systems and this sed command will happily modify option arguments that 
happen to contain that particular string.  As before, (POSIX) shell is not 
a good language in which to do this robustly (if you e.g. assume various 
bash features, it becomes easier).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Pedro Alves]

On Friday 29 October 2010 12:46:57, Andi Kleen wrote:
> --- /dev/null
> +++ b/gcc/lto/gcc-ranlib.in
> @@ -0,0 +1,8 @@
> +#!/bin/sh
> +# wrapper for ranlib with GCC LTO support
> +prefix=@prefix@
> +exec_prefix=@exec_prefix@
> +GCC_BINDIR=${GCC_BINDIR:-@bindir@}
> +AR=${AR:-`$GCC_BINDIR/gcc -print-prog-name=ar`}
> +
> +exec $AR -s --plugin `$GCC_BINDIR/gcc -print-lto-plugin` "$@"

Is this meant to be used by gcc users, rather then when building gcc?
At the risk of sounding obvious, I'd like to point out that wrapper
shell scripts don't work on Windows hosts.

-- 
Pedro Alves

Re: [PATCH 1/2] Add -print-lto-plugin

[Richard Guenther]

On Fri, Oct 29, 2010 at 1:46 PM, Andi Kleen <andi@firstfloor.org> wrote:
> From: Andi Kleen <ak@linux.intel.com>
>
> binutils have LTO linker plugin support now, but it requires hardcoding
> the path to gcc's libexec dir in the Makefile. Add an option to the
> gcc driver instead to print the full file name to avoid this.
>
> This is especially useful for ar or nm.

Is the reason for not using -print-file-name=liblto-plugin.so to not hardcode
that name?  (which you still did in the patch - not sure if it's that specific
name on all hosts).

Richard.

> gcc/
>
> 2010-10-02  Andi Kleen <ak@linux.intel.com>
>
>        * common.opt (-print-lto-plugin): Add.
>        * doc/invoke.texi (-print-lto-plugin): Document.
>        * gcc.c (display_help): Describe -print-lto-plugin.
>        (main.c): Handle print_lto_plugin.
> ---
>  gcc/common.opt      |    6 ++++++
>  gcc/doc/invoke.texi |    9 ++++++++-
>  gcc/gcc.c           |   13 +++++++++++++
>  3 files changed, 27 insertions(+), 1 deletions(-)
>
> diff --git a/gcc/common.opt b/gcc/common.opt
> index a74d60d..2b6c73b 100644
> --- a/gcc/common.opt
> +++ b/gcc/common.opt
> @@ -183,6 +183,9 @@ Driver JoinedOrMissing Alias(print-file-name=)
>  -print-libgcc-file-name
>  Driver Alias(print-libgcc-file-name)
>
> +-print-lto-plugin
> +Driver Alias(print-lto-plugin)
> +
>  -print-multi-directory
>  Driver Alias(print-multi-directory)
>
> @@ -1932,6 +1935,9 @@ Driver JoinedOrMissing Var(print_file_name)
>  print-libgcc-file-name
>  Driver
>
> +print-lto-plugin
> +Driver Var(print_lto_plugin)
> +
>  print-multi-directory
>  Driver Var(print_multi_directory)
>
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index 7ea042f..d6784ad 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -322,7 +322,7 @@ Objective-C and Objective-C++ Dialects}.
>  -fdebug-prefix-map=@var{old}=@var{new} @gol
>  -femit-struct-debug-baseonly -femit-struct-debug-reduced @gol
>  -femit-struct-debug-detailed@r{[}=@var{spec-list}@r{]} @gol
> --p  -pg  -print-file-name=@var{library}  -print-libgcc-file-name @gol
> +-p  -pg  -print-file-name=@var{library}  -print-libgcc-file-name  -print-lto-plugin @gol
>  -print-multi-directory  -print-multi-lib  -print-multi-os-directory @gol
>  -print-prog-name=@var{program}  -print-search-dirs  -Q @gol
>  -print-sysroot -print-sysroot-headers-suffix @gol
> @@ -5725,6 +5725,13 @@ Same as @option{-print-file-name=libgcc.a}.
>  This is useful when you use @option{-nostdlib} or @option{-nodefaultlibs}
>  but you do want to link with @file{libgcc.a}.  You can do
>
> +@item -print-lto-plugin
> +@opindex print-lto-plugin
> +Print the path name of the LTO linker plugin used by the compiler.
> +
> +This is useful to pass to ar, gold, ld or other binutils with linker
> +plugin support.
> +
>  @smallexample
>  gcc -nostdlib @var{files}@dots{} `gcc -print-libgcc-file-name`
>  @end smallexample
> diff --git a/gcc/gcc.c b/gcc/gcc.c
> index 13635d5..e6fc6d5 100644
> --- a/gcc/gcc.c
> +++ b/gcc/gcc.c
> @@ -2928,6 +2928,7 @@ display_help (void)
>   fputs (_("  -dumpmachine             Display the compiler's target processor\n"), stdout);
>   fputs (_("  -print-search-dirs       Display the directories in the compiler's search path\n"), stdout);
>   fputs (_("  -print-libgcc-file-name  Display the name of the compiler's companion library\n"), stdout);
> +  fputs (_("  -print-lto-plugin        Display the name of the compiler's LTO plugin\n"), stdout);
>   fputs (_("  -print-file-name=<lib>   Display the full path to library <lib>\n"), stdout);
>   fputs (_("  -print-prog-name=<prog>  Display the full path to compiler component <prog>\n"), stdout);
>   fputs (_("  -print-multi-directory   Display the root directory for versions of libgcc\n"), stdout);
> @@ -3222,6 +3223,7 @@ driver_handle_option (struct gcc_options *opts,
>     case OPT_print_sysroot:
>     case OPT_print_multi_os_directory:
>     case OPT_print_sysroot_headers_suffix:
> +    case OPT_print_lto_plugin:
>     case OPT_time:
>     case OPT_wrapper:
>       /* These options set the variables specified in common.opt
> @@ -6536,6 +6538,17 @@ main (int argc, char **argv)
>        fatal_error ("not configured with sysroot headers suffix");
>     }
>
> +  if (print_lto_plugin)
> +    {
> +      char *plugin;
> +
> +      plugin = find_a_file (&exec_prefixes, "liblto_plugin.so", R_OK, false);
> +      if (!plugin)
> +        fatal_error ("-print-lto-plugin specified, but no plugin found");
> +      printf ("%s\n", plugin);
> +      return (0);
> +    }
> +
>   if (print_help_list)
>     {
>       display_help ();
> --
> 1.7.1
>
>

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

On Fri, Oct 29, 2010 at 02:34:32PM +0100, Pedro Alves wrote:
> On Friday 29 October 2010 12:46:57, Andi Kleen wrote:
> > --- /dev/null
> > +++ b/gcc/lto/gcc-ranlib.in
> > @@ -0,0 +1,8 @@
> > +#!/bin/sh
> > +# wrapper for ranlib with GCC LTO support
> > +prefix=@prefix@
> > +exec_prefix=@exec_prefix@
> > +GCC_BINDIR=${GCC_BINDIR:-@bindir@}
> > +AR=${AR:-`$GCC_BINDIR/gcc -print-prog-name=ar`}
> > +
> > +exec $AR -s --plugin `$GCC_BINDIR/gcc -print-lto-plugin` "$@"
> 
> Is this meant to be used by gcc users, rather then when building gcc?

Both, but it's optional. You need it really only for LTO slim.

> At the risk of sounding obvious, I'd like to point out that wrapper
> shell scripts don't work on Windows hosts.

They would work with cygwin and such, right?

-Andi

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

On Fri, Oct 29, 2010 at 01:31:31PM +0000, Joseph S. Myers wrote:
> On Fri, 29 Oct 2010, Andi Kleen wrote:
> 
> > 	* gcc-ar.in, gcc-ld.in, gcc-ranlib.in: Add.
> 
> Installed scripts are a bad idea because GCC is portable to hosts that 
> cannot run shell scripts (the *build system* can run shell scripts, but 
> the *host* may be e.g. MinGW and not able to run them).  The portable way 
> to wrap subprocesses is to use C programs using the pexecute interface, as 
> in collect2 or lto-wrapper.

The scripts are really not intended to run everywhere, just on hosts
that support LTO (or really slim lto). That eliminates most of the
oddballs.

I think the only issue is Windows, but I think it's a reasonable
requirement to have a shell installed for slim LTO. Or actually
slim lto with makefiles that need ar/ranlib or call ld directly.

> > +ARGS=""
> 
> By building up a string like this you'll have lots of problems when 
> arguments contain spaces.  

I declare that just not supported right now.

> A C wrapper naturally avoids that.

Well if you want one feel free to write one. At least the scripts
do the job for me and they work for a gcc build and some other
projects.

> 
> > +prefix=@prefix@
> > +exec_prefix=@exec_prefix@
> 
> No.  GCC installations are meant to be relocatable; if the wrapper is 
> executed from somewhere other than the $bindir in which it is installed, 
> it must find other programs relative to the directory to which it has been 
> moved, rather than reusing configure-time absolute paths.  In a C wrapper 
> you can use make_relative_prefix.

Ok I can fix that in the script I think.

The good thing is that this allows dropping the ugly rules
for interacting with autoconf.

> 
> > +while test "x$1" != "x" ; do
> > +	case "$1" in
> > +	-r) 	R=1
> > +		N="$1" 
> > +		;;
> > +	-[olvO]|-f*|-nostdlib) 
> > +		N="$1" 
> > +		;;
> 
> -o and -l take arguments, which you don't seem to be allowing for here.

I'll move that thanks.

> 
> > +if test "x$R" != "x" ; then
> > +	# work around gold bug, -r and -fuse-linker-plugin
> > +	# do not interact well. It's not needed anyways.
> > +	# the gold bug has been fixed, but still keep this.
> > +	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`
> 
> Apart from the issues with spaces, echo will process some options on some 
> systems and this sed command will happily modify option arguments that 
> happen to contain that particular string.  As before, (POSIX) shell is not 
> a good language in which to do this robustly (if you e.g. assume various 
> bash features, it becomes easier).

Are these systems with funny echo supported by LTO?

In principle this could be dropped, but then everyone would need
to update their gold too.

-Andi

Re: [PATCH 1/2] Add -print-lto-plugin

[Andi Kleen]

On Fri, Oct 29, 2010 at 03:39:42PM +0200, Richard Guenther wrote:
> On Fri, Oct 29, 2010 at 1:46 PM, Andi Kleen <andi@firstfloor.org> wrote:
> > From: Andi Kleen <ak@linux.intel.com>
> >
> > binutils have LTO linker plugin support now, but it requires hardcoding
> > the path to gcc's libexec dir in the Makefile. Add an option to the
> > gcc driver instead to print the full file name to avoid this.
> >
> > This is especially useful for ar or nm.
> 
> Is the reason for not using -print-file-name=liblto-plugin.so to not hardcode
> that name?  (which you still did in the patch - not sure if it's that specific
> name on all hosts).

Yes, David Korn pointed out that doesn't work on Windows because of 
the postfix.

His COFF work is not merged yet, but he can simply switch the file name
around in gcc. on Windows when that gets merged. That's better than having
special cases everywhere in makefiles or in the wrappers.


-Andi

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andreas Schwab]

Andi Kleen <ak@linux.intel.com> writes:

> On Fri, Oct 29, 2010 at 01:31:31PM +0000, Joseph S. Myers wrote:
>> On Fri, 29 Oct 2010, Andi Kleen wrote:
>> > +if test "x$R" != "x" ; then
>> > +	# work around gold bug, -r and -fuse-linker-plugin
>> > +	# do not interact well. It's not needed anyways.
>> > +	# the gold bug has been fixed, but still keep this.
>> > +	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`
>> 
>> Apart from the issues with spaces, echo will process some options on some 
>> systems and this sed command will happily modify option arguments that 
>> happen to contain that particular string.  As before, (POSIX) shell is not 
>> a good language in which to do this robustly (if you e.g. assume various 
>> bash features, it becomes easier).
>
> Are these systems with funny echo supported by LTO?

You can do that without echo.

Andreas.

-- 
Andreas Schwab, schwab@redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

Re: [PATCH 1/2] Add -print-lto-plugin

[H.J. Lu]

On Fri, Oct 29, 2010 at 4:46 AM, Andi Kleen <andi@firstfloor.org> wrote:
> From: Andi Kleen <ak@linux.intel.com>
>
> binutils have LTO linker plugin support now, but it requires hardcoding
> the path to gcc's libexec dir in the Makefile. Add an option to the
> gcc driver instead to print the full file name to avoid this.
>
> This is especially useful for ar or nm.
>

There is

[hjl@gnu-6 gcc]$ grep LTOPLUGINSONAME configure.ac
AC_DEFINE_UNQUOTED(LTOPLUGINSONAME,"${host_lto_plugin_soname}",
[hjl@gnu-6 gcc]$

Why not add LTOPLUGINSONAME to Makefile? Then you can use
"gcc --print ...=$(LTOPLUGINSONAME)" to generate your scripts?



-- 
H.J.

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

On Fri, Oct 29, 2010 at 04:03:28PM +0200, Andreas Schwab wrote:
> Andi Kleen <ak@linux.intel.com> writes:
> 
> > On Fri, Oct 29, 2010 at 01:31:31PM +0000, Joseph S. Myers wrote:
> >> On Fri, 29 Oct 2010, Andi Kleen wrote:
> >> > +if test "x$R" != "x" ; then
> >> > +	# work around gold bug, -r and -fuse-linker-plugin
> >> > +	# do not interact well. It's not needed anyways.
> >> > +	# the gold bug has been fixed, but still keep this.
> >> > +	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`
> >> 
> >> Apart from the issues with spaces, echo will process some options on some 
> >> systems and this sed command will happily modify option arguments that 
> >> happen to contain that particular string.  As before, (POSIX) shell is not 
> >> a good language in which to do this robustly (if you e.g. assume various 
> >> bash features, it becomes easier).
> >
> > Are these systems with funny echo supported by LTO?
> 
> You can do that without echo.

How?  I was originally using bash ${../.../..}  but I was worried
that this wasn't portable enough, so i ended up with echo | sed instead.

-Andi

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Paolo Bonzini]

On 10/29/2010 01:46 PM, Andi Kleen wrote:
> From: Andi Kleen<ak@linux.intel.com>
>
> Earlier review resulted in a request for having gcc-... wrappers
> to handle the LTO symbol tables for ranlib/ar. This is needed for slim
> lto files, because these tools cannot read the symbol table
> otherwise. Essentially they just call binutils with
> the LTO plugin specified.
>
> Other compilers with LTO support tend to have similar tools.
>
> This patch implements those wrappers. I also added a wrapper for ld
> because I needed one for a large existing Makefile with a lot
> of complicated ld calls. It turns an ld command line into
> a gcc command line suitable for lto.
>
> The wrappers are also needed for a LTO slim of gcc. Right now
> they have to be manually specified for that.
>
> The wrappers require uptodate binutils (the upcoming release)
> with plugin support enabled.  There is currently no autoconf
> support to detect that. The wrappers query the gcc driver
> in an attempt to handle cross compilation and so naming everywhere.

Besides the problems that Joseph pointed out (I agree BTW that a rewrite 
in C is required for this to go in) I'm not sure I like the idea.

Is there any reason why the LTO plugin's additional capabilities should 
not be used _by default_ by ar/ranlib/ld (as a GNU extension to ELF, 
basically)?  I don't follow binutils development at all, what are other 
useful applications of plugins?

Paolo

Re: [PATCH 1/2] Add -print-lto-plugin

[Andi Kleen]

> [hjl@gnu-6 gcc]$ grep LTOPLUGINSONAME configure.ac
> AC_DEFINE_UNQUOTED(LTOPLUGINSONAME,"${host_lto_plugin_soname}",
> [hjl@gnu-6 gcc]$
> 
> Why not add LTOPLUGINSONAME to Makefile? Then you can use
> "gcc --print ...=$(LTOPLUGINSONAME)" to generate your scripts?

To which Makefile? The scripts should be called from arbitary
projects, not just the gcc makefile.

I'm not sure why you're so against that option anyways,
gcc has a couple of similar options to solve similar problems, 
why not have one more that is convenient?

-Andi

Re: [PATCH 1/2] Add -print-lto-plugin

[H.J. Lu]

On Fri, Oct 29, 2010 at 7:35 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> [hjl@gnu-6 gcc]$ grep LTOPLUGINSONAME configure.ac
>> AC_DEFINE_UNQUOTED(LTOPLUGINSONAME,"${host_lto_plugin_soname}",
>> [hjl@gnu-6 gcc]$
>>
>> Why not add LTOPLUGINSONAME to Makefile? Then you can use
>> "gcc --print ...=$(LTOPLUGINSONAME)" to generate your scripts?
>
> To which Makefile? The scripts should be called from arbitary
> projects, not just the gcc makefile.

You can use $(LTOPLUGINSONAME) when you generate your LTO
wrappers:

http://gcc.gnu.org/ml/gcc-patches/2010-10/msg02472.html

> I'm not sure why you're so against that option anyways,
> gcc has a couple of similar options to solve similar problems,
> why not have one more that is convenient?
>

I don't think it is necessary, especially with your LTO wrappers.


-- 
H.J.

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

> Besides the problems that Joseph pointed out (I agree BTW that a
> rewrite in C is required for this to go in) 

Ok I don't plan to do that right now because it seems wasteful
to me.

> Is there any reason why the LTO plugin's additional capabilities
> should not be used _by default_ by ar/ranlib/ld (as a GNU extension
> to ELF, basically)?  I don't follow binutils development at all,

That would mean you hardcode the LTO object format and cannot change it 
anymore, or only if you do a lockstep release with binutils.

Basically make it a standard or ship gcc always with binutils.

> what are other useful applications of plugins?

The main use is for sane LTO support in the linker.

It's useful for nm too with slim LTO files. I don't know of any
other uses, but in principle other tools could use it if they 
want do do operations on object files.

-Andi

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Paolo Bonzini]

On 10/29/2010 04:44 PM, Andi Kleen wrote:
>>>  Is there any reason why the LTO plugin's additional capabilities
>>>  should not be used_by default_  by ar/ranlib/ld (as a GNU extension
>>>  to ELF, basically)?  I don't follow binutils development at all,
>
> That would mean you hardcode the LTO object format and cannot change it
> anymore, or only if you do a lockstep release with binutils.
>
> Basically make it a standard or ship gcc always with binutils.

Understood, that's implied in "make a GNU extension to ELF" (though that 
was imprecise as it's also COFF and Mach-O at least).

>>>  what are other useful applications of plugins?
>
> The main use is for sane LTO support in the linker.
>
> It's useful for nm too with slim LTO files. I don't know of any
> other uses, but in principle other tools could use it if they
> want do do operations on object files.

ar/ranlib, ld, nm---it's still about LTO and about the LTO plugin only. 
  I'm trying to understand what other uses the plugin interface has 
beyond LTO.

Paolo

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Paolo Bonzini]

On 10/29/2010 04:30 PM, Andi Kleen wrote:
> >  You can do that without echo.
>
> How?  I was originally using bash ${../.../..}  but I was worried
> that this wasn't portable enough, so i ended up with echo | sed instead.

printf '%s\n' "$x"

Paolo

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Paolo Bonzini]

On 10/29/2010 04:30 PM, Andi Kleen wrote:
> >  You can do that without echo.
>
> How?  I was originally using bash ${../.../..}  but I was worried
> that this wasn't portable enough, so i ended up with echo | sed instead.

printf '%s\n' "$x"

Paolo

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

> ar/ranlib, ld, nm---it's still about LTO and about the LTO plugin
> only.  I'm trying to understand what other uses the plugin interface
> has beyond LTO.

It's really designed for LTO as I understand. Does it need any other uses?

-Andi

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Joseph S. Myers]

On Fri, 29 Oct 2010, Andi Kleen wrote:

> I think the only issue is Windows, but I think it's a reasonable
> requirement to have a shell installed for slim LTO. Or actually

We should not gratuitous reduce host portability.

There are cases where portability restrictions are reasonable - say the 
original LTO dependence on particular target object formats, or host 
dependencies for plugins.  But in this case the interfaces for calling 
subprocesses on many hosts already exist in libiberty and are widely used 
in GCC; not using them is gratuitous.

> > By building up a string like this you'll have lots of problems when 
> > arguments contain spaces.  
> 
> I declare that just not supported right now.

Just declaring things unsupported like that is inappropriate.  
Unsupported things need documentation, but it's clearly better just to 
support them in this case.

> > Apart from the issues with spaces, echo will process some options on some 
> > systems and this sed command will happily modify option arguments that 
> > happen to contain that particular string.  As before, (POSIX) shell is not 
> > a good language in which to do this robustly (if you e.g. assume various 
> > bash features, it becomes easier).
> 
> Are these systems with funny echo supported by LTO?

echo in bash will handle -n and -e.  POSIX echo interprets backslash 
sequences in its operands.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andreas Schwab]

Andi Kleen <andi@firstfloor.org> writes:

> On Fri, Oct 29, 2010 at 04:03:28PM +0200, Andreas Schwab wrote:
>> Andi Kleen <ak@linux.intel.com> writes:
>> 
>> > On Fri, Oct 29, 2010 at 01:31:31PM +0000, Joseph S. Myers wrote:
>> >> On Fri, 29 Oct 2010, Andi Kleen wrote:
>> >> > +if test "x$R" != "x" ; then
>> >> > +	# work around gold bug, -r and -fuse-linker-plugin
>> >> > +	# do not interact well. It's not needed anyways.
>> >> > +	# the gold bug has been fixed, but still keep this.
>> >> > +	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`
>> >> 
>> >> Apart from the issues with spaces, echo will process some options on some 
>> >> systems and this sed command will happily modify option arguments that 
>> >> happen to contain that particular string.  As before, (POSIX) shell is not 
>> >> a good language in which to do this robustly (if you e.g. assume various 
>> >> bash features, it becomes easier).
>> >
>> > Are these systems with funny echo supported by LTO?
>> 
>> You can do that without echo.
>
> How?

Iterate over $ARGS constructing a new ARGS.  That will also eliminate
the false match (the whitespace problem will remain, though).

Andreas.

-- 
Andreas Schwab, schwab@redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Ralf Wildenhues]

* Andi Kleen wrote on Fri, Oct 29, 2010 at 04:30:49PM CEST:
> On Fri, Oct 29, 2010 at 04:03:28PM +0200, Andreas Schwab wrote:
> > Andi Kleen <ak@linux.intel.com> writes:
> > > On Fri, Oct 29, 2010 at 01:31:31PM +0000, Joseph S. Myers wrote:
> > >> On Fri, 29 Oct 2010, Andi Kleen wrote:
> > >> > +if test "x$R" != "x" ; then
> > >> > +	# work around gold bug, -r and -fuse-linker-plugin
> > >> > +	# do not interact well. It's not needed anyways.
> > >> > +	# the gold bug has been fixed, but still keep this.
> > >> > +	ARGS=`echo $ARGS | sed -e s/-fuse-linker-plugin//`
> > >> 
> > >> Apart from the issues with spaces, echo will process some options on some 
> > >> systems and this sed command will happily modify option arguments that 
> > >> happen to contain that particular string.  As before, (POSIX) shell is not 
> > >> a good language in which to do this robustly (if you e.g. assume various 
> > >> bash features, it becomes easier).
> > >
> > > Are these systems with funny echo supported by LTO?
> > 
> > You can do that without echo.
> 
> How?  I was originally using bash ${../.../..}  but I was worried
> that this wasn't portable enough, so i ended up with echo | sed instead.

Go through "$@" with shift and set, and remember that "$@" is the
only array you have in portable shell (and the only way you can get
command-line arguments in a single variable is by quoting each arg by
itself; but in your script this isn't needed at all).  Look at the
compile script in toplevel gcc/ to see how it's done; otherwise I can
also write you a portable version of your code tomorrow, iff there is
consensus that a shell version is still desirable.

Cheers,
Ralf

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[H.J. Lu]

On Fri, Oct 29, 2010 at 4:46 AM, Andi Kleen <andi@firstfloor.org> wrote:
> From: Andi Kleen <ak@linux.intel.com>
>
> Earlier review resulted in a request for having gcc-... wrappers
> to handle the LTO symbol tables for ranlib/ar. This is needed for slim
> lto files, because these tools cannot read the symbol table
> otherwise. Essentially they just call binutils with
> the LTO plugin specified.
>
> Other compilers with LTO support tend to have similar tools.
>
> This patch implements those wrappers. I also added a wrapper for ld
> because I needed one for a large existing Makefile with a lot
> of complicated ld calls. It turns an ld command line into
> a gcc command line suitable for lto.
>
> The wrappers are also needed for a LTO slim of gcc. Right now
> they have to be manually specified for that.
>
> The wrappers require uptodate binutils (the upcoming release)
> with plugin support enabled.  There is currently no autoconf
> support to detect that. The wrappers query the gcc driver
> in an attempt to handle cross compilation and so naming everywhere.
>
> gcc/lto/
>
> 2010-10-26  Andi Kleen  <ak@linux.intel.com>
>
>        * Make-lang.in (AR_INSTALL_NAME, RANLIB_INSTALL_NAME, LD_INSTALL_NAME,
>        LTO_WRAPPERS): Add.
>        (lto.all.cross, lto.start.cross): Add dependency to LTO_WRAPPERS.
>        (lto.install.common): Install wrappers.
>        (lto.mostlyclean): Clean wrappers.
>        (gcc-ar, gcc-ranlib, gcc-ld): Add.
>        * config-lang.in (outputs): Add.
>        * gcc-ar.in, gcc-ld.in, gcc-ranlib.in: Add.
>
> gcc/
>
> 2010-10-26  Andi Kleen  <ak@linux.intel.com>
>
>        * doc/invoke.texi (gcc-ar, gcc-ld, gcc-ranlib): Document.

If I have gcc 4.5 installed under /usr/bin as system compiler and gcc 4.6
installed under /usr/gcc-4.6/bin, which plug-in DSO will /usr/gcc-4.6/bin/gcc-ld
use?


H.J.

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[Andi Kleen]

> If I have gcc 4.5 installed under /usr/bin as system compiler and gcc 4.6
> installed under /usr/gcc-4.6/bin, which plug-in DSO will /usr/gcc-4.6/bin/gcc-ld
> use?

The current version calls the gcc at install path and asks it.
Joseph asked for a relocatable version that asks the gcc
in the same directory as the script.

-Andi

Re: [PATCH 2/2] Add gcc-ar, gcc-ranlib, gcc-ld wrappers

[H.J. Lu]

On Fri, Oct 29, 2010 at 8:57 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> If I have gcc 4.5 installed under /usr/bin as system compiler and gcc 4.6
>> installed under /usr/gcc-4.6/bin, which plug-in DSO will /usr/gcc-4.6/bin/gcc-ld
>> use?
>
> The current version calls the gcc at install path and asks it.

That won't work with

# mv /usr/gcc-4.6 ~/tmp/
# ~/tmp/gcc-4.6/bin/gcc-ld ...

> Joseph asked for a relocatable version that asks the gcc
> in the same directory as the script.
>

That sounds better.


-- 
H.J.