[LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[William Maddox]

[-- Attachment #1: Type: text/plain, Size: 2261 bytes --]

This patch adds the missing '-flto-test' functionality to the 'gcc'
driver for LTO.
Diego strongly objected to the name '-flto-test' as used in the LTO driver
specification and in an earlier version of this patch.  He suggested
'-flto-single'
as a more descriptive name, and I have followed his suggestion here.

The intent of '-flto-single' is that 'gcc -flto-single' should behave
in a manner
functionally identical to 'gcc' without any LTO options.  In order to
exercise the
externalization and internalization of the IR, however, the usual compilation
pipeline via 'cc1' takes a detour through 'lto1'.  In this manner,  substantial
applications can be used as test cases by adding only a single flag to their
makefiles. Note that use of '-flto-single' does not enable testing the proper
combining of multiple compilation units in 'lto1'.  Furthermore, when compiling
a single source file all the way to an executable, simply performing a compile
and link with the '-flto' option achieves essentially the same effect in a more
satisfactorily end-to-end manner.  Thus it is unclear what value '-flto-single'
will have once LTO is functionally complete.

I could not find a way to deal with the '-flto-single' option within the specs
language implemented by 'gcc'.   I thus extended the specs language
to permit a spec function to be invoked as a predicate within the %{} construct,
for example:

    %{?lto-single():-flto}

An empty string (or NULL) returned from the function counts as false,
while all other values count as true.

       * gcc.c: (lto_single): New variable.  Flag to indicate
       lto-single mode.
       (LINK_COMMAND_SPEC,cc1_options): Add support for
       lto-single mode.
       (invoke_lto_single): New constant.
       Specs string to run hidden lto1 pass.
       (default_compilers): Add support for lto-single
       mode in specs for C compilations.
       (static_specs): Add entry for invoke_lto_single.
       (process_command): Recognize -flto-single option.
       (handle_spec_predicate): Parse and invoke spec predicate,
       similar to spec funtion.
       (handle_braces): Implement conditional on spec predicates.
       (lto_single_spec_predicate): New function. Spec predicate
       to test for lto-single mode.

[-- Attachment #2: lto-single-patch.txt --]
[-- Type: text/plain, Size: 12071 bytes --]

Index: gcc/gcc.c
===================================================================
--- gcc/gcc.c	(revision 130298)
+++ gcc/gcc.c	(working copy)
@@ -230,6 +230,13 @@
 
 static int use_pipes;
 
+/* Nonzero means we are running in lto-single mode, and each object file
+   resulting from a compilation should be immediately recompiled using
+   the LTO back end.  This makes it easy to exercise the LTO by adding
+   a single switch to existing makefiles.  */
+
+static int lto_single;
+
 /* The compiler version.  */
 
 static const char *compiler_version;
@@ -365,6 +372,7 @@
 static const char *version_compare_spec_function (int, const char **);
 static const char *include_spec_function (int, const char **);
 static const char *print_asm_header_spec_function (int, const char **);
+static const char *lto_single_spec_function (int, const char **);
 \f
 /* The Specs Language
 
@@ -522,7 +530,16 @@
  %{!.S:X} substitutes X, if NOT processing a file with suffix S.
  %{,S:X}  substitutes X, if processing a file which will use spec S.
  %{!,S:X} substitutes X, if NOT processing a file which will use spec S.
-	  
+
+ %{?function(args):X}
+  	  Call the named function FUNCTION, passing it ARGS.  ARGS is
+	  first processed as a nested spec string, then split into an
+	  argument vector in the usual fashion.  If the function
+	  returns a non-empty string, substitute X.
+ %{!?function(args):X}
+ 	  substitutes X, if FUNCTION returns an empty string.  ARGS is
+          processed as in the previous case.
+
  %{S|T:X} substitutes X if either -S or -T was given to CC.  This may be
 	  combined with '!', '.', ',', and '*' as above binding stronger
 	  than the OR.
@@ -832,8 +849,20 @@
  %{fsyntax-only:-o %j} %{-param*}\
  %{fmudflap|fmudflapth:-fno-builtin -fno-merge-constants}\
  %{coverage:-fprofile-arcs -ftest-coverage}\
- %{flto:-O2}";
+ %{flto:-O2} %{?lto-single(): -O2}";
 
+/* Do we need to preserve any assembler options here?
+   Note that we are going to ignore the object code, as
+   we are only interested in the .lto_info sections.  */
+static const char *invoke_lto_single =
+#ifdef AS_NEEDS_DASH_FOR_PIPED_INPUT
+"%{?lto-single(): -o %|.lto.s |\n as %(asm_options) %|.lto.s -o %g.lto.o \n\
+ lto1 %(cc1_options) %g.lto.o }";
+#else
+"%{?lto-single(): -o %|.lto.s |\n as %(asm_options) %m.lto.s -o %g.lto.o \n\
+ lto1 %(cc1_options) %g.lto.o }";
+#endif
+
 static const char *asm_options =
 "%{--target-help:%:print-asm-header()} "
 #if HAVE_GNU_AS
@@ -992,16 +1021,16 @@
 	  %{save-temps|traditional-cpp|no-integrated-cpp:%(trad_capable_cpp) \
 		%(cpp_options) -o %{save-temps:%b.i} %{!save-temps:%g.i} \n\
 		    cc1 -fpreprocessed %{save-temps:%b.i} %{!save-temps:%g.i} \
-			%(cc1_options)}\
+			%{?lto-single():-flto} %(cc1_options)}\
 	  %{!save-temps:%{!traditional-cpp:%{!no-integrated-cpp:\
-		cc1 %(cpp_unique_options) %(cc1_options)}}}\
-          %{!fsyntax-only:%(invoke_as)}} \
+		cc1 %(cpp_unique_options) %{?lto-single():-flto} %(cc1_options)}}}\
+          %{!fsyntax-only:%(invoke_lto_single) %(invoke_as)}} \
       %{combine:\
 	  %{save-temps|traditional-cpp|no-integrated-cpp:%(trad_capable_cpp) \
 		%(cpp_options) -o %{save-temps:%b.i} %{!save-temps:%g.i}}\
 	  %{!save-temps:%{!traditional-cpp:%{!no-integrated-cpp:\
-		cc1 %(cpp_unique_options) %(cc1_options)}}\
-                %{!fsyntax-only:%(invoke_as)}}}}}}", 0, 1, 1},
+		cc1 %(cpp_unique_options) %{?lto-single():-flto} %(cc1_options)}}\
+                %{!fsyntax-only:%(invoke_lto_single) %(invoke_as)}}}}}}", 0, 1, 1},
   {"-",
    "%{!E:%e-E or -x required when input is from standard input}\
     %(trad_capable_cpp) %(cpp_options) %(cpp_debug_options)", 0, 0, 0},
@@ -1023,7 +1052,8 @@
                     %W{o*:--output-pch=%*}%V}}}}}}", 0, 0, 0},
   {".i", "@cpp-output", 0, 1, 0},
   {"@cpp-output",
-   "%{!M:%{!MM:%{!E:cc1 -fpreprocessed %i %(cc1_options) %{!fsyntax-only:%(invoke_as)}}}}", 0, 1, 0},
+   "%{!M:%{!MM:%{!E:cc1 -fpreprocessed %i %{?lto-single():-flto} %(cc1_options) \
+%{!fsyntax-only:%(invoke_lto_single) %(invoke_as)}}}}", 0, 1, 0},
   {".s", "@assembler", 0, 1, 0},
   {"@assembler",
    "%{!M:%{!MM:%{!E:%{!S:as %(asm_debug) %(asm_options) %i %A }}}}", 0, 1, 0},
@@ -1573,6 +1603,7 @@
   INIT_STATIC_SPEC ("asm_final",		&asm_final_spec),
   INIT_STATIC_SPEC ("asm_options",		&asm_options),
   INIT_STATIC_SPEC ("invoke_as",		&invoke_as),
+  INIT_STATIC_SPEC ("invoke_lto_single",	&invoke_lto_single),
   INIT_STATIC_SPEC ("cpp",			&cpp_spec),
   INIT_STATIC_SPEC ("cpp_options",		&cpp_options),
   INIT_STATIC_SPEC ("cpp_debug_options",	&cpp_debug_options),
@@ -1639,6 +1670,7 @@
   { "version-compare",		version_compare_spec_function },
   { "include",			include_spec_function },
   { "print-asm-header",		print_asm_header_spec_function },
+  { "lto-single",		lto_single_spec_function },
 #ifdef EXTRA_SPEC_FUNCTIONS
   EXTRA_SPEC_FUNCTIONS
 #endif
@@ -3805,6 +3837,8 @@
 	  use_pipes = 1;
 	  n_switches++;
 	}
+      else if (strcmp (argv[i], "-flto-single") == 0)
+        lto_single = 1;
       else if (strcmp (argv[i], "-###") == 0)
 	{
 	  /* This is similar to -v except that there is no execution
@@ -4176,6 +4210,8 @@
 	;
       else if (strcmp (argv[i], "-time") == 0)
 	;
+      else if (strcmp (argv[i], "-flto-single") == 0)
+	;
       else if (strcmp (argv[i], "-###") == 0)
 	;
       else if (argv[i][0] == '-' && argv[i][1] != 0)
@@ -5573,6 +5609,73 @@
   return p;
 }
 
+/* Handle a spec predicate call in braces, i.e., %{?test-foo(): foobar }.
+
+   ARGS is processed as a spec in a separate context and split into an
+   argument vector in the normal fashion.  The predicate is a normal
+   spec function that returns an empty string (or NULL) if the predicate
+   is false, and some  other non-empty string value if true.  This allows
+   us to reuse existing machinery for evaluating the functions, and opens
+   the possibility of predicates that return useful values.
+
+   This function substantially duplicates handle_spec_function() above,
+   but note that the parsing of the function name differs slightly.
+   Maximal sharing would come at the expense of better error diagnostics
+   for non-predicate function calls.  */
+
+static const char *
+handle_spec_predicate (const char *p, bool *result)
+{
+  char *func, *args;
+  const char *endp, *funcval;
+  int count;
+
+  processing_spec_function++;
+
+  /* Get the function name.  */
+  for (endp = p; *endp != '\0'; endp++)
+    {
+      /* Only allow [A-Za-z0-9], -, and _ in function names.  */
+      if (!ISALNUM (*endp) && !(*endp == '-' || *endp == '_'))
+	  break;
+    }
+  if (*endp != '(')		/* ) */
+    fatal ("no arguments for spec function");
+  func = save_string (p, endp - p);
+  p = ++endp;
+
+  /* Get the arguments.  */
+  for (count = 0; *endp != '\0'; endp++)
+    {
+      /* ( */
+      if (*endp == ')')
+	{
+	  if (count == 0)
+	    break;
+	  count--;
+	}
+      else if (*endp == '(')	/* ) */
+	count++;
+    }
+  /* ( */
+  if (*endp != ')')
+    fatal ("malformed spec function arguments");
+  args = save_string (p, endp - p);
+  p = ++endp;
+
+  /* p now points to just past the end of the spec function expression.  */
+
+  funcval = eval_spec_function (func, args);
+  *result = ((funcval != NULL) && (funcval[0] != '\0'));
+
+  free (func);
+  free (args);
+
+  processing_spec_function--;
+
+  return p;
+}
+
 /* Inline subroutine of handle_braces.  Returns true if the current
    input suffix matches the atom bracketed by ATOM and END_ATOM.  */
 static inline bool
@@ -5660,6 +5763,7 @@
 
   bool a_is_suffix;
   bool a_is_spectype;
+  bool a_is_predicate;
   bool a_is_starred;
   bool a_is_negated;
   bool a_matched;
@@ -5672,6 +5776,8 @@
   bool n_way_choice   = false;
   bool n_way_matched  = false;
 
+  bool pred_result    = false;
+
 #define SKIP_WHITE() do { while (*p == ' ' || *p == '\t') p++; } while (0)
 
   do
@@ -5680,31 +5786,44 @@
 	goto invalid;
 
       /* Scan one "atom" (S in the description above of %{}, possibly
-	 with '!', '.', '@', ',', or '*' modifiers).  */
+	 with '!', '.', '@', ',', '?', or '*' modifiers).  */
       a_matched = false;
       a_is_suffix = false;
       a_is_starred = false;
       a_is_negated = false;
       a_is_spectype = false;
+      a_is_predicate = false;
 
       SKIP_WHITE();
       if (*p == '!')
 	p++, a_is_negated = true;
 
       SKIP_WHITE();
-      if (*p == '.')
-	p++, a_is_suffix = true;
-      else if (*p == ',')
-	p++, a_is_spectype = true;
+      if (*p == '?')
+	{
+	  /* The atom is taken to be the text of the function
+	     call itself.  This is done pro-forma only, and we
+	     do not expect the atom to be used.  */
+	  p++;
+	  atom = p;
+	  p = handle_spec_predicate(p, &pred_result);
+	  end_atom = p;
+	  a_is_predicate = true;
+	} else {
+	  if (*p == '.')
+	    p++, a_is_suffix = true;
+	  else if (*p == ',')
+	    p++, a_is_spectype = true;
 
-      atom = p;
-      while (ISIDNUM(*p) || *p == '-' || *p == '+' || *p == '='
-	     || *p == ',' || *p == '.' || *p == '@')
-	p++;
-      end_atom = p;
+	  atom = p;
+	  while (ISIDNUM(*p) || *p == '-' || *p == '+' || *p == '='
+		 || *p == ',' || *p == '.' || *p == '?' || *p == '@')
+	    p++;
+	  end_atom = p;
 
-      if (*p == '*')
-	p++, a_is_starred = 1;
+	  if (*p == '*')
+	    p++, a_is_starred = 1;
+	}
 
       SKIP_WHITE();
       switch (*p)
@@ -5713,7 +5832,7 @@
 	  /* Substitute the switch(es) indicated by the current atom.  */
 	  ordered_set = true;
 	  if (disjunct_set || n_way_choice || a_is_negated || a_is_suffix
-	      || a_is_spectype || atom == end_atom)
+	      || a_is_spectype || a_is_predicate || atom == end_atom)
 	    goto invalid;
 
 	  mark_matching_switches (atom, end_atom, a_is_starred);
@@ -5732,8 +5851,8 @@
 	  if (atom == end_atom)
 	    {
 	      if (!n_way_choice || disj_matched || *p == '|'
-		  || a_is_negated || a_is_suffix || a_is_spectype 
-		  || a_is_starred)
+		  || a_is_negated || a_is_suffix || a_is_spectype
+		  || a_is_predicate || a_is_starred)
 		goto invalid;
 
 	      /* An empty term may appear as the last choice of an
@@ -5744,7 +5863,7 @@
 	    }
 	  else
 	    {
-	      if ((a_is_suffix || a_is_spectype) && a_is_starred)
+	      if ((a_is_suffix || a_is_spectype || a_is_predicate) && a_is_starred)
 		goto invalid;
 	      
 	      if (!a_is_starred)
@@ -5758,7 +5877,9 @@
 		    a_matched = input_suffix_matches (atom, end_atom);
 		  else if (a_is_spectype)
 		    a_matched = input_spec_matches (atom, end_atom);
-		  else
+		  else if (a_is_predicate)
+		    a_matched = pred_result;
+                  else
 		    a_matched = switch_matches (atom, end_atom, a_is_starred);
 		  
 		  if (a_matched != a_is_negated)
@@ -7972,3 +8093,16 @@
   fflush (stdout);
   return NULL;
 }
+
+/* ?lto-single() spec predicate.  Return true, i.e., a non-empty string,
+   if the -flto-single switch was given to 'gcc'.  */
+
+static const char *
+lto_single_spec_function (int argc,
+                          const char **argv ATTRIBUTE_UNUSED)
+{
+  if (argc != 0)
+    abort ();
+
+  return ((lto_single) ? "lto-single" : NULL);
+}
Index: gcc/ChangeLog
===================================================================
--- gcc/ChangeLog	(revision 130298)
+++ gcc/ChangeLog	(working copy)
@@ -1,3 +1,21 @@
+2007-11-20  Bill Maddox  <maddox@google.com>
+
+	* gcc.c: (lto_single): New variable.  Flag to indicate
+	lto-single mode.
+	(LINK_COMMAND_SPEC,cc1_options): Add support for
+	lto-single mode.
+	(invoke_lto_single): New constant.
+	Specs string to run hidden lto1 pass.
+	(default_compilers): Add support for lto-single
+	mode in specs for C compilations.
+	(static_specs): Add entry for invoke_lto_single.
+	(process_command): Recognize -flto-single option.
+	(handle_spec_predicate): Parse and invoke spec predicate,
+	similar to spec funtion.
+	(handle_braces): Implement conditional on spec predicates.
+	(lto_single_spec_predicate): New function. Spec predicate
+	to test for lto-single mode.
+
 2007-11-13  Jakub Jelinek  <jakub@redhat.com>
 
 	PR tree-optimization/34063

Re: [LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[Mark Mitchell]

William Maddox wrote:
> This patch adds the missing '-flto-test' functionality to the 'gcc'
> driver for LTO.

Thanks!

> Diego strongly objected to the name '-flto-test' as used in the LTO driver
> specification and in an earlier version of this patch.  He suggested
> '-flto-single'
> as a more descriptive name, and I have followed his suggestion here.

OK; I have no strong feelings about it.  (The reason I suggested -test
was that this mode isn't useful for anything except testing.)

> I could not find a way to deal with the '-flto-single' option within the specs
> language implemented by 'gcc'.   I thus extended the specs language
> to permit a spec function to be invoked as a predicate within the %{} construct,
> for example:
> 
>     %{?lto-single():-flto}

Is that because of the -f in the option name?  So, something like:

  %{flto-single:-flto}

doesn't work?  If so, perhaps we should just drop the "-f" as that would
avoid creating this new infrastructure just for this case.

Nathan, I know you've been running the DejaGNU testsuite with LTO.  Have
you been using Bill's patch, or have you cobbled some other mechanism to
do that?

Thanks,

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713

Re: [LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[Diego Novillo]

Mark Mitchell wrote:

> Nathan, I know you've been running the DejaGNU testsuite with LTO.  Have
> you been using Bill's patch, or have you cobbled some other mechanism to
> do that?

It's already possible to run most of the testsuite with -flto already. 
We've been running compile.exp and execute.exp for a while now.  I 
patched both files to add -flto a few weeks ago.

There is a fair amount of working tests, which is nice.

Diego.

Re: [LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[Mark Mitchell]

Diego Novillo wrote:
> Mark Mitchell wrote:
> 
>> Nathan, I know you've been running the DejaGNU testsuite with LTO.  Have
>> you been using Bill's patch, or have you cobbled some other mechanism to
>> do that?
> 
> It's already possible to run most of the testsuite with -flto already.
> We've been running compile.exp and execute.exp for a while now.  I
> patched both files to add -flto a few weeks ago.

So, do you think we still need -flto-single?  It's purpose was to
facilitate testing; if you've already managed to get the testsuite to
pass -flto at compile-time and at link-time, then do we still think
-flto-single is worthwhile?

Thanks,

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713

Re: [LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[Diego Novillo]

Mark Mitchell wrote:

> So, do you think we still need -flto-single?  It's purpose was to
> facilitate testing; if you've already managed to get the testsuite to
> pass -flto at compile-time and at link-time, then do we still think
> -flto-single is worthwhile?

I'm not really thrilled with it, but it may be useful in that in the 
beginning it will allow more tests to go through.  Since each file is 
compiled to final object code using LTO independently, we don't run into 
the multi-file combination problems.

But I don't know if that is a good enough reason to keep this flag.  It 
won't be around for very long.  If people find it useful, I don't mind 
adding it.


Diego.

Re: [LTO] Patch to implement -flto-single (-flto-test) switch in gcc driver

[Mark Mitchell]

Diego Novillo wrote:
> Mark Mitchell wrote:
> 
>> So, do you think we still need -flto-single?  It's purpose was to
>> facilitate testing; if you've already managed to get the testsuite to
>> pass -flto at compile-time and at link-time, then do we still think
>> -flto-single is worthwhile?
> 
> I'm not really thrilled with it, but it may be useful in that in the
> beginning it will allow more tests to go through.

I feel bad about asking Bill to do this in the first place, but if you
don't think it's going to be useful in the medium term, then I think we
should just skip it.  It doesn't seem like a good idea to put it in,
only to take it out again.

Sorry,

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713