[PATCH][2/n] LTO option handling/merging rewrite

[PATCH][2/n] LTO option handling/merging rewrite

[Richard Guenther]

This moves the existing processing of user options from lto1 to
the lto driver (lto-wrapper).  It also changes the way we stream
user options from some custom binary format over to simply
streaming the original command-line as passed to the compiler
by the driver as a COLLECT_GCC_OPTIONS-like string.

lto-wrapper in this patch version (tries to) performs exactly
the same as lto1 tried to do - re-issue all target specific
options and a selected set of switches.

I chose to do this as an incremental step which should not
change behavior or regress in any way (fingers crossing).

From here we can think what would be the most sensible behavior
(and maybe start tagging options in the .opt files so they
get treatment based on some flag).

LTO bootstrapped and tested on x86_64-unknown-linux-gnu, ok?

Thanks,
Richard.

2011-10-28  Richard Guenther  <rguenther@suse.de>

	* lto-opts.c: Re-implement.
	* lto-streamer.h (lto_register_user_option): Remove.
	(lto_read_file_options): Likewise.
	(lto_reissue_options): Likewise.
	(lto_clear_user_options): Likewise.
	(lto_clear_file_options): Likewise.
	* opts-global.c (post_handling_callback): Remove.
	(set_default_handlers): Do not set post_handling_callback.
	(decode_options): Remove LTO specific code.
	* lto-wrapper.c (merge_and_complain): New function.
	(run_gcc): Read all input file options and
	prepend a merged set before the linker driver options.
	* gcc.c (driver_post_handling_callback): Remove.
	(set_option_handlers): Do not set post_handling_callback.
	* opts-common.c (handle_option): Do not call post_handling_callback.
	* opts.h (struct cl_option_handlers): Remove post_handling_callback.

	lto/
	* lto-lang.c (lto_post_options): Do not read file options.
	* lto.c (lto_read_all_file_options): Remove.
	(lto_init): Call lto_set_in_hooks here.


Index: trunk/gcc/lto-opts.c
===================================================================
*** trunk.orig/gcc/lto-opts.c	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/lto-opts.c	2011-10-28 12:15:52.000000000 +0200
***************
*** 1,6 ****
  /* LTO IL options.
  
!    Copyright 2009, 2010, 2011 Free Software Foundation, Inc.
     Contributed by Simon Baldwin <simonb@google.com>
  
  This file is part of GCC.
--- 1,6 ----
  /* LTO IL options.
  
!    Copyright 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
     Contributed by Simon Baldwin <simonb@google.com>
  
  This file is part of GCC.
*************** along with GCC; see the file COPYING3.
*** 33,422 ****
  #include "common/common-target.h"
  #include "diagnostic.h"
  #include "lto-streamer.h"
! 
! /* When a file is initially compiled, the options used when generating
!    the IL are not necessarily the same as those used when linking the
!    objects into the final executable.  In general, most build systems
!    will proceed with something along the lines of:
! 
!    	$ gcc <cc-flags> -flto -c f1.c -o f1.o
! 	$ gcc <cc-flags> -flto -c f2.c -o f2.o
! 	...
! 	$ gcc <cc-flags> -flto -c fN.c -o fN.o
! 
!    And the final link may or may not include the same <cc-flags> used
!    to generate the initial object files:
! 
!    	$ gcc <ld-flags> -flto -o prog f1.o ... fN.o
! 
!    Since we will be generating final code during the link step, some
!    of the flags used during the compile step need to be re-applied
!    during the link step.  For instance, flags in the -m family.
! 
!    The idea is to save a selected set of <cc-flags> in a special
!    section of the initial object files.  This section is then read
!    during linking and the options re-applied.
! 
!    FIXME lto.  Currently the scheme is limited in that only the
!    options saved on the first object file (f1.o) are read back during
!    the link step.  This means that the options used to compile f1.o
!    will be applied to ALL the object files in the final link step.
!    More work needs to be done to implement a merging and validation
!    mechanism, as this will not be enough for all cases.  */
! 
! /* Saved options hold the type of the option (currently CL_TARGET or
!    CL_COMMON), and the code, argument, and value.  */
! 
! typedef struct GTY(()) opt_d
! {
!   unsigned int type;
!   size_t code;
!   char *arg;
!   int value;
! } opt_t;
! 
! DEF_VEC_O (opt_t);
! DEF_VEC_ALLOC_O (opt_t, heap);
! 
! 
! /* Options are held in two vectors, one for those registered by
!    command line handling code, and the other for those read in from
!    any LTO IL input.  */
! static VEC(opt_t, heap) *user_options = NULL;
! static VEC(opt_t, heap) *file_options = NULL;
! 
! /* Iterate FROM in reverse, writing option codes not yet in CODES into *TO.
!    Mark each new option code encountered in CODES.  */
! 
! static void
! reverse_iterate_options (VEC(opt_t, heap) *from, VEC(opt_t, heap) **to,
! 			 bitmap codes)
! {
!   int i;
! 
!   for (i = VEC_length (opt_t, from); i > 0; i--)
!     {
!       const opt_t *const o = VEC_index (opt_t, from, i - 1);
! 
!       if (bitmap_set_bit (codes, o->code))
! 	VEC_safe_push (opt_t, heap, *to, o);
!     }
! }
! 
! /* Concatenate options vectors FIRST and SECOND, rationalize so that only the
!    final of any given option remains, and return the result.  */
! 
! static VEC(opt_t, heap) *
! concatenate_options (VEC(opt_t, heap) *first, VEC(opt_t, heap) *second)
! {
!   VEC(opt_t, heap) *results = NULL;
!   bitmap codes = lto_bitmap_alloc ();
! 
!   reverse_iterate_options (second, &results, codes);
!   reverse_iterate_options (first, &results, codes);
! 
!   lto_bitmap_free (codes);
!   return results;
! }
! 
! /* Clear the options vector in *OPTS_P and set it to NULL.  */
! 
! static void
! clear_options (VEC(opt_t, heap) **opts_p)
! {
!   int i;
!   opt_t *o;
! 
!   FOR_EACH_VEC_ELT (opt_t, *opts_p, i, o)
!     free (o->arg);
! 
!   VEC_free (opt_t, heap, *opts_p);
! }
! 
! /* Write LENGTH bytes from ADDR to STREAM.  */
! 
! static void
! output_data_stream (struct lto_output_stream *stream,
!                     const void *addr, size_t length)
! {
!   lto_output_data_stream (stream, addr, length);
! }
! 
! /* Write string STRING to STREAM.  */
! 
! static void
! output_string_stream (struct lto_output_stream *stream, const char *string)
! {
!   bool flag = false;
! 
!   if (string != NULL)
!     {
!       const size_t length = strlen (string);
! 
!       flag = true;
!       output_data_stream (stream, &flag, sizeof (flag));
!       output_data_stream (stream, &length, sizeof (length));
!       output_data_stream (stream, string, length);
!     }
!   else
!     output_data_stream (stream, &flag, sizeof (flag));
! }
! 
! /* Return a string from IB.  The string is allocated, and the caller is
!    responsible for freeing it.  */
! 
! static char *
! input_string_block (struct lto_input_block *ib)
! {
!   bool flag;
! 
!   lto_input_data_block (ib, &flag, sizeof (flag));
!   if (flag)
!     {
!       size_t length;
!       char *string;
! 
!       lto_input_data_block (ib, &length, sizeof (length));
!       string = (char *) xcalloc (1, length + 1);
!       lto_input_data_block (ib, string, length);
! 
!       return string;
!     }
!   else
!     return NULL;
! }
! 
! /* Return true if this option is one we need to save in LTO output files.
!    At present, we pass along all target options, and common options that
!    involve position independent code.
! 
!    TODO This list of options requires expansion and rationalization.
!    Among others, optimization options may well be appropriate here.  */
! 
! static bool
! register_user_option_p (size_t code, unsigned int type)
! {
!   if (type == CL_TARGET)
!     return true;
!   else if (type == CL_COMMON)
!     {
!       return (code == OPT_fPIC
! 	      || code == OPT_fpic
! 	      || code == OPT_fPIE
! 	      || code == OPT_fpie
! 	      || code == OPT_fcommon
! 	      || code == OPT_fexceptions);
!     }
! 
!   return false;
! }
! 
! /* Note command line option with the given TYPE and CODE, ARG, and VALUE.
!    If relevant to LTO, save it in the user options vector.  */
! 
! void
! lto_register_user_option (size_t code, const char *arg, int value,
! 			  unsigned int type)
! {
!   if (register_user_option_p (code, type))
!     {
!       opt_t o;
! 
!       o.type = type;
!       o.code = code;
!       if (arg != NULL)
! 	{
! 	  o.arg = (char *) xmalloc (strlen (arg) + 1);
! 	  strcpy (o.arg, arg);
! 	}
!       else
! 	o.arg = NULL;
!       o.value = value;
!       VEC_safe_push (opt_t, heap, user_options, &o);
!     }
! }
! 
! /* Empty the saved user options vector.  */
! 
! void
! lto_clear_user_options (void)
! {
!   clear_options (&user_options);
! }
! 
! /* Empty the saved file options vector.  */
! 
! void
! lto_clear_file_options (void)
! {
!   clear_options (&file_options);
! }
! 
! /* Concatenate the user options and any file options read from an LTO IL
!    file, and serialize them to STREAM.  File options precede user options
!    so that the latter override the former when reissued.  */
! 
! static void
! output_options (struct lto_output_stream *stream)
! {
!   VEC(opt_t, heap) *opts = concatenate_options (file_options, user_options);
!   const size_t length = VEC_length (opt_t, opts);
!   int i;
!   opt_t *o;
! 
!   output_data_stream (stream, &length, sizeof (length));
! 
!   FOR_EACH_VEC_ELT (opt_t, opts, i, o)
!     {
!       output_data_stream (stream, &o->type, sizeof (o->type));
!       output_data_stream (stream, &o->code, sizeof (o->code));
!       output_string_stream (stream, o->arg);
!       output_data_stream (stream, &o->value, sizeof (o->value));
!     }
! 
!   VEC_free (opt_t, heap, opts);
! }
  
  /* Write currently held options to an LTO IL section.  */
  
  void
  lto_write_options (void)
  {
-   char *const section_name = lto_get_section_name (LTO_section_opts, NULL, NULL);
    struct lto_output_stream stream;
!   struct lto_simple_header header;
!   struct lto_output_stream *header_stream;
! 
!   /* Targets and languages can provide defaults for -fexceptions but
!      we only process user options from the command-line.  Until we
!      serialize out a white list of options from the new global state
!      explicitly append important options as user options here.  */
!   if (flag_exceptions)
!     lto_register_user_option (OPT_fexceptions, NULL, 1, CL_COMMON);
! 
!   lto_begin_section (section_name, !flag_wpa);
!   free (section_name);
  
    memset (&stream, 0, sizeof (stream));
-   output_options (&stream);
- 
-   memset (&header, 0, sizeof (header));
-   header.lto_header.major_version = LTO_major_version;
-   header.lto_header.minor_version = LTO_minor_version;
-   header.lto_header.section_type = LTO_section_opts;
- 
-   header.compressed_size = 0;
-   header.main_size = stream.total_size;
- 
-   header_stream = ((struct lto_output_stream *)
- 		   xcalloc (1, sizeof (*header_stream)));
-   lto_output_data_stream (header_stream, &header, sizeof (header));
-   lto_write_stream (header_stream);
-   free (header_stream);
  
!   lto_write_stream (&stream);
!   lto_end_section ();
! }
! 
! /* Unserialize an options vector from IB, and append to file_options.  */
! 
! static void
! input_options (struct lto_input_block *ib)
! {
!   size_t length, i;
! 
!   lto_input_data_block (ib, &length, sizeof (length));
! 
!   for (i = 0; i < length; i++)
      {
!       opt_t o;
! 
!       lto_input_data_block (ib, &o.type, sizeof (o.type));
!       lto_input_data_block (ib, &o.code, sizeof (o.code));
!       o.arg = input_string_block (ib);
!       lto_input_data_block (ib, &o.value, sizeof (o.value));
!       VEC_safe_push (opt_t, heap, file_options, &o);
!     }
! }
! 
! /* Read options from an LTO IL section.  */
! 
! void
! lto_read_file_options (struct lto_file_decl_data *file_data)
! {
!   size_t len, l, skip;
!   const char *data, *p;
!   const struct lto_simple_header *header;
!   int32_t opts_offset;
!   struct lto_input_block ib;
! 
!   data = lto_get_section_data (file_data, LTO_section_opts, NULL, &len);
!   if (!data)
! 	  return;
! 
!   /* Option could be multiple sections merged (through ld -r) 
!      Keep reading all options.  This is ok right now because
!      the options just get mashed together anyways.
!      This will have to be done differently once lto-opts knows
!      how to associate options with different files. */
!   l = len;
!   p = data;
!   do 
!     { 
!       header = (const struct lto_simple_header *) p;
!       opts_offset = sizeof (*header);
! 
!       lto_check_version (header->lto_header.major_version,
! 			 header->lto_header.minor_version);
!       
!       LTO_INIT_INPUT_BLOCK (ib, p + opts_offset, 0, header->main_size);
!       input_options (&ib);
!       
!       skip = header->main_size + opts_offset;
!       l -= skip;
!       p += skip;
!     } 
!   while (l > 0);
! 
!   lto_free_section_data (file_data, LTO_section_opts, 0, data, len);
! }
  
! /* Concatenate the user options and any file options read from an LTO IL
!    file, and reissue them as if all had just been read in from the command
!    line.  As with serialization, file options precede user options.  */
  
! void
! lto_reissue_options (void)
! {
!   VEC(opt_t, heap) *opts = concatenate_options (file_options, user_options);
!   int i;
!   opt_t *o;
  
!   FOR_EACH_VEC_ELT (opt_t, opts, i, o)
!     {
!       void *flag_var = option_flag_var (o->code, &global_options);
  
!       if (flag_var)
! 	set_option (&global_options, &global_options_set,
! 		    o->code, o->value, o->arg,
! 		    DK_UNSPECIFIED, UNKNOWN_LOCATION, global_dc);
  
!       if (o->type == CL_TARGET)
  	{
! 	  struct cl_decoded_option decoded;
! 	  generate_option (o->code, o->arg, o->value, CL_TARGET, &decoded);
! 	  targetm_common.handle_option (&global_options, &global_options_set,
! 					&decoded, UNKNOWN_LOCATION);
  	}
-       else if (o->type == CL_COMMON)
- 	gcc_assert (flag_var);
-       else
- 	gcc_unreachable ();
      }
  
!   /* Flag_shlib is usually set by finish_options, but we are issuing flag_pic
!      too late.  */
!   if (flag_pic && !flag_pie)
!     flag_shlib = 1;
!   VEC_free (opt_t, heap, opts);
  }
--- 33,116 ----
  #include "common/common-target.h"
  #include "diagnostic.h"
  #include "lto-streamer.h"
! #include "toplev.h"
  
  /* Write currently held options to an LTO IL section.  */
  
  void
  lto_write_options (void)
  {
    struct lto_output_stream stream;
!   char *section_name;
!   struct obstack temporary_obstack;
!   unsigned int i, j;
!   char *args;
  
+   section_name = lto_get_section_name (LTO_section_opts, NULL, NULL);
+   lto_begin_section (section_name, false);
    memset (&stream, 0, sizeof (stream));
  
!   obstack_init (&temporary_obstack);
!   for (i = 1; i < save_decoded_options_count; ++i)
      {
!       struct cl_decoded_option *option = &save_decoded_options[i];
!       const char *q, *p;
  
!       /* Skip frontend and driver specific options here.  */
!       if (!(cl_options[option->opt_index].flags & (CL_COMMON|CL_TARGET|CL_LTO)))
! 	continue;
! 
!       /* Drop arguments created from the gcc driver that will be rejected
! 	 when passed on.  */
!       if (cl_options[option->opt_index].cl_reject_driver)
! 	continue;
  
!       /* Skip explicitly some common options that we do not need.  */
!       switch (option->opt_index)
! 	{
! 	case OPT_o:
! 	case OPT_dumpbase:
! 	case OPT_SPECIAL_input_file:
! 	  continue;
  
! 	default:
! 	  break;
! 	}
  
!       if (i != 1)
! 	obstack_grow (&temporary_obstack, " ", 1);
!       obstack_grow (&temporary_obstack, "'", 1);
!       q = option->canonical_option[0];
!       while ((p = strchr (q, '\'')))
! 	{
! 	  obstack_grow (&temporary_obstack, q, p - q);
! 	  obstack_grow (&temporary_obstack, "'\\''", 4);
! 	  q = ++p;
! 	}
!       obstack_grow (&temporary_obstack, q, strlen (q));
!       obstack_grow (&temporary_obstack, "'", 1);
  
!       for (j = 1; j < option->canonical_option_num_elements; ++j)
  	{
! 	  obstack_grow (&temporary_obstack, " '", 2);
! 	  q = option->canonical_option[j];
! 	  while ((p = strchr (q, '\'')))
! 	    {
! 	      obstack_grow (&temporary_obstack, q, p - q);
! 	      obstack_grow (&temporary_obstack, "'\\''", 4);
! 	      q = ++p;
! 	    }
! 	  obstack_grow (&temporary_obstack, q, strlen (q));
! 	  obstack_grow (&temporary_obstack, "'", 1);
  	}
      }
+   obstack_grow (&temporary_obstack, "\0", 1);
+   args = XOBFINISH (&temporary_obstack, char *);
+   lto_output_data_stream (&stream, args, strlen (args) + 1);
  
!   lto_write_stream (&stream);
!   lto_end_section ();
! 
!   obstack_free (&temporary_obstack, NULL);
!   free (section_name);
  }
Index: trunk/gcc/lto-streamer.h
===================================================================
*** trunk.orig/gcc/lto-streamer.h	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/lto-streamer.h	2011-10-28 11:51:24.000000000 +0200
*************** extern GTY(()) VEC(tree,gc) *lto_global_
*** 882,893 ****
  
  
  /* In lto-opts.c.  */
- extern void lto_register_user_option (size_t, const char *, int, unsigned int);
- extern void lto_read_file_options (struct lto_file_decl_data *);
  extern void lto_write_options (void);
- extern void lto_reissue_options (void);
- void lto_clear_user_options (void);
- void lto_clear_file_options (void);
  
  
  /* In lto-wpa-fixup.c  */
--- 882,888 ----
Index: trunk/gcc/lto/lto-lang.c
===================================================================
*** trunk.orig/gcc/lto/lto-lang.c	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/lto/lto-lang.c	2011-10-28 11:51:24.000000000 +0200
*************** lto_post_options (const char **pfilename
*** 692,699 ****
       support.  */
    flag_excess_precision_cmdline = EXCESS_PRECISION_FAST;
  
-   lto_read_all_file_options ();
- 
    /* Initialize the compiler back end.  */
    return false;
  }
--- 692,697 ----
Index: trunk/gcc/lto/lto.c
===================================================================
*** trunk.orig/gcc/lto/lto.c	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/lto/lto.c	2011-10-28 11:51:24.000000000 +0200
*************** lto_fixup_decls (struct lto_file_decl_da
*** 2494,2553 ****
      }
  }
  
- /* Read the options saved from each file in the command line.  Called
-    from lang_hooks.post_options which is called by process_options
-    right before all the options are used to initialize the compiler.
-    This assumes that decode_options has already run, so the
-    num_in_fnames and in_fnames are properly set.
- 
-    Note that this assumes that all the files had been compiled with
-    the same options, which is not a good assumption.  In general,
-    options ought to be read from all the files in the set and merged.
-    However, it is still unclear what the merge rules should be.  */
- 
- void
- lto_read_all_file_options (void)
- {
-   size_t i;
- 
-   /* Clear any file options currently saved.  */
-   lto_clear_file_options ();
- 
-   /* Set the hooks to read ELF sections.  */
-   lto_set_in_hooks (NULL, get_section_data, free_section_data);
-   if (!quiet_flag)
-     fprintf (stderr, "Reading command line options:");
- 
-   for (i = 0; i < num_in_fnames; i++)
-     {
-       struct lto_file_decl_data *file_data;
-       lto_file *file = lto_obj_file_open (in_fnames[i], false);
-       if (!file)
- 	break;
-       if (!quiet_flag)
- 	{
- 	  fprintf (stderr, " %s", in_fnames[i]);
- 	  fflush (stderr);
- 	}
- 
-       file_data = XCNEW (struct lto_file_decl_data);
-       file_data->file_name = file->filename;
-       file_data->section_hash_table = lto_obj_build_section_table (file, NULL);
- 
-       lto_read_file_options (file_data);
- 
-       lto_obj_file_close (file);
-       htab_delete (file_data->section_hash_table);
-       free (file_data);
-     }
- 
-   if (!quiet_flag)
-     fprintf (stderr, "\n");
- 
-   /* Apply globally the options read from all the files.  */
-   lto_reissue_options ();
- }
- 
  static GTY((length ("lto_stats.num_input_files + 1"))) struct lto_file_decl_data **all_file_decl_data;
  
  /* Turn file datas for sub files into a single array, so that they look
--- 2494,2499 ----
*************** lto_init (void)
*** 2921,2926 ****
--- 2867,2873 ----
    lto_process_name ();
    lto_streamer_hooks_init ();
    lto_reader_init ();
+   lto_set_in_hooks (NULL, get_section_data, free_section_data);
    memset (&lto_stats, 0, sizeof (lto_stats));
    bitmap_obstack_initialize (NULL);
    gimple_register_cfg_hooks ();
Index: trunk/gcc/opts-global.c
===================================================================
*** trunk.orig/gcc/opts-global.c	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/opts-global.c	2011-10-28 11:51:24.000000000 +0200
*************** unknown_option_callback (const struct cl
*** 160,178 ****
      return true;
  }
  
- /* Note that an option DECODED has been successfully handled with a
-    handler for mask MASK.  */
- 
- static void
- post_handling_callback (const struct cl_decoded_option *decoded ATTRIBUTE_UNUSED,
- 			unsigned int mask ATTRIBUTE_UNUSED)
- {
- #ifdef ENABLE_LTO
-   lto_register_user_option (decoded->opt_index, decoded->arg,
- 			    decoded->value, mask);
- #endif
- }
- 
  /* Handle a front-end option; arguments and return value as for
     handle_option.  */
  
--- 160,165 ----
*************** set_default_handlers (struct cl_option_h
*** 282,288 ****
  {
    handlers->unknown_option_callback = unknown_option_callback;
    handlers->wrong_lang_callback = complain_wrong_lang;
-   handlers->post_handling_callback = post_handling_callback;
    handlers->num_handlers = 3;
    handlers->handlers[0].handler = lang_handle_option;
    handlers->handlers[0].mask = initial_lang_mask;
--- 269,274 ----
*************** decode_options (struct gcc_options *opts
*** 314,324 ****
  				decoded_options, decoded_options_count,
  				loc, lang_mask, &handlers, dc);
  
- #ifdef ENABLE_LTO
-   /* Clear any options currently held for LTO.  */
-   lto_clear_user_options ();
- #endif
- 
    read_cmdline_options (opts, opts_set,
  			decoded_options, decoded_options_count,
  			loc, lang_mask,
--- 300,305 ----
Index: trunk/gcc/lto-wrapper.c
===================================================================
*** trunk.orig/gcc/lto-wrapper.c	2011-10-28 10:12:01.000000000 +0200
--- trunk/gcc/lto-wrapper.c	2011-10-28 12:15:35.000000000 +0200
*************** along with GCC; see the file COPYING3.
*** 45,50 ****
--- 45,58 ----
  #include "obstack.h"
  #include "opts.h"
  #include "options.h"
+ #include "simple-object.h"
+ 
+ /* From lto-streamer.h which we cannot include with -fkeep-inline-functions.
+    ???  Split out a lto-streamer-core.h.  */
+ 
+ #define LTO_SECTION_NAME_PREFIX         ".gnu.lto_"
+ 
+ /* End of lto-streamer.h copy.  */
  
  int debug;				/* true if -save-temps.  */
  int verbose;				/* true if -v.  */
*************** get_options_from_collect_gcc_options (co
*** 327,332 ****
--- 335,410 ----
    free (argv);
  }
  
+ /* Append OPTION to the options array DECODED_OPTIONS with size
+    DECODED_OPTIONS_COUNT if the option is not already present.  */
+ 
+ static void
+ append_option (struct cl_decoded_option **decoded_options,
+ 	       unsigned int *decoded_options_count,
+ 	       struct cl_decoded_option *option)
+ {
+   unsigned int i;
+ 
+   for (i = 0; i < *decoded_options_count; ++i)
+     if ((*decoded_options)[i].opt_index == option->opt_index)
+       return;
+ 
+   ++*decoded_options_count;
+   *decoded_options
+     = (struct cl_decoded_option *)
+ 	xrealloc (*decoded_options,
+ 		  (*decoded_options_count
+ 		   * sizeof (struct cl_decoded_option)));
+   memcpy (&(*decoded_options)[*decoded_options_count - 1], option,
+ 	  sizeof (struct cl_decoded_option));
+ }
+ 
+ /* Try to merge and complain about options FDECODED_OPTIONS when applied
+    ontop of DECODED_OPTIONS.  */
+ 
+ static void
+ merge_and_complain (struct cl_decoded_option **decoded_options,
+ 		    unsigned int *decoded_options_count,
+ 		    struct cl_decoded_option *fdecoded_options,
+ 		    unsigned int fdecoded_options_count)
+ {
+   unsigned int i;
+ 
+   /* ???  Merge options from files.  Most cases can be
+      handled by either unioning or intersecting
+      (for example -fwrapv is a case for unioning,
+      -ffast-math is for intersection).  Most complaints
+      about real conflicts between different options can
+      be deferred to the compiler proper.  Options that
+      we can neither safely handle by intersection nor
+      unioning would need to be complained about here.
+      Ideally we'd have a flag in the opt files that
+      tells whether to union or intersect or reject.
+      In absence of that it's unclear what a good default is.
+      It's also difficult to get positional handling correct.  */
+ 
+   /* The following does what the old LTO option code did,
+      union all target and a selected set of common options.  */
+   for (i = 0; i < fdecoded_options_count; ++i)
+     {
+       struct cl_decoded_option *foption = &fdecoded_options[i];
+       switch (foption->opt_index)
+ 	{
+ 	default:
+ 	  if (!(cl_options[foption->opt_index].flags & CL_TARGET))
+ 	    break;
+ 
+ 	  /* Fallthru.  */
+ 	case OPT_fPIC:
+ 	case OPT_fpic:
+ 	case OPT_fpie:
+ 	case OPT_fcommon:
+ 	case OPT_fexceptions:
+ 	  append_option (decoded_options, decoded_options_count, foption);
+ 	  break;
+ 	}
+     }
+ }
  
  /* Execute gcc. ARGC is the number of arguments. ARGV contains the arguments. */
  
*************** run_gcc (unsigned argc, char *argv[])
*** 342,347 ****
--- 420,427 ----
    int parallel = 0;
    int jobserver = 0;
    bool no_partition = false;
+   struct cl_decoded_option *fdecoded_options = NULL;
+   unsigned int fdecoded_options_count = 0;
    struct cl_decoded_option *decoded_options;
    unsigned int decoded_options_count;
    struct obstack argv_obstack;
*************** run_gcc (unsigned argc, char *argv[])
*** 359,369 ****
--- 439,563 ----
  					&decoded_options,
  					&decoded_options_count);
  
+   /* Look at saved options in the IL files.  */
+   for (i = 1; i < argc; ++i)
+     {
+       char *data, *p;
+       char *fopts;
+       int fd;
+       const char *errmsg;
+       int err;
+       off_t file_offset = 0, offset, length;
+       long loffset;
+       simple_object_read *sobj;
+       int consumed;
+       struct cl_decoded_option *f2decoded_options;
+       unsigned int f2decoded_options_count;
+       char *filename = argv[i];
+       if ((p = strrchr (argv[i], '@'))
+ 	  && p != argv[i] 
+ 	  && sscanf (p, "@%li%n", &loffset, &consumed) >= 1
+ 	  && strlen (p) == (unsigned int) consumed)
+ 	{
+ 	  filename = XNEWVEC (char, p - argv[i] + 1);
+ 	  memcpy (filename, argv[i], p - argv[i]);
+ 	  filename[p - argv[i]] = '\0';
+ 	  file_offset = (off_t) loffset;
+ 	}
+       fd = open (argv[i], O_RDONLY);
+       if (fd == -1)
+ 	continue;
+       sobj = simple_object_start_read (fd, file_offset, NULL, &errmsg, &err);
+       if (!sobj)
+ 	{
+ 	  close (fd);
+ 	  continue;
+ 	}
+       if (!simple_object_find_section (sobj, LTO_SECTION_NAME_PREFIX "." "opts",
+ 				       &offset, &length, &errmsg, &err))
+ 	{
+ 	  simple_object_release_read (sobj);
+ 	  close (fd);
+ 	  continue;
+ 	}
+       lseek (fd, file_offset + offset, SEEK_SET);
+       data = (char *)xmalloc (length);
+       read (fd, data, length);
+       fopts = data;
+       do
+ 	{
+ 	  get_options_from_collect_gcc_options (collect_gcc,
+ 						fopts, CL_LANG_ALL,
+ 						&f2decoded_options,
+ 						&f2decoded_options_count);
+ 	  if (!fdecoded_options)
+ 	    {
+ 	      fdecoded_options = f2decoded_options;
+ 	      fdecoded_options_count = f2decoded_options_count;
+ 	    }
+ 	  else
+ 	    merge_and_complain (&fdecoded_options,
+ 				&fdecoded_options_count,
+ 				f2decoded_options, f2decoded_options_count);
+ 
+ 	  fopts += strlen (fopts) + 1;
+ 	}
+       while (fopts - data < length);
+ 
+       free (data);
+       simple_object_release_read (sobj);
+       close (fd);
+     }
+ 
    /* Initalize the common arguments for the driver.  */
    obstack_init (&argv_obstack);
    obstack_ptr_grow (&argv_obstack, collect_gcc);
    obstack_ptr_grow (&argv_obstack, "-xlto");
    obstack_ptr_grow (&argv_obstack, "-c");
+ 
+   /* Append compiler driver arguments as far as they were merged.  */
+   for (j = 1; j < fdecoded_options_count; ++j)
+     {
+       struct cl_decoded_option *option = &fdecoded_options[j];
+ 
+       /* File options have been properly filtered by lto-opts.c.  */
+       switch (option->opt_index)
+ 	{
+ 	  /* Drop arguments that we want to take from the link line.  */
+ 	  case OPT_flto_:
+ 	  case OPT_flto:
+ 	  case OPT_flto_partition_none:
+ 	  case OPT_flto_partition_1to1:
+ 	  case OPT_flto_partition_balanced:
+ 	      continue;
+ 
+ 	  default:
+ 	      break;
+ 	}
+ 
+       /* For now do what the original LTO option code was doing - pass
+ 	 on any CL_TARGET flag and a few selected others.  */
+       switch (option->opt_index)
+ 	{
+ 	case OPT_fPIC:
+ 	case OPT_fpic:
+ 	case OPT_fpie:
+ 	case OPT_fcommon:
+ 	case OPT_fexceptions:
+ 	  break;
+ 
+ 	default:
+ 	  if (!(cl_options[option->opt_index].flags & CL_TARGET))
+ 	    continue;
+ 	}
+ 
+       /* Pass the option on.  */
+       for (i = 0; i < option->canonical_option_num_elements; ++i)
+ 	obstack_ptr_grow (&argv_obstack, option->canonical_option[i]);
+     }
+ 
+   /* Append linker driver arguments.  Compiler options from the linker
+      driver arguments will override / merge with those from the compiler.  */
    for (j = 1; j < decoded_options_count; ++j)
      {
        struct cl_decoded_option *option = &decoded_options[j];
Index: trunk/gcc/gcc.c
===================================================================
*** trunk.orig/gcc/gcc.c	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/gcc.c	2011-10-28 11:51:24.000000000 +0200
*************** driver_wrong_lang_callback (const struct
*** 3137,3152 ****
  		 &decoded->canonical_option[1], false);
  }
  
- /* Note that an option (index OPT_INDEX, argument ARG, value VALUE)
-    has been successfully handled with a handler for mask MASK.  */
- 
- static void
- driver_post_handling_callback (const struct cl_decoded_option *decoded ATTRIBUTE_UNUSED,
- 			       unsigned int mask ATTRIBUTE_UNUSED)
- {
-   /* Nothing to do here.  */
- }
- 
  static const char *spec_lang = 0;
  static int last_language_n_infiles;
  
--- 3137,3142 ----
*************** set_option_handlers (struct cl_option_ha
*** 3533,3539 ****
  {
    handlers->unknown_option_callback = driver_unknown_option_callback;
    handlers->wrong_lang_callback = driver_wrong_lang_callback;
-   handlers->post_handling_callback = driver_post_handling_callback;
    handlers->num_handlers = 3;
    handlers->handlers[0].handler = driver_handle_option;
    handlers->handlers[0].mask = CL_DRIVER;
--- 3523,3528 ----
Index: trunk/gcc/opts-common.c
===================================================================
*** trunk.orig/gcc/opts-common.c	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/opts-common.c	2011-10-28 11:51:24.000000000 +0200
*************** handle_option (struct gcc_options *opts,
*** 878,886 ****
  					    lang_mask, kind, loc,
  					    handlers, dc))
  	  return false;
- 	else
- 	  handlers->post_handling_callback (decoded,
- 					    handlers->handlers[i].mask);
        }
    
    return true;
--- 878,883 ----
Index: trunk/gcc/opts.h
===================================================================
*** trunk.orig/gcc/opts.h	2011-10-27 15:24:59.000000000 +0200
--- trunk/gcc/opts.h	2011-10-28 11:51:24.000000000 +0200
*************** struct cl_option_handlers
*** 291,300 ****
    void (*wrong_lang_callback) (const struct cl_decoded_option *decoded,
  			       unsigned int lang_mask);
  
-   /* Callback to call after the successful handling of any option.  */
-   void (*post_handling_callback) (const struct cl_decoded_option *decoded,
- 				  unsigned int mask);
- 
    /* The number of individual handlers.  */
    size_t num_handlers;
  
--- 291,296 ----

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Diego Novillo]

Isn't this the same patch as
http://gcc.gnu.org/ml/gcc-patches/2011-10/msg02348.html?


Diego.

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Joseph S. Myers]

On Fri, 28 Oct 2011, Richard Guenther wrote:

> + 	  /* Fallthru.  */
> + 	case OPT_fPIC:
> + 	case OPT_fpic:
> + 	case OPT_fpie:
> + 	case OPT_fcommon:
> + 	case OPT_fexceptions:
> + 	  append_option (decoded_options, decoded_options_count, foption);
> + 	  break;

No doubt this is what the previous code did, but in this case shouldn't 
"union" mean the biggest PIC status of any file wins (thus, if -fPIC was 
the PIC option that actually had effect on some object, that wins over an 
explicit -fno-PIC or -fpic on another object)?  In general whether the 
options are positive or negative matters, and I don't see that handled 
here.

(Actually, maybe the smallest PIC status should win - i.e. if any object 
is not PIC then the final code can be presumed to be non-PIC.)

(Using Negative in .opt files for groups of options such as -fPIC/-fpic 
would ensure that at most one survives from any one object, but you still 
need to work out what you want to do for merging.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Jack Howarth]

On Fri, Oct 28, 2011 at 01:43:07PM +0200, Richard Guenther wrote:
> 
> This moves the existing processing of user options from lto1 to
> the lto driver (lto-wrapper).  It also changes the way we stream
> user options from some custom binary format over to simply
> streaming the original command-line as passed to the compiler
> by the driver as a COLLECT_GCC_OPTIONS-like string.
> 
> lto-wrapper in this patch version (tries to) performs exactly
> the same as lto1 tried to do - re-issue all target specific
> options and a selected set of switches.
> 
> I chose to do this as an incremental step which should not
> change behavior or regress in any way (fingers crossing).
> 
> >From here we can think what would be the most sensible behavior
> (and maybe start tagging options in the .opt files so they
> get treatment based on some flag).
> 
> LTO bootstrapped and tested on x86_64-unknown-linux-gnu, ok?

LTO bootstraps on x86_64-apple-darwin11 applied to r180613 with no
regressions in lto.exp.
                  Jack

> 
> Thanks,
> Richard.
> 
> 2011-10-28  Richard Guenther  <rguenther@suse.de>
> 
> 	* lto-opts.c: Re-implement.
> 	* lto-streamer.h (lto_register_user_option): Remove.
> 	(lto_read_file_options): Likewise.
> 	(lto_reissue_options): Likewise.
> 	(lto_clear_user_options): Likewise.
> 	(lto_clear_file_options): Likewise.
> 	* opts-global.c (post_handling_callback): Remove.
> 	(set_default_handlers): Do not set post_handling_callback.
> 	(decode_options): Remove LTO specific code.
> 	* lto-wrapper.c (merge_and_complain): New function.
> 	(run_gcc): Read all input file options and
> 	prepend a merged set before the linker driver options.
> 	* gcc.c (driver_post_handling_callback): Remove.
> 	(set_option_handlers): Do not set post_handling_callback.
> 	* opts-common.c (handle_option): Do not call post_handling_callback.
> 	* opts.h (struct cl_option_handlers): Remove post_handling_callback.
> 
> 	lto/
> 	* lto-lang.c (lto_post_options): Do not read file options.
> 	* lto.c (lto_read_all_file_options): Remove.
> 	(lto_init): Call lto_set_in_hooks here.
> 
> 
> Index: trunk/gcc/lto-opts.c
> ===================================================================
> *** trunk.orig/gcc/lto-opts.c	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/lto-opts.c	2011-10-28 12:15:52.000000000 +0200
> ***************
> *** 1,6 ****
>   /* LTO IL options.
>   
> !    Copyright 2009, 2010, 2011 Free Software Foundation, Inc.
>      Contributed by Simon Baldwin <simonb@google.com>
>   
>   This file is part of GCC.
> --- 1,6 ----
>   /* LTO IL options.
>   
> !    Copyright 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
>      Contributed by Simon Baldwin <simonb@google.com>
>   
>   This file is part of GCC.
> *************** along with GCC; see the file COPYING3.
> *** 33,422 ****
>   #include "common/common-target.h"
>   #include "diagnostic.h"
>   #include "lto-streamer.h"
> ! 
> ! /* When a file is initially compiled, the options used when generating
> !    the IL are not necessarily the same as those used when linking the
> !    objects into the final executable.  In general, most build systems
> !    will proceed with something along the lines of:
> ! 
> !    	$ gcc <cc-flags> -flto -c f1.c -o f1.o
> ! 	$ gcc <cc-flags> -flto -c f2.c -o f2.o
> ! 	...
> ! 	$ gcc <cc-flags> -flto -c fN.c -o fN.o
> ! 
> !    And the final link may or may not include the same <cc-flags> used
> !    to generate the initial object files:
> ! 
> !    	$ gcc <ld-flags> -flto -o prog f1.o ... fN.o
> ! 
> !    Since we will be generating final code during the link step, some
> !    of the flags used during the compile step need to be re-applied
> !    during the link step.  For instance, flags in the -m family.
> ! 
> !    The idea is to save a selected set of <cc-flags> in a special
> !    section of the initial object files.  This section is then read
> !    during linking and the options re-applied.
> ! 
> !    FIXME lto.  Currently the scheme is limited in that only the
> !    options saved on the first object file (f1.o) are read back during
> !    the link step.  This means that the options used to compile f1.o
> !    will be applied to ALL the object files in the final link step.
> !    More work needs to be done to implement a merging and validation
> !    mechanism, as this will not be enough for all cases.  */
> ! 
> ! /* Saved options hold the type of the option (currently CL_TARGET or
> !    CL_COMMON), and the code, argument, and value.  */
> ! 
> ! typedef struct GTY(()) opt_d
> ! {
> !   unsigned int type;
> !   size_t code;
> !   char *arg;
> !   int value;
> ! } opt_t;
> ! 
> ! DEF_VEC_O (opt_t);
> ! DEF_VEC_ALLOC_O (opt_t, heap);
> ! 
> ! 
> ! /* Options are held in two vectors, one for those registered by
> !    command line handling code, and the other for those read in from
> !    any LTO IL input.  */
> ! static VEC(opt_t, heap) *user_options = NULL;
> ! static VEC(opt_t, heap) *file_options = NULL;
> ! 
> ! /* Iterate FROM in reverse, writing option codes not yet in CODES into *TO.
> !    Mark each new option code encountered in CODES.  */
> ! 
> ! static void
> ! reverse_iterate_options (VEC(opt_t, heap) *from, VEC(opt_t, heap) **to,
> ! 			 bitmap codes)
> ! {
> !   int i;
> ! 
> !   for (i = VEC_length (opt_t, from); i > 0; i--)
> !     {
> !       const opt_t *const o = VEC_index (opt_t, from, i - 1);
> ! 
> !       if (bitmap_set_bit (codes, o->code))
> ! 	VEC_safe_push (opt_t, heap, *to, o);
> !     }
> ! }
> ! 
> ! /* Concatenate options vectors FIRST and SECOND, rationalize so that only the
> !    final of any given option remains, and return the result.  */
> ! 
> ! static VEC(opt_t, heap) *
> ! concatenate_options (VEC(opt_t, heap) *first, VEC(opt_t, heap) *second)
> ! {
> !   VEC(opt_t, heap) *results = NULL;
> !   bitmap codes = lto_bitmap_alloc ();
> ! 
> !   reverse_iterate_options (second, &results, codes);
> !   reverse_iterate_options (first, &results, codes);
> ! 
> !   lto_bitmap_free (codes);
> !   return results;
> ! }
> ! 
> ! /* Clear the options vector in *OPTS_P and set it to NULL.  */
> ! 
> ! static void
> ! clear_options (VEC(opt_t, heap) **opts_p)
> ! {
> !   int i;
> !   opt_t *o;
> ! 
> !   FOR_EACH_VEC_ELT (opt_t, *opts_p, i, o)
> !     free (o->arg);
> ! 
> !   VEC_free (opt_t, heap, *opts_p);
> ! }
> ! 
> ! /* Write LENGTH bytes from ADDR to STREAM.  */
> ! 
> ! static void
> ! output_data_stream (struct lto_output_stream *stream,
> !                     const void *addr, size_t length)
> ! {
> !   lto_output_data_stream (stream, addr, length);
> ! }
> ! 
> ! /* Write string STRING to STREAM.  */
> ! 
> ! static void
> ! output_string_stream (struct lto_output_stream *stream, const char *string)
> ! {
> !   bool flag = false;
> ! 
> !   if (string != NULL)
> !     {
> !       const size_t length = strlen (string);
> ! 
> !       flag = true;
> !       output_data_stream (stream, &flag, sizeof (flag));
> !       output_data_stream (stream, &length, sizeof (length));
> !       output_data_stream (stream, string, length);
> !     }
> !   else
> !     output_data_stream (stream, &flag, sizeof (flag));
> ! }
> ! 
> ! /* Return a string from IB.  The string is allocated, and the caller is
> !    responsible for freeing it.  */
> ! 
> ! static char *
> ! input_string_block (struct lto_input_block *ib)
> ! {
> !   bool flag;
> ! 
> !   lto_input_data_block (ib, &flag, sizeof (flag));
> !   if (flag)
> !     {
> !       size_t length;
> !       char *string;
> ! 
> !       lto_input_data_block (ib, &length, sizeof (length));
> !       string = (char *) xcalloc (1, length + 1);
> !       lto_input_data_block (ib, string, length);
> ! 
> !       return string;
> !     }
> !   else
> !     return NULL;
> ! }
> ! 
> ! /* Return true if this option is one we need to save in LTO output files.
> !    At present, we pass along all target options, and common options that
> !    involve position independent code.
> ! 
> !    TODO This list of options requires expansion and rationalization.
> !    Among others, optimization options may well be appropriate here.  */
> ! 
> ! static bool
> ! register_user_option_p (size_t code, unsigned int type)
> ! {
> !   if (type == CL_TARGET)
> !     return true;
> !   else if (type == CL_COMMON)
> !     {
> !       return (code == OPT_fPIC
> ! 	      || code == OPT_fpic
> ! 	      || code == OPT_fPIE
> ! 	      || code == OPT_fpie
> ! 	      || code == OPT_fcommon
> ! 	      || code == OPT_fexceptions);
> !     }
> ! 
> !   return false;
> ! }
> ! 
> ! /* Note command line option with the given TYPE and CODE, ARG, and VALUE.
> !    If relevant to LTO, save it in the user options vector.  */
> ! 
> ! void
> ! lto_register_user_option (size_t code, const char *arg, int value,
> ! 			  unsigned int type)
> ! {
> !   if (register_user_option_p (code, type))
> !     {
> !       opt_t o;
> ! 
> !       o.type = type;
> !       o.code = code;
> !       if (arg != NULL)
> ! 	{
> ! 	  o.arg = (char *) xmalloc (strlen (arg) + 1);
> ! 	  strcpy (o.arg, arg);
> ! 	}
> !       else
> ! 	o.arg = NULL;
> !       o.value = value;
> !       VEC_safe_push (opt_t, heap, user_options, &o);
> !     }
> ! }
> ! 
> ! /* Empty the saved user options vector.  */
> ! 
> ! void
> ! lto_clear_user_options (void)
> ! {
> !   clear_options (&user_options);
> ! }
> ! 
> ! /* Empty the saved file options vector.  */
> ! 
> ! void
> ! lto_clear_file_options (void)
> ! {
> !   clear_options (&file_options);
> ! }
> ! 
> ! /* Concatenate the user options and any file options read from an LTO IL
> !    file, and serialize them to STREAM.  File options precede user options
> !    so that the latter override the former when reissued.  */
> ! 
> ! static void
> ! output_options (struct lto_output_stream *stream)
> ! {
> !   VEC(opt_t, heap) *opts = concatenate_options (file_options, user_options);
> !   const size_t length = VEC_length (opt_t, opts);
> !   int i;
> !   opt_t *o;
> ! 
> !   output_data_stream (stream, &length, sizeof (length));
> ! 
> !   FOR_EACH_VEC_ELT (opt_t, opts, i, o)
> !     {
> !       output_data_stream (stream, &o->type, sizeof (o->type));
> !       output_data_stream (stream, &o->code, sizeof (o->code));
> !       output_string_stream (stream, o->arg);
> !       output_data_stream (stream, &o->value, sizeof (o->value));
> !     }
> ! 
> !   VEC_free (opt_t, heap, opts);
> ! }
>   
>   /* Write currently held options to an LTO IL section.  */
>   
>   void
>   lto_write_options (void)
>   {
> -   char *const section_name = lto_get_section_name (LTO_section_opts, NULL, NULL);
>     struct lto_output_stream stream;
> !   struct lto_simple_header header;
> !   struct lto_output_stream *header_stream;
> ! 
> !   /* Targets and languages can provide defaults for -fexceptions but
> !      we only process user options from the command-line.  Until we
> !      serialize out a white list of options from the new global state
> !      explicitly append important options as user options here.  */
> !   if (flag_exceptions)
> !     lto_register_user_option (OPT_fexceptions, NULL, 1, CL_COMMON);
> ! 
> !   lto_begin_section (section_name, !flag_wpa);
> !   free (section_name);
>   
>     memset (&stream, 0, sizeof (stream));
> -   output_options (&stream);
> - 
> -   memset (&header, 0, sizeof (header));
> -   header.lto_header.major_version = LTO_major_version;
> -   header.lto_header.minor_version = LTO_minor_version;
> -   header.lto_header.section_type = LTO_section_opts;
> - 
> -   header.compressed_size = 0;
> -   header.main_size = stream.total_size;
> - 
> -   header_stream = ((struct lto_output_stream *)
> - 		   xcalloc (1, sizeof (*header_stream)));
> -   lto_output_data_stream (header_stream, &header, sizeof (header));
> -   lto_write_stream (header_stream);
> -   free (header_stream);
>   
> !   lto_write_stream (&stream);
> !   lto_end_section ();
> ! }
> ! 
> ! /* Unserialize an options vector from IB, and append to file_options.  */
> ! 
> ! static void
> ! input_options (struct lto_input_block *ib)
> ! {
> !   size_t length, i;
> ! 
> !   lto_input_data_block (ib, &length, sizeof (length));
> ! 
> !   for (i = 0; i < length; i++)
>       {
> !       opt_t o;
> ! 
> !       lto_input_data_block (ib, &o.type, sizeof (o.type));
> !       lto_input_data_block (ib, &o.code, sizeof (o.code));
> !       o.arg = input_string_block (ib);
> !       lto_input_data_block (ib, &o.value, sizeof (o.value));
> !       VEC_safe_push (opt_t, heap, file_options, &o);
> !     }
> ! }
> ! 
> ! /* Read options from an LTO IL section.  */
> ! 
> ! void
> ! lto_read_file_options (struct lto_file_decl_data *file_data)
> ! {
> !   size_t len, l, skip;
> !   const char *data, *p;
> !   const struct lto_simple_header *header;
> !   int32_t opts_offset;
> !   struct lto_input_block ib;
> ! 
> !   data = lto_get_section_data (file_data, LTO_section_opts, NULL, &len);
> !   if (!data)
> ! 	  return;
> ! 
> !   /* Option could be multiple sections merged (through ld -r) 
> !      Keep reading all options.  This is ok right now because
> !      the options just get mashed together anyways.
> !      This will have to be done differently once lto-opts knows
> !      how to associate options with different files. */
> !   l = len;
> !   p = data;
> !   do 
> !     { 
> !       header = (const struct lto_simple_header *) p;
> !       opts_offset = sizeof (*header);
> ! 
> !       lto_check_version (header->lto_header.major_version,
> ! 			 header->lto_header.minor_version);
> !       
> !       LTO_INIT_INPUT_BLOCK (ib, p + opts_offset, 0, header->main_size);
> !       input_options (&ib);
> !       
> !       skip = header->main_size + opts_offset;
> !       l -= skip;
> !       p += skip;
> !     } 
> !   while (l > 0);
> ! 
> !   lto_free_section_data (file_data, LTO_section_opts, 0, data, len);
> ! }
>   
> ! /* Concatenate the user options and any file options read from an LTO IL
> !    file, and reissue them as if all had just been read in from the command
> !    line.  As with serialization, file options precede user options.  */
>   
> ! void
> ! lto_reissue_options (void)
> ! {
> !   VEC(opt_t, heap) *opts = concatenate_options (file_options, user_options);
> !   int i;
> !   opt_t *o;
>   
> !   FOR_EACH_VEC_ELT (opt_t, opts, i, o)
> !     {
> !       void *flag_var = option_flag_var (o->code, &global_options);
>   
> !       if (flag_var)
> ! 	set_option (&global_options, &global_options_set,
> ! 		    o->code, o->value, o->arg,
> ! 		    DK_UNSPECIFIED, UNKNOWN_LOCATION, global_dc);
>   
> !       if (o->type == CL_TARGET)
>   	{
> ! 	  struct cl_decoded_option decoded;
> ! 	  generate_option (o->code, o->arg, o->value, CL_TARGET, &decoded);
> ! 	  targetm_common.handle_option (&global_options, &global_options_set,
> ! 					&decoded, UNKNOWN_LOCATION);
>   	}
> -       else if (o->type == CL_COMMON)
> - 	gcc_assert (flag_var);
> -       else
> - 	gcc_unreachable ();
>       }
>   
> !   /* Flag_shlib is usually set by finish_options, but we are issuing flag_pic
> !      too late.  */
> !   if (flag_pic && !flag_pie)
> !     flag_shlib = 1;
> !   VEC_free (opt_t, heap, opts);
>   }
> --- 33,116 ----
>   #include "common/common-target.h"
>   #include "diagnostic.h"
>   #include "lto-streamer.h"
> ! #include "toplev.h"
>   
>   /* Write currently held options to an LTO IL section.  */
>   
>   void
>   lto_write_options (void)
>   {
>     struct lto_output_stream stream;
> !   char *section_name;
> !   struct obstack temporary_obstack;
> !   unsigned int i, j;
> !   char *args;
>   
> +   section_name = lto_get_section_name (LTO_section_opts, NULL, NULL);
> +   lto_begin_section (section_name, false);
>     memset (&stream, 0, sizeof (stream));
>   
> !   obstack_init (&temporary_obstack);
> !   for (i = 1; i < save_decoded_options_count; ++i)
>       {
> !       struct cl_decoded_option *option = &save_decoded_options[i];
> !       const char *q, *p;
>   
> !       /* Skip frontend and driver specific options here.  */
> !       if (!(cl_options[option->opt_index].flags & (CL_COMMON|CL_TARGET|CL_LTO)))
> ! 	continue;
> ! 
> !       /* Drop arguments created from the gcc driver that will be rejected
> ! 	 when passed on.  */
> !       if (cl_options[option->opt_index].cl_reject_driver)
> ! 	continue;
>   
> !       /* Skip explicitly some common options that we do not need.  */
> !       switch (option->opt_index)
> ! 	{
> ! 	case OPT_o:
> ! 	case OPT_dumpbase:
> ! 	case OPT_SPECIAL_input_file:
> ! 	  continue;
>   
> ! 	default:
> ! 	  break;
> ! 	}
>   
> !       if (i != 1)
> ! 	obstack_grow (&temporary_obstack, " ", 1);
> !       obstack_grow (&temporary_obstack, "'", 1);
> !       q = option->canonical_option[0];
> !       while ((p = strchr (q, '\'')))
> ! 	{
> ! 	  obstack_grow (&temporary_obstack, q, p - q);
> ! 	  obstack_grow (&temporary_obstack, "'\\''", 4);
> ! 	  q = ++p;
> ! 	}
> !       obstack_grow (&temporary_obstack, q, strlen (q));
> !       obstack_grow (&temporary_obstack, "'", 1);
>   
> !       for (j = 1; j < option->canonical_option_num_elements; ++j)
>   	{
> ! 	  obstack_grow (&temporary_obstack, " '", 2);
> ! 	  q = option->canonical_option[j];
> ! 	  while ((p = strchr (q, '\'')))
> ! 	    {
> ! 	      obstack_grow (&temporary_obstack, q, p - q);
> ! 	      obstack_grow (&temporary_obstack, "'\\''", 4);
> ! 	      q = ++p;
> ! 	    }
> ! 	  obstack_grow (&temporary_obstack, q, strlen (q));
> ! 	  obstack_grow (&temporary_obstack, "'", 1);
>   	}
>       }
> +   obstack_grow (&temporary_obstack, "\0", 1);
> +   args = XOBFINISH (&temporary_obstack, char *);
> +   lto_output_data_stream (&stream, args, strlen (args) + 1);
>   
> !   lto_write_stream (&stream);
> !   lto_end_section ();
> ! 
> !   obstack_free (&temporary_obstack, NULL);
> !   free (section_name);
>   }
> Index: trunk/gcc/lto-streamer.h
> ===================================================================
> *** trunk.orig/gcc/lto-streamer.h	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/lto-streamer.h	2011-10-28 11:51:24.000000000 +0200
> *************** extern GTY(()) VEC(tree,gc) *lto_global_
> *** 882,893 ****
>   
>   
>   /* In lto-opts.c.  */
> - extern void lto_register_user_option (size_t, const char *, int, unsigned int);
> - extern void lto_read_file_options (struct lto_file_decl_data *);
>   extern void lto_write_options (void);
> - extern void lto_reissue_options (void);
> - void lto_clear_user_options (void);
> - void lto_clear_file_options (void);
>   
>   
>   /* In lto-wpa-fixup.c  */
> --- 882,888 ----
> Index: trunk/gcc/lto/lto-lang.c
> ===================================================================
> *** trunk.orig/gcc/lto/lto-lang.c	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/lto/lto-lang.c	2011-10-28 11:51:24.000000000 +0200
> *************** lto_post_options (const char **pfilename
> *** 692,699 ****
>        support.  */
>     flag_excess_precision_cmdline = EXCESS_PRECISION_FAST;
>   
> -   lto_read_all_file_options ();
> - 
>     /* Initialize the compiler back end.  */
>     return false;
>   }
> --- 692,697 ----
> Index: trunk/gcc/lto/lto.c
> ===================================================================
> *** trunk.orig/gcc/lto/lto.c	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/lto/lto.c	2011-10-28 11:51:24.000000000 +0200
> *************** lto_fixup_decls (struct lto_file_decl_da
> *** 2494,2553 ****
>       }
>   }
>   
> - /* Read the options saved from each file in the command line.  Called
> -    from lang_hooks.post_options which is called by process_options
> -    right before all the options are used to initialize the compiler.
> -    This assumes that decode_options has already run, so the
> -    num_in_fnames and in_fnames are properly set.
> - 
> -    Note that this assumes that all the files had been compiled with
> -    the same options, which is not a good assumption.  In general,
> -    options ought to be read from all the files in the set and merged.
> -    However, it is still unclear what the merge rules should be.  */
> - 
> - void
> - lto_read_all_file_options (void)
> - {
> -   size_t i;
> - 
> -   /* Clear any file options currently saved.  */
> -   lto_clear_file_options ();
> - 
> -   /* Set the hooks to read ELF sections.  */
> -   lto_set_in_hooks (NULL, get_section_data, free_section_data);
> -   if (!quiet_flag)
> -     fprintf (stderr, "Reading command line options:");
> - 
> -   for (i = 0; i < num_in_fnames; i++)
> -     {
> -       struct lto_file_decl_data *file_data;
> -       lto_file *file = lto_obj_file_open (in_fnames[i], false);
> -       if (!file)
> - 	break;
> -       if (!quiet_flag)
> - 	{
> - 	  fprintf (stderr, " %s", in_fnames[i]);
> - 	  fflush (stderr);
> - 	}
> - 
> -       file_data = XCNEW (struct lto_file_decl_data);
> -       file_data->file_name = file->filename;
> -       file_data->section_hash_table = lto_obj_build_section_table (file, NULL);
> - 
> -       lto_read_file_options (file_data);
> - 
> -       lto_obj_file_close (file);
> -       htab_delete (file_data->section_hash_table);
> -       free (file_data);
> -     }
> - 
> -   if (!quiet_flag)
> -     fprintf (stderr, "\n");
> - 
> -   /* Apply globally the options read from all the files.  */
> -   lto_reissue_options ();
> - }
> - 
>   static GTY((length ("lto_stats.num_input_files + 1"))) struct lto_file_decl_data **all_file_decl_data;
>   
>   /* Turn file datas for sub files into a single array, so that they look
> --- 2494,2499 ----
> *************** lto_init (void)
> *** 2921,2926 ****
> --- 2867,2873 ----
>     lto_process_name ();
>     lto_streamer_hooks_init ();
>     lto_reader_init ();
> +   lto_set_in_hooks (NULL, get_section_data, free_section_data);
>     memset (&lto_stats, 0, sizeof (lto_stats));
>     bitmap_obstack_initialize (NULL);
>     gimple_register_cfg_hooks ();
> Index: trunk/gcc/opts-global.c
> ===================================================================
> *** trunk.orig/gcc/opts-global.c	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/opts-global.c	2011-10-28 11:51:24.000000000 +0200
> *************** unknown_option_callback (const struct cl
> *** 160,178 ****
>       return true;
>   }
>   
> - /* Note that an option DECODED has been successfully handled with a
> -    handler for mask MASK.  */
> - 
> - static void
> - post_handling_callback (const struct cl_decoded_option *decoded ATTRIBUTE_UNUSED,
> - 			unsigned int mask ATTRIBUTE_UNUSED)
> - {
> - #ifdef ENABLE_LTO
> -   lto_register_user_option (decoded->opt_index, decoded->arg,
> - 			    decoded->value, mask);
> - #endif
> - }
> - 
>   /* Handle a front-end option; arguments and return value as for
>      handle_option.  */
>   
> --- 160,165 ----
> *************** set_default_handlers (struct cl_option_h
> *** 282,288 ****
>   {
>     handlers->unknown_option_callback = unknown_option_callback;
>     handlers->wrong_lang_callback = complain_wrong_lang;
> -   handlers->post_handling_callback = post_handling_callback;
>     handlers->num_handlers = 3;
>     handlers->handlers[0].handler = lang_handle_option;
>     handlers->handlers[0].mask = initial_lang_mask;
> --- 269,274 ----
> *************** decode_options (struct gcc_options *opts
> *** 314,324 ****
>   				decoded_options, decoded_options_count,
>   				loc, lang_mask, &handlers, dc);
>   
> - #ifdef ENABLE_LTO
> -   /* Clear any options currently held for LTO.  */
> -   lto_clear_user_options ();
> - #endif
> - 
>     read_cmdline_options (opts, opts_set,
>   			decoded_options, decoded_options_count,
>   			loc, lang_mask,
> --- 300,305 ----
> Index: trunk/gcc/lto-wrapper.c
> ===================================================================
> *** trunk.orig/gcc/lto-wrapper.c	2011-10-28 10:12:01.000000000 +0200
> --- trunk/gcc/lto-wrapper.c	2011-10-28 12:15:35.000000000 +0200
> *************** along with GCC; see the file COPYING3.
> *** 45,50 ****
> --- 45,58 ----
>   #include "obstack.h"
>   #include "opts.h"
>   #include "options.h"
> + #include "simple-object.h"
> + 
> + /* From lto-streamer.h which we cannot include with -fkeep-inline-functions.
> +    ???  Split out a lto-streamer-core.h.  */
> + 
> + #define LTO_SECTION_NAME_PREFIX         ".gnu.lto_"
> + 
> + /* End of lto-streamer.h copy.  */
>   
>   int debug;				/* true if -save-temps.  */
>   int verbose;				/* true if -v.  */
> *************** get_options_from_collect_gcc_options (co
> *** 327,332 ****
> --- 335,410 ----
>     free (argv);
>   }
>   
> + /* Append OPTION to the options array DECODED_OPTIONS with size
> +    DECODED_OPTIONS_COUNT if the option is not already present.  */
> + 
> + static void
> + append_option (struct cl_decoded_option **decoded_options,
> + 	       unsigned int *decoded_options_count,
> + 	       struct cl_decoded_option *option)
> + {
> +   unsigned int i;
> + 
> +   for (i = 0; i < *decoded_options_count; ++i)
> +     if ((*decoded_options)[i].opt_index == option->opt_index)
> +       return;
> + 
> +   ++*decoded_options_count;
> +   *decoded_options
> +     = (struct cl_decoded_option *)
> + 	xrealloc (*decoded_options,
> + 		  (*decoded_options_count
> + 		   * sizeof (struct cl_decoded_option)));
> +   memcpy (&(*decoded_options)[*decoded_options_count - 1], option,
> + 	  sizeof (struct cl_decoded_option));
> + }
> + 
> + /* Try to merge and complain about options FDECODED_OPTIONS when applied
> +    ontop of DECODED_OPTIONS.  */
> + 
> + static void
> + merge_and_complain (struct cl_decoded_option **decoded_options,
> + 		    unsigned int *decoded_options_count,
> + 		    struct cl_decoded_option *fdecoded_options,
> + 		    unsigned int fdecoded_options_count)
> + {
> +   unsigned int i;
> + 
> +   /* ???  Merge options from files.  Most cases can be
> +      handled by either unioning or intersecting
> +      (for example -fwrapv is a case for unioning,
> +      -ffast-math is for intersection).  Most complaints
> +      about real conflicts between different options can
> +      be deferred to the compiler proper.  Options that
> +      we can neither safely handle by intersection nor
> +      unioning would need to be complained about here.
> +      Ideally we'd have a flag in the opt files that
> +      tells whether to union or intersect or reject.
> +      In absence of that it's unclear what a good default is.
> +      It's also difficult to get positional handling correct.  */
> + 
> +   /* The following does what the old LTO option code did,
> +      union all target and a selected set of common options.  */
> +   for (i = 0; i < fdecoded_options_count; ++i)
> +     {
> +       struct cl_decoded_option *foption = &fdecoded_options[i];
> +       switch (foption->opt_index)
> + 	{
> + 	default:
> + 	  if (!(cl_options[foption->opt_index].flags & CL_TARGET))
> + 	    break;
> + 
> + 	  /* Fallthru.  */
> + 	case OPT_fPIC:
> + 	case OPT_fpic:
> + 	case OPT_fpie:
> + 	case OPT_fcommon:
> + 	case OPT_fexceptions:
> + 	  append_option (decoded_options, decoded_options_count, foption);
> + 	  break;
> + 	}
> +     }
> + }
>   
>   /* Execute gcc. ARGC is the number of arguments. ARGV contains the arguments. */
>   
> *************** run_gcc (unsigned argc, char *argv[])
> *** 342,347 ****
> --- 420,427 ----
>     int parallel = 0;
>     int jobserver = 0;
>     bool no_partition = false;
> +   struct cl_decoded_option *fdecoded_options = NULL;
> +   unsigned int fdecoded_options_count = 0;
>     struct cl_decoded_option *decoded_options;
>     unsigned int decoded_options_count;
>     struct obstack argv_obstack;
> *************** run_gcc (unsigned argc, char *argv[])
> *** 359,369 ****
> --- 439,563 ----
>   					&decoded_options,
>   					&decoded_options_count);
>   
> +   /* Look at saved options in the IL files.  */
> +   for (i = 1; i < argc; ++i)
> +     {
> +       char *data, *p;
> +       char *fopts;
> +       int fd;
> +       const char *errmsg;
> +       int err;
> +       off_t file_offset = 0, offset, length;
> +       long loffset;
> +       simple_object_read *sobj;
> +       int consumed;
> +       struct cl_decoded_option *f2decoded_options;
> +       unsigned int f2decoded_options_count;
> +       char *filename = argv[i];
> +       if ((p = strrchr (argv[i], '@'))
> + 	  && p != argv[i] 
> + 	  && sscanf (p, "@%li%n", &loffset, &consumed) >= 1
> + 	  && strlen (p) == (unsigned int) consumed)
> + 	{
> + 	  filename = XNEWVEC (char, p - argv[i] + 1);
> + 	  memcpy (filename, argv[i], p - argv[i]);
> + 	  filename[p - argv[i]] = '\0';
> + 	  file_offset = (off_t) loffset;
> + 	}
> +       fd = open (argv[i], O_RDONLY);
> +       if (fd == -1)
> + 	continue;
> +       sobj = simple_object_start_read (fd, file_offset, NULL, &errmsg, &err);
> +       if (!sobj)
> + 	{
> + 	  close (fd);
> + 	  continue;
> + 	}
> +       if (!simple_object_find_section (sobj, LTO_SECTION_NAME_PREFIX "." "opts",
> + 				       &offset, &length, &errmsg, &err))
> + 	{
> + 	  simple_object_release_read (sobj);
> + 	  close (fd);
> + 	  continue;
> + 	}
> +       lseek (fd, file_offset + offset, SEEK_SET);
> +       data = (char *)xmalloc (length);
> +       read (fd, data, length);
> +       fopts = data;
> +       do
> + 	{
> + 	  get_options_from_collect_gcc_options (collect_gcc,
> + 						fopts, CL_LANG_ALL,
> + 						&f2decoded_options,
> + 						&f2decoded_options_count);
> + 	  if (!fdecoded_options)
> + 	    {
> + 	      fdecoded_options = f2decoded_options;
> + 	      fdecoded_options_count = f2decoded_options_count;
> + 	    }
> + 	  else
> + 	    merge_and_complain (&fdecoded_options,
> + 				&fdecoded_options_count,
> + 				f2decoded_options, f2decoded_options_count);
> + 
> + 	  fopts += strlen (fopts) + 1;
> + 	}
> +       while (fopts - data < length);
> + 
> +       free (data);
> +       simple_object_release_read (sobj);
> +       close (fd);
> +     }
> + 
>     /* Initalize the common arguments for the driver.  */
>     obstack_init (&argv_obstack);
>     obstack_ptr_grow (&argv_obstack, collect_gcc);
>     obstack_ptr_grow (&argv_obstack, "-xlto");
>     obstack_ptr_grow (&argv_obstack, "-c");
> + 
> +   /* Append compiler driver arguments as far as they were merged.  */
> +   for (j = 1; j < fdecoded_options_count; ++j)
> +     {
> +       struct cl_decoded_option *option = &fdecoded_options[j];
> + 
> +       /* File options have been properly filtered by lto-opts.c.  */
> +       switch (option->opt_index)
> + 	{
> + 	  /* Drop arguments that we want to take from the link line.  */
> + 	  case OPT_flto_:
> + 	  case OPT_flto:
> + 	  case OPT_flto_partition_none:
> + 	  case OPT_flto_partition_1to1:
> + 	  case OPT_flto_partition_balanced:
> + 	      continue;
> + 
> + 	  default:
> + 	      break;
> + 	}
> + 
> +       /* For now do what the original LTO option code was doing - pass
> + 	 on any CL_TARGET flag and a few selected others.  */
> +       switch (option->opt_index)
> + 	{
> + 	case OPT_fPIC:
> + 	case OPT_fpic:
> + 	case OPT_fpie:
> + 	case OPT_fcommon:
> + 	case OPT_fexceptions:
> + 	  break;
> + 
> + 	default:
> + 	  if (!(cl_options[option->opt_index].flags & CL_TARGET))
> + 	    continue;
> + 	}
> + 
> +       /* Pass the option on.  */
> +       for (i = 0; i < option->canonical_option_num_elements; ++i)
> + 	obstack_ptr_grow (&argv_obstack, option->canonical_option[i]);
> +     }
> + 
> +   /* Append linker driver arguments.  Compiler options from the linker
> +      driver arguments will override / merge with those from the compiler.  */
>     for (j = 1; j < decoded_options_count; ++j)
>       {
>         struct cl_decoded_option *option = &decoded_options[j];
> Index: trunk/gcc/gcc.c
> ===================================================================
> *** trunk.orig/gcc/gcc.c	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/gcc.c	2011-10-28 11:51:24.000000000 +0200
> *************** driver_wrong_lang_callback (const struct
> *** 3137,3152 ****
>   		 &decoded->canonical_option[1], false);
>   }
>   
> - /* Note that an option (index OPT_INDEX, argument ARG, value VALUE)
> -    has been successfully handled with a handler for mask MASK.  */
> - 
> - static void
> - driver_post_handling_callback (const struct cl_decoded_option *decoded ATTRIBUTE_UNUSED,
> - 			       unsigned int mask ATTRIBUTE_UNUSED)
> - {
> -   /* Nothing to do here.  */
> - }
> - 
>   static const char *spec_lang = 0;
>   static int last_language_n_infiles;
>   
> --- 3137,3142 ----
> *************** set_option_handlers (struct cl_option_ha
> *** 3533,3539 ****
>   {
>     handlers->unknown_option_callback = driver_unknown_option_callback;
>     handlers->wrong_lang_callback = driver_wrong_lang_callback;
> -   handlers->post_handling_callback = driver_post_handling_callback;
>     handlers->num_handlers = 3;
>     handlers->handlers[0].handler = driver_handle_option;
>     handlers->handlers[0].mask = CL_DRIVER;
> --- 3523,3528 ----
> Index: trunk/gcc/opts-common.c
> ===================================================================
> *** trunk.orig/gcc/opts-common.c	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/opts-common.c	2011-10-28 11:51:24.000000000 +0200
> *************** handle_option (struct gcc_options *opts,
> *** 878,886 ****
>   					    lang_mask, kind, loc,
>   					    handlers, dc))
>   	  return false;
> - 	else
> - 	  handlers->post_handling_callback (decoded,
> - 					    handlers->handlers[i].mask);
>         }
>     
>     return true;
> --- 878,883 ----
> Index: trunk/gcc/opts.h
> ===================================================================
> *** trunk.orig/gcc/opts.h	2011-10-27 15:24:59.000000000 +0200
> --- trunk/gcc/opts.h	2011-10-28 11:51:24.000000000 +0200
> *************** struct cl_option_handlers
> *** 291,300 ****
>     void (*wrong_lang_callback) (const struct cl_decoded_option *decoded,
>   			       unsigned int lang_mask);
>   
> -   /* Callback to call after the successful handling of any option.  */
> -   void (*post_handling_callback) (const struct cl_decoded_option *decoded,
> - 				  unsigned int mask);
> - 
>     /* The number of individual handlers.  */
>     size_t num_handlers;
>   
> --- 291,296 ----

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Richard Guenther]

On Fri, Oct 28, 2011 at 5:48 PM, Joseph S. Myers
<joseph@codesourcery.com> wrote:
> On Fri, 28 Oct 2011, Richard Guenther wrote:
>
>> +       /* Fallthru.  */
>> +     case OPT_fPIC:
>> +     case OPT_fpic:
>> +     case OPT_fpie:
>> +     case OPT_fcommon:
>> +     case OPT_fexceptions:
>> +       append_option (decoded_options, decoded_options_count, foption);
>> +       break;
>
> No doubt this is what the previous code did, but in this case shouldn't
> "union" mean the biggest PIC status of any file wins (thus, if -fPIC was
> the PIC option that actually had effect on some object, that wins over an
> explicit -fno-PIC or -fpic on another object)?  In general whether the
> options are positive or negative matters, and I don't see that handled
> here.

I tried to look at what we get for -fno-pic vs. -fpic and -fno-pic is completely
dropped from the decoded options list (not sure what happens on targets
with -fpic as default).  So it seems at most one state (the non-default one)
survives here.

But maybe I'm missing something - how can I reliably check if there is
a negative form of an option, and if, if the option is the negative form?

> (Actually, maybe the smallest PIC status should win - i.e. if any object
> is not PIC then the final code can be presumed to be non-PIC.)
>
> (Using Negative in .opt files for groups of options such as -fPIC/-fpic
> would ensure that at most one survives from any one object, but you still
> need to work out what you want to do for merging.)

Sure - I suppose we can fix that as a followup - the patch tries to do what
we do now, just at a different place (the driver).  Thus I tried to preserve
all bugs as well ;)

Richard.

> --
> Joseph S. Myers
> joseph@codesourcery.com
>

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Joseph S. Myers]

On Sat, 29 Oct 2011, Richard Guenther wrote:

> I tried to look at what we get for -fno-pic vs. -fpic and -fno-pic is completely
> dropped from the decoded options list (not sure what happens on targets
> with -fpic as default).  So it seems at most one state (the non-default one)
> survives here.

I don't see where that would happening, and don't see that myself; 
-fno-pic gets included in COLLECT_GCC_OPTIONS for me if it's the last 
-fpic/-fno-pic option on the driver command line.

> But maybe I'm missing something - how can I reliably check if there is
> a negative form of an option, and if, if the option is the negative form?

The "value" field of a cl_decoded_option is 1 for a positive option, 0 for 
the negative form, for an option accepting positive and negative forms 
(not marked with RejectNegative / CL_REJECT_NEGATIVE.  (It's a bug if an 
option allows a negative form but also takes a UInteger argument, since 
such an argument also uses the "value" field.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Richard Guenther]

On Fri, Oct 28, 2011 at 4:48 PM, Diego Novillo <dnovillo@google.com> wrote:
> Isn't this the same patch as
> http://gcc.gnu.org/ml/gcc-patches/2011-10/msg02348.html?

No, it's updated to do exactly what we do now.  Changes to behavior to be split
to 3/n (like warning about mismatches or fixing the way we handle fPIC).

Richard.

>
> Diego.
>

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Richard Guenther]

On Fri, 28 Oct 2011, Joseph S. Myers wrote:

> On Fri, 28 Oct 2011, Richard Guenther wrote:
> 
> > + 	  /* Fallthru.  */
> > + 	case OPT_fPIC:
> > + 	case OPT_fpic:
> > + 	case OPT_fpie:
> > + 	case OPT_fcommon:
> > + 	case OPT_fexceptions:
> > + 	  append_option (decoded_options, decoded_options_count, foption);
> > + 	  break;
> 
> No doubt this is what the previous code did, but in this case shouldn't 
> "union" mean the biggest PIC status of any file wins (thus, if -fPIC was 
> the PIC option that actually had effect on some object, that wins over an 
> explicit -fno-PIC or -fpic on another object)?  In general whether the 
> options are positive or negative matters, and I don't see that handled 
> here.

Thinking about this again I think -fPIC and -fno-PIC in two objects
should cause a diagnostic.  The whole point of this (and the old) code
is to make sure we use -fPIC at link-time, but we should only do so
if there is consistent use in the separate CUs.  The same probably
applies to target specific opts - a -mavx on one unit shouldn't really
cause -mavx being used for the whole program at link-time (even if
it currently does ...).

The following slight adjustment to the patch marks the code identifying
"duplicates" accordingly - that it does what the old code did.  I'll
try to clean this up as a followup (I basically want to make sure
we do not regress with the switch to handling things in the driver,
changing behavior on purpose will just make bisecting this harder).

I've also arranged to do more pruning on the saved options on the
writing side (drop all diagnostic options and options that are
also handled by the driver).  What we want to preserve are
IL changing and optimization / target options.

Thus, I really want to go forward with this ... as this also eliminates
the most often seen reason for LTO bytecode incompatibility on branches
(OPT_ enum changes).

Re-LTO-bootstrap and test running on x86_64-unknown-linux-gnu, ok for 
trunk?

Thanks,
Richard.

2011-10-28  Richard Guenther  <rguenther@suse.de>

	* lto-opts.c: Re-implement.
	* lto-streamer.h (lto_register_user_option): Remove.
	(lto_read_file_options): Likewise.
	(lto_reissue_options): Likewise.
	(lto_clear_user_options): Likewise.
	(lto_clear_file_options): Likewise.
	* opts-global.c (post_handling_callback): Remove.
	(set_default_handlers): Do not set post_handling_callback.
	(decode_options): Remove LTO specific code.
	* lto-wrapper.c (merge_and_complain): New function.
	(run_gcc): Read all input file options and
	prepend a merged set before the linker driver options.
	* gcc.c (driver_post_handling_callback): Remove.
	(set_option_handlers): Do not set post_handling_callback.
	* opts-common.c (handle_option): Do not call post_handling_callback.
	* opts.h (struct cl_option_handlers): Remove post_handling_callback.

	lto/
	* lto-lang.c (lto_post_options): Do not read file options.
	* lto.c (lto_read_all_file_options): Remove.
	(lto_init): Call lto_set_in_hooks here.


Index: trunk/gcc/lto-opts.c
===================================================================
*** trunk.orig/gcc/lto-opts.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/lto-opts.c	2011-11-02 12:22:43.000000000 +0100
***************
*** 1,6 ****
  /* LTO IL options.
  
!    Copyright 2009, 2010, 2011 Free Software Foundation, Inc.
     Contributed by Simon Baldwin <simonb@google.com>
  
  This file is part of GCC.
--- 1,6 ----
  /* LTO IL options.
  
!    Copyright 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
     Contributed by Simon Baldwin <simonb@google.com>
  
  This file is part of GCC.
*************** along with GCC; see the file COPYING3.
*** 33,422 ****
  #include "common/common-target.h"
  #include "diagnostic.h"
  #include "lto-streamer.h"
! 
! /* When a file is initially compiled, the options used when generating
!    the IL are not necessarily the same as those used when linking the
!    objects into the final executable.  In general, most build systems
!    will proceed with something along the lines of:
! 
!    	$ gcc <cc-flags> -flto -c f1.c -o f1.o
! 	$ gcc <cc-flags> -flto -c f2.c -o f2.o
! 	...
! 	$ gcc <cc-flags> -flto -c fN.c -o fN.o
! 
!    And the final link may or may not include the same <cc-flags> used
!    to generate the initial object files:
! 
!    	$ gcc <ld-flags> -flto -o prog f1.o ... fN.o
! 
!    Since we will be generating final code during the link step, some
!    of the flags used during the compile step need to be re-applied
!    during the link step.  For instance, flags in the -m family.
! 
!    The idea is to save a selected set of <cc-flags> in a special
!    section of the initial object files.  This section is then read
!    during linking and the options re-applied.
! 
!    FIXME lto.  Currently the scheme is limited in that only the
!    options saved on the first object file (f1.o) are read back during
!    the link step.  This means that the options used to compile f1.o
!    will be applied to ALL the object files in the final link step.
!    More work needs to be done to implement a merging and validation
!    mechanism, as this will not be enough for all cases.  */
! 
! /* Saved options hold the type of the option (currently CL_TARGET or
!    CL_COMMON), and the code, argument, and value.  */
! 
! typedef struct GTY(()) opt_d
! {
!   unsigned int type;
!   size_t code;
!   char *arg;
!   int value;
! } opt_t;
! 
! DEF_VEC_O (opt_t);
! DEF_VEC_ALLOC_O (opt_t, heap);
! 
! 
! /* Options are held in two vectors, one for those registered by
!    command line handling code, and the other for those read in from
!    any LTO IL input.  */
! static VEC(opt_t, heap) *user_options = NULL;
! static VEC(opt_t, heap) *file_options = NULL;
! 
! /* Iterate FROM in reverse, writing option codes not yet in CODES into *TO.
!    Mark each new option code encountered in CODES.  */
! 
! static void
! reverse_iterate_options (VEC(opt_t, heap) *from, VEC(opt_t, heap) **to,
! 			 bitmap codes)
! {
!   int i;
! 
!   for (i = VEC_length (opt_t, from); i > 0; i--)
!     {
!       const opt_t *const o = VEC_index (opt_t, from, i - 1);
! 
!       if (bitmap_set_bit (codes, o->code))
! 	VEC_safe_push (opt_t, heap, *to, o);
!     }
! }
! 
! /* Concatenate options vectors FIRST and SECOND, rationalize so that only the
!    final of any given option remains, and return the result.  */
! 
! static VEC(opt_t, heap) *
! concatenate_options (VEC(opt_t, heap) *first, VEC(opt_t, heap) *second)
! {
!   VEC(opt_t, heap) *results = NULL;
!   bitmap codes = lto_bitmap_alloc ();
! 
!   reverse_iterate_options (second, &results, codes);
!   reverse_iterate_options (first, &results, codes);
! 
!   lto_bitmap_free (codes);
!   return results;
! }
! 
! /* Clear the options vector in *OPTS_P and set it to NULL.  */
! 
! static void
! clear_options (VEC(opt_t, heap) **opts_p)
! {
!   int i;
!   opt_t *o;
! 
!   FOR_EACH_VEC_ELT (opt_t, *opts_p, i, o)
!     free (o->arg);
! 
!   VEC_free (opt_t, heap, *opts_p);
! }
! 
! /* Write LENGTH bytes from ADDR to STREAM.  */
! 
! static void
! output_data_stream (struct lto_output_stream *stream,
!                     const void *addr, size_t length)
! {
!   lto_output_data_stream (stream, addr, length);
! }
! 
! /* Write string STRING to STREAM.  */
! 
! static void
! output_string_stream (struct lto_output_stream *stream, const char *string)
! {
!   bool flag = false;
! 
!   if (string != NULL)
!     {
!       const size_t length = strlen (string);
! 
!       flag = true;
!       output_data_stream (stream, &flag, sizeof (flag));
!       output_data_stream (stream, &length, sizeof (length));
!       output_data_stream (stream, string, length);
!     }
!   else
!     output_data_stream (stream, &flag, sizeof (flag));
! }
! 
! /* Return a string from IB.  The string is allocated, and the caller is
!    responsible for freeing it.  */
! 
! static char *
! input_string_block (struct lto_input_block *ib)
! {
!   bool flag;
! 
!   lto_input_data_block (ib, &flag, sizeof (flag));
!   if (flag)
!     {
!       size_t length;
!       char *string;
! 
!       lto_input_data_block (ib, &length, sizeof (length));
!       string = (char *) xcalloc (1, length + 1);
!       lto_input_data_block (ib, string, length);
! 
!       return string;
!     }
!   else
!     return NULL;
! }
! 
! /* Return true if this option is one we need to save in LTO output files.
!    At present, we pass along all target options, and common options that
!    involve position independent code.
! 
!    TODO This list of options requires expansion and rationalization.
!    Among others, optimization options may well be appropriate here.  */
! 
! static bool
! register_user_option_p (size_t code, unsigned int type)
! {
!   if (type == CL_TARGET)
!     return true;
!   else if (type == CL_COMMON)
!     {
!       return (code == OPT_fPIC
! 	      || code == OPT_fpic
! 	      || code == OPT_fPIE
! 	      || code == OPT_fpie
! 	      || code == OPT_fcommon
! 	      || code == OPT_fexceptions);
!     }
! 
!   return false;
! }
! 
! /* Note command line option with the given TYPE and CODE, ARG, and VALUE.
!    If relevant to LTO, save it in the user options vector.  */
! 
! void
! lto_register_user_option (size_t code, const char *arg, int value,
! 			  unsigned int type)
! {
!   if (register_user_option_p (code, type))
!     {
!       opt_t o;
! 
!       o.type = type;
!       o.code = code;
!       if (arg != NULL)
! 	{
! 	  o.arg = (char *) xmalloc (strlen (arg) + 1);
! 	  strcpy (o.arg, arg);
! 	}
!       else
! 	o.arg = NULL;
!       o.value = value;
!       VEC_safe_push (opt_t, heap, user_options, &o);
!     }
! }
! 
! /* Empty the saved user options vector.  */
! 
! void
! lto_clear_user_options (void)
! {
!   clear_options (&user_options);
! }
! 
! /* Empty the saved file options vector.  */
! 
! void
! lto_clear_file_options (void)
! {
!   clear_options (&file_options);
! }
! 
! /* Concatenate the user options and any file options read from an LTO IL
!    file, and serialize them to STREAM.  File options precede user options
!    so that the latter override the former when reissued.  */
! 
! static void
! output_options (struct lto_output_stream *stream)
! {
!   VEC(opt_t, heap) *opts = concatenate_options (file_options, user_options);
!   const size_t length = VEC_length (opt_t, opts);
!   int i;
!   opt_t *o;
! 
!   output_data_stream (stream, &length, sizeof (length));
! 
!   FOR_EACH_VEC_ELT (opt_t, opts, i, o)
!     {
!       output_data_stream (stream, &o->type, sizeof (o->type));
!       output_data_stream (stream, &o->code, sizeof (o->code));
!       output_string_stream (stream, o->arg);
!       output_data_stream (stream, &o->value, sizeof (o->value));
!     }
! 
!   VEC_free (opt_t, heap, opts);
! }
  
  /* Write currently held options to an LTO IL section.  */
  
  void
  lto_write_options (void)
  {
-   char *const section_name = lto_get_section_name (LTO_section_opts, NULL, NULL);
    struct lto_output_stream stream;
!   struct lto_simple_header header;
!   struct lto_output_stream *header_stream;
! 
!   /* Targets and languages can provide defaults for -fexceptions but
!      we only process user options from the command-line.  Until we
!      serialize out a white list of options from the new global state
!      explicitly append important options as user options here.  */
!   if (flag_exceptions)
!     lto_register_user_option (OPT_fexceptions, NULL, 1, CL_COMMON);
! 
!   lto_begin_section (section_name, !flag_wpa);
!   free (section_name);
  
    memset (&stream, 0, sizeof (stream));
-   output_options (&stream);
- 
-   memset (&header, 0, sizeof (header));
-   header.lto_header.major_version = LTO_major_version;
-   header.lto_header.minor_version = LTO_minor_version;
-   header.lto_header.section_type = LTO_section_opts;
- 
-   header.compressed_size = 0;
-   header.main_size = stream.total_size;
- 
-   header_stream = ((struct lto_output_stream *)
- 		   xcalloc (1, sizeof (*header_stream)));
-   lto_output_data_stream (header_stream, &header, sizeof (header));
-   lto_write_stream (header_stream);
-   free (header_stream);
  
!   lto_write_stream (&stream);
!   lto_end_section ();
! }
! 
! /* Unserialize an options vector from IB, and append to file_options.  */
! 
! static void
! input_options (struct lto_input_block *ib)
! {
!   size_t length, i;
! 
!   lto_input_data_block (ib, &length, sizeof (length));
! 
!   for (i = 0; i < length; i++)
      {
!       opt_t o;
! 
!       lto_input_data_block (ib, &o.type, sizeof (o.type));
!       lto_input_data_block (ib, &o.code, sizeof (o.code));
!       o.arg = input_string_block (ib);
!       lto_input_data_block (ib, &o.value, sizeof (o.value));
!       VEC_safe_push (opt_t, heap, file_options, &o);
!     }
! }
! 
! /* Read options from an LTO IL section.  */
! 
! void
! lto_read_file_options (struct lto_file_decl_data *file_data)
! {
!   size_t len, l, skip;
!   const char *data, *p;
!   const struct lto_simple_header *header;
!   int32_t opts_offset;
!   struct lto_input_block ib;
! 
!   data = lto_get_section_data (file_data, LTO_section_opts, NULL, &len);
!   if (!data)
! 	  return;
! 
!   /* Option could be multiple sections merged (through ld -r) 
!      Keep reading all options.  This is ok right now because
!      the options just get mashed together anyways.
!      This will have to be done differently once lto-opts knows
!      how to associate options with different files. */
!   l = len;
!   p = data;
!   do 
!     { 
!       header = (const struct lto_simple_header *) p;
!       opts_offset = sizeof (*header);
! 
!       lto_check_version (header->lto_header.major_version,
! 			 header->lto_header.minor_version);
!       
!       LTO_INIT_INPUT_BLOCK (ib, p + opts_offset, 0, header->main_size);
!       input_options (&ib);
!       
!       skip = header->main_size + opts_offset;
!       l -= skip;
!       p += skip;
!     } 
!   while (l > 0);
! 
!   lto_free_section_data (file_data, LTO_section_opts, 0, data, len);
! }
  
! /* Concatenate the user options and any file options read from an LTO IL
!    file, and reissue them as if all had just been read in from the command
!    line.  As with serialization, file options precede user options.  */
  
! void
! lto_reissue_options (void)
! {
!   VEC(opt_t, heap) *opts = concatenate_options (file_options, user_options);
!   int i;
!   opt_t *o;
  
!   FOR_EACH_VEC_ELT (opt_t, opts, i, o)
!     {
!       void *flag_var = option_flag_var (o->code, &global_options);
  
!       if (flag_var)
! 	set_option (&global_options, &global_options_set,
! 		    o->code, o->value, o->arg,
! 		    DK_UNSPECIFIED, UNKNOWN_LOCATION, global_dc);
  
!       if (o->type == CL_TARGET)
  	{
! 	  struct cl_decoded_option decoded;
! 	  generate_option (o->code, o->arg, o->value, CL_TARGET, &decoded);
! 	  targetm_common.handle_option (&global_options, &global_options_set,
! 					&decoded, UNKNOWN_LOCATION);
  	}
-       else if (o->type == CL_COMMON)
- 	gcc_assert (flag_var);
-       else
- 	gcc_unreachable ();
      }
  
!   /* Flag_shlib is usually set by finish_options, but we are issuing flag_pic
!      too late.  */
!   if (flag_pic && !flag_pie)
!     flag_shlib = 1;
!   VEC_free (opt_t, heap, opts);
  }
--- 33,121 ----
  #include "common/common-target.h"
  #include "diagnostic.h"
  #include "lto-streamer.h"
! #include "toplev.h"
  
  /* Write currently held options to an LTO IL section.  */
  
  void
  lto_write_options (void)
  {
    struct lto_output_stream stream;
!   char *section_name;
!   struct obstack temporary_obstack;
!   unsigned int i, j;
!   char *args;
  
+   section_name = lto_get_section_name (LTO_section_opts, NULL, NULL);
+   lto_begin_section (section_name, false);
    memset (&stream, 0, sizeof (stream));
  
!   obstack_init (&temporary_obstack);
!   for (i = 1; i < save_decoded_options_count; ++i)
      {
!       struct cl_decoded_option *option = &save_decoded_options[i];
!       const char *q, *p;
  
!       /* Skip frontend and driver specific options here.  */
!       if (!(cl_options[option->opt_index].flags & (CL_COMMON|CL_TARGET|CL_LTO)))
! 	continue;
! 
!       /* Drop options created from the gcc driver that will be rejected
! 	 when passed on to the driver again.  */
!       if (cl_options[option->opt_index].cl_reject_driver)
! 	continue;
! 
!       /* Also drop all options that are handled by the driver as well,
!          which includes things like -o and -v or -fhelp for example.
! 	 We do not need those.  Also drop all diagnostic options.  */
!       if (cl_options[option->opt_index].flags & (CL_DRIVER|CL_WARNING))
! 	continue;
  
!       /* Skip explicitly some common options that we do not need.  */
!       switch (option->opt_index)
! 	{
! 	case OPT_dumpbase:
! 	case OPT_SPECIAL_input_file:
! 	  continue;
  
! 	default:
! 	  break;
! 	}
  
!       if (i != 1)
! 	obstack_grow (&temporary_obstack, " ", 1);
!       obstack_grow (&temporary_obstack, "'", 1);
!       q = option->canonical_option[0];
!       while ((p = strchr (q, '\'')))
! 	{
! 	  obstack_grow (&temporary_obstack, q, p - q);
! 	  obstack_grow (&temporary_obstack, "'\\''", 4);
! 	  q = ++p;
! 	}
!       obstack_grow (&temporary_obstack, q, strlen (q));
!       obstack_grow (&temporary_obstack, "'", 1);
  
!       for (j = 1; j < option->canonical_option_num_elements; ++j)
  	{
! 	  obstack_grow (&temporary_obstack, " '", 2);
! 	  q = option->canonical_option[j];
! 	  while ((p = strchr (q, '\'')))
! 	    {
! 	      obstack_grow (&temporary_obstack, q, p - q);
! 	      obstack_grow (&temporary_obstack, "'\\''", 4);
! 	      q = ++p;
! 	    }
! 	  obstack_grow (&temporary_obstack, q, strlen (q));
! 	  obstack_grow (&temporary_obstack, "'", 1);
  	}
      }
+   obstack_grow (&temporary_obstack, "\0", 1);
+   args = XOBFINISH (&temporary_obstack, char *);
+   lto_output_data_stream (&stream, args, strlen (args) + 1);
  
!   lto_write_stream (&stream);
!   lto_end_section ();
! 
!   obstack_free (&temporary_obstack, NULL);
!   free (section_name);
  }
Index: trunk/gcc/lto-streamer.h
===================================================================
*** trunk.orig/gcc/lto-streamer.h	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/lto-streamer.h	2011-11-02 11:21:49.000000000 +0100
*************** extern GTY(()) VEC(tree,gc) *lto_global_
*** 882,893 ****
  
  
  /* In lto-opts.c.  */
- extern void lto_register_user_option (size_t, const char *, int, unsigned int);
- extern void lto_read_file_options (struct lto_file_decl_data *);
  extern void lto_write_options (void);
- extern void lto_reissue_options (void);
- void lto_clear_user_options (void);
- void lto_clear_file_options (void);
  
  
  /* In lto-wpa-fixup.c  */
--- 882,888 ----
Index: trunk/gcc/lto/lto-lang.c
===================================================================
*** trunk.orig/gcc/lto/lto-lang.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/lto/lto-lang.c	2011-11-02 11:21:49.000000000 +0100
*************** lto_post_options (const char **pfilename
*** 692,699 ****
       support.  */
    flag_excess_precision_cmdline = EXCESS_PRECISION_FAST;
  
-   lto_read_all_file_options ();
- 
    /* Initialize the compiler back end.  */
    return false;
  }
--- 692,697 ----
Index: trunk/gcc/lto/lto.c
===================================================================
*** trunk.orig/gcc/lto/lto.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/lto/lto.c	2011-11-02 11:21:49.000000000 +0100
*************** lto_fixup_decls (struct lto_file_decl_da
*** 2494,2553 ****
      }
  }
  
- /* Read the options saved from each file in the command line.  Called
-    from lang_hooks.post_options which is called by process_options
-    right before all the options are used to initialize the compiler.
-    This assumes that decode_options has already run, so the
-    num_in_fnames and in_fnames are properly set.
- 
-    Note that this assumes that all the files had been compiled with
-    the same options, which is not a good assumption.  In general,
-    options ought to be read from all the files in the set and merged.
-    However, it is still unclear what the merge rules should be.  */
- 
- void
- lto_read_all_file_options (void)
- {
-   size_t i;
- 
-   /* Clear any file options currently saved.  */
-   lto_clear_file_options ();
- 
-   /* Set the hooks to read ELF sections.  */
-   lto_set_in_hooks (NULL, get_section_data, free_section_data);
-   if (!quiet_flag)
-     fprintf (stderr, "Reading command line options:");
- 
-   for (i = 0; i < num_in_fnames; i++)
-     {
-       struct lto_file_decl_data *file_data;
-       lto_file *file = lto_obj_file_open (in_fnames[i], false);
-       if (!file)
- 	break;
-       if (!quiet_flag)
- 	{
- 	  fprintf (stderr, " %s", in_fnames[i]);
- 	  fflush (stderr);
- 	}
- 
-       file_data = XCNEW (struct lto_file_decl_data);
-       file_data->file_name = file->filename;
-       file_data->section_hash_table = lto_obj_build_section_table (file, NULL);
- 
-       lto_read_file_options (file_data);
- 
-       lto_obj_file_close (file);
-       htab_delete (file_data->section_hash_table);
-       free (file_data);
-     }
- 
-   if (!quiet_flag)
-     fprintf (stderr, "\n");
- 
-   /* Apply globally the options read from all the files.  */
-   lto_reissue_options ();
- }
- 
  static GTY((length ("lto_stats.num_input_files + 1"))) struct lto_file_decl_data **all_file_decl_data;
  
  /* Turn file datas for sub files into a single array, so that they look
--- 2494,2499 ----
*************** lto_init (void)
*** 2921,2926 ****
--- 2867,2873 ----
    lto_process_name ();
    lto_streamer_hooks_init ();
    lto_reader_init ();
+   lto_set_in_hooks (NULL, get_section_data, free_section_data);
    memset (&lto_stats, 0, sizeof (lto_stats));
    bitmap_obstack_initialize (NULL);
    gimple_register_cfg_hooks ();
Index: trunk/gcc/opts-global.c
===================================================================
*** trunk.orig/gcc/opts-global.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/opts-global.c	2011-11-02 11:21:49.000000000 +0100
*************** unknown_option_callback (const struct cl
*** 160,178 ****
      return true;
  }
  
- /* Note that an option DECODED has been successfully handled with a
-    handler for mask MASK.  */
- 
- static void
- post_handling_callback (const struct cl_decoded_option *decoded ATTRIBUTE_UNUSED,
- 			unsigned int mask ATTRIBUTE_UNUSED)
- {
- #ifdef ENABLE_LTO
-   lto_register_user_option (decoded->opt_index, decoded->arg,
- 			    decoded->value, mask);
- #endif
- }
- 
  /* Handle a front-end option; arguments and return value as for
     handle_option.  */
  
--- 160,165 ----
*************** set_default_handlers (struct cl_option_h
*** 282,288 ****
  {
    handlers->unknown_option_callback = unknown_option_callback;
    handlers->wrong_lang_callback = complain_wrong_lang;
-   handlers->post_handling_callback = post_handling_callback;
    handlers->num_handlers = 3;
    handlers->handlers[0].handler = lang_handle_option;
    handlers->handlers[0].mask = initial_lang_mask;
--- 269,274 ----
*************** decode_options (struct gcc_options *opts
*** 314,324 ****
  				decoded_options, decoded_options_count,
  				loc, lang_mask, &handlers, dc);
  
- #ifdef ENABLE_LTO
-   /* Clear any options currently held for LTO.  */
-   lto_clear_user_options ();
- #endif
- 
    read_cmdline_options (opts, opts_set,
  			decoded_options, decoded_options_count,
  			loc, lang_mask,
--- 300,305 ----
Index: trunk/gcc/lto-wrapper.c
===================================================================
*** trunk.orig/gcc/lto-wrapper.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/lto-wrapper.c	2011-11-02 12:07:20.000000000 +0100
*************** along with GCC; see the file COPYING3.
*** 45,50 ****
--- 45,58 ----
  #include "obstack.h"
  #include "opts.h"
  #include "options.h"
+ #include "simple-object.h"
+ 
+ /* From lto-streamer.h which we cannot include with -fkeep-inline-functions.
+    ???  Split out a lto-streamer-core.h.  */
+ 
+ #define LTO_SECTION_NAME_PREFIX         ".gnu.lto_"
+ 
+ /* End of lto-streamer.h copy.  */
  
  int debug;				/* true if -save-temps.  */
  int verbose;				/* true if -v.  */
*************** get_options_from_collect_gcc_options (co
*** 327,332 ****
--- 335,412 ----
    free (argv);
  }
  
+ /* Append OPTION to the options array DECODED_OPTIONS with size
+    DECODED_OPTIONS_COUNT.  */
+ 
+ static void
+ append_option (struct cl_decoded_option **decoded_options,
+ 	       unsigned int *decoded_options_count,
+ 	       struct cl_decoded_option *option)
+ {
+   ++*decoded_options_count;
+   *decoded_options
+     = (struct cl_decoded_option *)
+ 	xrealloc (*decoded_options,
+ 		  (*decoded_options_count
+ 		   * sizeof (struct cl_decoded_option)));
+   memcpy (&(*decoded_options)[*decoded_options_count - 1], option,
+ 	  sizeof (struct cl_decoded_option));
+ }
+ 
+ /* Try to merge and complain about options FDECODED_OPTIONS when applied
+    ontop of DECODED_OPTIONS.  */
+ 
+ static void
+ merge_and_complain (struct cl_decoded_option **decoded_options,
+ 		    unsigned int *decoded_options_count,
+ 		    struct cl_decoded_option *fdecoded_options,
+ 		    unsigned int fdecoded_options_count)
+ {
+   unsigned int i, j;
+ 
+   /* ???  Merge options from files.  Most cases can be
+      handled by either unioning or intersecting
+      (for example -fwrapv is a case for unioning,
+      -ffast-math is for intersection).  Most complaints
+      about real conflicts between different options can
+      be deferred to the compiler proper.  Options that
+      we can neither safely handle by intersection nor
+      unioning would need to be complained about here.
+      Ideally we'd have a flag in the opt files that
+      tells whether to union or intersect or reject.
+      In absence of that it's unclear what a good default is.
+      It's also difficult to get positional handling correct.  */
+ 
+   /* The following does what the old LTO option code did,
+      union all target and a selected set of common options.  */
+   for (i = 0; i < fdecoded_options_count; ++i)
+     {
+       struct cl_decoded_option *foption = &fdecoded_options[i];
+       switch (foption->opt_index)
+ 	{
+ 	default:
+ 	  if (!(cl_options[foption->opt_index].flags & CL_TARGET))
+ 	    break;
+ 
+ 	  /* Fallthru.  */
+ 	case OPT_fPIC:
+ 	case OPT_fpic:
+ 	case OPT_fpie:
+ 	case OPT_fcommon:
+ 	case OPT_fexceptions:
+ 	  /* Do what the old LTO code did - collect exactly one option
+ 	     setting per OPT code, we pick the first we encounter.
+ 	     ???  This doesn't make too much sense, but when it doesn't
+ 	     then we should complain.  */
+ 	  for (j = 0; j < *decoded_options_count; ++j)
+ 	    if ((*decoded_options)[j].opt_index == foption->opt_index)
+ 	      break;
+ 	  if (j == *decoded_options_count)
+ 	    append_option (decoded_options, decoded_options_count, foption);
+ 	  break;
+ 	}
+     }
+ }
  
  /* Execute gcc. ARGC is the number of arguments. ARGV contains the arguments. */
  
*************** run_gcc (unsigned argc, char *argv[])
*** 342,347 ****
--- 422,429 ----
    int parallel = 0;
    int jobserver = 0;
    bool no_partition = false;
+   struct cl_decoded_option *fdecoded_options = NULL;
+   unsigned int fdecoded_options_count = 0;
    struct cl_decoded_option *decoded_options;
    unsigned int decoded_options_count;
    struct obstack argv_obstack;
*************** run_gcc (unsigned argc, char *argv[])
*** 359,369 ****
--- 441,565 ----
  					&decoded_options,
  					&decoded_options_count);
  
+   /* Look at saved options in the IL files.  */
+   for (i = 1; i < argc; ++i)
+     {
+       char *data, *p;
+       char *fopts;
+       int fd;
+       const char *errmsg;
+       int err;
+       off_t file_offset = 0, offset, length;
+       long loffset;
+       simple_object_read *sobj;
+       int consumed;
+       struct cl_decoded_option *f2decoded_options;
+       unsigned int f2decoded_options_count;
+       char *filename = argv[i];
+       if ((p = strrchr (argv[i], '@'))
+ 	  && p != argv[i] 
+ 	  && sscanf (p, "@%li%n", &loffset, &consumed) >= 1
+ 	  && strlen (p) == (unsigned int) consumed)
+ 	{
+ 	  filename = XNEWVEC (char, p - argv[i] + 1);
+ 	  memcpy (filename, argv[i], p - argv[i]);
+ 	  filename[p - argv[i]] = '\0';
+ 	  file_offset = (off_t) loffset;
+ 	}
+       fd = open (argv[i], O_RDONLY);
+       if (fd == -1)
+ 	continue;
+       sobj = simple_object_start_read (fd, file_offset, NULL, &errmsg, &err);
+       if (!sobj)
+ 	{
+ 	  close (fd);
+ 	  continue;
+ 	}
+       if (!simple_object_find_section (sobj, LTO_SECTION_NAME_PREFIX "." "opts",
+ 				       &offset, &length, &errmsg, &err))
+ 	{
+ 	  simple_object_release_read (sobj);
+ 	  close (fd);
+ 	  continue;
+ 	}
+       lseek (fd, file_offset + offset, SEEK_SET);
+       data = (char *)xmalloc (length);
+       read (fd, data, length);
+       fopts = data;
+       do
+ 	{
+ 	  get_options_from_collect_gcc_options (collect_gcc,
+ 						fopts, CL_LANG_ALL,
+ 						&f2decoded_options,
+ 						&f2decoded_options_count);
+ 	  if (!fdecoded_options)
+ 	    {
+ 	      fdecoded_options = f2decoded_options;
+ 	      fdecoded_options_count = f2decoded_options_count;
+ 	    }
+ 	  else
+ 	    merge_and_complain (&fdecoded_options,
+ 				&fdecoded_options_count,
+ 				f2decoded_options, f2decoded_options_count);
+ 
+ 	  fopts += strlen (fopts) + 1;
+ 	}
+       while (fopts - data < length);
+ 
+       free (data);
+       simple_object_release_read (sobj);
+       close (fd);
+     }
+ 
    /* Initalize the common arguments for the driver.  */
    obstack_init (&argv_obstack);
    obstack_ptr_grow (&argv_obstack, collect_gcc);
    obstack_ptr_grow (&argv_obstack, "-xlto");
    obstack_ptr_grow (&argv_obstack, "-c");
+ 
+   /* Append compiler driver arguments as far as they were merged.  */
+   for (j = 1; j < fdecoded_options_count; ++j)
+     {
+       struct cl_decoded_option *option = &fdecoded_options[j];
+ 
+       /* File options have been properly filtered by lto-opts.c.  */
+       switch (option->opt_index)
+ 	{
+ 	  /* Drop arguments that we want to take from the link line.  */
+ 	  case OPT_flto_:
+ 	  case OPT_flto:
+ 	  case OPT_flto_partition_none:
+ 	  case OPT_flto_partition_1to1:
+ 	  case OPT_flto_partition_balanced:
+ 	      continue;
+ 
+ 	  default:
+ 	      break;
+ 	}
+ 
+       /* For now do what the original LTO option code was doing - pass
+ 	 on any CL_TARGET flag and a few selected others.  */
+       switch (option->opt_index)
+ 	{
+ 	case OPT_fPIC:
+ 	case OPT_fpic:
+ 	case OPT_fpie:
+ 	case OPT_fcommon:
+ 	case OPT_fexceptions:
+ 	  break;
+ 
+ 	default:
+ 	  if (!(cl_options[option->opt_index].flags & CL_TARGET))
+ 	    continue;
+ 	}
+ 
+       /* Pass the option on.  */
+       for (i = 0; i < option->canonical_option_num_elements; ++i)
+ 	obstack_ptr_grow (&argv_obstack, option->canonical_option[i]);
+     }
+ 
+   /* Append linker driver arguments.  Compiler options from the linker
+      driver arguments will override / merge with those from the compiler.  */
    for (j = 1; j < decoded_options_count; ++j)
      {
        struct cl_decoded_option *option = &decoded_options[j];
Index: trunk/gcc/gcc.c
===================================================================
*** trunk.orig/gcc/gcc.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/gcc.c	2011-11-02 11:21:49.000000000 +0100
*************** driver_wrong_lang_callback (const struct
*** 3137,3152 ****
  		 &decoded->canonical_option[1], false);
  }
  
- /* Note that an option (index OPT_INDEX, argument ARG, value VALUE)
-    has been successfully handled with a handler for mask MASK.  */
- 
- static void
- driver_post_handling_callback (const struct cl_decoded_option *decoded ATTRIBUTE_UNUSED,
- 			       unsigned int mask ATTRIBUTE_UNUSED)
- {
-   /* Nothing to do here.  */
- }
- 
  static const char *spec_lang = 0;
  static int last_language_n_infiles;
  
--- 3137,3142 ----
*************** set_option_handlers (struct cl_option_ha
*** 3533,3539 ****
  {
    handlers->unknown_option_callback = driver_unknown_option_callback;
    handlers->wrong_lang_callback = driver_wrong_lang_callback;
-   handlers->post_handling_callback = driver_post_handling_callback;
    handlers->num_handlers = 3;
    handlers->handlers[0].handler = driver_handle_option;
    handlers->handlers[0].mask = CL_DRIVER;
--- 3523,3528 ----
Index: trunk/gcc/opts-common.c
===================================================================
*** trunk.orig/gcc/opts-common.c	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/opts-common.c	2011-11-02 11:21:49.000000000 +0100
*************** handle_option (struct gcc_options *opts,
*** 878,886 ****
  					    lang_mask, kind, loc,
  					    handlers, dc))
  	  return false;
- 	else
- 	  handlers->post_handling_callback (decoded,
- 					    handlers->handlers[i].mask);
        }
    
    return true;
--- 878,883 ----
Index: trunk/gcc/opts.h
===================================================================
*** trunk.orig/gcc/opts.h	2011-10-28 14:20:11.000000000 +0200
--- trunk/gcc/opts.h	2011-11-02 11:21:49.000000000 +0100
*************** struct cl_option_handlers
*** 291,300 ****
    void (*wrong_lang_callback) (const struct cl_decoded_option *decoded,
  			       unsigned int lang_mask);
  
-   /* Callback to call after the successful handling of any option.  */
-   void (*post_handling_callback) (const struct cl_decoded_option *decoded,
- 				  unsigned int mask);
- 
    /* The number of individual handlers.  */
    size_t num_handlers;
  
--- 291,296 ----

Re: [PATCH][2/n] LTO option handling/merging rewrite

[Andreas Schwab]

On Nov 02 2011, Richard Guenther wrote:

> 	lto/
> 	* lto-lang.c (lto_post_options): Do not read file options.
> 	* lto.c (lto_read_all_file_options): Remove.

This fails to update the documentation.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."