PATCH: distcc and ccache speedup: adds directives-only preprocessing

PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

[-- Attachment #1: Type: text/plain, Size: 2914 bytes --]

This patch decreases distcc build and ccache lookup times by providing
an optimized
"directives-only" preprocessing mode.

Distcc improves build performance by preprocessing files locally,
transmitting the preprocessed code to one or more remote build
servers, and performing compilation on the remote servers.  In
practice, the time to preprocess files locally is the limiting factor
in further reducing build time.  This patch adds a -directives-only
flag which instructs the preprocessor to treat directives normally but
bypass tokenization of other preprocessing tokens.  Additional
preprocessing, such as macro expansion outside of directives, is
performed during the compilation phase.

Ccache reduces recompilation time by caching the results of previous
compilations and detecting when a compilation is repeated.  It
generates a lookup key from the preprocessed source code.  The
-directives-only flag improves lookup performance by reducing key
generation time.

For a typical c++ file, the new option reduces preprocessing time by
about 30%.  This in turn results in a similar decrease in overall distcc
build and ccache lookup times.

Usage is straightforward.  The -directives-only flag is simply added
to the compiler options passed to distcc or ccache.

Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.

Ollie

:ADDPATCH libcpp:

2007-02-12  Ollie Wild  <aaw@google.com>

        * gcc.dg/cpp/dir-only-1.c: New test.
        * gcc.dg/cpp/dir-only-1.h: New file.
        * gcc.dg/cpp/dir-only-2.c: New test.

2007-02-12  Ollie Wild  <aaw@google.com>

        * directives-only.c: New file.
        * internal.h (struct _cpp_dir_only_callbacks): New.
        (_cpp_preprocess_dir_only): New function.
        * files.c (_cpp_stack_file): Add directives_only check.
        * cpplib.h (cpp_init_special_builtins): New function.
        (cpp_options): Add directives_only.
        * init.c (cpp_init_special_builtins): New function.
        (cpp_init_builtins): Move builtin_array initialization to
        cpp_init_special_builtins.
        (post_options): Check directives_only before setting
        pfile->state.prevent_expansion = 1.
        * Makefile.in (libcpp_a_OBJS): Add directives-only.o.
        (libcpp_a_SOURCES): Add directives-only.c.

2007-02-12  Ollie Wild  <aaw@google.com>

        * c-ppoutput.c (print_lines_directives_only): New function.
        (scan_translation_unit_directives_only): New function.
        (cb_define_directives_only): New function.
        (preprocess_file): Add call to scan_translation_unit_directives_only.
        (init_pp_ouput): Register cb_define_directives_only.
        * c-opts.c (c_common_handle_option): Add OPT_directives_only.
        (finish_options): Add builtin macro initialization for
        -directives-only + -fpreprocessed.
        * c.opt (directives-only): New.
        * doc/cppopts.texi (directives-only): New.

[-- Attachment #2: directives-only.txt --]
[-- Type: text/plain, Size: 18293 bytes --]

Index: gcc/doc/cppopts.texi
===================================================================
--- gcc/doc/cppopts.texi	(revision 121856)
+++ gcc/doc/cppopts.texi	(working copy)
@@ -650,6 +650,22 @@ Output @samp{#include} directives in add
 preprocessing.
 @end table
 
+@item -directives-only
+@opindex directives-only
+This option provides a simplified preprocessor to improve the
+performance of distributed build systems such as distcc.  It's
+behavior depends on a number of other flags.
+
+If the @option{-E} option is enabled, it suppresses things like macro
+expansion, trigraph conversion, and escaped newline splicing
+outside of directives.  All directives are processed normally, except that
+macro definitions are output similar to the @option{-dD} option.
+
+If the @option{-fpreprocessed} option is enabled, it suppresses
+predefinition of most builtin and command line macros.  This
+prevents duplicate definition of macros output with the @option{-E}
+option.
+
 @item -P
 @opindex P
 Inhibit generation of linemarkers in the output from the preprocessor.
Index: gcc/testsuite/gcc.dg/cpp/dir-only-1.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-1.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-1.c	(revision 0)
@@ -0,0 +1,67 @@
+/* Copyright 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -directives-only } */
+
+/* Tests scan_translation_unit_directives_only()'s handling of corner cases. */
+
+/* Ignore directives inside block comments...
+#error directive inside block comment
+*/
+
+// Escaped newline doesn't terminate line comment \
+#error directive inside line comment
+
+/* A comment canot start inside a string. */
+const char *c1 = "/*";
+#define NOT_IN_COMMENT
+const char *c2 = "*/";
+#ifndef NOT_IN_COMMENT
+#error Comment started inside a string literal
+#endif
+
+/* Escaped newline handling. */
+int i; \
+#error ignored escaped newline
+  \
+  \
+#define BOL
+#ifndef BOL
+#error escaped newline did not preserve beginning of line
+#endif
+
+/* Handles \\ properly at the end of a string. */
+"string ends in \\"/*
+#error Missed string terminator.
+*/
+
+/* Handles macro expansion in preprocessing directives. */
+#define HEADER "dir-only-1.h"
+#include HEADER
+#ifndef GOT_HEADER
+#error Failed to include header.
+#endif
+
+/\
+*
+#define IN_COMMENT
+*/
+#ifdef IN_COMMENT
+#error Escaped newline breaks block comment initiator.
+#endif
+
+/*
+*\
+/
+#define NOT_IN_COMMENT2
+/**/
+#ifndef NOT_IN_COMMENT2
+#error Escaped newline breaks block comment terminator.
+#endif
+
+/* Test escaped newline inside character escape sequence. */
+"\\
+\"/*
+#error Missed string terminator
+*/
Index: gcc/testsuite/gcc.dg/cpp/dir-only-1.h
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-1.h	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-1.h	(revision 0)
@@ -0,0 +1,3 @@
+/* Copyright 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+#define GOT_HEADER
Index: gcc/testsuite/gcc.dg/cpp/dir-only-2.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-2.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-2.c	(revision 0)
@@ -0,0 +1,10 @@
+/* Copyright 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fpreprocessed -directives-only -DNOT_SET" } */
+
+/* Tests -directives-only + -fpreprocessed. */
+
+/* Check this is not defined. */
+#define NOT_SET
Index: gcc/c.opt
===================================================================
--- gcc/c.opt	(revision 121856)
+++ gcc/c.opt	(working copy)
@@ -462,6 +462,10 @@ d
 C ObjC C++ ObjC++ Joined
 ; Documented in common.opt.  FIXME - what about -dI, -dD, -dN and -dD?
 
+directives-only
+C ObjC C++ ObjC++
+Preprocess directives only.
+
 faccess-control
 C++ ObjC++
 Enforce class member access control semantics
Index: gcc/c-opts.c
===================================================================
--- gcc/c-opts.c	(revision 121856)
+++ gcc/c-opts.c	(working copy)
@@ -560,6 +560,10 @@ c_common_handle_option (size_t scode, co
       handle_OPT_d (arg);
       break;
 
+    case OPT_directives_only:
+      cpp_opts->directives_only = 1;
+      break;
+
     case OPT_fcond_mismatch:
       if (!c_dialect_cxx ())
 	{
@@ -1461,6 +1465,8 @@ finish_options (void)
 	    }
 	}
     }
+  else if (cpp_opts->directives_only)
+    cpp_init_special_builtins (parse_in);
 
   include_cursor = 0;
   push_command_line_include ();
Index: gcc/c-ppoutput.c
===================================================================
--- gcc/c-ppoutput.c	(revision 121856)
+++ gcc/c-ppoutput.c	(working copy)
@@ -41,6 +41,8 @@ static struct
 
 /* General output routines.  */
 static void scan_translation_unit (cpp_reader *);
+static void print_lines_directives_only (int, const void *, size_t);
+static void scan_translation_unit_directives_only (cpp_reader *);
 static void scan_translation_unit_trad (cpp_reader *);
 static void account_for_newlines (const unsigned char *, size_t);
 static int dump_macro (cpp_reader *, cpp_hashnode *, void *);
@@ -52,6 +54,8 @@ static void maybe_print_line (source_loc
    in specific modes.  */
 static void cb_line_change (cpp_reader *, const cpp_token *, int);
 static void cb_define (cpp_reader *, source_location, cpp_hashnode *);
+static void cb_define_directives_only (cpp_reader *, source_location,
+				       cpp_hashnode *);
 static void cb_undef (cpp_reader *, source_location, cpp_hashnode *);
 static void cb_include (cpp_reader *, source_location, const unsigned char *,
 			const char *, int, const cpp_token **);
@@ -75,6 +79,8 @@ preprocess_file (cpp_reader *pfile)
     }
   else if (cpp_get_options (pfile)->traditional)
     scan_translation_unit_trad (pfile);
+  else if (cpp_get_options (pfile)->directives_only)
+    scan_translation_unit_directives_only (pfile);
   else
     scan_translation_unit (pfile);
 
@@ -114,7 +120,12 @@ init_pp_output (FILE *out_stream)
       cb->read_pch = cb_read_pch;
     }
 
-  if (flag_dump_macros == 'N' || flag_dump_macros == 'D')
+  if (cpp_get_options (parse_in)->directives_only)
+    {
+      cb->define = cb_define_directives_only;
+      cb->undef  = cb_undef;
+    }
+  else if (flag_dump_macros == 'N' || flag_dump_macros == 'D')
     {
       cb->define = cb_define;
       cb->undef  = cb_undef;
@@ -179,6 +190,26 @@ scan_translation_unit (cpp_reader *pfile
     }
 }
 
+static void
+print_lines_directives_only (int lines, const void *buf, size_t size)
+{
+  print.src_line += lines;
+  fwrite (buf, 1, size, print.outf);
+}
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+static void
+scan_translation_unit_directives_only (cpp_reader *pfile)
+{
+  struct _cpp_dir_only_callbacks cb;
+
+  cb.print_lines = print_lines_directives_only;
+  cb.maybe_print_line = maybe_print_line;
+
+  _cpp_preprocess_dir_only (pfile, &cb);
+}
+
 /* Adjust print.src_line for newlines embedded in output.  */
 static void
 account_for_newlines (const unsigned char *str, size_t len)
@@ -324,6 +355,20 @@ cb_define (cpp_reader *pfile, source_loc
     fputs ((const char *) NODE_NAME (node), print.outf);
 
   putc ('\n', print.outf);
+  print.src_line++;
+}
+
+static void
+cb_define_directives_only (cpp_reader *pfile, source_location line,
+			   cpp_hashnode *node)
+{
+  maybe_print_line (line);
+  fputs ("#define ", print.outf);
+
+  fputs ((const char *) cpp_macro_definition (pfile, node),
+	 print.outf);
+
+  putc ('\n', print.outf);
   if (linemap_lookup (&line_table, line)->to_line != 0)
     print.src_line++;
 }
Index: libcpp/Makefile.in
===================================================================
--- libcpp/Makefile.in	(revision 121856)
+++ libcpp/Makefile.in	(working copy)
@@ -69,14 +69,14 @@ INCLUDES = -I$(srcdir) -I. -I$(srcdir)/.
 
 ALL_CFLAGS = $(CFLAGS) $(WARN_CFLAGS) $(INCLUDES) $(CPPFLAGS)
 
-libcpp_a_OBJS = charset.o directives.o errors.o expr.o files.o \
-	identifiers.o init.o lex.o line-map.o macro.o mkdeps.o \
-	pch.o symtab.o traditional.o
+libcpp_a_OBJS = charset.o directives.o directives-only.o errors.o \
+	expr.o files.o identifiers.o init.o lex.o line-map.o macro.o \
+	mkdeps.o pch.o symtab.o traditional.o
 makedepend_OBJS = makedepend.o
 
-libcpp_a_SOURCES = charset.c directives.c errors.c expr.c files.c \
-	identifiers.c init.c lex.c line-map.c macro.c mkdeps.c \
-	pch.c symtab.c traditional.c
+libcpp_a_SOURCES = charset.c directives.c directives-only.c errors.c \
+	expr.c files.c identifiers.c init.c lex.c line-map.c macro.c \
+	mkdeps.c pch.c symtab.c traditional.c
 
 all: libcpp.a makedepend$(EXEEXT) $(USED_CATALOGS)
 
Index: libcpp/files.c
===================================================================
--- libcpp/files.c	(revision 121856)
+++ libcpp/files.c	(working copy)
@@ -722,7 +722,8 @@ _cpp_stack_file (cpp_reader *pfile, _cpp
 
   /* Stack the buffer.  */
   buffer = cpp_push_buffer (pfile, file->buffer, file->st.st_size,
-			    CPP_OPTION (pfile, preprocessed));
+			    CPP_OPTION (pfile, preprocessed)
+			    && !CPP_OPTION (pfile, directives_only));
   buffer->file = file;
   buffer->sysp = sysp;
 
Index: libcpp/include/cpplib.h
===================================================================
--- libcpp/include/cpplib.h	(revision 121856)
+++ libcpp/include/cpplib.h	(working copy)
@@ -440,6 +440,9 @@ struct cpp_options
 
   /* True means error callback should be used for diagnostics.  */
   bool client_diagnostic;
+
+  /* True disables tokenization outside of preprocessing directives. */
+  bool directives_only;
 };
 
 /* Callback for header lookup for HEADER, which is the name of a
@@ -643,6 +646,10 @@ extern struct deps *cpp_get_deps (cpp_re
    too.  If there was an error opening the file, it returns NULL.  */
 extern const char *cpp_read_main_file (cpp_reader *, const char *);
 
+/* Set up built-ins with special behavior.  Use cpp_init_builtins()
+   instead unless your know what you are doing.  */
+extern void cpp_init_special_builtins (cpp_reader *);
+
 /* Set up built-ins like __FILE__.  */
 extern void cpp_init_builtins (cpp_reader *, int);
 
Index: libcpp/init.c
===================================================================
--- libcpp/init.c	(revision 121856)
+++ libcpp/init.c	(working copy)
@@ -349,11 +349,8 @@ mark_named_operators (cpp_reader *pfile)
     }
 }
 
-/* Read the builtins table above and enter them, and language-specific
-   macros, into the hash table.  HOSTED is true if this is a hosted
-   environment.  */
 void
-cpp_init_builtins (cpp_reader *pfile, int hosted)
+cpp_init_special_builtins (cpp_reader *pfile)
 {
   const struct builtin *b;
   size_t n = ARRAY_SIZE (builtin_array);
@@ -362,10 +359,7 @@ cpp_init_builtins (cpp_reader *pfile, in
     n -= 2;
   else if (! CPP_OPTION (pfile, stdc_0_in_system_headers)
 	   || CPP_OPTION (pfile, std))
-    {
-      n--;
-      _cpp_define_builtin (pfile, "__STDC__ 1");
-    }
+    n--;
 
   for (b = builtin_array; b < builtin_array + n; b++)
     {
@@ -374,6 +368,20 @@ cpp_init_builtins (cpp_reader *pfile, in
       hp->flags |= NODE_BUILTIN | NODE_WARN;
       hp->value.builtin = (enum builtin_type) b->value;
     }
+}
+
+/* Read the builtins table above and enter them, and language-specific
+   macros, into the hash table.  HOSTED is true if this is a hosted
+   environment.  */
+void
+cpp_init_builtins (cpp_reader *pfile, int hosted)
+{
+  cpp_init_special_builtins (pfile);
+
+  if (!CPP_OPTION (pfile, traditional)
+      && (! CPP_OPTION (pfile, stdc_0_in_system_headers)
+	  || CPP_OPTION (pfile, std)))
+    _cpp_define_builtin (pfile, "__STDC__ 1");
 
   if (CPP_OPTION (pfile, cplusplus))
     _cpp_define_builtin (pfile, "__cplusplus 1");
@@ -621,7 +629,8 @@ post_options (cpp_reader *pfile)
      preprocessed text.  Read preprocesed source in ISO mode.  */
   if (CPP_OPTION (pfile, preprocessed))
     {
-      pfile->state.prevent_expansion = 1;
+      if (!CPP_OPTION (pfile, directives_only))
+	pfile->state.prevent_expansion = 1;
       CPP_OPTION (pfile, traditional) = 0;
     }
 
Index: libcpp/directives-only.c
===================================================================
--- libcpp/directives-only.c	(revision 0)
+++ libcpp/directives-only.c	(revision 0)
@@ -0,0 +1,185 @@
+/* CPP Library - directive only preprocessing for distributed compilation.
+   Copyright (C) 2006, 2007
+   Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.
+
+This program is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 2, or (at your option) any
+later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */
+
+#include "config.h"
+#include "system.h"
+#include "cpplib.h"
+#include "internal.h"
+
+/* DO (Directive only) flags. */
+#define DO_BOL		 (1 << 0) /* At the beginning of a logical line. */
+#define DO_STRING	 (1 << 1) /* In a string constant. */
+#define DO_CHAR		 (1 << 2) /* In a character constant. */
+#define DO_BLOCK_COMMENT (1 << 3) /* In a block comment. */
+#define DO_LINE_COMMENT	 (1 << 4) /* In a single line "//-style" comment. */
+
+#define DO_LINE_SPECIAL (DO_STRING | DO_CHAR | DO_LINE_COMMENT)
+#define DO_SPECIAL	(DO_LINE_SPECIAL | DO_BLOCK_COMMENT)
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+void
+_cpp_preprocess_dir_only (cpp_reader *pfile,
+			  const struct _cpp_dir_only_callbacks *cb)
+{
+  struct cpp_buffer *buffer;
+  const unsigned char *cur, *base, *next_line, *rlimit;
+  cppchar_t c, last_c;
+  unsigned flags;
+  int lines, escaped_lines;
+
+ restart:
+  /* Buffer initialization ala _cpp_clean_line(). */
+  buffer = pfile->buffer;
+  buffer->cur_note = buffer->notes_used = 0;
+  buffer->cur = buffer->line_base = buffer->next_line;
+  buffer->need_line = false;
+
+  /* Scan initialization. */
+  next_line = cur = base = buffer->cur;
+  rlimit = buffer->rlimit;
+  flags = DO_BOL;
+  lines = 0;
+  escaped_lines = 0;
+
+  for (last_c = '\n', c = *cur; cur < rlimit; last_c = c, c = *++cur)
+    {
+      if (last_c == '#')
+	{
+	  if ((c != '#') && (flags & DO_BOL) && !(flags & DO_SPECIAL))
+	  {
+	    struct line_maps *line_table;
+
+	    if (!pfile->state.skipping && next_line != base)
+	      cb->print_lines (lines, base, next_line - base);
+
+	    /* Prep things for directive handling. */
+	    buffer->next_line = next_line;
+	    buffer->need_line = true;
+	    _cpp_get_fresh_line (pfile);
+	    while (*buffer->cur++ != '#');
+
+	    _cpp_handle_directive (pfile, 0 /* ignore indented */);
+
+	    /* Sanitize the line settings.  Duplicate #include's can mess
+	       things up. */
+	    line_table = pfile->line_table;
+	    line_table->highest_location = line_table->highest_line;
+
+	    /* The if block prevents us from outputing line information when
+	       the file ends with a directive and no newline.  Note that we
+	       must use pfile->buffer, not buffer. */
+	    if (pfile->buffer->cur != pfile->buffer->rlimit)
+	      cb->maybe_print_line (pfile->line_table->highest_line);
+
+	    goto restart;
+	  }
+
+	  flags &= ~DO_BOL;
+	}
+
+      switch (c)
+	{
+	case '/':
+	  flags &= ~DO_BOL;
+
+	  if ((flags & DO_BLOCK_COMMENT) && last_c == '*')
+	    flags &= ~DO_BLOCK_COMMENT;
+	  else if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_LINE_COMMENT;
+
+	  break;
+
+	case '*':
+	  flags &= ~DO_BOL;
+
+	  if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_BLOCK_COMMENT;
+
+	  break;
+
+	case '\'':
+	case '"':
+	  {
+	    unsigned state = (c == '"') ? DO_STRING : DO_CHAR;
+
+	    flags &= ~DO_BOL;
+
+	    if (!(flags & DO_SPECIAL))
+	      flags |= state;
+	    else if ((flags & state) && last_c != '\\')
+	      flags &= ~state;
+
+	    break;
+	  }
+
+	case '\\':
+	  {
+	    const unsigned char *tmp = cur + 1;
+
+	    /* Skip over escaped newlines. */
+	    while (is_nvspace (*tmp) && tmp < rlimit)
+	      tmp++;
+	    if (*tmp == '\r')
+	      tmp++;
+	    if (*tmp == '\n' && tmp < rlimit)
+	      {
+		escaped_lines++;
+		cur = tmp;
+		c = last_c;
+	      }
+	    else if ((flags & (DO_STRING | DO_CHAR)) && last_c == '\\')
+	      c = '\0';
+
+	    break;
+	  }
+
+	case '\n':
+	  do
+	    {
+	      CPP_INCREMENT_LINE (pfile, 0);
+	      lines++;
+	    }
+	  while (escaped_lines--);
+	  escaped_lines = 0;
+	  next_line = cur + 1;
+	  flags &= ~DO_LINE_SPECIAL;
+	  flags |= DO_BOL;
+	  break;
+
+	case '#':
+	  /* Don't update DO_BOL yet. */
+	  break;
+
+	case ' ': case '\t': case '\f': case '\v': case '\0':
+	  break;
+
+	default:
+	  flags &= ~DO_BOL;
+	  break;
+	}
+    }
+
+  if (!pfile->state.skipping && cur != base)
+    cb->print_lines (lines, base, cur - base);
+
+  _cpp_pop_buffer (pfile);
+  if (pfile->buffer)
+    goto restart;
+}
Index: libcpp/internal.h
===================================================================
--- libcpp/internal.h	(revision 121856)
+++ libcpp/internal.h	(working copy)
@@ -568,6 +568,17 @@ extern void _cpp_do_file_change (cpp_rea
 				 unsigned int, unsigned int);
 extern void _cpp_pop_buffer (cpp_reader *);
 
+/* In directives.c */
+struct _cpp_dir_only_callbacks
+{
+  /* Called to print a block of lines. */
+  void (*print_lines) (int, const void *, size_t);
+  void (*maybe_print_line) (source_location);
+};
+
+extern void _cpp_preprocess_dir_only (cpp_reader *,
+				      const struct _cpp_dir_only_callbacks *);
+
 /* In traditional.c.  */
 extern bool _cpp_scan_out_logical_line (cpp_reader *, cpp_macro *);
 extern bool _cpp_read_logical_line_trad (cpp_reader *);

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Alexandre Oliva]

On Feb 12, 2007, "Ollie Wild" <aaw@google.com> wrote:

> Usage is straightforward.  The -directives-only flag is simply added
> to the compiler options passed to distcc or ccache.

This flag name conflicts with the existing -dCHARS and -dLETTERS
options already present in cpp and gcc, respectively.

I suggest something like -dF (fast) or some -f option.

> Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.

Have you run the C testsuite using this option?  It would be nice to
make sure it doesn't break any of the corner cases of tokenization.

You should also make sure it doesn't interfere destructively with
the -dCHARS options documented in the cpp manual.  It seems to me like
it might.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

> > Usage is straightforward.  The -directives-only flag is simply added
> > to the compiler options passed to distcc or ccache.
>
> This flag name conflicts with the existing -dCHARS and -dLETTERS
> options already present in cpp and gcc, respectively.

Ah, I was unaware of the -dLETTERS option in gcc.  At the moment, it
doesn't actually interfere -- both versions work and are callable --
but I agree that this could be confusing.

> I suggest something like -dF (fast) or some -f option.

I don't like the -dF suggestion because this doesn't fit the mold of a
dump operation.  During preprocessing, it does share the macro output
portion, but that's about it.  It also includes special handling
during the compilation stage: if -fpreprocessed is set, it disables
automatic definition of most builtin macros.

How does -fdirectives-only sound?

> > Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.
>
> Have you run the C testsuite using this option?  It would be nice to
> make sure it doesn't break any of the corner cases of tokenization.

I did add a couple test cases specific to this patch.  I also compiled
a large subset of Google's internal codebase with this.  The test case
includes the various issues I ran into, some feedback from Ian Taylor,
as well as some corner cases of my own devising.

The directives-only option is a null-op unless compilation is done as
a two stage process: first preprocessing, then compilation.  I'd need
to modify the C testsuite accordingly in order to do a full run.  I'm
not terribly familiar with dejagnu in general and gcc's test code in
particular, but if you could give me some pointers, I'd be willing to
try it out.


> You should also make sure it doesn't interfere destructively with
> the -dCHARS options documented in the cpp manual.  It seems to me like
> it might.

It does interfere with -dN and -dD.  These are fundamentally
incompatible with directives-only, though, since directives-only
preprocessing requires macro definitions to be output (to enable
deferral of macro expansion).  At the moment, directives-only takes
precedence, though this could be changed.

What would you suggest?  Is there a good rationale for choosing one
over the other?

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Eric Christopher]

>
> How does -fdirectives-only sound?
>
> What would you suggest?  Is there a good rationale for choosing one
> over the other?

I think this sounds good. Lacking a libcpp maintainer we'll have to  
get someone global to approve it, but IMO this sounds better than  
anything else.

-eric

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

> > How does -fdirectives-only sound?
> >
> > What would you suggest?  Is there a good rationale for choosing one
> > over the other?
>
> I think this sounds good. Lacking a libcpp maintainer we'll have to
> get someone global to approve it, but IMO this sounds better than
> anything else.

OK.  I've attached an updated patch which replaces the
-directives-only option with a -fdirectives-only option.  Otherwise,
it's identical to the previous patch.

Ollie

:ADDPATCH libcpp:

2007-02-12  Ollie Wild  <aaw@google.com>

       * gcc.dg/cpp/dir-only-1.c: New test.
       * gcc.dg/cpp/dir-only-1.h: New file.
       * gcc.dg/cpp/dir-only-2.c: New test.

2007-02-12  Ollie Wild  <aaw@google.com>

       * directives-only.c: New file.
       * internal.h (struct _cpp_dir_only_callbacks): New.
       (_cpp_preprocess_dir_only): New function.
       * files.c (_cpp_stack_file): Add directives_only check.
       * cpplib.h (cpp_init_special_builtins): New function.
       (cpp_options): Add directives_only.
       * init.c (cpp_init_special_builtins): New function.
       (cpp_init_builtins): Move builtin_array initialization to
       cpp_init_special_builtins.
       (post_options): Check directives_only before setting
       pfile->state.prevent_expansion = 1.
       * Makefile.in (libcpp_a_OBJS): Add directives-only.o.
       (libcpp_a_SOURCES): Add directives-only.c.

2007-02-12  Ollie Wild  <aaw@google.com>

       * c-ppoutput.c (print_lines_directives_only): New function.
       (scan_translation_unit_directives_only): New function.
       (cb_define_directives_only): New function.
       (preprocess_file): Add call to scan_translation_unit_directives_only.
       (init_pp_ouput): Register cb_define_directives_only.
       * c-opts.c (c_common_handle_option): Add OPT_fdirectives_only.
       (finish_options): Add builtin macro initialization for
       -fdirectives-only + -fpreprocessed.
       * c.opt (fdirectives-only): New.
       * doc/cppopts.texi (fdirectives-only): New.

[-- Attachment #2: directives-only.txt --]
[-- Type: text/plain, Size: 18321 bytes --]

Index: gcc/doc/cppopts.texi
===================================================================
--- gcc/doc/cppopts.texi	(revision 121893)
+++ gcc/doc/cppopts.texi	(working copy)
@@ -510,6 +510,22 @@ Search @var{dir} only for header files r
 @xref{Search Path}.
 @end ifset
 
+@item -fdirectives-only
+@opindex fdirectives-only
+This option provides a simplified preprocessor to improve the
+performance of distributed build systems such as distcc.  It's
+behavior depends on a number of other flags.
+
+If the @option{-E} option is enabled, it suppresses things like macro
+expansion, trigraph conversion, and escaped newline splicing
+outside of directives.  All directives are processed normally, except that
+macro definitions are output similar to the @option{-dD} option.
+
+If the @option{-fpreprocessed} option is enabled, it suppresses
+predefinition of most builtin and command line macros.  This
+prevents duplicate definition of macros output with the @option{-E}
+option.
+
 @item -fdollars-in-identifiers
 @opindex fdollars-in-identifiers
 @anchor{fdollars-in-identifiers}
Index: gcc/testsuite/gcc.dg/cpp/dir-only-1.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-1.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-1.c	(revision 0)
@@ -0,0 +1,67 @@
+/* Copyright 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* Tests scan_translation_unit_directives_only()'s handling of corner cases. */
+
+/* Ignore directives inside block comments...
+#error directive inside block comment
+*/
+
+// Escaped newline doesn't terminate line comment \
+#error directive inside line comment
+
+/* A comment canot start inside a string. */
+const char *c1 = "/*";
+#define NOT_IN_COMMENT
+const char *c2 = "*/";
+#ifndef NOT_IN_COMMENT
+#error Comment started inside a string literal
+#endif
+
+/* Escaped newline handling. */
+int i; \
+#error ignored escaped newline
+  \
+  \
+#define BOL
+#ifndef BOL
+#error escaped newline did not preserve beginning of line
+#endif
+
+/* Handles \\ properly at the end of a string. */
+"string ends in \\"/*
+#error Missed string terminator.
+*/
+
+/* Handles macro expansion in preprocessing directives. */
+#define HEADER "dir-only-1.h"
+#include HEADER
+#ifndef GOT_HEADER
+#error Failed to include header.
+#endif
+
+/\
+*
+#define IN_COMMENT
+*/
+#ifdef IN_COMMENT
+#error Escaped newline breaks block comment initiator.
+#endif
+
+/*
+*\
+/
+#define NOT_IN_COMMENT2
+/**/
+#ifndef NOT_IN_COMMENT2
+#error Escaped newline breaks block comment terminator.
+#endif
+
+/* Test escaped newline inside character escape sequence. */
+"\\
+\"/*
+#error Missed string terminator
+*/
Index: gcc/testsuite/gcc.dg/cpp/dir-only-1.h
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-1.h	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-1.h	(revision 0)
@@ -0,0 +1,3 @@
+/* Copyright 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+#define GOT_HEADER
Index: gcc/testsuite/gcc.dg/cpp/dir-only-2.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-2.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-2.c	(revision 0)
@@ -0,0 +1,10 @@
+/* Copyright 2006, 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fpreprocessed -fdirectives-only -DNOT_SET" } */
+
+/* Tests -fdirectives-only + -fpreprocessed. */
+
+/* Check this is not defined. */
+#define NOT_SET
Index: gcc/c.opt
===================================================================
--- gcc/c.opt	(revision 121893)
+++ gcc/c.opt	(working copy)
@@ -504,6 +504,10 @@ fdefault-inline
 C++ ObjC++
 Inline member functions by default
 
+fdirectives-only
+C ObjC C++ ObjC++
+Preprocess directives only.
+
 fdollars-in-identifiers
 C ObjC C++ ObjC++
 Permit '$' as an identifier character
Index: gcc/c-opts.c
===================================================================
--- gcc/c-opts.c	(revision 121893)
+++ gcc/c-opts.c	(working copy)
@@ -607,6 +607,10 @@ c_common_handle_option (size_t scode, co
 	disable_builtin_function (arg);
       break;
 
+    case OPT_fdirectives_only:
+      cpp_opts->directives_only = 1;
+      break;
+
     case OPT_fdollars_in_identifiers:
       cpp_opts->dollars_in_ident = value;
       break;
@@ -1461,6 +1465,8 @@ finish_options (void)
 	    }
 	}
     }
+  else if (cpp_opts->directives_only)
+    cpp_init_special_builtins (parse_in);
 
   include_cursor = 0;
   push_command_line_include ();
Index: gcc/c-ppoutput.c
===================================================================
--- gcc/c-ppoutput.c	(revision 121893)
+++ gcc/c-ppoutput.c	(working copy)
@@ -41,6 +41,8 @@ static struct
 
 /* General output routines.  */
 static void scan_translation_unit (cpp_reader *);
+static void print_lines_directives_only (int, const void *, size_t);
+static void scan_translation_unit_directives_only (cpp_reader *);
 static void scan_translation_unit_trad (cpp_reader *);
 static void account_for_newlines (const unsigned char *, size_t);
 static int dump_macro (cpp_reader *, cpp_hashnode *, void *);
@@ -52,6 +54,8 @@ static void maybe_print_line (source_loc
    in specific modes.  */
 static void cb_line_change (cpp_reader *, const cpp_token *, int);
 static void cb_define (cpp_reader *, source_location, cpp_hashnode *);
+static void cb_define_directives_only (cpp_reader *, source_location,
+				       cpp_hashnode *);
 static void cb_undef (cpp_reader *, source_location, cpp_hashnode *);
 static void cb_include (cpp_reader *, source_location, const unsigned char *,
 			const char *, int, const cpp_token **);
@@ -75,6 +79,8 @@ preprocess_file (cpp_reader *pfile)
     }
   else if (cpp_get_options (pfile)->traditional)
     scan_translation_unit_trad (pfile);
+  else if (cpp_get_options (pfile)->directives_only)
+    scan_translation_unit_directives_only (pfile);
   else
     scan_translation_unit (pfile);
 
@@ -114,7 +120,12 @@ init_pp_output (FILE *out_stream)
       cb->read_pch = cb_read_pch;
     }
 
-  if (flag_dump_macros == 'N' || flag_dump_macros == 'D')
+  if (cpp_get_options (parse_in)->directives_only)
+    {
+      cb->define = cb_define_directives_only;
+      cb->undef  = cb_undef;
+    }
+  else if (flag_dump_macros == 'N' || flag_dump_macros == 'D')
     {
       cb->define = cb_define;
       cb->undef  = cb_undef;
@@ -179,6 +190,26 @@ scan_translation_unit (cpp_reader *pfile
     }
 }
 
+static void
+print_lines_directives_only (int lines, const void *buf, size_t size)
+{
+  print.src_line += lines;
+  fwrite (buf, 1, size, print.outf);
+}
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+static void
+scan_translation_unit_directives_only (cpp_reader *pfile)
+{
+  struct _cpp_dir_only_callbacks cb;
+
+  cb.print_lines = print_lines_directives_only;
+  cb.maybe_print_line = maybe_print_line;
+
+  _cpp_preprocess_dir_only (pfile, &cb);
+}
+
 /* Adjust print.src_line for newlines embedded in output.  */
 static void
 account_for_newlines (const unsigned char *str, size_t len)
@@ -324,6 +355,20 @@ cb_define (cpp_reader *pfile, source_loc
     fputs ((const char *) NODE_NAME (node), print.outf);
 
   putc ('\n', print.outf);
+  print.src_line++;
+}
+
+static void
+cb_define_directives_only (cpp_reader *pfile, source_location line,
+			   cpp_hashnode *node)
+{
+  maybe_print_line (line);
+  fputs ("#define ", print.outf);
+
+  fputs ((const char *) cpp_macro_definition (pfile, node),
+	 print.outf);
+
+  putc ('\n', print.outf);
   if (linemap_lookup (&line_table, line)->to_line != 0)
     print.src_line++;
 }
Index: libcpp/Makefile.in
===================================================================
--- libcpp/Makefile.in	(revision 121893)
+++ libcpp/Makefile.in	(working copy)
@@ -69,14 +69,14 @@ INCLUDES = -I$(srcdir) -I. -I$(srcdir)/.
 
 ALL_CFLAGS = $(CFLAGS) $(WARN_CFLAGS) $(INCLUDES) $(CPPFLAGS)
 
-libcpp_a_OBJS = charset.o directives.o errors.o expr.o files.o \
-	identifiers.o init.o lex.o line-map.o macro.o mkdeps.o \
-	pch.o symtab.o traditional.o
+libcpp_a_OBJS = charset.o directives.o directives-only.o errors.o \
+	expr.o files.o identifiers.o init.o lex.o line-map.o macro.o \
+	mkdeps.o pch.o symtab.o traditional.o
 makedepend_OBJS = makedepend.o
 
-libcpp_a_SOURCES = charset.c directives.c errors.c expr.c files.c \
-	identifiers.c init.c lex.c line-map.c macro.c mkdeps.c \
-	pch.c symtab.c traditional.c
+libcpp_a_SOURCES = charset.c directives.c directives-only.c errors.c \
+	expr.c files.c identifiers.c init.c lex.c line-map.c macro.c \
+	mkdeps.c pch.c symtab.c traditional.c
 
 all: libcpp.a makedepend$(EXEEXT) $(USED_CATALOGS)
 
Index: libcpp/files.c
===================================================================
--- libcpp/files.c	(revision 121893)
+++ libcpp/files.c	(working copy)
@@ -722,7 +722,8 @@ _cpp_stack_file (cpp_reader *pfile, _cpp
 
   /* Stack the buffer.  */
   buffer = cpp_push_buffer (pfile, file->buffer, file->st.st_size,
-			    CPP_OPTION (pfile, preprocessed));
+			    CPP_OPTION (pfile, preprocessed)
+			    && !CPP_OPTION (pfile, directives_only));
   buffer->file = file;
   buffer->sysp = sysp;
 
Index: libcpp/include/cpplib.h
===================================================================
--- libcpp/include/cpplib.h	(revision 121893)
+++ libcpp/include/cpplib.h	(working copy)
@@ -440,6 +440,9 @@ struct cpp_options
 
   /* True means error callback should be used for diagnostics.  */
   bool client_diagnostic;
+
+  /* True disables tokenization outside of preprocessing directives. */
+  bool directives_only;
 };
 
 /* Callback for header lookup for HEADER, which is the name of a
@@ -643,6 +646,10 @@ extern struct deps *cpp_get_deps (cpp_re
    too.  If there was an error opening the file, it returns NULL.  */
 extern const char *cpp_read_main_file (cpp_reader *, const char *);
 
+/* Set up built-ins with special behavior.  Use cpp_init_builtins()
+   instead unless your know what you are doing.  */
+extern void cpp_init_special_builtins (cpp_reader *);
+
 /* Set up built-ins like __FILE__.  */
 extern void cpp_init_builtins (cpp_reader *, int);
 
Index: libcpp/init.c
===================================================================
--- libcpp/init.c	(revision 121893)
+++ libcpp/init.c	(working copy)
@@ -349,11 +349,8 @@ mark_named_operators (cpp_reader *pfile)
     }
 }
 
-/* Read the builtins table above and enter them, and language-specific
-   macros, into the hash table.  HOSTED is true if this is a hosted
-   environment.  */
 void
-cpp_init_builtins (cpp_reader *pfile, int hosted)
+cpp_init_special_builtins (cpp_reader *pfile)
 {
   const struct builtin *b;
   size_t n = ARRAY_SIZE (builtin_array);
@@ -362,10 +359,7 @@ cpp_init_builtins (cpp_reader *pfile, in
     n -= 2;
   else if (! CPP_OPTION (pfile, stdc_0_in_system_headers)
 	   || CPP_OPTION (pfile, std))
-    {
-      n--;
-      _cpp_define_builtin (pfile, "__STDC__ 1");
-    }
+    n--;
 
   for (b = builtin_array; b < builtin_array + n; b++)
     {
@@ -374,6 +368,20 @@ cpp_init_builtins (cpp_reader *pfile, in
       hp->flags |= NODE_BUILTIN | NODE_WARN;
       hp->value.builtin = (enum builtin_type) b->value;
     }
+}
+
+/* Read the builtins table above and enter them, and language-specific
+   macros, into the hash table.  HOSTED is true if this is a hosted
+   environment.  */
+void
+cpp_init_builtins (cpp_reader *pfile, int hosted)
+{
+  cpp_init_special_builtins (pfile);
+
+  if (!CPP_OPTION (pfile, traditional)
+      && (! CPP_OPTION (pfile, stdc_0_in_system_headers)
+	  || CPP_OPTION (pfile, std)))
+    _cpp_define_builtin (pfile, "__STDC__ 1");
 
   if (CPP_OPTION (pfile, cplusplus))
     _cpp_define_builtin (pfile, "__cplusplus 1");
@@ -621,7 +629,8 @@ post_options (cpp_reader *pfile)
      preprocessed text.  Read preprocesed source in ISO mode.  */
   if (CPP_OPTION (pfile, preprocessed))
     {
-      pfile->state.prevent_expansion = 1;
+      if (!CPP_OPTION (pfile, directives_only))
+	pfile->state.prevent_expansion = 1;
       CPP_OPTION (pfile, traditional) = 0;
     }
 
Index: libcpp/directives-only.c
===================================================================
--- libcpp/directives-only.c	(revision 0)
+++ libcpp/directives-only.c	(revision 0)
@@ -0,0 +1,185 @@
+/* CPP Library - directive only preprocessing for distributed compilation.
+   Copyright (C) 2006, 2007
+   Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.
+
+This program is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 2, or (at your option) any
+later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */
+
+#include "config.h"
+#include "system.h"
+#include "cpplib.h"
+#include "internal.h"
+
+/* DO (Directive only) flags. */
+#define DO_BOL		 (1 << 0) /* At the beginning of a logical line. */
+#define DO_STRING	 (1 << 1) /* In a string constant. */
+#define DO_CHAR		 (1 << 2) /* In a character constant. */
+#define DO_BLOCK_COMMENT (1 << 3) /* In a block comment. */
+#define DO_LINE_COMMENT	 (1 << 4) /* In a single line "//-style" comment. */
+
+#define DO_LINE_SPECIAL (DO_STRING | DO_CHAR | DO_LINE_COMMENT)
+#define DO_SPECIAL	(DO_LINE_SPECIAL | DO_BLOCK_COMMENT)
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+void
+_cpp_preprocess_dir_only (cpp_reader *pfile,
+			  const struct _cpp_dir_only_callbacks *cb)
+{
+  struct cpp_buffer *buffer;
+  const unsigned char *cur, *base, *next_line, *rlimit;
+  cppchar_t c, last_c;
+  unsigned flags;
+  int lines, escaped_lines;
+
+ restart:
+  /* Buffer initialization ala _cpp_clean_line(). */
+  buffer = pfile->buffer;
+  buffer->cur_note = buffer->notes_used = 0;
+  buffer->cur = buffer->line_base = buffer->next_line;
+  buffer->need_line = false;
+
+  /* Scan initialization. */
+  next_line = cur = base = buffer->cur;
+  rlimit = buffer->rlimit;
+  flags = DO_BOL;
+  lines = 0;
+  escaped_lines = 0;
+
+  for (last_c = '\n', c = *cur; cur < rlimit; last_c = c, c = *++cur)
+    {
+      if (last_c == '#')
+	{
+	  if ((c != '#') && (flags & DO_BOL) && !(flags & DO_SPECIAL))
+	  {
+	    struct line_maps *line_table;
+
+	    if (!pfile->state.skipping && next_line != base)
+	      cb->print_lines (lines, base, next_line - base);
+
+	    /* Prep things for directive handling. */
+	    buffer->next_line = next_line;
+	    buffer->need_line = true;
+	    _cpp_get_fresh_line (pfile);
+	    while (*buffer->cur++ != '#');
+
+	    _cpp_handle_directive (pfile, 0 /* ignore indented */);
+
+	    /* Sanitize the line settings.  Duplicate #include's can mess
+	       things up. */
+	    line_table = pfile->line_table;
+	    line_table->highest_location = line_table->highest_line;
+
+	    /* The if block prevents us from outputing line information when
+	       the file ends with a directive and no newline.  Note that we
+	       must use pfile->buffer, not buffer. */
+	    if (pfile->buffer->cur != pfile->buffer->rlimit)
+	      cb->maybe_print_line (pfile->line_table->highest_line);
+
+	    goto restart;
+	  }
+
+	  flags &= ~DO_BOL;
+	}
+
+      switch (c)
+	{
+	case '/':
+	  flags &= ~DO_BOL;
+
+	  if ((flags & DO_BLOCK_COMMENT) && last_c == '*')
+	    flags &= ~DO_BLOCK_COMMENT;
+	  else if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_LINE_COMMENT;
+
+	  break;
+
+	case '*':
+	  flags &= ~DO_BOL;
+
+	  if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_BLOCK_COMMENT;
+
+	  break;
+
+	case '\'':
+	case '"':
+	  {
+	    unsigned state = (c == '"') ? DO_STRING : DO_CHAR;
+
+	    flags &= ~DO_BOL;
+
+	    if (!(flags & DO_SPECIAL))
+	      flags |= state;
+	    else if ((flags & state) && last_c != '\\')
+	      flags &= ~state;
+
+	    break;
+	  }
+
+	case '\\':
+	  {
+	    const unsigned char *tmp = cur + 1;
+
+	    /* Skip over escaped newlines. */
+	    while (is_nvspace (*tmp) && tmp < rlimit)
+	      tmp++;
+	    if (*tmp == '\r')
+	      tmp++;
+	    if (*tmp == '\n' && tmp < rlimit)
+	      {
+		escaped_lines++;
+		cur = tmp;
+		c = last_c;
+	      }
+	    else if ((flags & (DO_STRING | DO_CHAR)) && last_c == '\\')
+	      c = '\0';
+
+	    break;
+	  }
+
+	case '\n':
+	  do
+	    {
+	      CPP_INCREMENT_LINE (pfile, 0);
+	      lines++;
+	    }
+	  while (escaped_lines--);
+	  escaped_lines = 0;
+	  next_line = cur + 1;
+	  flags &= ~DO_LINE_SPECIAL;
+	  flags |= DO_BOL;
+	  break;
+
+	case '#':
+	  /* Don't update DO_BOL yet. */
+	  break;
+
+	case ' ': case '\t': case '\f': case '\v': case '\0':
+	  break;
+
+	default:
+	  flags &= ~DO_BOL;
+	  break;
+	}
+    }
+
+  if (!pfile->state.skipping && cur != base)
+    cb->print_lines (lines, base, cur - base);
+
+  _cpp_pop_buffer (pfile);
+  if (pfile->buffer)
+    goto restart;
+}
Index: libcpp/internal.h
===================================================================
--- libcpp/internal.h	(revision 121893)
+++ libcpp/internal.h	(working copy)
@@ -568,6 +568,17 @@ extern void _cpp_do_file_change (cpp_rea
 				 unsigned int, unsigned int);
 extern void _cpp_pop_buffer (cpp_reader *);
 
+/* In directives.c */
+struct _cpp_dir_only_callbacks
+{
+  /* Called to print a block of lines. */
+  void (*print_lines) (int, const void *, size_t);
+  void (*maybe_print_line) (source_location);
+};
+
+extern void _cpp_preprocess_dir_only (cpp_reader *,
+				      const struct _cpp_dir_only_callbacks *);
+
 /* In traditional.c.  */
 extern bool _cpp_scan_out_logical_line (cpp_reader *, cpp_macro *);
 extern bool _cpp_read_logical_line_trad (cpp_reader *);

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Alexandre Oliva]

On Feb 13, 2007, "Ollie Wild" <aaw@google.com> wrote:

> I did add a couple test cases specific to this patch.  I also compiled
> a large subset of Google's internal codebase with this.

I was more worried about oddities that probably no sane program should
rely on but that Standard-compliant compilers must handle correctly.

> The directives-only option is a null-op unless compilation is done as
> a two stage process: first preprocessing, then compilation.  I'd need
> to modify the C testsuite accordingly in order to do a full run.  I'm
> not terribly familiar with dejagnu in general and gcc's test code in
> particular, but if you could give me some pointers, I'd be willing to
> try it out.

For a one-shot run (probably good enough) I'd try replacing xgcc in
the build tree with a script that runs ccache xgcc.orig in such a way
that -fdirectives-only is exercised, and running the C and C++
testsuites to get an idea of whether the patch breaks anything covered
by the GCC testsuite.

> It does interfere with -dN and -dD.  These are fundamentally
> incompatible with directives-only, though, since directives-only
> preprocessing requires macro definitions to be output (to enable
> deferral of macro expansion).  At the moment, directives-only takes
> precedence, though this could be changed.

Ideally we'd get errors for conflicting options.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

> For a one-shot run (probably good enough) I'd try replacing xgcc in
> the build tree with a script that runs ccache xgcc.orig in such a way
> that -fdirectives-only is exercised, and running the C and C++
> testsuites to get an idea of whether the patch breaks anything covered
> by the GCC testsuite.

Thanks.  I ran the gcc and g++ testsuites using your suggestion.
Aside from cpp tests, the results with and without -fdirectives-only
are identical.

The cpp tests are more complicated.  A lot of these fail (about 136
for gcc), but the vast majority are expected and benign.  For example,
since -fdirectives-only moves trigraph processing to the compilation
phase, any test which look for a trigraph warning fails -- the warning
is output by the subsequent compilation phase.

That said, I did run into a few real issues.  The main one is my
handling of include guards.  I'm not properly detecting cases where
code appears outside of an include guard (cf. mi7.c).  There are also
some warnings which I don't handle but easily could (e.g. unterminated
comments) and some incompatible command line options (e.g. -Wunused
and -dD) which should emit warnings or errors.

I'll clean these up and resubmit.

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

[-- Attachment #1: Type: text/plain, Size: 3369 bytes --]

> That said, I did run into a few real issues.  The main one is my
> handling of include guards.  I'm not properly detecting cases where
> code appears outside of an include guard (cf. mi7.c).  There are also
> some warnings which I don't handle but easily could (e.g. unterminated
> comments) and some incompatible command line options (e.g. -Wunused
> and -dD) which should emit warnings or errors.
>
> I'll clean these up and resubmit.

I've finished updating the patch.  Here's an overview of the changes I've made:

- Added errors if -fdirectives-only is used in conjunction with -traditional or
  -Wunused-macros.
- Fixed include guard handling.  Faulty (or missing) include guards
now correctly trigger
  multiple inclusion.
- Adds an error for unterminated block comments.
- Fixes incorrect handling of preprocessing directives preceded by
block comments.
- Allows -dCHARS arguments to be used in conjunction with -fdirectives-only.
- Adds support for '-E -fdirectives-only -fpreprocessed', which causes
a two-stage pass to
  return the same output as a single '-E' pass.
- Adds additional test cases.

Using ccache to enforce two-stage processing, the new code still
correctly executes all non-cpp test cases.  The cpp test cases which
fail now do so in predictable and desirable fashion.

Ollie

:ADDPATCH libcpp:

2007-02-27  Ollie Wild  <aaw@google.com>

        * gcc.dg/cpp/dir-only-1.c: New test.
        * gcc.dg/cpp/dir-only-1.h: New file.
        * gcc.dg/cpp/dir-only-2.c: New test.
        * gcc.dg/cpp/dir-only-3.c: New test.
        * gcc.dg/cpp/dir-only-3a.h: New file.
        * gcc.dg/cpp/dir-only-3b.h: New file.
        * gcc.dg/cpp/dir-only-4.c: New test.
        * gcc.dg/cpp/dir-only-5.c: New test.
        * gcc.dg/cpp/dir-only-6.c: New test.

2007-02-27  Ollie Wild  <aaw@google.com>

        * directives-only.c: New file.
        * internal.h (struct _cpp_dir_only_callbacks): New.
        (_cpp_preprocess_dir_only): New function.
        * directives.c (_cpp_handle_directive): Check directives_only before
        disabling execution of indented directives.
        * files.c (_cpp_stack_file): Add directives_only check.
        * include/cpplib.h (struct cpp_options): Add directives_only.
        (cpp_init_special_builtins): New function.
        * init.c (cpp_init_special_builtins): New function.
        (cpp_init_builtins): Move builtin_array initialization to
        cpp_init_special_builtins.
        (post_options): Check directives_only before setting
        pfile->state.prevent_expansion = 1.
        * Makefile.in (libcpp_a_OBJS): Add directives-only.o.
        (libcpp_a_SOURCES): Add directives-only.c.

2007-02-27  Ollie Wild  <aaw@google.com>

        * c-ppoutput.c (print_lines_directives_only): New function.
        (scan_translation_unit_directives_only): New function.
        (preprocess_file): Add call to scan_translation_unit_directives_only.
        * c-opts.c (c_common_handle_option): Add OPT_fdirectives_only.
        (sanitize_cpp_opts): Add default flag_dump_macros setting for
        -fdirectives-only.  Add errors for -fdirectives-only conflict with
        -Wunused-macros and -traditional.
        (finish_options): Add builtin macro initialization for
        -fdirectives-only + -fpreprocessed.
        * c.opt (fdirectives-only): New.
        * doc/cppopts.texi (fdirectives-only): New.

[-- Attachment #2: directives-only.txt --]
[-- Type: text/plain, Size: 23253 bytes --]

Index: gcc/doc/cppopts.texi
===================================================================
--- gcc/doc/cppopts.texi	(revision 122387)
+++ gcc/doc/cppopts.texi	(working copy)
@@ -510,6 +510,22 @@ Search @var{dir} only for header files r
 @xref{Search Path}.
 @end ifset
 
+@item -fdirectives-only
+@opindex fdirectives-only
+This option provides a simplified preprocessor to improve the
+performance of distributed build systems such as distcc.  It's
+behavior depends on a number of other flags.
+
+If the @option{-E} option is enabled, it suppresses things like macro
+expansion, trigraph conversion, and escaped newline splicing
+outside of directives.  All directives are processed normally, except that
+macro definitions are output similar to the @option{-dD} option.
+
+If the @option{-fpreprocessed} option is enabled, it suppresses
+predefinition of most builtin and command line macros.  This
+prevents duplicate definition of macros output with the @option{-E}
+option.
+
 @item -fdollars-in-identifiers
 @opindex fdollars-in-identifiers
 @anchor{fdollars-in-identifiers}
Index: gcc/testsuite/gcc.dg/cpp/dir-only-1.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-1.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-1.c	(revision 0)
@@ -0,0 +1,73 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* Tests scan_translation_unit_directives_only()'s handling of corner cases. */
+
+/* Ignore directives inside block comments...
+#error directive inside block comment
+*/
+
+// Escaped newline doesn't terminate line comment \
+#error directive inside line comment
+
+/* A comment canot start inside a string. */
+const char *c1 = "/*";
+#define NOT_IN_COMMENT
+const char *c2 = "*/";
+#ifndef NOT_IN_COMMENT
+#error Comment started inside a string literal
+#endif
+
+/* Escaped newline handling. */
+int i; \
+#error ignored escaped newline
+  \
+  \
+#define BOL
+#ifndef BOL
+#error escaped newline did not preserve beginning of line
+#endif
+
+/* Handles \\ properly at the end of a string. */
+"string ends in \\"/*
+#error Missed string terminator.
+*/
+
+/* Handles macro expansion in preprocessing directives. */
+#define HEADER "dir-only-1.h"
+#include HEADER
+#ifndef GOT_HEADER
+#error Failed to include header.
+#endif
+
+/\
+*
+#define IN_COMMENT
+*/
+#ifdef IN_COMMENT
+#error Escaped newline breaks block comment initiator.
+#endif
+
+/*
+*\
+/
+#define NOT_IN_COMMENT2
+/**/
+#ifndef NOT_IN_COMMENT2
+#error Escaped newline breaks block comment terminator.
+#endif
+
+/* Test escaped newline inside character escape sequence. */
+"\\
+\"/*
+#error Missed string terminator
+*/
+
+/* Block comments don't mask trailing preprocessing
+   directive. */ #define NOT_MASKED
+#ifndef NOT_MASKED
+#error Comment masks trailing directive.
+#endif
Index: gcc/testsuite/gcc.dg/cpp/dir-only-1.h
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-1.h	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-1.h	(revision 0)
@@ -0,0 +1,3 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+#define GOT_HEADER
Index: gcc/testsuite/gcc.dg/cpp/dir-only-2.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-2.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-2.c	(revision 0)
@@ -0,0 +1,12 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fpreprocessed -fdirectives-only -DNOT_SET" } */
+
+/* Tests -fdirectives-only + -fpreprocessed. */
+
+/* Check this is not defined. */
+#ifdef NOT_SET
+#error Command line macro not disabled.
+#endif
Index: gcc/testsuite/gcc.dg/cpp/dir-only-3.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-3.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-3.c	(revision 0)
@@ -0,0 +1,13 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -H" } */
+/* { dg-error "dir-only-3a\.h\n\[^\n\]*dir-only-3b\.h\n\[^\n\]*dir-only-3a\.h\n" "include guard check" { target *-*-* } 0 } */
+
+/* Tests include guards. */
+
+#include "dir-only-3a.h"
+#include "dir-only-3b.h"
+#include "dir-only-3b.h"
+#include "dir-only-3a.h"
Index: gcc/testsuite/gcc.dg/cpp/dir-only-4.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-4.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-4.c	(revision 0)
@@ -0,0 +1,6 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -Wunused-macros" } */
+/* { dg-error "-fdirectives-only is incompatible with -Wunused_macros\n" "-Wunused-macros check" { target *-*-* } 0 } */
Index: gcc/testsuite/gcc.dg/cpp/dir-only-3a.h
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-3a.h	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-3a.h	(revision 0)
@@ -0,0 +1,8 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+extern int outside_guard
+
+#ifndef DIR_ONLY_3A_H
+#define DIR_ONLY_3A_H
+#endif
Index: gcc/testsuite/gcc.dg/cpp/dir-only-5.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-5.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-5.c	(revision 0)
@@ -0,0 +1,6 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -traditional" } */
+/* { dg-error "-fdirectives-only is incompatible with -traditional\n" "-traditional check" { target *-*-* } 0 } */
Index: gcc/testsuite/gcc.dg/cpp/dir-only-3b.h
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-3b.h	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-3b.h	(revision 0)
@@ -0,0 +1,9 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+#ifndef DIR_ONLY_3B_H
+#define DIR_ONLY_3B_H
+
+extern int inside guard;
+
+#endif
Index: gcc/testsuite/gcc.dg/cpp/dir-only-6.c
===================================================================
--- gcc/testsuite/gcc.dg/cpp/dir-only-6.c	(revision 0)
+++ gcc/testsuite/gcc.dg/cpp/dir-only-6.c	(revision 0)
@@ -0,0 +1,7 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* { dg-error "unterminated comment" }
Index: gcc/c.opt
===================================================================
--- gcc/c.opt	(revision 122387)
+++ gcc/c.opt	(working copy)
@@ -508,6 +508,10 @@ fdefault-inline
 C++ ObjC++
 Inline member functions by default
 
+fdirectives-only
+C ObjC C++ ObjC++
+Preprocess directives only.
+
 fdollars-in-identifiers
 C ObjC C++ ObjC++
 Permit '$' as an identifier character
Index: gcc/c-opts.c
===================================================================
--- gcc/c-opts.c	(revision 122387)
+++ gcc/c-opts.c	(working copy)
@@ -610,6 +610,10 @@ c_common_handle_option (size_t scode, co
 	disable_builtin_function (arg);
       break;
 
+    case OPT_fdirectives_only:
+      cpp_opts->directives_only = 1;
+      break;
+
     case OPT_fdollars_in_identifiers:
       cpp_opts->dollars_in_ident = value;
       break;
@@ -1355,6 +1359,11 @@ sanitize_cpp_opts (void)
   if (flag_dump_macros == 'M')
     flag_no_output = 1;
 
+  /* By default, -fdirectives-only implies -dD.  This allows subsequent phases
+     to perform proper macro expansion.  */
+  if (cpp_opts->directives_only && !cpp_opts->preprocessed && !flag_dump_macros)
+    flag_dump_macros = 'D';
+
   /* Disable -dD, -dN and -dI if normal output is suppressed.  Allow
      -dM since at least glibc relies on -M -dM to work.  */
   /* Also, flag_no_output implies flag_no_line_commands, always.  */
@@ -1385,6 +1394,14 @@ sanitize_cpp_opts (void)
      actually output the current directory?  */
   if (flag_working_directory == -1)
     flag_working_directory = (debug_info_level != DINFO_LEVEL_NONE);
+
+  if (cpp_opts->directives_only)
+    {
+      if (warn_unused_macros)
+	error ("-fdirectives-only is incompatible with -Wunused_macros");
+      if (cpp_opts->traditional)
+	error ("-fdirectives-only is incompatible with -traditional");
+    }
 }
 
 /* Add include path with a prefix at the front of its name.  */
@@ -1468,6 +1485,8 @@ finish_options (void)
 	    }
 	}
     }
+  else if (cpp_opts->directives_only)
+    cpp_init_special_builtins (parse_in);
 
   include_cursor = 0;
   push_command_line_include ();
Index: gcc/c-ppoutput.c
===================================================================
--- gcc/c-ppoutput.c	(revision 122387)
+++ gcc/c-ppoutput.c	(working copy)
@@ -41,6 +41,8 @@ static struct
 
 /* General output routines.  */
 static void scan_translation_unit (cpp_reader *);
+static void print_lines_directives_only (int, const void *, size_t);
+static void scan_translation_unit_directives_only (cpp_reader *);
 static void scan_translation_unit_trad (cpp_reader *);
 static void account_for_newlines (const unsigned char *, size_t);
 static int dump_macro (cpp_reader *, cpp_hashnode *, void *);
@@ -75,6 +77,9 @@ preprocess_file (cpp_reader *pfile)
     }
   else if (cpp_get_options (pfile)->traditional)
     scan_translation_unit_trad (pfile);
+  else if (cpp_get_options (pfile)->directives_only
+	   && !cpp_get_options (pfile)->preprocessed)
+    scan_translation_unit_directives_only (pfile);
   else
     scan_translation_unit (pfile);
 
@@ -179,6 +184,26 @@ scan_translation_unit (cpp_reader *pfile
     }
 }
 
+static void
+print_lines_directives_only (int lines, const void *buf, size_t size)
+{
+  print.src_line += lines;
+  fwrite (buf, 1, size, print.outf);
+}
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+static void
+scan_translation_unit_directives_only (cpp_reader *pfile)
+{
+  struct _cpp_dir_only_callbacks cb;
+
+  cb.print_lines = print_lines_directives_only;
+  cb.maybe_print_line = maybe_print_line;
+
+  _cpp_preprocess_dir_only (pfile, &cb);
+}
+
 /* Adjust print.src_line for newlines embedded in output.  */
 static void
 account_for_newlines (const unsigned char *str, size_t len)
Index: libcpp/Makefile.in
===================================================================
--- libcpp/Makefile.in	(revision 122387)
+++ libcpp/Makefile.in	(working copy)
@@ -69,14 +69,14 @@ INCLUDES = -I$(srcdir) -I. -I$(srcdir)/.
 
 ALL_CFLAGS = $(CFLAGS) $(WARN_CFLAGS) $(INCLUDES) $(CPPFLAGS)
 
-libcpp_a_OBJS = charset.o directives.o errors.o expr.o files.o \
-	identifiers.o init.o lex.o line-map.o macro.o mkdeps.o \
-	pch.o symtab.o traditional.o
+libcpp_a_OBJS = charset.o directives.o directives-only.o errors.o \
+	expr.o files.o identifiers.o init.o lex.o line-map.o macro.o \
+	mkdeps.o pch.o symtab.o traditional.o
 makedepend_OBJS = makedepend.o
 
-libcpp_a_SOURCES = charset.c directives.c errors.c expr.c files.c \
-	identifiers.c init.c lex.c line-map.c macro.c mkdeps.c \
-	pch.c symtab.c traditional.c
+libcpp_a_SOURCES = charset.c directives.c directives-only.c errors.c \
+	expr.c files.c identifiers.c init.c lex.c line-map.c macro.c \
+	mkdeps.c pch.c symtab.c traditional.c
 
 all: libcpp.a makedepend$(EXEEXT) $(USED_CATALOGS)
 
Index: libcpp/directives.c
===================================================================
--- libcpp/directives.c	(revision 122387)
+++ libcpp/directives.c	(working copy)
@@ -424,8 +424,13 @@ _cpp_handle_directive (cpp_reader *pfile
 	 does not cause '#define foo bar' to get executed when
 	 compiled with -save-temps, we recognize directives in
 	 -fpreprocessed mode only if the # is in column 1.  macro.c
-	 puts a space in front of any '#' at the start of a macro.  */
+	 puts a space in front of any '#' at the start of a macro.
+	 
+	 We exclude the -fdirectives-only case because macro expansion
+	 has not been performed yet, and block comments can cause spaces
+	 to preceed the directive.  */
       if (CPP_OPTION (pfile, preprocessed)
+	  && !CPP_OPTION (pfile, directives_only)
 	  && (indented || !(dir->flags & IN_I)))
 	{
 	  skip = 0;
Index: libcpp/files.c
===================================================================
--- libcpp/files.c	(revision 122387)
+++ libcpp/files.c	(working copy)
@@ -722,7 +722,8 @@ _cpp_stack_file (cpp_reader *pfile, _cpp
 
   /* Stack the buffer.  */
   buffer = cpp_push_buffer (pfile, file->buffer, file->st.st_size,
-			    CPP_OPTION (pfile, preprocessed));
+			    CPP_OPTION (pfile, preprocessed)
+			    && !CPP_OPTION (pfile, directives_only));
   buffer->file = file;
   buffer->sysp = sysp;
 
Index: libcpp/include/cpplib.h
===================================================================
--- libcpp/include/cpplib.h	(revision 122387)
+++ libcpp/include/cpplib.h	(working copy)
@@ -440,6 +440,9 @@ struct cpp_options
 
   /* True means error callback should be used for diagnostics.  */
   bool client_diagnostic;
+
+  /* True disables tokenization outside of preprocessing directives. */
+  bool directives_only;
 };
 
 /* Callback for header lookup for HEADER, which is the name of a
@@ -643,6 +646,10 @@ extern struct deps *cpp_get_deps (cpp_re
    too.  If there was an error opening the file, it returns NULL.  */
 extern const char *cpp_read_main_file (cpp_reader *, const char *);
 
+/* Set up built-ins with special behavior.  Use cpp_init_builtins()
+   instead unless your know what you are doing.  */
+extern void cpp_init_special_builtins (cpp_reader *);
+
 /* Set up built-ins like __FILE__.  */
 extern void cpp_init_builtins (cpp_reader *, int);
 
Index: libcpp/init.c
===================================================================
--- libcpp/init.c	(revision 122387)
+++ libcpp/init.c	(working copy)
@@ -349,11 +349,8 @@ mark_named_operators (cpp_reader *pfile)
     }
 }
 
-/* Read the builtins table above and enter them, and language-specific
-   macros, into the hash table.  HOSTED is true if this is a hosted
-   environment.  */
 void
-cpp_init_builtins (cpp_reader *pfile, int hosted)
+cpp_init_special_builtins (cpp_reader *pfile)
 {
   const struct builtin *b;
   size_t n = ARRAY_SIZE (builtin_array);
@@ -362,10 +359,7 @@ cpp_init_builtins (cpp_reader *pfile, in
     n -= 2;
   else if (! CPP_OPTION (pfile, stdc_0_in_system_headers)
 	   || CPP_OPTION (pfile, std))
-    {
-      n--;
-      _cpp_define_builtin (pfile, "__STDC__ 1");
-    }
+    n--;
 
   for (b = builtin_array; b < builtin_array + n; b++)
     {
@@ -374,6 +368,20 @@ cpp_init_builtins (cpp_reader *pfile, in
       hp->flags |= NODE_BUILTIN | NODE_WARN;
       hp->value.builtin = (enum builtin_type) b->value;
     }
+}
+
+/* Read the builtins table above and enter them, and language-specific
+   macros, into the hash table.  HOSTED is true if this is a hosted
+   environment.  */
+void
+cpp_init_builtins (cpp_reader *pfile, int hosted)
+{
+  cpp_init_special_builtins (pfile);
+
+  if (!CPP_OPTION (pfile, traditional)
+      && (! CPP_OPTION (pfile, stdc_0_in_system_headers)
+	  || CPP_OPTION (pfile, std)))
+    _cpp_define_builtin (pfile, "__STDC__ 1");
 
   if (CPP_OPTION (pfile, cplusplus))
     _cpp_define_builtin (pfile, "__cplusplus 1");
@@ -621,7 +629,8 @@ post_options (cpp_reader *pfile)
      preprocessed text.  Read preprocesed source in ISO mode.  */
   if (CPP_OPTION (pfile, preprocessed))
     {
-      pfile->state.prevent_expansion = 1;
+      if (!CPP_OPTION (pfile, directives_only))
+	pfile->state.prevent_expansion = 1;
       CPP_OPTION (pfile, traditional) = 0;
     }
 
Index: libcpp/directives-only.c
===================================================================
--- libcpp/directives-only.c	(revision 0)
+++ libcpp/directives-only.c	(revision 0)
@@ -0,0 +1,229 @@
+/* CPP Library - directive only preprocessing for distributed compilation.
+   Copyright (C) 2007
+   Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.
+
+This program is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 2, or (at your option) any
+later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */
+
+#include "config.h"
+#include "system.h"
+#include "cpplib.h"
+#include "internal.h"
+
+/* DO (Directive only) flags. */
+#define DO_BOL		 (1 << 0) /* At the beginning of a logical line. */
+#define DO_STRING	 (1 << 1) /* In a string constant. */
+#define DO_CHAR		 (1 << 2) /* In a character constant. */
+#define DO_BLOCK_COMMENT (1 << 3) /* In a block comment. */
+#define DO_LINE_COMMENT	 (1 << 4) /* In a single line "//-style" comment. */
+
+#define DO_LINE_SPECIAL (DO_STRING | DO_CHAR | DO_LINE_COMMENT)
+#define DO_SPECIAL	(DO_LINE_SPECIAL | DO_BLOCK_COMMENT)
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+void
+_cpp_preprocess_dir_only (cpp_reader *pfile,
+			  const struct _cpp_dir_only_callbacks *cb)
+{
+  struct cpp_buffer *buffer;
+  const unsigned char *cur, *base, *next_line, *rlimit;
+  cppchar_t c, last_c;
+  unsigned flags;
+  int lines, col;
+  source_location loc;
+
+ restart:
+  /* Buffer initialization ala _cpp_clean_line(). */
+  buffer = pfile->buffer;
+  buffer->cur_note = buffer->notes_used = 0;
+  buffer->cur = buffer->line_base = buffer->next_line;
+  buffer->need_line = false;
+
+  /* This isn't really needed.  It prevents a compiler warning, though. */
+  loc = pfile->line_table->highest_line;
+
+  /* Scan initialization. */
+  next_line = cur = base = buffer->cur;
+  rlimit = buffer->rlimit;
+  flags = DO_BOL;
+  lines = 0;
+  col = 1;
+
+  for (last_c = '\n', c = *cur; cur < rlimit; last_c = c, c = *++cur, ++col)
+    {
+      /* Skip over escaped newlines. */
+      if (__builtin_expect (c == '\\', false))
+	{
+	  const unsigned char *tmp = cur + 1;
+
+	  while (is_nvspace (*tmp) && tmp < rlimit)
+	    tmp++;
+	  if (*tmp == '\r')
+	    tmp++;
+	  if (*tmp == '\n' && tmp < rlimit)
+	    {
+	      CPP_INCREMENT_LINE (pfile, 0);
+	      lines++;
+	      col = 0;
+	      cur = tmp;
+	      c = last_c;
+	      continue;
+	    }
+	}
+
+      if (__builtin_expect (last_c == '#', false) && !(flags & DO_SPECIAL))
+	{
+	  if (c != '#' && (flags & DO_BOL))
+	  {
+	    struct line_maps *line_table;
+
+	    if (!pfile->state.skipping && next_line != base)
+	      cb->print_lines (lines, base, next_line - base);
+
+	    /* Prep things for directive handling. */
+	    buffer->next_line = cur;
+	    buffer->need_line = true;
+	    _cpp_get_fresh_line (pfile);
+
+	    /* Ensure proper column numbering for generated error messages. */
+	    buffer->line_base -= col - 1;
+
+	    _cpp_handle_directive (pfile, 0 /* ignore indented */);
+
+	    /* Sanitize the line settings.  Duplicate #include's can mess
+	       things up. */
+	    line_table = pfile->line_table;
+	    line_table->highest_location = line_table->highest_line;
+
+	    /* The if block prevents us from outputing line information when
+	       the file ends with a directive and no newline.  Note that we
+	       must use pfile->buffer, not buffer. */
+	    if (pfile->buffer->cur != pfile->buffer->rlimit)
+	      cb->maybe_print_line (pfile->line_table->highest_line);
+
+	    goto restart;
+	  }
+
+	  flags &= ~DO_BOL;
+	  pfile->mi_valid = false;
+	}
+      else if (__builtin_expect (last_c == '/', false) \
+	       && !(flags & DO_SPECIAL) && c != '*' && c != '/')
+	{
+	  /* If a previous slash is not starting a block comment, clear the
+	     DO_BOL flag.  */
+	  flags &= ~DO_BOL;
+	  pfile->mi_valid = false;
+	}
+
+      switch (c)
+	{
+	case '/':
+	  if ((flags & DO_BLOCK_COMMENT) && last_c == '*')
+	    {
+	      flags &= ~DO_BLOCK_COMMENT;
+	      c = 0;
+	    }
+	  else if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_LINE_COMMENT;
+	  else if (!(flags & DO_SPECIAL))
+	    /* Mark the position for possible error reporting. */
+	    LINEMAP_POSITION_FOR_COLUMN (loc, pfile->line_table, col);
+
+	  break;
+
+	case '*':
+	  if (!(flags & DO_SPECIAL))
+	    {
+	      if (last_c == '/')
+		flags |= DO_BLOCK_COMMENT;
+	      else
+		{
+		  flags &= ~DO_BOL;
+		  pfile->mi_valid = false;
+		}
+	    }
+
+	  break;
+
+	case '\'':
+	case '"':
+	  {
+	    unsigned state = (c == '"') ? DO_STRING : DO_CHAR;
+
+	    if (!(flags & DO_SPECIAL))
+	      {
+		flags |= state;
+		flags &= ~DO_BOL;
+		pfile->mi_valid = false;
+	      }
+	    else if ((flags & state) && last_c != '\\')
+	      flags &= ~state;
+
+	    break;
+	  }
+
+	case '\\':
+	  {
+	    if ((flags & (DO_STRING | DO_CHAR)) && last_c == '\\')
+	      c = 0;
+
+	    if (!(flags & DO_SPECIAL))
+	      {
+		flags &= ~DO_BOL;
+		pfile->mi_valid = false;
+	      }
+
+	    break;
+	  }
+
+	case '\n':
+	  CPP_INCREMENT_LINE (pfile, 0);
+	  lines++;
+	  col = 0;
+	  flags &= ~DO_LINE_SPECIAL;
+	  if (!(flags & DO_SPECIAL))
+	    flags |= DO_BOL;
+	  break;
+
+	case '#':
+	  next_line = cur;
+	  /* Don't update DO_BOL yet. */
+	  break;
+
+	case ' ': case '\t': case '\f': case '\v': case '\0':
+	  break;
+
+	default:
+	  if (!(flags & DO_SPECIAL))
+	    {
+	      flags &= ~DO_BOL;
+	      pfile->mi_valid = false;
+	    }
+	  break;
+	}
+    }
+
+  if (flags & DO_BLOCK_COMMENT)
+    cpp_error_with_line (pfile, CPP_DL_ERROR, loc, 0, "unterminated comment");
+
+  if (!pfile->state.skipping && cur != base)
+    cb->print_lines (lines, base, cur - base);
+
+  _cpp_pop_buffer (pfile);
+  if (pfile->buffer)
+    goto restart;
+}
Index: libcpp/internal.h
===================================================================
--- libcpp/internal.h	(revision 122387)
+++ libcpp/internal.h	(working copy)
@@ -568,6 +568,17 @@ extern void _cpp_do_file_change (cpp_rea
 				 unsigned int, unsigned int);
 extern void _cpp_pop_buffer (cpp_reader *);
 
+/* In directives.c */
+struct _cpp_dir_only_callbacks
+{
+  /* Called to print a block of lines. */
+  void (*print_lines) (int, const void *, size_t);
+  void (*maybe_print_line) (source_location);
+};
+
+extern void _cpp_preprocess_dir_only (cpp_reader *,
+				      const struct _cpp_dir_only_callbacks *);
+
 /* In traditional.c.  */
 extern bool _cpp_scan_out_logical_line (cpp_reader *, cpp_macro *);
 extern bool _cpp_read_logical_line_trad (cpp_reader *);

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

It's been a couple weeks since I posted this patch so I wanted to send
out a quick reminder.  My last email with the latest patch is at
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.


Here's an updated summary:

This patch decreases distcc build and ccache lookup times by providing
an optimized "directives-only" preprocessing mode.

Distcc improves build performance by preprocessing files locally,
transmitting the preprocessed code to one or more remote build
servers, and performing compilation on the remote servers.  In
practice, the time to preprocess files locally is the limiting factor
in further reducing build time.  This patch adds a -fdirectives-only
flag which instructs the preprocessor to treat directives normally but
bypass tokenization of other preprocessing tokens.  Additional
preprocessing, such as macro expansion outside of directives, is
performed during the compilation phase.

Ccache reduces recompilation time by caching the results of previous
compilations and detecting when a compilation is repeated.  It
generates a lookup key from the preprocessed source code.  The
-directives-only flag improves lookup performance by reducing key
generation time.

For a typical c++ file, the new option reduces preprocessing time by
about 30%.  This in turn results in a similar decrease in overall
distcc build and ccache lookup times.

Usage is straightforward.  The -fdirectives-only flag is simply added
to the compiler options passed to distcc or ccache.

Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.
Validated two-stage processing by rerunning the C/C++ testsuites with
xgcc and g++ replaced by scripts which invoke ccache.

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

It's been about a month since I posted this patch so I wanted to send
out another reminder.  The patch submission is at
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.


Here's a refresher summary:

This patch decreases distcc build and ccache lookup times by providing
an optimized "directives-only" preprocessing mode.

Distcc improves build performance by preprocessing files locally,
transmitting the preprocessed code to one or more remote build
servers, and performing compilation on the remote servers.  In
practice, the time to preprocess files locally is the limiting factor
in further reducing build time.  This patch adds a -fdirectives-only
flag which instructs the preprocessor to treat directives normally but
bypass tokenization of other preprocessing tokens.  Additional
preprocessing, such as macro expansion outside of directives, is
performed during the compilation phase.

Ccache reduces recompilation time by caching the results of previous
compilations and detecting when a compilation is repeated.  It
generates a lookup key from the preprocessed source code.  The
-directives-only flag improves lookup performance by reducing key
generation time.

For a typical c++ file, the new option reduces preprocessing time by
about 30%.  This in turn results in a similar decrease in overall
distcc build and ccache lookup times.

Usage is straightforward.  The -fdirectives-only flag is simply added
to the compiler options passed to distcc or ccache.

Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.
Validated two-stage processing by rerunning the C/C++ testsuites with
xgcc and g++ replaced by scripts which invoke ccache.

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Mike Stump]

On Mar 26, 2007, at 3:31 PM, Ollie Wild wrote:
> It's been about a month since I posted this patch so I wanted to send
> out another reminder.  The patch submission is at
> http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.

I wish a maintainer would review this patch.  I think its interesting  
from a compilation speed perspective.  I wish the SC would invent a  
solution to patches like this falling through the cracks.  Maybe  
appoint a front end compilation speed maintainer?

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

I heard a little rumor that gcc now has an official preprocessor
maintainer, so I thought I'd take this opportunity to send out a
shameless plug for my favorite patch.  The submission is at
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.


Here's a refresher summary:

This patch decreases distcc build and ccache lookup times by providing
an optimized "directives-only" preprocessing mode.

Distcc improves build performance by preprocessing files locally,
transmitting the preprocessed code to one or more remote build
servers, and performing compilation on the remote servers.  In
practice, the time to preprocess files locally is the limiting factor
in further reducing build time.  This patch adds a -fdirectives-only
flag which instructs the preprocessor to treat directives normally but
bypass tokenization of other preprocessing tokens.  Additional
preprocessing, such as macro expansion outside of directives, is
performed during the compilation phase.

Ccache reduces recompilation time by caching the results of previous
compilations and detecting when a compilation is repeated.  It
generates a lookup key from the preprocessed source code.  The
-directives-only flag improves lookup performance by reducing key
generation time.

For a typical c++ file, the new option reduces preprocessing time by
about 30%.  This in turn results in a similar decrease in overall
distcc build and ccache lookup times.

Usage is straightforward.  The -fdirectives-only flag is simply added
to the compiler options passed to distcc or ccache.

Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.
Validated two-stage processing by rerunning the C/C++ testsuites with
xgcc and g++ replaced by scripts which invoke ccache.

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Tom Tromey]

>>>>> "Ollie" == Ollie Wild <aaw@google.com> writes:

Ollie> I heard a little rumor that gcc now has an official preprocessor
Ollie> maintainer, so I thought I'd take this opportunity to send out a
Ollie> shameless plug for my favorite patch.  The submission is at
Ollie> http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.

I haven't yet done any of my maintainer duties... unfortunately this
week has been busy and I haven't had time to get to them :(.  I do
plan to get to the patch backlog soon.

I didn't read your patch in detail but I did take a look at it.  I'm
not super fond of the idea of having another cpp operation mode, but
the application does seem important and useful enough to warrant it.

I'm not sure whether or not I could actually approve this patch
though.  I'm not sure how far the non-algorithmic rules stretch.

Tom

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Mike Stump]

On Apr 17, 2007, at 10:38 PM, Ollie Wild wrote:
> This patch decreases distcc build and ccache lookup times by providing
> an optimized "directives-only" preprocessing mode.

I did a quick review over the code, looks good.  I reviewed it for  
safety (default behavior) and it looks reasonable.  The concept  
sounds reasonable and could enhance the scaling factor of distcc  
builds, which makes it desirable for me, as that means a build time  
speed up.  I don't know of any near term technology that can gives a  
30% speedup, save new hardware.  Personally, I'd like it for 4.2 as  
well.  It looks safe enough to backport.

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Mike Stump]

On Apr 18, 2007, at 11:57 AM, Tom Tromey wrote:
> I'm not sure whether or not I could actually approve this patch
> though.  I'm not sure how far the non-algorithmic rules stretch.

My take, I didn't see any change in algorithm in the patch.  What it  
does is provide a new option to run half of the existing algorithm...

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Dirk Mueller]

On Wednesday, 18. April 2007, Ollie Wild wrote:

> Here's a refresher summary:

This seems to be more or less a newly implemented cpp, that is a lot simpler 
than the old one. have you looked into getting the other one faster instead? 


Dirk

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

On 18 Apr 2007 12:57:45 -0600, Tom Tromey <tromey@redhat.com> wrote:
> I'm not sure whether or not I could actually approve this patch
> though.  I'm not sure how far the non-algorithmic rules stretch.

Indeed.  I hadn't understood the non-algorithmic distinction.  Would
one of the C or C++ front end maintainers mind taking a look?

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Mark Mitchell]

Ollie Wild wrote:
> On 18 Apr 2007 12:57:45 -0600, Tom Tromey <tromey@redhat.com> wrote:
>> I'm not sure whether or not I could actually approve this patch
>> though.  I'm not sure how far the non-algorithmic rules stretch.
> 
> Indeed.  I hadn't understood the non-algorithmic distinction.  Would
> one of the C or C++ front end maintainers mind taking a look?

I will take a look, if you point me at the URL.  I can't commit to doing
this immediately, but I will do my best.  If one of the other C/C++
maintainers beat me to it, even better. :-)

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

> I will take a look, if you point me at the URL.  I can't commit to doing
> this immediately, but I will do my best.  If one of the other C/C++
> maintainers beat me to it, even better. :-)

Thanks, Mark.

The latest patch is at
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.  A summary is
available at http://gcc.gnu.org/ml/gcc-patches/2007-03/msg00786.html.

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

> > I will take a look, if you point me at the URL.  I can't commit to doing
> > this immediately, but I will do my best.  If one of the other C/C++
> > maintainers beat me to it, even better. :-)
>
> Thanks, Mark.
>
> The latest patch is at
> http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html.  A summary is
> available at http://gcc.gnu.org/ml/gcc-patches/2007-03/msg00786.html.

I just wanted to send out a quick reminder since I haven't heard back
on this for a while.  I imagine Mark is pretty overworked at the
moment with the pending 4.2 release.  Would one of the other C/C++
maintainers like to take a stab at this?

Ollie

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Ollie Wild]

[-- Attachment #1: Type: text/plain, Size: 4101 bytes --]

> I just wanted to send out a quick reminder since I haven't heard back
> on this for a while.  I imagine Mark is pretty overworked at the
> moment with the pending 4.2 release.  Would one of the other C/C++
> maintainers like to take a stab at this?

Michael Chastain discovered a bug wherein I was incorrectly handling
files without terminating newline characters.  The attached patch is
identical to the one at
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html except that it
properly handles non-newline terminated files inside
scan_translation_unit_directives_only().


A quick summary recap:

This patch decreases distcc build and ccache lookup times by providing
an optimized "directives-only" preprocessing mode.

Distcc improves build performance by preprocessing files locally,
transmitting the preprocessed code to one or more remote build
servers, and performing compilation on the remote servers.  In
practice, the time to preprocess files locally is the limiting factor
in further reducing build time.  This patch adds a -fdirectives-only
flag which instructs the preprocessor to treat directives normally but
bypass tokenization of other preprocessing tokens.  Additional
preprocessing, such as macro expansion outside of directives, is
performed during the compilation phase.

Ccache reduces recompilation time by caching the results of previous
compilations and detecting when a compilation is repeated.  It
generates a lookup key from the preprocessed source code.  The
-directives-only flag improves lookup performance by reducing key
generation time.

For a typical c++ file, the new option reduces preprocessing time by
about 30%.  This in turn results in a similar decrease in overall
distcc build and ccache lookup times.

Usage is straightforward.  The -fdirectives-only flag is simply added
to the compiler options passed to distcc or ccache.

Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.
Validated two-stage processing by rerunning the C/C++ testsuites with
xgcc and g++ replaced by scripts which invoke ccache.

Ollie

:ADDPATCH libcpp:

2007-06-14 Ollie Wild <aaw@google.com>

       * gcc.dg/cpp/dir-only-1.c: New test.
       * gcc.dg/cpp/dir-only-1.h: New file.
       * gcc.dg/cpp/dir-only-2.c: New test.
       * gcc.dg/cpp/dir-only-3.c: New test.
       * gcc.dg/cpp/dir-only-3a.h: New file.
       * gcc.dg/cpp/dir-only-3b.h: New file.
       * gcc.dg/cpp/dir-only-4.c: New test.
       * gcc.dg/cpp/dir-only-5.c: New test.
       * gcc.dg/cpp/dir-only-6.c: New test.


2007-06-14 Ollie Wild <aaw@google.com>

       * directives-only.c: New file.
       * internal.h (struct _cpp_dir_only_callbacks): New.
       (_cpp_preprocess_dir_only): New function.
       * directives.c (_cpp_handle_directive): Check directives_only before
       disabling execution of indented directives.
       * files.c (_cpp_stack_file): Add directives_only check.
       * include/cpplib.h (struct cpp_options): Add directives_only.
       (cpp_init_special_builtins): New function.
       * init.c (cpp_init_special_builtins): New function.
       (cpp_init_builtins): Move builtin_array initialization to
       cpp_init_special_builtins.
       (post_options): Check directives_only before setting
       pfile->state.prevent_expansion = 1.
       * Makefile.in (libcpp_a_OBJS): Add directives-only.o.
       (libcpp_a_SOURCES): Add directives-only.c.


2007-06-14 Ollie Wild <aaw@google.com>

       * c-ppoutput.c (print_lines_directives_only): New function.
       (scan_translation_unit_directives_only): New function.
       (preprocess_file): Add call to scan_translation_unit_directives_only.
       * c-opts.c (c_common_handle_option): Add OPT_fdirectives_only.
       (sanitize_cpp_opts): Add default flag_dump_macros setting for
       -fdirectives-only.  Add errors for -fdirectives-only conflict with
       -Wunused-macros and -traditional.
       (finish_options): Add builtin macro initialization for
       -fdirectives-only + -fpreprocessed.
       * c.opt (fdirectives-only): New.
       * doc/cppopts.texi (fdirectives-only): New.

[-- Attachment #2: directives-only.patch --]
[-- Type: text/x-patch, Size: 23067 bytes --]

diff --git a/gcc/c-opts.c b/gcc/c-opts.c
index d0fafc6..79a49a3 100644
--- a/gcc/c-opts.c
+++ b/gcc/c-opts.c
@@ -616,6 +616,10 @@ c_common_handle_option (size_t scode, const char *arg, int value)
 	disable_builtin_function (arg);
       break;
 
+    case OPT_fdirectives_only:
+      cpp_opts->directives_only = 1;
+      break;
+
     case OPT_fdollars_in_identifiers:
       cpp_opts->dollars_in_ident = value;
       break;
@@ -1409,6 +1413,11 @@ sanitize_cpp_opts (void)
   if (flag_dump_macros == 'M')
     flag_no_output = 1;
 
+  /* By default, -fdirectives-only implies -dD.  This allows subsequent phases
+     to perform proper macro expansion.  */
+  if (cpp_opts->directives_only && !cpp_opts->preprocessed && !flag_dump_macros)
+    flag_dump_macros = 'D';
+
   /* Disable -dD, -dN and -dI if normal output is suppressed.  Allow
      -dM since at least glibc relies on -M -dM to work.  */
   /* Also, flag_no_output implies flag_no_line_commands, always.  */
@@ -1439,6 +1448,14 @@ sanitize_cpp_opts (void)
      actually output the current directory?  */
   if (flag_working_directory == -1)
     flag_working_directory = (debug_info_level != DINFO_LEVEL_NONE);
+
+  if (cpp_opts->directives_only)
+    {
+      if (warn_unused_macros)
+	error ("-fdirectives-only is incompatible with -Wunused_macros");
+      if (cpp_opts->traditional)
+	error ("-fdirectives-only is incompatible with -traditional");
+    }
 }
 
 /* Add include path with a prefix at the front of its name.  */
@@ -1522,6 +1539,8 @@ finish_options (void)
 	    }
 	}
     }
+  else if (cpp_opts->directives_only)
+    cpp_init_special_builtins (parse_in);
 
   include_cursor = 0;
   push_command_line_include ();
diff --git a/gcc/c-ppoutput.c b/gcc/c-ppoutput.c
index 10540b7..62b350b 100644
--- a/gcc/c-ppoutput.c
+++ b/gcc/c-ppoutput.c
@@ -41,6 +41,8 @@ static struct
 
 /* General output routines.  */
 static void scan_translation_unit (cpp_reader *);
+static void print_lines_directives_only (int, const void *, size_t);
+static void scan_translation_unit_directives_only (cpp_reader *);
 static void scan_translation_unit_trad (cpp_reader *);
 static void account_for_newlines (const unsigned char *, size_t);
 static int dump_macro (cpp_reader *, cpp_hashnode *, void *);
@@ -75,6 +77,9 @@ preprocess_file (cpp_reader *pfile)
     }
   else if (cpp_get_options (pfile)->traditional)
     scan_translation_unit_trad (pfile);
+  else if (cpp_get_options (pfile)->directives_only
+	   && !cpp_get_options (pfile)->preprocessed)
+    scan_translation_unit_directives_only (pfile);
   else
     scan_translation_unit (pfile);
 
@@ -179,6 +184,26 @@ scan_translation_unit (cpp_reader *pfile)
     }
 }
 
+static void
+print_lines_directives_only (int lines, const void *buf, size_t size)
+{
+  print.src_line += lines;
+  fwrite (buf, 1, size, print.outf);
+}
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+static void
+scan_translation_unit_directives_only (cpp_reader *pfile)
+{
+  struct _cpp_dir_only_callbacks cb;
+
+  cb.print_lines = print_lines_directives_only;
+  cb.maybe_print_line = maybe_print_line;
+
+  _cpp_preprocess_dir_only (pfile, &cb);
+}
+
 /* Adjust print.src_line for newlines embedded in output.  */
 static void
 account_for_newlines (const unsigned char *str, size_t len)
diff --git a/gcc/c.opt b/gcc/c.opt
index 63e2bda..dc6cff1 100644
--- a/gcc/c.opt
+++ b/gcc/c.opt
@@ -524,6 +524,10 @@ fdefault-inline
 C++ ObjC++
 Inline member functions by default
 
+fdirectives-only
+C ObjC C++ ObjC++
+Preprocess directives only.
+
 fdollars-in-identifiers
 C ObjC C++ ObjC++
 Permit '$' as an identifier character
diff --git a/gcc/doc/cppopts.texi b/gcc/doc/cppopts.texi
index 17235bd..14187b6 100644
--- a/gcc/doc/cppopts.texi
+++ b/gcc/doc/cppopts.texi
@@ -518,6 +518,22 @@ Search @var{dir} only for header files requested with
 If @var{dir} begins with @code{=}, then the @code{=} will be replaced
 by the sysroot prefix; see @option{--sysroot} and @option{-isysroot}.
 
+@item -fdirectives-only
+@opindex fdirectives-only
+This option provides a simplified preprocessor to improve the
+performance of distributed build systems such as distcc.  It's
+behavior depends on a number of other flags.
+
+If the @option{-E} option is enabled, it suppresses things like macro
+expansion, trigraph conversion, and escaped newline splicing
+outside of directives.  All directives are processed normally, except that
+macro definitions are output similar to the @option{-dD} option.
+
+If the @option{-fpreprocessed} option is enabled, it suppresses
+predefinition of most builtin and command line macros.  This
+prevents duplicate definition of macros output with the @option{-E}
+option.
+
 @item -fdollars-in-identifiers
 @opindex fdollars-in-identifiers
 @anchor{fdollars-in-identifiers}
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-1.c b/gcc/testsuite/gcc.dg/cpp/dir-only-1.c
new file mode 100644
index 0000000..3c22616
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-1.c
@@ -0,0 +1,73 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* Tests scan_translation_unit_directives_only()'s handling of corner cases. */
+
+/* Ignore directives inside block comments...
+#error directive inside block comment
+*/
+
+// Escaped newline doesn't terminate line comment \
+#error directive inside line comment
+
+/* A comment canot start inside a string. */
+const char *c1 = "/*";
+#define NOT_IN_COMMENT
+const char *c2 = "*/";
+#ifndef NOT_IN_COMMENT
+#error Comment started inside a string literal
+#endif
+
+/* Escaped newline handling. */
+int i; \
+#error ignored escaped newline
+  \
+  \
+#define BOL
+#ifndef BOL
+#error escaped newline did not preserve beginning of line
+#endif
+
+/* Handles \\ properly at the end of a string. */
+"string ends in \\"/*
+#error Missed string terminator.
+*/
+
+/* Handles macro expansion in preprocessing directives. */
+#define HEADER "dir-only-1.h"
+#include HEADER
+#ifndef GOT_HEADER
+#error Failed to include header.
+#endif
+
+/\
+*
+#define IN_COMMENT
+*/
+#ifdef IN_COMMENT
+#error Escaped newline breaks block comment initiator.
+#endif
+
+/*
+*\
+/
+#define NOT_IN_COMMENT2
+/**/
+#ifndef NOT_IN_COMMENT2
+#error Escaped newline breaks block comment terminator.
+#endif
+
+/* Test escaped newline inside character escape sequence. */
+"\\
+\"/*
+#error Missed string terminator
+*/
+
+/* Block comments don't mask trailing preprocessing
+   directive. */ #define NOT_MASKED
+#ifndef NOT_MASKED
+#error Comment masks trailing directive.
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-1.h b/gcc/testsuite/gcc.dg/cpp/dir-only-1.h
new file mode 100644
index 0000000..96dbcc0
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-1.h
@@ -0,0 +1,3 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+#define GOT_HEADER
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-2.c b/gcc/testsuite/gcc.dg/cpp/dir-only-2.c
new file mode 100644
index 0000000..489b4d6
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-2.c
@@ -0,0 +1,12 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fpreprocessed -fdirectives-only -DNOT_SET" } */
+
+/* Tests -fdirectives-only + -fpreprocessed. */
+
+/* Check this is not defined. */
+#ifdef NOT_SET
+#error Command line macro not disabled.
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-3.c b/gcc/testsuite/gcc.dg/cpp/dir-only-3.c
new file mode 100644
index 0000000..d97c23f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-3.c
@@ -0,0 +1,13 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -H" } */
+/* { dg-error "dir-only-3a\.h\n\[^\n\]*dir-only-3b\.h\n\[^\n\]*dir-only-3a\.h\n" "include guard check" { target *-*-* } 0 } */
+
+/* Tests include guards. */
+
+#include "dir-only-3a.h"
+#include "dir-only-3b.h"
+#include "dir-only-3b.h"
+#include "dir-only-3a.h"
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-3a.h b/gcc/testsuite/gcc.dg/cpp/dir-only-3a.h
new file mode 100644
index 0000000..6644bbf
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-3a.h
@@ -0,0 +1,8 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+extern int outside_guard
+
+#ifndef DIR_ONLY_3A_H
+#define DIR_ONLY_3A_H
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-3b.h b/gcc/testsuite/gcc.dg/cpp/dir-only-3b.h
new file mode 100644
index 0000000..4edaa7b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-3b.h
@@ -0,0 +1,9 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+#ifndef DIR_ONLY_3B_H
+#define DIR_ONLY_3B_H
+
+extern int inside guard;
+
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-4.c b/gcc/testsuite/gcc.dg/cpp/dir-only-4.c
new file mode 100644
index 0000000..a7b5f04
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-4.c
@@ -0,0 +1,6 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -Wunused-macros" } */
+/* { dg-error "-fdirectives-only is incompatible with -Wunused_macros\n" "-Wunused-macros check" { target *-*-* } 0 } */
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-5.c b/gcc/testsuite/gcc.dg/cpp/dir-only-5.c
new file mode 100644
index 0000000..643a4d7
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-5.c
@@ -0,0 +1,6 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -traditional" } */
+/* { dg-error "-fdirectives-only is incompatible with -traditional\n" "-traditional check" { target *-*-* } 0 } */
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-6.c b/gcc/testsuite/gcc.dg/cpp/dir-only-6.c
new file mode 100644
index 0000000..0023205
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-6.c
@@ -0,0 +1,7 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* { dg-error "unterminated comment" }
diff --git a/libcpp/Makefile.in b/libcpp/Makefile.in
index 166f1fa..960a42d 100644
--- a/libcpp/Makefile.in
+++ b/libcpp/Makefile.in
@@ -69,14 +69,14 @@ INCLUDES = -I$(srcdir) -I. -I$(srcdir)/../include @INCINTL@ \
 
 ALL_CFLAGS = $(CFLAGS) $(WARN_CFLAGS) $(INCLUDES) $(CPPFLAGS)
 
-libcpp_a_OBJS = charset.o directives.o errors.o expr.o files.o \
-	identifiers.o init.o lex.o line-map.o macro.o mkdeps.o \
-	pch.o symtab.o traditional.o
+libcpp_a_OBJS = charset.o directives.o directives-only.o errors.o \
+	expr.o files.o identifiers.o init.o lex.o line-map.o macro.o \
+	mkdeps.o pch.o symtab.o traditional.o
 makedepend_OBJS = makedepend.o
 
-libcpp_a_SOURCES = charset.c directives.c errors.c expr.c files.c \
-	identifiers.c init.c lex.c line-map.c macro.c mkdeps.c \
-	pch.c symtab.c traditional.c
+libcpp_a_SOURCES = charset.c directives.c directives-only.c errors.c \
+	expr.c files.c identifiers.c init.c lex.c line-map.c macro.c \
+	mkdeps.c pch.c symtab.c traditional.c
 
 all: libcpp.a makedepend$(EXEEXT) $(USED_CATALOGS)
 
diff --git a/libcpp/directives-only.c b/libcpp/directives-only.c
new file mode 100644
index 0000000..d50cebb
--- /dev/null
+++ b/libcpp/directives-only.c
@@ -0,0 +1,240 @@
+/* CPP Library - directive only preprocessing for distributed compilation.
+   Copyright (C) 2007
+   Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.
+
+This program is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 2, or (at your option) any
+later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */
+
+#include "config.h"
+#include "system.h"
+#include "cpplib.h"
+#include "internal.h"
+
+/* DO (Directive only) flags. */
+#define DO_BOL		 (1 << 0) /* At the beginning of a logical line. */
+#define DO_STRING	 (1 << 1) /* In a string constant. */
+#define DO_CHAR		 (1 << 2) /* In a character constant. */
+#define DO_BLOCK_COMMENT (1 << 3) /* In a block comment. */
+#define DO_LINE_COMMENT	 (1 << 4) /* In a single line "//-style" comment. */
+
+#define DO_LINE_SPECIAL (DO_STRING | DO_CHAR | DO_LINE_COMMENT)
+#define DO_SPECIAL	(DO_LINE_SPECIAL | DO_BLOCK_COMMENT)
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+void
+_cpp_preprocess_dir_only (cpp_reader *pfile,
+			  const struct _cpp_dir_only_callbacks *cb)
+{
+  struct cpp_buffer *buffer;
+  const unsigned char *cur, *base, *next_line, *rlimit;
+  cppchar_t c, last_c;
+  unsigned flags;
+  int lines, col;
+  source_location loc;
+
+ restart:
+  /* Buffer initialization ala _cpp_clean_line(). */
+  buffer = pfile->buffer;
+  buffer->cur_note = buffer->notes_used = 0;
+  buffer->cur = buffer->line_base = buffer->next_line;
+  buffer->need_line = false;
+
+  /* This isn't really needed.  It prevents a compiler warning, though. */
+  loc = pfile->line_table->highest_line;
+
+  /* Scan initialization. */
+  next_line = cur = base = buffer->cur;
+  rlimit = buffer->rlimit;
+  flags = DO_BOL;
+  lines = 0;
+  col = 1;
+
+  for (last_c = '\n', c = *cur; cur < rlimit; last_c = c, c = *++cur, ++col)
+    {
+      /* Skip over escaped newlines. */
+      if (__builtin_expect (c == '\\', false))
+	{
+	  const unsigned char *tmp = cur + 1;
+
+	  while (is_nvspace (*tmp) && tmp < rlimit)
+	    tmp++;
+	  if (*tmp == '\r')
+	    tmp++;
+	  if (*tmp == '\n' && tmp < rlimit)
+	    {
+	      CPP_INCREMENT_LINE (pfile, 0);
+	      lines++;
+	      col = 0;
+	      cur = tmp;
+	      c = last_c;
+	      continue;
+	    }
+	}
+
+      if (__builtin_expect (last_c == '#', false) && !(flags & DO_SPECIAL))
+	{
+	  if (c != '#' && (flags & DO_BOL))
+	  {
+	    struct line_maps *line_table;
+
+	    if (!pfile->state.skipping && next_line != base)
+	      cb->print_lines (lines, base, next_line - base);
+
+	    /* Prep things for directive handling. */
+	    buffer->next_line = cur;
+	    buffer->need_line = true;
+	    _cpp_get_fresh_line (pfile);
+
+	    /* Ensure proper column numbering for generated error messages. */
+	    buffer->line_base -= col - 1;
+
+	    _cpp_handle_directive (pfile, 0 /* ignore indented */);
+
+	    /* Sanitize the line settings.  Duplicate #include's can mess
+	       things up. */
+	    line_table = pfile->line_table;
+	    line_table->highest_location = line_table->highest_line;
+
+	    /* The if block prevents us from outputing line information when
+	       the file ends with a directive and no newline.  Note that we
+	       must use pfile->buffer, not buffer. */
+	    if (pfile->buffer->next_line < pfile->buffer->rlimit)
+	      cb->maybe_print_line (pfile->line_table->highest_line);
+
+	    goto restart;
+	  }
+
+	  flags &= ~DO_BOL;
+	  pfile->mi_valid = false;
+	}
+      else if (__builtin_expect (last_c == '/', false) \
+	       && !(flags & DO_SPECIAL) && c != '*' && c != '/')
+	{
+	  /* If a previous slash is not starting a block comment, clear the
+	     DO_BOL flag.  */
+	  flags &= ~DO_BOL;
+	  pfile->mi_valid = false;
+	}
+
+      switch (c)
+	{
+	case '/':
+	  if ((flags & DO_BLOCK_COMMENT) && last_c == '*')
+	    {
+	      flags &= ~DO_BLOCK_COMMENT;
+	      c = 0;
+	    }
+	  else if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_LINE_COMMENT;
+	  else if (!(flags & DO_SPECIAL))
+	    /* Mark the position for possible error reporting. */
+	    LINEMAP_POSITION_FOR_COLUMN (loc, pfile->line_table, col);
+
+	  break;
+
+	case '*':
+	  if (!(flags & DO_SPECIAL))
+	    {
+	      if (last_c == '/')
+		flags |= DO_BLOCK_COMMENT;
+	      else
+		{
+		  flags &= ~DO_BOL;
+		  pfile->mi_valid = false;
+		}
+	    }
+
+	  break;
+
+	case '\'':
+	case '"':
+	  {
+	    unsigned state = (c == '"') ? DO_STRING : DO_CHAR;
+
+	    if (!(flags & DO_SPECIAL))
+	      {
+		flags |= state;
+		flags &= ~DO_BOL;
+		pfile->mi_valid = false;
+	      }
+	    else if ((flags & state) && last_c != '\\')
+	      flags &= ~state;
+
+	    break;
+	  }
+
+	case '\\':
+	  {
+	    if ((flags & (DO_STRING | DO_CHAR)) && last_c == '\\')
+	      c = 0;
+
+	    if (!(flags & DO_SPECIAL))
+	      {
+		flags &= ~DO_BOL;
+		pfile->mi_valid = false;
+	      }
+
+	    break;
+	  }
+
+	case '\n':
+	  CPP_INCREMENT_LINE (pfile, 0);
+	  lines++;
+	  col = 0;
+	  flags &= ~DO_LINE_SPECIAL;
+	  if (!(flags & DO_SPECIAL))
+	    flags |= DO_BOL;
+	  break;
+
+	case '#':
+	  next_line = cur;
+	  /* Don't update DO_BOL yet. */
+	  break;
+
+	case ' ': case '\t': case '\f': case '\v': case '\0':
+	  break;
+
+	default:
+	  if (!(flags & DO_SPECIAL))
+	    {
+	      flags &= ~DO_BOL;
+	      pfile->mi_valid = false;
+	    }
+	  break;
+	}
+    }
+
+  if (flags & DO_BLOCK_COMMENT)
+    cpp_error_with_line (pfile, CPP_DL_ERROR, loc, 0, "unterminated comment");
+
+  if (!pfile->state.skipping && cur != base)
+    {
+      /* If the file was not newline terminated, add rlimit, which is
+         guaranteed to point to a newline, to the end of our range.  */
+      if (cur[-1] != '\n')
+	{
+	  cur++;
+	  CPP_INCREMENT_LINE (pfile, 0);
+	  lines++;
+	}
+
+      cb->print_lines (lines, base, cur - base);
+    }
+
+  _cpp_pop_buffer (pfile);
+  if (pfile->buffer)
+    goto restart;
+}
diff --git a/libcpp/directives.c b/libcpp/directives.c
index ccb9f32..38ca949 100644
--- a/libcpp/directives.c
+++ b/libcpp/directives.c
@@ -424,8 +424,13 @@ _cpp_handle_directive (cpp_reader *pfile, int indented)
 	 does not cause '#define foo bar' to get executed when
 	 compiled with -save-temps, we recognize directives in
 	 -fpreprocessed mode only if the # is in column 1.  macro.c
-	 puts a space in front of any '#' at the start of a macro.  */
+	 puts a space in front of any '#' at the start of a macro.
+	 
+	 We exclude the -fdirectives-only case because macro expansion
+	 has not been performed yet, and block comments can cause spaces
+	 to preceed the directive.  */
       if (CPP_OPTION (pfile, preprocessed)
+	  && !CPP_OPTION (pfile, directives_only)
 	  && (indented || !(dir->flags & IN_I)))
 	{
 	  skip = 0;
diff --git a/libcpp/files.c b/libcpp/files.c
index b20c38e..30f77a5 100644
--- a/libcpp/files.c
+++ b/libcpp/files.c
@@ -788,7 +788,8 @@ _cpp_stack_file (cpp_reader *pfile, _cpp_file *file, bool import)
 
   /* Stack the buffer.  */
   buffer = cpp_push_buffer (pfile, file->buffer, file->st.st_size,
-			    CPP_OPTION (pfile, preprocessed));
+			    CPP_OPTION (pfile, preprocessed)
+			    && !CPP_OPTION (pfile, directives_only));
   buffer->file = file;
   buffer->sysp = sysp;
 
diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
index 0edcf65..333e134 100644
--- a/libcpp/include/cpplib.h
+++ b/libcpp/include/cpplib.h
@@ -440,6 +440,9 @@ struct cpp_options
 
   /* True means error callback should be used for diagnostics.  */
   bool client_diagnostic;
+
+  /* True disables tokenization outside of preprocessing directives. */
+  bool directives_only;
 };
 
 /* Callback for header lookup for HEADER, which is the name of a
@@ -644,6 +647,10 @@ extern struct deps *cpp_get_deps (cpp_reader *);
    too.  If there was an error opening the file, it returns NULL.  */
 extern const char *cpp_read_main_file (cpp_reader *, const char *);
 
+/* Set up built-ins with special behavior.  Use cpp_init_builtins()
+   instead unless your know what you are doing.  */
+extern void cpp_init_special_builtins (cpp_reader *);
+
 /* Set up built-ins like __FILE__.  */
 extern void cpp_init_builtins (cpp_reader *, int);
 
diff --git a/libcpp/init.c b/libcpp/init.c
index 71583df..62f4f95 100644
--- a/libcpp/init.c
+++ b/libcpp/init.c
@@ -350,11 +350,8 @@ mark_named_operators (cpp_reader *pfile)
     }
 }
 
-/* Read the builtins table above and enter them, and language-specific
-   macros, into the hash table.  HOSTED is true if this is a hosted
-   environment.  */
 void
-cpp_init_builtins (cpp_reader *pfile, int hosted)
+cpp_init_special_builtins (cpp_reader *pfile)
 {
   const struct builtin *b;
   size_t n = ARRAY_SIZE (builtin_array);
@@ -363,10 +360,7 @@ cpp_init_builtins (cpp_reader *pfile, int hosted)
     n -= 2;
   else if (! CPP_OPTION (pfile, stdc_0_in_system_headers)
 	   || CPP_OPTION (pfile, std))
-    {
-      n--;
-      _cpp_define_builtin (pfile, "__STDC__ 1");
-    }
+    n--;
 
   for (b = builtin_array; b < builtin_array + n; b++)
     {
@@ -375,6 +369,20 @@ cpp_init_builtins (cpp_reader *pfile, int hosted)
       hp->flags |= NODE_BUILTIN | NODE_WARN;
       hp->value.builtin = (enum builtin_type) b->value;
     }
+}
+
+/* Read the builtins table above and enter them, and language-specific
+   macros, into the hash table.  HOSTED is true if this is a hosted
+   environment.  */
+void
+cpp_init_builtins (cpp_reader *pfile, int hosted)
+{
+  cpp_init_special_builtins (pfile);
+
+  if (!CPP_OPTION (pfile, traditional)
+      && (! CPP_OPTION (pfile, stdc_0_in_system_headers)
+	  || CPP_OPTION (pfile, std)))
+    _cpp_define_builtin (pfile, "__STDC__ 1");
 
   if (CPP_OPTION (pfile, cplusplus))
     _cpp_define_builtin (pfile, "__cplusplus 1");
@@ -622,7 +630,8 @@ post_options (cpp_reader *pfile)
      preprocessed text.  Read preprocesed source in ISO mode.  */
   if (CPP_OPTION (pfile, preprocessed))
     {
-      pfile->state.prevent_expansion = 1;
+      if (!CPP_OPTION (pfile, directives_only))
+	pfile->state.prevent_expansion = 1;
       CPP_OPTION (pfile, traditional) = 0;
     }
 
diff --git a/libcpp/internal.h b/libcpp/internal.h
index d000cfd..8561088 100644
--- a/libcpp/internal.h
+++ b/libcpp/internal.h
@@ -575,6 +575,17 @@ extern void _cpp_do_file_change (cpp_reader *, enum lc_reason, const char *,
 				 unsigned int, unsigned int);
 extern void _cpp_pop_buffer (cpp_reader *);
 
+/* In directives.c */
+struct _cpp_dir_only_callbacks
+{
+  /* Called to print a block of lines. */
+  void (*print_lines) (int, const void *, size_t);
+  void (*maybe_print_line) (source_location);
+};
+
+extern void _cpp_preprocess_dir_only (cpp_reader *,
+				      const struct _cpp_dir_only_callbacks *);
+
 /* In traditional.c.  */
 extern bool _cpp_scan_out_logical_line (cpp_reader *, cpp_macro *);
 extern bool _cpp_read_logical_line_trad (cpp_reader *);

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Andrew Pinski]

On 6/14/07, Ollie Wild <aaw@google.com> wrote:
> Michael Chastain discovered a bug wherein I was incorrectly handling
> files without terminating newline characters.  The attached patch is
> identical to the one at
> http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html except that it
> properly handles non-newline terminated files inside
> scan_translation_unit_directives_only().

See this is why we should have kept the warning, oh well.

-- Pinski

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[Mike Stump]

On Apr 19, 2007, at 2:30 AM, François-Xavier Coudert wrote:
> I don't know what personal incentive you have for this, but we  
> really need it to get released at some point.

Putting this patch into 4.2 isn't going to affect the 4.2 timeline or  
release, other than to make the compiler faster for those that use  
distcc type technology.

A 30% leap in compile time may well be more important to us than all  
the regressions combined.  This is important to us, because we have  
users where it is important to them.  It's important to me, as it  
makes my users happier.  30 fixed regressions that none of my users  
hit affects them in no way.  If any of the regressions do affect  
them, we'd fix them.

The competition is still 2x faster...

Re: PATCH: distcc and ccache speedup: adds directives-only preprocessing

[François-Xavier Coudert]

> Personally, I'd like it for 4.2 as well. It looks safe enough to backport.

It's the second time I see such a request in a short time, so I'll say
it just to see if I'm the only one thinking this way: I'm quite
surprised that, with the discussion going on about how the 4.2.0
release is long-overdue, oscillating between stage 3 and release
candidate because of the slow rate at which important bugs get fixed,
some people still think of it as a good time to fit in new features.

I don't know what personal incentive you have for this (although I
think I recall a mention of Apple going for 4.2, in a mail about the
ABI stability of libgomp), but we really need it to get released at
some point.

FX