public inbox for gcc-patches@gcc.gnu.org
 help / color / mirror / Atom feed
* PATCH PING: distcc and ccache speedup: adds directives-only preprocessing
@ 2007-07-25 16:10 Ian Lance Taylor
  2007-07-25 16:27 ` Andreas Schwab
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Ian Lance Taylor @ 2007-07-25 16:10 UTC (permalink / raw)
  To: gnu, wilson; +Cc: mark, aaw, tromey, gcc-patches

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

This is the patch to speed up the preprocessor that I mentioned at the
summit.  I've reviewed the patch a few times now, and it looks good to
me.  It has been in active use at Google for every compilation for
about six months with no problems.

I would approve the patch if I could, but, unfortunately, since it
adds a new option and a new way of reading files, it is probably
non-algorithmic.  That means that I can't approve it, nor can Tom.  We
need an approval from a global write maintainer, or from a C or C++
front-end maintainer.

Can somebody take a look?  Thanks.

Ian


[-- Attachment #2: Type: message/rfc822, Size: 31254 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 4101 bytes --]

> I just wanted to send out a quick reminder since I haven't heard back
> on this for a while.  I imagine Mark is pretty overworked at the
> moment with the pending 4.2 release.  Would one of the other C/C++
> maintainers like to take a stab at this?

Michael Chastain discovered a bug wherein I was incorrectly handling
files without terminating newline characters.  The attached patch is
identical to the one at
http://gcc.gnu.org/ml/gcc-patches/2007-02/msg02178.html except that it
properly handles non-newline terminated files inside
scan_translation_unit_directives_only().


A quick summary recap:

This patch decreases distcc build and ccache lookup times by providing
an optimized "directives-only" preprocessing mode.

Distcc improves build performance by preprocessing files locally,
transmitting the preprocessed code to one or more remote build
servers, and performing compilation on the remote servers.  In
practice, the time to preprocess files locally is the limiting factor
in further reducing build time.  This patch adds a -fdirectives-only
flag which instructs the preprocessor to treat directives normally but
bypass tokenization of other preprocessing tokens.  Additional
preprocessing, such as macro expansion outside of directives, is
performed during the compilation phase.

Ccache reduces recompilation time by caching the results of previous
compilations and detecting when a compilation is repeated.  It
generates a lookup key from the preprocessed source code.  The
-directives-only flag improves lookup performance by reducing key
generation time.

For a typical c++ file, the new option reduces preprocessing time by
about 30%.  This in turn results in a similar decrease in overall
distcc build and ccache lookup times.

Usage is straightforward.  The -fdirectives-only flag is simply added
to the compiler options passed to distcc or ccache.

Tested with a C/C++ bootstrap and testsuite run on i686-pc-linux-gnu.
Validated two-stage processing by rerunning the C/C++ testsuites with
xgcc and g++ replaced by scripts which invoke ccache.

Ollie

:ADDPATCH libcpp:

2007-06-14 Ollie Wild <aaw@google.com>

       * gcc.dg/cpp/dir-only-1.c: New test.
       * gcc.dg/cpp/dir-only-1.h: New file.
       * gcc.dg/cpp/dir-only-2.c: New test.
       * gcc.dg/cpp/dir-only-3.c: New test.
       * gcc.dg/cpp/dir-only-3a.h: New file.
       * gcc.dg/cpp/dir-only-3b.h: New file.
       * gcc.dg/cpp/dir-only-4.c: New test.
       * gcc.dg/cpp/dir-only-5.c: New test.
       * gcc.dg/cpp/dir-only-6.c: New test.


2007-06-14 Ollie Wild <aaw@google.com>

       * directives-only.c: New file.
       * internal.h (struct _cpp_dir_only_callbacks): New.
       (_cpp_preprocess_dir_only): New function.
       * directives.c (_cpp_handle_directive): Check directives_only before
       disabling execution of indented directives.
       * files.c (_cpp_stack_file): Add directives_only check.
       * include/cpplib.h (struct cpp_options): Add directives_only.
       (cpp_init_special_builtins): New function.
       * init.c (cpp_init_special_builtins): New function.
       (cpp_init_builtins): Move builtin_array initialization to
       cpp_init_special_builtins.
       (post_options): Check directives_only before setting
       pfile->state.prevent_expansion = 1.
       * Makefile.in (libcpp_a_OBJS): Add directives-only.o.
       (libcpp_a_SOURCES): Add directives-only.c.


2007-06-14 Ollie Wild <aaw@google.com>

       * c-ppoutput.c (print_lines_directives_only): New function.
       (scan_translation_unit_directives_only): New function.
       (preprocess_file): Add call to scan_translation_unit_directives_only.
       * c-opts.c (c_common_handle_option): Add OPT_fdirectives_only.
       (sanitize_cpp_opts): Add default flag_dump_macros setting for
       -fdirectives-only.  Add errors for -fdirectives-only conflict with
       -Wunused-macros and -traditional.
       (finish_options): Add builtin macro initialization for
       -fdirectives-only + -fpreprocessed.
       * c.opt (fdirectives-only): New.
       * doc/cppopts.texi (fdirectives-only): New.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2.1.2: directives-only.patch --]
[-- Type: text/x-patch; name=directives-only.patch, Size: 23067 bytes --]

diff --git a/gcc/c-opts.c b/gcc/c-opts.c
index d0fafc6..79a49a3 100644
--- a/gcc/c-opts.c
+++ b/gcc/c-opts.c
@@ -616,6 +616,10 @@ c_common_handle_option (size_t scode, const char *arg, int value)
 	disable_builtin_function (arg);
       break;
 
+    case OPT_fdirectives_only:
+      cpp_opts->directives_only = 1;
+      break;
+
     case OPT_fdollars_in_identifiers:
       cpp_opts->dollars_in_ident = value;
       break;
@@ -1409,6 +1413,11 @@ sanitize_cpp_opts (void)
   if (flag_dump_macros == 'M')
     flag_no_output = 1;
 
+  /* By default, -fdirectives-only implies -dD.  This allows subsequent phases
+     to perform proper macro expansion.  */
+  if (cpp_opts->directives_only && !cpp_opts->preprocessed && !flag_dump_macros)
+    flag_dump_macros = 'D';
+
   /* Disable -dD, -dN and -dI if normal output is suppressed.  Allow
      -dM since at least glibc relies on -M -dM to work.  */
   /* Also, flag_no_output implies flag_no_line_commands, always.  */
@@ -1439,6 +1448,14 @@ sanitize_cpp_opts (void)
      actually output the current directory?  */
   if (flag_working_directory == -1)
     flag_working_directory = (debug_info_level != DINFO_LEVEL_NONE);
+
+  if (cpp_opts->directives_only)
+    {
+      if (warn_unused_macros)
+	error ("-fdirectives-only is incompatible with -Wunused_macros");
+      if (cpp_opts->traditional)
+	error ("-fdirectives-only is incompatible with -traditional");
+    }
 }
 
 /* Add include path with a prefix at the front of its name.  */
@@ -1522,6 +1539,8 @@ finish_options (void)
 	    }
 	}
     }
+  else if (cpp_opts->directives_only)
+    cpp_init_special_builtins (parse_in);
 
   include_cursor = 0;
   push_command_line_include ();
diff --git a/gcc/c-ppoutput.c b/gcc/c-ppoutput.c
index 10540b7..62b350b 100644
--- a/gcc/c-ppoutput.c
+++ b/gcc/c-ppoutput.c
@@ -41,6 +41,8 @@ static struct
 
 /* General output routines.  */
 static void scan_translation_unit (cpp_reader *);
+static void print_lines_directives_only (int, const void *, size_t);
+static void scan_translation_unit_directives_only (cpp_reader *);
 static void scan_translation_unit_trad (cpp_reader *);
 static void account_for_newlines (const unsigned char *, size_t);
 static int dump_macro (cpp_reader *, cpp_hashnode *, void *);
@@ -75,6 +77,9 @@ preprocess_file (cpp_reader *pfile)
     }
   else if (cpp_get_options (pfile)->traditional)
     scan_translation_unit_trad (pfile);
+  else if (cpp_get_options (pfile)->directives_only
+	   && !cpp_get_options (pfile)->preprocessed)
+    scan_translation_unit_directives_only (pfile);
   else
     scan_translation_unit (pfile);
 
@@ -179,6 +184,26 @@ scan_translation_unit (cpp_reader *pfile)
     }
 }
 
+static void
+print_lines_directives_only (int lines, const void *buf, size_t size)
+{
+  print.src_line += lines;
+  fwrite (buf, 1, size, print.outf);
+}
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+static void
+scan_translation_unit_directives_only (cpp_reader *pfile)
+{
+  struct _cpp_dir_only_callbacks cb;
+
+  cb.print_lines = print_lines_directives_only;
+  cb.maybe_print_line = maybe_print_line;
+
+  _cpp_preprocess_dir_only (pfile, &cb);
+}
+
 /* Adjust print.src_line for newlines embedded in output.  */
 static void
 account_for_newlines (const unsigned char *str, size_t len)
diff --git a/gcc/c.opt b/gcc/c.opt
index 63e2bda..dc6cff1 100644
--- a/gcc/c.opt
+++ b/gcc/c.opt
@@ -524,6 +524,10 @@ fdefault-inline
 C++ ObjC++
 Inline member functions by default
 
+fdirectives-only
+C ObjC C++ ObjC++
+Preprocess directives only.
+
 fdollars-in-identifiers
 C ObjC C++ ObjC++
 Permit '$' as an identifier character
diff --git a/gcc/doc/cppopts.texi b/gcc/doc/cppopts.texi
index 17235bd..14187b6 100644
--- a/gcc/doc/cppopts.texi
+++ b/gcc/doc/cppopts.texi
@@ -518,6 +518,22 @@ Search @var{dir} only for header files requested with
 If @var{dir} begins with @code{=}, then the @code{=} will be replaced
 by the sysroot prefix; see @option{--sysroot} and @option{-isysroot}.
 
+@item -fdirectives-only
+@opindex fdirectives-only
+This option provides a simplified preprocessor to improve the
+performance of distributed build systems such as distcc.  It's
+behavior depends on a number of other flags.
+
+If the @option{-E} option is enabled, it suppresses things like macro
+expansion, trigraph conversion, and escaped newline splicing
+outside of directives.  All directives are processed normally, except that
+macro definitions are output similar to the @option{-dD} option.
+
+If the @option{-fpreprocessed} option is enabled, it suppresses
+predefinition of most builtin and command line macros.  This
+prevents duplicate definition of macros output with the @option{-E}
+option.
+
 @item -fdollars-in-identifiers
 @opindex fdollars-in-identifiers
 @anchor{fdollars-in-identifiers}
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-1.c b/gcc/testsuite/gcc.dg/cpp/dir-only-1.c
new file mode 100644
index 0000000..3c22616
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-1.c
@@ -0,0 +1,73 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* Tests scan_translation_unit_directives_only()'s handling of corner cases. */
+
+/* Ignore directives inside block comments...
+#error directive inside block comment
+*/
+
+// Escaped newline doesn't terminate line comment \
+#error directive inside line comment
+
+/* A comment canot start inside a string. */
+const char *c1 = "/*";
+#define NOT_IN_COMMENT
+const char *c2 = "*/";
+#ifndef NOT_IN_COMMENT
+#error Comment started inside a string literal
+#endif
+
+/* Escaped newline handling. */
+int i; \
+#error ignored escaped newline
+  \
+  \
+#define BOL
+#ifndef BOL
+#error escaped newline did not preserve beginning of line
+#endif
+
+/* Handles \\ properly at the end of a string. */
+"string ends in \\"/*
+#error Missed string terminator.
+*/
+
+/* Handles macro expansion in preprocessing directives. */
+#define HEADER "dir-only-1.h"
+#include HEADER
+#ifndef GOT_HEADER
+#error Failed to include header.
+#endif
+
+/\
+*
+#define IN_COMMENT
+*/
+#ifdef IN_COMMENT
+#error Escaped newline breaks block comment initiator.
+#endif
+
+/*
+*\
+/
+#define NOT_IN_COMMENT2
+/**/
+#ifndef NOT_IN_COMMENT2
+#error Escaped newline breaks block comment terminator.
+#endif
+
+/* Test escaped newline inside character escape sequence. */
+"\\
+\"/*
+#error Missed string terminator
+*/
+
+/* Block comments don't mask trailing preprocessing
+   directive. */ #define NOT_MASKED
+#ifndef NOT_MASKED
+#error Comment masks trailing directive.
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-1.h b/gcc/testsuite/gcc.dg/cpp/dir-only-1.h
new file mode 100644
index 0000000..96dbcc0
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-1.h
@@ -0,0 +1,3 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+#define GOT_HEADER
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-2.c b/gcc/testsuite/gcc.dg/cpp/dir-only-2.c
new file mode 100644
index 0000000..489b4d6
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-2.c
@@ -0,0 +1,12 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fpreprocessed -fdirectives-only -DNOT_SET" } */
+
+/* Tests -fdirectives-only + -fpreprocessed. */
+
+/* Check this is not defined. */
+#ifdef NOT_SET
+#error Command line macro not disabled.
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-3.c b/gcc/testsuite/gcc.dg/cpp/dir-only-3.c
new file mode 100644
index 0000000..d97c23f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-3.c
@@ -0,0 +1,13 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -H" } */
+/* { dg-error "dir-only-3a\.h\n\[^\n\]*dir-only-3b\.h\n\[^\n\]*dir-only-3a\.h\n" "include guard check" { target *-*-* } 0 } */
+
+/* Tests include guards. */
+
+#include "dir-only-3a.h"
+#include "dir-only-3b.h"
+#include "dir-only-3b.h"
+#include "dir-only-3a.h"
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-3a.h b/gcc/testsuite/gcc.dg/cpp/dir-only-3a.h
new file mode 100644
index 0000000..6644bbf
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-3a.h
@@ -0,0 +1,8 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+extern int outside_guard
+
+#ifndef DIR_ONLY_3A_H
+#define DIR_ONLY_3A_H
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-3b.h b/gcc/testsuite/gcc.dg/cpp/dir-only-3b.h
new file mode 100644
index 0000000..4edaa7b
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-3b.h
@@ -0,0 +1,9 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+#ifndef DIR_ONLY_3B_H
+#define DIR_ONLY_3B_H
+
+extern int inside guard;
+
+#endif
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-4.c b/gcc/testsuite/gcc.dg/cpp/dir-only-4.c
new file mode 100644
index 0000000..a7b5f04
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-4.c
@@ -0,0 +1,6 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -Wunused-macros" } */
+/* { dg-error "-fdirectives-only is incompatible with -Wunused_macros\n" "-Wunused-macros check" { target *-*-* } 0 } */
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-5.c b/gcc/testsuite/gcc.dg/cpp/dir-only-5.c
new file mode 100644
index 0000000..643a4d7
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-5.c
@@ -0,0 +1,6 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options "-fdirectives-only -traditional" } */
+/* { dg-error "-fdirectives-only is incompatible with -traditional\n" "-traditional check" { target *-*-* } 0 } */
diff --git a/gcc/testsuite/gcc.dg/cpp/dir-only-6.c b/gcc/testsuite/gcc.dg/cpp/dir-only-6.c
new file mode 100644
index 0000000..0023205
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/cpp/dir-only-6.c
@@ -0,0 +1,7 @@
+/* Copyright 2007 Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.  */
+
+/* { dg-do preprocess } */
+/* { dg-options -fdirectives-only } */
+
+/* { dg-error "unterminated comment" }
diff --git a/libcpp/Makefile.in b/libcpp/Makefile.in
index 166f1fa..960a42d 100644
--- a/libcpp/Makefile.in
+++ b/libcpp/Makefile.in
@@ -69,14 +69,14 @@ INCLUDES = -I$(srcdir) -I. -I$(srcdir)/../include @INCINTL@ \
 
 ALL_CFLAGS = $(CFLAGS) $(WARN_CFLAGS) $(INCLUDES) $(CPPFLAGS)
 
-libcpp_a_OBJS = charset.o directives.o errors.o expr.o files.o \
-	identifiers.o init.o lex.o line-map.o macro.o mkdeps.o \
-	pch.o symtab.o traditional.o
+libcpp_a_OBJS = charset.o directives.o directives-only.o errors.o \
+	expr.o files.o identifiers.o init.o lex.o line-map.o macro.o \
+	mkdeps.o pch.o symtab.o traditional.o
 makedepend_OBJS = makedepend.o
 
-libcpp_a_SOURCES = charset.c directives.c errors.c expr.c files.c \
-	identifiers.c init.c lex.c line-map.c macro.c mkdeps.c \
-	pch.c symtab.c traditional.c
+libcpp_a_SOURCES = charset.c directives.c directives-only.c errors.c \
+	expr.c files.c identifiers.c init.c lex.c line-map.c macro.c \
+	mkdeps.c pch.c symtab.c traditional.c
 
 all: libcpp.a makedepend$(EXEEXT) $(USED_CATALOGS)
 
diff --git a/libcpp/directives-only.c b/libcpp/directives-only.c
new file mode 100644
index 0000000..d50cebb
--- /dev/null
+++ b/libcpp/directives-only.c
@@ -0,0 +1,240 @@
+/* CPP Library - directive only preprocessing for distributed compilation.
+   Copyright (C) 2007
+   Free Software Foundation, Inc.
+   Contributed by Ollie Wild <aaw@google.com>.
+
+This program is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation; either version 2, or (at your option) any
+later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */
+
+#include "config.h"
+#include "system.h"
+#include "cpplib.h"
+#include "internal.h"
+
+/* DO (Directive only) flags. */
+#define DO_BOL		 (1 << 0) /* At the beginning of a logical line. */
+#define DO_STRING	 (1 << 1) /* In a string constant. */
+#define DO_CHAR		 (1 << 2) /* In a character constant. */
+#define DO_BLOCK_COMMENT (1 << 3) /* In a block comment. */
+#define DO_LINE_COMMENT	 (1 << 4) /* In a single line "//-style" comment. */
+
+#define DO_LINE_SPECIAL (DO_STRING | DO_CHAR | DO_LINE_COMMENT)
+#define DO_SPECIAL	(DO_LINE_SPECIAL | DO_BLOCK_COMMENT)
+
+/* Writes out the preprocessed file, handling spacing and paste
+   avoidance issues.  */
+void
+_cpp_preprocess_dir_only (cpp_reader *pfile,
+			  const struct _cpp_dir_only_callbacks *cb)
+{
+  struct cpp_buffer *buffer;
+  const unsigned char *cur, *base, *next_line, *rlimit;
+  cppchar_t c, last_c;
+  unsigned flags;
+  int lines, col;
+  source_location loc;
+
+ restart:
+  /* Buffer initialization ala _cpp_clean_line(). */
+  buffer = pfile->buffer;
+  buffer->cur_note = buffer->notes_used = 0;
+  buffer->cur = buffer->line_base = buffer->next_line;
+  buffer->need_line = false;
+
+  /* This isn't really needed.  It prevents a compiler warning, though. */
+  loc = pfile->line_table->highest_line;
+
+  /* Scan initialization. */
+  next_line = cur = base = buffer->cur;
+  rlimit = buffer->rlimit;
+  flags = DO_BOL;
+  lines = 0;
+  col = 1;
+
+  for (last_c = '\n', c = *cur; cur < rlimit; last_c = c, c = *++cur, ++col)
+    {
+      /* Skip over escaped newlines. */
+      if (__builtin_expect (c == '\\', false))
+	{
+	  const unsigned char *tmp = cur + 1;
+
+	  while (is_nvspace (*tmp) && tmp < rlimit)
+	    tmp++;
+	  if (*tmp == '\r')
+	    tmp++;
+	  if (*tmp == '\n' && tmp < rlimit)
+	    {
+	      CPP_INCREMENT_LINE (pfile, 0);
+	      lines++;
+	      col = 0;
+	      cur = tmp;
+	      c = last_c;
+	      continue;
+	    }
+	}
+
+      if (__builtin_expect (last_c == '#', false) && !(flags & DO_SPECIAL))
+	{
+	  if (c != '#' && (flags & DO_BOL))
+	  {
+	    struct line_maps *line_table;
+
+	    if (!pfile->state.skipping && next_line != base)
+	      cb->print_lines (lines, base, next_line - base);
+
+	    /* Prep things for directive handling. */
+	    buffer->next_line = cur;
+	    buffer->need_line = true;
+	    _cpp_get_fresh_line (pfile);
+
+	    /* Ensure proper column numbering for generated error messages. */
+	    buffer->line_base -= col - 1;
+
+	    _cpp_handle_directive (pfile, 0 /* ignore indented */);
+
+	    /* Sanitize the line settings.  Duplicate #include's can mess
+	       things up. */
+	    line_table = pfile->line_table;
+	    line_table->highest_location = line_table->highest_line;
+
+	    /* The if block prevents us from outputing line information when
+	       the file ends with a directive and no newline.  Note that we
+	       must use pfile->buffer, not buffer. */
+	    if (pfile->buffer->next_line < pfile->buffer->rlimit)
+	      cb->maybe_print_line (pfile->line_table->highest_line);
+
+	    goto restart;
+	  }
+
+	  flags &= ~DO_BOL;
+	  pfile->mi_valid = false;
+	}
+      else if (__builtin_expect (last_c == '/', false) \
+	       && !(flags & DO_SPECIAL) && c != '*' && c != '/')
+	{
+	  /* If a previous slash is not starting a block comment, clear the
+	     DO_BOL flag.  */
+	  flags &= ~DO_BOL;
+	  pfile->mi_valid = false;
+	}
+
+      switch (c)
+	{
+	case '/':
+	  if ((flags & DO_BLOCK_COMMENT) && last_c == '*')
+	    {
+	      flags &= ~DO_BLOCK_COMMENT;
+	      c = 0;
+	    }
+	  else if (!(flags & DO_SPECIAL) && last_c == '/')
+	    flags |= DO_LINE_COMMENT;
+	  else if (!(flags & DO_SPECIAL))
+	    /* Mark the position for possible error reporting. */
+	    LINEMAP_POSITION_FOR_COLUMN (loc, pfile->line_table, col);
+
+	  break;
+
+	case '*':
+	  if (!(flags & DO_SPECIAL))
+	    {
+	      if (last_c == '/')
+		flags |= DO_BLOCK_COMMENT;
+	      else
+		{
+		  flags &= ~DO_BOL;
+		  pfile->mi_valid = false;
+		}
+	    }
+
+	  break;
+
+	case '\'':
+	case '"':
+	  {
+	    unsigned state = (c == '"') ? DO_STRING : DO_CHAR;
+
+	    if (!(flags & DO_SPECIAL))
+	      {
+		flags |= state;
+		flags &= ~DO_BOL;
+		pfile->mi_valid = false;
+	      }
+	    else if ((flags & state) && last_c != '\\')
+	      flags &= ~state;
+
+	    break;
+	  }
+
+	case '\\':
+	  {
+	    if ((flags & (DO_STRING | DO_CHAR)) && last_c == '\\')
+	      c = 0;
+
+	    if (!(flags & DO_SPECIAL))
+	      {
+		flags &= ~DO_BOL;
+		pfile->mi_valid = false;
+	      }
+
+	    break;
+	  }
+
+	case '\n':
+	  CPP_INCREMENT_LINE (pfile, 0);
+	  lines++;
+	  col = 0;
+	  flags &= ~DO_LINE_SPECIAL;
+	  if (!(flags & DO_SPECIAL))
+	    flags |= DO_BOL;
+	  break;
+
+	case '#':
+	  next_line = cur;
+	  /* Don't update DO_BOL yet. */
+	  break;
+
+	case ' ': case '\t': case '\f': case '\v': case '\0':
+	  break;
+
+	default:
+	  if (!(flags & DO_SPECIAL))
+	    {
+	      flags &= ~DO_BOL;
+	      pfile->mi_valid = false;
+	    }
+	  break;
+	}
+    }
+
+  if (flags & DO_BLOCK_COMMENT)
+    cpp_error_with_line (pfile, CPP_DL_ERROR, loc, 0, "unterminated comment");
+
+  if (!pfile->state.skipping && cur != base)
+    {
+      /* If the file was not newline terminated, add rlimit, which is
+         guaranteed to point to a newline, to the end of our range.  */
+      if (cur[-1] != '\n')
+	{
+	  cur++;
+	  CPP_INCREMENT_LINE (pfile, 0);
+	  lines++;
+	}
+
+      cb->print_lines (lines, base, cur - base);
+    }
+
+  _cpp_pop_buffer (pfile);
+  if (pfile->buffer)
+    goto restart;
+}
diff --git a/libcpp/directives.c b/libcpp/directives.c
index ccb9f32..38ca949 100644
--- a/libcpp/directives.c
+++ b/libcpp/directives.c
@@ -424,8 +424,13 @@ _cpp_handle_directive (cpp_reader *pfile, int indented)
 	 does not cause '#define foo bar' to get executed when
 	 compiled with -save-temps, we recognize directives in
 	 -fpreprocessed mode only if the # is in column 1.  macro.c
-	 puts a space in front of any '#' at the start of a macro.  */
+	 puts a space in front of any '#' at the start of a macro.
+	 
+	 We exclude the -fdirectives-only case because macro expansion
+	 has not been performed yet, and block comments can cause spaces
+	 to preceed the directive.  */
       if (CPP_OPTION (pfile, preprocessed)
+	  && !CPP_OPTION (pfile, directives_only)
 	  && (indented || !(dir->flags & IN_I)))
 	{
 	  skip = 0;
diff --git a/libcpp/files.c b/libcpp/files.c
index b20c38e..30f77a5 100644
--- a/libcpp/files.c
+++ b/libcpp/files.c
@@ -788,7 +788,8 @@ _cpp_stack_file (cpp_reader *pfile, _cpp_file *file, bool import)
 
   /* Stack the buffer.  */
   buffer = cpp_push_buffer (pfile, file->buffer, file->st.st_size,
-			    CPP_OPTION (pfile, preprocessed));
+			    CPP_OPTION (pfile, preprocessed)
+			    && !CPP_OPTION (pfile, directives_only));
   buffer->file = file;
   buffer->sysp = sysp;
 
diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
index 0edcf65..333e134 100644
--- a/libcpp/include/cpplib.h
+++ b/libcpp/include/cpplib.h
@@ -440,6 +440,9 @@ struct cpp_options
 
   /* True means error callback should be used for diagnostics.  */
   bool client_diagnostic;
+
+  /* True disables tokenization outside of preprocessing directives. */
+  bool directives_only;
 };
 
 /* Callback for header lookup for HEADER, which is the name of a
@@ -644,6 +647,10 @@ extern struct deps *cpp_get_deps (cpp_reader *);
    too.  If there was an error opening the file, it returns NULL.  */
 extern const char *cpp_read_main_file (cpp_reader *, const char *);
 
+/* Set up built-ins with special behavior.  Use cpp_init_builtins()
+   instead unless your know what you are doing.  */
+extern void cpp_init_special_builtins (cpp_reader *);
+
 /* Set up built-ins like __FILE__.  */
 extern void cpp_init_builtins (cpp_reader *, int);
 
diff --git a/libcpp/init.c b/libcpp/init.c
index 71583df..62f4f95 100644
--- a/libcpp/init.c
+++ b/libcpp/init.c
@@ -350,11 +350,8 @@ mark_named_operators (cpp_reader *pfile)
     }
 }
 
-/* Read the builtins table above and enter them, and language-specific
-   macros, into the hash table.  HOSTED is true if this is a hosted
-   environment.  */
 void
-cpp_init_builtins (cpp_reader *pfile, int hosted)
+cpp_init_special_builtins (cpp_reader *pfile)
 {
   const struct builtin *b;
   size_t n = ARRAY_SIZE (builtin_array);
@@ -363,10 +360,7 @@ cpp_init_builtins (cpp_reader *pfile, int hosted)
     n -= 2;
   else if (! CPP_OPTION (pfile, stdc_0_in_system_headers)
 	   || CPP_OPTION (pfile, std))
-    {
-      n--;
-      _cpp_define_builtin (pfile, "__STDC__ 1");
-    }
+    n--;
 
   for (b = builtin_array; b < builtin_array + n; b++)
     {
@@ -375,6 +369,20 @@ cpp_init_builtins (cpp_reader *pfile, int hosted)
       hp->flags |= NODE_BUILTIN | NODE_WARN;
       hp->value.builtin = (enum builtin_type) b->value;
     }
+}
+
+/* Read the builtins table above and enter them, and language-specific
+   macros, into the hash table.  HOSTED is true if this is a hosted
+   environment.  */
+void
+cpp_init_builtins (cpp_reader *pfile, int hosted)
+{
+  cpp_init_special_builtins (pfile);
+
+  if (!CPP_OPTION (pfile, traditional)
+      && (! CPP_OPTION (pfile, stdc_0_in_system_headers)
+	  || CPP_OPTION (pfile, std)))
+    _cpp_define_builtin (pfile, "__STDC__ 1");
 
   if (CPP_OPTION (pfile, cplusplus))
     _cpp_define_builtin (pfile, "__cplusplus 1");
@@ -622,7 +630,8 @@ post_options (cpp_reader *pfile)
      preprocessed text.  Read preprocesed source in ISO mode.  */
   if (CPP_OPTION (pfile, preprocessed))
     {
-      pfile->state.prevent_expansion = 1;
+      if (!CPP_OPTION (pfile, directives_only))
+	pfile->state.prevent_expansion = 1;
       CPP_OPTION (pfile, traditional) = 0;
     }
 
diff --git a/libcpp/internal.h b/libcpp/internal.h
index d000cfd..8561088 100644
--- a/libcpp/internal.h
+++ b/libcpp/internal.h
@@ -575,6 +575,17 @@ extern void _cpp_do_file_change (cpp_reader *, enum lc_reason, const char *,
 				 unsigned int, unsigned int);
 extern void _cpp_pop_buffer (cpp_reader *);
 
+/* In directives.c */
+struct _cpp_dir_only_callbacks
+{
+  /* Called to print a block of lines. */
+  void (*print_lines) (int, const void *, size_t);
+  void (*maybe_print_line) (source_location);
+};
+
+extern void _cpp_preprocess_dir_only (cpp_reader *,
+				      const struct _cpp_dir_only_callbacks *);
+
 /* In traditional.c.  */
 extern bool _cpp_scan_out_logical_line (cpp_reader *, cpp_macro *);
 extern bool _cpp_read_logical_line_trad (cpp_reader *);

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2007-07-30 18:34 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-07-25 16:10 PATCH PING: distcc and ccache speedup: adds directives-only preprocessing Ian Lance Taylor
2007-07-25 16:27 ` Andreas Schwab
2007-07-26  7:16 ` Mark Mitchell
2007-07-26 10:51   ` Dave Korn
2007-07-26 19:37   ` Ian Lance Taylor
2007-07-26 22:10   ` Ollie Wild
2007-07-29 19:29     ` Mark Mitchell
2007-07-30 18:53       ` Ollie Wild
2007-07-27 23:17 ` Michael Meissner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).