* compile server design document
@ 2003-02-27 5:55 Per Bothner
2003-02-27 6:13 ` Tolga Dalman
` (7 more replies)
0 siblings, 8 replies; 20+ messages in thread
From: Per Bothner @ 2003-02-27 5:55 UTC (permalink / raw)
To: gcc
[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]
As I've hinted, I have been working for Apple on a "compile server".
The idea is that the compiler when in server mode will re-use trees
from header files. I've attached a semi-detailed design documnent.
This is in texinfo format (so it can later be merged into gccint.text).
To generate html do: makeinfo --html gcc-server.texi
I'll submit a patch for the first prototype shortly, though note
that this is far from clean enough to check in as is. Before the
actual server, there is some amount of work in terms of splitting
up compiler initialization into two parts: one-time initialization,
and per-compilation initialization. I hope people will work with
me on this.
We're quite exited about this approach. We think it can lead
to substantial compile-time speed-ups, and possibly other benefits.
A preliminary benchmark suggested that re-using a large set of
header files (the Apple Carbon framework) was over 3x as fast as
normal processing.
--
--Per Bothner
per@bothner.com pbothner@apple.com http://www.bothner.com/per/
[-- Attachment #2: gcc-server.texi --]
[-- Type: application/x-texinfo, Size: 35713 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
@ 2003-02-27 6:13 ` Tolga Dalman
2003-02-27 7:08 ` Timothy J. Wood
` (6 subsequent siblings)
7 siblings, 0 replies; 20+ messages in thread
From: Tolga Dalman @ 2003-02-27 6:13 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
On Wed, 26 Feb 2003 17:27:12 -0800 Per Bothner <pbothner@apple.com> wrote:
> As I've hinted, I have been working for Apple on a "compile server".
> The idea is that the compiler when in server mode will re-use trees
> from header files. I've attached a semi-detailed design documnent.
> This is in texinfo format (so it can later be merged into gccint.text).
> To generate html do: makeinfo --html gcc-server.texi
>
> I'll submit a patch for the first prototype shortly, though note
> that this is far from clean enough to check in as is. Before the
> actual server, there is some amount of work in terms of splitting
> up compiler initialization into two parts: one-time initialization,
> and per-compilation initialization. I hope people will work with
> me on this.
>
> We're quite exited about this approach. We think it can lead
> to substantial compile-time speed-ups, and possibly other benefits.
> A preliminary benchmark suggested that re-using a large set of
> header files (the Apple Carbon framework) was over 3x as fast as
> normal processing.
what's the current status of the project? i mean, if you have had preliminary tests, we can expect some results soon, eh?
however, i've just looked over the paper, and the concept appears quite mature
(i haven't read it thoroughfully enough to judge properly, of course).
i'll be waiting for any patches ;)
Tolga Dalman.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
2003-02-27 6:13 ` Tolga Dalman
@ 2003-02-27 7:08 ` Timothy J. Wood
2003-02-27 7:27 ` Per Bothner
2003-02-27 10:14 ` Per Bothner
` (5 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Timothy J. Wood @ 2003-02-27 7:08 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
Cool!
Some comments:
- s/becomes we may allow/because.../
- The proposal doesn't explicitly say anything about threads, but you
could have each connection to the socket fork a thread. Are you
planning on having this be threaded or having dual processor
machines just start two servers? If you have two servers, the IDE
should really record 'file affinity' to avoid eventually growing both
servers to their maximum size.
- This design would let you get rid of all the PFE/PCH code for
persisting compiler state, assuming you want to dedicate VM
space to keeping a server alive. But, do you think it would be
worth the effort to keep the 'persist the compiler state' code
around so that a server could be kick started.
As an example, each IDE project might have its own
persistent server database stored in the build directory with the
.o files for that workspace. Thus, if I go through 15 different
IDE project while doing a large build, I might save some time.
This disallows cross-project sharing but would mean that you
could have a large number of projects 'active' w/o having a
absolutely huge VM sucking server running.
Basically, I'm just wondering if you have any data on this one
way or the other.
- Caching... what are your ideas on long lived servers? This is
especially relevant if you don't have per- IDE project servers
(since one server could get handed thousands of files and grow
without bound)
- 'initially' and 'init' seem less than ideal for the naming scheme.
More descriptive, shorter and more unique names are possible:
'init'/'reset' or 'startup'/'perfile', for example
- The 'depends-on-set' functionality seems like it would be very
useful in higher levels than the preprocessor (forgive me if it
misunderstood it... :) Consider the case:
foo.h:
... many other decls ...
extern void foo(float x);
... some more decls ...
foo.c:
void foo(float x) {...}
bar.c:
void bar(float x) { foo(x); }
If each decl was a 'fragment' then changing the type of the argument
to 'foo' would cause only foo.c and bar.c to be recompiled (instead of
all the files that include foo.h).
-tim
On Wednesday, February 26, 2003, at 05:27 PM, Per Bothner wrote:
> As I've hinted, I have been working for Apple on a "compile server".
> The idea is that the compiler when in server mode will re-use trees
> from header files. I've attached a semi-detailed design documnent.
> This is in texinfo format (so it can later be merged into gccint.text).
> To generate html do: makeinfo --html gcc-server.texi
>
> I'll submit a patch for the first prototype shortly, though note
> that this is far from clean enough to check in as is. Before the
> actual server, there is some amount of work in terms of splitting
> up compiler initialization into two parts: one-time initialization,
> and per-compilation initialization. I hope people will work with
> me on this.
>
> We're quite exited about this approach. We think it can lead
> to substantial compile-time speed-ups, and possibly other benefits.
> A preliminary benchmark suggested that re-using a large set of
> header files (the Apple Carbon framework) was over 3x as fast as
> normal processing.
> --
> --Per Bothner
> per@bothner.com pbothner@apple.com http://www.bothner.com/per/
> <gcc-server.texi>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 7:08 ` Timothy J. Wood
@ 2003-02-27 7:27 ` Per Bothner
0 siblings, 0 replies; 20+ messages in thread
From: Per Bothner @ 2003-02-27 7:27 UTC (permalink / raw)
To: Timothy J. Wood; +Cc: gcc
Timothy J. Wood wrote:
> - s/becomes we may allow/because.../
Fixed in the master.
> - The proposal doesn't explicitly say anything about threads, but you
> could have each connection to the socket fork a thread. Are you
> planning on having this be threaded
NO. That would require a more radical re-write of Gcc. Gcc just
keeps too much state in global variables. Plus you have to deal with
locking etc.
> or having dual processor machines just start two servers?
I think that makes more sense.
> - This design would let you get rid of all the PFE/PCH code for
> persisting compiler state, assuming you want to dedicate VM
> space to keeping a server alive. But, do you think it would be
> worth the effort to keep the 'persist the compiler state' code
> around so that a server could be kick started.
My gut feeling is no. But at a later point it might be worth
an experiment.
>
> As an example, each IDE project might have its own
> persistent server database stored in the build directory with the
> .o files for that workspace. Thus, if I go through 15 different
> IDE project while doing a large build, I might save some time.
Perhaps for release 2.0 of the compile server ...
>
> This disallows cross-project sharing but would mean that you
> could have a large number of projects 'active' w/o having a
> absolutely huge VM sucking server running.
>
> Basically, I'm just wondering if you have any data on this one
> way or the other.
No.
> - Caching... what are your ideas on long lived servers? This is
> especially relevant if you don't have per- IDE project servers
> (since one server could get handed thousands of files and grow
> without bound)
Well, right now it is a per-directory server. Per IDE-project
may be a future refinement.
>
> - 'initially' and 'init' seem less than ideal for the naming scheme.
> More descriptive, shorter and more unique names are possible:
> 'init'/'reset' or 'startup'/'perfile', for example
'startup'/'init' perhaps?
> - The 'depends-on-set' functionality seems like it would be very
> useful in higher levels than the preprocessor (forgive me if it
> misunderstood it... :) Consider the case:
>
> foo.h:
>
> ... many other decls ...
> extern void foo(float x);
> ... some more decls ...
>
> foo.c:
> void foo(float x) {...}
>
> bar.c:
> void bar(float x) { foo(x); }
>
> If each decl was a 'fragment' then changing the type of the argument
> to 'foo' would cause only foo.c and bar.c to be recompiled (instead of
> all the files that include foo.h).
There are interesting things you can do. For example, an IDE could
tell the compile server to only re-compile a single function.
The initial focus of the compile server is avoiding redundant heade
file compilation, but there are lots of things one could do beyond that.
--
--Per Bothner
per@bothner.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
2003-02-27 6:13 ` Tolga Dalman
2003-02-27 7:08 ` Timothy J. Wood
@ 2003-02-27 10:14 ` Per Bothner
2003-02-27 11:28 ` Joseph S. Myers
` (4 subsequent siblings)
7 siblings, 0 replies; 20+ messages in thread
From: Per Bothner @ 2003-02-27 10:14 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
[-- Attachment #1: Type: text/plain, Size: 363 bytes --]
Attached is a recent snapshot of the compile server.
This predates the C++ work which I've just started,
and is C-only. I.e. you should configure with
--enable-languages=c if you want to try it.
But mainly this is for illustration. It is certainly
not being proposed for checking into cvs as is!
--
--Per Bothner
per@bothner.com http://www.bothner.com/per/
[-- Attachment #2: gcc-server.patch --]
[-- Type: text/plain, Size: 56041 bytes --]
Index: c-common.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-common.h,v
retrieving revision 1.167
diff -c -r1.167 c-common.h
*** c-common.h 18 Jan 2003 02:26:40 -0000 1.167
--- c-common.h 8 Feb 2003 20:05:02 -0000
***************
*** 180,185 ****
--- 180,236 ----
struct cpp_hashnode node;
};
+ #define DECL_FRAGMENT(DECL) \
+ ((struct c_include_fragment *) DECL_CHECK (DECL)->decl.defining_fragment)
+ #define SET_DECL_FRAGMENT(DECL, FRAGMENT) \
+ (DECL_CHECK (DECL)->decl.defining_fragment = (tree) (FRAGMENT))
+ #define C_FRAGMENT(FRAG) ((struct c_include_fragment *) (FRAG)->start_marker)
+
+ struct c_include_fragment GTY(())
+ {
+ struct tree_common common;
+ const char *name;
+ /* Value of c_timestamp when this fragment was last read (parsed). */
+ int read_timestamp;
+ /* Value of c_timestamp when this fragment was last logically included. */
+ int include_timestamp;
+ /* True if a declaration in this fragmemt was used in current_c_fragment. */
+ unsigned used_in_current : 1;
+ unsigned valid : 1;
+ /* A TREE_VEC whose members are other c_include_fragments. An element is in
+ uses_fragments iff the current fragments uses (references) a declartion
+ in that element. */
+ tree uses_fragments;
+ tree bindings;
+ };
+
+ extern int main_timestamp;
+ extern int c_timestamp;
+ extern GTY(()) struct c_include_fragment *current_c_fragment;
+ extern int currently_nested;
+
+ extern void register_decl_dependency PARAMS ((tree));
+ extern struct c_include_fragment * alloc_include_fragment PARAMS ((void));
+ extern void remember_fragment_start PARAMS ((struct c_include_fragment *));
+ extern void remember_fragment_end PARAMS ((struct c_include_fragment *));
+ extern void restore_from_fragment PARAMS ((struct c_include_fragment *));
+ extern void note_fragment_binding_1 PARAMS ((tree));
+ extern void note_fragment_binding_2 PARAMS ((tree, tree));
+ extern void note_fragment_binding_3 PARAMS ((tree, tree, tree));
+ extern void setup_globals PARAMS ((void));
+
+ extern void* cb_push_fragment PARAMS ((cpp_reader*, cpp_fragment*, const char*, int));
+ extern void cb_pop_fragment PARAMS ((cpp_reader*, cpp_fragment*));
+
+ #if 0
+ #define FRAGMENT_OLDEST_DECL(NODE) \
+ (((struct c_include_fragment *) (NODE))->oldest_decl)
+ #define FRAGMENT_NEWEST_DECL(NODE) \
+ (((struct c_include_fragment *) (NODE))->newest_decl)
+ #define FRAGMENT_PREVIOUS_FRAGMENT_DECL(NODE) \
+ (((struct c_include_fragment *) (NODE))->previous_fragment_decl)
+ #endif
+
#define wchar_type_node c_global_trees[CTI_WCHAR_TYPE]
#define signed_wchar_type_node c_global_trees[CTI_SIGNED_WCHAR_TYPE]
#define unsigned_wchar_type_node c_global_trees[CTI_UNSIGNED_WCHAR_TYPE]
***************
*** 916,921 ****
--- 967,974 ----
extern tree convert_and_check PARAMS ((tree, tree));
extern void overflow_warning PARAMS ((tree));
extern void unsigned_conversion_warning PARAMS ((tree, tree));
+
+ extern void c_initially_decl_processing PARAMS ((void));
/* Read the rest of the current #-directive line. */
extern char *get_directive_line PARAMS ((void));
Index: cpphash.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cpphash.h,v
retrieving revision 1.175
diff -c -r1.175 cpphash.h
*** cpphash.h 21 Jan 2003 00:12:52 -0000 1.175
--- cpphash.h 8 Feb 2003 20:05:03 -0000
***************
*** 267,272 ****
--- 267,297 ----
unsigned char printed; /* Nonzero if something output at line. */
};
+ struct cpp_fragment
+ {
+ const char *name; /* actual path name of file REDUNDANT */
+ cpp_fragment *next;
+ const unsigned char *start;
+ const unsigned char *end;
+ unsigned was_reused : 1;
+ /*unsigned int valid : 1;*/
+ void *start_marker; /* Returned from front-end - kludge. */
+ unsigned int start_line;
+ unsigned int end_line;
+ const unsigned char *end_line_base;
+ };
+
+ /* Used by compile server to restore previous macro state. */
+ struct cpp_macro_note
+ {
+ cpp_hashnode *node;
+ struct cpp_macro *macro; /* null for an #undef */
+ };
+ extern void _cpp_note_macro PARAMS ((cpp_reader *,
+ cpp_hashnode *, cpp_macro *));
+ extern void _cpp_restore_macros PARAMS ((cpp_reader *,
+ struct cpp_macro_note *, int));
+
/* Represents the contents of a file cpplib has read in. */
struct cpp_buffer
{
***************
*** 323,328 ****
--- 348,355 ----
/* Used for buffer overlays by cpptrad.c. */
const uchar *saved_cur, *saved_rlimit;
+
+ cpp_fragment *saved_current_fragment;
};
/* A cpp_reader encapsulates the "state" of a pre-processor run.
***************
*** 343,348 ****
--- 370,376 ----
struct line_maps line_maps;
const struct line_map *map;
unsigned int line;
+ const char *main_input_filename;
/* The line of the '#' of the current directive. */
unsigned int directive_line;
***************
*** 386,391 ****
--- 414,426 ----
/* Tree of other included files. See cppfiles.c. */
struct splay_tree_s *all_include_files;
+
+ cpp_fragment *current_fragment;
+
+ int do_note_macros;
+ struct cpp_macro_note *macro_notes;
+ int macro_notes_alloc_length;
+ int macro_notes_count;
/* Current maximum length of directory names in the search path
for include files. (Altered as we get more of them.) */
Index: cpplib.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cpplib.h,v
retrieving revision 1.242
diff -c -r1.242 cpplib.h
*** cpplib.h 28 Jan 2003 19:30:00 -0000 1.242
--- cpplib.h 8 Feb 2003 20:05:04 -0000
***************
*** 39,44 ****
--- 39,45 ----
typedef struct cpp_hashnode cpp_hashnode;
typedef struct cpp_macro cpp_macro;
typedef struct cpp_callbacks cpp_callbacks;
+ typedef struct cpp_fragment cpp_fragment;
struct answer;
struct file_name_map_list;
***************
*** 419,424 ****
--- 420,428 ----
void (*file_change) PARAMS ((cpp_reader *, const struct line_map *));
void (*include) PARAMS ((cpp_reader *, unsigned int,
const unsigned char *, const cpp_token *));
+ void* (*push_fragment) PARAMS ((cpp_reader *, cpp_fragment *,
+ const char*, int));
+ void (*pop_fragment) PARAMS ((cpp_reader *, cpp_fragment *));
void (*define) PARAMS ((cpp_reader *, unsigned int, cpp_hashnode *));
void (*undef) PARAMS ((cpp_reader *, unsigned int, cpp_hashnode *));
void (*ident) PARAMS ((cpp_reader *, unsigned int, const cpp_string *));
***************
*** 540,545 ****
--- 544,551 ----
extern int cpp_handle_options PARAMS ((cpp_reader *, int, char **));
extern int cpp_handle_option PARAMS ((cpp_reader *, int, char **));
+ /* FIXME this comment needs fixing, but leave it until later,
+ due to gcc-3.3 and gcc-3.4 type fix causing patch hassles. */
/* This function reads the file, but does not start preprocessing. It
returns the name of the original file; this is the same as the
input file, except for preprocessed input. This will generate at
***************
*** 550,557 ****
pointer. Otherwise you should pass in an initialized hash table
that cpplib will share; this technique is used by the C front
ends. */
! extern const char *cpp_read_main_file PARAMS ((cpp_reader *, const char *,
! struct ht *));
/* Deferred handling of command line options that can generate debug
callbacks, such as -D and -imacros. Call this after
--- 556,563 ----
pointer. Otherwise you should pass in an initialized hash table
that cpplib will share; this technique is used by the C front
ends. */
! extern const char *cpp_read_main_file PARAMS ((cpp_reader *, const char *));
! extern void cpp_init_tables PARAMS ((cpp_reader *, struct ht *));
/* Deferred handling of command line options that can generate debug
callbacks, such as -D and -imacros. Call this after
***************
*** 590,595 ****
--- 596,604 ----
extern const unsigned char *cpp_macro_definition PARAMS ((cpp_reader *,
const cpp_hashnode *));
extern void _cpp_backup_tokens PARAMS ((cpp_reader *, unsigned int));
+ extern void _cpp_start_fragment PARAMS ((cpp_reader *));
+ extern void _cpp_push_fragment PARAMS ((cpp_reader *, cpp_fragment*));
+ extern void _cpp_pop_fragment PARAMS ((cpp_reader *, cpp_fragment*));
/* Evaluate a CPP_CHAR or CPP_WCHAR token. */
extern cppchar_t
Index: c-tree.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-tree.h,v
retrieving revision 1.109
diff -c -r1.109 c-tree.h
*** c-tree.h 16 Sep 2002 18:33:18 -0000 1.109
--- c-tree.h 8 Feb 2003 20:05:04 -0000
***************
*** 244,249 ****
--- 244,250 ----
/* in c-objc-common.c */
extern int c_disregard_inline_limits PARAMS ((tree));
extern int c_cannot_inline_tree_fn PARAMS ((tree *));
+ extern void c_objc_common_initially PARAMS ((void));
extern const char *c_objc_common_init PARAMS ((const char *));
extern int c_missing_noreturn_ok_p PARAMS ((tree));
extern void c_objc_common_finish_file PARAMS ((void));
Index: flags.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/flags.h,v
retrieving revision 1.96
diff -c -r1.96 flags.h
*** flags.h 24 Jan 2003 15:17:24 -0000 1.96
--- flags.h 8 Feb 2003 20:05:04 -0000
***************
*** 664,669 ****
--- 664,671 ----
/* Nonzero means disable transformations observable by signaling NaNs. */
extern int flag_signaling_nans;
+ extern int is_server;
+
/* True if the given mode has a NaN representation and the treatment of
NaN operands is important. Certain optimizations, such as folding
x * 0 into x, are not correct for NaN operands, and are normally
Index: langhooks-def.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/langhooks-def.h,v
retrieving revision 1.40
diff -c -r1.40 langhooks-def.h
*** langhooks-def.h 16 Dec 2002 18:19:40 -0000 1.40
--- langhooks-def.h 8 Feb 2003 20:05:05 -0000
***************
*** 85,90 ****
--- 85,91 ----
#define LANG_HOOKS_NAME "GNU unknown"
#define LANG_HOOKS_IDENTIFIER_SIZE sizeof (struct lang_identifier)
#define LANG_HOOKS_INIT lhd_do_nothing
+ #define LANG_HOOKS_INITIALLY lhd_do_nothing
#define LANG_HOOKS_FINISH lhd_do_nothing
#define LANG_HOOKS_PARSE_FILE lhd_do_nothing_i
#define LANG_HOOKS_CLEAR_BINDING_STACK lhd_clear_binding_stack
***************
*** 228,233 ****
--- 229,235 ----
LANG_HOOKS_INIT_OPTIONS, \
LANG_HOOKS_DECODE_OPTION, \
LANG_HOOKS_POST_OPTIONS, \
+ LANG_HOOKS_INITIALLY, \
LANG_HOOKS_INIT, \
LANG_HOOKS_FINISH, \
LANG_HOOKS_PARSE_FILE, \
Index: langhooks.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/langhooks.h,v
retrieving revision 1.49
diff -c -r1.49 langhooks.h
*** langhooks.h 16 Dec 2002 18:19:41 -0000 1.49
--- langhooks.h 8 Feb 2003 20:05:05 -0000
***************
*** 206,217 ****
immediately and the finish hook is not called. */
bool (*post_options) PARAMS ((void));
! /* Called after post_options, to initialize the front end. The main
! input filename is passed, which may be NULL; the front end should
return the original filename (e.g. foo.i -> foo.c). Return NULL
to indicate a serious error of some sort; in that case no
! compilation is performed, and the finish hook is called
! immediately. */
const char * (*init) PARAMS ((const char *));
/* Called at the end of compilation, as a finalizer. */
--- 206,221 ----
immediately and the finish hook is not called. */
bool (*post_options) PARAMS ((void));
! /* Called after post_options, to initialize the front end.
! This only gets call once, even if we're invoked as a server. */
! void (*initially) PARAMS ((void));
!
! /* Called (after initially) to initially the front-end for a main file.
! If we're invoked as a server, this is called once for each request.
! The input filename is passed, which may be NULL; the front end should
return the original filename (e.g. foo.i -> foo.c). Return NULL
to indicate a serious error of some sort; in that case no
! compilation is performed, and the finish hook is called immediately. */
const char * (*init) PARAMS ((const char *));
/* Called at the end of compilation, as a finalizer. */
Index: tree.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/tree.h,v
retrieving revision 1.379
diff -c -r1.379 tree.h
*** tree.h 31 Jan 2003 14:46:47 -0000 1.379
--- tree.h 8 Feb 2003 20:05:07 -0000
***************
*** 1792,1797 ****
--- 1792,1799 ----
{
struct tree_common common;
location_t locus;
+ /* FIXME make part of locus - or of context! */
+ tree defining_fragment;
unsigned int uid;
tree size;
ENUM_BITFIELD(machine_mode) mode : 8;
Index: c-common.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-common.c,v
retrieving revision 1.399
diff -c -r1.399 c-common.c
*** c-common.c 30 Jan 2003 07:23:57 -0000 1.399
--- c-common.c 8 Feb 2003 20:05:11 -0000
***************
*** 38,43 ****
--- 38,44 ----
#include "tm_p.h"
#include "obstack.h"
#include "cpplib.h"
+ #include "cpphash.h"
#include "target.h"
#include "langhooks.h"
#include "except.h" /* For USING_SJLJ_EXCEPTIONS. */
***************
*** 3185,3190 ****
--- 3186,3195 ----
static void c_init_attributes PARAMS ((void));
+ static void reserve_fragment_binding PARAMS ((int));
+
+ static int common_nodes_and_builtins_done;
+
/* Build tree nodes and builtin functions common to both C and C++ language
frontends. */
***************
*** 3225,3230 ****
--- 3230,3239 ----
tree va_list_ref_type_node;
tree va_list_arg_type_node;
+ setup_globals ();
+ if (common_nodes_and_builtins_done++ > 0)
+ return; /* FIXME - never happens? */
+
/* Define `int' and `char' first so that dbx will output them first. */
record_builtin_type (RID_INT, NULL, integer_type_node);
record_builtin_type (RID_CHAR, "char", char_type_node);
***************
*** 4725,4730 ****
--- 4734,4973 ----
return val;
}
\f
+ /* A stack of include fragments that are used by current_c_fragment
+ or other not-yet popped headers. The ones in current_c_fragment
+ are whose at index from 0 to current_fragment_deps_end (exclusive). */
+ static GTY (()) tree current_fragment_deps_stack;
+ /* The number of elements in current_include_deps_stack that are in use. */
+ static int current_fragment_deps_end;
+
+ struct c_include_fragment *current_c_fragment;
+
+ extern void register_fragment_dependency PARAMS ((struct c_include_fragment*));
+
+ void
+ register_decl_dependency (used)
+ tree used;
+ {
+ struct c_include_fragment *fragment;
+ const char *file = DECL_SOURCE_FILE (used);
+ if (file == NULL)
+ return;
+ if (strcmp (file, "<built-in>") == 0)
+ return;
+
+ fragment = DECL_FRAGMENT (used);
+ if (fragment == NULL)
+ return;
+ register_fragment_dependency (fragment);
+ }
+
+ tree fragment_bindings_stack;
+ int fragment_bindings_end;
+
+ static void
+ reserve_fragment_binding (space_needed)
+ int space_needed;
+ {
+ if (fragment_bindings_stack == NULL_TREE)
+ fragment_bindings_stack = make_tree_vec (50);
+ else if (TREE_VEC_LENGTH (fragment_bindings_stack)
+ < fragment_bindings_end + space_needed)
+ {
+ /* Re-size fragment_bindings_stack. */
+ int i = fragment_bindings_end;
+ tree new_vec = make_tree_vec (2 * i);
+ while (--i >= 0)
+ TREE_VEC_ELT (new_vec, i)
+ = TREE_VEC_ELT (fragment_bindings_stack, i);
+ fragment_bindings_stack = new_vec;
+ }
+ fragment_bindings_end += space_needed;
+ }
+
+ void
+ note_fragment_binding_1 (tree1)
+ tree tree1;
+ {
+ reserve_fragment_binding (1);
+ TREE_VEC_ELT (fragment_bindings_stack, fragment_bindings_end - 1) = tree1;
+ }
+
+ void
+ note_fragment_binding_2 (tree1, tree2)
+ tree tree1, tree2;
+ {
+ reserve_fragment_binding (2);
+ TREE_VEC_ELT (fragment_bindings_stack, fragment_bindings_end - 2) = tree1;
+ TREE_VEC_ELT (fragment_bindings_stack, fragment_bindings_end - 1) = tree2;
+ }
+
+ void
+ note_fragment_binding_3 (tree1, tree2, tree3)
+ tree tree1, tree2, tree3;
+ {
+ reserve_fragment_binding (3);
+ TREE_VEC_ELT (fragment_bindings_stack, fragment_bindings_end - 3) = tree1;
+ TREE_VEC_ELT (fragment_bindings_stack, fragment_bindings_end - 2) = tree2;
+ TREE_VEC_ELT (fragment_bindings_stack, fragment_bindings_end - 1) = tree3;
+ }
+
+ void
+ register_fragment_dependency (used)
+ struct c_include_fragment* used;
+ {
+ if (! used->used_in_current && used != current_c_fragment
+ && current_c_fragment != NULL)
+ {
+ if (current_fragment_deps_stack == NULL_TREE)
+ current_fragment_deps_stack = make_tree_vec (50);
+ else if (TREE_VEC_LENGTH (current_fragment_deps_stack)
+ == current_fragment_deps_end)
+ {
+ /* Re-size current_fragment_deps_stack. */
+ int i = current_fragment_deps_end;
+ tree new_vec = make_tree_vec (2 * i);
+ while (--i >= 0)
+ TREE_VEC_ELT (new_vec, i)
+ = TREE_VEC_ELT (current_fragment_deps_stack, i);
+ current_fragment_deps_stack = new_vec;
+ }
+ TREE_VEC_ELT (current_fragment_deps_stack, current_fragment_deps_end)
+ = (tree) used;
+ current_fragment_deps_end++;
+ used->used_in_current = 1;
+ }
+ }
+
+ int main_timestamp;
+ int c_timestamp;
+ /* Inside an incomple enum, for example. */
+ int currently_nested;
+
+ void*
+ cb_push_fragment (reader, fragment, name, line)
+ cpp_reader* reader ATTRIBUTE_UNUSED;
+ cpp_fragment *fragment;
+ const char* name;
+ int line;
+ {
+ struct c_include_fragment* st = C_FRAGMENT (fragment);
+ void *ret;
+ bool valid = 0;
+ if (st != NULL)
+ {
+ int i;
+ valid = st->valid && st->include_timestamp < main_timestamp && ! currently_nested;
+
+ if (valid)
+ {
+ /* Check dependencies. */
+ tree d = st->uses_fragments;
+ for (i = d == NULL_TREE ? 0 : TREE_VEC_LENGTH (d); --i >= 0; )
+ {
+ struct c_include_fragment *uses
+ = (struct c_include_fragment *) TREE_VEC_ELT (d, i);
+ if (uses->include_timestamp < main_timestamp
+ || uses->read_timestamp == 0
+ || uses->read_timestamp > st->read_timestamp)
+ {
+ valid = 0;
+ break;
+ }
+ }
+ }
+
+ if (! valid)
+ {
+ st->valid = 0;
+ if (! quiet_flag)
+ fprintf (stderr, "(invalidating cached fragment %s:%d)\n", name, line);
+ }
+ else
+ {
+ if (! quiet_flag)
+ fprintf (stderr, "(reusing cached fragment %s:%d)\n", name, line);
+ restore_from_fragment (st);
+ ret = NULL;
+ }
+ }
+ else
+ {
+ st = alloc_include_fragment ();
+ st->name = name;
+ st->valid = 0;
+ }
+ st->include_timestamp = ++c_timestamp;
+ if (! valid)
+ {
+ st->read_timestamp = 0;
+ st->uses_fragments = NULL_TREE;
+ st->bindings = NULL_TREE;
+ /*st->used_in_current = 0;*/
+
+ ret = st;
+ current_fragment_deps_end = 0;
+ fragment_bindings_end = 0;
+ st->read_timestamp = st->include_timestamp;
+ }
+ current_c_fragment = st;
+ /* Note fragment->was_resued is redundant - it's same as
+ st->include_timestamp > st->read_timestamp */
+ fragment->was_reused = ret == NULL;
+ return ret;
+ }
+
+ void
+ cb_pop_fragment (reader, fragment)
+ cpp_reader *reader ATTRIBUTE_UNUSED;
+ cpp_fragment *fragment;
+ {
+ struct c_include_fragment* st = C_FRAGMENT (fragment);
+ if (st != NULL)
+ {
+ int i;
+
+ int uses_fragments_count = current_fragment_deps_end;
+ tree uses_fragments = (uses_fragments_count == 0 ? NULL_TREE
+ : make_tree_vec (uses_fragments_count));
+
+ tree bindings = (fragment_bindings_end == 0 ? NULL_TREE
+ : make_tree_vec (fragment_bindings_end));
+ for (i = 0; i < fragment_bindings_end; i++)
+ {
+ TREE_VEC_ELT (bindings, i)
+ = TREE_VEC_ELT (fragment_bindings_stack, i);
+ /* Clean up for the sake of garbage collection. ??? */
+ TREE_VEC_ELT (fragment_bindings_stack, i) = NULL;
+ }
+ st->bindings = bindings;
+ fragment_bindings_end = 0;
+
+ for (i = 0; i < current_fragment_deps_end; i++)
+ {
+ tree uses = TREE_VEC_ELT (current_fragment_deps_stack, i);
+ TREE_VEC_ELT (uses_fragments, i) = uses;
+ ((struct c_include_fragment*) uses)->used_in_current = 0;
+ /* Clean up for the sake of garbage collection. ??? */
+ TREE_VEC_ELT (current_fragment_deps_stack, i) = NULL;
+ }
+ st->uses_fragments = uses_fragments;
+ current_fragment_deps_end = 0;
+ for (i = 0; i < current_fragment_deps_end; i++)
+ {
+ ((struct c_include_fragment*) TREE_VEC_ELT (current_fragment_deps_stack, i))->used_in_current = 1;
+ }
+
+ current_c_fragment = NULL;
+ st->valid = ! currently_nested;
+ }
+ #if 0
+ fprintf(stderr, "(pop deps start:%d end:%d for %s ret:%s)\n",
+ current_fragment_deps_start, current_fragment_deps_end, st->name,
+ (st ? "non-null":"null"));
+ #endif
+ }
+ \f
/* Define NAME with value TYPE precision. */
static void
builtin_define_type_precision (name, type)
***************
*** 4918,4923 ****
--- 5161,5169 ----
/* -undef turns off target-specific built-ins. */
if (flag_undef)
return;
+
+ if (is_server)
+ pfile->do_note_macros = 1;
if (c_language == clk_cplusplus)
{
Index: c-decl.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-decl.c,v
retrieving revision 1.361
diff -c -r1.361 c-decl.c
*** c-decl.c 24 Jan 2003 15:17:24 -0000 1.361
--- c-decl.c 8 Feb 2003 20:05:16 -0000
***************
*** 43,48 ****
--- 43,49 ----
#include "ggc.h"
#include "tm_p.h"
#include "cpplib.h"
+ #include "cpphash.h"
#include "target.h"
#include "debug.h"
#include "timevar.h"
***************
*** 344,349 ****
--- 345,474 ----
}
}
\f
+ static int c_init_decl_done;
+ static GTY(()) tree builtins_end;
+
+ static void restore_fragment_bindings PARAMS ((struct c_include_fragment *));
+
+ static void
+ restore_fragment_bindings (fragment)
+ struct c_include_fragment *fragment;
+ {
+ int i;
+ tree bindings = fragment->bindings;
+ int len;
+ struct binding_level *b = current_binding_level;
+ if (bindings == NULL_TREE)
+ return;
+ len = TREE_VEC_LENGTH (bindings);
+ for (i = 0; i < len; )
+ {
+ tree x = TREE_VEC_ELT (bindings, i);
+
+ if (TREE_CODE_CLASS (TREE_CODE (x)) == 'd')
+ {
+ tree n = DECL_NAME (x);
+ if (n != NULL_TREE)
+ IDENTIFIER_GLOBAL_VALUE (n) = x;
+ TREE_CHAIN (x) = b->names;
+ b->names = x;
+ i++;
+ }
+ else if (TREE_CODE (x) == TREE_LIST) /* pushtag */
+ {
+ tree type = TREE_VALUE (x);
+ TREE_CHAIN (x) = b->tags;
+ TYPE_SIZE (type) = NULL_TREE;
+ TYPE_FIELDS (type) = NULL_TREE;
+ b->tags = x;
+ i++;
+ }
+ else if (TREE_CODE_CLASS (TREE_CODE (x)) == 't')
+ { /* finish_struct */
+ tree type = x;
+ TYPE_FIELDS (type) = TREE_VEC_ELT (bindings, i + 1);
+ TYPE_SIZE (type) = TREE_VEC_ELT (bindings, i + 2);
+ i += 3;
+ }
+ else
+ abort ();
+ }
+ }
+
+ void
+ restore_from_fragment (fragment)
+ struct c_include_fragment *fragment;
+ {
+ restore_fragment_bindings (fragment);
+ }
+
+ extern int clear_identifiers PARAMS ((cpp_reader*, cpp_hashnode*, void*));
+ int
+ clear_identifiers (pfile, node, v)
+ cpp_reader *pfile ATTRIBUTE_UNUSED;
+ cpp_hashnode *node;
+ void *v ATTRIBUTE_UNUSED;
+ {
+ tree tnode = HT_IDENT_TO_GCC_IDENT (node);
+ struct lang_identifier * lang_id = (struct lang_identifier *) tnode;
+ node->flags &= ~ NODE_POISONED; /* ??? also clear NODE_DIAGNOSTIC? */
+ if (TREE_CODE (tnode) != IDENTIFIER_NODE)
+ abort();
+ if (lang_id != NULL)
+ lang_id->global_value = NULL_TREE;
+ if (node->type == NT_MACRO && !(node->flags & NODE_BUILTIN))
+ {
+ cpp_macro *macro = node->value.macro;
+ macro->used = 0;
+
+ _cpp_free_definition (node);
+ }
+
+ return 1;
+ }
+
+ void setup_globals ()
+ {
+ current_function_decl = NULL;
+ named_labels = NULL;
+ current_binding_level = NULL_BINDING_LEVEL;
+
+ /* Make the binding_level structure for global names. */
+ pushlevel (0);
+ global_binding_level = current_binding_level;
+ }
+
+ void
+ c_init_decl_processing ()
+ {
+
+ main_timestamp = ++c_timestamp;
+ if (c_init_decl_done++ == 0)
+ {
+ setup_globals ();
+ parse_in->do_note_macros = 0;
+ }
+ else
+ {
+ tree d;
+ struct c_include_fragment *fragment;
+
+ cpp_forall_identifiers (parse_in, clear_identifiers, NULL);
+ _cpp_restore_macros (parse_in,
+ parse_in->macro_notes, parse_in->macro_notes_count);
+
+ global_binding_level->names = builtins_end;
+ global_binding_level->tags = NULL_TREE;
+
+ for (d = builtins_end; d != NULL_TREE; d = TREE_CHAIN (d))
+ {
+ tree n = DECL_NAME (d);
+ if (n != NULL_TREE)
+ IDENTIFIER_GLOBAL_VALUE (n) = d;
+ }
+ }
+ }
+ \f
/* Reuse or create a struct for this binding level. */
static struct binding_level *
***************
*** 831,836 ****
--- 956,966 ----
b->tags = tree_cons (name, type, b->tags);
+ if (b == global_binding_level && is_server)
+ {
+ note_fragment_binding_1 (b->tags);
+ }
+
/* Create a fake NULL-named TYPE_DECL node whose TREE_TYPE will be the
tagged type we just added to the current binding level. This fake
NULL-named TYPE_DECL node helps dwarfout.c to know when it needs
***************
*** 1839,1844 ****
--- 1969,1979 ----
TREE_PUBLIC (name) = 1;
IDENTIFIER_GLOBAL_VALUE (name) = x;
+ if (is_server)
+ {
+ SET_DECL_FRAGMENT (x, current_c_fragment);
+ note_fragment_binding_1 (x);
+ }
/* We no longer care about any previous block level declarations. */
IDENTIFIER_LIMBO_VALUE (name) = 0;
***************
*** 2432,2437 ****
--- 2567,2574 ----
val = IDENTIFIER_LOCAL_VALUE (name);
else
val = IDENTIFIER_GLOBAL_VALUE (name);
+ if (val != NULL_TREE && is_server)
+ register_decl_dependency (val);
return val;
}
***************
*** 2462,2468 ****
Make definitions for built-in primitive functions. */
void
! c_init_decl_processing ()
{
tree endlink;
tree ptr_ftype_void, ptr_ftype_ptr;
--- 2599,2605 ----
Make definitions for built-in primitive functions. */
void
! c_initially_decl_processing ()
{
tree endlink;
tree ptr_ftype_void, ptr_ftype_ptr;
***************
*** 2470,2484 ****
/* Adds some ggc roots, and reserved words for c-parse.in. */
c_parse_init ();
- current_function_decl = NULL;
- named_labels = NULL;
- current_binding_level = NULL_BINDING_LEVEL;
free_binding_level = NULL_BINDING_LEVEL;
- /* Make the binding_level structure for global names. */
- pushlevel (0);
- global_binding_level = current_binding_level;
-
build_common_tree_nodes (flag_signed_char);
c_common_nodes_and_builtins ();
--- 2607,2614 ----
***************
*** 2509,2514 ****
--- 2639,2645 ----
make_fname_decl = c_make_fname_decl;
start_fname_decls ();
+ builtins_end = global_binding_level->names;
}
/* Create the VAR_DECL for __FUNCTION__ etc. ID is the name to give the
***************
*** 5302,5307 ****
--- 5433,5443 ----
layout_type (t);
+ if (is_server)
+ {
+ note_fragment_binding_3 (t, fieldlist, TYPE_SIZE (t));
+ }
+
/* Delete all zero-width bit-fields from the fieldlist */
{
tree *fieldlistp = &fieldlist;
***************
*** 5450,5455 ****
--- 5586,5593 ----
if (flag_short_enums)
TYPE_PACKED (enumtype) = 1;
+ currently_nested++;
+
return enumtype;
}
***************
*** 5470,5475 ****
--- 5608,5615 ----
int precision, unsign;
int toplevel = (global_binding_level == current_binding_level);
+ currently_nested--;
+
if (in_parm_level_p ())
warning ("enum defined inside parms");
***************
*** 5562,5567 ****
--- 5702,5712 ----
}
TYPE_VALUES (enumtype) = values;
+ }
+
+ if (is_server)
+ {
+ note_fragment_binding_3 (enumtype, values, TYPE_SIZE (enumtype));
}
/* Fix up all variant types of this enum type. */
Index: c-lang.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-lang.c,v
retrieving revision 1.101
diff -c -r1.101 c-lang.c
*** c-lang.c 16 Dec 2002 18:19:02 -0000 1.101
--- c-lang.c 8 Feb 2003 20:05:16 -0000
***************
*** 37,42 ****
--- 37,44 ----
#undef LANG_HOOKS_NAME
#define LANG_HOOKS_NAME "GNU C"
+ #undef LANG_HOOKS_INITIALLY
+ #define LANG_HOOKS_INITIALLY c_objc_common_initially
#undef LANG_HOOKS_INIT
#define LANG_HOOKS_INIT c_objc_common_init
#undef LANG_HOOKS_FINISH
Index: c-lex.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-lex.c,v
retrieving revision 1.193
diff -c -r1.193 c-lex.c
*** c-lex.c 13 Jan 2003 00:14:58 -0000 1.193
--- c-lex.c 8 Feb 2003 20:05:17 -0000
***************
*** 139,146 ****
/* Start it at 0. */
lineno = 0;
! return cpp_read_main_file (parse_in, filename, ident_hash);
}
/* A thin wrapper around the real parser that initializes the
integrated preprocessor after debug output has been initialized.
--- 139,149 ----
/* Start it at 0. */
lineno = 0;
! return cpp_read_main_file (parse_in, filename);
}
+
+ /* kludge to reduce patch conflicts with Apple PFE */
+ #define cpp_finish_options(parse_in) ((void)0)
/* A thin wrapper around the real parser that initializes the
integrated preprocessor after debug output has been initialized.
Index: c-objc-common.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-objc-common.c,v
retrieving revision 1.20
diff -c -r1.20 c-objc-common.c
*** c-objc-common.c 10 Jan 2003 02:21:58 -0000 1.20
--- c-objc-common.c 8 Feb 2003 20:05:17 -0000
***************
*** 28,33 ****
--- 28,34 ----
#include "integrate.h"
#include "expr.h"
#include "c-tree.h"
+ #include "c-pragma.h"
#include "function.h"
#include "flags.h"
#include "toplev.h"
***************
*** 237,242 ****
--- 238,279 ----
return false;
return true;
+ }
+
+ /* APPLE local hack - remove for FSF */
+ void print_INCLUDE_FRAGMENT () { abort(); }
+
+ struct c_include_fragment *
+ alloc_include_fragment ()
+ {
+ int length = sizeof (struct c_include_fragment);
+ tree t = ggc_alloc_tree (length);
+ memset ((PTR) t, 0, length);
+ TREE_SET_CODE (t, INCLUDE_FRAGMENT);
+ return (struct c_include_fragment *) t;
+ }
+
+ /* Initialization common to C and Objective-C front ends. */
+ void
+ c_objc_common_initially ()
+ {
+ cpp_init_tables (parse_in, ident_hash);
+
+ /*
+ if (! CPP_OPTION (pfile, preprocessed))
+ {
+ struct pending_option *p;
+
+ _cpp_do_file_change (pfile, LC_RENAME, _("<built-in>"), 1, 0);
+ }
+ */
+ init_pragma ();
+
+ c_initially_decl_processing ();
+
+ c_common_initially ();
+
+ cpp_finish_options (parse_in);
}
/* Initialization common to C and Objective-C front ends. */
Index: c-opts.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-opts.c,v
retrieving revision 1.28
diff -c -r1.28 c-opts.c
*** c-opts.c 28 Jan 2003 19:29:57 -0000 1.28
--- c-opts.c 8 Feb 2003 20:05:18 -0000
***************
*** 1430,1439 ****
cpp_preprocess_file (parse_in, in_fname, out_stream);
}
! /* Front end initialization common to C, ObjC and C++. */
! const char *
! c_common_init (filename)
! const char *filename;
{
/* Set up preprocessor arithmetic. Must be done after call to
c_common_nodes_and_builtins for type nodes to be good. */
--- 1430,1437 ----
cpp_preprocess_file (parse_in, in_fname, out_stream);
}
! void
! c_common_initially ()
{
/* Set up preprocessor arithmetic. Must be done after call to
c_common_nodes_and_builtins for type nodes to be good. */
***************
*** 1446,1452 ****
--- 1444,1461 ----
/* Register preprocessor built-ins before calls to
cpp_main_file. */
cpp_get_callbacks (parse_in)->register_builtins = cb_register_builtins;
+ if (is_server)
+ {
+ cpp_get_callbacks (parse_in)->push_fragment = cb_push_fragment;
+ cpp_get_callbacks (parse_in)->pop_fragment = cb_pop_fragment;
+ }
+ }
+ /* Front end initialization common to C, ObjC and C++. */
+ const char *
+ c_common_init (filename)
+ const char *filename;
+ {
/* NULL is passed up to toplev.c and we exit quickly. */
if (flag_preprocess_only)
{
***************
*** 1457,1465 ****
/* Do this before initializing pragmas, as then cpplib's hash table
has been set up. NOTE: we are using our own file name here, not
the one supplied. */
filename = init_c_lex (in_fname);
-
- init_pragma ();
return filename;
}
--- 1466,1473 ----
/* Do this before initializing pragmas, as then cpplib's hash table
has been set up. NOTE: we are using our own file name here, not
the one supplied. */
+ in_fname = filename;
filename = init_c_lex (in_fname);
return filename;
}
Index: cppfiles.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cppfiles.c,v
retrieving revision 1.160
diff -c -r1.160 cppfiles.c
*** cppfiles.c 10 Jan 2003 02:21:58 -0000 1.160
--- cppfiles.c 8 Feb 2003 20:05:22 -0000
***************
*** 94,99 ****
--- 94,100 ----
/* location in search path where file was
found, for #include_next and sysp. */
const unsigned char *buffer; /* pointer to cached file contents */
+ cpp_fragment *fragments; /* head of fragment chain */
struct stat st; /* copy of stat(2) data for file */
int fd; /* fd open on file (short term storage only) */
int err_no; /* errno obtained if opening a file failed */
***************
*** 121,127 ****
/* The cmacro works like this: If it's NULL, the file is to be
included again. If it's NEVER_REREAD, the file is never to be
included again. Otherwise it is a macro hashnode, and the file is
! to be included again if the macro is defined. */
#define NEVER_REREAD ((const cpp_hashnode *) -1)
#define DO_NOT_REREAD(inc) \
((inc)->cmacro && ((inc)->cmacro == NEVER_REREAD \
--- 122,128 ----
/* The cmacro works like this: If it's NULL, the file is to be
included again. If it's NEVER_REREAD, the file is never to be
included again. Otherwise it is a macro hashnode, and the file is
! to be included again if the macro is undefined. */
#define NEVER_REREAD ((const cpp_hashnode *) -1)
#define DO_NOT_REREAD(inc) \
((inc)->cmacro && ((inc)->cmacro == NEVER_REREAD \
***************
*** 148,153 ****
--- 149,155 ----
static int read_include_file PARAMS ((cpp_reader *, struct include_file *));
static bool stack_include_file PARAMS ((cpp_reader *, struct include_file *));
static void purge_cache PARAMS ((struct include_file *));
+ static void purge_fragments PARAMS ((struct include_file *));
static void destroy_node PARAMS ((splay_tree_value));
static int report_missing_guard PARAMS ((splay_tree_node, void *));
static splay_tree_node find_or_create_entry PARAMS ((cpp_reader *,
***************
*** 467,472 ****
--- 469,479 ----
/* from_stage3 */ CPP_OPTION (pfile, preprocessed), 0);
fp->inc = inc;
fp->inc->refcnt++;
+ pfile->current_fragment = inc->fragments;
+ #if 0
+ if (inc->fragments)
+ _cpp_push_fragment (pfile, inc->fragments);
+ #endif
/* Initialize controlling macro state. */
pfile->mi_valid = true;
***************
*** 481,486 ****
--- 488,531 ----
return true;
}
+ void
+ _cpp_push_fragment (pfile, fragment)
+ cpp_reader *pfile;
+ cpp_fragment *fragment;
+ {
+ if (pfile->cb.push_fragment != NULL)
+ {
+ void *start
+ = pfile->cb.push_fragment (pfile, fragment,
+ fragment->name,
+ SOURCE_LINE (lookup_line (&pfile->line_maps, pfile->line), pfile->line));
+ if (start == NULL)
+ {
+ pfile->buffer->cur = fragment->end;
+ pfile->buffer->line_base = fragment->end_line_base;
+ pfile->line += fragment->end_line - fragment->start_line;
+ return;
+ }
+ fragment->start_marker = start;
+ fragment->start_line = pfile->line;
+ }
+ }
+
+ void
+ _cpp_pop_fragment (pfile, fragment)
+ cpp_reader *pfile;
+ cpp_fragment *fragment;
+ {
+ if (pfile->cb.pop_fragment != NULL
+ && ! pfile->state.skipping
+ && ! fragment->was_reused)
+ {
+ pfile->cb.pop_fragment (pfile, fragment);
+ fragment->end_line = pfile->line;
+ fragment->end_line_base = pfile->buffer->line_base;
+ }
+ }
+
/* Read the file referenced by INC into the file cache.
If fd points to a plain file, we might be able to mmap it; we can
***************
*** 500,505 ****
--- 545,551 ----
cpp_reader *pfile;
struct include_file *inc;
{
+ cpp_fragment *fragment;
ssize_t size, offset, count;
uchar *buf;
#if MMAP_THRESHOLD
***************
*** 601,606 ****
--- 647,668 ----
}
inc->buffer = buf;
+
+ if (pfile->cb.pop_fragment != NULL)
+ {
+ size = strlen (inc->name);
+ if (size > 2 && inc->name[size-1] != 'c') /* FIXME */
+ {
+ purge_fragments (inc);
+ fragment = xcnew (cpp_fragment);
+ fragment->name = inc->name;
+ fragment->next = NULL;
+ fragment->start = buf;
+ fragment->end = 0;
+ inc->fragments = fragment;
+ }
+ }
+
return 0;
perror_fail:
***************
*** 631,636 ****
--- 693,713 ----
free ((PTR) inc->buffer);
inc->buffer = NULL;
}
+ purge_fragments (inc);
+ }
+
+ static void
+ purge_fragments (inc)
+ struct include_file *inc;
+ {
+ cpp_fragment *fragment = inc->fragments;;
+ while (fragment != NULL)
+ {
+ cpp_fragment *next = fragment->next;
+ free ((PTR) fragment);
+ fragment = next;
+ }
+ inc->fragments = NULL;
}
/* Return 1 if the file named by FNAME has been included before in
***************
*** 895,902 ****
--- 972,981 ----
pfile->mi_valid = false;
inc->refcnt--;
+ #if 0
if (inc->refcnt == 0 && DO_NOT_REREAD (inc))
purge_cache (inc);
+ #endif
}
/* Returns the first place in the include chain to start searching for
Index: cppinit.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cppinit.c,v
retrieving revision 1.266
diff -c -r1.266 cppinit.c
*** cppinit.c 28 Jan 2003 19:29:57 -0000 1.266
--- cppinit.c 8 Feb 2003 20:05:23 -0000
***************
*** 911,924 ****
deps_add_target (pfile->deps, target, quote);
}
! /* This is called after options have been parsed, and partially
! processed. Setup for processing input from the file named FNAME,
! or stdin if it is the empty string. Return the original filename
! on success (e.g. foo.i->foo.c), or NULL on failure. */
! const char *
! cpp_read_main_file (pfile, fname, table)
cpp_reader *pfile;
- const char *fname;
hash_table *table;
{
static const char *const lang_env_vars[] =
--- 911,919 ----
deps_add_target (pfile->deps, target, quote);
}
! void
! cpp_init_tables (pfile, table)
cpp_reader *pfile;
hash_table *table;
{
static const char *const lang_env_vars[] =
***************
*** 963,969 ****
--- 958,975 ----
}
fprintf (stderr, _("End of search list.\n"));
}
+ }
+ /* This is called after options have been parsed, and partially
+ processed. Setup for processing input from the file named FNAME,
+ or stdin if it is the empty string. Return the original filename
+ on success (e.g. foo.i->foo.c), or NULL on failure. */
+ const char *
+ cpp_read_main_file (pfile, fname)
+ cpp_reader *pfile;
+ const char *fname;
+ {
+ pfile->main_input_filename = fname;
if (CPP_OPTION (pfile, deps.style) != DEPS_NONE)
{
if (!pfile->deps)
***************
*** 1078,1084 ****
{
/* All done; restore the line map from <command line>. */
_cpp_do_file_change (pfile, LC_RENAME,
! pfile->line_maps.maps[0].to_file, 1, 0);
/* Don't come back here again. */
pfile->next_include_file = NULL;
}
--- 1084,1091 ----
{
/* All done; restore the line map from <command line>. */
_cpp_do_file_change (pfile, LC_RENAME,
! pfile->main_input_filename, 1, 0);
! /* pfile->line_maps.maps[0].to_file, 1, 0);*/
/* Don't come back here again. */
pfile->next_include_file = NULL;
}
Index: cpplib.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cpplib.c,v
retrieving revision 1.329
diff -c -r1.329 cpplib.c
*** cpplib.c 28 Jan 2003 19:30:00 -0000 1.329
--- cpplib.c 8 Feb 2003 20:05:25 -0000
***************
*** 238,249 ****
--- 238,261 ----
start_directive (pfile)
cpp_reader *pfile;
{
+ cpp_fragment *fragment;
+ const unsigned char *cur;
+
/* Setup in-directive state. */
pfile->state.in_directive = 1;
pfile->state.save_comments = 0;
/* Some handlers need the position of the # for diagnostics. */
pfile->directive_line = pfile->line;
+
+ cur = pfile->buffer->cur - 1;
+ if ((fragment = pfile->current_fragment) != NULL
+ && (fragment->end == NULL
+ || fragment->end == cur))
+ {
+ fragment->end = cur;
+ _cpp_pop_fragment (pfile, fragment);
+ }
}
/* Called when leaving a directive, _Pragma or command-line directive. */
***************
*** 277,282 ****
--- 289,335 ----
pfile->state.in_expression = 0;
pfile->state.angled_headers = 0;
pfile->directive = 0;
+ _cpp_start_fragment (pfile);
+ }
+
+ void
+ _cpp_start_fragment (pfile)
+ cpp_reader *pfile;
+ {
+ cpp_fragment *fragment;
+ if ((fragment = pfile->current_fragment) != NULL
+ && ! pfile->state.skipping)
+ {
+ const unsigned char *cur = pfile->buffer->cur;
+ for (;;)
+ {
+ cpp_fragment *next_fragment = fragment->next;
+ if (next_fragment == NULL || next_fragment->start > cur)
+ break;
+ fragment = next_fragment;
+ }
+ if (fragment->start < cur)
+ {
+ cpp_fragment *next_fragment;
+ if (1 /*fragment->start != fragment->end*/
+ /* && contains more than whitespace ... FIXME */)
+ {
+ #if 0
+ if (fragment->start != fragment->end)
+ fragment->start_marker = NULL; /* FIXME */
+ #endif
+ next_fragment = xcnew (cpp_fragment);
+ next_fragment->name = fragment->name;
+ next_fragment->next = fragment->next;
+ fragment->next = next_fragment;
+ fragment = next_fragment;
+ }
+ fragment->start = cur;
+ fragment->end = 0;
+ }
+ pfile->current_fragment = fragment;
+ _cpp_push_fragment (pfile, fragment);
+ }
}
/* Prepare to handle the directive in pfile->directive. */
***************
*** 520,525 ****
--- 573,634 ----
return NULL;
}
+ void
+ _cpp_restore_macros (pfile, notes, count)
+ cpp_reader *pfile ATTRIBUTE_UNUSED;
+ struct cpp_macro_note *notes;
+ int count;
+ {
+ for (; --count >= 0; notes++)
+ {
+ cpp_hashnode *node = notes->node;
+ struct cpp_macro *macro = notes->macro;
+ if (macro != NULL)
+ {
+ /* #define */
+ node->type = NT_MACRO;
+ node->value.macro = macro;
+ }
+ else
+ {
+ /* #undef */
+ _cpp_free_definition (node);
+ }
+ }
+ }
+
+ void
+ _cpp_note_macro (pfile, node, macro)
+ cpp_reader *pfile;
+ cpp_hashnode *node;
+ struct cpp_macro *macro;
+ {
+ int count, alloc;
+ if (! pfile->do_note_macros)
+ return;
+
+ count = pfile->macro_notes_count;
+ if (pfile->macro_notes == NULL)
+ {
+ alloc = 64;
+ pfile->macro_notes = xmalloc (alloc * sizeof (struct cpp_macro_note));
+ pfile->macro_notes_alloc_length = alloc;
+ }
+ else if (count >= (alloc = pfile->macro_notes_alloc_length))
+ {
+ struct cpp_macro_note *new_notes
+ = xmalloc (2 * alloc * sizeof (struct cpp_macro_note));
+ memcpy (new_notes, pfile->macro_notes,
+ count * sizeof (struct cpp_macro_note));
+ free (pfile->macro_notes);
+ pfile->macro_notes = new_notes;
+ pfile->macro_notes_alloc_length = 2 * alloc;
+ }
+ pfile->macro_notes[count].node = node;
+ pfile->macro_notes[count].macro = macro;
+ pfile->macro_notes_count = count + 1;
+ }
+
/* Process a #define directive. Most work is done in cppmacro.c. */
static void
do_define (pfile)
***************
*** 534,542 ****
pfile->state.save_comments =
! CPP_OPTION (pfile, discard_comments_in_macro_exp);
if (_cpp_create_definition (pfile, node))
! if (pfile->cb.define)
! (*pfile->cb.define) (pfile, pfile->directive_line, node);
}
}
--- 643,655 ----
pfile->state.save_comments =
! CPP_OPTION (pfile, discard_comments_in_macro_exp);
+
if (_cpp_create_definition (pfile, node))
! {
! _cpp_note_macro (pfile, node, node->value.macro);
! if (pfile->cb.define)
! (*pfile->cb.define) (pfile, pfile->directive_line, node);
! }
}
}
***************
*** 554,559 ****
--- 667,674 ----
if (pfile->cb.undef)
(*pfile->cb.undef) (pfile, pfile->directive_line, node);
+ _cpp_note_macro (pfile, node, NULL);
+
if (node->flags & NODE_WARN)
cpp_error (pfile, DL_WARNING, "undefining \"%s\"", NODE_NAME (node));
***************
*** 2030,2036 ****
--- 2145,2153 ----
new->return_at_eof = return_at_eof;
new->saved_flags = BOL;
+ new->saved_current_fragment = pfile->current_fragment;
pfile->buffer = new;
+ pfile->current_fragment = NULL;
return new;
}
***************
*** 2054,2060 ****
--- 2171,2184 ----
/* In case of a missing #endif. */
pfile->state.skipping = 0;
+ if (pfile->current_fragment != 0)
+ {
+ cpp_fragment *fragment = pfile->current_fragment;
+ fragment->end = buffer->cur;
+ _cpp_pop_fragment (pfile, fragment);
+ }
/* _cpp_do_file_change expects pfile->buffer to be the new one. */
+ pfile->current_fragment = buffer->saved_current_fragment;
pfile->buffer = buffer->prev;
/* Free the buffer object now; we may want to push a new buffer
***************
*** 2074,2079 ****
--- 2198,2204 ----
files left to push. */
if (!pfile->buffer->prev)
_cpp_maybe_push_include_file (pfile);
+ _cpp_start_fragment (pfile);
}
}
}
Index: cppmain.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cppmain.c,v
retrieving revision 1.106
diff -c -r1.106 cppmain.c
*** cppmain.c 16 Dec 2002 18:19:18 -0000 1.106
--- cppmain.c 8 Feb 2003 20:05:25 -0000
***************
*** 72,81 ****
setup_callbacks (pfile);
! if (cpp_read_main_file (pfile, in_fname, NULL))
{
cpp_options *options = &pfile->opts;
! cpp_finish_options (pfile);
/* A successful cpp_read_main_file guarantees that we can call
cpp_scan_nooutput or cpp_get_token next. */
--- 72,81 ----
setup_callbacks (pfile);
! if (cpp_read_main_file (pfile, in_fname))
{
cpp_options *options = &pfile->opts;
! /*cpp_finish_options (pfile);*/
/* A successful cpp_read_main_file guarantees that we can call
cpp_scan_nooutput or cpp_get_token next. */
Index: fix-header.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/fix-header.c,v
retrieving revision 1.85
diff -c -r1.85 fix-header.c
*** fix-header.c 16 Dec 2002 18:19:26 -0000 1.85
--- fix-header.c 8 Feb 2003 20:05:26 -0000
***************
*** 637,643 ****
if (cpp_errors (scan_in))
exit (FATAL_EXIT_CODE);
! if (! cpp_read_main_file (scan_in, in_fname, NULL))
exit (FATAL_EXIT_CODE);
cpp_finish_options (scan_in);
--- 637,644 ----
if (cpp_errors (scan_in))
exit (FATAL_EXIT_CODE);
! cpp_init_tables (scan_in, NULL);
! if (! cpp_read_main_file (scan_in, in_fname))
exit (FATAL_EXIT_CODE);
cpp_finish_options (scan_in);
Index: ggc-page.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/ggc-page.c,v
retrieving revision 1.61
diff -c -r1.61 ggc-page.c
*** ggc-page.c 30 Jan 2003 18:14:06 -0000 1.61
--- ggc-page.c 8 Feb 2003 20:05:29 -0000
***************
*** 1523,1528 ****
--- 1523,1529 ----
void
ggc_collect ()
{
+ #if 0
/* Avoid frequent unnecessary work by skipping collection if the
total allocations haven't expanded much since the last
collection. */
***************
*** 1564,1569 ****
--- 1565,1571 ----
if (!quiet_flag)
fprintf (stderr, "%luk}", (unsigned long) G.allocated / 1024);
+ #endif
}
/* Print allocation statistics. */
Index: regclass.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/regclass.c,v
retrieving revision 1.168
diff -c -r1.168 regclass.c
*** regclass.c 31 Jan 2003 23:34:13 -0000 1.168
--- regclass.c 8 Feb 2003 20:05:31 -0000
***************
*** 49,55 ****
#endif
static void init_reg_sets_1 PARAMS ((void));
- static void init_reg_modes PARAMS ((void));
static void init_reg_autoinc PARAMS ((void));
/* If we have auto-increment or auto-decrement and we can have secondary
--- 49,54 ----
***************
*** 549,555 ****
These values are used to record death information for individual registers
(as opposed to a multi-register mode). */
! static void
init_reg_modes ()
{
int i;
--- 548,554 ----
These values are used to record death information for individual registers
(as opposed to a multi-register mode). */
! void
init_reg_modes ()
{
int i;
***************
*** 577,584 ****
/* This finishes what was started by init_reg_sets, but couldn't be done
until after register usage was specified. */
init_reg_sets_1 ();
-
- init_reg_modes ();
init_reg_autoinc ();
}
--- 576,581 ----
Index: toplev.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/toplev.c,v
retrieving revision 1.703
diff -c -r1.703 toplev.c
*** toplev.c 26 Jan 2003 14:40:15 -0000 1.703
--- toplev.c 8 Feb 2003 20:05:34 -0000
***************
*** 142,147 ****
--- 142,150 ----
/* Copy of arguments to toplev_main. */
int save_argc;
char **save_argv;
+
+ int is_server;
+ static bool no_backend;
\f
/* Name of current original source file (what was input to cpp).
This comes from each #-command in the actual input. */
***************
*** 982,987 ****
--- 985,991 ----
static const lang_independent_options f_options[] =
{
+ {"server", &is_server, 1, N_("run compiler in server mode") },
{"eliminate-dwarf2-dups", &flag_eliminate_dwarf2_dups, 1,
N_("Perform DWARF2 duplicate elimination") },
{"float-store", &flag_float_store, 1,
***************
*** 5214,5230 ****
static void
backend_init ()
{
- /* init_emit_once uses reg_raw_mode and therefore must be called
- after init_regs which initialized reg_raw_mode. */
init_regs ();
- init_emit_once (debug_info_level == DINFO_LEVEL_NORMAL
- || debug_info_level == DINFO_LEVEL_VERBOSE
- #ifdef VMS_DEBUGGING_INFO
- /* Enable line number info for traceback */
- || debug_info_level > DINFO_LEVEL_NONE
- #endif
- || flag_test_coverage
- || warn_notreached);
init_fake_stack_mems ();
init_alias_once ();
init_loop ();
--- 5218,5224 ----
***************
*** 5352,5359 ****
}
\f
/* Initialize the compiler, and compile the input file. */
! static void
! do_compile ()
{
/* All command line options have been parsed; allow the front end to
perform consistency checks, etc. */
--- 5346,5353 ----
}
\f
/* Initialize the compiler, and compile the input file. */
! static bool
! init_compile ()
{
/* All command line options have been parsed; allow the front end to
perform consistency checks, etc. */
***************
*** 5362,5370 ****
/* The bulk of command line switch processing. */
process_options ();
! /* If an error has already occurred, give up. */
! if (errorcount)
! return;
if (aux_base_name)
/*NOP*/;
--- 5356,5380 ----
/* The bulk of command line switch processing. */
process_options ();
! /* init_emit_once uses reg_raw_mode and therefore must be called
! after init_reg_modes which initialized reg_raw_mode. */
! init_reg_modes ();
! init_emit_once (debug_info_level == DINFO_LEVEL_NORMAL
! || debug_info_level == DINFO_LEVEL_VERBOSE
! #ifdef VMS_DEBUGGING_INFO
! /* Enable line number info for traceback */
! || debug_info_level > DINFO_LEVEL_NONE
! #endif
! || flag_test_coverage
! || warn_notreached);
!
! (*lang_hooks.initially) ();
! return no_backend;
! }
!
! static void
! do_compile ()
! {
if (aux_base_name)
/*NOP*/;
***************
*** 5398,5403 ****
--- 5408,5446 ----
timevar_print (stderr);
}
\f
+ extern int
+ server_get_filename (void);
+ int
+ server_get_filename ()
+ {
+ char buffer[1024];
+ int len;
+
+ if (fgets (buffer, 1024, stdin) == NULL)
+ return 0;
+ len = strlen (buffer);
+ if (len == 0)
+ return 0;
+ if (buffer[len-1] == '\n')
+ buffer[--len] = '\0';
+ filename = xmalloc (len+1);
+ strcpy (filename, buffer);
+
+ if (fgets (buffer, 1024, stdin) == NULL)
+ return 0;
+ len = strlen (buffer);
+ if (len == 0)
+ return 0;
+ if (buffer[len-1] == '\n')
+ buffer[--len] = '\0';
+ asm_file_name = xmalloc (len+1);
+ strcpy (asm_file_name, buffer);
+
+ if (! quiet_flag)
+ fprintf (stderr, "serve '%s' -o '%s'\n", filename, asm_file_name);
+ return 1;
+ }
+
/* Entry point of cc1, cc1plus, jc1, f771, etc.
Decode command args, then call compile_file.
Exit code is FATAL_EXIT_CODE if can't open files or if there were
***************
*** 5419,5425 ****
/* Exit early if we can (e.g. -help). */
if (!exit_after_options)
! do_compile ();
if (errorcount || sorrycount)
return (FATAL_EXIT_CODE);
--- 5462,5482 ----
/* Exit early if we can (e.g. -help). */
if (!exit_after_options)
! {
! no_backend = init_compile ();
!
! /* If an error has already occurred, give up. */
! if (! errorcount)
! {
! if (is_server)
! {
! while (server_get_filename ())
! do_compile ();
! }
! else
! do_compile ();
! }
! }
if (errorcount || sorrycount)
return (FATAL_EXIT_CODE);
Index: c-common.def
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-common.def,v
retrieving revision 1.13
diff -c -r1.13 c-common.def
*** c-common.def 29 Sep 2002 13:16:42 -0000 1.13
--- c-common.def 8 Feb 2003 20:05:35 -0000
***************
*** 28,33 ****
--- 28,35 ----
/* A node to remember a source position. */
DEFTREECODE (SRCLOC, "srcloc", 'x', 2)
+ DEFTREECODE (INCLUDE_FRAGMENT, "include_fragment", 'x', 2)
+
DEFTREECODE (SIZEOF_EXPR, "sizeof_expr", '1', 1)
DEFTREECODE (ARROW_EXPR, "arrow_expr", 'e', 1)
DEFTREECODE (ALIGNOF_EXPR, "alignof_expr", '1', 1)
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
` (2 preceding siblings ...)
2003-02-27 10:14 ` Per Bothner
@ 2003-02-27 11:28 ` Joseph S. Myers
2003-02-27 19:52 ` Per Bothner
2003-02-27 12:23 ` Jan Hubicka
` (3 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Joseph S. Myers @ 2003-02-27 11:28 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
On Wed, 26 Feb 2003, Per Bothner wrote:
> As I've hinted, I have been working for Apple on a "compile server".
> The idea is that the compiler when in server mode will re-use trees
> from header files. I've attached a semi-detailed design documnent.
> This is in texinfo format (so it can later be merged into gccint.text).
> To generate html do: makeinfo --html gcc-server.texi
The dependency system seems very complicated (necessarily) and potentially
fragile. Note that any place where an identifier is lexed as not
currently declared as a typedef, this must be treated as having a negative
dependency, and any declaration of an identifier must probably depend on
either any previous declaration in scope or on the absence of a previous
declaration.
Is there any prior art for this in other compilers, and if so can you
point to their documentation for what dependencies are handled and how?
This is an area where seeing the testsuite for the dependency handling
would be more useful at this point than the source code for an
implementation of it.
--
Joseph S. Myers
jsm28@cam.ac.uk
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
` (3 preceding siblings ...)
2003-02-27 11:28 ` Joseph S. Myers
@ 2003-02-27 12:23 ` Jan Hubicka
2003-02-28 3:36 ` Per Bothner
2003-02-27 14:10 ` Steven Bosscher
` (2 subsequent siblings)
7 siblings, 1 reply; 20+ messages in thread
From: Jan Hubicka @ 2003-02-27 12:23 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
> As I've hinted, I have been working for Apple on a "compile server".
> The idea is that the compiler when in server mode will re-use trees
> from header files. I've attached a semi-detailed design documnent.
> This is in texinfo format (so it can later be merged into gccint.text).
> To generate html do: makeinfo --html gcc-server.texi
>
> I'll submit a patch for the first prototype shortly, though note
> that this is far from clean enough to check in as is. Before the
> actual server, there is some amount of work in terms of splitting
> up compiler initialization into two parts: one-time initialization,
> and per-compilation initialization. I hope people will work with
> me on this.
>
> We're quite exited about this approach. We think it can lead
> to substantial compile-time speed-ups, and possibly other benefits.
> A preliminary benchmark suggested that re-using a large set of
> header files (the Apple Carbon framework) was over 3x as fast as
> normal processing.
So the main difference from current PCH scheme is that you don't reload
the PCH data from the disc each time, instead you keep them in memory
accross compilations?
Honza
> --
> --Per Bothner
> per@bothner.com pbothner@apple.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
` (4 preceding siblings ...)
2003-02-27 12:23 ` Jan Hubicka
@ 2003-02-27 14:10 ` Steven Bosscher
2003-02-27 23:04 ` Devang Patel
2003-03-03 10:18 ` Alexandre Oliva
7 siblings, 0 replies; 20+ messages in thread
From: Steven Bosscher @ 2003-02-27 14:10 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
Op do 27-02-2003, om 02:27 schreef Per Bothner:
> As I've hinted, I have been working for Apple on a "compile server".
> The idea is that the compiler when in server mode will re-use trees
> from header files. I've attached a semi-detailed design documnent.
Good stuff!
> This is in texinfo format (so it can later be merged into gccint.text).
> To generate html do: makeinfo --html gcc-server.texi
Or texi2pdf gcc-server.texi.
Also uncovers a minor error:
You say on lines 201 and 671: "@itemize bullet",
should be "@itemize @bullet".
Greetz
Steven
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 11:28 ` Joseph S. Myers
@ 2003-02-27 19:52 ` Per Bothner
2003-02-27 20:32 ` Joseph S. Myers
0 siblings, 1 reply; 20+ messages in thread
From: Per Bothner @ 2003-02-27 19:52 UTC (permalink / raw)
To: Joseph S. Myers; +Cc: Per Bothner, gcc
Joseph S. Myers wrote:
> The dependency system seems very complicated (necessarily) and potentially
> fragile.
It could be. The compile server is an option, and should not
be the default, certainly not anytime soon.
I argue that it may be acceptable for the compile server to
get confused in certain pathological cases, but we want to
minimize the chance of that happening. Ideally, we want to
compile exactly the same code and emit exactly the same
errors messages with and without the server, but approaching
that goal is an incremental process that will require experience
plus smart thinking.
> Note that any place where an identifier is lexed as not
> currently declared as a typedef, this must be treated as having a negative
> dependency,
I think this is only an issue when an identifier is being declared,
since when the identifier is *used*, a positive dependency is noted.
I also think this could only be an issue for non-local declarations,
since otherwise the duplicate definition will be caught.
> and any declaration of an identifier must probably depend on
> either any previous declaration in scope or on the absence of a previous
> declaration.
Again, I think only an issue for local declaration.
These are the kind of nasty corner cases that we will need to think
about, but I don't see them as a priority, since I don't think
they'll happen in real code, buggy or not.
> Is there any prior art for this in other compilers, and if so can you
> point to their documentation for what dependencies are handled and how?
Daniel Berlin mentioned some similariies with CodeStore, a persistent
code cache/database used by Montana, an incremental C++ compiler (used
in VisualAge for C++). Google "montana codestore" finds stuff I should
look at.
> This is an area where seeing the testsuite for the dependency handling
> would be more useful at this point than the source code for an
> implementation of it.
Hm. I guess I need to learn how to write such tests in an
other-than-ad-hoc manner.
--
--Per Bothner
per@bothner.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 19:52 ` Per Bothner
@ 2003-02-27 20:32 ` Joseph S. Myers
2003-02-28 3:58 ` Per Bothner
0 siblings, 1 reply; 20+ messages in thread
From: Joseph S. Myers @ 2003-02-27 20:32 UTC (permalink / raw)
To: Per Bothner; +Cc: Per Bothner, gcc
On Thu, 27 Feb 2003, Per Bothner wrote:
> > Note that any place where an identifier is lexed as not
> > currently declared as a typedef, this must be treated as having a negative
> > dependency,
>
> I think this is only an issue when an identifier is being declared,
> since when the identifier is *used*, a positive dependency is noted.
> I also think this could only be an issue for non-local declarations,
> since otherwise the duplicate definition will be caught.
In C90 you have implicit function declarations, so given a declaration of
x in an outer scope,
foo(x);
redeclares x if foo is a typedef, but implicitly declares and calls a
function foo if foo isn't declared. (No doubt header files should not
contain that sort of thing in inline functions, but I think compile-time
performance optimisations ought to start from the basis of always being
correct, even in corner cases, and then optimise particular cases where a
dependency isn't needed when it can be shown this is correct in that
particular case.)
> > This is an area where seeing the testsuite for the dependency handling
> > would be more useful at this point than the source code for an
> > implementation of it.
>
> Hm. I guess I need to learn how to write such tests in an
> other-than-ad-hoc manner.
Go through the standards sentence-by-sentence to find all ways in which
the interpretation or validity of code can depend on previous code in the
same translation unit. For C this should mostly be covered (apart from
the preprocessor cases) by every reference to an identifier having
dependencies on any relevant declaration in scope, or on the absence of
such - but the declared type of an identifier could be a composite type
arising from several source code declarations, and for GCC warnings there
may be other cases, e.g. checking based on extern declarations that have
gone out of scope, but warnings for which may still be useful. And
explicit declarations may modify built-in function declarations (which you
wanted to initialise only once), e.g. adding attributes to them.
The tests would probably be of the form: run one compilation that uses a
header that uses or declares an identifier in a particular way (a test for
every standard case of how it might be used or declared). Then run
another compilation in which the context for that identifier's use or
declaration is different (in every way in the standard in which it might
be different leading to different warnings or code generation) and check
that the diagnostics and code are identical to those without the server.
--
Joseph S. Myers
jsm28@cam.ac.uk
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
` (5 preceding siblings ...)
2003-02-27 14:10 ` Steven Bosscher
@ 2003-02-27 23:04 ` Devang Patel
2003-02-28 0:38 ` Per Bothner
2003-03-03 10:18 ` Alexandre Oliva
7 siblings, 1 reply; 20+ messages in thread
From: Devang Patel @ 2003-02-27 23:04 UTC (permalink / raw)
To: Per Bothner; +Cc: Devang Patel, gcc
Thanks for taking time to write detailed document. It helps to
understand more about compile server approach.
- "This assumes that (normally) a header file consists mainly of
declarations (including macro definitions), and the "meaning" of these
declarations does not chnge across compilations."
In C++ headers, inline function definitions are everywhere. But even
<stdio.h> brings in this little jewel
static __inline int __sputc(int _c, FILE *_p) {
if (--_p->_w >= 0 || (_p->_w >= _p->_lbfsize && (char)_c !=
'\n'))
return (*_p->_p++ = _c);
else
return (__swbuf(_c, _p));
}
- Do you have any info/data/rough estimate about fragments per header
for popular headers like stdio.h, iostream and Carbon.h ?
- "Depending on lack of a macro bindings" issue. In C++, namespaces
also creates similar problem.
- "Types defined in multiple locations". Do you plan to detect and
report violation of extended single one-definition rule?
- And last question is a implementation detail. Any idea how client
will know if server went in infinite loop or an ICE?
Thanks,
-Devang
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 23:04 ` Devang Patel
@ 2003-02-28 0:38 ` Per Bothner
0 siblings, 0 replies; 20+ messages in thread
From: Per Bothner @ 2003-02-28 0:38 UTC (permalink / raw)
To: Devang Patel; +Cc: gcc
Devang Patel wrote:
> - "This assumes that (normally) a header file consists mainly of
> declarations (including macro definitions), and the "meaning" of these
> declarations does not chnge across compilations."
>
> In C++ headers, inline function definitions are everywhere. But even
> <stdio.h> brings in this little jewel
>
> static __inline int __sputc(int _c, FILE *_p) {
> if (--_p->_w >= 0 || (_p->_w >= _p->_lbfsize && (char)_c != '\n'))
> return (*_p->_p++ = _c);
> else
> return (__swbuf(_c, _p));
> }
A definition is also a declaration. I don't off-hand see a problem
with this, beyond the general dependency-tracking issues. There is
one complication in that gcc mushes seclaration (including forward
declarations) and a definition into a single tree. We may need to
un-mush them, in case some other compilation unit just defines a
declaration. This is similar to the issue of declarations vs
definiions of struct tags, and can I think be handled the same
way (abstractly): When we see a combined declaration/definition,
treat as a separate declaration followed by a definition. This
allows a separate compilation to re-use the declaration without
the definition.
> - Do you have any info/data/rough estimate about fragments per header
> for popular headers like stdio.h, iostream and Carbon.h ?
Not really. The systems ones will probably have more conditionals and
other magic than typical headers.
> - "Depending on lack of a macro bindings" issue. In C++, namespaces also
> creates similar problem.
Yes, the presence or absence of using declarations and directives
may also complicate things.
> - "Types defined in multiple locations". Do you plan to detect and
> report violation of extended single one-definition rule?
I think that would be desirable, as a warning at least. I don't
think it would automatically fall out of the compile server work,
but the compile server at least makes it easier to make these
more global checks. It shouldn't be hard or expensive to add.
> - And last question is a implementation detail. Any idea how client will
> know if server went in infinite loop or an ICE?
No. If the server dies, hopefully there is some way the client
will receive an error report when waiting for the reponse, but
I don't know enough of it. Infinite loops are trickier. But
then how does make know when gcc does into an infinite loop>
--
--Per Bothner
per@bothner.com pbothner@apple.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 12:23 ` Jan Hubicka
@ 2003-02-28 3:36 ` Per Bothner
0 siblings, 0 replies; 20+ messages in thread
From: Per Bothner @ 2003-02-28 3:36 UTC (permalink / raw)
To: Jan Hubicka; +Cc: gcc
Jan Hubicka wrote:
> So the main difference from current PCH scheme is that you don't reload
> the PCH data from the disc each time, instead you keep them in memory
> accross compilations?
Basically, yes. And because of that, you don't have to
explicitly create or manage PCH files, and it is more
versatile in terms of include order.
--
--Per Bothner
per@bothner.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 20:32 ` Joseph S. Myers
@ 2003-02-28 3:58 ` Per Bothner
2003-02-28 14:21 ` Fergus Henderson
0 siblings, 1 reply; 20+ messages in thread
From: Per Bothner @ 2003-02-28 3:58 UTC (permalink / raw)
To: Joseph S. Myers; +Cc: gcc
Joseph S. Myers wrote:
> In C90 you have implicit function declarations, so given a declaration of
> x in an outer scope,
>
> foo(x);
>
> redeclares x if foo is a typedef, but implicitly declares and calls a
> function foo if foo isn't declared.
The problem is if foo was implicitly declared when the fragment was
first parsed, but there is an explicit, possibly conflicting declaration
when the fragment is re-used. Can the compile server detect this?
One option is to disallow implicit declarations (or at least emit
a warning) when using the compile server. People will just have
clean up their code if they want to use the compile server.
Another option is that the fragment has a dependency on the implicitly
declared function. When we decide whether we can re-use the fragment,
we note that there is a conflicting definition.
> (No doubt header files should not
> contain that sort of thing in inline functions, but I think compile-time
> performance optimisations ought to start from the basis of always being
> correct, even in corner cases, and then optimise particular cases where a
> dependency isn't needed when it can be shown this is correct in that
> particular case.)
That's very nice, but I think unrealistic. You could say the same thing
about run-time performance improvements, but they go in, and we later
discover bugs. We also sometimes check in changes that we *know*
cause regressions, because we think on the whole it improves things.
Note also the correctness is a matter of implementation matching
the specification. We are perfectly free to document that "if
you use the compile server, don't do XXX. You shouldn't do XXX anyway,
but the compile server may give unpredictable results". This isn't
that different from strict aliasing, which is actually *much more*
likely to cause real hard-to-detect problems.
Finally, I am not proposing the compiler server will be the default,
at least not for the foreseeable future. I'm proposing it as an
*option* that will work well with reasonably clean code, and
speed up compiling that. As a side benefit, it may catch some
consistency errors that we otherwise don't catch, or it may enable
whole-program optimizations, or it may have other benefits.
We will then *incrementally* add more consistency checks,
A "fully-correct" compile server, even if possible and desirable,
is something that will take a while. It should not stand in the
way of a "normally-correct" compile server that people can try if
they wish to.
--
--Per Bothner
per@bothner.com pbothner@apple.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-28 3:58 ` Per Bothner
@ 2003-02-28 14:21 ` Fergus Henderson
0 siblings, 0 replies; 20+ messages in thread
From: Fergus Henderson @ 2003-02-28 14:21 UTC (permalink / raw)
To: Per Bothner; +Cc: Joseph S. Myers, gcc
On 27-Feb-2003, Per Bothner <pbothner@apple.com> wrote:
>
> One option is to disallow implicit declarations (or at least emit
> a warning) when using the compile server. People will just have
> clean up their code if they want to use the compile server.
I think that would be completely acceptable.
--
Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit
The University of Melbourne | of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-02-27 5:55 compile server design document Per Bothner
` (6 preceding siblings ...)
2003-02-27 23:04 ` Devang Patel
@ 2003-03-03 10:18 ` Alexandre Oliva
2003-03-04 21:38 ` Per Bothner
7 siblings, 1 reply; 20+ messages in thread
From: Alexandre Oliva @ 2003-03-03 10:18 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
On Feb 26, 2003, Per Bothner <pbothner@apple.com> wrote:
> The idea is that the compiler when in server mode will re-use trees
> from header files.
I'm yet to give your text and patch a closer look, but at a first
overview, it doesn't seem to be very friendly to preprocessed files,
which GCC has been more and more commonly fed by people who use ccache
and distcc. Is there any effort in the direction of improving compile
performance in the ccache+distcc scenario, by say recognizing
fragments (as you call them) from header files even when they're
encountered in a previously-preprocessed file?
--
Alexandre Oliva Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist Professional serial bug killer
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-03-03 10:18 ` Alexandre Oliva
@ 2003-03-04 21:38 ` Per Bothner
2003-03-04 21:44 ` Michael Matz
0 siblings, 1 reply; 20+ messages in thread
From: Per Bothner @ 2003-03-04 21:38 UTC (permalink / raw)
To: Alexandre Oliva; +Cc: gcc
Alexandre Oliva wrote:
>
> I'm yet to give your text and patch a closer look, but at a first
> overview, it doesn't seem to be very friendly to preprocessed files,
> which GCC has been more and more commonly fed by people who use ccache
> and distcc.
Gcc is currently optimized for *not* creating preprocessed files.
So systems like distcc that use preprocessed files are deliberately
giving up some performance (and the "mainstream" of work in gcc
compilation-speed) improvements - in the hope of gains in other
ways. In addition ccache has to calculate a hash of each input
file. My hope is that that the compile server can eliminate the
need for ccache.
Distributed compilation is still useful, but I'm not that comfortable
with distcc's approach of shipping preprocessed files. It substantially
increases network bandwidth. Of course this isn't really a problem
if you're on a fast network - but in that case why not instead get the
header files using (say) NFS or FTP - and have them be cached by the
compile server?
My preferred approach would be just running one or more compile
servers per node.
> Is there any effort in the direction of improving compile
> performance in the ccache+distcc scenario, by say recognizing
> fragments (as you call them) from header files even when they're
> encountered in a previously-preprocessed file?
It's not something I've considered, nor it's not a priority for
me at this point. The problem is that you have to quickly
recognize that a header file fragment in a preprocessed file
is the same as an already-seen fragment. Presumably you can
do this using hashing.
--
--Per Bothner
per@bothner.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-03-04 21:38 ` Per Bothner
@ 2003-03-04 21:44 ` Michael Matz
2003-03-04 21:56 ` Per Bothner
0 siblings, 1 reply; 20+ messages in thread
From: Michael Matz @ 2003-03-04 21:44 UTC (permalink / raw)
To: Per Bothner; +Cc: Alexandre Oliva, gcc
Hi,
On Tue, 4 Mar 2003, Per Bothner wrote:
> Distributed compilation is still useful, but I'm not that comfortable
> with distcc's approach of shipping preprocessed files. It substantially
> increases network bandwidth. Of course this isn't really a problem
> if you're on a fast network - but in that case why not instead get the
> header files using (say) NFS or FTP - and have them be cached by the
> compile server?
I tell you: because setting up NFS servers would be another thing to do.
This really is a hurdle. There were other distributed compiling
environments in the past (pmake and friends) for a long time, which more
or less all were based on some kind of network file system. Still those
never made it to mainstream use (not even in all companies which could
theoretically make use of that). A part of the problems are also locking
problems, or problems with timestamps and synchronisation. Whatever,
using NFS for this is not easy. With teambuilder and distcc this changed.
Simply install/run a daemon, and be done with it. With distcc it's a bit
more difficult, as you also have to configure the clients with a list of
to be used compiler servers, and it doesn't have a central scheduler
(compared to teambuilder).
Anyway, if you say, that using preprocessed files is increasing network
load, the same happens for NFS/AFS based systems. There is no big
difference if the content for the preprocessed contents comes in directly
as preprocessed file, or from header files over NFS, at least not in the
direction you indicate. On the contrary, the preprocessed file will most
probably be smaller, than the whole files used for building it, as for
instance all comments are stripped. You might answer, that one should not
take common headers from NFS, but instead have them locally on each
compile server. That would add a _huge_ maintenace cost in having them
consistent.
I tried to use all three approaches (NFS+pmake, distcc and teambuilder)
with KDE. pmake never really worked that well, distcc/teambuilder just
worked. So, while in theory totally equivalent solutions, in practice
there's a huge difference. compile servers not using preprocessed files
are practically unusable ;)
Ciao,
Michael.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-03-04 21:44 ` Michael Matz
@ 2003-03-04 21:56 ` Per Bothner
2003-03-04 22:32 ` Michael Matz
0 siblings, 1 reply; 20+ messages in thread
From: Per Bothner @ 2003-03-04 21:56 UTC (permalink / raw)
To: Michael Matz; +Cc: gcc
Michael Matz wrote:
> I tell you: because setting up NFS servers would be another thing to do.
> This really is a hurdle. ... With teambuilder and distcc this changed.
> Simply install/run a daemon, and be done with it.
So why not write a user-space daemon on the user side that can
ship header files as the compilation daemon requests it? This
could be builtin to the process that ships off compilation
requests to the compilation farm. Or use ftp or a trivial
user-mode http server?
> Anyway, if you say, that using preprocessed files is increasing network
> load, the same happens for NFS/AFS based systems. There is no big
> difference if the content for the preprocessed contents comes in directly
> as preprocessed file, or from header files over NFS, at least not in the
> direction you indicate.
If you ship preprocessed files you need to copy the header
file contents for *each* file you compile. If you use a
remote compile server it can (and will) cache the contents
of the header files between compilation requests. It seems
much more efficient to me. Also, you don't have to complicate
the compile server logic, except that you need to be able to
request a file over the network. That's a simple localized
change.
> I tried to use all three approaches (NFS+pmake, distcc and teambuilder)
> with KDE. pmake never really worked that well, distcc/teambuilder just
> worked. So, while in theory totally equivalent solutions, in practice
> there's a huge difference. compile servers not using preprocessed files
> are practically unusable ;)
The conclusion does not follow.
--
--Per Bothner
per@bothner.com http://www.bothner.com/per/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: compile server design document
2003-03-04 21:56 ` Per Bothner
@ 2003-03-04 22:32 ` Michael Matz
0 siblings, 0 replies; 20+ messages in thread
From: Michael Matz @ 2003-03-04 22:32 UTC (permalink / raw)
To: Per Bothner; +Cc: gcc
Hi,
On Tue, 4 Mar 2003, Per Bothner wrote:
> > Simply install/run a daemon, and be done with it.
>
> So why not write a user-space daemon on the user side that can
> ship header files as the compilation daemon requests it?
Sure, sure. This all could be done. If someone does it ;-) (btw. a
poor-mans version of such daemon exists: gcc -E 1/2 ;-) )
> > Anyway, if you say, that using preprocessed files is increasing network
> > load, the same happens for NFS/AFS based systems. There is no big
> > difference if the content for the preprocessed contents comes in directly
> > as preprocessed file, or from header files over NFS, at least not in the
> > direction you indicate.
>
> If you ship preprocessed files you need to copy the header
> file contents for *each* file you compile. If you use a
> remote compile server it can (and will) cache the contents
> of the header files between compilation requests. It seems
> much more efficient to me.
It is. But it also adds a whole new level of complexity. Namely a
consistency protocol. Think about headers which are generated by the
build process, maybe shadowing global headers. You would also need to add
such a thing, and then you have the problem to explain, why you didn't
simply use a (caching like AFS) networking file system, because _those_
already do have means for consistency. This is the basic difference:
either you do the combining client-side (on the caller) --> simple
maintenable system
or you do it server-side --> complex system.
Given the expected (from experience) performance advantage of the complex
system (namely not much), while compiling C++ code, I would choose the
simple system. Yes, I constrained the use to one language, in fact one,
where compiling takes _much_ longer than preprocessing, or tranfering
files over network. But this is indeed the most usefull/realistic case.
Ciao,
Michael.
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2003-03-04 22:03 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-02-27 5:55 compile server design document Per Bothner
2003-02-27 6:13 ` Tolga Dalman
2003-02-27 7:08 ` Timothy J. Wood
2003-02-27 7:27 ` Per Bothner
2003-02-27 10:14 ` Per Bothner
2003-02-27 11:28 ` Joseph S. Myers
2003-02-27 19:52 ` Per Bothner
2003-02-27 20:32 ` Joseph S. Myers
2003-02-28 3:58 ` Per Bothner
2003-02-28 14:21 ` Fergus Henderson
2003-02-27 12:23 ` Jan Hubicka
2003-02-28 3:36 ` Per Bothner
2003-02-27 14:10 ` Steven Bosscher
2003-02-27 23:04 ` Devang Patel
2003-02-28 0:38 ` Per Bothner
2003-03-03 10:18 ` Alexandre Oliva
2003-03-04 21:38 ` Per Bothner
2003-03-04 21:44 ` Michael Matz
2003-03-04 21:56 ` Per Bothner
2003-03-04 22:32 ` Michael Matz
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).