[libcpp] RFA: Add support for comments retrieval

[libcpp] RFA: Add support for comments retrieval

[Arnaud Charlet]

This is a patch that adds support for easy retrieval of C/C++ comments
parsed by libcpp.

We have another patch depending on this one that would then take advantage
of this new "cpp_get_comments" function. This other patch will add the
capability to dump GCC GENERIC trees using Ada syntax, thus providing
a way to generate Ada specs automatically from C/C++ header files.
In this context, being able to also dump the original comments from the
header files is very useful.

I'm sure this function could then be used in other contexts (e.g. other
kind of source code analyzer/plug-ins based on GCC).

Tested on i686-pc-linux-gnu, OK for trunk ?

2008-09-26  Matthew Gingell  <gingell@adacore.com>
	    Arnaud Charlet  <charlet@adacore.com>

	* include/cpplib.h (cpp_comments, cpp_comment_table): New structs.
	(cpp_get_comments): New function.

	* lex.c (store_comment, cpp_get_comments): New function.s 
	(comments): New struct.
	(save_comment): Store comments in comments struct.

Index: include/cpplib.h
===================================================================
--- include/cpplib.h	(revision 140660)
+++ include/cpplib.h	(working copy)
@@ -870,6 +870,23 @@ extern const char *cpp_type2name (enum c
 extern cppchar_t cpp_parse_escape (cpp_reader *, const unsigned char ** pstr,
 				   const unsigned char *limit, int wide);
 
+typedef struct
+{
+  const char *comment;
+  source_location sloc;
+} cpp_comment;
+
+typedef struct
+{
+  cpp_comment *entries;
+  int count;
+  int allocated;
+} cpp_comment_table;
+
+/* Returns the table of comments encountered by the preprocessor. This
+   table is only populated when pfile->state.save_comments is true. */
+extern cpp_comment_table cpp_get_comments (void);
+
 /* In hash.c */
 
 /* Lookup an identifier in the hashtable.  Puts the identifier in the
Index: lex.c
===================================================================
--- lex.c	(revision 140660)
+++ lex.c	(working copy)
@@ -55,6 +55,7 @@ static int skip_line_comment (cpp_reader
 static void skip_whitespace (cpp_reader *, cppchar_t);
 static void lex_string (cpp_reader *, cpp_token *, const uchar *);
 static void save_comment (cpp_reader *, cpp_token *, const uchar *, cppchar_t);
+static void store_comment (cpp_token *);
 static void create_literal (cpp_reader *, cpp_token *, const uchar *,
 			    unsigned int, enum cpp_ttype);
 static bool warn_in_comment (cpp_reader *, _cpp_line_note *);
@@ -670,6 +671,44 @@ lex_string (cpp_reader *pfile, cpp_token
   create_literal (pfile, token, base, cur - base, type);
 }
 
+/* Table for storing comments in the pfile->state.save_comments case.
+   In other cases there is no overhead. */
+
+static cpp_comment_table comments = {NULL, 0, 0};
+
+/* Return the comment table. The client may not make any assumption
+   about the ordering of the table.  */
+extern cpp_comment_table
+cpp_get_comments (void) 
+{
+  return comments;
+}
+
+/* Append a comment to the end of the comment table. */
+static void 
+store_comment (cpp_token *token) 
+{
+  if (comments.allocated == 0)
+    {
+      comments.allocated = 256; 
+      comments.entries = 
+	(cpp_comment *) xmalloc (comments.allocated * sizeof (cpp_comment));
+    }
+
+  if (comments.count == comments.allocated)
+    {
+      comments.allocated *= 2;
+      comments.entries = 
+	(cpp_comment *) xrealloc (comments.entries, 
+				  comments.allocated * sizeof (cpp_comment));
+    }
+
+  comments.entries [comments.count].comment = 
+    xstrdup ((char *) (token->val.str.text));
+  comments.entries [comments.count].sloc = token->src_loc;
+  comments.count++;
+}
+
 /* The stored comment includes the comment start and any terminator.  */
 static void
 save_comment (cpp_reader *pfile, cpp_token *token, const unsigned char *from,
@@ -709,6 +748,9 @@ save_comment (cpp_reader *pfile, cpp_tok
       buffer[clen - 2] = '*';
       buffer[clen - 1] = '/';
     }
+
+  /* Finally store this comment for use by clients of libcpp. */
+  store_comment (token);
 }
 
 /* Allocate COUNT tokens for RUN.  */

Re: [libcpp] RFA: Add support for comments retrieval

[Basile STARYNKEVITCH]

Arnaud Charlet wrote:
> This is a patch that adds support for easy retrieval of C/C++ comments
> parsed by libcpp.
> 
> We have another patch depending on this one that would then take advantage
> of this new "cpp_get_comments" function. This other patch will add the
> capability to dump GCC GENERIC trees using Ada syntax, thus providing
> a way to generate Ada specs automatically from C/C++ header files.
> In this context, being able to also dump the original comments from the
> header files is very useful.
> 
> I'm sure this function could then be used in other contexts (e.g. other
> kind of source code analyzer/plug-ins based on GCC).

I agree, but in that case I suggest adding some documentation somewhere, 
probably a small paragraph in some *.texi file. I am currently not able 
to figure out how could I use such a function, even if it interests me a 
lot. My previous thinking was that comments is the first thing a lexer 
wants to be removed!


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: [libcpp] RFA: Add support for comments retrieval

[Tom Tromey]

>>>>> "Arnaud" == Arnaud Charlet <charlet@adacore.com> writes:

Arnaud> This is a patch that adds support for easy retrieval of C/C++ comments
Arnaud> parsed by libcpp.

I think the general idea is ok.

There are a few changes I would like.  Some of these are nits :)

Arnaud> 	* lex.c (store_comment, cpp_get_comments): New function.s 

Trailing 's'.

Arnaud> +typedef struct
Arnaud> +{
Arnaud> +  const char *comment;
Arnaud> +  source_location sloc;
Arnaud> +} cpp_comment;

Please put a comment on the struct and the various fields.

Arnaud> +typedef struct
Arnaud> +{
Arnaud> +  cpp_comment *entries;
Arnaud> +  int count;
Arnaud> +  int allocated;
Arnaud> +} cpp_comment_table;

Ditto.

Arnaud> +/* Table for storing comments in the pfile->state.save_comments case.
Arnaud> +   In other cases there is no overhead. */
Arnaud> +
Arnaud> +static cpp_comment_table comments = {NULL, 0, 0};

Instead of a global, I think this should be in struct cpp_reader.
This implies that cpp_get_comments needs a cpp_reader argument.
Also, destroying the cpp_reader should free the comment entries.

I wonder if we should have a new option (in cpp_options) controlling
this.

Tom

Re: [libcpp] RFA: Add support for comments retrieval

[Arnaud Charlet]

> I agree, but in that case I suggest adding some documentation somewhere, 
> probably a small paragraph in some *.texi file. I am currently not able to 

It's documented in include/cpplib.h, but if people think more documentation
elsewhere would be helpful, that's fine with me, lathough I do not really
see why this function would be different from others that are documented
in the sources directly.

> figure out how could I use such a function, even if it interests me a lot. 
> My previous thinking was that comments is the first thing a lexer wants to 
> be removed!

Note that the comment handling is optional in libcpp, triggered with the
-C switch, so this patch only affects this mode.

Arno

Re: [libcpp] RFA: Add support for comments retrieval

[Arnaud Charlet]

> I think the general idea is ok.

Thanks.

> There are a few changes I would like.  Some of these are nits :)

Sure, thanks for the quick review.
Here is a new version taking your comments into account:

2008-09-26  Matthew Gingell  <gingell@adacore.com>
	    Arnaud Charlet  <charlet@adacore.com>

	* include/cpplib.h (cpp_comments, cpp_comment_table): New structs.
	(cpp_get_comments): New function.

	* internal.h (struct cpp_reader): Add comments field.

	* init.c (cpp_destroy): Free comments.

	* lex.c (store_comment, cpp_get_comments): New functions.
	(comments): New struct.
	(save_comment): Store comments in comments struct.

Index: include/cpplib.h
===================================================================
--- include/cpplib.h	(revision 140660)
+++ include/cpplib.h	(working copy)
@@ -870,6 +870,28 @@ extern const char *cpp_type2name (enum c
 extern cppchar_t cpp_parse_escape (cpp_reader *, const unsigned char ** pstr,
 				   const unsigned char *limit, int wide);
 
+/* Structure used to hold a comment block at a given location in the
+   source code.  */
+
+typedef struct
+{
+  const char *comment;   /* string containing comment at a given location.  */
+  source_location sloc;  /* source location for the given comment.  */
+} cpp_comment;
+
+/* Structure holding all comments for a given cpp_reader.  */
+
+typedef struct
+{
+  cpp_comment *entries;  /* table of comment entries.  */
+  int count;             /* number of actual entries entered in the table.  */
+  int allocated;         /* number of entries allocated currently.  */
+} cpp_comment_table;
+
+/* Returns the table of comments encountered by the preprocessor. This
+   table is only populated when pfile->state.save_comments is true. */
+extern cpp_comment_table *cpp_get_comments (cpp_reader *);
+
 /* In hash.c */
 
 /* Lookup an identifier in the hashtable.  Puts the identifier in the
Index: init.c
===================================================================
--- init.c	(revision 140660)
+++ init.c	(working copy)
@@ -245,6 +245,7 @@ cpp_destroy (cpp_reader *pfile)
 {
   cpp_context *context, *contextn;
   tokenrun *run, *runn;
+  int i;
 
   free (pfile->op_stack);
 
@@ -287,6 +288,14 @@ cpp_destroy (cpp_reader *pfile)
       free (context);
     }
 
+  if (pfile->comments.entries)
+    {
+      for (i = 0; i < pfile->comments.count; i++)
+	free (pfile->comments.entries [i].comment);
+
+      free (pfile->comments.entries);
+    }
+
   free (pfile);
 }
 
Index: internal.h
===================================================================
--- internal.h	(revision 140660)
+++ internal.h	(working copy)
@@ -471,6 +471,9 @@ struct cpp_reader
 
   /* Next value of __COUNTER__ macro. */
   unsigned int counter;
+
+  /* Table of comments, when state.save_comments is true.  */
+  cpp_comment_table comments;
 };
 
 /* Character classes.  Based on the more primitive macros in safe-ctype.h.
Index: lex.c
===================================================================
--- lex.c	(revision 140660)
+++ lex.c	(working copy)
@@ -55,6 +55,7 @@ static int skip_line_comment (cpp_reader
 static void skip_whitespace (cpp_reader *, cppchar_t);
 static void lex_string (cpp_reader *, cpp_token *, const uchar *);
 static void save_comment (cpp_reader *, cpp_token *, const uchar *, cppchar_t);
+static void store_comment (cpp_reader *, cpp_token *);
 static void create_literal (cpp_reader *, cpp_token *, const uchar *,
 			    unsigned int, enum cpp_ttype);
 static bool warn_in_comment (cpp_reader *, _cpp_line_note *);
@@ -670,6 +671,39 @@ lex_string (cpp_reader *pfile, cpp_token
   create_literal (pfile, token, base, cur - base, type);
 }
 
+/* Return the comment table. The client may not make any assumption
+   about the ordering of the table.  */
+cpp_comment_table *
+cpp_get_comments (cpp_reader *pfile)
+{
+  return &pfile->comments;
+}
+
+/* Append a comment to the end of the comment table. */
+static void 
+store_comment (cpp_reader *pfile, cpp_token *token) 
+{
+  if (pfile->comments.allocated == 0)
+    {
+      pfile->comments.allocated = 256; 
+      pfile->comments.entries = (cpp_comment *) xmalloc
+	(pfile->comments.allocated * sizeof (cpp_comment));
+    }
+
+  if (pfile->comments.count == pfile->comments.allocated)
+    {
+      pfile->comments.allocated *= 2;
+      pfile->comments.entries = (cpp_comment *) xrealloc
+	(pfile->comments.entries,
+	 pfile->comments.allocated * sizeof (cpp_comment));
+    }
+
+  pfile->comments.entries [pfile->comments.count].comment = 
+    xstrdup ((char *) (token->val.str.text));
+  pfile->comments.entries [pfile->comments.count].sloc = token->src_loc;
+  pfile->comments.count++;
+}
+
 /* The stored comment includes the comment start and any terminator.  */
 static void
 save_comment (cpp_reader *pfile, cpp_token *token, const unsigned char *from,
@@ -709,6 +743,9 @@ save_comment (cpp_reader *pfile, cpp_tok
       buffer[clen - 2] = '*';
       buffer[clen - 1] = '/';
     }
+
+  /* Finally store this comment for use by clients of libcpp. */
+  store_comment (pfile, token);
 }
 
 /* Allocate COUNT tokens for RUN.  */

Re: [libcpp] RFA: Add support for comments retrieval

[Paolo Bonzini]

> +typedef struct
> +{
> +  const char *comment;   /* string containing comment at a given location.  */

Should be

  /* Text of the comment including (or is it excluding?) the
     terminators.  */
  const char *comment;

  /* Source location for the given comment.  */
  source_location sloc;

... and likewise for the other struct.

Thanks,

Paolo

Re: [libcpp] RFA: Add support for comments retrieval

[Arnaud Charlet]

> Should be
> 
>   /* Text of the comment including (or is it excluding?) the
>      terminators.  */
>   const char *comment;
> 
>   /* Source location for the given comment.  */
>   source_location sloc;
> 
> ... and likewise for the other struct.
> 
> Thanks,

Absolutely, will fix.

Is the patch OK with these changes ?

Arno

Re: [libcpp] RFA: Add support for comments retrieval

[Arnaud Charlet]

> Is the patch OK with these changes ?

Appended for completeness.

2008-09-26  Matthew Gingell  <gingell@adacore.com>
	    Arnaud Charlet  <charlet@adacore.com>

	* include/cpplib.h (cpp_comments, cpp_comment_table): New structs.
	(cpp_get_comments): New function.

	* internal.h (struct cpp_reader): Add comments field.

	* init.c (cpp_destroy): Free comments.

	* lex.c (store_comment, cpp_get_comments): New functions.
	(comments): New struct.
	(save_comment): Store comments in comments struct.

Index: include/cpplib.h
===================================================================
--- include/cpplib.h	(revision 140660)
+++ include/cpplib.h	(working copy)
@@ -870,6 +870,36 @@ extern const char *cpp_type2name (enum c
 extern cppchar_t cpp_parse_escape (cpp_reader *, const unsigned char ** pstr,
 				   const unsigned char *limit, int wide);
 
+/* Structure used to hold a comment block at a given location in the
+   source code.  */
+
+typedef struct
+{
+  /* Text of the comment including the terminators.  */
+  char *comment;
+
+  /* source location for the given comment.  */
+  source_location sloc;
+} cpp_comment;
+
+/* Structure holding all comments for a given cpp_reader.  */
+
+typedef struct
+{
+  /* table of comment entries.  */
+  cpp_comment *entries;
+
+  /* number of actual entries entered in the table.  */
+  int count;
+
+  /* number of entries allocated currently.  */
+  int allocated;
+} cpp_comment_table;
+
+/* Returns the table of comments encountered by the preprocessor. This
+   table is only populated when pfile->state.save_comments is true. */
+extern cpp_comment_table *cpp_get_comments (cpp_reader *);
+
 /* In hash.c */
 
 /* Lookup an identifier in the hashtable.  Puts the identifier in the
Index: init.c
===================================================================
--- init.c	(revision 140660)
+++ init.c	(working copy)
@@ -245,6 +245,7 @@ cpp_destroy (cpp_reader *pfile)
 {
   cpp_context *context, *contextn;
   tokenrun *run, *runn;
+  int i;
 
   free (pfile->op_stack);
 
@@ -287,6 +288,14 @@ cpp_destroy (cpp_reader *pfile)
       free (context);
     }
 
+  if (pfile->comments.entries)
+    {
+      for (i = 0; i < pfile->comments.count; i++)
+	free (pfile->comments.entries [i].comment);
+
+      free (pfile->comments.entries);
+    }
+
   free (pfile);
 }
 
Index: internal.h
===================================================================
--- internal.h	(revision 140660)
+++ internal.h	(working copy)
@@ -471,6 +471,9 @@ struct cpp_reader
 
   /* Next value of __COUNTER__ macro. */
   unsigned int counter;
+
+  /* Table of comments, when state.save_comments is true.  */
+  cpp_comment_table comments;
 };
 
 /* Character classes.  Based on the more primitive macros in safe-ctype.h.
Index: lex.c
===================================================================
--- lex.c	(revision 140660)
+++ lex.c	(working copy)
@@ -55,6 +55,7 @@ static int skip_line_comment (cpp_reader
 static void skip_whitespace (cpp_reader *, cppchar_t);
 static void lex_string (cpp_reader *, cpp_token *, const uchar *);
 static void save_comment (cpp_reader *, cpp_token *, const uchar *, cppchar_t);
+static void store_comment (cpp_reader *, cpp_token *);
 static void create_literal (cpp_reader *, cpp_token *, const uchar *,
 			    unsigned int, enum cpp_ttype);
 static bool warn_in_comment (cpp_reader *, _cpp_line_note *);
@@ -670,6 +671,39 @@ lex_string (cpp_reader *pfile, cpp_token
   create_literal (pfile, token, base, cur - base, type);
 }
 
+/* Return the comment table. The client may not make any assumption
+   about the ordering of the table.  */
+cpp_comment_table *
+cpp_get_comments (cpp_reader *pfile)
+{
+  return &pfile->comments;
+}
+
+/* Append a comment to the end of the comment table. */
+static void 
+store_comment (cpp_reader *pfile, cpp_token *token) 
+{
+  if (pfile->comments.allocated == 0)
+    {
+      pfile->comments.allocated = 256; 
+      pfile->comments.entries = (cpp_comment *) xmalloc
+	(pfile->comments.allocated * sizeof (cpp_comment));
+    }
+
+  if (pfile->comments.count == pfile->comments.allocated)
+    {
+      pfile->comments.allocated *= 2;
+      pfile->comments.entries = (cpp_comment *) xrealloc
+	(pfile->comments.entries,
+	 pfile->comments.allocated * sizeof (cpp_comment));
+    }
+
+  pfile->comments.entries [pfile->comments.count].comment = 
+    xstrdup ((char *) (token->val.str.text));
+  pfile->comments.entries [pfile->comments.count].sloc = token->src_loc;
+  pfile->comments.count++;
+}
+
 /* The stored comment includes the comment start and any terminator.  */
 static void
 save_comment (cpp_reader *pfile, cpp_token *token, const unsigned char *from,
@@ -709,6 +743,9 @@ save_comment (cpp_reader *pfile, cpp_tok
       buffer[clen - 2] = '*';
       buffer[clen - 1] = '/';
     }
+
+  /* Finally store this comment for use by clients of libcpp. */
+  store_comment (pfile, token);
 }
 
 /* Allocate COUNT tokens for RUN.  */

Re: [libcpp] RFA: Add support for comments retrieval

[Tom Tromey]

>>>>> "Arnaud" == Arnaud Charlet <charlet@adacore.com> writes:

>> Is the patch OK with these changes ?

Arnaud> Appended for completeness.

I have two trivial nits.  It is ok with these changes.

Normally I would say that this should be delayed until Stage 1, but I
think it is reasonably small and safe, and so it can go in now.

Arnaud> 2008-09-26  Matthew Gingell  <gingell@adacore.com>
Arnaud> 	    Arnaud Charlet  <charlet@adacore.com>

Arnaud> 	* include/cpplib.h (cpp_comments, cpp_comment_table): New structs.
Arnaud> 	(cpp_get_comments): New function.

Arnaud> 	* internal.h (struct cpp_reader): Add comments field.

[...]

For a single change which spans multiple files, I don't put blank
lines between the file entries in the ChangeLog.

Arnaud> +	free (pfile->comments.entries [i].comment);

No space before the '['.  There are a couple cases of this.

Tom

Re: [libcpp] RFA: Add support for comments retrieval

[Arnaud Charlet]

> I have two trivial nits.  It is ok with these changes.
> 
> Normally I would say that this should be delayed until Stage 1, but I
> think it is reasonably small and safe, and so it can go in now.

Understood, thanks.

Here is the final patch with nits fixed, retested and committed
on trunk.

2008-10-05  Matthew Gingell  <gingell@adacore.com>
	    Arnaud Charlet  <charlet@adacore.com>

	* include/cpplib.h (cpp_comments, cpp_comment_table): New structs.
	(cpp_get_comments): New function.
	* internal.h (struct cpp_reader): Add comments field.
	* init.c (cpp_destroy): Free comments.
	* lex.c (store_comment, cpp_get_comments): New functions.
	(comments): New struct.
	(save_comment): Store comments in comments struct.

Index: include/cpplib.h
===================================================================
--- include/cpplib.h	(revision 140660)
+++ include/cpplib.h	(working copy)
@@ -870,6 +870,36 @@ extern const char *cpp_type2name (enum c
 extern cppchar_t cpp_parse_escape (cpp_reader *, const unsigned char ** pstr,
 				   const unsigned char *limit, int wide);
 
+/* Structure used to hold a comment block at a given location in the
+   source code.  */
+
+typedef struct
+{
+  /* Text of the comment including the terminators.  */
+  char *comment;
+
+  /* source location for the given comment.  */
+  source_location sloc;
+} cpp_comment;
+
+/* Structure holding all comments for a given cpp_reader.  */
+
+typedef struct
+{
+  /* table of comment entries.  */
+  cpp_comment *entries;
+
+  /* number of actual entries entered in the table.  */
+  int count;
+
+  /* number of entries allocated currently.  */
+  int allocated;
+} cpp_comment_table;
+
+/* Returns the table of comments encountered by the preprocessor. This
+   table is only populated when pfile->state.save_comments is true. */
+extern cpp_comment_table *cpp_get_comments (cpp_reader *);
+
 /* In hash.c */
 
 /* Lookup an identifier in the hashtable.  Puts the identifier in the
Index: init.c
===================================================================
--- init.c	(revision 140660)
+++ init.c	(working copy)
@@ -245,6 +245,7 @@ cpp_destroy (cpp_reader *pfile)
 {
   cpp_context *context, *contextn;
   tokenrun *run, *runn;
+  int i;
 
   free (pfile->op_stack);
 
@@ -287,6 +288,14 @@ cpp_destroy (cpp_reader *pfile)
       free (context);
     }
 
+  if (pfile->comments.entries)
+    {
+      for (i = 0; i < pfile->comments.count; i++)
+	free (pfile->comments.entries[i].comment);
+
+      free (pfile->comments.entries);
+    }
+
   free (pfile);
 }
 
Index: internal.h
===================================================================
--- internal.h	(revision 140660)
+++ internal.h	(working copy)
@@ -471,6 +471,9 @@ struct cpp_reader
 
   /* Next value of __COUNTER__ macro. */
   unsigned int counter;
+
+  /* Table of comments, when state.save_comments is true.  */
+  cpp_comment_table comments;
 };
 
 /* Character classes.  Based on the more primitive macros in safe-ctype.h.
Index: lex.c
===================================================================
--- lex.c	(revision 140660)
+++ lex.c	(working copy)
@@ -55,6 +55,7 @@ static int skip_line_comment (cpp_reader
 static void skip_whitespace (cpp_reader *, cppchar_t);
 static void lex_string (cpp_reader *, cpp_token *, const uchar *);
 static void save_comment (cpp_reader *, cpp_token *, const uchar *, cppchar_t);
+static void store_comment (cpp_reader *, cpp_token *);
 static void create_literal (cpp_reader *, cpp_token *, const uchar *,
 			    unsigned int, enum cpp_ttype);
 static bool warn_in_comment (cpp_reader *, _cpp_line_note *);
@@ -670,6 +671,51 @@ lex_string (cpp_reader *pfile, cpp_token
   create_literal (pfile, token, base, cur - base, type);
 }
 
+/* Return the comment table. The client may not make any assumption
+   about the ordering of the table.  */
+cpp_comment_table *
+cpp_get_comments (cpp_reader *pfile)
+{
+  return &pfile->comments;
+}
+
+/* Append a comment to the end of the comment table. */
+static void 
+store_comment (cpp_reader *pfile, cpp_token *token) 
+{
+  int len;
+
+  if (pfile->comments.allocated == 0)
+    {
+      pfile->comments.allocated = 256; 
+      pfile->comments.entries = (cpp_comment *) xmalloc
+	(pfile->comments.allocated * sizeof (cpp_comment));
+    }
+
+  if (pfile->comments.count == pfile->comments.allocated)
+    {
+      pfile->comments.allocated *= 2;
+      pfile->comments.entries = (cpp_comment *) xrealloc
+	(pfile->comments.entries,
+	 pfile->comments.allocated * sizeof (cpp_comment));
+    }
+
+  len = token->val.str.len;
+
+  /* Copy comment. Note, token may not be NULL terminated. */
+  pfile->comments.entries[pfile->comments.count].comment = 
+    (char *) xmalloc (sizeof (char) * (len + 1));
+  memcpy (pfile->comments.entries[pfile->comments.count].comment,
+	  token->val.str.text, len);
+  pfile->comments.entries[pfile->comments.count].comment[len] = '\0';
+
+  /* Set source location. */
+  pfile->comments.entries[pfile->comments.count].sloc = token->src_loc;
+
+  /* Increment the count of entries in the comment table. */
+  pfile->comments.count++;
+}
+
 /* The stored comment includes the comment start and any terminator.  */
 static void
 save_comment (cpp_reader *pfile, cpp_token *token, const unsigned char *from,
@@ -709,6 +755,9 @@ save_comment (cpp_reader *pfile, cpp_tok
       buffer[clen - 2] = '*';
       buffer[clen - 1] = '/';
     }
+
+  /* Finally store this comment for use by clients of libcpp. */
+  store_comment (pfile, token);
 }
 
 /* Allocate COUNT tokens for RUN.  */