[PATCH 0/7] Tracking locations of tokens resulting from macro expansion

[Dodji Seketeli <dodji@redhat.com>]

Hello,

This is an update of the patch set that I initially posted to
http://gcc.gnu.org/ml/gcc-patches/2010-12/msg00858.html.

The main goals achieved by this set are the following:

- Decrease the overall memory consumption.  On the tests I have done
  on a reasonably big C++ program compiled with the previous patch
  set, I have noticed a global memory consumption peak increase of
  13%, for the whole compilation.  With this new patch set, the global
  memory consumption peak is down to 0.6%.  Details of the figures are
  appended at the end of this message.  To do this, I have quite
  heavily modified the patch "Generate virtual locations for tokens"
  and I have added the patch "Reduce memory waste due to
  non-power-of-2 allocs".  The changes of these patches are explained
  in their respective preambles.

- Change the diagnostics output as suggested by Paolo and Gaby in the
  sub-thread http://gcc.gnu.org/ml/gcc-patches/2010-12/msg00987.html.

Besides, I have hopefully addressed most of the comments that were
made in the initial discussion thread.  Gaby suggested that I drop the
-ftrack-macro-expansion flag that I have added to enable this feature.
Now that the memory consumption has been reduce to something that is
more acceptable, I guess this goal is closer now.  However, I have
kept the flag for now as I believe there is value in keeping it to
ease testing and performance comparison measures on real life
examples.  When I am comfortable with those aspects, I'll happily
remove that flag, e.g, after a period of enabling it by default.

Please find below a summary of the patches of this set as well as the
details of the memory consumption gain I measured.

  Linemap infrastructure for virtual locations
  Generate virtual locations for tokens
  Emit macro expansion related diagnostics
  Support -fdebug-cpp option
  Add line map statistics to -fmem-report output
  Kill pedantic warnings on system headers macros
  Reduce memory waste due to non-power-of-2 allocs

 gcc/Makefile.in                                 |    2 +-
 gcc/ada/gcc-interface/trans.c                   |   10 +-
 gcc/c-decl.c                                    |   17 +-
 gcc/c-family/c-lex.c                            |   10 +-
 gcc/c-family/c-opts.c                           |   17 +
 gcc/c-family/c-pch.c                            |    2 +-
 gcc/c-family/c-ppoutput.c                       |   98 ++-
 gcc/c-family/c.opt                              |   12 +
 gcc/c-parser.c                                  |   12 +-
 gcc/c-tree.h                                    |    2 +-
 gcc/cp/error.c                                  |    5 +-
 gcc/diagnostic.c                                |   25 +-
 gcc/diagnostic.h                                |    2 +-
 gcc/doc/cppopts.texi                            |   29 +
 gcc/doc/invoke.texi                             |    6 +-
 gcc/fortran/cpp.c                               |   22 +-
 gcc/input.c                                     |  105 ++-
 gcc/input.h                                     |   22 +-
 gcc/java/jcf-parse.c                            |    2 +-
 gcc/testsuite/g++.dg/cpp0x/initlist15.C         |    1 +
 gcc/testsuite/g++.old-deja/g++.robertl/eb43.C   |    4 +
 gcc/testsuite/g++.old-deja/g++.robertl/eb79.C   |    4 +
 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-1.c |   21 +
 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-2.c |   21 +
 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-3.c |   14 +
 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-4.c |   14 +
 gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-1.c  |   32 +
 gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-2.c  |   34 +
 gcc/testsuite/gcc.dg/cpp/syshdr3.c              |   16 +
 gcc/testsuite/gcc.dg/cpp/syshdr3.h              |    7 +
 gcc/testsuite/gcc.dg/nofixed-point-2.c          |    6 +-
 gcc/testsuite/gcc.target/i386/sse-vect-types.c  |    6 +
 gcc/toplev.c                                    |    4 +
 gcc/tree-diagnostic.c                           |  181 +++-
 gcc/tree-diagnostic.h                           |    3 +-
 libcpp/directives-only.c                        |    7 +-
 libcpp/directives.c                             |   21 +-
 libcpp/errors.c                                 |   21 +-
 libcpp/expr.c                                   |  176 ++--
 libcpp/files.c                                  |   24 +-
 libcpp/include/cpp-id-data.h                    |    6 +
 libcpp/include/cpplib.h                         |   15 +-
 libcpp/include/line-map.h                       |  827 +++++++++++++--
 libcpp/init.c                                   |    5 +-
 libcpp/internal.h                               |   60 +-
 libcpp/lex.c                                    |  107 ++-
 libcpp/line-map.c                               |  831 ++++++++++++--
 libcpp/macro.c                                  | 1350 ++++++++++++++++++++---
 libcpp/traditional.c                            |    7 +-
 49 files changed, 3656 insertions(+), 569 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-1.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-2.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-3.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/macro-exp-tracking-4.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-1.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/pragma-diagnostic-2.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/syshdr3.c
 create mode 100644 gcc/testsuite/gcc.dg/cpp/syshdr3.h

Below are the details of the memory consumption I have measured with
both the previous and current versions of this patch set.

[ For abbreviations, there is a legend at the end of the document ]

Memory peak, and GC memory UAEOC increase incurred by macro tokens
location tracking, before the work:

			|pre-processing only	|full compilation|
------------------------|-----------------------|----------------|
% memory peak increase	|	19.9%'		|      8%	 |
% GC memory UAEOC	|	283%		|      9.82%	 |


Memory peak increase incurred by macro tokens location tracking,
after the work:

			|pre-processing only	|full compilation|
------------------------|-----------------------|----------------|
% memory peak increase	|	19.2%'		|      0.6%	 |
% GC memory UAEOC  	|	250.5%		|      4.9%	 |

Number of expanded macros:                     96524
Average number of tokens per macro expansion:  21


                        |C0-p-n |C0-p-t |C0-f-n |C0-f-t |C1-p-n |C1-p-t |C1-f-n |C1-f-t |
----------------------------------------------------------------------------------------
maps allocated		|310K	|21M	|310K	|21M	|255K	|20M	|255K	|20M	|
maps used		|213K	|20M	|213K	|20M	|213K	|20M	|213K	|20M	|
macro maps used		|0	|19M	|0	|19M	|0	|19M	|0	|19M	|
map allocation waste	|374K	|10.7M	|374	|10.7M	|64	|4.35M	|64	|4.35M	|
GC memory used at EOC	|9.9M	|38M	|285M	|313M	|9.7M	|34M	|285M	|299M	|
G.M.U.A.O.E overhead	|140K	|396K	|4M	|4.3M	|140K	|386K	|4M	|4M	|
total GC mem allocation	|11M	|47M	|617M	|653.7M	|10M	|36M	|616.5M	|642.6M	|
total GC mem overhead	|1M	|11M	|54M	|64.31M	|919K	|5.2M	|53.5M	|58M	|
peak memory usage	|166M	|199M	|658M	|711MB	|158,3M	|188.7M	|658M	|662,4M	|


Legend:
=======

C0-p-n: Code base 0 (before this work), for pre-processing only, without
        macro tokens location tracking activated.

C0-p-t: Code base 0 (before this work), for pre-processing only, with
        macro tokens location tracking activated.

C0-f-n: Code base 0 (before this work), for full compilation, without
        macro tokens location tracking activated.

C0-f-t: Code base 0 (before this work), for full compilation, with
        macro tokens location tracking activated.

C1-p-n: Code base 1 (after this work), for pre-processing only,
        without macro tokens location tracking activated.

C1-p-t: Code base 1 (after this work), for pre-processing only, with
        macro tokens location tracking activated.

C1-f-n: Code base 1 (after this work), for full compilation, without
        macro tokens location tracking activated.

C1-f-t: Code base 1 (after this work), for full compilation, with
        macro tokens location tracking activated.

EOC   	      : End of compilation.
G.M.U.A.O.E   : GC memory still used at end of compilation.
UAEOC	      : Used at end of compilation