Re: preprocessor/2706: #defines expanded when -fpreprocessed given

Re: preprocessor/2706: #defines expanded when -fpreprocessed given

[Zack Weinberg]

The following reply was made to PR preprocessor/2706; it has been noted by GNATS.

From: "Zack Weinberg" <zackw@Stanford.EDU>
To: Nathan Sidwell <nathan@codesourcery.com>
Cc: Neil Booth <neil@daikokuya.demon.co.uk>, gcc-gnats@gcc.gnu.org
Subject: Re: preprocessor/2706: #defines expanded when -fpreprocessed given
Date: Wed, 2 May 2001 17:00:33 -0700

 On Wed, May 02, 2001 at 05:00:15PM +0100, Nathan Sidwell wrote:
 > 
 > > It might be better to emit a diagnostic from cpplib, along the lines
 > > of "'#define' not honored in already-preprocessed source", and discard
 > > the pseudo-directive.  The diagnostics we give right now are rather
 > > unhelpful.  Thoughts?
 > That was the kind of error I expected. What triggered this was cc1plus
 > silently swallowing a #include.
 
 I looked into providing this.  In order for you to get the nice error
 both with and without -save-temps, it needs to come from c-lex.c.
 Unfortunately, cpplib does not export enough information to do this
 reliably.  I need to know when a token is the first one on a line, and
 I need to be able to discard a line, both from the context of cpplib's
 user.  cpplib throws away the information about tokens being first on
 a line deep inside its inner loop.  The routine to skip the rest of a
 line is private; if I simply export it it doesn't work properly
 (caller gets an EOF as the next token processed).
 
 I propose we live with the unfriendly error messages for now, until
 cpplib can be adjusted to give c-lex what it needs.  I'm testing a
 slightly-revised patch right now on the 3.0 branch.
 
 zw

Re: preprocessor/2706: #defines expanded when -fpreprocessed given

[Nathan Sidwell]

The following reply was made to PR preprocessor/2706; it has been noted by GNATS.

From: Nathan Sidwell <nathan@codesourcery.com>
To: Zack Weinberg <zackw@stanford.edu>
Cc: gcc-gnats@gcc.gnu.org
Subject: Re: preprocessor/2706: #defines expanded when -fpreprocessed given
Date: Wed, 02 May 2001 17:00:15 +0100

 Zack Weinberg wrote:
 
 > 
 > If you could write up some comprehensive test cases for this that'd be
 > greatly appreciated.  I'm bootstrapping this in mainline and will
 > apply it there; I'd like Neil Booth's reaction before it goes onto the
 > branch.
 thanks for looking at this, I'll make some tests
 
 > It might be better to emit a diagnostic from cpplib, along the lines
 > of "'#define' not honored in already-preprocessed source", and discard
 > the pseudo-directive.  The diagnostics we give right now are rather
 > unhelpful.  Thoughts?
 That was the kind of error I expected. What triggered this was cc1plus
 silently swallowing a #include.
 
 nathan
 -- 
 Dr Nathan Sidwell   ::   http://www.codesourcery.com   ::   CodeSourcery LLC
          'But that's a lie.' - 'Yes it is. What's your point?'
 nathan@codesourcery.com : http://www.cs.bris.ac.uk/~nathan/ : nathan@acm.org

Re: preprocessor/2706: #defines expanded when -fpreprocessed given

[Zack Weinberg]

The following reply was made to PR preprocessor/2706; it has been noted by GNATS.

From: "Zack Weinberg" <zackw@stanford.edu>
To: nathan@codesourcery.com
Cc: gcc-gnats@gcc.gnu.org
Subject: Re: preprocessor/2706: #defines expanded when -fpreprocessed given
Date: Tue, 1 May 2001 10:21:33 -0700

 On Tue, May 01, 2001 at 01:38:19PM -0000, nathan@codesourcery.com wrote:
 > 
 > >Description:
 > the attached source is ill-formed, however it is accepted
 > as cpp appears to expand HASH to # and then process
 > the resultant #define. To make things more confusing 
 > when rereading the preprocessed source , the #defines are
 > again expanded. This is a 2.95 regression and will be
 > a real headache to anybody trying to track down obsucre
 > preprocessor related problems in their source!
 > 
 > There appear to be 3 bugs, but could all have the same
 > cause.
 > 1) a line is macro expanded and the result interpretted
 > for preprocessor directives
 > 2) no diagnostic is issued for finding something like
 > #define in a preprocessed source
 > 3) macros are expanded in preprocessed source.
 
 Ugh, is this ever a mess.
 
 First off, this only happens with -save-temps.  If you use the
 integrated preprocessor, 3.1 gives
 
 cpp.C:2: syntax error before '#' token
 cpp.C:2: parse error before string constant
 cpp.C:3: syntax error before '#' token
 
 which is not the ideal error message, but at least it rejects the
 code.
 
 With -save-temps, things get hairier.  See, at -g3 we want to
 communicate #defines to the compiler for possible inclusion in the
 debug information.  With the integrated preprocessor this happens via
 a callback routine.  With the external preprocessor, we have to
 communicate it in-band somehow.  Presently this is done by repeating
 the original #define in the intermediate file.  When we come back and
 reread the file, cpplib is supposed to process the #define and call
 the compiler's callback but not actually intern the macro.
 
 The first bug was that the code which looked for repeated #define in
 the intermediate file had no way of distinguishing a repeated
 directive from a fake directive resulting from expansion.  The second
 bug was that irrespective of whether directives were honored in
 preprocessed files, they were silently discarded from the token stream
 passed on to the compiler, so your 'HASH include "thing"' was
 invisible.  The third bug was that, yep, the check preventing any
 macro expansion in intermediate files had gotten lost.
 
 The second and third bugs are straightforward to fix.  The first is
 harder.  I don't see any solution which is better than a kluge, but a
 kluge should do in this context.  In the appended patch I require
 '#define' to appear starting in column 1 and with no space between the
 # and the directive name.  That suffices to prevent 'HASH <directive>'
 from being misinterpreted.  You can be cleverer, though:
 
 #define DEF #define
 DEF foo bar
 int foo(void) { return 23; }
 // compile this with -save-temps.  Did you get an error?
 // Check the .s file, did the compiler generate a function named 'bar'?
 
 So I also forcibly insert a space between # and define in that case.
 
 If you could write up some comprehensive test cases for this that'd be
 greatly appreciated.  I'm bootstrapping this in mainline and will
 apply it there; I'd like Neil Booth's reaction before it goes onto the
 branch.
 
 It might be better to emit a diagnostic from cpplib, along the lines
 of "'#define' not honored in already-preprocessed source", and discard
 the pseudo-directive.  The diagnostics we give right now are rather
 unhelpful.  Thoughts?
 
 zw
 
 	* cpplib.c (_cpp_handle_directive): In preprocessed source,
 	process IN_I directives only if the # is in column 1 and the
 	directive name starts in column 2.  Preserve things which look
 	like directives, but are not honored, in the output stream
 	from preprocessed source.
 	* cppmacro.c (cpp_get_token): Don't ever expand macros when
 	rescanning preprocessed source.
 	* cppmain.c (scan_buffer): Insert a space between a '#' and a
 	directive name, so that this cannot be confused with a directive
 	repeated in the output by a callback routine.
 
 ===================================================================
 Index: cpplib.c
 --- cpplib.c	2001/04/11 19:28:23	1.249
 +++ cpplib.c	2001/05/01 17:15:34
 @@ -311,7 +311,32 @@ _cpp_handle_directive (pfile, indented)
  
        /* If we are rescanning preprocessed input, only directives tagged
  	 with IN_I are honored, and the warnings below are suppressed.  */
 -      if (! CPP_OPTION (pfile, preprocessed) || dir->flags & IN_I)
 +      if (CPP_OPTION (pfile, preprocessed))
 +	{
 +	  /* Kluge alert.  In order to be sure that code like this
 +	     #define HASH #
 +	     HASH define foo bar
 +	     does not cause '#define foo bar' to get executed when
 +	     compiled with -save-temps, we recognize directives in
 +	     -fpreprocessed mode only if the # is in column 1 and the
 +	     directive name starts in column 2.  This output can only
 +	     be generated by the directive callbacks in cppmain.c (see
 +	     also the special case in scan_buffer).  */
 +	  if (dir->flags & IN_I && !indented && !(dname.flags & PREV_WHITE))
 +	    (*dir->handler) (pfile);
 +	  /* That check misses '# 123' linemarkers.  Let them through too.  */
 +	  else if (dname.type == CPP_NUMBER)
 +	    (*dir->handler) (pfile);
 +	  else
 +	    {
 +	      /* We don't want to process this directive.  Put back the
 +		 tokens so caller will see them (and issue an error,
 +		 probably).  */
 +	      _cpp_push_token (pfile, &dname, &pfile->directive_pos);
 +	      skip = 0;
 +	    }
 +	}
 +      else
  	{
  	  /* Traditionally, a directive is ignored unless its # is in
  	     column 1.  Therefore in code intended to work with K+R
 ===================================================================
 Index: cppmacro.c
 --- cppmacro.c	2001/03/27 21:04:55	1.50
 +++ cppmacro.c	2001/05/01 17:15:35
 @@ -947,7 +947,8 @@ cpp_get_token (pfile, token)
        /* Handle macros and the _Pragma operator.  */
        if (token->val.node->type == NT_MACRO
  	  && !pfile->state.prevent_expansion
 -	  && !(token->flags & NO_EXPAND))
 +	  && !(token->flags & NO_EXPAND)
 +	  && !CPP_OPTION (pfile, preprocessed))
  	{
  	  cpp_hashnode *node = token->val.node;
  
 ===================================================================
 Index: cppmain.c
 --- cppmain.c	2001/03/04 12:02:02	1.63
 +++ cppmain.c	2001/05/01 17:21:16
 @@ -238,6 +238,14 @@ scan_buffer (pfile)
  		       == AVOID_LPASTE
  		   && cpp_avoid_paste (pfile, &tokens[1 - index], token))
  	    token->flags |= PREV_WHITE;
 +	  /* Special case '# <directive name>': insert a space between
 +	     the # and the token.  This will prevent it from being
 +	     treated as a directive when this code is re-
 +	     preprocessed.
 +	     XXX Should do this only at the beginning of a line, but how?  */
 +	  else if (token->type == CPP_NAME && token->val.node->directive_index
 +		   && tokens[1 - index].type == CPP_HASH)
 +	    token->flags |= PREV_WHITE;
  
  	  cpp_output_token (token, print.outf);
  	  print.printed = 1;

preprocessor/2706: #defines expanded when -fpreprocessed given

[nathan]

>Number:         2706
>Category:       preprocessor
>Synopsis:       #defines expanded when -fpreprocessed given
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    unassigned
>State:          open
>Class:          accepts-illegal
>Submitter-Id:   net
>Arrival-Date:   Tue May 01 06:46:01 PDT 2001
>Closed-Date:
>Last-Modified:
>Originator:     Nathan
>Release:        1May2001 CVS 3.1 and 3.0
>Organization:
>Environment:

>Description:
the attached source is ill-formed, however it is accepted
as cpp appears to expand HASH to # and then process
the resultant #define. To make things more confusing 
when rereading the preprocessed source , the #defines are
again expanded. This is a 2.95 regression and will be
a real headache to anybody trying to track down obsucre
preprocessor related problems in their source!

There appear to be 3 bugs, but could all have the same
cause.
1) a line is macro expanded and the result interpretted
for preprocessor directives
2) no diagnostic is issued for finding something like
#define in a preprocessed source
3) macros are expanded in preprocessed source.

current tree gives, 
nathan@uha:275>./g++ -B ./ -save-temps current/cpp.C -c
nathan@uha:276>more cpp.ii                             
# 2 "current/cpp.C"
# define X "string"
# include "thing"

char const *foo ()
{
    return X;
}
nathan@uha:277>./cc1plus -fpreprocessed cpp.ii -quiet  

2.95 says
nathan@uha:278>g++ -save-temps current/cpp.C -c
current/cpp.C:3: undefined or invalid # directive
current/cpp.C: In function `const char * foo()':
current/cpp.C:7: `X' undeclared (first use this function)
current/cpp.C:7: (Each undeclared identifier is reported only once
current/cpp.C:7: for each function it appears in.)
current/cpp.C:8: warning: control reaches end of non-void function `foo()'
nathan@uha:279>more cpp.ii
# 1 "current/cpp.C"

#  define X "string"
#  include "thing"

char const *foo ()
{
    return X;
}
nathan@uha:280>g++ cpp.ii -c 
current/cpp.C:3: undefined or invalid # directive
current/cpp.C: In function `const char * foo()':
current/cpp.C:7: `X' undeclared (first use this function)
current/cpp.C:7: (Each undeclared identifier is reported only once
current/cpp.C:7: for each function it appears in.)
current/cpp.C:8: warning: control reaches end of non-void function `foo()'

>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
----gnatsweb-attachment----
Content-Type: application/octet-stream; name="cpp.C"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="cpp.C"

I2RlZmluZSBIQVNIICMKSEFTSCBkZWZpbmUgWCAic3RyaW5nIgpIQVNIIGluY2x1ZGUgInRoaW5n
IgoKY2hhciBjb25zdCAqZm9vICgpCnsKICAgIHJldHVybiBYOwp9Cg==