Re: warning: pasting would not give a valid preprocessing token

Re: warning: pasting would not give a valid preprocessing token

[Neil Booth]

[There's no point cc-ing Zack; his mail address is no more]

Byron,

I hope this fixes it for you.  I'm bootstrapping now, and will then
run the testsuite.

This area is a bit ugly and in bad need of a cleanup.  I need to
extract more flexibility from the lexer first, though.  The problem
was in the way we handle macro arguments: each argument is a list of
pointers to the tokens making up the argument.  These tend to point to
the original tokens (in this case, the comma in the macro definition
in the hash table).  That token has a PASTE_LEFT flag to say it's on
the left hand side of a ##.  The problem is that after processing the
paste, and correctly not emitting a warning about invalid pastes, we
maintained a pointer to the same comma for the nested macro
replacements (since it is part of an argument itself).  Unforunately,
of course, that comma still has its paste flag set, so the paste
mechanism gets re-invoked in later unintended places; hence the
(repeated) warnings.

This is a nice testcase for quite a deep nested macro bug.  I
simplified your case to the one below, which I'll commit along with
the fix.  If you have a chance, please give it a whirl.

It also fixes a bug in ON_REST_ARG - the macro was testing the flags
of the wrong thing (easy to do since #defines are typeless).  Any
relief the original fix brought might therefore be undone; I'd be
interested in knowing if the warnings it "cured" come back with this
patch.

OK to commit?

Neil.

	* cpplex.c (ON_REST_ARG): Correct the test.
	(maybe_paste_with_next): Duplicate a token that fail pasting,
	and clear its PASTE_LEFT flag, so that nested pasting attempts
	do not occur.
	* gcc.dg/cpp/paste10.c: Testcase.

Index: cpplex.c
===================================================================
RCS file: /cvs/gcc/egcs/gcc/cpplex.c,v
retrieving revision 1.95
diff -u -p -r1.95 cpplex.c
--- cpplex.c	2000/09/12 03:42:29	1.95
+++ cpplex.c	2000/09/14 22:28:32
@@ -201,9 +201,9 @@ TOKEN_LEN (token)
 #define IS_ARG_CONTEXT(c) ((c)->flags & CONTEXT_ARG)
 #define CURRENT_CONTEXT(pfile) ((pfile)->contexts + (pfile)->cur_context)
 #define ON_REST_ARG(c) \
- (((c)->flags & VAR_ARGS) \
-  && ((c)-1)->u.list->tokens[((c)-1)->posn - 1].val.aux \
-      == (unsigned int) (((c)-1)->u.list->paramc - 1))
+ (((c)->u.list->flags & VAR_ARGS) \
+  && (c)->u.list->tokens[(c)->posn - 1].val.aux \
+      == (unsigned int) ((c)->u.list->paramc - 1))
 
 #define ASSIGN_FLAGS_AND_POS(d, s) \
   do {(d)->flags = (s)->flags & (PREV_WHITE | BOL | PASTE_LEFT); \
@@ -2787,14 +2787,17 @@ maybe_paste_with_next (pfile, token)
 		     the special extended semantics (see above).  */
 		  if (token->type == CPP_COMMA
 		      && IS_ARG_CONTEXT (CURRENT_CONTEXT (pfile))
-		      && ON_REST_ARG (CURRENT_CONTEXT (pfile)))
+		      && ON_REST_ARG (CURRENT_CONTEXT (pfile) - 1))
 		    /* no warning */;
 		  else
 		    cpp_warning (pfile,
 			"pasting would not give a valid preprocessing token");
 		}
 	      _cpp_push_token (pfile, second);
-	      return token;
+	      /* A short term hack to safely clear the PASTE_LEFT flag.  */
+	      pasted = duplicate_token (pfile, token);
+	      pasted->flags &= ~PASTE_LEFT;
+	      return pasted;
 	    }
 
 	  if (type == CPP_NAME || type == CPP_NUMBER)
Index: cmdlne-C.c
===================================================================
RCS file: /cvs/gcc/egcs/gcc/testsuite/gcc.dg/cpp/cmdlne-C.c,v
retrieving revision 1.2
diff -u -p -r1.2 cmdlne-C.c
--- cmdlne-C.c	2000/07/11 13:51:39	1.2
+++ cmdlne-C.c	2000/09/14 22:47:29
@@ -10,7 +10,7 @@
    the beginning of a directive turns it into a non-directive.  */
 
 #define simple no comments
-#/**/define bad_directive		/* { dg-error "invalid" } */
+
 #define/**/obj_like/**/(some)/**/thing/**/
 #define fun_like(/**/x/**/,/**/y/**/)/**/
 /**/#define not_a_macro
Index: paste10.c
===================================================================
RCS file: paste10.c
diff -N paste10.c
--- /dev/null	Tue May  5 13:32:27 1998
+++ paste10.c	Thu Sep 14 15:47:29 2000
@@ -0,0 +1,14 @@
+/* Copyright (C) 2000 Free Software Foundation, Inc.  */
+
+/* { dg-do preprocess } */
+
+/* This testcase used to produce a bogus "invalid paste" warning, owing
+   to not clearing a PASTE_LEFT flag.  */
+
+#define strcpy(src) __strcpy_small (src)
+
+#define __strcpy_small(src) src
+
+#define tprintf(format, args...) sprintf(format, ## args)
+
+strcpy(tprintf("<%s>", test))

Re: warning: pasting would not give a valid preprocessing token

[Neil Booth]

Please ignore the unintended diff to cmdline-C.c; that's in my local
tree and completely unrelated.

Neil.

Re: warning: pasting would not give a valid preprocessing token

[Jeffrey A Law]

  In message < 20000914235302.A9969@daikokuya.demon.co.uk >you write:
  > [There's no point cc-ing Zack; his mail address is no more]
  > 
  > Byron,
  > 
  > I hope this fixes it for you.  I'm bootstrapping now, and will then
  > run the testsuite.
  > 
  > This area is a bit ugly and in bad need of a cleanup.  I need to
  > extract more flexibility from the lexer first, though.  The problem
  > was in the way we handle macro arguments: each argument is a list of
  > pointers to the tokens making up the argument.  These tend to point to
  > the original tokens (in this case, the comma in the macro definition
  > in the hash table).  That token has a PASTE_LEFT flag to say it's on
  > the left hand side of a ##.  The problem is that after processing the
  > paste, and correctly not emitting a warning about invalid pastes, we
  > maintained a pointer to the same comma for the nested macro
  > replacements (since it is part of an argument itself).  Unforunately,
  > of course, that comma still has its paste flag set, so the paste
  > mechanism gets re-invoked in later unintended places; hence the
  > (repeated) warnings.
  > 
  > This is a nice testcase for quite a deep nested macro bug.  I
  > simplified your case to the one below, which I'll commit along with
  > the fix.  If you have a chance, please give it a whirl.
  > 
  > It also fixes a bug in ON_REST_ARG - the macro was testing the flags
  > of the wrong thing (easy to do since #defines are typeless).  Any
  > relief the original fix brought might therefore be undone; I'd be
  > interested in knowing if the warnings it "cured" come back with this
  > patch.
  > 
  > OK to commit?
  > 
  > Neil.
  > 
  > 	* cpplex.c (ON_REST_ARG): Correct the test.
  > 	(maybe_paste_with_next): Duplicate a token that fail pasting,
  > 	and clear its PASTE_LEFT flag, so that nested pasting attempts
  > 	do not occur.
  > 	* gcc.dg/cpp/paste10.c: Testcase.
Assuming the bootstrap & test passes, this is fine to install.

jeff

Re: warning: pasting would not give a valid preprocessing token

[Byron Stanoszek]

On Thu, 14 Sep 2000, Neil Booth wrote:

> Byron,
> 
> I hope this fixes it for you.  I'm bootstrapping now, and will then
> run the testsuite.

This works like a charm. Thanks for the speedy bugfix!

 -Byron

> 	* cpplex.c (ON_REST_ARG): Correct the test.
> 	(maybe_paste_with_next): Duplicate a token that fail pasting,
> 	and clear its PASTE_LEFT flag, so that nested pasting attempts
> 	do not occur.
> 	* gcc.dg/cpp/paste10.c: Testcase.

-- 
Byron Stanoszek                         Ph: (330) 644-3059
Systems Programmer                      Fax: (330) 644-8110
Commercial Timesharing Inc.             Email: bstanoszek@comtime.com

warning: pasting would not give a valid preprocessing token

[Byron Stanoszek]

I've seen this warning in many of my .c files when compiling with versions of
2.96 from mid-july and newer.. has anyone else seen this before, know what it
is, and how to fix it?

warning: pasting would not give a valid preprocessing token

-- 
Byron Stanoszek                         Ph: (330) 644-3059
Systems Programmer                      Fax: (330) 644-8110
Commercial Timesharing Inc.             Email: bstanoszek@comtime.com

Re: warning: pasting would not give a valid preprocessing token

[Geoff Keating]

Byron Stanoszek <gandalf@winds.org> writes:

> I've seen this warning in many of my .c files when compiling with versions of
> 2.96 from mid-july and newer.. has anyone else seen this before, know what it
> is, and how to fix it?
> 
> warning: pasting would not give a valid preprocessing token

It means you're writing something like

#define paste(a, b) a##b
paste(*,foo)

because '*foo' is not a token, the tokens you want are '*' and 'foo'.
The solution is to replace the ## with a space.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: warning: pasting would not give a valid preprocessing token

[Byron Stanoszek]

On 11 Aug 2000, Geoff Keating wrote:

> Byron Stanoszek <gandalf@winds.org> writes:
> 
> > I've seen this warning in many of my .c files when compiling with versions of
> > 2.96 from mid-july and newer.. has anyone else seen this before, know what it
> > is, and how to fix it?
> > 
> > warning: pasting would not give a valid preprocessing token
> 
> It means you're writing something like
> 
> #define paste(a, b) a##b
> paste(*,foo)
> 
> because '*foo' is not a token, the tokens you want are '*' and 'foo'.
> The solution is to replace the ## with a space.

Strange, but true. I see the problem now. I have a #define like this:

/* tprintf(): prints to a temporary variable 16k in size */
extern char global_buff[16384];
#define tprintf(format, args...) (sprintf(global_buff, format, ## args), \
                                  (char *)global_buff)

But in my .c file, I use string concatenation like this:

  notify(player, tprintf("Your knowledge of %s is too low to speak "
         "it fluently.", language[ptr->num].name));

Can you tell me what the best way there is to get around that?

Also, this brings up another question with variable argument macros.

My intent is to use tprintf with 0 or more `args'. However, this is only
possible by using the ## operator in front of args. If that ## is removed,
then it only allows 1 or more arguments. Here is an example:

with ##:

tprintf("Foo\n");  =>  sprintf(global_buff, "Foo");

without ##:

tprintf("Foo\n");  =>  sprintf(global_buff, "Foo", );

(which results in a nice GCC error).


Any way to [correctly] achieve the zero-or-more arguments there without
munging 'format' (which is a mandatory part of that macro).

Thanks.

-- 
Byron Stanoszek                         Ph: (330) 644-3059
Systems Programmer                      Fax: (330) 644-8110
Commercial Timesharing Inc.             Email: bstanoszek@comtime.com

Re: warning: pasting would not give a valid preprocessing token

[Zack Weinberg]

On Fri, Aug 11, 2000 at 11:58:11PM -0400, Byron Stanoszek wrote:
> On 11 Aug 2000, Geoff Keating wrote:
> 
> > Byron Stanoszek <gandalf@winds.org> writes:
> > 
> > > I've seen this warning in many of my .c files when compiling
> > > with versions of 2.96 from mid-july and newer.. has anyone else
> > > seen this before, know what it is, and how to fix it?
> > > 
> > > warning: pasting would not give a valid preprocessing token
> > 
> > It means you're writing something like
> > 
> > #define paste(a, b) a##b
> > paste(*,foo)
> > 
> > because '*foo' is not a token, the tokens you want are '*' and 'foo'.
> > The solution is to replace the ## with a space.
> 
> Strange, but true. I see the problem now. I have a #define like this:
> 
> /* tprintf(): prints to a temporary variable 16k in size */
> extern char global_buff[16384];
> #define tprintf(format, args...) (sprintf(global_buff, format, ## args), \
>                                   (char *)global_buff)
> 
> But in my .c file, I use string concatenation like this:
> 
>   notify(player, tprintf("Your knowledge of %s is too low to speak "
>          "it fluently.", language[ptr->num].name));
> 
> Can you tell me what the best way there is to get around that?

Very recent versions of gcc should not give that warning for the
specific case of , ## <rest argument>.

You could also write your tprintf() macro like this:

#define tprintf(args...) (snprintf(buff, BUFSIZ, args), (char *)buff)

you'll still get an error for tprintf(/* nothing */) but
tprintf("string") will be fine.

zw

Re: warning: pasting would not give a valid preprocessing token

[Byron Stanoszek]

On Fri, 11 Aug 2000, Zack Weinberg wrote:

> Very recent versions of gcc should not give that warning for the
> specific case of , ## <rest argument>.

Well, I just tried it on the August 11 snapshot. So unless you put in a patch
over the night, it still issues those warnings.

> You could also write your tprintf() macro like this:
> 
> #define tprintf(args...) (snprintf(buff, BUFSIZ, args), (char *)buff)
> 
> you'll still get an error for tprintf(/* nothing */) but
> tprintf("string") will be fine.

I could do that, and I thought about it; (sprintf(buff, args)) instead of
snprintf because I want to remain compatible on systems that do not have
glibc. But I was hoping to make the code easier to read by keeping that
'format' variable in there. Again, the only time I get that warning is when
there are two tokens with string concatenation ("abc" "def", args...). GCC
didn't complain about this before, so I was wondering if it could get fixed.

Thanks.

-- 
Byron Stanoszek                         Ph: (330) 644-3059
Systems Programmer                      Fax: (330) 644-8110
Commercial Timesharing Inc.             Email: bstanoszek@comtime.com

Re: warning: pasting would not give a valid preprocessing token

[Zack Weinberg]

On Sat, Aug 12, 2000 at 11:18:56AM -0400, Byron Stanoszek wrote:
> On Fri, 11 Aug 2000, Zack Weinberg wrote:
> 
> > Very recent versions of gcc should not give that warning for the
> > specific case of , ## <rest argument>.
> 
> Well, I just tried it on the August 11 snapshot. So unless you put in a patch
> over the night, it still issues those warnings.

A month later (apologies)... I believe this has now been fixed.
Please update and let me know if you still have problems.  You want
this change:

2000-09-04  Jakub Jelinek  <jakub@redhat.com>

        * cpplex.c (ON_REST_ARG): Check VAR_ARGS flag of current context,
        use posn - 1 to index into tokens array.
        (maybe_paste_with_next): Adjust caller.

It may or may not be in the 2000-09-04 snapshot, but will be in the
-11 snapshot when that comes out.

zw

Re: warning: pasting would not give a valid preprocessing token

[Byron Stanoszek]

On Wed, 6 Sep 2000, Zack Weinberg wrote:

> On Sat, Aug 12, 2000 at 11:18:56AM -0400, Byron Stanoszek wrote:
> > On Fri, 11 Aug 2000, Zack Weinberg wrote:
> > 
> > > Very recent versions of gcc should not give that warning for the
> > > specific case of , ## <rest argument>.
> > 
> > Well, I just tried it on the August 11 snapshot. So unless you put in a patch
> > over the night, it still issues those warnings.
> 
> A month later (apologies)... I believe this has now been fixed.
> Please update and let me know if you still have problems.  You want
> this change:
> 
> 2000-09-04  Jakub Jelinek  <jakub@redhat.com>
> 
>         * cpplex.c (ON_REST_ARG): Check VAR_ARGS flag of current context,
>         use posn - 1 to index into tokens array.
>         (maybe_paste_with_next): Adjust caller.

Close, but no cigar. Actually, it hushed all of the warnings in every single .c
file (about 45 of them individually), except for this one line. I took apart
the .c file and made a small testcase that you can compile as described below.

I do confirm, however, that the other warnings that were caused by ANSI String
Concatenation are gone now.

This is using the CVS 09/06/2000 version:

---
winds:~/mare/src> cat test.c
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

/* tprintf(): prints to a temporary variable 16k in size */
extern char global_buff[16384];
#define tprintf(format, args...) (sprintf(global_buff, format, ## args), \
                                  (char *)global_buff)

extern char *wtime(unsigned long, int);
extern char *center(char *, int, char);

void test()
{
  char buf[256];
  strcpy(buf, center(tprintf("<%s>", strchr(wtime(-1L, 1), ' ')+1), 63, ' '));
}

winds:~/mare/src> gcc -Wall -O3 test.c -c
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
test.c:16:77: warning: pasting would not give a valid preprocessing token
winds:~/mare/src> 

That's what my strcpy() function looked like.
Narrowing down the test case, this code also causes the same warnings:

  char buf[256], *test="Test";
  strcpy(buf, tprintf("<%s>", test));

However both of these do not give the warnings:

  strcpy(buf, tprintf("<>"));

  tprintf("<%s>", test);

The warnings still occur if I use the form ({ sprintf(..); global_buff; })
in the #define.

Hope this helps in tracking down the problem.

-- 
Byron Stanoszek                         Ph: (330) 644-3059
Systems Programmer                      Fax: (330) 644-8110
Commercial Timesharing Inc.             Email: bstanoszek@comtime.com

Re: warning: pasting would not give a valid preprocessing token

[Zack Weinberg]

On Thu, Sep 07, 2000 at 03:04:13PM -0400, Byron Stanoszek wrote:
> 
> Close, but no cigar. Actually, it hushed all of the warnings in
> every single .c file (about 45 of them individually), except for
> this one line. I took apart the .c file and made a small testcase
> that you can compile as described below.

...
> #include <stdlib.h>
> #include <stdio.h>
> #include <string.h>
> 
> /* tprintf(): prints to a temporary variable 16k in size */
> extern char global_buff[16384];
> #define tprintf(format, args...) (sprintf(global_buff, format, ## args), \
>                                   (char *)global_buff)
> 
> extern char *wtime(unsigned long, int);
> extern char *center(char *, int, char);
> 
> void test()
> {
>   char buf[256];
>   strcpy(buf, center(tprintf("<%s>", strchr(wtime(-1L, 1), ' ')+1), 63, ' '));
> }
> 
> winds:~/mare/src> gcc -Wall -O3 test.c -c
> test.c:16:77: warning: pasting would not give a valid preprocessing token

This is being caused by interference from glibc's damn string macros.
I don't have time to investigate any deeper now or in the near future,
sorry.  Neil, if you get a chance could you look at this?

zw