macros with traditional cpp

macros with traditional cpp

[Philip Blundell]

The preprocessor used to accept code like this both with and without 
-traditional:

	#define foo(x...) x
	foo(one,two)

Now, -traditional causes an error like:

t.c:1: badly punctuated parameter list in #define

Is this intentional?

Thanks

p.

Re: macros with traditional cpp

[Jamie Lokier]

Philip Blundell wrote:
> The preprocessor used to accept code like this both with and without 
> -traditional:
> 
> 	#define foo(x...) x
> 	foo(one,two)
> 
> Now, -traditional causes an error like:
> 
> t.c:1: badly punctuated parameter list in #define
> 
> Is this intentional?

Ah yes, that is one form of a varargs bug I've been meaning to report.
I've got macros which work fine with GCC 2.95, but don't work with _or_
without -traditional now.  (Sorry, I'll try to provide an example
tomorrow).

It's a bummer -- I have to use an older preprocessor because I've a file
full of those kinds of macros and I can't think of a way to rewrite them
to work with the current preprocessor, or with C99 varargs.

I do hope the old behaviour is restored, and furthermore, the GNU
extension form isn't deprecated.  I simply can't think of a way to write
many of my varargs macros in C99 syntax.

-- Jamie

Re: macros with traditional cpp

[Zack Weinberg]

On Sun, Sep 24, 2000 at 10:57:10PM +0200, Jamie Lokier wrote:
> Philip Blundell wrote:
> > The preprocessor used to accept code like this both with and without 
> > -traditional:
> > 
> > 	#define foo(x...) x
> > 	foo(one,two)
> > 
> > Now, -traditional causes an error like:
> > 
> > t.c:1: badly punctuated parameter list in #define
> > 
> > Is this intentional?

Yes.  The traditional preprocessor is intended to be a faithful
reimplementation of the original K+R C preprocessor, which did not
support varargs macros.  It would have *been* the original K+R C
preprocessor if I'd been able to get RMS and SCO to sign off on it.

Anyhow, code like the above simply can't be used with -traditional
anymore.  If you're using -traditional for assembly, try it without.
It'll probably work with only minor tweaks, like using C comments
instead of assembler comments.

> Ah yes, that is one form of a varargs bug I've been meaning to report.
> I've got macros which work fine with GCC 2.95, but don't work with _or_
> without -traditional now.  (Sorry, I'll try to provide an example
> tomorrow).

Please do provide an example.  I wouldn't be surprised to find genuine
bugs in the varargs macro code.

The new semantics are not that different; in most cases it will be possible
to convert macros.

zw

Varargs macros subtly broken

[Jamie Lokier]

Zack Weinberg wrote:
> > Ah yes, that is one form of a varargs bug I've been meaning to report.
> > I've got macros which work fine with GCC 2.95, but don't work with _or_
> > without -traditional now.  (Sorry, I'll try to provide an example
> > tomorrow).
> 
> Please do provide an example.  I wouldn't be surprised to find genuine
> bugs in the varargs macro code.
> 
> The new semantics are not that different; in most cases it will be possible
> to convert macros.

I see that now.  Attached are examples of macros that used to work but
are broken now.  Preprocessing fails with errors.

I've included examples using the old syntax, and also converted ones for
__VA_ARGS__ syntax.  Both exhibit the same problems with the current
preprocessor

Btw, I found this while trying to build our application using RedHat 7.0
Beta -- they are shipping GCC 2.96.2 which has the problem CPP.  Ew.

I tested today with the CVS snapshot -- same problems, but the warnings
about token pasting have been fixed.

-- Jamie

/* This file illustrates a few changes in preprocessor behaviour using
   varargs from EGCS 1.1.2 to GCC 2.96.2.

   Observe: gnu_count expansions differ.
            gnu_repeat is the same.
            gnu_index makes preprocessing fail.

   I've included some attempts at using the C99 syntax.
   I haven't bothered with c99_index.

   Observe: c99_count functionality can't quite do the same as gnu_count.
            c99_repeat is fine though.

   Bonus observations:

        - ## swallows the token in rest arg macros, even with
          __VA_ARGS__.  According to the CPP manual, this is an
          unreliable feature that is deprecated, and it's use is warned
          about.  Actually there is no warning.  As you can see from
          this code, that feature is essential, though only when the
          discarded token is ",", to get the correct results here.  Can
          we have a decision to not deprecate it at least in the comma
          case, provided that is compatible with ISO C99?  (Presumably
          ISO C99 doesn't define the behaviour of pasting "," before
          another token anyway -- leaving a loophole which GCC can
          exploit to discard it).

        - The output has spurious newlines, followed by a corrective "#
          LINE" directive.

   -- Jamie Lokier <jamie.lokier@cern.ch>, 25/Sep/2000

   Permission is granted to use, copy, modify and distribute this file
   freely for any purpose whatsoever.  This file is free software, and
   there's no warranty. */

/********** Definitions. ***********/

#define dup3(x)            x,x,x

/* Count elements in a list (0 to 10 max). */
#define gnu_count(y...)   _gnu_count1 ( , ##y)
#define _gnu_count1(y...) _gnu_count2 (y,10,9,8,7,6,5,4,3,2,1,0)
#define _gnu_count2(_,x0,x1,x2,x3,x4,x5,x6,x7,x8,x9,n,ys...) n

/* Tail of a list. */
#define gnu_tail(y...)    _gnu_tail (y)
#define _gnu_tail(x,y...) y

/* Repeat N times. */
#define gnu_repeat(n, x) gnu_tail (_gnu_repeat (n, x))
#define _gnu_repeat(n, x) _gnu_repeat_##n (x)
#define _gnu_repeat_0(x)
#define _gnu_repeat_1(x) ,x
#define _gnu_repeat_2(x) ,x,x
#define _gnu_repeat_3(x) ,x,x,x
#define _gnu_repeat_4(x) ,x,x,x,x
#define _gnu_repeat_5(x) ,x,x,x,x,x

#define _gnu_keep(xs...) xs
#define _gnu_discard(xs...)
#define _gnu_split_r(n,xs...) _gnu_split_rd (n,_gnu_keep,_gnu_discard xs)
#define _gnu_split_d(n,xs...) _gnu_split_rd (n,_gnu_discard,_gnu_keep xs)
#define _gnu_split_rd(n,xs...) _gnu_split_##n (xs)
#define _gnu_split_0(a,b,xs...) a() b(xs)
#define _gnu_split_1(a,b,x0,xs...) a(x0) b(xs)
#define _gnu_split_2(a,b,x0,x1,xs...) a(x0,x1) b(xs)
#define _gnu_split_3(a,b,x0,x1,x2,xs...) a(x0,x1,x2) b(xs)
#define _gnu_split_4(a,b,x0,x1,x2,x3,xs...) a(x0,x1,x2,x3) b(xs)
#define _gnu_split_5(a,b,x0,x1,x2,x3,x4,xs...) a(x0,x1,x2,x3,x4) b(xs)

/* List manipulations.  Surprise: index zero is the rightmost element. */
#define gnu_take(n, xs...) \
  _gnu_split_d (_gnu_count1 ( , ## xs), _gnu_repeat (n, _gnu_error) , ## xs)
#define gnu_drop(n, xs...) \
  _gnu_split_d (n,,_gnu_split_r  (_gnu_count1 ( , ## xs), _gnu_repeat (n, _gnu_error) , ## xs))
#define gnu_index(pos, xs...) gnu_take (1, gnu_drop (pos , ## xs))

/* C99 __VA_ARGS__ versions */
#define c99_count(...)    _c99_count1 ( , ##__VA_ARGS__)/* If only ## worked.*/
#define _c99_count1(...)  _c99_count2 (__VA_ARGS__,10,9,8,7,6,5,4,3,2,1,0)
#define _c99_count2(_,x0,x1,x2,x3,x4,x5,x6,x7,x8,x9,n,...) n

#define c99_tail(...)     _c99_tail (__VA_ARGS__)
#define _c99_tail(x,...)  __VA_ARGS__

#define _c99_keep(...)    __VA_ARGS__
#define _c99_discard(...)
#define _c99_split_r(n,...) _c99_split_rd(n,_c99_keep,_c99_discard __VA_ARGS__)
#define _c99_split_d(n,...) _c99_split_rd(n,_c99_discard,_c99_keep __VA_ARGS__)
#define _c99_split_rd(n,...) _c99_split_##n (__VA_ARGS__)
#define _c99_split_0(a,b,...) a() b(__VA_ARGS__)
#define _c99_split_1(a,b,x0,...) a(x0) b(__VA_ARGS__)
#define _c99_split_2(a,b,x0,x1,...) a(x0,x1) b(__VA_ARGS__)
#define _c99_split_3(a,b,x0,x1,x2,...) a(x0,x1,x2) b(__VA_ARGS__)
#define _c99_split_4(a,b,x0,x1,x2,x3,...) a(x0,x1,x2,x3) b(__VA_ARGS__)
#define _c99_split_5(a,b,x0,x1,x2,x3,x4,...) a(x0,x1,x2,x3,x4) b(__VA_ARGS__)

/* List manipulations.  Surprise: index zero is the rightmost element. */
#define c99_take(n, ...) \
  _c99_split_d (_c99_count1 ( , ## __VA_ARGS__), _c99_repeat (n, _c99_error) , ## __VA_ARGS__)
#define c99_drop(n, ...) \
  _c99_split_d (n,,_c99_split_r  (_c99_count1 ( , ## __VA_ARGS__), _c99_repeat (n, _c99_error) , ## __VA_ARGS__))
#define c99_index(pos, ...) c99_take (1, c99_drop (pos , ## __VA_ARGS__))

/************** Expansions **************/

/* Correct answers are 0, 0, 1, 2, 10 (works with older GNU CPPs).
   Current CVS gives 0, 1, 1, 2, 10. */
_gnu_count1 ();
gnu_count ();
gnu_count (A);
gnu_count (,);
gnu_count (A, B, C, D, E, F, G, H, I, J);

/* These are fine. */
_gnu_tail (dup3 ("1"), dup3 ("2")); /* Should be "2", "2", "2". */
gnu_tail  (dup3 ("1"), dup3 ("2")); /* Should be "1", "1", "2", "2", "2". */

/* Correct answers are empty, "x" and "x", "x", "x". */
gnu_repeat (0, "x");
gnu_repeat (1, "x");
gnu_repeat (3, "x");

/* Correct answers are "e", "b", "a", empty.
   Current CVS barfs with `not enough arguments for macro "_gnu_split_N"'
   where N varies for each line. */
gnu_index (0, "a", "b", "c", "d", "e");
gnu_index (3, "a", "b", "c", "d", "e");
gnu_index (4, "a", "b", "c", "d", "e");
gnu_index (5, "a", "b", "c", "d", "e");

/************* C99 tests *************/

/* The answers I would like are 0, 0, 1, 2, 10 as for the non-C99 version.
   Current CVS gives 0, 1, 1, 2, 10. */
_c99_count1 ();
c99_count ();
c99_count (A);
c99_count (,);
c99_count (A, B, C, D, E, F, G, H, I, J);

/* These are fine. */
_c99_tail (dup3 ("1"), dup3 ("2")); /* Should be "2", "2", "2". */
c99_tail  (dup3 ("1"), dup3 ("2")); /* Should be "1", "1", "2", "2", "2". */

/* Correct answers are "e", "b", "a", empty.
   Current CVS barfs with `not enough arguments for macro "_c99_split_N"'
   where N varies for each line. */
c99_index (0, "a", "b", "c", "d", "e");
c99_index (3, "a", "b", "c", "d", "e");
c99_index (4, "a", "b", "c", "d", "e");
c99_index (5, "a", "b", "c", "d", "e");

Re: Varargs macros subtly broken

[Zack Weinberg]

On Mon, Sep 25, 2000 at 06:42:54PM +0200, Jamie Lokier wrote:
> Zack Weinberg wrote:
> > > Ah yes, that is one form of a varargs bug I've been meaning to report.
> > > I've got macros which work fine with GCC 2.95, but don't work with _or_
> > > without -traditional now.  (Sorry, I'll try to provide an example
> > > tomorrow).
> > 
> > Please do provide an example.  I wouldn't be surprised to find genuine
> > bugs in the varargs macro code.
> > 
> > The new semantics are not that different; in most cases it will be possible
> > to convert macros.
> 
> I see that now.  Attached are examples of macros that used to work but
> are broken now.  Preprocessing fails with errors.
> 
> I've included examples using the old syntax, and also converted ones for
> __VA_ARGS__ syntax.  Both exhibit the same problems with the current
> preprocessor

Let me go over the intended semantics.  There are three extensions in
GCC to C99 vararg macros.  First:

#define macro(a, b, ...)  a, b, __VA_ARGS__

means precisely the same as

#define macro(a, b, __VA_ARGS__...) a, b, __VA_ARGS__

and you may replace __VA_ARGS__ by any identifier:

#define macro(a, b, rest...) a, b, rest

In other words, __VA_ARGS__ is the default rest-arg name, and you can
use something more euphonious if you wish.  Note that this means that
__VA_ARGS__ is not special outside of vararg macros - contrary to C99
where it is always reserved.  Also, the other two extensions apply
whether or not you give the rest argument a name.

Second: given

#define macro(a, b, rest...) a, b, rest

then

macro(a, b)

is permitted by GCC, and has the same expansion as

macro(a, b, )

However, if -pedantic is on, a warning issues.

Third: if you write

#define macro(a, b, rest...) a, b, ## rest

and then use macro with only two arguments, then the comma after b is
deleted.  This does not happen if you write macro(a, b, ).  You can
only count on this behavior if the token immediately preceding the ##
is a comma.  Other immediate predecessors may or may not be deleted
depending on undocumented implementation details.

This is the biggest change from GCC 2.95.  Previous versions of the
preprocessor deleted the sequence of nonwhitespace characters
immediately before the ##, no matter what they were.  This version
only deletes one token, and only does so if you give no rest args at
all; an empty rest arg doesn't trigger deletion.  And you cannot rely
on the preceding token being deleted unless it is a comma.

Also, there's a new warning, which issues whenever the tokens on
either side of ## cannot be glued into one token.  This warning is on
by default, and may be suppressed by -Wno-paste.  A macro written to
use the third extension would normally trigger this warning.  However,
as a special case, the warning is suppressed for ", ## rest" where
rest is a rest argument.  If you put something else before the ## the
warning may or may not happen.

I do not have time to investigate your example in detail, but my
immediate reaction is that you're trying to do things outside of cpp's
intended scope.  If you want M4, you know where to find it.

zw

Re: Varargs macros subtly broken

[Jamie Lokier]

Zack Weinberg wrote:
> macro(a, b)
> 
> is permitted by GCC, and has the same expansion as
> 
> macro(a, b, )

Not quite, as you point out below.

> Third: if you write
> 
> #define macro(a, b, rest...) a, b, ## rest
> 
> and then use macro with only two arguments, then the comma after b is
> deleted.  This does not happen if you write macro(a, b, ).

Good, this is like older versions of GCC.

> You can only count on this behavior if the token immediately preceding
> the ## is a comma.  Other immediate predecessors may or may not be
> deleted depending on undocumented implementation details.

Cool.

For other tokens, perhaps C99 requires that the token preceding
##__VA_ARGS__ is not deleted, if it is possible to paste them?

> This is the biggest change from GCC 2.95.  Previous versions of the
> preprocessor deleted the sequence of nonwhitespace characters
> immediately before the ##, no matter what they were.  This version
> only deletes one token, and only does so if you give no rest args at
> all; an empty rest arg doesn't trigger deletion.  And you cannot rely
> on the preceding token being deleted unless it is a comma.

Btw, older versions of GCC also distinguish between no rest args, and
an empty one.  I expect you know this.

> Also, there's a new warning, which issues whenever the tokens on
> either side of ## cannot be glued into one token.  This warning is on
> by default, and may be suppressed by -Wno-paste.  A macro written to
> use the third extension would normally trigger this warning.  However,
> as a special case, the warning is suppressed for ", ## rest" where
> rest is a rest argument.  If you put something else before the ## the
> warning may or may not happen.

Perhaps more sensible logic would be to warn if the tokens are
non-pastable and you did not delete the preceding token?  ", ## rest"
naturally does not require a warning in that case, but other cases where
the previous token is not deleted would trigger the warning.  (I don't
know which cases they are but you alluded to them earlier).

> I do not have time to investigate your example in detail, but my
> immediate reaction is that you're trying to do things outside of cpp's
> intended scope.  If you want M4, you know where to find it.

It's code I've been working with for years, that worked fine over many
many versions of GCC and has recently broken between GCC 2.95.2 and GCC 2.96.

If GCC _has_ to change to comply with a standard, fair enough but I
think that in this case there is a genuine bug somewhere in the
preprocessor, to do with nested varargs processing.  My macros happen to
trigger it.  Hopefully Neil can work out what's going on.

thanks,
-- Jamie

Re: Varargs macros subtly broken

[Neil Booth]

Jamie Lokier wrote:-

> For other tokens, perhaps C99 requires that the token preceding
> ##__VA_ARGS__ is not deleted, if it is possible to paste them?

C99 has no concept of token deletion like this at all.  It's purely an
extension.  Judging from the C99 rationale that was published, they
considered handling the extra-comma problem, but decided not to do
anything, and it didn't seem like they will in future either.  Zack
and I want to keep it useful for what it was used for, but at minimal
deviation from the standard.

> Perhaps more sensible logic would be to warn if the tokens are
> non-pastable and you did not delete the preceding token?  ", ## rest"
> naturally does not require a warning in that case, but other cases where
> the previous token is not deleted would trigger the warning.  (I don't
> know which cases they are but you alluded to them earlier).

I think that's what we do now.  I doubt any cases other than comma will
cause deletion in the future (they don't now).

> If GCC _has_ to change to comply with a standard, fair enough but I
> think that in this case there is a genuine bug somewhere in the
> preprocessor, to do with nested varargs processing.  My macros happen to
> trigger it.  Hopefully Neil can work out what's going on.

I suspect you're right about it being a bug.  I'll try and simplify the
test-case a bit.  Most can shrink to 2 or 3 simple macros.

Neil.

Re: Varargs macros subtly broken

[Jamie Lokier]

Neil Booth wrote:
> > For other tokens, perhaps C99 requires that the token preceding
> > ##__VA_ARGS__ is not deleted, if it is possible to paste them?
> 
> C99 has no concept of token deletion like this at all.  It's purely an
> extension.

Precisely the reason we why should not, in general, delete tokens.

However deleting the comma may be permitted by the standard precisely
because you can't paste a comma with another token anyway.  Hmm.

I wonder what happens to argument prescan of __VA_ARGS__ in the presence
of ##.

-- Jamie

Re: Varargs macros subtly broken

[Neil Booth]

Jamie Lokier wrote:-

> If GCC _has_ to change to comply with a standard, fair enough but I
> think that in this case there is a genuine bug somewhere in the
> preprocessor, to do with nested varargs processing.  My macros happen to
> trigger it.  Hopefully Neil can work out what's going on.

It's a bug, and it's nasty.  "subtly broken" is fitting.  The
expansion algorithm assumes that if it has come across an identifier
during macro expansion that has previously failed expansion, then it
cannot be expanded again.  There are 2 possibilities: either the
identifier has already been expanded, and hence is permanently "off",
or maybe it was a function-like macro without a following '('.  If it
didn't have a following '(' previously one won't magically appear in
future expansions.  Seems reasonable if you think about it, but sadly
your testcase has proven this last assumption wrong.  Consider an
argument to a (necessarily varargs) macro of the form

"F, b"

where F is a function-like macro (very common in your *cough* macros).
F cannot be expanded here because no '(' follows it.  However, this
argument can be substituted within the expansion list of a macro, say
SomeMacro, causing it to be *split up* as so:

#define SomeMacro(...) A(__VA_ARGS__)
#define A(a, b) a(b)

SomeMacro(F, b)

And here, in the expansion of A, F now has the required '(' after it,
and so undergoes expansion itself.  What a shame.

Needless to say, I don't think I can fix this within the current code.
It's convinced me that even staying within the C99 standard
single-pass expansion is doomed to fail.  We need to revert to re-scan
passes, and this will take a while.  The nice thing is that the code
will be easier to understand again.

At least we get 99.9999% of macro expansions correct, so it's not too
bad if it remains broken for a short while longer.

Neil.

Re: Varargs macros subtly broken

[Jamie Lokier]

Neil Booth wrote:
> "F, b"
> 
> where F is a function-like macro (very common in your *cough* macros).
> F cannot be expanded here because no '(' follows it.  However, this
> argument can be substituted within the expansion list of a macro, say
> SomeMacro, causing it to be *split up* as so:
> 
> #define SomeMacro(...) A(__VA_ARGS__)
> #define A(a, b) a(b)
> 
> SomeMacro(F, b)
> 
> And here, in the expansion of A, F now has the required '(' after it,
> and so undergoes expansion itself.  What a shame.
> 
> Needless to say, I don't think I can fix this within the current code.

*boggle* you mean this is broken even without varargs?  Ouch!

Lots of my programs use function-like macros in this way, for the most
basic things like hash tables and sorting functions.
E.g. "FAST_MERGE_SORT (a, b, STRCMP_LIKE_COMPARISON_MACRO)".

I'll have to see if they still compile.  Oh, they do.  What am I missing
from the above explanation?

-- Jamie

Re: Varargs macros subtly broken

[Neil Booth]

Jamie Lokier wrote:-

> *boggle* you mean this is broken even without varargs?  Ouch!

No, only in the varargs case.

> Lots of my programs use function-like macros in this way, for the most
> basic things like hash tables and sorting functions.
> E.g. "FAST_MERGE_SORT (a, b, STRCMP_LIKE_COMPARISON_MACRO)".
> 
> I'll have to see if they still compile.  Oh, they do.  What am I missing
> from the above explanation?

Just that the normal case is taken care of <g>.

Neil.

Re: Varargs macros subtly broken

[Jamie Lokier]

Neil Booth wrote:
>> *boggle* you mean this is broken even without varargs?  Ouch!
>
>No, only in the varargs case.

Ah, I see.  Here is my simplified test case, probably like yours:

  #define half(x)      ((x) / 2)
  #define apply(...)   apply2 (__VA_ARGS__)
  #define apply2(f,x)  f (x)

  apply (half, X)

Expands to `half (X)' when it should expand to `((X) / 2)'.

-- Jamie

Re: macros with traditional cpp

[Richard Henderson]

On Sun, Sep 24, 2000 at 08:48:20PM +0100, Philip Blundell wrote:
> The preprocessor used to accept code like this both with and without 
> -traditional:
> 
> 	#define foo(x...) x
> 	foo(one,two)

It did?  I could have sworn I'd always had problems with that.
Oh, wait, I had problems with

	#define foo(x,y...) x , ## y

> Is this intentional?

It was probably not intentional that macro varargs ever worked
with -tradition, given that my test case failed.

My advice is to not use -traditional with your assembly.


r~

Re: macros with traditional cpp

[Richard Henderson]

On Sun, Sep 24, 2000 at 10:14:53PM +0100, Neil Booth wrote:
> > Oh, wait, I had problems with
> > 
> > 	#define foo(x,y...) x , ## y
> 
> Hmm, if problems with this still exist, I'd like to know.

No, no, the problems were with -traditional only, and they
existed in previous versions of gcc as well.


r~