[gcc r12-727] preprocessor: Fix cpp_avoid_paste for digit separators

[gcc r12-727] preprocessor: Fix cpp_avoid_paste for digit separators

[Joseph Myers]

https://gcc.gnu.org/g:3e3fdf3d5217e5a2d075ca399b557b2e886dcd18

commit r12-727-g3e3fdf3d5217e5a2d075ca399b557b2e886dcd18
Author: Joseph Myers <joseph@codesourcery.com>
Date:   Tue May 11 18:54:32 2021 +0000

    preprocessor: Fix cpp_avoid_paste for digit separators
    
    The libcpp function cpp_avoid_paste is used to insert whitespace in
    preprocessed output where needed to avoid two consecutive
    preprocessing tokens, that logically (e.g. when stringized) do not
    have whitespace between them, from being incorrectly lexed as one when
    the preprocessed input is reread by a compiler.
    
    This fails to allow for digit separators, so meaning that invalid
    code, that has a CPP_NUMBER (from a macro expansion) followed by a
    character literal, can result in preprocessed output with a valid use
    of digit separators, so that required syntax errors do not occur when
    compiling with -save-temps.  Fix this by handling that case in
    cpp_avoid_paste (as with other cases in cpp_avoid_paste, this doesn't
    try to check whether the language version in use supports digit
    separators; it's always OK to have unnecessary whitespace in
    preprocessed output).
    
    Note: there are other cases, with various kinds of wide character or
    string literal following a CPP_NUMBER, where spurious pasting of
    preprocessing tokens can occur but the sequence of tokens remains
    invalid both before and after that pasting.  Maybe cpp_avoid_paste
    should also handle those cases (and similar cases after a CPP_NAME),
    to ensure the sequence of preprocessing tokens in preprocessed output
    is exactly right, whether or not it affects whether syntax errors
    occur.  This patch only addresses the case with digit separators where
    invalid code can fail to be diagnosed without the space inserted.
    
    Bootstrapped with no regressions for x86_64-pc-linux-gnu.
    
    libcpp/
            * lex.c (cpp_avoid_paste): Do not allow pasting CPP_NUMBER with
            CPP_CHAR.
    
    gcc/testsuite/
            * g++.dg/cpp1y/digit-sep-paste.C, gcc.dg/c2x-digit-separators-3.c:
            New tests.

Diff:
---
 gcc/testsuite/g++.dg/cpp1y/digit-sep-paste.C  | 11 +++++++++++
 gcc/testsuite/gcc.dg/c2x-digit-separators-3.c | 12 ++++++++++++
 libcpp/lex.c                                  |  1 +
 3 files changed, 24 insertions(+)

diff --git a/gcc/testsuite/g++.dg/cpp1y/digit-sep-paste.C b/gcc/testsuite/g++.dg/cpp1y/digit-sep-paste.C
new file mode 100644
index 00000000000..41fb967ef8d
--- /dev/null
+++ b/gcc/testsuite/g++.dg/cpp1y/digit-sep-paste.C
@@ -0,0 +1,11 @@
+// Test token pasting with digit separators avoided for preprocessed output.
+// { dg-do compile { target c++14 } }
+// { dg-options "-save-temps" }
+
+#define ZERO 0
+
+int
+f ()
+{
+  return ZERO'0'0; /* { dg-error "expected" } */
+}
diff --git a/gcc/testsuite/gcc.dg/c2x-digit-separators-3.c b/gcc/testsuite/gcc.dg/c2x-digit-separators-3.c
new file mode 100644
index 00000000000..cddb88fa880
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/c2x-digit-separators-3.c
@@ -0,0 +1,12 @@
+/* Test C2x digit separators.  Test token pasting avoided for preprocessed
+   output.  */
+/* { dg-do compile } */
+/* { dg-options "-std=c2x -save-temps" } */
+
+#define ZERO 0
+
+int
+f (void)
+{
+  return ZERO'0'0; /* { dg-error "expected" } */
+}
diff --git a/libcpp/lex.c b/libcpp/lex.c
index b7ce85a0331..36cd2e30630 100644
--- a/libcpp/lex.c
+++ b/libcpp/lex.c
@@ -3725,6 +3725,7 @@ cpp_avoid_paste (cpp_reader *pfile, const cpp_token *token1,
 				|| b == CPP_NAME
 				|| b == CPP_CHAR || b == CPP_STRING); /* L */
     case CPP_NUMBER:	return (b == CPP_NUMBER || b == CPP_NAME
+				|| b == CPP_CHAR
 				|| c == '.' || c == '+' || c == '-');
 				      /* UCNs */
     case CPP_OTHER:	return ((token1->val.str.text[0] == '\\'