Ping: [PATCH v2] libcpp: Handle extended characters in user-defined literal suffix [PR103902]

[Lewis Hyatt <lhyatt@gmail.com>]

May I please ping this one?
https://gcc.gnu.org/pipermail/gcc-patches/2023-March/613247.html

Thanks!

-Lewis

On Thu, Mar 2, 2023 at 6:21 PM Lewis Hyatt <lhyatt@gmail.com> wrote:
>
> The PR complains that we do not handle UTF-8 in the suffix for a user-defined
> literal, such as:
>
> bool operator ""_π (unsigned long long);
>
> In fact we don't handle any extended identifier characters there, whether
> UTF-8, UCNs, or the $ sign. We do handle it fine if the optional space after
> the "" tokens is included, since then the identifier is lexed in the "normal"
> way as its own token. But when it is lexed as part of the string token, this
> is handled in lex_string() with a one-off loop that is not aware of extended
> characters.
>
> This patch fixes it by adding a new function scan_cur_identifier() that can be
> used to lex an identifier while in the middle of lexing another token.
>
> BTW, the other place that has been mis-lexing identifiers is
> lex_identifier_intern(), which is used to implement #pragma push_macro
> and #pragma pop_macro. This does not support extended characters either.
> I will add that in a subsequent patch, because it can't directly reuse the
> new function, but rather needs to lex from a string instead of a cpp_buffer.
>
> With scan_cur_identifier(), we do also correctly warn about bidi and
> normalization issues in the extended identifiers comprising the suffix.
>
> libcpp/ChangeLog:
>
>         PR preprocessor/103902
>         * lex.cc (identifier_diagnostics_on_lex): New function refactoring
>         some common code.
>         (lex_identifier_intern): Use the new function.
>         (lex_identifier): Don't run identifier diagnostics here, rather let
>         the call site do it when needed.
>         (_cpp_lex_direct): Adjust the call sites of lex_identifier ()
>         acccordingly.
>         (struct scan_id_result): New struct.
>         (scan_cur_identifier): New function.
>         (create_literal2): New function.
>         (lit_accum::create_literal2): New function.
>         (is_macro): Folded into new function...
>         (maybe_ignore_udl_macro_suffix): ...here.
>         (is_macro_not_literal_suffix): Folded likewise.
>         (lex_raw_string): Handle UTF-8 in UDL suffix via scan_cur_identifier ().
>         (lex_string): Likewise.
>
> gcc/testsuite/ChangeLog:
>
>         PR preprocessor/103902
>         * g++.dg/cpp0x/udlit-extended-id-1.C: New test.
>         * g++.dg/cpp0x/udlit-extended-id-2.C: New test.
>         * g++.dg/cpp0x/udlit-extended-id-3.C: New test.
>         * g++.dg/cpp0x/udlit-extended-id-4.C: New test.
> ---
>
> Notes:
>     Hello-
>
>     This is the updated version of the patch, incorporating feedback from Jakub
>     and Jason, most recently discussed here:
>
>     https://gcc.gnu.org/pipermail/gcc-patches/2023-February/612073.html
>
>     Please let me know how it looks? It is simpler than before with the new
>     approach. Thanks!
>
>     One thing to note. As Jason clarified for me, a usage like this:
>
>      #pragma GCC poison _x
>     const char * operator "" _x (const char *, unsigned long);
>
>     The space between the "" and the _x is currently allowed but will be
>     deprecated in C++23. GCC currently will complain about the poisoned use of
>     _x in this case, and this patch, which is just focused on handling UTF-8
>     properly, does not change this. But it seems that it would be correct
>     not to apply poison in this case. I can try to follow up with a patch to do
>     so, if it seems worthwhile? Given the syntax is deprecated, maybe it's not
>     worth it...
>
>     For the time being, this patch does add a testcase for the above and xfails
>     it. For the case where no space is present, which is the part touched by the
>     present patch, existing behavior is preserved correctly and no diagnostics
>     such as poison are issued for the UDL suffix. (Contrary to v1 of this
>     patch.)
>
>     Thanks! bootstrap + regtested all languages on x86-64 Linux with
>     no regressions.
>
>     -Lewis
>
>  .../g++.dg/cpp0x/udlit-extended-id-1.C        |  68 ++++
>  .../g++.dg/cpp0x/udlit-extended-id-2.C        |   6 +
>  .../g++.dg/cpp0x/udlit-extended-id-3.C        |  15 +
>  .../g++.dg/cpp0x/udlit-extended-id-4.C        |  14 +
>  libcpp/lex.cc                                 | 382 ++++++++++--------
>  5 files changed, 317 insertions(+), 168 deletions(-)
>  create mode 100644 gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-1.C
>  create mode 100644 gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-2.C
>  create mode 100644 gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-3.C
>  create mode 100644 gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-4.C
>
> diff --git a/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-1.C b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-1.C
> new file mode 100644
> index 00000000000..411d4fdd0ba
> --- /dev/null
> +++ b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-1.C
> @@ -0,0 +1,68 @@
> +// { dg-do run { target c++11 } }
> +// { dg-additional-options "-Wno-error=normalized" }
> +#include <cstring>
> +using namespace std;
> +
> +constexpr unsigned long long operator "" _π (unsigned long long x)
> +{
> +  return 3 * x;
> +}
> +
> +/* Historically we didn't parse properly as part of the "" token, so check that
> +   as well.  */
> +constexpr unsigned long long operator ""_Π2 (unsigned long long x)
> +{
> +  return 4 * x;
> +}
> +
> +char x1[1_π];
> +char x2[2_Π2];
> +
> +static_assert (sizeof x1 == 3, "test1");
> +static_assert (sizeof x2 == 8, "test2");
> +
> +const char * operator "" _1σ (const char *s, unsigned long)
> +{
> +  return s + 1;
> +}
> +
> +const char * operator ""_Σ2 (const char *s, unsigned long)
> +{
> +  return s + 2;
> +}
> +
> +const char * operator "" _\U000000e61 (const char *s, unsigned long)
> +{
> +  return "ae";
> +}
> +
> +const char* operator ""_\u01532 (const char *s, unsigned long)
> +{
> +  return "oe";
> +}
> +
> +bool operator "" _\u0BC7\u0BBE (unsigned long long); // { dg-warning "not in NFC" }
> +bool operator ""_\u0B47\U00000B3E (unsigned long long); // { dg-warning "not in NFC" }
> +
> +#define xτy
> +const char * str = ""xτy; // { dg-warning "invalid suffix on literal" }
> +
> +int main()
> +{
> +  if (3_π != 9)
> +    __builtin_abort ();
> +  if (4_Π2 != 16)
> +    __builtin_abort ();
> +  if (strcmp ("abc"_1σ, "bc"))
> +    __builtin_abort ();
> +  if (strcmp ("abcd"_Σ2, "cd"))
> +    __builtin_abort ();
> +  if (strcmp (R"(abcdef)"_1σ, "bcdef"))
> +    __builtin_abort ();
> +  if (strcmp (R"(abcdef)"_Σ2, "cdef"))
> +    __builtin_abort ();
> +  if (strcmp ("xyz"_æ1, "ae"))
> +    __builtin_abort ();
> +  if (strcmp ("xyz"_œ2, "oe"))
> +    __builtin_abort ();
> +}
> diff --git a/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-2.C b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-2.C
> new file mode 100644
> index 00000000000..05a2804a463
> --- /dev/null
> +++ b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-2.C
> @@ -0,0 +1,6 @@
> +// { dg-do compile { target c++11 } }
> +// { dg-additional-options "-Wbidi-chars=any,ucn" }
> +bool operator ""_d\u202ae\u202cf (unsigned long long); // { dg-line line1 }
> +// { dg-error "universal character \\\\u202a is not valid in an identifier" "test1" { target *-*-* } line1 }
> +// { dg-error "universal character \\\\u202c is not valid in an identifier" "test2" { target *-*-* } line1 }
> +// { dg-warning "found problematic Unicode character" "test3" { target *-*-* } line1 }
> diff --git a/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-3.C b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-3.C
> new file mode 100644
> index 00000000000..11292e476e3
> --- /dev/null
> +++ b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-3.C
> @@ -0,0 +1,15 @@
> +// { dg-do compile { target c++11 } }
> +
> +// Check that we do not look for poisoned identifier when it is a suffix.
> +int _ħ;
> +#pragma GCC poison _ħ
> +const char * operator ""_ħ (const char *, unsigned long); // { dg-bogus "poisoned" }
> +bool operator ""_ħ (unsigned long long x); // { dg-bogus "poisoned" }
> +bool b = 1_ħ; // { dg-bogus "poisoned" }
> +const char *x = "hbar"_ħ; // { dg-bogus "poisoned" }
> +
> +/* Ideally, we should not warn here either, but this is not implemented yet.  This
> +   syntax has been deprecated for C++23.  */
> +#pragma GCC poison _ħ2
> +const char * operator "" _ħ2 (const char *, unsigned long); // { dg-bogus "poisoned" "" { xfail *-*-*} }
> +const char *x2 = "hbar2"_ħ2; // { dg-bogus "poisoned" }
> diff --git a/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-4.C b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-4.C
> new file mode 100644
> index 00000000000..d1683c4d892
> --- /dev/null
> +++ b/gcc/testsuite/g++.dg/cpp0x/udlit-extended-id-4.C
> @@ -0,0 +1,14 @@
> +// { dg-options "-std=c++98 -Wc++11-compat" }
> +#define END ;
> +#define εND ;
> +#define EηD ;
> +#define EN\u0394 ;
> +
> +const char *s1 = "s1"END // { dg-warning "requires a space between string literal and macro" }
> +const char *s2 = "s2"εND // { dg-warning "requires a space between string literal and macro" }
> +const char *s3 = "s3"EηD // { dg-warning "requires a space between string literal and macro" }
> +const char *s4 = "s4"ENΔ // { dg-warning "requires a space between string literal and macro" }
> +
> +/* Make sure we did not skip the token also in the case that it wasn't found to
> +   be a macro; compilation should fail here.  */
> +const char *s5 = "s5"NØT_A_MACRO; // { dg-error "expected ',' or ';' before" }
> diff --git a/libcpp/lex.cc b/libcpp/lex.cc
> index 45ea16a91bc..062935e2371 100644
> --- a/libcpp/lex.cc
> +++ b/libcpp/lex.cc
> @@ -2057,8 +2057,11 @@ warn_about_normalization (cpp_reader *pfile,
>      }
>  }
>
> -/* Returns TRUE if the sequence starting at buffer->cur is valid in
> -   an identifier.  FIRST is TRUE if this starts an identifier.  */
> +/* Returns TRUE if the byte sequence starting at buffer->cur is a valid
> +   extended character in an identifier.  If FIRST is TRUE, then the character
> +   must be valid at the beginning of an identifier as well.  If the return
> +   value is TRUE, then pfile->buffer->cur has been moved to point to the next
> +   byte after the extended character.  */
>
>  static bool
>  forms_identifier_p (cpp_reader *pfile, int first,
> @@ -2154,6 +2157,47 @@ maybe_va_opt_error (cpp_reader *pfile)
>      }
>  }
>
> +/* Helper function to perform diagnostics that are needed (rarely)
> +   when an identifier is lexed.  */
> +static void
> +identifier_diagnostics_on_lex (cpp_reader *pfile, cpp_hashnode *node)
> +{
> +  if (__builtin_expect (!(node->flags & NODE_DIAGNOSTIC)
> +                       || pfile->state.skipping, 1))
> +    return;
> +
> +  /* It is allowed to poison the same identifier twice.  */
> +  if ((node->flags & NODE_POISONED) && !pfile->state.poisoned_ok)
> +    cpp_error (pfile, CPP_DL_ERROR, "attempt to use poisoned \"%s\"",
> +              NODE_NAME (node));
> +
> +  /* Constraint 6.10.3.5: __VA_ARGS__ should only appear in the
> +     replacement list of a variadic macro.  */
> +  if (node == pfile->spec_nodes.n__VA_ARGS__
> +      && !pfile->state.va_args_ok)
> +    {
> +      if (CPP_OPTION (pfile, cplusplus))
> +       cpp_error (pfile, CPP_DL_PEDWARN,
> +                  "__VA_ARGS__ can only appear in the expansion"
> +                  " of a C++11 variadic macro");
> +      else
> +       cpp_error (pfile, CPP_DL_PEDWARN,
> +                  "__VA_ARGS__ can only appear in the expansion"
> +                  " of a C99 variadic macro");
> +    }
> +
> +  /* __VA_OPT__ should only appear in the replacement list of a
> +     variadic macro.  */
> +  if (node == pfile->spec_nodes.n__VA_OPT__)
> +    maybe_va_opt_error (pfile);
> +
> +  /* For -Wc++-compat, warn about use of C++ named operators.  */
> +  if (node->flags & NODE_WARN_OPERATOR)
> +    cpp_warning (pfile, CPP_W_CXX_OPERATOR_NAMES,
> +                "identifier \"%s\" is a special operator name in C++",
> +                NODE_NAME (node));
> +}
> +
>  /* Helper function to get the cpp_hashnode of the identifier BASE.  */
>  static cpp_hashnode *
>  lex_identifier_intern (cpp_reader *pfile, const uchar *base)
> @@ -2173,41 +2217,7 @@ lex_identifier_intern (cpp_reader *pfile, const uchar *base)
>    hash = HT_HASHFINISH (hash, len);
>    result = CPP_HASHNODE (ht_lookup_with_hash (pfile->hash_table,
>                                               base, len, hash, HT_ALLOC));
> -
> -  /* Rarely, identifiers require diagnostics when lexed.  */
> -  if (__builtin_expect ((result->flags & NODE_DIAGNOSTIC)
> -                       && !pfile->state.skipping, 0))
> -    {
> -      /* It is allowed to poison the same identifier twice.  */
> -      if ((result->flags & NODE_POISONED) && !pfile->state.poisoned_ok)
> -       cpp_error (pfile, CPP_DL_ERROR, "attempt to use poisoned \"%s\"",
> -                  NODE_NAME (result));
> -
> -      /* Constraint 6.10.3.5: __VA_ARGS__ should only appear in the
> -        replacement list of a variadic macro.  */
> -      if (result == pfile->spec_nodes.n__VA_ARGS__
> -         && !pfile->state.va_args_ok)
> -       {
> -         if (CPP_OPTION (pfile, cplusplus))
> -           cpp_error (pfile, CPP_DL_PEDWARN,
> -                      "__VA_ARGS__ can only appear in the expansion"
> -                      " of a C++11 variadic macro");
> -         else
> -           cpp_error (pfile, CPP_DL_PEDWARN,
> -                      "__VA_ARGS__ can only appear in the expansion"
> -                      " of a C99 variadic macro");
> -       }
> -
> -      if (result == pfile->spec_nodes.n__VA_OPT__)
> -       maybe_va_opt_error (pfile);
> -
> -      /* For -Wc++-compat, warn about use of C++ named operators.  */
> -      if (result->flags & NODE_WARN_OPERATOR)
> -       cpp_warning (pfile, CPP_W_CXX_OPERATOR_NAMES,
> -                    "identifier \"%s\" is a special operator name in C++",
> -                    NODE_NAME (result));
> -    }
> -
> +  identifier_diagnostics_on_lex (pfile, result);
>    return result;
>  }
>
> @@ -2221,7 +2231,9 @@ _cpp_lex_identifier (cpp_reader *pfile, const char *name)
>    return result;
>  }
>
> -/* Lex an identifier starting at BUFFER->CUR - 1.  */
> +/* Lex an identifier starting at BASE.  BUFFER->CUR is expected to point
> +   one past the first character at BASE, which may be a (possibly multi-byte)
> +   character if STARTS_UCN is true.  */
>  static cpp_hashnode *
>  lex_identifier (cpp_reader *pfile, const uchar *base, bool starts_ucn,
>                 struct normalize_state *nst, cpp_hashnode **spelling)
> @@ -2270,42 +2282,51 @@ lex_identifier (cpp_reader *pfile, const uchar *base, bool starts_ucn,
>        *spelling = result;
>      }
>
> -  /* Rarely, identifiers require diagnostics when lexed.  */
> -  if (__builtin_expect ((result->flags & NODE_DIAGNOSTIC)
> -                       && !pfile->state.skipping, 0))
> -    {
> -      /* It is allowed to poison the same identifier twice.  */
> -      if ((result->flags & NODE_POISONED) && !pfile->state.poisoned_ok)
> -       cpp_error (pfile, CPP_DL_ERROR, "attempt to use poisoned \"%s\"",
> -                  NODE_NAME (result));
> -
> -      /* Constraint 6.10.3.5: __VA_ARGS__ should only appear in the
> -        replacement list of a variadic macro.  */
> -      if (result == pfile->spec_nodes.n__VA_ARGS__
> -         && !pfile->state.va_args_ok)
> -       {
> -         if (CPP_OPTION (pfile, cplusplus))
> -           cpp_error (pfile, CPP_DL_PEDWARN,
> -                      "__VA_ARGS__ can only appear in the expansion"
> -                      " of a C++11 variadic macro");
> -         else
> -           cpp_error (pfile, CPP_DL_PEDWARN,
> -                      "__VA_ARGS__ can only appear in the expansion"
> -                      " of a C99 variadic macro");
> -       }
> +  return result;
> +}
>
> -      /* __VA_OPT__ should only appear in the replacement list of a
> -        variadic macro.  */
> -      if (result == pfile->spec_nodes.n__VA_OPT__)
> -       maybe_va_opt_error (pfile);
> -
> -      /* For -Wc++-compat, warn about use of C++ named operators.  */
> -      if (result->flags & NODE_WARN_OPERATOR)
> -       cpp_warning (pfile, CPP_W_CXX_OPERATOR_NAMES,
> -                    "identifier \"%s\" is a special operator name in C++",
> -                    NODE_NAME (result));
> -    }
> +/* Struct to hold the return value of the scan_cur_identifier () helper
> +   function below.  */
>
> +struct scan_id_result
> +{
> +  cpp_hashnode *node;
> +  normalize_state nst;
> +
> +  scan_id_result ()
> +    : node (nullptr)
> +  {
> +    nst = INITIAL_NORMALIZE_STATE;
> +  }
> +
> +  explicit operator bool () const { return node; }
> +};
> +
> +/* Helper function to scan an entire identifier beginning at
> +   pfile->buffer->cur, and possibly containing extended characters (UCNs
> +   and/or UTF-8).  Returns the cpp_hashnode for the identifier on success, or
> +   else nullptr, as well as a normalize_state so that normalization warnings
> +   may be issued once the token lexing is complete.  */
> +
> +static scan_id_result
> +scan_cur_identifier (cpp_reader *pfile)
> +{
> +  const auto buffer = pfile->buffer;
> +  const auto begin = buffer->cur;
> +  scan_id_result result;
> +  if (ISIDST (*buffer->cur))
> +    {
> +      ++buffer->cur;
> +      cpp_hashnode *ignore;
> +      result.node = lex_identifier (pfile, begin, false, &result.nst, &ignore);
> +    }
> +  else if (forms_identifier_p (pfile, true, &result.nst))
> +    {
> +      /* buffer->cur has been moved already by the call
> +        to forms_identifier_p.  */
> +      cpp_hashnode *ignore;
> +      result.node = lex_identifier (pfile, begin, true, &result.nst, &ignore);
> +    }
>    return result;
>  }
>
> @@ -2365,6 +2386,24 @@ create_literal (cpp_reader *pfile, cpp_token *token, const uchar *base,
>    token->val.str.text = cpp_alloc_token_string (pfile, base, len);
>  }
>
> +/* Like create_literal(), but construct it from two separate strings
> +   which are concatenated.  LEN2 may be 0 if no second string is
> +   required.  */
> +static void
> +create_literal2 (cpp_reader *pfile, cpp_token *token, const uchar *base1,
> +                unsigned int len1, const uchar *base2, unsigned int len2,
> +                enum cpp_ttype type)
> +{
> +  token->type = type;
> +  token->val.str.len = len1 + len2;
> +  uchar *const dest = _cpp_unaligned_alloc (pfile, len1 + len2 + 1);
> +  memcpy (dest, base1, len1);
> +  if (len2)
> +    memcpy (dest+len1, base2, len2);
> +  dest[len1 + len2] = 0;
> +  token->val.str.text = dest;
> +}
> +
>  const uchar *
>  cpp_alloc_token_string (cpp_reader *pfile,
>                         const unsigned char *ptr, unsigned len)
> @@ -2403,6 +2442,11 @@ struct lit_accum {
>        rpos = NULL;
>      return c;
>    }
> +
> +  void create_literal2 (cpp_reader *pfile, cpp_token *token,
> +                       const uchar *base1, unsigned int len1,
> +                       const uchar *base2, unsigned int len2,
> +                       enum cpp_ttype type);
>  };
>
>  /* Subroutine of lex_raw_string: Append LEN chars from BASE to the buffer
> @@ -2445,45 +2489,57 @@ lit_accum::read_begin (cpp_reader *pfile)
>    rpos = BUFF_FRONT (last);
>  }
>
> -/* Returns true if a macro has been defined.
> -   This might not work if compile with -save-temps,
> -   or preprocess separately from compilation.  */
> +/* Helper function to check if a string format macro, say from inttypes.h, is
> +   placed touching a string literal, in which case it could be parsed as a C++11
> +   user-defined string literal thus breaking the program.  User-defined literals
> +   outside of namespace std must start with a single underscore, so assume
> +   anything of that form really is a UDL suffix.  We don't need to worry about
> +   UDLs defined inside namespace std because their names are reserved, so cannot
> +   be used as macro names in valid programs.  Return TRUE if the UDL should be
> +   ignored for now and preserved for potential macro expansion.  */
>
>  static bool
> -is_macro(cpp_reader *pfile, const uchar *base)
> +maybe_ignore_udl_macro_suffix (cpp_reader *pfile, location_t src_loc,
> +                              const uchar *suffix_begin, cpp_hashnode *node)
>  {
> -  const uchar *cur = base;
> -  if (! ISIDST (*cur))
> +  if ((suffix_begin[0] == '_' && suffix_begin[1] != '_')
> +      || !cpp_macro_p (node))
>      return false;
> -  unsigned int hash = HT_HASHSTEP (0, *cur);
> -  ++cur;
> -  while (ISIDNUM (*cur))
> -    {
> -      hash = HT_HASHSTEP (hash, *cur);
> -      ++cur;
> -    }
> -  hash = HT_HASHFINISH (hash, cur - base);
>
> -  cpp_hashnode *result = CPP_HASHNODE (ht_lookup_with_hash (pfile->hash_table,
> -                                       base, cur - base, hash, HT_NO_INSERT));
> -
> -  return result && cpp_macro_p (result);
> +  /* Maybe raise a warning here; caller should arrange not to consume
> +     the tokens.  */
> +  if (CPP_OPTION (pfile, warn_literal_suffix) && !pfile->state.skipping)
> +    cpp_warning_with_line (pfile, CPP_W_LITERAL_SUFFIX, src_loc, 0,
> +                          "invalid suffix on literal; C++11 requires a space "
> +                          "between literal and string macro");
> +  return true;
>  }
>
> -/* Returns true if a literal suffix does not have the expected form
> -   and is defined as a macro.  */
> -
> -static bool
> -is_macro_not_literal_suffix(cpp_reader *pfile, const uchar *base)
> +/* Like create_literal2(), but also prepend all the accumulated data from
> +   the lit_accum struct.  */
> +void
> +lit_accum::create_literal2 (cpp_reader *pfile, cpp_token *token,
> +                           const uchar *base1, unsigned int len1,
> +                           const uchar *base2, unsigned int len2,
> +                           enum cpp_ttype type)
>  {
> -  /* User-defined literals outside of namespace std must start with a single
> -     underscore, so assume anything of that form really is a UDL suffix.
> -     We don't need to worry about UDLs defined inside namespace std because
> -     their names are reserved, so cannot be used as macro names in valid
> -     programs.  */
> -  if (base[0] == '_' && base[1] != '_')
> -    return false;
> -  return is_macro (pfile, base);
> +  const unsigned int tot_len = accum + len1 + len2;
> +  uchar *dest = _cpp_unaligned_alloc (pfile, tot_len + 1);
> +  token->type = type;
> +  token->val.str.len = tot_len;
> +  token->val.str.text = dest;
> +  for (_cpp_buff *buf = first; buf; buf = buf->next)
> +    {
> +      size_t len = BUFF_FRONT (buf) - buf->base;
> +      memcpy (dest, buf->base, len);
> +      dest += len;
> +    }
> +  memcpy (dest, base1, len1);
> +  dest += len1;
> +  if (len2)
> +    memcpy (dest, base2, len2);
> +  dest += len2;
> +  *dest = '\0';
>  }
>
>  /* Lexes a raw string.  The stored string contains the spelling,
> @@ -2758,26 +2814,25 @@ lex_raw_string (cpp_reader *pfile, cpp_token *token, const uchar *base)
>
>    if (CPP_OPTION (pfile, user_literals))
>      {
> -      /* If a string format macro, say from inttypes.h, is placed touching
> -        a string literal it could be parsed as a C++11 user-defined string
> -        literal thus breaking the program.  */
> -      if (is_macro_not_literal_suffix (pfile, pos))
> -       {
> -         /* Raise a warning, but do not consume subsequent tokens.  */
> -         if (CPP_OPTION (pfile, warn_literal_suffix) && !pfile->state.skipping)
> -           cpp_warning_with_line (pfile, CPP_W_LITERAL_SUFFIX,
> -                                  token->src_loc, 0,
> -                                  "invalid suffix on literal; C++11 requires "
> -                                  "a space between literal and string macro");
> -       }
> -      /* Grab user defined literal suffix.  */
> -      else if (ISIDST (*pos))
> -       {
> -         type = cpp_userdef_string_add_type (type);
> -         ++pos;
> +      const uchar *const suffix_begin = pos;
> +      pfile->buffer->cur = pos;
>
> -         while (ISIDNUM (*pos))
> -           ++pos;
> +      if (const auto sr = scan_cur_identifier (pfile))
> +       {
> +         if (maybe_ignore_udl_macro_suffix (pfile, token->src_loc,
> +                                            suffix_begin, sr.node))
> +             pfile->buffer->cur = suffix_begin;
> +         else
> +           {
> +             type = cpp_userdef_string_add_type (type);
> +             accum.create_literal2 (pfile, token, base, suffix_begin - base,
> +                                    NODE_NAME (sr.node), NODE_LEN (sr.node),
> +                                    type);
> +             if (accum.first)
> +               _cpp_release_buff (pfile, accum.first);
> +             warn_about_normalization (pfile, token, &sr.nst, true);
> +             return;
> +           }
>         }
>      }
>
> @@ -2787,21 +2842,8 @@ lex_raw_string (cpp_reader *pfile, cpp_token *token, const uchar *base)
>      create_literal (pfile, token, base, pos - base, type);
>    else
>      {
> -      size_t extra_len = pos - base;
> -      uchar *dest = _cpp_unaligned_alloc (pfile, accum.accum + extra_len + 1);
> -
> -      token->type = type;
> -      token->val.str.len = accum.accum + extra_len;
> -      token->val.str.text = dest;
> -      for (_cpp_buff *buf = accum.first; buf; buf = buf->next)
> -       {
> -         size_t len = BUFF_FRONT (buf) - buf->base;
> -         memcpy (dest, buf->base, len);
> -         dest += len;
> -       }
> +      accum.create_literal2 (pfile, token, base, pos - base, nullptr, 0, type);
>        _cpp_release_buff (pfile, accum.first);
> -      memcpy (dest, base, extra_len);
> -      dest[extra_len] = '\0';
>      }
>  }
>
> @@ -2908,39 +2950,40 @@ lex_string (cpp_reader *pfile, cpp_token *token, const uchar *base)
>      cpp_error (pfile, CPP_DL_PEDWARN, "missing terminating %c character",
>                (int) terminator);
>
> +  pfile->buffer->cur = cur;
> +  const uchar *const suffix_begin = cur;
> +
>    if (CPP_OPTION (pfile, user_literals))
>      {
> -      /* If a string format macro, say from inttypes.h, is placed touching
> -        a string literal it could be parsed as a C++11 user-defined string
> -        literal thus breaking the program.  */
> -      if (is_macro_not_literal_suffix (pfile, cur))
> -       {
> -         /* Raise a warning, but do not consume subsequent tokens.  */
> -         if (CPP_OPTION (pfile, warn_literal_suffix) && !pfile->state.skipping)
> -           cpp_warning_with_line (pfile, CPP_W_LITERAL_SUFFIX,
> -                                  token->src_loc, 0,
> -                                  "invalid suffix on literal; C++11 requires "
> -                                  "a space between literal and string macro");
> -       }
> -      /* Grab user defined literal suffix.  */
> -      else if (ISIDST (*cur))
> +      if (const auto sr = scan_cur_identifier (pfile))
>         {
> -         type = cpp_userdef_char_add_type (type);
> -         type = cpp_userdef_string_add_type (type);
> -          ++cur;
> -
> -         while (ISIDNUM (*cur))
> -           ++cur;
> +         if (maybe_ignore_udl_macro_suffix (pfile, token->src_loc,
> +                                            suffix_begin, sr.node))
> +           pfile->buffer->cur = suffix_begin;
> +         else
> +           {
> +             /* Grab user defined literal suffix.  */
> +             type = cpp_userdef_char_add_type (type);
> +             type = cpp_userdef_string_add_type (type);
> +             create_literal2 (pfile, token, base, suffix_begin - base,
> +                              NODE_NAME (sr.node), NODE_LEN (sr.node), type);
> +             warn_about_normalization (pfile, token, &sr.nst, true);
> +             return;
> +           }
>         }
>      }
>    else if (CPP_OPTION (pfile, cpp_warn_cxx11_compat)
> -          && is_macro (pfile, cur)
>            && !pfile->state.skipping)
> -    cpp_warning_with_line (pfile, CPP_W_CXX11_COMPAT,
> -                          token->src_loc, 0, "C++11 requires a space "
> -                          "between string literal and macro");
> +    {
> +      const auto sr = scan_cur_identifier (pfile);
> +      /* Maybe raise a warning, but do not consume the tokens.  */
> +      pfile->buffer->cur = suffix_begin;
> +      if (sr && cpp_macro_p (sr.node))
> +       cpp_warning_with_line (pfile, CPP_W_CXX11_COMPAT,
> +                              token->src_loc, 0, "C++11 requires a space "
> +                              "between string literal and macro");
> +    }
>
> -  pfile->buffer->cur = cur;
>    create_literal (pfile, token, base, cur - base, type);
>  }
>
> @@ -3915,9 +3958,10 @@ _cpp_lex_direct (cpp_reader *pfile)
>        result->type = CPP_NAME;
>        {
>         struct normalize_state nst = INITIAL_NORMALIZE_STATE;
> -       result->val.node.node = lex_identifier (pfile, buffer->cur - 1, false,
> -                                               &nst,
> -                                               &result->val.node.spelling);
> +       const auto node = lex_identifier (pfile, buffer->cur - 1, false, &nst,
> +                                         &result->val.node.spelling);
> +       result->val.node.node = node;
> +       identifier_diagnostics_on_lex (pfile, node);
>         warn_about_normalization (pfile, result, &nst, true);
>        }
>
> @@ -4220,8 +4264,10 @@ _cpp_lex_direct (cpp_reader *pfile)
>         if (forms_identifier_p (pfile, true, &nst))
>           {
>             result->type = CPP_NAME;
> -           result->val.node.node = lex_identifier (pfile, base, true, &nst,
> -                                                   &result->val.node.spelling);
> +           const auto node = lex_identifier (pfile, base, true, &nst,
> +                                             &result->val.node.spelling);
> +           result->val.node.node = node;
> +           identifier_diagnostics_on_lex (pfile, node);
>             warn_about_normalization (pfile, result, &nst, true);
>             break;
>           }
> @@ -4353,7 +4399,7 @@ cpp_digraph2name (enum cpp_ttype type)
>  }
>
>  /* Write the spelling of an identifier IDENT, using UCNs, to BUFFER.
> -   The buffer must already contain the enough space to hold the
> +   The buffer must already contain enough space to hold the
>     token's spelling.  Returns a pointer to the character after the
>     last character written.  */
>  unsigned char *
> @@ -4375,7 +4421,7 @@ _cpp_spell_ident_ucns (unsigned char *buffer, cpp_hashnode *ident)
>  }
>
>  /* Write the spelling of a token TOKEN to BUFFER.  The buffer must
> -   already contain the enough space to hold the token's spelling.
> +   already contain enough space to hold the token's spelling.
>     Returns a pointer to the character after the last character written.
>     FORSTRING is true if this is to be the spelling after translation
>     phase 1 (with the original spelling of extended identifiers), false