rust frontend and UTF-8/unicode processing/properties

[Mark Wielaard <mark@klomp.org>]

Hi,

For the gcc rust frontend I was thinking of importing a couple of
gnulib modules to help with UTF-8 processing, conversion to/from
unicode codepoints and determining various properties of those
codepoints. But it seems gcc doesn't yet have any gnulib modules
imported, and maybe other frontends already have helpers to this that
the gcc rust frontend could reuse.

Rust only accepts valid UTF-8 encoded source files, which may or may
not start with UTF-8 BOM character. Whitespace is any codepoint with
the Pattern_White_Space property. Identifiers can start with any
codepoint with the XID_start property plus zero or one codepoints with
XID_continue property. It isn't required, but highly desirable to
detect confusable identifiers according to tr39/Confusable_Detection.

Other names might be constraint to Alphabetic and/or Number categories
(Nd, Nl, No), textual types can only contain Unicode Scalar Values
(any Unicode codepoint except high-surrogate and low-surrogates),
strings in source code can contain unicode escapes (24 bit, up to 6
digits codepoints) but are internally stored as UTF-8 (and must not
encode any surrogates).

Do other gcc frontends handle any of the above already in a way that
might be reusable for other frontends?

Thanks,

Mark