Re: rust frontend and UTF-8/unicode processing/properties

[Philip Herron <philip.herron@embecosm.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 2753 bytes --]

On 18/07/2021 23:23, Jason Merrill via Gcc-rust wrote:
> On Sun, Jul 18, 2021 at 1:13 PM Ian Lance Taylor via Gcc
> <gcc@gcc.gnu.org <mailto:gcc@gcc.gnu.org>> wrote:
>
>     On Sun, Jul 18, 2021 at 6:23 AM Mark Wielaard <mark@klomp.org
>     <mailto:mark@klomp.org>> wrote:
>     >
>     > For the gcc rust frontend I was thinking of importing a couple of
>     > gnulib modules to help with UTF-8 processing, conversion to/from
>     > unicode codepoints and determining various properties of those
>     > codepoints. But it seems gcc doesn't yet have any gnulib modules
>     > imported, and maybe other frontends already have helpers to this
>     that
>     > the gcc rust frontend could reuse.
>     >
>     > Rust only accepts valid UTF-8 encoded source files, which may or may
>     > not start with UTF-8 BOM character. Whitespace is any codepoint with
>     > the Pattern_White_Space property. Identifiers can start with any
>     > codepoint with the XID_start property plus zero or one
>     codepoints with
>     > XID_continue property. It isn't required, but highly desirable to
>     > detect confusable identifiers according to
>     tr39/Confusable_Detection.
>     >
>     > Other names might be constraint to Alphabetic and/or Number
>     categories
>     > (Nd, Nl, No), textual types can only contain Unicode Scalar Values
>     > (any Unicode codepoint except high-surrogate and low-surrogates),
>     > strings in source code can contain unicode escapes (24 bit, up to 6
>     > digits codepoints) but are internally stored as UTF-8 (and must not
>     > encode any surrogates).
>     >
>     > Do other gcc frontends handle any of the above already in a way that
>     > might be reusable for other frontends?
>
>     I don't know that this is particularly helpful, but the Go frontend
>     has this kind of code in gcc/go/gofrontend/lex.cc.  E.g.,
>     Lex::fetch_char, Lex::advance_one_utf8_char, unicode_space,
>     unicode_digits, unicode_letters, Lex::is_unicode_space, etc.  But you
>     probably won't be able to use the code directly, and the code in the
>     gofrontend directory is also shared with GoLLVM so it can't trivially
>     be moved.
>
>
> I believe the UTF-8 handling for the C family front ends is all in
> libcpp; I don't think it's factored in a way to be useful to other
> front ends.
>
> Jason
>
I think it would be ideal to reuse code if possible, it seems like this
project could be an opportunity to at try and make patches to try and
reuse code from other parts of GCC. I bootstrapped the front-end code
from gccgo, which has really helped the project to get going, I would
like to try and give back where I can.

Thanks

--Phil


[-- Attachment #1.1.2: Type: text/html, Size: 4461 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 665 bytes --]