Re: Fwd: New contributor tasks

[Mark Wielaard <mark@klomp.org>]

Hi Joel,

On Sat, Jul 17, 2021 at 10:25:48PM +0800, The Other wrote:
> > - Full unicode/utf8 support in the lexer. Currently the lexer only
> >   explicitly interprets the input as UTF8 for string parseing. It
> >   should really treat all input as UTF-8. gnulib has some handy
> >   modules we could use to read/convert from/to utf8 (unistr/u8-to-u32,
> >   unistr/u32-to-u8) and test various unicode properties
> >   (unictype/property-white-space, unictype/property-xid-continue,
> >   unictype/property-xid-start). I don't know if we can import those or
> >   if gcc already has these kind of UTF-8/unicode support functions for
> >   other languages?
> 
> At the time of writing the lexer, I was under the impression that Rust only
> supported UTF-8 in strings. The Rust Reference seems to have changed now to
> show that it supports UTF-8 in identifiers as well. I believe that the C++
> frontend, at least, has its own specific hardcoded UTF-8 handling for
> identifiers and strings (rather than using a library).
> 
> There could be issues with lookahead of several bytes (which the lexer uses
> liberally) if using UTF-8 in strings, depending on the exact implementation
> of whatever library you use (or function you write).

The whole source file should be valid UTF-8. You can use it in
comments too. And any invalid UTF-8 encoding means the file isn't a
valid Rust source file. So the simplest is to make the lexer handle
UTF-8 and handle one codepoint (UCS4/32bits) at a time. Lookahead then
also simply works per codepoint. We would still store strings as
UTF-8. gnulib contains various helpers to convert to/from utf-8/ucs4
and to test various unicode properties of codepoints. I'll ask on the
gcc mailinglist whether to use the C++ frontend support or import the
gnulib helpers.

> >> - rust-macro-expand tries to handle both macros and attributes, is
> >>  this by design?  Should we handle different passes for different
> >>  (inert or not) attributes that run before or after macro expansion?
> > As for macro and cfg expansion Joel some stuff already in place but i do
> > think they need to be separated into distinct passes which would be a
> > good first start with the expand folder.
> 
> That is a good question. Technically, rust-macro-expand only handles cfg
> expansion at the moment. You can read and discuss more about that here:
> https://github.com/Rust-GCC/gccrs/issues/563

I have to think about whether it makes sense to handle the cfg
attribute and the !cfg macro rules in hte same pass/expansion. The
!cfg macro seems so simple it could be handled immediately by the
parser since it only relies on the compiler/host attributes and simply
generates a true or false token.

In general it seems attribute expansion cannot be simply done by one
AttributeVisitor pass because the effect can be at different stages of
parsing (and they can even affect what the lexer accepts -
e.g. whether identifiers as unicode strings are accepted). For example
the various lint attributes can warn/error/etc when lowering the final
AST (CamelCaseStructs for example), after type checking or after
lifeness analysis. So maybe we need to design a pass for each
different attribute and not try to combine them (except maybe to
recognize and validate the attribute syntax).

Cheers,

Mark