Re: Implementing Universal Character Names in identifiers

[loewis@informatik.hu-berlin.de (Martin v. Löwis)]

Zack Weinberg <zack@codesourcery.com> writes:

> http://gcc.gnu.org/projects/cpplib.html#charset contains some
> discussion of the plan - comments would be appreciated.

Sounds all good. For mangling, I'd take a step back, though:
if the assembler does not support UTF-8, you lose.

As for Java: gcj mangles, say, \u0388 as __U388_.

> What you wrote in response to this is interesting but doesn't
> address the issue of Unicode normalization of identifiers.  It
> sounds more like an extended discussion of the previous point.  I'm
> talking about the process described in UAX 15
> (http://www.unicode.org/unicode/reports/tr15/) and in particular
> annex 7 of that document ("Programming Language Identifiers").

I see. For the characters allowed in C99, normalization is almost a
non-issue, since nearly every identifier will already be in NFC.

One exception are 20 characters which have a different canonical
equivalent (e.g. U+1f71 or U+2126); that might be a defect in ISO TR
10176 (they should not be allowed in identifiers - even though TR#15
allows them as well).

The other exception are 80 characters which have a non-zero combining
class; those would might need to be re-ordered under a normal form.

I agree with Joseph that we are not entitled to perform normalization
of the identifiers - there is nothing in the standard that says that
\u1f71 is the same identifier as \u03ac.

So *if* normalization matters, I think we should require the input to
be already normalized, and refuse compilation if it isn't. This could
be accomplished with a small database: we need to ban additional
characters, and we need to record the combining class for those 80
combining characters.

> Ugh.  IMO, this is a defect in both standards - they should simply
> reference UAX15a7 and be done with it.  It's been around since 1998,
> so they don't really have an excuse for not using it.

I completely disagree. The real problem with normalization (until
Unicode 3.1, 2001-05-16) is that it depends on the version of the
Unicode database: different normalization algorithms might normalize
the same string into different code point sequences. This is really
bad, and has been fixed, by setting Unicode 3.1 as the composition
version.

Even with that change, C++ would *still* have needed to pick a
particular version of the Unicode database, or else programs using
extended identifiers would not be portable across conforming
implementations.

So I think the approach of ISO TR 10176 is really sensible: it is
*better* than UAX15a7, as it gives a precise guideline, instead of a
wishy-washy one.

It is unfortunate that C++ would use a draft of that, but there is
really nobody to blame for that, I guess.

>  - GCC enforces the precise lists in C99 and C++98 only in -pedantic
>    mode.

I think GCC should be really careful when extending the language, and
there should be really good reasons for doing so. In the specific
case, I would require a real user with a real need before accepting
more characters than mandated by the language spec.

Merely saying that "C++ does not allow the FEMININE ORDINAL INDICATOR"
is not good enough. The user would need to say: I want the identifier
Internet\u00aa - which won't happen, because (to my understanding),
the FEMININE ORDINAL INDICATOR only applies to numbers, and it could
not be used there, anyway. I believe the same holds for many of the
other characters that would be allowed under UAX15a7 but are currently
banned - nobody would use them in identifiers even if they could.

So I very much doubt that the need for more characters than allowed in
C++98 shows up in the near time. When it does, the standards might
have been revised, so we just update the implementation. If some users
do have a real need, we can still reconsider, and maybe incorporate
the Unicode 4.5 database (in 2006).

> I am not sure about "few" some years down the road, when people start
> _using_ the ability to write identifiers in their own languages.  In
> any case, using "fwrite(ptr, 1, 1, file)" is just silly when
> "putc(*ptr, file)" will do.

Oops, yes. On the general remark: I'm not sure how to significantly
improve performance. I think cpplib should output \u00c0 if the input
was \u00c0 - and perhaps even if the input was in a native character
set. So direct copying to the output will not be appropriate.

Regards,
Martin