Re: revised proposal for GCC and non-Ascii source files

[Paul Eggert <eggert@twinsun.com>]

   Date: Mon, 04 Jan 1999 16:15:46 -0500
   From: Zack Weinberg <zack@rabi.columbia.edu>

   The preprocessor may have trouble with native extended chars as
   first characters, but if we can get isalpha() to cooperate, it
   should work.

There should be no trouble with isalpha cooperating, since GCC
shouldn't use isalpha to detect identifiers.

To detect C alphanumerics, GCC must use its own function or table.
GCC shouldn't use isalpha unless we want it to detect alphanumeric
bytes in the current locale, which is normally not what is wanted.
It might be OK for GCC to use isalpha for some obscure low-level stuff,
e.g. detecting whether an MS-DOS file name has a drive specifier.
But GCC shouldn't use isalpha for detecting identifiers, because
isalpha doesn't work with multibyte chars.

Parts of GCC already do the right thing here.  E.g. cccp.c uses
is_idchar instead of isalpha.  Some other parts of GCC use isalpha
where they shouldn't, though, and we'll have to fix these parts if we
want gcc to work with multibyte chars properly.

I already did most of this job for the back end and the C front end,
in the internationalization patch that has been applied to GCC2:

ftp://alpha.gnu.org/gnu/testgcc-980705-intl.patch.gz

However, this patch didn't cover the C++ front end, and there are
probably some other places that I missed that we'll just have to find
if and when they come up.  Also, this patch hasn't been fully applied
to EGCS yet, and there's undoubtedly some integration work needed
there since GCC2 and EGCS have diverged.  At some point I'd like to
get it working properly with EGCS as well as GCC2.  I understand that
a merge is in progress and so I'll wait for it to finish before
hacking any further on isalpha problems.

   This raises the issue of how we tell native extended character X
   from native ASCII character %.  I'm beginning to suspect we need
   the more general locale information, not just the charset.

I reluctantly agree.  Another reason we need the locale info is
because iconv (which needs only the charset) doesn't let us determine
character boundaries reliably; we need mbrlen and mbsinit for that,
and they need the LC_CTYPE locale.

This is why I'm changing the word `charset' to `ctype' in my next
version of the proposal for non-Ascii source files.  GCC will need to
know the LC_CTYPE locale (which I am calling the `ctype'), and from
that GCC can use nl_langinfo (CODESET) to determine the charset.