Re: [PATCH v3] Add new C.UTF-8 locale (Bug 17318)

[Florian Weimer <fweimer@redhat.com>]

* Carlos O'Donell via Libc-alpha:

> diff --git a/locale/programs/charmap.c b/locale/programs/charmap.c
> index 3d51e702dc..77085cff72 100644
> --- a/locale/programs/charmap.c
> +++ b/locale/programs/charmap.c
> @@ -49,7 +49,7 @@ static void new_width (struct linereader *cmfile, struct charmap_t *result,

> +  /* POSIX explicitly requires that ellipsis processing do the
> +     following: "Bytes shall be treated as unsigned octets, and carry
> +     shall be propagated between the bytes as necessary to represent the
> +     range."  It then goes on to say that such a declaration should
> +     never be specified because it creates NULL bytes.  Therefore we

NUL or null, I think.

> +     error on this condition (see charmap_new_char).  However this still
> +     leaves a problem for encodings which use less than the full 8-bits,
> +     like UTF-8, and in such encodings you can use an ellipsis to
> +     silently and accidentally create invalid ranges.  In UTF-8 you have
> +     only the first 6-bits of the first byte and if your ellipsis covers

UTF-8 is variable length even in the leader byte, so “only the first
6-bits of the first byte” seems wrong.

> +/* This function takes the Unicode code point CP and encodes it into
> +   a UTF-8 byte stream that must be NBYTES long and is stored into
> +   the unsigned character array at BYTES.
> +
> +   If CP requires more than NBYTES to be encoded then we return an
> +   error of -1.
> +
> +   If CP is not within any of the valid Unicode code point ranges
> +   then we return an error of -2.
> +
> +   Otherwise we return the number of bytes encoded.  */
> +static int
> +output_utf8_bytes (unsigned int cp, size_t nbytes, unsigned char *bytes)
> +{
> +  /* We need at least 1 byte.  */
> +  if (nbytes < 1)
> +    return -1;
> +
> +  /* One byte range.  */
> +  if (cp >= 0x0 && cp <= 0x7f)
> +    {
> +      bytes[0] = cp & 0x7f;
> +      return 1;
> +    }

0x7f is superfluous and confusing here, as discussed before.

> diff --git a/localedata/charmaps/UTF-8 b/localedata/charmaps/UTF-8
> index 8cce47cd97..c70d359744 100644
> --- a/localedata/charmaps/UTF-8
> +++ b/localedata/charmaps/UTF-8
> @@ -895,12 +895,14 @@ CHARMAP

> +<UD800>..<UDB7F> /xed/xa0/x80 <Non Private Use High Surrogate>
> +<UDB80>..<UDBFF> /xed/xae/x80 <Private Use High Surrogate>
> +<UDC00>..<UDFFF> /xed/xb0/x80 <Low Surrogate>
> +<UE000>..<UF8FF> /xee/x80/x80 <Private Use>

Technically this isn't right.  We don't want mappings for those
characters because it might introduce in other locale files that use
those characters.  But may be just need to be careful.

I'm surprised that this doesn't lead to testsuite failures because it's
inconsistent with the gconv converters.  Maybe we don't use this
anywhere?

The other invalid-ish Unicode codepoints (U+FFFE, U+FFFF) are actually
valid UTF-8 and handled by gconv, so including them seems okay.

> diff --git a/localedata/locales/C b/localedata/locales/C
> new file mode 100644
> index 0000000000..418e7c90a5
> --- /dev/null
> +++ b/localedata/locales/C
> @@ -0,0 +1,192 @@

> +% One rule, sort forward, for all code points to give code point
> +% order sorting for Unicode.
> +LC_COLLATE
> +order_start forward
> +<U00000000>
> +..
> +<U0000007F>
> +<U00000080>
> +..
> +<U000007FF>
> +<U00000800>
> +..
> +<U0000FFFF>
> +<U00010000>
> +..
> +<U0010FFFF>
> +UNDEFINED
> +order_end
> +END LC_COLLATE

Why are multiple ranges required here?

> diff --git a/localedata/locales/i18n_ctype b/localedata/locales/i18n_ctype
> index c63e0790fc..c92bb95148 100644
> --- a/localedata/locales/i18n_ctype
> +++ b/localedata/locales/i18n_ctype
> @@ -26,7 +26,7 @@ fax       ""
>  language  ""
>  territory "Earth"
>  revision  "13.0.0"
> -date      "2020-06-25"
> +date      "2021-02-17"
>  category  "i18n:2012";LC_CTYPE
>  END LC_IDENTIFICATION

Those date changes seem spurious.  Is this no-op file regeneration
really needed?

> diff --git a/localedata/unicode-gen/utf8_gen.py b/localedata/unicode-gen/utf8_gen.py
> index 899840923a..42fc5efcb9 100755
> --- a/localedata/unicode-gen/utf8_gen.py
> +++ b/localedata/unicode-gen/utf8_gen.py

>  def convert_to_hex(code_point):
>      '''Converts a code point to a hexadecimal UTF-8 representation
> +    like /x**/x**/x** without using any python library functions.
> +    This avoids problems with the encode function, including an
> +    inability to output the surrogate code points.

You can use chr(code_point).encode('UTF-8', 'surrogatepass') and the
Python encoder.

I reviewed the other changes and spot-checked the generated charmap.
Those parts look okay.

The question is whether we actually need a UTF-8 charmap.  If not, we
can teach charmap.c to generate the UTF-8 data on the fly in
charmap_find_value and charmap_find_symbol.  But I consider this part of
the data representation changes we discussed earlier.

Thanks,
Florian