Re: [PATCH v4 4/4] Add generic C.UTF-8 locale (Bug 17318)

[Carlos O'Donell <carlos@redhat.com>]

On 4/29/21 10:13 AM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> We add a new C.UTF-8 locale.  This locale is not builtin to glibc, but
>> is provided as a distinct locale.  The locale provides full support
>> for UTF-8 and this includes full code point sorting via collation
>> (excludes surrogates).  Unfortuantely given the present implementation
>> in glibc this results in 28MiB of LC_COLLATE data for all possible
>> Unicode code points.  Future improvements may reduce this size. Such
>> improvements likely require a shortcut for the collation data that
>> relies on C.UTF-8 single-byte sorting being equivalent to strcmp.
>>
>> The new locale is NOT added to SUPPORTED.  Minimal test data for
>> specific code points (minus those not supported by collate-test) is
>> provided in C.UTF-8.in, and this verifies code point sorting is
>> working reasonably across the range.
>>
>> The next step is to reduce LC_COLLATE to a manageable size before we
>> enable the locale in SUPPORTED. Fully testing C.UTF-8 collation can
>> add ~5-7 minutes to the locale testing (collate-test, and xfrm-test
>> twice) so we don't enable full testing of all code points until we can
>> parallelize the sort-test test. Testing sort-test with C.UTF-8 minimal
>> test data passes cleanly.
> 
> Can you compare this locale with what is in Fedora and Debian, for the
> non-collaction/CTYPE aspects?

Oh, doing this review in more detail for you found a potential defect.
Thank you for encouraging a more detailed review.

I see that C has the first work day as Monday, but in C.UTF-8 we have
switched to Sunday, possibly by accident, and my initial review didn't
catch this. I'll spin a v5 which is also going to be smaller after this
patch: https://sourceware.org/pipermail/libc-alpha/2021-April/125595.html.

Debian (sid, 2.31-11) vs Upstream:
- LC_IDENTIFICATION, contains old date, maintainer @debian address etc.
  - No substantive differences.
- LC_CTYPE, includes "translit_combining" which is wrong for a C locale IMO.
  - Upstream C.UTF-8 includes no transliteration, all characters pass
    through because UTF-8 supports all such characters.
- LC_COLLATE, split ranges differently, *includes* surrogates, uses UNDEFINED correctly.
  - Upstream C.UTF-8 *excludes* surrogates, but otherwise covers same set.
- LC_MONETARY, copy "POSIX"
  - Upstream C.UTF-8 explicitly defines fields, difference in 'negative_sign'
    where upstream will use "-" and POSIX uses "". This aligns with existing builtin
    C locale.
- LC_NUMERIC, copy "POSIX"
  - Upstream C.UTF-8 explicitly defines fields, no difference from POSIX.
- LC_TIME, first_workday is 2 (otherwise the same)
  - Upstream C.UTF-8 set first_workday to 1, this is a bug my patch.
- LC_MESSAGES, only defines yesexpr and noexpr.
  - Upstream C.UTF-8 defines yesexpr, noexpr, yesstr and nostr. Superset of data.
- LC_PAPER, copy "i18n"
  - Upstream C.UTF-8 explicitly defines fields, no differences.
- LC_NAME, copy "i18n"
  - Upstream C.UTF-8 explicitly defines fields, no differences.
- LC_ADDRESS, copy "i18n"
  - Upstream C.UTF-8 explicitly defines fields, no differences.
- LC_TELEPHONE, defines tel_int_fmt.
  - Upstream C.UTF-8 explicitly defines tel_int_fmt, no differences.
- LC_MEASUREMENT, copy "i18n"
  - Upstream C.UTF-8 explicitly defines measurement, no differences.

> Are there other distributions which ship a downstream C.UTF-8 locale?

Yes, Gentoo. I spoke to Andreas Huettel from the gentoo-toolchain team
and they are using Mike Fabian's original C.UTF-8 which is harmonized
and identical (including the first_workday bug) to what I'm proposing.

I think it would be safe for Debian, Ubuntu, Gentoo, Fedora, CentOS Stream,
and RHEL to switch to the new C.UTF-8 locale from upstream.

-- 
Cheers,
Carlos.