Re: [PATCH v6 2/2] POSIX locale covers every byte [BZ# 29511]

[Florian Weimer <fweimer@redhat.com>]

* Florian Weimer:

> * наб:
>
>> This is a logistically trivial patch,
>> largely duplicating the extant ASCII code with the error path changed
>
> I wouldn't say it's trivial in the commit message. 8-)
>
>> There are two user-facing changes:
>>   * nl_langinfo(CODESET) is "POSIX" instead of "ANSI_X3.4-1968"
>>   * mbrtowc() and friends return b if b <= 0x7F else <UDF00>+b
>>
>> Since Issue 7 TC 2/Issue 8, the C/POSIX locale, effectively,
>>   (a) is 1-byte, stateless, and contains 256 characters
>>   (b) they collate in byte order
>>   (c) the first 128 characters are equivalent to ASCII (like previous)
>> cf. https://www.austingroupbugs.net/view.php?id=663 for a summary of
>> changes to the standard;
>> in short, this means that mbrtowc() must never fail and must return
>>   b if b <= 0x7F else ab+c for all bytes b
>>   where c is some constant >=0x80
>>     and a is a positive integer constant
>>
>> By strategically picking c=<UDF00> we land at the tail-end of the
>> Unicode Low Surrogate Area at DC00-DFFF, described as
>>   > Isolated surrogate code points have no interpretation;
>>   > consequently, no character code charts or names lists
>>   > are provided for this range.
>> and match musl
>
> Sadly this doesn't match Python and PEP 540:
>
>>>> b'\x80'.decode('UTF-8', errors='surrogateescape')
> '\udc80'
>
> I believe the implementation translates this to 0xDF80 instead.
>
> Not sure what is more important here, musl compatibility or Python
> compatibility.  Cc:ing Victor in case he as comments.  I should probably
> ask on the musl list as well as how this divergence came to pass.

Raised on the musl list here:

  Choice of wchar_t mapping for non-ASCII bytes in the POSIX locale
  <https://www.openwall.com/lists/musl/2022/11/10/1>

> This change definitely needs a NEWS entry.

(With this I meant the change overall, not the encoding.)

Thanks,
Florian