Re: [PATCH] POSIX locale covers every byte [BZ# 29511]

[Florian Weimer <fweimer@redhat.com>]

* наб via Libc-alpha:

> This is a trivial patch, largely duplicating the extant ASCII code
>
> There are two user-facing changes:
>   * nl_langinfo(CODESET) is "POSIX" instead of "ANSI_X3.4-1968"
>   * mbrtowc() and friends return b if b <= 0x7F else <UDF00>+b
>
> Since Issue 7 TC 2/Issue 8, the C/POSIX locale, effectively:
>   (a) is 1-byte, stateless, and contains 256 characters
>   (b) which collate in byte order
>   (c) the first 128 characters are equivalent to ASCII (like previous)
> cf. https://www.austingroupbugs.net/view.php?id=663 for a summary of
> changes to the standard;
> in short, this means that mbrtowc() must never fail and must return
>   b if b <= 0x7F else ab+c for all bytes b
>   where c is some constant >=0x80
>     and a is a positive integer constant
>
> By strategically picking c=<UDF00> we land at the tail-end of the
> Unicode Low Surrogate Area at DC00-DFFF, described as
>   > Isolated surrogate code points have no interpretation;
>   > consequently, no character code charts or names lists
>   > are provided for this range.
> and match musl

We don't match Python and its surrogateescape encoding (PEP 838).  It
maps invalid bytes in the 0x80…0xff range to U+DC80…U+DCFF.  It may make
more sense to align with that.

What worries me is that this effectively closes the door for using UTF-8
(or some variant, such as Python's) with the C locale.  I used to assume
that POSIX allows that, but they now say this was just a mistake.

Anyway, regarding mechanics, we'll need a new localedata/charmaps/POSIX
charmap, I think.  This charmap then can be tested against the gconv
converter.

You should put the new converters into a separate file (not
iconv/gconv_simple.c), then the s390x version will use that
automatically.
> diff --git a/localedata/locales/POSIX b/localedata/locales/POSIX
> index 7ec7f1c577..fc34a6abc1 100644
> --- a/localedata/locales/POSIX
> +++ b/localedata/locales/POSIX
> @@ -97,6 +97,20 @@ END LC_CTYPE
>  LC_COLLATE
>  % This is the POSIX Locale definition for the LC_COLLATE category.

Isn't this just the C locale?  We don't have a separate file for that.

> diff --git a/wcsmbs/wcsmbsload.c b/wcsmbs/wcsmbsload.c
> index 0f0f55f9ed..f87099bcf5 100644
> --- a/wcsmbs/wcsmbsload.c
> +++ b/wcsmbs/wcsmbsload.c
> @@ -33,10 +33,10 @@ static const struct __gconv_step to_wc =
>    .__shlib_handle = NULL,
>    .__modname = NULL,
>    .__counter = INT_MAX,
> -  .__from_name = (char *) "ANSI_X3.4-1968//TRANSLIT",
> +  .__from_name = (char *) "POSIX",
>    .__to_name = (char *) "INTERNAL",
> -  .__fct = __gconv_transform_ascii_internal,
> -  .__btowc_fct = __gconv_btwoc_ascii,
> +  .__fct = __gconv_transform_posix_internal,
> +  .__btowc_fct = __gconv_btwoc_posix,
>    .__init_fct = NULL,
>    .__end_fct = NULL,
>    .__min_needed_from = 1,
> @@ -53,8 +53,8 @@ static const struct __gconv_step to_mb =
>    .__modname = NULL,
>    .__counter = INT_MAX,
>    .__from_name = (char *) "INTERNAL",
> -  .__to_name = (char *) "ANSI_X3.4-1968//TRANSLIT",
> -  .__fct = __gconv_transform_internal_ascii,
> +  .__to_name = (char *) "POSIX",
> +  .__fct = __gconv_transform_internal_posix,
>    .__btowc_fct = NULL,
>    .__init_fct = NULL,
>    .__end_fct = NULL,

This makes the comment on __wcsmbs_gconv_fcts_c in the same file
obsolete.

Thanks,
Florian