[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com
             Status|UNCONFIRMED                 |RESOLVED
           See Also|                            |http://bugs.debian.org/1017
                   |                            |852
              Flags|                            |security-
         Resolution|---                         |INVALID

--- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
I don't see how this is a bug, sorry.

There is no well-defined conversion from US-ASCII to the internal UCS-4
encoding, so reporting an error for input bytes that do not have a mapping
makes sense.

If another glibc function does not behave as expected (fnmatch?), please file a
separate bug for that.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[nabijaczleweli at nabijaczleweli dot xyz]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

--- Comment #2 from наб <nabijaczleweli at nabijaczleweli dot xyz> ---
I don't see how that's relevant? The POSIX locale is /not/ US-ASCII: it's
/some/ encoding that contains the PCS and NPCS as the first 128 characters
(this coincides with 7-bit ASCII) /and/ contains mappings from all 256 single
bytes to a character (and back).

This is very obviously listed in the first hunk, which I'll reiterate again:
POSIX.1, Issue 7 TC 2, XBD, 6.2 "Character Encoding" starts with:
  > The POSIX locale shall contain 256 single-byte characters including the
characters in Portable Character Set and Non-Portable Control Characters, which
have the properties listed in LC_CTYPE. 

The bug here isn't "glibc doesn't decode ASCII [0xfa, 0] to my favourite
character", it's "glibc treats POSIX locale as-if it were ASCII" – which it's
not.

From your response it appears you also haven't consulted bug 663 (an oversight
on my part, it seems I only linked it directly in the Debian bug, and here
indirectly): https://www.austingroupbugs.net/view.php?id=663 – it contains a
summary of changes that make the Issue 7 TC 2/Issue 8 POSIX locale
as-described.

POSIX.1, Issue 7 TC 2, XSH, mbrtowc(), ERRORS:
-- >8 --
 [EILSEQ]
    An invalid character sequence is detected. [CX]In the POSIX locale an
EILSEQ error cannot occur since all byte values are valid characters.[/CX]
-- >8 --
(the same text is seen in mbrlen(), mbstowcs(), &c., &c.).

POSIX.1, Issue 7 TC 2, XRAT, XBD, A.6.2 Character Encoding:
-- >8 --
Earlier versions of this standard did not state the requirement that the POSIX
locale contains 256 single-byte characters. This was an oversight; the
intention was always that the POSIX locale should have an 8-bit-clean
single-byte encoding.
-- >8 --

I hope this convinces you that this is not RESOLVED INVALID, but, indeed, a
conformance error.

A different, less messy than the one in my patch (or just making it a KOI-8
variant, which is admittedly the worst one so far), solution I arrived at today
would be to map [0, 0x7F] to [U+0, U+7F] as is done currently and B[0x80, 0xFF]
to C[U+FFFFFF80, U+FFFFFFFF] or C[U+FFFFFF00, U+FFFFFF7F] — I don't think
Unicode will reach that high any-time soon – or to pick a PUA and map to
C[U+100000, U+10007F]/C[U+E080, U+E0FF]/you get the point here – "using glibc"
reasonably fits as a "private agreement between collaborating users".

Changing the CODESET is also, presumably, a given: naively, to "POSIX", I
guess?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
         Resolution|INVALID                     |---

--- Comment #3 from Florian Weimer <fweimer at redhat dot com> ---
Sorry, I had assumed that C was specifically worded in such a way that
implementations could default to UTF-8 if they wanted, and POSIX kept this
possibility.

Not sure what to do about this issue. We had this idea to switch to the UTF-8
character set by default for a long time. It's strange that POSIX no longer
allows that.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[nabijaczleweli at nabijaczleweli dot xyz]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

--- Comment #4 from наб <nabijaczleweli at nabijaczleweli dot xyz> ---
Out of curiosity, I checked what musl does (represented here by current Void,
musl-1.1.24_10):
-- >8 --
bash-5.1# ./b | paste - - - - - - - -
CODESET=ASCII
00: 0, 0        01: 1, 1        02: 1, 2        03: 1, 3        04: 1, 4       
05: 1, 5        06: 1, 6        07: 1, 7
08: 1, 8        09: 1, 9        0a: 1, a        0b: 1, b        0c: 1, c       
0d: 1, d        0e: 1, e        0f: 1, f
10: 1, 10       11: 1, 11       12: 1, 12       13: 1, 13       14: 1, 14      
15: 1, 15       16: 1, 16       17: 1, 17
18: 1, 18       19: 1, 19       1a: 1, 1a       1b: 1, 1b       1c: 1, 1c      
1d: 1, 1d       1e: 1, 1e       1f: 1, 1f
20: 1, 20       21: 1, 21       22: 1, 22       23: 1, 23       24: 1, 24      
25: 1, 25       26: 1, 26       27: 1, 27
28: 1, 28       29: 1, 29       2a: 1, 2a       2b: 1, 2b       2c: 1, 2c      
2d: 1, 2d       2e: 1, 2e       2f: 1, 2f
30: 1, 30       31: 1, 31       32: 1, 32       33: 1, 33       34: 1, 34      
35: 1, 35       36: 1, 36       37: 1, 37
38: 1, 38       39: 1, 39       3a: 1, 3a       3b: 1, 3b       3c: 1, 3c      
3d: 1, 3d       3e: 1, 3e       3f: 1, 3f
40: 1, 40       41: 1, 41       42: 1, 42       43: 1, 43       44: 1, 44      
45: 1, 45       46: 1, 46       47: 1, 47
48: 1, 48       49: 1, 49       4a: 1, 4a       4b: 1, 4b       4c: 1, 4c      
4d: 1, 4d       4e: 1, 4e       4f: 1, 4f
50: 1, 50       51: 1, 51       52: 1, 52       53: 1, 53       54: 1, 54      
55: 1, 55       56: 1, 56       57: 1, 57
58: 1, 58       59: 1, 59       5a: 1, 5a       5b: 1, 5b       5c: 1, 5c      
5d: 1, 5d       5e: 1, 5e       5f: 1, 5f
60: 1, 60       61: 1, 61       62: 1, 62       63: 1, 63       64: 1, 64      
65: 1, 65       66: 1, 66       67: 1, 67
68: 1, 68       69: 1, 69       6a: 1, 6a       6b: 1, 6b       6c: 1, 6c      
6d: 1, 6d       6e: 1, 6e       6f: 1, 6f
70: 1, 70       71: 1, 71       72: 1, 72       73: 1, 73       74: 1, 74      
75: 1, 75       76: 1, 76       77: 1, 77
78: 1, 78       79: 1, 79       7a: 1, 7a       7b: 1, 7b       7c: 1, 7c      
7d: 1, 7d       7e: 1, 7e       7f: 1, 7f
80: 1, df80     81: 1, df81     82: 1, df82     83: 1, df83     84: 1, df84    
85: 1, df85     86: 1, df86     87: 1, df87
88: 1, df88     89: 1, df89     8a: 1, df8a     8b: 1, df8b     8c: 1, df8c    
8d: 1, df8d     8e: 1, df8e     8f: 1, df8f
90: 1, df90     91: 1, df91     92: 1, df92     93: 1, df93     94: 1, df94    
95: 1, df95     96: 1, df96     97: 1, df97
98: 1, df98     99: 1, df99     9a: 1, df9a     9b: 1, df9b     9c: 1, df9c    
9d: 1, df9d     9e: 1, df9e     9f: 1, df9f
a0: 1, dfa0     a1: 1, dfa1     a2: 1, dfa2     a3: 1, dfa3     a4: 1, dfa4    
a5: 1, dfa5     a6: 1, dfa6     a7: 1, dfa7
a8: 1, dfa8     a9: 1, dfa9     aa: 1, dfaa     ab: 1, dfab     ac: 1, dfac    
ad: 1, dfad     ae: 1, dfae     af: 1, dfaf
b0: 1, dfb0     b1: 1, dfb1     b2: 1, dfb2     b3: 1, dfb3     b4: 1, dfb4    
b5: 1, dfb5     b6: 1, dfb6     b7: 1, dfb7
b8: 1, dfb8     b9: 1, dfb9     ba: 1, dfba     bb: 1, dfbb     bc: 1, dfbc    
bd: 1, dfbd     be: 1, dfbe     bf: 1, dfbf
c0: 1, dfc0     c1: 1, dfc1     c2: 1, dfc2     c3: 1, dfc3     c4: 1, dfc4    
c5: 1, dfc5     c6: 1, dfc6     c7: 1, dfc7
c8: 1, dfc8     c9: 1, dfc9     ca: 1, dfca     cb: 1, dfcb     cc: 1, dfcc    
cd: 1, dfcd     ce: 1, dfce     cf: 1, dfcf
d0: 1, dfd0     d1: 1, dfd1     d2: 1, dfd2     d3: 1, dfd3     d4: 1, dfd4    
d5: 1, dfd5     d6: 1, dfd6     d7: 1, dfd7
d8: 1, dfd8     d9: 1, dfd9     da: 1, dfda     db: 1, dfdb     dc: 1, dfdc    
dd: 1, dfdd     de: 1, dfde     df: 1, dfdf
e0: 1, dfe0     e1: 1, dfe1     e2: 1, dfe2     e3: 1, dfe3     e4: 1, dfe4    
e5: 1, dfe5     e6: 1, dfe6     e7: 1, dfe7
e8: 1, dfe8     e9: 1, dfe9     ea: 1, dfea     eb: 1, dfeb     ec: 1, dfec    
ed: 1, dfed     ee: 1, dfee     ef: 1, dfef
f0: 1, dff0     f1: 1, dff1     f2: 1, dff2     f3: 1, dff3     f4: 1, dff4    
f5: 1, dff5     f6: 1, dff6     f7: 1, dff7
f8: 1, dff8     f9: 1, dff9     fa: 1, dffa     fb: 1, dffb     fc: 1, dffc    
fd: 1, dffd     fe: 1, dffe     ff: 1, dfff
-- >8 --

Starting at U+DF80 falls into the Low Surrogate Area at DC00-DFFF, which the
chart describes as:
> Isolated surrogate code points have no interpretation; consequently, no character code charts or names lists are provided for this range.

Which is perfect. I'm attaching a diff that adds a POSIX charset that does the
same, and uses it for default/C/POSIX locale, alongside some new tests.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[nabijaczleweli at nabijaczleweli dot xyz]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

наб <nabijaczleweli at nabijaczleweli dot xyz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #14286|0                           |1
        is obsolete|                            |

--- Comment #5 from наб <nabijaczleweli at nabijaczleweli dot xyz> ---
Created attachment 14305
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14305&action=edit
CHARSET=POSIX for default/C/POSIX locale, [0x80, 0xFF] in [U+DF80, U+DFFF]

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[bruno at clisp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

Bruno Haible <bruno at clisp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bruno at clisp dot org

--- Comment #6 from Bruno Haible <bruno at clisp dot org> ---
I think this is a duplicate of bug #19932.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/29511] default/C/POSIX locale is 7-bit (127 characters), must be 8-bit (256 characters) since POSIX Issue 7 TC2/Issue 8

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29511

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.