[Bug localedata/20865] New: iconv: cp950 does not contain EUDC/PUA mappings

[Bug localedata/20865] New: iconv: cp950 does not contain EUDC/PUA mappings

[arthur200126 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20865

            Bug ID: 20865
           Summary: iconv: cp950 does not contain EUDC/PUA mappings
           Product: glibc
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: arthur200126 at gmail dot com
                CC: libc-locales at sourceware dot org
  Target Milestone: ---

Microsoft's cp950 mapping contains sequential mappings from Big5's Extended
User-defined Characters (EUDC) to Unicode PUA. Such mappings are used by a
number of Big5 extensions, including HKSCS which uses these PUA code points
when a character is not yet available in the target UCS version.

The following sessions come from GNU bash running in a UTF-8 console. $''
denotes bash's ANSI C-style quoting, where \xhh generates a raw hex byte and
\uhhhh generates the representation of U+hhhh under current locale.

Currently glibc's cp950 implementation does not contain these mappings:

# iconv (Ubuntu GLIBC 2.23-0ubuntu4) 2.23
ubuntu$ iconv -f cp950 -t utf-32le <<< $'\x81\x40' | hexdump -C
iconv: illegal input sequence at position 0
ubuntu$ iconv -t cp950 -f utf-8 <<< $'\ueeb8' | hexdump -C
iconv: illegal input sequence at position 0

The desired behavior for decoding can be seen in libiconv:

# iconv (GNU libiconv 1.14)
cygwin$ iconv -f cp950 -t utf-32le <<< $'\x81\x40' | hexdump -C
00000000  b8 ee 00 00 0a 00 00 00                           |........|
00000008

Note that libiconv is not interested in doing the reverse:

cygwin$ iconv -t cp950 -f utf-8 <<< $'\ueeb8' | hexdump -C
iconv: illegal input sequence at position 0

libiconv's mapping:
http://git.savannah.gnu.org/cgit/libiconv.git/tree/lib/cp950.h#n72

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/20865] charmaps: cp950 does not contain EUDC/PUA mappings

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20865

Mike Frysinger <vapier at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|iconv: cp950 does not       |charmaps: cp950 does not
                   |contain EUDC/PUA mappings   |contain EUDC/PUA mappings

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

--- Comment #1 from Mike Frysinger <vapier at gentoo dot org> ---
we currently alias CP950 to BIG5.  i don't think we want to add all these
mappings to BIG5 since it explicitly carves out that spaces as "reserved":
  https://en.wikipedia.org/wiki/Big5#A_more_detailed_look_at_the_organization

which means we need to copy BIG5 to CP950 and add the MS extensions.  then drop
the alias of CP950->BIG5.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/20865] charmaps: cp950 does not contain EUDC/PUA mappings

[arthur200126 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20865

--- Comment #2 from Mingye Wang <arthur200126 at gmail dot com> ---
> I don't think we want to add all these mappings to BIG5 since it
> explicitly carves out that spaces as "reserved":
If you consider how GB 18030's user-defined areas are mapped, "reserved for
user-defined characters" may be acceptable for PUA. The "reserved, not for
user-defined" part is not mapped to PUA in cp950 and big5-2003.


> which means we need to copy BIG5 to CP950 and add the MS extensions
I went back to libiconv's charts and read a few lines up too see all of MS's
modifications & extensions; now it does sound messy enough for a split. (0xA1F2
and 0xA1F3 are entertainingly weird.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/20865] charmaps: cp950 does not contain EUDC/PUA mappings

[arthur200126 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20865

--- Comment #3 from Mingye Wang <arthur200126 at gmail dot com> ---
> now it does sound messy enough for a split.

Hmm, glibc is already using the "MICSFT/WINDOWS" cp950 definition for BIG5's
charmap. So it's not that bad...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/20865] charmaps: cp950 does not contain EUDC/PUA mappings

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20865

Mike FABIAN <maiku.fabian at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maiku.fabian at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/20865] charmaps: cp950 needs Windows EUDC/PUA mappings

[arthur200126 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20865

Mingye Wang <arthur200126 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|charmaps: cp950 does not    |charmaps: cp950 needs
                   |contain EUDC/PUA mappings   |Windows EUDC/PUA mappings

-- 
You are receiving this mail because:
You are on the CC list for the bug.