[Bug localedata/11837] GB18030-2005 is not supported!

[Bug localedata/11837] GB18030-2005 is not supported!

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING
                 CC|                            |drepper.fsp at gmail dot
                   |                            |com

--- Comment #1 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-05-09 23:14:45 UTC ---
Well, and where is the data?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |FIXED

--- Comment #2 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-05-17 05:43:14 UTC ---
I've checked in a patch.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |

--- Comment #3 from Andreas Schwab <schwab@linux-m68k.org> 2011-06-14 12:47:04 UTC ---
That doesn't appear to work.

$ printf "\xf0\xa0\xb3\x90\n" | iconv -t gb18030
iconv: illegal input sequence at position 0

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |

--- Comment #5 from Andreas Schwab <schwab@linux-m68k.org> 2011-07-07 06:48:30 UTC ---
GB18030 defines a mapping for *every* Unicode character, even the
unassigned/reserved ones.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #4 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-07-07 03:55:35 UTC ---
(In reply to comment #3)
> That doesn't appear to work.
> 
> $ printf "\xf0\xa0\xb3\x90\n" | iconv -t gb18030
> iconv: illegal input sequence at position 0

That's expected.  Previous mappings were wrong.  The official GB18030 mapping
doesn't define a mapping for U20cd0.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #6 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-07-08 16:47:13 UTC ---
Nog(In reply to comment #5)
> GB18030 defines a mapping for *every* Unicode character, even the
> unassigned/reserved ones.

It says how they would be mapped.  But this is not what converters are supposed
to do.  The only official mappings available don't do that.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |

--- Comment #7 from Andreas Schwab <schwab@linux-m68k.org> 2011-07-11 07:16:22 UTC ---
GB18030 is defined to map every Unicode character.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #8 from Rich Felker <bugdal at aerifal dot cx> 2011-07-16 00:44:36 UTC ---
GB18030 is defined to map not just every Unicode *character*, but every
*Unicode Scalar Value*. That means every number in the ranges 0x0000-0xD7FF and
0xE000-0x10FFFF is mapped. This property is what makes it a true UTF and not
merely a legacy DBCS.

Mr. Drepper, if you claim GB18030 should not successfully map unassigned
codepoints, what about the converters between UTF-8, UTF-16, and UTF-32? Should
they also reject unassigned codepoints? Despite being horribly ugly and having
all the harmful properties of legacy DBCS, GB18030 is a UTF and should be
treated the same as other UTFs.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[an.euroford at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

An Yang <an.euroford at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |an.euroford at gmail dot
                   |                            |com

--- Comment #9 from An Yang <an.euroford at gmail dot com> 2011-08-06 17:16:22 UTC ---
The system can convert or display all of Chinese Characters in Unicode6.0 CJK
Ext-A/B/C/D.

But glibc have a bug related with pinyin sort, it can NOT sort any characters
in CJK Ext-A/B/C/D, it just drop all of them.

I'll file a new bug.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #10 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-10-29 17:18:38 UTC ---
Stop reopening this.  The canonical source for the conversion does exactly what
the glibc code does.  Anything else does not have any value and only creates
problems.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[an.euroford at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

--- Comment #11 from An Yang <an.euroford at gmail dot com> 2011-10-31 03:50:55 UTC ---
Hi Ulrich Drepper,

Take it easy.

I'm sure something is wrong in Fedora/RHEL and any other Linux which use glibc,
please see http://sourceware.org/bugzilla/show_bug.cgi?id=13063, and make
comments there.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[leemars at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

leemars at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |leemars at gmail dot com

--- Comment #12 from leemars at gmail dot com 2011-11-21 14:37:19 UTC ---
Is it possible to rollback the commit ee30c380b8f7c9253c87103c58c5201268d30181
"Update GB18030 to 2005 version"? or maybe consider to cherry-pick the commit
2a57bd797c9a0f9d79436b8960019506c28c5889 "Repair GB18030 charmap" and commit
3d828a61cdc5ccd5e907e880cff45130169a543e "Fix more bugs in GB18030 charmap"? At
least we need a workable version.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[liyangdal at hotmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Li Yang <liyangdal at hotmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
                 CC|                            |liyangdal at hotmail dot
                   |                            |com
         Resolution|FIXED                       |

--- Comment #13 from Li Yang <liyangdal at hotmail dot com> 2011-11-25 17:45:22 UTC ---
In fact, it worked well before this change has been committed.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[bruno at clisp dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Bruno Haible <bruno at clisp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bruno at clisp dot org

--- Comment #14 from Bruno Haible <bruno at clisp dot org> 2012-01-26 16:25:49 UTC ---
As a result of this mess, openSUSE 12.1 is now shipping with yet another
GB18030 converter: the one by Anthony Fok <anthony@thizlinux.com>, 2002.
And it is broken as well: It cannot convert the character
U+C50B HANGUL SYLLABLE SSEUH to GB18030:

$ printf '\x00\x00\xc5\x0B' | LC_ALL=C /usr/bin/iconv -f UCS-4BE -t GB18030 |
od -t x1 | head -n 1
/usr/bin/iconv: illegal input sequence at position 0
0000000

Expected output:

$ printf '\x00\x00\xc5\x0B' | LC_ALL=C /usr/bin/iconv -f UCS-4BE -t GB18030 |
od -t x1 | head -n 1
0000000 83 32 da 36

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

--- Comment #15 from Rich Felker <bugdal at aerifal dot cx> 2012-01-26 17:33:45 UTC ---
> That's expected.  Previous mappings were wrong.  The official GB18030 mapping
> doesn't define a mapping for U20cd0.

This is false. The official GB18030 defines a mapping for every Unicode Scalar
Value, as it is a UTF. Why do you refuse the simple, standards-conformant fix
that would make all of these issues go away?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug localedata/11837] GB18030-2005 is not supported!

[carlos_odonell at mentor dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11837

Carlos O'Donell <carlos_odonell at mentor dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos_odonell at mentor
                   |                            |dot com
         AssignedTo|libc-locales at sources dot |unassigned at sourceware
                   |redhat.com                  |dot org
   Target Milestone|---                         |2.16
              Flags|                            |review?(schwab@linux-m68k.o
                   |                            |rg)

--- Comment #16 from Carlos O'Donell <carlos_odonell at mentor dot com> 2012-05-09 12:34:11 UTC ---
OK, we want to get this fixed for 2.16. Setting milestone.

Andreas, Could you please post your patch to libc-alpha again, we'll have a
quick review and then check it in as incremental progress. I'd like to see 2.16
have better support for GB18030.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.