[PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Mike Frysinger]

Microsoft long ago added a mapping for 0x80 to the Euro sign to their
CP936.  While GBK 1.0 doesn't include this mapping, it is compatible,
and Microsoft and glibc alias the two codepages.  We could split them
apart so GBK wouldn't include the mapping, but that seems like a lot
of work for little gain.

2016-11-25  Mike Frysinger  <vapier@gentoo.org>

	[BZ #20864]
	* localedata/GBK: Map 0x80 to U20AC.
---
 localedata/charmaps/GBK | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/localedata/charmaps/GBK b/localedata/charmaps/GBK
index 4c35f1426c64..4186896f2a9b 100644
--- a/localedata/charmaps/GBK
+++ b/localedata/charmaps/GBK
@@ -3,10 +3,15 @@
 <mb_cur_min> 1
 <comment_char> %
 <escape_char> /
+% https://en.wikipedia.org/wiki/GBK
 
+% https://www-01.ibm.com/software/globalization/ccsid/ccsid936.html
 % alias CP936
+
+% ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit
 % alias MS936
 % alias WINDOWS-936
+
 CHARMAP
 <U0000>     /x00         NULL (NUL)
 <U0001>     /x01         START OF HEADING (SOH)
@@ -136,6 +141,8 @@ CHARMAP
 <U007D>     /x7d         RIGHT CURLY BRACKET
 <U007E>     /x7e         TILDE
 <U007F>     /x7f         DELETE (DEL)
+% This isn't in GBK 1.0, but Microsoft added it to their CP936 page.
+<U20AC>     /x80         EURO SIGN
 
 <U4E02>     /x81/x40     <CJK>
 <U4E04>     /x81/x41     <CJK>
-- 
2.10.2

Re: [PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Florian Weimer]

On 11/25/2016 05:17 PM, Mike Frysinger wrote:
> Microsoft long ago added a mapping for 0x80 to the Euro sign to their
> CP936.  While GBK 1.0 doesn't include this mapping, it is compatible,
> and Microsoft and glibc alias the two codepages.  We could split them
> apart so GBK wouldn't include the mapping, but that seems like a lot
> of work for little gain.
>
> 2016-11-25  Mike Frysinger  <vapier@gentoo.org>
>
> 	[BZ #20864]
> 	* localedata/GBK: Map 0x80 to U20AC.

Looks okay to me.

Thanks,
Florian

Re: [PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Andreas Schwab]

FAIL: iconvdata/tst-tables
failed: ./tst-table.sh /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/ /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/iconvdata/ GBK 

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: [PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Florian Weimer]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

On 11/28/2016 10:23 AM, Andreas Schwab wrote:
> FAIL: iconvdata/tst-tables
> failed: ./tst-table.sh /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/ /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/iconvdata/ GBK

Here's one way to fix it.  Doesn't look really pretty, but it seems that 
this new 0x80 mapping is quite an outlier.

Thanks,
Florian


[-- Attachment #2: gbk.patch --]
[-- Type: text/x-patch, Size: 1571 bytes --]

gconv: Adjust GBK to support the Euro sign

Commit aa4d00ca39e604ac4e9fead401ccd4483e11a281 only updated the
data used by locales.

2016-11-29  Florian Weimer  <fweimer@redhat.com>

	* iconvdata/gbk.c (BODY): Add Euro sign support (both directions).

diff --git a/iconvdata/gbk.c b/iconvdata/gbk.c
index fc32a50..d39e398 100644
--- a/iconvdata/gbk.c
+++ b/iconvdata/gbk.c
@@ -13148,8 +13148,17 @@ static const char __gbk_from_ucs4_tab12[][2] =
       if (__builtin_expect (ch <= 0x80, 0)				      \
 	  || __builtin_expect (ch > 0xfe, 0))				      \
 	{								      \
-	  /* This is illegal.  */					      \
-	  STANDARD_FROM_LOOP_ERR_HANDLER (1);				      \
+	  if (__glibc_likely (ch == 0x80))				      \
+	    {								      \
+	      /* Exception for the Euro sign (see CP936).  */		      \
+	      ch = 0x20AC;						      \
+	      ++inptr;							      \
+	    }								      \
+	  else								      \
+	    {								      \
+	      /* This is illegal.  */					      \
+	      STANDARD_FROM_LOOP_ERR_HANDLER (1);			      \
+	    }								      \
 	}								      \
       else								      \
 	{								      \
@@ -13292,6 +13301,10 @@ static const char __gbk_from_ucs4_tab12[][2] =
 	case 0x2010 ... 0x203b:						      \
 	  cp = __gbk_from_ucs4_tab4[ch - 0x2010];			      \
 	  break;							      \
+	case 0x20AC:							      \
+	  /* Exception for the Euro sign (see CP396).  */		      \
+	  cp = "\x80";							      \
+	  break;							      \
 	case 0x2103 ... 0x22bf:						      \
 	  cp = __gbk_from_ucs4_tab5[ch - 0x2103];			      \
 	  break;							      \

Re: [PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Zack Weinberg]

On Tue, Nov 29, 2016 at 5:08 AM, Florian Weimer <fweimer@redhat.com> wrote:
> On 11/28/2016 10:23 AM, Andreas Schwab wrote:
>>
>> FAIL: iconvdata/tst-tables
>> failed: ./tst-table.sh
>> /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/
>> /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/iconvdata/
>> GBK
>
> Here's one way to fix it.  Doesn't look really pretty, but it seems that
> this new 0x80 mapping is quite an outlier.

+1 from me - this is consistent with the behavior specified for GBK in
the web "Encoding Standard"
(https://encoding.spec.whatwg.org/#gbk-decoder) which has been
obsessively tuned for maximum compatibility with existing content.

zw

Re: [PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Mike Frysinger]

On Tue, Nov 29, 2016 at 5:08 AM, Florian Weimer <fweimer@redhat.com> wrote:
> On 11/28/2016 10:23 AM, Andreas Schwab wrote:
>> FAIL: iconvdata/tst-tables
>> failed: ./tst-table.sh
>> /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/
>> /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/iconvdata/
>> GBK
>
> Here's one way to fix it.  Doesn't look really pretty, but it seems that
> this new 0x80 mapping is quite an outlier.

i'm not super familiar with the gconv modules, but this seems to make sense

should use lowercase hex constants though to match surrounding code style

my dev box is offline pending a move, so i won't be able to push
anything for a while
-mike

Re: [PATCH] localedata: GBK: add mapping for 0x80->Euro sign [BZ #20864]

[Florian Weimer]

On 11/29/2016 06:04 PM, Mike Frysinger wrote:
> On Tue, Nov 29, 2016 at 5:08 AM, Florian Weimer <fweimer@redhat.com> wrote:
>> On 11/28/2016 10:23 AM, Andreas Schwab wrote:
>>> FAIL: iconvdata/tst-tables
>>> failed: ./tst-table.sh
>>> /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/
>>> /home/abuild/rpmbuild/BUILD/glibc-2.24.90.20161127.gb964e06/cc-base/iconvdata/
>>> GBK
>>
>> Here's one way to fix it.  Doesn't look really pretty, but it seems that
>> this new 0x80 mapping is quite an outlier.
>
> i'm not super familiar with the gconv modules, but this seems to make sense
>
> should use lowercase hex constants though to match surrounding code style

Right, I'll fix this before committing it.

Thanks,
Florian