[Bug localedata/22371] New: U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[Bug localedata/22371] New: U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[nakajima.akira at nttcom dot co.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

            Bug ID: 22371
           Summary: U+FFE2 and U+FFE4, iconv does not convert to
                    HALFWIDTH(EUC-JISX0213)
           Product: glibc
           Version: 2.26
            Status: UNCONFIRMED
          Severity: minor
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: nakajima.akira at nttcom dot co.jp
                CC: libc-locales at sourceware dot org
  Target Milestone: ---

When converting to EUC-JISX0213,
iconv does not convert 'FULLWIDTH NOT SIGN' (U+FFE2) to 'NOT SIGN' (U+00AC).

# printf '\xef\xbf\xa2' | iconv -c -f UTF-8 -t EUC-JISX0213 | od -tx1
0000000


Nearby characters are converted to FULLWIDTH to HALFWIDTH.

# printf '\xef\xbf\xa3' | iconv -c -f UTF-8 -t EUC-JISX0213 | od -tx1
0000000 a1 b1


nkf converts to 'a2 cc'.

# printf '\xef\xbf\xa2' | nkf --ic=UTF-8 --oc=EUC-JISX0213 | od -tx1
0000000 a2 cc

'a2 cc' is 'NOT SIGN' of HALFWIDTH in EUC-JISX0213.
 (Same character of FULLWIDTH does not exist in EUC-JISX0213).


========================================
As similarly,
iconv does not convert 'FULLWIDTH BROKEN BAR' (U+FFE4)  to 'BROKEN BAR'
(U+00A6).

# printf '\xef\xbf\xa4' | iconv -c -f UTF-8 -t EUC-JISX0213 | od -tx1
0000000

# printf '\xef\xbf\xa4' | unorm --normalization=nfkd | od -tx1
0000000 c2 a6

HALFWIDTH (U+00A6) is exists in EUC-JISX0213 at 'a9 a5'.
http://charset.uic.jp/show/eucjisx0213/

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |INVALID

--- Comment #1 from Andreas Schwab <schwab@linux-m68k.org> ---
You cannot have two characters mapped to the same target character.  All
mappings must be injective.  iconv does not do any normalisation.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at redhat dot com

--- Comment #2 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Akira Nakajima from comment #0)
> When converting to EUC-JISX0213,
> iconv does not convert 'FULLWIDTH NOT SIGN' (U+FFE2) to 'NOT SIGN' (U+00AC).
> 
> # printf '\xef\xbf\xa2' | iconv -c -f UTF-8 -t EUC-JISX0213 | od -tx1
> 0000000
> 
> 
> Nearby characters are converted to FULLWIDTH to HALFWIDTH.
> 
> # printf '\xef\xbf\xa3' | iconv -c -f UTF-8 -t EUC-JISX0213 | od -tx1
> 0000000 a1 b1
> 
> 
> nkf converts to 'a2 cc'.

You can attain a similar behaviour with iconv if you use '//TRANSLIT' to
transliterate.

printf '\xef\xbf\xa2' | iconv -c -f UTF-8 -t EUC-JISX0213//TRANSLIT | od -tx1
0000000 a2 cc
0000002

Without transliteration the 'FULLWIDTH NOT SIGN' has no direct representation
in the target character map, and as Andreas explains, the conversion mapping
must be injective for iconv. The <U00AC> is already assigned to the NOT SIGN.

Note that once transliterated you cannot go back, the mapping with
transliteration is not injective.

> As similarly,
> iconv does not convert 'FULLWIDTH BROKEN BAR' (U+FFE4)  to 'BROKEN BAR'
> (U+00A6).
> 
> # printf '\xef\xbf\xa4' | iconv -c -f UTF-8 -t EUC-JISX0213 | od -tx1
> 0000000
> 
> # printf '\xef\xbf\xa4' | unorm --normalization=nfkd | od -tx1
> 0000000 c2 a6
> 
> HALFWIDTH (U+00A6) is exists in EUC-JISX0213 at 'a9 a5'.
> http://charset.uic.jp/show/eucjisx0213/

printf '\xef\xbf\xa4' | iconv -c -f UTF-8 -t EUC-JISX0213//TRANSLIT | od -tx1
0000000 a9 a5
0000002

Again, with transliteration you can change one into the other.

Keep in mind that transliteration is locale specific.

In this case glibc has a 'translit_wide' specifically for wide
transliterations, and it *is* part of our language and locale neutral
transliterations that is included by our global i18n locale specification. Thus
all locales "should" have this neutral transformation available.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[nakajima.akira at nttcom dot co.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

--- Comment #3 from Akira Nakajima <nakajima.akira at nttcom dot co.jp> ---
I didn't know TRANSLATE option. Thanks for teaching.
But I don't understand following works.

Following are all FULLWIDTH chars, and result of convert UTF-8 to EUC-JISX0213.

        char  EUC-JISX0213             EUC-JISX0213//TRANSLIT
U+FFE0  ￠    null                     a1 f1(¢ HALF = U+00A2)
U+FFE1  ￡    null                     a1 f2(£ HALF = U+00A3)
U+FFE2  ￢    null                     a2 cc(¬ HALF = U+00AC)
U+FFE3  ￣    a1 b1(‾ HALF = U+203E)   a1 b1
U+FFE4  ￤    null                     a9 a5(¦ HALF = U+00A6)
U+FFE5  ￥    a1 ef(\ HALF = U+00A5)   a1 ef

I can understand that iconv convert FULLWIDTH to HALFWIDTH with //TRANSLIT
 ,because same FULLWIDTH chars are not exist in EUC-JISX0213, but HALFWIDTH
exist.

But U+FFE3 and U+FFE5 are converted to HALFWIDTH therefore with and without
//TRANSLIT.
Is this works correct?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

--- Comment #4 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Akira Nakajima from comment #3)
> I didn't know TRANSLATE option. Thanks for teaching.
> But I don't understand following works.
> 
> Following are all FULLWIDTH chars, and result of convert UTF-8 to
> EUC-JISX0213.
> 
>         char  EUC-JISX0213             EUC-JISX0213//TRANSLIT
> U+FFE0  ￠    null                     a1 f1(¢ HALF = U+00A2)
> U+FFE1  ￡    null                     a1 f2(£ HALF = U+00A3)
> U+FFE2  ￢    null                     a2 cc(¬ HALF = U+00AC)
> U+FFE3  ￣    a1 b1(‾ HALF = U+203E)   a1 b1

In EUC-JISX0213 the character /xa1/xb1 is a FULLWIDTH MACRON, it is not U+203E.

See, page 3 of the PDF, note that the 8th bit is always set:
https://www.itscj.ipsj.or.jp/iso-ir/228.pdf

Therefore there is a direct representation in EUC-JISX0213 for U+FFE3, which is
why you do not need to transliterate.

> U+FFE4  ￤    null                     a9 a5(¦ HALF = U+00A6)
> U+FFE5  ￥    a1 ef(\ HALF = U+00A5)   a1 ef
> 
> I can understand that iconv convert FULLWIDTH to HALFWIDTH with //TRANSLIT
>  ,because same FULLWIDTH chars are not exist in EUC-JISX0213, but HALFWIDTH
> exist.

In general that statement is true.

However, there are 2 characters in your list that *do* have direct EUC-JISX0213
equivalents.

> But U+FFE3 and U+FFE5 are converted to HALFWIDTH therefore with and without
> //TRANSLIT.
> Is this works correct?

No.

U+FFE3 "FULLWIDTH MACRON" is available directly in EUC_JISX0213 as /xa1/xb1
(full width macron).

U+FFE5 "FULLWIDTH YEN SIGN" is available directly in EUC_JISX0213 as /xa1/ef
(full width yen sign).

No transliteration is required.

Your confusion appears to be that you think EUC_JISX0213 character /xa1/xb1 is
halfwidth, but the standard does not show that.

I hope that answers your questions.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

Mike FABIAN <maiku.fabian at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maiku.fabian at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[nakajima.akira at nttcom dot co.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

--- Comment #5 from Akira Nakajima <nakajima.akira at nttcom dot co.jp> ---
\xa1\xef is mapped to U+00A5(HALFWIDTH YEN) in EUC-JISX0213 and EUC-JIS-2004 by
following URL.
\xa1\xb1 is as same.

"However, with Unicode 3.2.0 the mappings differ in 3 codepoints."
http://search.cpan.org/~dankogai/Encode-JIS2K-0.03/JIS2K.pm#what_is_JIS_X_0213_anyway?

=============================================
http://charset.uic.jp/show/eucjisx0213/
http://x0213.org/codetable/euc-jis-2004-with-char.txt
char    JIS     Unicode
￣       0xA1B1  U+203E  # OVERLINE      Windows: U+FFE3
―       0xA1BD  U+2014  # EM DASH       Windows: U+2015
￥       0xA1EF  U+00A5  # YEN SIGN      Windows: U+FFE5
=============================================

=============================================
perl 5.24.3

# perl -e 'use Encode; use Encode::JISX0213; print encode("euc-jisx0213",
"\x{00a5}");' | od -tx1
0000000 a1 ef

# perl -e 'use Encode; use Encode::JISX0213; print encode("euc-jisx0213",
"\x{ffe5}");' | od -tx1
0000000 a1 ef
=============================================



But Python and "/usr/local/share/i18n/charmaps/EUC-JISX0213.gz"
 have mapping to U+FFE5.
I don't know which one is correct.

=============================================
Python 3.6.2

# python3 -c "print(u'\u00a5'.encode('euc-jisx0213'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'euc_jisx0213' codec can't encode character '\xa5' in
position 0: illegal multibyte sequence

# python3 -c "print(u'\uffe5'.encode('euc-jisx0213'))"
b'\xa1\xef'
=============================================

=============================================
/usr/local/share/i18n/charmaps/EUC-JISX0213.gz (Fedora 26)

<UFFE3>     /xa1/xb1     FULLWIDTH MACRON
<UFFE5>     /xa1/xef     FULLWIDTH YEN SIGN
=============================================

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

--- Comment #6 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Akira Nakajima from comment #5)
> \xa1\xef is mapped to U+00A5(HALFWIDTH YEN) in EUC-JISX0213 and EUC-JIS-2004
> by following URL.
> \xa1\xb1 is as same.
> 
> "However, with Unicode 3.2.0 the mappings differ in 3 codepoints."
> http://search.cpan.org/~dankogai/Encode-JIS2K-0.03/JIS2K.
> pm#what_is_JIS_X_0213_anyway?
> 
> =============================================
> http://charset.uic.jp/show/eucjisx0213/
> http://x0213.org/codetable/euc-jis-2004-with-char.txt
> char    JIS     Unicode
> ￣	0xA1B1	U+203E	# OVERLINE	Windows: U+FFE3
> ―	0xA1BD	U+2014	# EM DASH	Windows: U+2015
> ￥	0xA1EF	U+00A5	# YEN SIGN	Windows: U+FFE5
> =============================================

This is not an authoritative source. I do not believe this is correct. Notice
that there is a comment that on Windows it is U+FFE5, which is the FULLWIDTH
YEN. That is in my opinion correct. Therefore Windows and Linux have the same
representation e.g. FULLWIDTH YEN. I suggest discussing this with the author of
the document.

> =============================================
> perl 5.24.3
> 
> # perl -e 'use Encode; use Encode::JISX0213; print encode("euc-jisx0213",
> "\x{00a5}");' | od -tx1
> 0000000 a1 ef
> # perl -e 'use Encode; use Encode::JISX0213; print encode("euc-jisx0213",
> "\x{ffe5}");' | od -tx1
> 0000000 a1 ef
> =============================================

This behaviour is expected. You are encoding a unicode code point into
EUC-JISX0213. There is no representation of YEN, so the output is FULLWIDTH YEN
in both cases e.g. /xa1/xef.

> But Python and "/usr/local/share/i18n/charmaps/EUC-JISX0213.gz"
>  have mapping to U+FFE5.
> I don't know which one is correct.

This is correct. As it is on Windows.

> =============================================
> Python 3.6.2
> 
> # python3 -c "print(u'\u00a5'.encode('euc-jisx0213'))"
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> UnicodeEncodeError: 'euc_jisx0213' codec can't encode character '\xa5' in
> position 0: illegal multibyte sequence

Correct. There is no YEN in EUC-JISX0213. This python behaviour is based on
glibc's character map.

> # python3 -c "print(u'\uffe5'.encode('euc-jisx0213'))"
> b'\xa1\xef'
> =============================================
> 
> =============================================
> /usr/local/share/i18n/charmaps/EUC-JISX0213.gz (Fedora 26)
> 
> <UFFE3>     /xa1/xb1     FULLWIDTH MACRON
> <UFFE5>     /xa1/xef     FULLWIDTH YEN SIGN
> =============================================

These are IMO correct.

In my previous post I referenced *official* Japanese ISO-IR documents, and I
will reference them again:

See, page 3 of the PDF, note that the 8th bit is always set:
https://www.itscj.ipsj.or.jp/iso-ir/228.pdf

You can see that the macron is a FULLWIDTH MACRO, and with the yen sign the
FULLWIDTH YEN is selected because it provides compatibility with Windows.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[nakajima.akira at nttcom dot co.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

--- Comment #7 from Akira Nakajima <nakajima.akira at nttcom dot co.jp> ---
Created attachment 10576
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10576&action=edit
JIS X 0213

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/22371] U+FFE2 and U+FFE4, iconv does not convert to HALFWIDTH(EUC-JISX0213)

[nakajima.akira at nttcom dot co.jp]

https://sourceware.org/bugzilla/show_bug.cgi?id=22371

--- Comment #8 from Akira Nakajima <nakajima.akira at nttcom dot co.jp> ---
I found following sources.

=====================================================
JIS X 0208 -> Unicode

https://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0208.TXT
SJIS    JIS     JIS X 0208 -> Unicode
0x8150  0x2131  0xFFE3  # FULLWIDTH MACRON
0x818F  0x216F  0xFFE5  # FULLWIDTH YEN SIGN
=====================================================


=====================================================
JIS X 0213 -> Unicode

I couldn't find English source, but found Japanese source at JIS HomePage.
 (can't download, only see at Web)
http://www.jisc.go.jp/index.html

This page is Japanese.
To reach source, Please see attached PNG.

01.PNG : Click 「JIS検索」 at Right Upper.
02.PNG : Input 「X0213」 and Press Enter.
03:PNG : Click 「X0213_007 （PDFファイル：2881KB 別ウィンドウでリンク）」

See page 5 and 16

SJIS    JIS     JIS X 0213 -> Unicode(JIS X 0221)
0x8150  0x2131  U+203E  # OVERLINE
0x818F  0x216F  U+00A5  # YEN SIGN
=====================================================


I noticed Yesterday that this conversion of iconv is famous in many blogs and
books.
Their view are that this is not bug, this is specification.
They who know much more detail than me say so.
Therefore, this will be specification of iconv.

-- 
You are receiving this mail because:
You are on the CC list for the bug.