[Bug localedata/17588] New: Update UTF-8 charmap and width to Unicode 7.0.0

[Bug localedata/17588] New: Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

            Bug ID: 17588
           Summary: Update UTF-8 charmap and width to Unicode 7.0.0
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: pravin.d.s at gmail dot com
                CC: libc-locales at sourceware dot org

Forked from #14094. Good to have separate bugs for UTF-8 and i18n file update.
Tracking changes and issues will be more clearer in long term.
*************************************************************
 Joseph Myers 2012-05-10 20:27:32 UTC

The Unicode locale data - character map and LC_CTYPE information - should be
updated from Unicode 6.1 (the character map is currently based on 6.0, and
LC_CTYPE is currently based on 5.0).  This should be done with proper
automation and wiki documentation being added of how to do future updates.  I
identified the following tasks at
<http://sourceware.org/ml/libc-alpha/2012-05/msg00590.html>:

* Ensure the character type data in localedata/charmaps/i18n can be
  properly reproduced from Unicode 5.0 data using gen-unicode-ctype.c,
  adapting gen-unicode-ctype.c as needed to replicate any changes that
  may have been made not using that program.

* Update the character type data to Unicode 6.1, removing any local
  hacks from gen-unicode-ctype.c that are no longer needed.
  (10646:2012, corresponding to Unicode 6.1, appears to be in
  publication stage so should be out very soon.)

* Ensure the character data in localedata/charmaps/UTF-8 can be
  reproduced in some automated fashion from Unicode 6.0, locating any
  previously used automation for this or creating some new automation
  if any previous automation can't be found.

* Update the character data to Unicode 6.1, removing any local hacks
  in the automation from the previous step.

* Document thoroughly on the wiki how the automation works and how to
  do updates to new Unicode versions.

[reply] [−] Comment 1 Rich Felker 2012-05-11 03:25:47 UTC

One of the major "local hacks" can be fixed, fixing many other problems at the
same time, by switching to using the Unicode "Alphabetic" property (from
DerivedCoreProperties.txt) instead of just categories L* for class alpha. Right
now there are many languages whose letters are considered non-alphabetic by
glibc because they're in category Mn or Mc or even Cf. There are "local hacks"
to fix this for maybe one or two languages, but using the right Unicode
property would fix it for all languages.
*******************************************************

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-26581-listarch-glibc-bugs=sources.redhat.com@sourceware.org Wed Nov 12 10:13:33 2014
Return-Path: <glibc-bugs-return-26581-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 23322 invoked by alias); 12 Nov 2014 10:13:33 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 22671 invoked by uid 48); 12 Nov 2014 10:13:26 -0000
From: "pravin.d.s at gmail dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0
Date: Wed, 12 Nov 2014 10:13:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: unspecified
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: pravin.d.s at gmail dot com
X-Bugzilla-Status: NEW
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: unassigned at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields: blocked
Message-ID: <bug-17588-131-xS6LUohNPs@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-17588-131@http.sourceware.org/bugzilla/>
References: <bug-17588-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2014-11/txt/msg00073.txt.bz2
Content-length: 370

https://sourceware.org/bugzilla/show_bug.cgi?id\x17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |14094

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maiku.fabian at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at sourceware dot org   |pravin.d.s at gmail dot com

--- Comment #1 from Pravin S <pravin.d.s at gmail dot com> ---
Created attachment 7926
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7926&action=edit
Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0

1. utf8-gen.py to generate UTF-8 file
2. utf8-compatibility.py : to check backward compatibility of newly generated
UTF-8 file
3. Report of new UTF-8 file backward compatibility is available AT
https://raw.githubusercontent.com/pravins/glibc-i18n/master/report-utf8

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #7926|0                           |1
        is obsolete|                            |
                 CC|                            |pravin.d.s at gmail dot com

--- Comment #2 from Pravin S <pravin.d.s at gmail dot com> ---
Created attachment 7958
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7958&action=edit
Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0

Mike did review on it earlir and done updates to glibc-i18n git.
https://github.com/pravins/glibc-i18n

I have updated patch based on those improvement.

Latest report on backward compatibility is available AT
https://raw.githubusercontent.com/pravins/glibc-i18n/master/report-utf8  

Note: Please file word Analysis, it is done after report is generated to make
sure changes are correct.

Mike please review patch and give your comments.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

--- Comment #3 from Mike FABIAN <maiku.fabian at gmail dot com> ---
(In reply to Pravin S from comment #2)
> Created attachment 7958 [details]
> Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0
> 
> Mike did review on it earlir and done updates to glibc-i18n git.
> https://github.com/pravins/glibc-i18n
> 
> I have updated patch based on those improvement.
> 
> Latest report on backward compatibility is available AT
> https://raw.githubusercontent.com/pravins/glibc-i18n/master/report-utf8  
> 
> Note: Please file word Analysis, it is done after report is generated to
> make sure changes are correct.
> 
> Mike please review patch and give your comments.

To check whether the new generated UTF-8 file is correct,
I ran the utf8-compatibility.py script (updated version) like this:

python3 utf8-compatibility.py -o ../glibc/localedata/charmaps/UTF-8 -n UTF-8 
-u unicode7-0/UnicodeData.txt -e unicode7-0/EastAsianWidth.txt -c
Report on CHARMAP:
This character might be missing in the generated charmap:  <U9F80>..<U9FC3>
************************************************************

Report on WIDTH:
Total changed characters in newly generated WIDTH:  88827
changed width: 0x00ad : 1->0 eaw=A category=Cf bidi=BN  name=SOFT HYPHEN
...
changed width: 0xa960 : 1->2 eaw=W category=Lo bidi=L   name=HANGUL CHOSEONG
TIKEUT-MIEUM
...
many such lines
...

Now I look at these lines, for example the above mentioned change
where the width of a character changes from 1 to 2 and the character has
East Asian Width “W” and the category “Lo” is certainly correct
(This character was not in the old UTF-8 file, only characters with
width 0 and 2 are in the file, 1 is the default width, every character
not in the UTF-8 file gets the default width 1).

As this change looks correct, I remove all lines like this from my Emacs
buffer with:

    “M-x flush-lines RET 1->2 eaw=W category=Lo”

Removing lines with obviously correct changes like this quickly
reduces the number of lines to look at and after a while I have only

changed width: 0x00ad : 1->0 eaw=A category=Cf bidi=BN  name=SOFT HYPHEN
changed width: 0x3248 : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER TEN
ON BLACK SQUARE
changed width: 0x3249 : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
TWENTY ON BLACK SQUARE
changed width: 0x324a : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
THIRTY ON BLACK SQUARE
changed width: 0x324b : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
FORTY ON BLACK SQUARE
changed width: 0x324c : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
FIFTY ON BLACK SQUARE
changed width: 0x324d : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
SIXTY ON BLACK SQUARE
changed width: 0x324e : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
SEVENTY ON BLACK SQUARE
changed width: 0x324f : 2->1 eaw=A category=No bidi=L   name=CIRCLED NUMBER
EIGHTY ON BLACK SQUARE

The change for the characters with eaw=A (East Asian Width
“Ambiguous”) where the width changed from 2 to 1 is also correct, I think.
The UTF-8 file is a generic file, not especially for an East Asian locale,
so the “Ambiguous” characters should not have width 2.

Then only the soft hyphen remains which puzzles me a bit:

changed width: 0x00ad : 1->0 eaw=A category=Cf bidi=BN  name=SOFT HYPHEN

Our script gives width 0 to this character because of category=Cf.

But the display width of the soft hyphen depends on whether it is
in the middle of a line (invisible then) or happens to be at the end
of a line where it should be visible (and doesn’t it have a width greater
than zero if it is visible?).
But still giving width 0 to the soft hyphen in the UTF-8 file seems the
right thing to me.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-26674-listarch-glibc-bugs=sources.redhat.com@sourceware.org Fri Nov 21 12:36:42 2014
Return-Path: <glibc-bugs-return-26674-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 8120 invoked by alias); 21 Nov 2014 12:36:41 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 8076 invoked by uid 48); 21 Nov 2014 12:36:37 -0000
From: "maiku.fabian at gmail dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0
Date: Fri, 21 Nov 2014 12:36:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: unspecified
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: maiku.fabian at gmail dot com
X-Bugzilla-Status: ASSIGNED
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: pravin.d.s at gmail dot com
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields:
Message-ID: <bug-17588-131-HVLtqxBETP@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-17588-131@http.sourceware.org/bugzilla/>
References: <bug-17588-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2014-11/txt/msg00166.txt.bz2
Content-length: 2083

https://sourceware.org/bugzilla/show_bug.cgi?id\x17588

--- Comment #4 from Mike FABIAN <maiku.fabian at gmail dot com> ---
Here is another one where I have a little bit of doubt left:

changed width: 0x1929 : 0->1 eaw=N category=Mc bidi=L   name=LIMBU SUBJOINED
LETTER YA

Why is this combining characters listed with width 0 in the current UTF-8 file?

In our newly generated UTF-8 file it has width 1 (because it is removed from
that  file).

The comment in the existing UTF-8 file in glibc says:

% Character width according to Unicode 5.0.0.
% - Default width is 1.
% - Double-width characters have width 2; generated from
%        "grep '^[^;]*;[WF]' EastAsianWidth.txt"
%   and  "grep '^[^;]*;[^WF]' EastAsianWidth.txt"
% - Non-spacing characters have width 0; generated from PropList.txt or
%   "grep '^[^;]*;[^;]*;[^;]*;[^;]*;NSM;' UnicodeData.txt"
% - Format control characters have width 0; generated from
%   "grep '^[^;]*;[^;]*;Cf;' UnicodeData.txt"
% - Zero width characters have width 0; generated from
%   "grep '^[^;]*;ZERO WIDTH ' UnicodeData.txt"

This does *not* mention combining characters as needing width 0,
these grep patters to not include some combining characters.

The combining characters with category=Mn get width 0 because the
also have bidi=NSM, for example:

changed width: 0x1a1b : 1->0 eaw=N category=Mn bidi=NSM name=BUGINESE VOWEL
SIGN AE

but the combining characters with category=Mc are not matched by
the above grep patterns, because they do *not* have bidi=NSM.
That seems correct, considering they have a positive advance width:

Mn     Nonspacing_Mark  a nonspacing combining mark (zero advance width)
Mc     Spacing_Mark      a spacing combining mark (positive advance width)
Me     Enclosing_Mark      an enclosing combining mark

(http://www.unicode.org/reports/tr44)

But how did these get into the existing UTF-8 file in glibc?

Looks like the existing UTF-8 file in glibc was edited manually
and not just created using the grep patterns in the comment.

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

--- Comment #5 from Mike FABIAN <maiku.fabian at gmail dot com> ---
localedata/Changelog entry from the patch from comment#2:

>    * scripts/utf8-gen.py: New script for generating UTF-8 CHARMAP from
>    latest UnicodeData.txt.
> 
>    * scripts/utf-compatibility.py: New script for testing backward

- The script is actually called “utf8-compatibility.py”, not
  “utf-compatibility.py”
- The patch puts the scripts “utf8-gen.py” and “utf8-compatibility.py”
  into the “localedata/” directory,  not the “scripts/” directory.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-26676-listarch-glibc-bugs=sources.redhat.com@sourceware.org Fri Nov 21 18:19:37 2014
Return-Path: <glibc-bugs-return-26676-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 2544 invoked by alias); 21 Nov 2014 18:19:37 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 2515 invoked by uid 48); 21 Nov 2014 18:19:33 -0000
From: "jsm28 at gcc dot gnu.org" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug stdio/17633] New: perror fileno namespace
Date: Fri, 21 Nov 2014 18:19:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: stdio
X-Bugzilla-Version: 2.21
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: jsm28 at gcc dot gnu.org
X-Bugzilla-Status: NEW
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: unassigned at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags:
X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status bug_severity priority component assigned_to reporter
Message-ID: <bug-17633-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2014-11/txt/msg00168.txt.bz2
Content-length: 526

https://sourceware.org/bugzilla/show_bug.cgi?id\x17633

            Bug ID: 17633
           Summary: perror fileno namespace
           Product: glibc
           Version: 2.21
            Status: NEW
          Severity: normal
          Priority: P2
         Component: stdio
          Assignee: unassigned at sourceware dot org
          Reporter: jsm28 at gcc dot gnu.org

perror, an ISO C function, uses fileno, which is not an ISO C function.

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #7958|0                           |1
        is obsolete|                            |

--- Comment #6 from Pravin S <pravin.d.s at gmail dot com> ---
Created attachment 7969
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7969&action=edit
Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0

Good catch Mike. Latest patch attached.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

--- Comment #7 from Mike FABIAN <maiku.fabian at gmail dot com> ---
When I try to apply the latest patch

https://sourceware.org/bugzilla/attachment.cgi?id=7969

I get:

    $ git am bug-17588-13064.patch
    Applying: updated UTF-8 (charmap and width) to Unicode 7.0
    /local/mfabian/src/glibc/.git/rebase-apply/patch:20: trailing whitespace.
            * localedata/utf8-gen.py: New script for generating UTF-8 CHARMAP
from 
    /local/mfabian/src/glibc/.git/rebase-apply/patch:35221: trailing
whitespace.
    # Contributed by 
    error: patch failed: localedata/charmaps/UTF-8:134
    error: localedata/charmaps/UTF-8: patch does not apply
    Patch failed at 0001 updated UTF-8 (charmap and width) to Unicode 7.0
    The copy of the patch that failed is found in:
       /local/mfabian/src/glibc/.git/rebase-apply/patch
    When you have resolved this problem, run "git am --continue".
    If you prefer to skip this patch, run "git am --skip" instead.
    To restore the original branch and stop patching, run "git am --abort".

Applying  it ignoring whitespace works:

    $ git am --ignore-space-change bug-17588-13064.patch
    Applying: updated UTF-8 (charmap and width) to Unicode 7.0
    /local/mfabian/src/glibc/.git/rebase-apply/patch:20: trailing whitespace.
            * localedata/utf8-gen.py: New script for generating UTF-8 CHARMAP
from 
    /local/mfabian/src/glibc/.git/rebase-apply/patch:35221: trailing
whitespace.
    # Contributed by 
    warning: 2 lines add whitespace errors.

But then we get a very inconsitent use of white space, for example:


    @@ -2192,6 +2256,7 @@ CHARMAP
     <U097D>     /xe0/xa5/xbd DEVANAGARI LETTER GLOTTAL STOP
     <U097E>     /xe0/xa5/xbe DEVANAGARI LETTER DDDA
     <U097F>     /xe0/xa5/xbf DEVANAGARI LETTER BBA
    +<U0980>     /xe0/xa6/x80         BENGALI ANJI
     <U0981>     /xe0/xa6/x81 BENGALI SIGN CANDRABINDU
     <U0982>     /xe0/xa6/x82 BENGALI SIGN ANUSVARA
     <U0983>     /xe0/xa6/x83 BENGALI SIGN VISARGA

Probably it is better to always use only a single space after
the the UTF-8 byte sequcence. That would make some lines change
only in white space, for example

<U0000>     /x00         NULL

would change to

<U0000>     /x00 NULL

but the end result looks more consistent.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #7969|0                           |1
        is obsolete|                            |

--- Comment #8 from Pravin S <pravin.d.s at gmail dot com> ---
Created attachment 7980
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7980&action=edit
Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0

Agree with you Mike.

Earlier created patch by ignoring space thinking it will be easy to review.
Thank you for pointing that applying such patch create inconsistency in final
UTF-8 file.

Yes, not reason to use mutliple space after utf8 hex filed. 

Created new patch without ignoring space.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[maiku.fabian at gmail dot com]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 9632 bytes --]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

--- Comment #9 from Mike FABIAN <maiku.fabian at gmail dot com> ---
I built glibc with the patch from comment#8.

I produces some FAILs in “make check”:

    FAIL: localedata/cs_CZ.UTF-8/LC_CTYPE
    ... similar FAILs ...

Shortly after starting “make check” one sees:

    ./charmaps/UTF-8:42734: unknown character `U00009FCD'
    ... similar messages ...

All the above problems are cause by ranges of reserved code points
which are listed in EastAsianWidth.txt like this:

    9FCD..9FFF;W     # Cn    [51] <reserved-9FCD>..<reserved-9FFF>

and these code points are not in UnicodeData.txt.

Therefore, they are not generated into the CHARMAP section
of glibc’s UTF-8 file and it causes the above problems if they
are generated into the WIDTH section of glibc’s  UTF-8 file.

This can be fixed by not generating reserved code points into
the WIDTH section, i.e. by ignoring the  reserved  code points
mentioned in EastAsianWidth.txt. Patch for utf8-gen.py:

diff --git a/utf8-gen.py b/utf8-gen.py
index 57875b6..20b68bb 100755
--- a/utf8-gen.py
+++ b/utf8-gen.py
@@ -218,6 +218,8 @@ if __name__ == "__main__":
         write_comments(outfile, 1)
         elines = []
         for line in easta_file.readlines():
+                if re.match(r'.*<reserved-.+>\.\.<reserved-.+>.*', line):
+                        continue
                 if re.match(r'^[^;]*;[WF]', line):
                         elines.append(line.strip())
         process_width(outfile, flines, elines)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-26784-listarch-glibc-bugs=sources.redhat.com@sourceware.org Wed Dec 03 09:59:20 2014
Return-Path: <glibc-bugs-return-26784-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 30733 invoked by alias); 3 Dec 2014 09:59:20 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 30661 invoked by uid 48); 3 Dec 2014 09:59:14 -0000
From: "maiku.fabian at gmail dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/14094] Update locale data to Unicode 7.0.0
Date: Wed, 03 Dec 2014 09:59:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: 2.21
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: maiku.fabian at gmail dot com
X-Bugzilla-Status: ASSIGNED
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: pravin.d.s at gmail dot com
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields:
Message-ID: <bug-14094-131-MOycuzHapv@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-14094-131@http.sourceware.org/bugzilla/>
References: <bug-14094-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2014-12/txt/msg00027.txt.bz2
Content-length: 3386

https://sourceware.org/bugzilla/show_bug.cgi?id=14094

--- Comment #34 from Mike FABIAN <maiku.fabian at gmail dot com> ---
When I generate a new glibc/localedata/locales/i18n file
using gen-unicode-ctype.py from comment#33 and build
glibc with that and then run the tests with “make check”, I get
one failure:

    FAIL: localedata/tst-ctype

Looking why it fails I find in ./localedata/tst-ctype.out:

    Locale-specific tests for `lower'
      islower('ª' = '\xaa') is true
      islower('º' = '\xba') is true
    Locale-specific tests for `lower'
    ...
    2 errors for `de_DE.ISO-8859-1' locale

The new “lower” character class generated by gen-unicode-ctype.py
contains U+00AA ª FEMININE ORDINAL INDICATOR and U+00BA º MASCULINE
ORDINAL INDICATOR.

The test tst-ctype run by “make check” wants them *not* to be lower case.

DerivedCoreProperties.txt lists both as lower case though:

    00AA          ; Lowercase # Lo       FEMININE ORDINAL INDICATOR
    00BA          ; Lowercase # Lo       MASCULINE ORDINAL INDICATOR

That’s why gen-unicode-ctype.py adds them to the “lower” character
class, it adds all characters found in DerivedCoreProperties.txt
marked as “Lowercase” to the character class “lower”.

I wonder what needs to be done here.

Is the test in glibc wrong?

If so, it could be fixed by a patch like this:

$ git show | iconv -f iso-8859-1 -t utf-8
commit 25c913674386011a44b6270579a894b2e8200d25
Author: Mike FABIAN <mfabian@redhat.com>
Date:   Wed Dec 3 10:05:42 2014 +0100

    Fix test case localedata/tst-ctype-de_DE.ISO-8859-1.in

    DerivedCoreProperties.txt from Unicode 7.0.0 lists
    the characters U+00AA (ª) and U+00BA (º) as lower case:

    00AA          ; Lowercase # Lo       FEMININE ORDINAL INDICATOR
    00BA          ; Lowercase # Lo       MASCULINE ORDINAL INDICATOR

diff --git a/localedata/tst-ctype-de_DE.ISO-8859-1.in
b/localedata/tst-ctype-de_DE.ISO-8859-1.in
index f71d76c..e124a52 100644
--- a/localedata/tst-ctype-de_DE.ISO-8859-1.in
+++ b/localedata/tst-ctype-de_DE.ISO-8859-1.in
@@ -1,5 +1,5 @@
 lower    ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏ
-        000000000000000000000100000000000000000000000000
+        000000000010000000000100001000000000000000000000
 lower   ÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ
         000000000000000111111111111111111111111011111111
 upper    ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏ

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-26785-listarch-glibc-bugs=sources.redhat.com@sourceware.org Wed Dec 03 11:49:27 2014
Return-Path: <glibc-bugs-return-26785-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 6139 invoked by alias); 3 Dec 2014 11:49:27 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 6105 invoked by uid 48); 3 Dec 2014 11:49:23 -0000
From: "pravin.d.s at gmail dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0
Date: Wed, 03 Dec 2014 11:49:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: unspecified
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: pravin.d.s at gmail dot com
X-Bugzilla-Status: ASSIGNED
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: pravin.d.s at gmail dot com
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields: attachments.isobsolete attachments.created
Message-ID: <bug-17588-131-DUHIElb1jH@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-17588-131@http.sourceware.org/bugzilla/>
References: <bug-17588-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2014-12/txt/msg00028.txt.bz2
Content-length: 1283

https://sourceware.org/bugzilla/show_bug.cgi?id\x17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #7980|0                           |1
        is obsolete|                            |

--- Comment #10 from Pravin S <pravin.d.s at gmail dot com> ---
Created attachment 7987
  --> https://sourceware.org/bugzilla/attachment.cgi?idy87&actioníit
Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0

2014-12-01  Pravin Satpute  <psatpute@redhat.com>

        [BZ #17588 #13064]
        * charmaps/UTF-8: Updated UTF-8 CHARMAP and WIDTH to Unicode 7.0.0.

        * localedata/utf8-gen.py: New script for generating UTF-8 CHARMAP from
        latest UnicodeData.txt.

        * localedata/utf8-compatibility.py: New script for testing backward
        compatibility of newly generated UTF-8 file.
       Reviewed and improved by Mike FABIAN <mfabian@redhat.com>

------------------------------------------------------------------------------

Yes, i also able to reproduce same issues while building glibc with patch. This
patch fixes those issues.

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[pravin.d.s at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Pravin S <pravin.d.s at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #7987|0                           |1
        is obsolete|                            |

--- Comment #11 from Pravin S <pravin.d.s at gmail dot com> ---
Created attachment 8009
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8009&action=edit
Patch to update UTF-8 CHARMAP and WIDTH to unicode 7.0

2014-12-12  Pravin Satpute  <psatpute@redhat.com>

        [BZ #17588 #13064]
        * charmaps/UTF-8: Updated UTF-8 CHARMAP and WIDTH to Unicode 7.0.0.

        * localedata/utf8_gen.py: New script for generating UTF-8 CHARMAP from 
        latest UnicodeData.txt.

        * localedata/utf8_compatibility.py: New script for testing backward
        compatibility of newly generated UTF-8 file.
       Reviewed and improved by Mike FABIAN <mfabian@redhat.com>

*******************************************************************************
In this patch Mike fixed pylint warning raised by glibc/scripts/pylint.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

--- Comment #12 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, master has been updated
       via  4a4839c94a4c93ffc0d5b95c69a08b02a57007f2 (commit)
      from  e4a399dc3dbb3228eb39af230ad11bc42a018c93 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=4a4839c94a4c93ffc0d5b95c69a08b02a57007f2

commit 4a4839c94a4c93ffc0d5b95c69a08b02a57007f2
Author: Alexandre Oliva <aoliva@redhat.com>
Date:   Fri Feb 20 20:14:59 2015 -0200

    Unicode 7.0.0 update; added generator scripts.

    for  localedata/ChangeLog

        [BZ #17588]
        [BZ #13064]
        [BZ #14094]
        [BZ #17998]
        * unicode-gen/Makefile: New.
        * unicode-gen/unicode-license.txt: New, from Unicode.
        * unicode-gen/UnicodeData.txt: New, from Unicode.
        * unicode-gen/DerivedCoreProperties.txt: New, from Unicode.
        * unicode-gen/EastAsianWidth.txt: New, from Unicode.
        * unicode-gen/gen_unicode_ctype.py: New generator, from Mike
        FABIAN <mfabian@redhat.com>.
        * unicode-gen/ctype_compatibility.py: New verifier, from
        Pravin Satpute <psatpute@redhat.com> and Mike FABIAN.
        * unicode-gen/ctype_compatibility_test_cases.py: New verifier
        module, from Mike FABIAN.
        * unicode-gen/utf8_gen.py: New generator, from Pravin Satpute
        and Mike FABIAN.
        * unicode-gen/utf8_compatibility.py: New verifier, from Pravin
        Satpute and Mike FABIAN.
        * charmaps/UTF-8: Update.
        * locales/i18n: Update.
        * gen-unicode-ctype.c: Remove.
        * tst-ctype-de_DE.ISO-8859-1.in: Adjust, islower now returns
        true for ordinal indicators.

-----------------------------------------------------------------------

Summary of changes:
 NEWS                                               |   11 +-
 localedata/ChangeLog                               |   27 +
 localedata/charmaps/UTF-8                          |11946 ++++++---
 localedata/gen-unicode-ctype.c                     |  784 -
 localedata/locales/i18n                            | 2652 +-
 localedata/tst-ctype-de_DE.ISO-8859-1.in           |    2 +-
 localedata/unicode-gen/DerivedCoreProperties.txt   |10794 ++++++++
 localedata/unicode-gen/EastAsianWidth.txt          | 2121 ++
 localedata/unicode-gen/Makefile                    |   99 +
 localedata/unicode-gen/UnicodeData.txt             |27268 ++++++++++++++++++++
 localedata/unicode-gen/ctype_compatibility.py      |  546 +
 .../unicode-gen/ctype_compatibility_test_cases.py  |  951 +
 localedata/unicode-gen/gen_unicode_ctype.py        |  751 +
 localedata/unicode-gen/unicode-license.txt         |   50 +
 localedata/unicode-gen/utf8_compatibility.py       |  399 +
 localedata/unicode-gen/utf8_gen.py                 |  286 +
 16 files changed, 53305 insertions(+), 5382 deletions(-)
 delete mode 100644 localedata/gen-unicode-ctype.c
 create mode 100644 localedata/unicode-gen/DerivedCoreProperties.txt
 create mode 100644 localedata/unicode-gen/EastAsianWidth.txt
 create mode 100644 localedata/unicode-gen/Makefile
 create mode 100644 localedata/unicode-gen/UnicodeData.txt
 create mode 100755 localedata/unicode-gen/ctype_compatibility.py
 create mode 100644 localedata/unicode-gen/ctype_compatibility_test_cases.py
 create mode 100755 localedata/unicode-gen/gen_unicode_ctype.py
 create mode 100644 localedata/unicode-gen/unicode-license.txt
 create mode 100755 localedata/unicode-gen/utf8_compatibility.py
 create mode 100755 localedata/unicode-gen/utf8_gen.py

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/17588] Update UTF-8 charmap and width to Unicode 7.0.0

[aoliva at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17588

Alexandre Oliva <aoliva at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
                 CC|                            |aoliva at sourceware dot org
         Resolution|---                         |FIXED

--- Comment #13 from Alexandre Oliva <aoliva at sourceware dot org> ---
Fixed

-- 
You are receiving this mail because:
You are on the CC list for the bug.