[Bug localedata/18568] New: Update locale data to Unicode 8.0

[Bug localedata/18568] New: Update locale data to Unicode 8.0

[jsm28 at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

            Bug ID: 18568
           Summary: Update locale data to Unicode 8.0
           Product: glibc
           Version: 2.21
            Status: NEW
          Severity: normal
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: jsm28 at gcc dot gnu.org
                CC: aoliva at sourceware dot org, libc-locales at sourceware dot org,
                    maiku.fabian at gmail dot com, pravin.d.s at gmail dot com
  Target Milestone: ---

Now that Unicode 8.0 has been released, the locale data that was updated for
Unicode 7.0 (bug 14094, bug 17588) should be updated to 8.0.  Hopefully, given
the automation that was developed for the last update, this one should be a lot
simpler to do.

ISO/IEC 10646:2014/Amd 1:2015 was published 2015-05-15.  So the
__STDC_ISO_10646__ value should be 201505L.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[myllynen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

Marko Myllynen <myllynen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |myllynen at redhat dot com

--- Comment #1 from Marko Myllynen <myllynen at redhat dot com> ---
The following Unicode Blog posting provides some comments about required
implementation changes, however I haven't investigated how relevant these are
from glibc perspective.

http://blog.unicode.org/2015/03/unicode-80-beta-review.html

Unicode 8.0.0 comprises several changes which require careful migration in
implementations, including the conversion of Cherokee to a bicameral script, a
different encoding model for New Tai Lue, and additional character repertoire.
Implementers need to change code and check assumptions regarding case mappings
New Tai Lue syllables, Han character ranges, and confusables.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #2 from Mike FABIAN <maiku.fabian at gmail dot com> ---
I made a patch for this and posted it to the libc-alpha mailing list:

https://sourceware.org/ml/libc-alpha/2015-06/msg00748.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #3 from Mike FABIAN <maiku.fabian at gmail dot com> ---
(In reply to Marko Myllynen from comment #1)
> The following Unicode Blog posting provides some comments about required
> implementation changes, however I haven't investigated how relevant these
> are from glibc perspective.

I think for glibc, this has little impact.

> http://blog.unicode.org/2015/03/unicode-80-beta-review.html
> 
> Unicode 8.0.0 comprises several changes which require careful migration in
> implementations, including the conversion of Cherokee to a bicameral script,

That means Cherokee has upper and lower case now, lots of characters
get added to "toupper", "tolower", "totitle" in the "i18n" file.

> a different encoding model for New Tai Lue, and additional character
> repertoire.

Many New Tai Lue characters which were combining characters before are
not combining characters anymore.

> Implementers need to change code and check assumptions regarding
> case mappings New Tai Lue syllables, Han character ranges, and confusables.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[pabs3 at bonedaddy dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

Paul Wise <pabs3 at bonedaddy dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pabs3 at bonedaddy dot net

--- Comment #4 from Paul Wise <pabs3 at bonedaddy dot net> ---
What about removing the Unicode locale data from the glibc repository and
tarball and creating the glibc versions from the Unicode data at build time?
This would mean a new Unicode release would just need a rebuild of glibc to
include the new data?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[joseph at codesourcery dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #5 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
The results of a glibc build and install should depend only on the glibc 
source tree (preferably, be byte-for-byte reproducible in different 
builds), not on any network resources.  In addition, it's common for 
manual review and scripts changes to be needed as part of a Unicode 
update.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[pabs3 at bonedaddy dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #6 from Paul Wise <pabs3 at bonedaddy dot net> ---
That should be possible, just require the Unicode data at build time, just like
glibc requires gcc at build time. The Unicode data is already packaged in some
distributions as a unicode-data package. No network resources needed.

Depending on the Unicode data instead of embedding copies of it doesn't prevent
review and script changes.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #7 from Andreas Schwab <schwab@linux-m68k.org> ---
You are missing the point.  glibc requires a fixed version of the unicode data
which cannot be updated without updating the glibc sources.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[pabs3 at bonedaddy dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #8 from Paul Wise <pabs3 at bonedaddy dot net> ---
Why is that?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at redhat dot com

--- Comment #9 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Paul Wise from comment #8)
> Why is that?

Stability. We want to test and validate one version of Unicode data and update
the implementation to match.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [Bug localedata/18568] Update locale data to Unicode 8.0

[Keld Simonsen]

On Mon, Jun 29, 2015 at 08:04:58PM +0000, carlos at redhat dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18568
> 
> Carlos O'Donell <carlos at redhat dot com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |carlos at redhat dot com
> 
> --- Comment #9 from Carlos O'Donell <carlos at redhat dot com> ---
> (In reply to Paul Wise from comment #8)
> > Why is that?
> 
> Stability. We want to test and validate one version of Unicode data and update
> the implementation to match.

I think we should keep LC_CTYPE and LC_COLLATE data in  sync.
And then build LC_COLLATE on the latest approved ISO 14651 data.

Vbest regards
Keld

[Bug localedata/18568] Update locale data to Unicode 8.0

[joseph at codesourcery dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #11 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
Collation is the subject of bug 14095, not this one (and as noted there, 
is probably a lot more work to update because of all the local changes 
over the past 15 years - though maybe once the initial catching up is 
done, future Unicode updates can then update everything together).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[keld at keldix dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #10 from keld at keldix dot com <keld at keldix dot com> ---
On Mon, Jun 29, 2015 at 08:04:58PM +0000, carlos at redhat dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18568
> 
> Carlos O'Donell <carlos at redhat dot com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |carlos at redhat dot com
> 
> --- Comment #9 from Carlos O'Donell <carlos at redhat dot com> ---
> (In reply to Paul Wise from comment #8)
> > Why is that?
> 
> Stability. We want to test and validate one version of Unicode data and update
> the implementation to match.

I think we should keep LC_CTYPE and LC_COLLATE data in  sync.
And then build LC_COLLATE on the latest approved ISO 14651 data.

Vbest regards
Keld

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[pabs3 at bonedaddy dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #12 from Paul Wise <pabs3 at bonedaddy dot net> ---
On Mon, Jun 29, 2015 at 08:04:58PM +0000, carlos at redhat dot com wrote:

> Stability. We want to test and validate one version of Unicode
> data and update the implementation to match.

Is that testing and validation automated or manual?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #13 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Paul Wise from comment #12)
> On Mon, Jun 29, 2015 at 08:04:58PM +0000, carlos at redhat dot com wrote:
> 
> > Stability. We want to test and validate one version of Unicode
> > data and update the implementation to match.
> 
> Is that testing and validation automated or manual?

In the past the testing and validation was entirely manual, which is why
automatically migrating to a new version of Unicode was difficult and
potentially error prone. With the recent update by the Fedora i18n team (Mike
Fabian, Pravin Satpute) and with help of Alexandre Oliva (Red Hat), we were
able to automate the update of the character encoding and ctype tables. This
update included regression testing scripts that verify backwards compatibility
automatically. So in some ways we have automatic testing for that component of
the locales, but given the broader changes in Unicode 8.0, someone should do
the work and verify the results.

If you want to try update to Unicode 8.0, you can do so by looking at
localedata/unicode-gen/Makefile, updating the external properties files, and
running `make all` in that unicode-gen directory (distinct from glibc build).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #14 from Mike FABIAN <maiku.fabian at gmail dot com> ---
(In reply to Carlos O'Donell from comment #13)

> compatibility automatically. So in some ways we have automatic testing for
> that component of the locales, but given the broader changes in Unicode 8.0,
> someone should do the work and verify the results.
> 
> If you want to try update to Unicode 8.0, you can do so by looking at
> localedata/unicode-gen/Makefile, updating the external properties files, and
> running `make all` in that unicode-gen directory (distinct from glibc build).

My latest patch set doing the update to Unicode 8.0 (including updating
the translit_* files) is here:

https://sourceware.org/ml/libc-alpha/2015-06/msg00934.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |review-

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

--- Comment #15 from Mike FABIAN <maiku.fabian at gmail dot com> ---
The latest patch is here: 

https://sourceware.org/ml/libc-alpha/2015-07/msg00836.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18568] Update locale data to Unicode 8.0

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18568

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED
   Target Milestone|---                         |2.23

--- Comment #16 from Carlos O'Donell <carlos at redhat dot com> ---
This is now fixed.

commit 23256f5ed889266223380c02b2750d19e3fe666b
Author: Mike FABIAN <mfabian@redhat.com>
Date:   Thu Dec 10 00:30:51 2015 -0500

    Update to Unicode 8.0.0.

    Update __STDC_ISO_10646__ to 201505L for Unicode 8.0.0.
    Update character encoding, ctype, and transliteration tables.
    New scripts autogenerate transliteration tables.

-- 
You are receiving this mail because:
You are on the CC list for the bug.