From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 29690 invoked by alias); 29 Jan 2011 15:10:27 -0000 Received: (qmail 29681 invoked by uid 22791); 29 Jan 2011 15:10:26 -0000 X-SWARE-Spam-Status: No, hits=-6.8 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,SPF_HELO_PASS,T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Sat, 29 Jan 2011 15:10:21 +0000 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id p0TFAJdn008016 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 29 Jan 2011 10:10:19 -0500 Received: from [10.3.113.146] (ovpn-113-146.phx2.redhat.com [10.3.113.146]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p0TFAIsn032271; Sat, 29 Jan 2011 10:10:19 -0500 Message-ID: <4D442DDA.4050807@redhat.com> Date: Sat, 29 Jan 2011 17:51:00 -0000 From: Eric Blake User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc14 Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.7 MIME-Version: 1.0 To: cygwin@cygwin.com, bug-gnu-libiconv@gnu.org Subject: Re: Bug in libiconv? References: <201101282312.50298.bruno@clisp.org> <20110129123014.GA8671@calimero.vinschen.de> In-Reply-To: <20110129123014.GA8671@calimero.vinschen.de> OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigEB77E732F4D2D606FC5E086B" X-IsSubscribed: yes Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com X-SW-Source: 2011-01/txt/msg00417.txt.bz2 --------------enigEB77E732F4D2D606FC5E086B Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-length: 2744 On 01/29/2011 05:30 AM, Corinna Vinschen wrote: >> But when characters outside the basic plane, such as >> U+12345 (CUNEIFORM SIGN URU TIMES KI), are encoded by 2 consecutive wcha= r_t >> values, values of type wchar_t don't correspond to ISO/IEC 10646 charact= ers. >> (Or maybe I'm underestimating what "coded representations" means...?) >=20 > I don't read that from your above quote. The core is that the *type* > wchar_t is a *coded* *representation* of the characters defined in > 10646. At no point it says that a single wchar_t value must represent a > single character from 10646. So I take it that UTF-16 is a valid, coded > representation of the characters from 10646. POSIX is clear that wchar_t must be wide enough so that 1 wchar_t is one character. Which limits a 2-byte wchar_t to just the Unicode basic plane. There's nothing cygwin can do about this other than break LOTS of ABI to support a 4-byte wchar_t to supply all of Unicode. http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html#tag= _06_03 "All wide-character codes in a given process consist of an equal number of bits. This is in contrast to characters, which can consist of a variable number of bytes. The byte or byte sequence that represents a character can also be represented as a wide-character code. Wide-character codes thus provide a uniform size for manipulating text data." So, using UTF-16 surrogate encodings for characters outside the basic plane violates POSIX, but it's the best we can do for those characters. > I've put a lot of effort in 2009 and early 2010 to make the wchar_t > representation in Cygwin and newlib as much Unicode 5.2 compatible as > possible. Even the wcrtomb and mbrtowc functions in newlib are capable > of dealing with UTF-16 surrogates. And I appreciate that effort - even though it means wchar_t is just as painful as multi-byte char characters in that an array of wchar_t is not necessarily that many characters long, but only when surrogates are involved. >=20 > However, given that Windows XP basically only supports the charset from > Unicode 4.0, and given that Cygwin's support for east-asian double and > triple byte codesets (Big5, GBK, eucKR, eucJP, and a SJIS/CP932 bastard) > still requires the underlying Windows conversion functions, I've set > __STDC_ISO_10646__ to a value which reflects Unicode 4.0 (200305L) for > Cygwin 1.7.8. Someday when gcc has better support for C+1x 16- and 32-bit characters (regardless of the sizing of wchar_t), then we can add all the new 32-bit character APIs that use Unicode unimpeded, without breaking existing ones that use wchar_t. --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enigEB77E732F4D2D606FC5E086B Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" Content-length: 619 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJNRC3aAAoJEKeha0olJ0NqQ98IAIE5vfjLBgcVJ7thBoEFZv/W KwsuOkwMAygUqM/kK3gL5EnHEqK/LSsLkNz/Qbdva+mcpzRCB570ys4hbthnjJTt PIOAvDqGWa7PYN7DqKayDnIp8tWmgtkgsoCvXhiF+XuKB8zbhlpQo5g+PkKbU7Y3 2FdBOtHIqLPli2Xhv6+Nie+OkvLLaQV3DJuaqVuPtxUWAsliFM0P7sbzk6CZ9Nq0 ZUvsPuZUN3tfcM98vMrRfL0wJLHSprF61nPX8vJrHdRFMTVRPy56acW5L65mfxs3 QQgoE3ht8HK5Ve//olXQOMnkjyqiTqHuN62uTu6ja4FmW8aB6DYDw8RgLe9dwGQ= =eZ/A -----END PGP SIGNATURE----- --------------enigEB77E732F4D2D606FC5E086B--