From: Eric Blake <eblake@redhat.com>
To: cygwin@cygwin.com, bug-gnu-libiconv@gnu.org
Subject: Re: Bug in libiconv?
Date: Sat, 29 Jan 2011 17:51:00 -0000 [thread overview]
Message-ID: <4D442DDA.4050807@redhat.com> (raw)
In-Reply-To: <20110129123014.GA8671@calimero.vinschen.de>
[-- Attachment #1: Type: text/plain, Size: 2788 bytes --]
On 01/29/2011 05:30 AM, Corinna Vinschen wrote:
>> But when characters outside the basic plane, such as
>> U+12345 (CUNEIFORM SIGN URU TIMES KI), are encoded by 2 consecutive wchar_t
>> values, values of type wchar_t don't correspond to ISO/IEC 10646 characters.
>> (Or maybe I'm underestimating what "coded representations" means...?)
>
> I don't read that from your above quote. The core is that the *type*
> wchar_t is a *coded* *representation* of the characters defined in
> 10646. At no point it says that a single wchar_t value must represent a
> single character from 10646. So I take it that UTF-16 is a valid, coded
> representation of the characters from 10646.
POSIX is clear that wchar_t must be wide enough so that 1 wchar_t is one
character. Which limits a 2-byte wchar_t to just the Unicode basic
plane. There's nothing cygwin can do about this other than break LOTS
of ABI to support a 4-byte wchar_t to supply all of Unicode.
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html#tag_06_03
"All wide-character codes in a given process consist of an equal number
of bits. This is in contrast to characters, which can consist of a
variable number of bytes. The byte or byte sequence that represents a
character can also be represented as a wide-character code.
Wide-character codes thus provide a uniform size for manipulating text
data."
So, using UTF-16 surrogate encodings for characters outside the basic
plane violates POSIX, but it's the best we can do for those characters.
> I've put a lot of effort in 2009 and early 2010 to make the wchar_t
> representation in Cygwin and newlib as much Unicode 5.2 compatible as
> possible. Even the wcrtomb and mbrtowc functions in newlib are capable
> of dealing with UTF-16 surrogates.
And I appreciate that effort - even though it means wchar_t is just as
painful as multi-byte char characters in that an array of wchar_t is not
necessarily that many characters long, but only when surrogates are
involved.
>
> However, given that Windows XP basically only supports the charset from
> Unicode 4.0, and given that Cygwin's support for east-asian double and
> triple byte codesets (Big5, GBK, eucKR, eucJP, and a SJIS/CP932 bastard)
> still requires the underlying Windows conversion functions, I've set
> __STDC_ISO_10646__ to a value which reflects Unicode 4.0 (200305L) for
> Cygwin 1.7.8.
Someday when gcc has better support for C+1x 16- and 32-bit characters
(regardless of the sizing of wchar_t), then we can add all the new
32-bit character APIs that use Unicode unimpeded, without breaking
existing ones that use wchar_t.
--
Eric Blake eblake@redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]
next prev parent reply other threads:[~2011-01-29 15:10 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-29 2:15 Bruno Haible
2011-01-29 12:34 ` Charles Wilson
2011-01-29 13:20 ` Charles Wilson
2011-01-29 17:15 ` Corinna Vinschen
2011-01-29 16:02 ` Corinna Vinschen
2011-01-29 17:51 ` Eric Blake [this message]
2011-01-29 18:12 ` Corinna Vinschen
2011-01-29 18:28 ` Eric Blake
2011-01-30 11:34 ` Corinna Vinschen
2011-01-30 11:43 ` Corinna Vinschen
2011-01-30 2:40 ` Corinna Vinschen
-- strict thread matches above, loose matches on Subject: below --
2011-02-02 18:58 Bruno Haible
2011-02-02 21:20 ` Corinna Vinschen
2011-02-02 22:57 ` Charles Wilson
2011-01-27 16:06 simrw
2011-01-26 13:39 simrw
2011-01-26 13:50 ` Corinna Vinschen
2011-01-26 17:01 ` Charles Wilson
2011-01-26 22:39 ` Corinna Vinschen
2011-01-25 6:36 Corinna Vinschen
2011-01-25 11:15 ` Charles Wilson
2011-01-25 15:04 ` Corinna Vinschen
2011-01-25 18:58 ` Charles Wilson
2011-01-25 20:11 ` Corinna Vinschen
2011-01-28 22:13 ` Charles Wilson
2011-01-27 5:46 ` Charles Wilson
2011-01-27 16:05 ` Corinna Vinschen
2011-01-27 17:18 ` Charles Wilson
2011-01-27 3:53 ` Charles Wilson
2011-01-27 16:21 ` Corinna Vinschen
2011-01-27 17:39 ` Charles Wilson
2011-01-27 18:05 ` Corinna Vinschen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D442DDA.4050807@redhat.com \
--to=eblake@redhat.com \
--cc=bug-gnu-libiconv@gnu.org \
--cc=cygwin@cygwin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).