* [Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
@ 2020-06-02 10:37 ` fweimer at redhat dot com
2020-06-02 11:33 ` fweimer at redhat dot com
` (10 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: fweimer at redhat dot com @ 2020-06-02 10:37 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
Florian Weimer <fweimer at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://sourceware.org/bugz
| |illa/show_bug.cgi?id=26034
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
2020-06-02 10:37 ` [Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8 fweimer at redhat dot com
@ 2020-06-02 11:33 ` fweimer at redhat dot com
2020-06-02 11:38 ` [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629 fweimer at redhat dot com
` (9 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: fweimer at redhat dot com @ 2020-06-02 11:33 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
Florian Weimer <fweimer at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also|https://sourceware.org/bugz |
|illa/show_bug.cgi?id=26034 |
CC| |johannes at sipsolutions dot net
--- Comment #5 from Florian Weimer <fweimer at redhat dot com> ---
*** Bug 26034 has been marked as a duplicate of this bug. ***
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
2020-06-02 10:37 ` [Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8 fweimer at redhat dot com
2020-06-02 11:33 ` fweimer at redhat dot com
@ 2020-06-02 11:38 ` fweimer at redhat dot com
2020-06-05 20:42 ` johannes at sipsolutions dot net
` (8 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: fweimer at redhat dot com @ 2020-06-02 11:38 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
Florian Weimer <fweimer at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|iconv allows encoding |Restrict UTF-8 to 17
|characters above U+10FFFF |planes, as required by RFC
|in UTF-8 |3629
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (2 preceding siblings ...)
2020-06-02 11:38 ` [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629 fweimer at redhat dot com
@ 2020-06-05 20:42 ` johannes at sipsolutions dot net
2020-06-05 21:39 ` schwab@linux-m68k.org
` (7 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: johannes at sipsolutions dot net @ 2020-06-05 20:42 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #6 from Johannes Berg <johannes at sipsolutions dot net> ---
I was looking around for the ISO, but only found this:
https://unicode.org/L2/L2010/10038-fcd10646-main.pdf
which does in fact also specify only up to 0x10ffff. So maybe that *did* get
settled, which the original report mentioned.
https://www.unicode.org/versions/Unicode13.0.0/UnicodeStandard-13.0.pdf
seems to say the same, and the unicode website says:
"This version of the Unicode Standard is also synchronized with ISO/IEC
10646:2020, sixth edition."
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (3 preceding siblings ...)
2020-06-05 20:42 ` johannes at sipsolutions dot net
@ 2020-06-05 21:39 ` schwab@linux-m68k.org
2020-06-06 21:53 ` johannes at sipsolutions dot net
` (6 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: schwab@linux-m68k.org @ 2020-06-05 21:39 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #7 from Andreas Schwab <schwab@linux-m68k.org> ---
RFC 2044 defines UTF-8 as a 1-6 octet encoding, referencing ISO/IEC
10646-1:1993 as the source. This was eventually updated by RFC 3629, which
introduced the U+10FFFF limit, but citing ISO/IEC 10646-1:2000 as without that
limit.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (4 preceding siblings ...)
2020-06-05 21:39 ` schwab@linux-m68k.org
@ 2020-06-06 21:53 ` johannes at sipsolutions dot net
2020-06-07 5:37 ` fw at deneb dot enyo.de
` (5 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: johannes at sipsolutions dot net @ 2020-06-06 21:53 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #8 from Johannes Berg <johannes at sipsolutions dot net> ---
Oh, ok. The original comment here seemed to imply that ISO was the last one to
hold out for more space than the others.
To carry over some discussion from the bug I originally filed (which was since
closed as duplicate in favour of this one):
This came up because Python does this conversion using mbstowcs() and/or
mbrtowc(), but then later goes to check that valid characters were returned.
The python discussion is here:
https://bugs.python.org/issue35883
Note that this isn't just about the range, but also the RFC prohibits the
surrogate pair reservations:
RFC 3629:
The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16
encoding form (as surrogate pairs) and do not directly represent
characters.
(Python internally may actually allow using this in an UTF-8-like encoded
string [that they call utf-8b] to carry arbitrary bytes around.)
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (5 preceding siblings ...)
2020-06-06 21:53 ` johannes at sipsolutions dot net
@ 2020-06-07 5:37 ` fw at deneb dot enyo.de
2020-06-07 6:17 ` schwab@linux-m68k.org
` (4 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: fw at deneb dot enyo.de @ 2020-06-07 5:37 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
Florian Weimer <fw at deneb dot enyo.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |fw at deneb dot enyo.de
--- Comment #9 from Florian Weimer <fw at deneb dot enyo.de> ---
(In reply to Andreas Schwab from comment #7)
> RFC 2044 defines UTF-8 as a 1-6 octet encoding, referencing ISO/IEC
> 10646-1:1993 as the source. This was eventually updated by RFC 3629, which
> introduced the U+10FFFF limit, but citing ISO/IEC 10646-1:2000 as without
> that limit.
Where? I think RFC 3629 still claims that the six byte limit per codepoint does
not exist, in section 10:
Another security issue occurs when encoding to UTF-8: the ISO/IEC
10646 description of UTF-8 allows encoding character numbers up to
U+7FFFFFFF, yielding sequences of up to 6 bytes. There is therefore
a risk of buffer overflow if the range of character numbers is not
explicitly limited to U+10FFFF or if buffer sizing doesn't take into
account the possibility of 5- and 6-byte sequences.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (6 preceding siblings ...)
2020-06-07 5:37 ` fw at deneb dot enyo.de
@ 2020-06-07 6:17 ` schwab@linux-m68k.org
2020-06-30 13:05 ` fw at deneb dot enyo.de
` (3 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: schwab@linux-m68k.org @ 2020-06-07 6:17 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #10 from Andreas Schwab <schwab@linux-m68k.org> ---
How does that disagree with what I wrote?
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (7 preceding siblings ...)
2020-06-07 6:17 ` schwab@linux-m68k.org
@ 2020-06-30 13:05 ` fw at deneb dot enyo.de
2020-06-30 13:18 ` schwab@linux-m68k.org
` (2 subsequent siblings)
11 siblings, 0 replies; 12+ messages in thread
From: fw at deneb dot enyo.de @ 2020-06-30 13:05 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #11 from Florian Weimer <fw at deneb dot enyo.de> ---
(In reply to Andreas Schwab from comment #7)
> RFC 2044 defines UTF-8 as a 1-6 octet encoding, referencing ISO/IEC
> 10646-1:1993 as the source. This was eventually updated by RFC 3629, which
> introduced the U+10FFFF limit, but citing ISO/IEC 10646-1:2000 as without
> that limit.
This is very misleading. I have a copy of ISO/IEC 10646-1 : 2000(E), bought
directly from ISO, and Annex D (which is normative) still specifies 7FFF FFFF
as the maximum UCS-4 value.
If UTF-8 is restricted to 17 planes in ISO 10646, this restriction has been
introduced in a later version of the standard.
I don't think this matters because RFC 3629 is a publicly accessible standard,
so I think this is what we should follow anyway.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (8 preceding siblings ...)
2020-06-30 13:05 ` fw at deneb dot enyo.de
@ 2020-06-30 13:18 ` schwab@linux-m68k.org
2020-06-30 14:56 ` joseph at codesourcery dot com
2023-03-01 14:37 ` roman.zilka at gmail dot com
11 siblings, 0 replies; 12+ messages in thread
From: schwab@linux-m68k.org @ 2020-06-30 13:18 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #12 from Andreas Schwab <schwab@linux-m68k.org> ---
> This is very misleading. I have a copy of ISO/IEC 10646-1 : 2000(E), bought
> directly from ISO, and Annex D (which is normative) still specifies 7FFF
> FFFF as the maximum UCS-4 value.
In which way does that differ from what I wrote?
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (9 preceding siblings ...)
2020-06-30 13:18 ` schwab@linux-m68k.org
@ 2020-06-30 14:56 ` joseph at codesourcery dot com
2023-03-01 14:37 ` roman.zilka at gmail dot com
11 siblings, 0 replies; 12+ messages in thread
From: joseph at codesourcery dot com @ 2020-06-30 14:56 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
--- Comment #13 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
The limit was in ISO 10646 in the 2011 edition but not in the 2003
edition.
https://sourceware.org/legacy-ml/libc-alpha/2012-09/msg00112.html has my
notes on a previous investigation of when the limit was introduced.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread* [Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629
[not found] <bug-2373-131@http.sourceware.org/bugzilla/>
` (10 preceding siblings ...)
2020-06-30 14:56 ` joseph at codesourcery dot com
@ 2023-03-01 14:37 ` roman.zilka at gmail dot com
11 siblings, 0 replies; 12+ messages in thread
From: roman.zilka at gmail dot com @ 2023-03-01 14:37 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=2373
Roman Žilka <roman.zilka at gmail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |roman.zilka at gmail dot com
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 12+ messages in thread