[Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8

[Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://sourceware.org/bugz
                   |                            |illa/show_bug.cgi?id=26034

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] iconv allows encoding characters above U+10FFFF in UTF-8

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|https://sourceware.org/bugz |
                   |illa/show_bug.cgi?id=26034  |
                 CC|                            |johannes at sipsolutions dot net

--- Comment #5 from Florian Weimer <fweimer at redhat dot com> ---
*** Bug 26034 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|iconv allows encoding       |Restrict UTF-8 to 17
                   |characters above U+10FFFF   |planes, as required by RFC
                   |in UTF-8                    |3629

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[johannes at sipsolutions dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #6 from Johannes Berg <johannes at sipsolutions dot net> ---
I was looking around for the ISO, but only found this:

https://unicode.org/L2/L2010/10038-fcd10646-main.pdf

which does in fact also specify only up to 0x10ffff. So maybe that *did* get
settled, which the original report mentioned.


https://www.unicode.org/versions/Unicode13.0.0/UnicodeStandard-13.0.pdf

seems to say the same, and the unicode website says:

"This version of the Unicode Standard is also synchronized with ISO/IEC
10646:2020, sixth edition."

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #7 from Andreas Schwab <schwab@linux-m68k.org> ---
RFC 2044 defines UTF-8 as a 1-6 octet encoding, referencing ISO/IEC
10646-1:1993 as the source.  This was eventually updated by RFC 3629, which
introduced the U+10FFFF limit, but citing ISO/IEC 10646-1:2000 as without that
limit.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[johannes at sipsolutions dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #8 from Johannes Berg <johannes at sipsolutions dot net> ---
Oh, ok. The original comment here seemed to imply that ISO was the last one to
hold out for more space than the others.


To carry over some discussion from the bug I originally filed (which was since
closed as duplicate in favour of this one):

This came up because Python does this conversion using mbstowcs() and/or
mbrtowc(), but then later goes to check that valid characters were returned.

The python discussion is here:

https://bugs.python.org/issue35883


Note that this isn't just about the range, but also the RFC prohibits the
surrogate pair reservations:


RFC 3629:

   The definition of UTF-8 prohibits encoding character numbers between
   U+D800 and U+DFFF, which are reserved for use with the UTF-16
   encoding form (as surrogate pairs) and do not directly represent
   characters.


(Python internally may actually allow using this in an UTF-8-like encoded
string [that they call utf-8b] to carry arbitrary bytes around.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[fw at deneb dot enyo.de]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

Florian Weimer <fw at deneb dot enyo.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fw at deneb dot enyo.de

--- Comment #9 from Florian Weimer <fw at deneb dot enyo.de> ---
(In reply to Andreas Schwab from comment #7)
> RFC 2044 defines UTF-8 as a 1-6 octet encoding, referencing ISO/IEC
> 10646-1:1993 as the source.  This was eventually updated by RFC 3629, which
> introduced the U+10FFFF limit, but citing ISO/IEC 10646-1:2000 as without
> that limit.

Where? I think RFC 3629 still claims that the six byte limit per codepoint does
not exist, in section 10:

   Another security issue occurs when encoding to UTF-8: the ISO/IEC
   10646 description of UTF-8 allows encoding character numbers up to
   U+7FFFFFFF, yielding sequences of up to 6 bytes.  There is therefore
   a risk of buffer overflow if the range of character numbers is not
   explicitly limited to U+10FFFF or if buffer sizing doesn't take into
   account the possibility of 5- and 6-byte sequences.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #10 from Andreas Schwab <schwab@linux-m68k.org> ---
How does that disagree with what I wrote?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[fw at deneb dot enyo.de]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #11 from Florian Weimer <fw at deneb dot enyo.de> ---
(In reply to Andreas Schwab from comment #7)
> RFC 2044 defines UTF-8 as a 1-6 octet encoding, referencing ISO/IEC
> 10646-1:1993 as the source.  This was eventually updated by RFC 3629, which
> introduced the U+10FFFF limit, but citing ISO/IEC 10646-1:2000 as without
> that limit.

This is very misleading. I have a copy of ISO/IEC 10646-1 : 2000(E), bought
directly from ISO, and Annex D (which is normative) still specifies 7FFF FFFF
as the maximum UCS-4 value.

If UTF-8 is restricted to 17 planes in ISO 10646, this restriction has been
introduced in a later version of the standard.

I don't think this matters because RFC 3629 is a publicly accessible standard,
so I think this is what we should follow anyway.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #12 from Andreas Schwab <schwab@linux-m68k.org> ---
> This is very misleading. I have a copy of ISO/IEC 10646-1 : 2000(E), bought
> directly from ISO, and Annex D (which is normative) still specifies 7FFF
> FFFF as the maximum UCS-4 value.

In which way does that differ from what I wrote?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[joseph at codesourcery dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

--- Comment #13 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
The limit was in ISO 10646 in the 2011 edition but not in the 2003 
edition.  
https://sourceware.org/legacy-ml/libc-alpha/2012-09/msg00112.html has my 
notes on a previous investigation of when the limit was introduced.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug locale/2373] Restrict UTF-8 to 17 planes, as required by RFC 3629

[roman.zilka at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2373

Roman Žilka <roman.zilka at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |roman.zilka at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.