public inbox for gcc-bugs@sourceware.org help / color / mirror / Atom feed
From: "dmjpp at hotmail dot com" <gcc-bugzilla@gcc.gnu.org> To: gcc-bugs@gcc.gnu.org Subject: [Bug libstdc++/108976] New: codecvt for Unicode allows surrogate code points Date: Tue, 28 Feb 2023 23:20:58 +0000 [thread overview] Message-ID: <bug-108976-4@http.gcc.gnu.org/bugzilla/> (raw) https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108976 Bug ID: 108976 Summary: codecvt for Unicode allows surrogate code points Product: gcc Version: 13.0 Status: UNCONFIRMED Severity: normal Priority: P3 Component: libstdc++ Assignee: unassigned at gcc dot gnu.org Reporter: dmjpp at hotmail dot com Target Milestone: --- Text in valid Unicode should never contain surrogate code POINTS. Those are only allowed in UTF-16, but only as code UNITS and must be properly paired. UTF-8 text in its strictest form must not contain surrogates but in a slightly relaxed form surrogates can be easily encoded as 3-byte sequences. Same can be said for UTF-32 and UCS-2. Only UTF-16 is immune to the error of surrogate code POINT (they are treated as UNITS). Codecvts in libstdc++ currently allow surrogate code points in some cases. Here is a minimal reproduction (asserts are the correct behavior): #include <locale> #include <cassert> void u32() { using namespace std; auto& f = use_facet<codecvt<char32_t, char, mbstate_t>>(locale::classic()); char u8str[] = "\uC800\uCBFF\uCC00\uCFFF"; u8str[0] = u8str[3] = u8str[6] = u8str[9] = 0xED; // turn the C into D. // now the string is D800, DBFF, DC00 and DFFF encoded in relaxed UTF-8 // that allows surrogate code points. char32_t u32str[] = {0xD800, 0xDBFF, 0xDC00, 0xDFFF, 0}; char32_t u32out[1]; const char* from_next; char32_t* to_next; mbstate_t st = {}; auto res = f.in(st, u8str, u8str+3, from_next, u32out, u32out+1, to_next); assert(res == f.error); assert(from_next == u8str); assert(to_next == u32out); st = {}; auto l = f.length(st, u8str, u8str+3, 1); assert(l == 0); char u8out[3]; const char32_t* from_next2; char* to_next2; st = {}; res = f.out(st, u32str, u32str+1, from_next2, u8out, u8out+3, to_next2); assert(res == f.error); assert(from_next2 == u32str); assert(to_next2 == u8out); } void u16() { using namespace std; auto& f = use_facet<codecvt<char16_t, char, mbstate_t>>(locale::classic()); char u8str[] = "\uC800\uCBFF\uCC00\uCFFF"; u8str[0] = u8str[3] = u8str[6] = u8str[9] = 0xED; // turn the C into D. // now the string is D800, DBFF, DC00 and DFFF encoded in relaxed UTF-8 // that allows surrogates. char16_t u16out[1]; const char* from_next; char16_t* to_next; mbstate_t st = {}; auto res = f.in(st, u8str, u8str+3, from_next, u16out, u16out+1, to_next); assert(res == f.error); assert(from_next == u8str); assert(to_next == u16out); st = {}; auto l = f.length(st, u8str, u8str+3, 1); assert(l == 0); } int main() { u32(); u16(); } From reading the file codecvt.cc the following conversions have the bug: - From UTF-8 to any other encoding. - From UTF-32/UCS-4 to any other encoding. Those that read from UCS-2 seem to me like they properly report the error. Reading from UTF-16 can not have this bug by definition. From what I checked, the functions for reading UTF-16 properly treat unpaired surrogate code units as error.
next reply other threads:[~2023-02-28 23:20 UTC|newest] Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-02-28 23:20 dmjpp at hotmail dot com [this message] 2023-03-02 11:03 ` [Bug libstdc++/108976] " redi at gcc dot gnu.org 2023-03-02 11:08 ` dmjpp at hotmail dot com 2023-03-02 11:17 ` redi at gcc dot gnu.org 2023-03-07 20:17 ` dmjpp at hotmail dot com 2023-03-07 21:43 ` redi at gcc dot gnu.org 2023-03-08 14:11 ` dmjpp at hotmail dot com 2023-04-18 13:45 ` dmjpp at hotmail dot com 2023-09-29 15:01 ` cvs-commit at gcc dot gnu.org 2024-01-05 16:34 ` dmjpp at hotmail dot com 2024-01-05 18:57 ` redi at gcc dot gnu.org 2024-01-13 11:56 ` dmjpp at hotmail dot com 2024-01-13 12:25 ` redi at gcc dot gnu.org 2024-05-21 9:14 ` jakub at gcc dot gnu.org 2024-05-21 22:03 ` cvs-commit at gcc dot gnu.org 2024-05-21 22:03 ` redi at gcc dot gnu.org
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=bug-108976-4@http.gcc.gnu.org/bugzilla/ \ --to=gcc-bugzilla@gcc.gnu.org \ --cc=gcc-bugs@gcc.gnu.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).