From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <gcc-bugzilla@gcc.gnu.org>
Received: by sourceware.org (Postfix, from userid 48)
	id 6C5CF3858D39; Tue, 28 Feb 2023 23:20:59 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6C5CF3858D39
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org;
	s=default; t=1677626459;
	bh=IL/BNSKSm0OsZArRS2f/1QbmXHIni291ebQSVrW0eY8=;
	h=From:To:Subject:Date:From;
	b=h57sFPTf93Cw5kBtQlPZmCN6I7zLotiGs+SKlDQdtmv2cqoS9Jg8jDhxy5MT++/M2
	 6PDK4pXSo66ovKQA5YIhL6EN3UnSVRgTXzTR+oFLhDmGx23xH+ecrYAmgoCHEMirLQ
	 qI6WDMWFvAA0CErstbXc1O8mpqHGY7ijC4m1fN5w=
From: "dmjpp at hotmail dot com" <gcc-bugzilla@gcc.gnu.org>
To: gcc-bugs@gcc.gnu.org
Subject: [Bug libstdc++/108976] New: codecvt for Unicode allows surrogate
 code points
Date: Tue, 28 Feb 2023 23:20:58 +0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: gcc
X-Bugzilla-Component: libstdc++
X-Bugzilla-Version: 13.0
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: normal
X-Bugzilla-Who: dmjpp at hotmail dot com
X-Bugzilla-Status: UNCONFIRMED
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: P3
X-Bugzilla-Assigned-To: unassigned at gcc dot gnu.org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status
 bug_severity priority component assigned_to reporter target_milestone
Message-ID: <bug-108976-4@http.gcc.gnu.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://gcc.gnu.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
List-Id: <gcc-bugs.sourceware.org>

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D108976

            Bug ID: 108976
           Summary: codecvt for Unicode allows surrogate code points
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libstdc++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: dmjpp at hotmail dot com
  Target Milestone: ---

Text in valid Unicode should never contain surrogate code POINTS. Those are
only allowed in UTF-16, but only as code UNITS and must be properly paired.

UTF-8 text in its strictest form must not contain surrogates but in a sligh=
tly
relaxed form surrogates can be easily encoded as 3-byte sequences. Same can=
 be
said for UTF-32 and UCS-2. Only UTF-16 is immune to the error of surrogate =
code
POINT (they are treated as UNITS).

Codecvts in libstdc++ currently allow surrogate code points in some cases. =
Here
is a minimal reproduction (asserts are the correct behavior):

#include <locale>
#include <cassert>

void u32()
{
    using namespace std;
    auto& f =3D use_facet<codecvt<char32_t, char, mbstate_t>>(locale::class=
ic());

    char u8str[] =3D "\uC800\uCBFF\uCC00\uCFFF";
    u8str[0] =3D u8str[3] =3D u8str[6] =3D u8str[9] =3D 0xED; // turn the C=
 into D.
    // now the string is D800, DBFF, DC00 and DFFF encoded in relaxed UTF-8
    // that allows surrogate code points.
    char32_t u32str[] =3D {0xD800, 0xDBFF, 0xDC00, 0xDFFF, 0};

    char32_t u32out[1];
    const char* from_next;
    char32_t* to_next;
    mbstate_t st =3D {};
    auto res =3D f.in(st, u8str, u8str+3, from_next, u32out, u32out+1, to_n=
ext);
    assert(res =3D=3D f.error);
    assert(from_next =3D=3D u8str);
    assert(to_next =3D=3D u32out);

    st =3D {};
    auto l =3D f.length(st, u8str, u8str+3, 1);
    assert(l =3D=3D 0);

    char u8out[3];
    const char32_t* from_next2;
    char* to_next2;
    st =3D {};
    res =3D f.out(st, u32str, u32str+1, from_next2, u8out, u8out+3, to_next=
2);
    assert(res =3D=3D f.error);
    assert(from_next2 =3D=3D u32str);
    assert(to_next2 =3D=3D u8out);
}
void u16()
{
    using namespace std;
    auto& f =3D use_facet<codecvt<char16_t, char, mbstate_t>>(locale::class=
ic());

    char u8str[] =3D "\uC800\uCBFF\uCC00\uCFFF";
    u8str[0] =3D u8str[3] =3D u8str[6] =3D u8str[9] =3D 0xED; // turn the C=
 into D.
    // now the string is D800, DBFF, DC00 and DFFF encoded in relaxed UTF-8
    // that allows surrogates.

    char16_t u16out[1];
    const char* from_next;
    char16_t* to_next;
    mbstate_t st =3D {};
    auto res =3D f.in(st, u8str, u8str+3, from_next, u16out, u16out+1, to_n=
ext);
    assert(res =3D=3D f.error);
    assert(from_next =3D=3D u8str);
    assert(to_next =3D=3D u16out);

    st =3D {};
    auto l =3D f.length(st, u8str, u8str+3, 1);
    assert(l =3D=3D 0);
}
int main()
{
    u32();
    u16();
}

>From reading the file codecvt.cc the following conversions have the bug:

- From UTF-8 to any other encoding.
- From UTF-32/UCS-4 to any other encoding.

Those that read from UCS-2 seem to me like they properly report the error.
Reading from UTF-16 can not have this bug by definition. From what I checke=
d,
the functions for reading UTF-16 properly treat unpaired surrogate code uni=
ts
as error.=