From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp72.ord1c.emailsrvr.com (smtp72.ord1c.emailsrvr.com [108.166.43.72]) by sourceware.org (Postfix) with ESMTPS id 550833858C3A for ; Sun, 27 Feb 2022 16:53:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 550833858C3A X-Auth-ID: tom@honermann.net Received: by smtp18.relay.ord1c.emailsrvr.com (Authenticated sender: tom-AT-honermann.net) with ESMTPSA id 32B6AE00D0 for ; Sun, 27 Feb 2022 11:53:20 -0500 (EST) Content-Type: multipart/mixed; boundary="------------gxsxyxD5czdfVvyVhg94qHQo" Message-ID: <29a48f8e-0c31-072c-ec36-8b62a0e1b430@honermann.net> Date: Sun, 27 Feb 2022 11:53:19 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Content-Language: en-US From: Tom Honermann Subject: [PATCH 1/3]: C++20 P0482R6 and C2X N2653: Fix for bug 25744, mbrtowc with Big5-HKSCS To: libc-alpha X-Classification-ID: 94f9124f-ec9c-43af-bb50-51b69921ce7b-1-1 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Feb 2022 16:53:23 -0000 This is a multi-part message in MIME format. --------------gxsxyxD5czdfVvyVhg94qHQo Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit This patch for bug 25744 [1] updates the Big5-HKSCS converter to properly maintain the lowest 3 bits of the mbstate_t __count data member. This change is necessary to ensure that state is correctly preserved when the converter encounters an incomplete multibyte character. More details are available in bug 25744 [1]. The code changes are styled to match how these bits are maintained by converters such as iso-2022-jp.c, ibm930.c, and others. Running 'grep __count' in the 'iconvdata' directory suggests that a number of other converters, euc-jisx0213.c for example, also fail to preserve these bits in some cases, though it may be that negative effects are not observed for those converters. This patch does not attempt to address such issues with other converters. Tested on Linux x86_64. Tom. [1]: Bug 25744 "mbrtowc with Big5-HKSCS returns 2 instead of 1 when consuming the second byte of certain double byte characters" https://sourceware.org/bugzilla/show_bug.cgi?id=25744 --------------gxsxyxD5czdfVvyVhg94qHQo Content-Type: text/x-patch; charset=UTF-8; name="n2653-1.patch" Content-Disposition: attachment; filename="n2653-1.patch" Content-Transfer-Encoding: base64 Y29tbWl0IDk1ODBmYzRlN2ZhMGNlMzNjMDQ5YjBjMmQ2MWI5ODQwNWZkZDJhZTMKQXV0aG9y OiBUb20gSG9uZXJtYW5uIDx0b21AaG9uZXJtYW5uLm5ldD4KRGF0ZTogICBXZWQgSmFuIDUg MTg6MDI6MjQgMjAyMiAtMDUwMAoKICAgIENvcnJlY3QgdGhlIEJpZzUtSEtTQ1MgY29udmVy dGVyIHRvIHByZXNlcnZlIGxvdyBvcmRlciBzdGF0ZSBiaXRzLgogICAgCiAgICBCWjogaHR0 cHM6Ly9zb3VyY2V3YXJlLm9yZy9idWd6aWxsYS9zaG93X2J1Zy5jZ2k/aWQ9MjU3NDQKCmRp ZmYgLS1naXQgYS9pY29udmRhdGEvYmlnNWhrc2NzLmMgYi9pY29udmRhdGEvYmlnNWhrc2Nz LmMKaW5kZXggYTI4YjE4YTVlYy4uZDEyMzg5YjJlMyAxMDA2NDQKLS0tIGEvaWNvbnZkYXRh L2JpZzVoa3Njcy5jCisrKyBiL2ljb252ZGF0YS9iaWc1aGtzY3MuYwpAQCAtMTc3NjksNyAr MTc3NjksNyBAQCBzdGF0aWMgc3RydWN0CiAgICB0aGUgb3V0cHV0IHN0YXRlIHRvIHRoZSBp bml0aWFsIHN0YXRlLiAgVGhpcyBoYXMgdG8gYmUgZG9uZSBkdXJpbmcgdGhlCiAgICBmbHVz aGluZy4gICovCiAjZGVmaW5lIEVNSVRfU0hJRlRfVE9fSU5JVCBcCi0gIGlmIChkYXRhLT5f X3N0YXRlcC0+X19jb3VudCAhPSAwKQkJCQkJICAgICAgXAorICBpZiAoKGRhdGEtPl9fc3Rh dGVwLT5fX2NvdW50ID4+IDMpICE9IDApCQkJCSAgICAgIFwKICAgICB7CQkJCQkJCQkJICAg ICAgXAogICAgICAgaWYgKEZST01fRElSRUNUSU9OKQkJCQkJCSAgICAgIFwKIAl7CQkJCQkJ CQkgICAgICBcCkBAIC0xNzc3OCw3ICsxNzc3OCw3IEBAIHN0YXRpYyBzdHJ1Y3QKIAkgICAg ICAvKiBXcml0ZSBvdXQgdGhlIGxhc3QgY2hhcmFjdGVyLiAgKi8JCQkgICAgICBcCiAJICAg ICAgKigodWludDMyX3QgKikgb3V0YnVmKSA9IGRhdGEtPl9fc3RhdGVwLT5fX2NvdW50ID4+ IDM7CSAgICAgIFwKIAkgICAgICBvdXRidWYgKz0gc2l6ZW9mICh1aW50MzJfdCk7CQkJCSAg ICAgIFwKLQkgICAgICBkYXRhLT5fX3N0YXRlcC0+X19jb3VudCA9IDA7CQkJCSAgICAgIFwK KwkgICAgICBkYXRhLT5fX3N0YXRlcC0+X19jb3VudCAmPSA3OwkJCQkgICAgICBcCiAJICAg IH0JCQkJCQkJCSAgICAgIFwKIAkgIGVsc2UJCQkJCQkJCSAgICAgIFwKIAkgICAgLyogV2Ug ZG9uJ3QgaGF2ZSBlbm91Z2ggcm9vbSBpbiB0aGUgb3V0cHV0IGJ1ZmZlci4gICovCSAgICAg IFwKQEAgLTE3NzkyLDcgKzE3NzkyLDcgQEAgc3RhdGljIHN0cnVjdAogCSAgICAgIHVpbnQz Ml90IGxhc3R0d28gPSBkYXRhLT5fX3N0YXRlcC0+X19jb3VudCA+PiAzOwkJICAgICAgXAog CSAgICAgICpvdXRidWYrKyA9IChsYXN0dHdvID4+IDgpICYgMHhmZjsJCQkgICAgICBcCiAJ ICAgICAgKm91dGJ1ZisrID0gbGFzdHR3byAmIDB4ZmY7CQkJCSAgICAgIFwKLQkgICAgICBk YXRhLT5fX3N0YXRlcC0+X19jb3VudCA9IDA7CQkJCSAgICAgIFwKKwkgICAgICBkYXRhLT5f X3N0YXRlcC0+X19jb3VudCAmPSA3OwkJCQkgICAgICBcCiAJICAgIH0JCQkJCQkJCSAgICAg IFwKIAkgIGVsc2UJCQkJCQkJCSAgICAgIFwKIAkgICAgLyogV2UgZG9uJ3QgaGF2ZSBlbm91 Z2ggcm9vbSBpbiB0aGUgb3V0cHV0IGJ1ZmZlci4gICovCSAgICAgIFwKQEAgLTE3ODc4LDcg KzE3ODc4LDcgQEAgc3RhdGljIHN0cnVjdAogCQkJCQkJCQkJICAgICAgXAogCQkvKiBPdGhl cndpc2Ugc3RvcmUgb25seSB0aGUgZmlyc3QgY2hhcmFjdGVyIG5vdywgYW5kCSAgICAgIFwK IAkJICAgcHV0IHRoZSBzZWNvbmQgb25lIGludG8gdGhlIHF1ZXVlLiAgKi8JCSAgICAgIFwK LQkJKnN0YXRlcCA9IGNoMiA8PCAzOwkJCQkJICAgICAgXAorCQkqc3RhdGVwID0gKGNoMiA8 PCAzKSB8ICgqc3RhdGVwICYgNyk7CQkJICAgICAgXAogCQkvKiBUZWxsIHRoZSBjYWxsZXIg d2h5IHdlIHRlcm1pbmF0ZSB0aGUgbG9vcC4gICovCSAgICAgIFwKIAkJcmVzdWx0ID0gX19H Q09OVl9GVUxMX09VVFBVVDsJCQkJICAgICAgXAogCQlicmVhazsJCQkJCQkJICAgICAgXApA QCAtMTc4OTUsNyArMTc4OTUsNyBAQCBzdGF0aWMgc3RydWN0CiAgICAgICB9CQkJCQkJCQkJ ICAgICAgXAogICAgIGVsc2UJCQkJCQkJCSAgICAgIFwKICAgICAgIC8qIENsZWFyIHRoZSBx dWV1ZSBhbmQgcHJvY2VlZCB0byBvdXRwdXQgdGhlIHNhdmVkIGNoYXJhY3Rlci4gICovCSAg ICAgIFwKLSAgICAgICpzdGF0ZXAgPSAwOwkJCQkJCQkgICAgICBcCisgICAgICAqc3RhdGVw ICY9IDc7CQkJCQkJCSAgICAgIFwKIAkJCQkJCQkJCSAgICAgIFwKICAgICBwdXQzMiAob3V0 cHRyLCBjaCk7CQkJCQkJCSAgICAgIFwKICAgICBvdXRwdHIgKz0gNDsJCQkJCQkJICAgICAg XApAQCAtMTc5NDYsNyArMTc5NDYsNyBAQCBzdGF0aWMgc3RydWN0CiAJICB9CQkJCQkJCQkg ICAgICBcCiAJKm91dHB0cisrID0gKGNoID4+IDgpICYgMHhmZjsJCQkJCSAgICAgIFwKIAkq b3V0cHRyKysgPSBjaCAmIDB4ZmY7CQkJCQkJICAgICAgXAotCSpzdGF0ZXAgPSAwOwkJCQkJ CQkgICAgICBcCisJKnN0YXRlcCAmPSA3OwkJCQkJCQkgICAgICBcCiAJaW5wdHIgKz0gNDsJ CQkJCQkJICAgICAgXAogCWNvbnRpbnVlOwkJCQkJCQkgICAgICBcCiAJCQkJCQkJCQkgICAg ICBcCkBAIC0xNzk1OSw3ICsxNzk1OSw3IEBAIHN0YXRpYyBzdHJ1Y3QKIAkgIH0JCQkJCQkJ CSAgICAgIFwKIAkqb3V0cHRyKysgPSAobGFzdHR3byA+PiA4KSAmIDB4ZmY7CQkJCSAgICAg IFwKIAkqb3V0cHRyKysgPSBsYXN0dHdvICYgMHhmZjsJCQkJCSAgICAgIFwKLQkqc3RhdGVw ID0gMDsJCQkJCQkJICAgICAgXAorCSpzdGF0ZXAgJj0gNzsJCQkJCQkJICAgICAgXAogCWNv bnRpbnVlOwkJCQkJCQkgICAgICBcCiAgICAgICB9CQkJCQkJCQkJICAgICAgXAogCQkJCQkJ CQkJICAgICAgXApAQCAtMTc5OTYsNyArMTc5OTYsNyBAQCBzdGF0aWMgc3RydWN0CiAJICAg LyogQ2hlY2sgZm9yIHBvc3NpYmxlIGNvbWJpbmluZyBjaGFyYWN0ZXIuICAqLwkJICAgICAg XAogCSAgICBpZiAoX19nbGliY191bmxpa2VseSAoY2ggPT0gMHhjYSB8fCBjaCA9PSAweGVh KSkJCSAgICAgIFwKIAkgICAgICB7CQkJCQkJCQkgICAgICBcCi0JCSpzdGF0ZXAgPSAoKGNw WzBdIDw8IDgpIHwgY3BbMV0pIDw8IDM7CQkJICAgICAgXAorCQkqc3RhdGVwID0gKCgoY3Bb MF0gPDwgOCkgfCBjcFsxXSkgPDwgMykgfCAoKnN0YXRlcCAmIDcpOyAgICAgIFwKIAkJaW5w dHIgKz0gNDsJCQkJCQkgICAgICBcCiAJCWNvbnRpbnVlOwkJCQkJCSAgICAgIFwKIAkgICAg ICB9CQkJCQkJCQkgICAgICBcCmRpZmYgLS1naXQgYS9pY29udmRhdGEvdHN0LWljb252LWJp ZzUtaGtzY3MtdG8tMnVjczQuYyBiL2ljb252ZGF0YS90c3QtaWNvbnYtYmlnNS1oa3Njcy10 by0ydWNzNC5jCmluZGV4IDk2MDFiNmMxZDkuLmUxNDcyZGMyZTIgMTAwNjQ0Ci0tLSBhL2lj b252ZGF0YS90c3QtaWNvbnYtYmlnNS1oa3Njcy10by0ydWNzNC5jCisrKyBiL2ljb252ZGF0 YS90c3QtaWNvbnYtYmlnNS1oa3Njcy10by0ydWNzNC5jCkBAIC0xMjgsNiArMTI4LDcxIEBA IGNoZWNrX2NvbnZlcnNpb24gKHN0cnVjdCB0ZXN0ZGF0YSB0ZXN0KQogICAgICAgcHJpbnRm ICgiZXJyb3I6IFJlc3VsdCBvZiB0aGlyZCBjb252ZXJzaW9uIHdhcyB3cm9uZy5cbiIpOwog ICAgICAgZXJyKys7CiAgICAgfQorCisgIC8qIE5vdyBwZXJmb3JtIHRoZSBzYW1lIHRlc3Qg YXMgYWJvdmUgY29uc3VtaW5nIG9uZSBieXRlIGF0IGEgdGltZS4gICovCisgIG1icyA9IHRl c3QuaW5wdXQ7CisgIG1lbXNldCAoJnN0LCAwLCBzaXplb2YgKHN0KSk7CisKKyAgLyogQ29u c3VtZSB0aGUgZmlyc3QgYnl0ZTsgZXhwZWN0IGFuIGluY29tcGxldGUgbXVsdGlieXRlIGNo YXJhY3Rlci4gICovCisgIHJldCA9IG1icnRvd2MgKCZ3YywgbWJzLCAxLCAmc3QpOworICBp ZiAocmV0ICE9IC0yKQorICAgIHsKKyAgICAgIHByaW50ZiAoImVycm9yOiBGaXJzdCBieXRl IGNvbnZlcnNpb24gcmV0dXJuZWQgJXpkLlxuIiwgcmV0KTsKKyAgICAgIGVycisrOworICAg IH0KKyAgLyogQWR2YW5jZSBwYXN0IHRoZSBmaXJzdCBjb25zdW1lZCBieXRlLiAgKi8KKyAg bWJzICs9IDE7CisgIC8qIENvbnN1bWUgdGhlIHNlY29uZCBieXRlOyBleHBlY3QgdGhlIGZp cnN0IHdjaGFyX3QuICAqLworICByZXQgPSBtYnJ0b3djICgmd2MsIG1icywgMSwgJnN0KTsK KyAgaWYgKHJldCAhPSAxKQorICAgIHsKKyAgICAgIHByaW50ZiAoImVycm9yOiBTZWNvbmQg Ynl0ZSBjb252ZXJzaW9uIHJldHVybmVkICV6ZC5cbiIsIHJldCk7CisgICAgICBlcnIrKzsK KyAgICB9CisgIC8qIEFkdmFuY2UgcGFzdCB0aGUgc2Vjb25kIGNvbnN1bWVkIGJ5dGUuICAq LworICBtYnMgKz0gMTsKKyAgaWYgKHdjICE9IHRlc3QuZXhwZWN0ZWRbMF0pCisgICAgewor ICAgICAgcHJpbnRmICgiZXJyb3I6IFJlc3VsdCBvZiBmaXJzdCB3Y2hhcl90IGNvbnZlcnNp b24gd2FzIHdyb25nLlxuIik7CisgICAgICBlcnIrKzsKKyAgICB9CisgIC8qIENvbnN1bWUg bm8gYnl0ZXM7IGV4cGVjdCB0aGUgc2Vjb25kIHdjaGFyX3QuICAqLworICByZXQgPSBtYnJ0 b3djICgmd2MsIG1icywgMSwgJnN0KTsKKyAgaWYgKHJldCAhPSAwKQorICAgIHsKKyAgICAg IHByaW50ZiAoImVycm9yOiBGaXJzdCBhdHRlbXB0IG9mIHRoaXJkIGJ5dGUgY29udmVyc2lv biByZXR1cm5lZCAlemQuXG4iLCByZXQpOworICAgICAgZXJyKys7CisgICAgfQorICAvKiBE byBub3QgYWR2YW5jZSBwYXN0IHRoZSB0aGlyZCBieXRlLiAgKi8KKyAgbWJzICs9IDA7Cisg IGlmICh3YyAhPSB0ZXN0LmV4cGVjdGVkWzFdKQorICAgIHsKKyAgICAgIHByaW50ZiAoImVy cm9yOiBSZXN1bHQgb2Ygc2Vjb25kIHdjaGFyX3QgY29udmVyc2lvbiB3YXMgd3JvbmcuXG4i KTsKKyAgICAgIGVycisrOworICAgIH0KKyAgLyogQWZ0ZXIgdGhlIHNlY29uZCB3Y2hhcl90 IGNvbnZlcnNpb24sIHRoZSBjb252ZXJ0ZXIgc2hvdWxkIGJlIGluCisgICAgIHRoZSBpbml0 aWFsIHN0YXRlIHNpbmNlIHRoZSB0d28gaW5wdXQgQklHNS1IS1NDUyBieXRlcyBoYXZlIGJl ZW4KKyAgICAgY29uc3VtZWQgYW5kIHRoZSB0d28gd2NoYXJfdCdzIGhhdmUgYmVlbiBvdXRw dXQuICAqLworICBpZiAobWJzaW5pdCAoJnN0KSA9PSAwKQorICAgIHsKKyAgICAgIHByaW50 ZiAoImVycm9yOiBDb252ZXJ0ZXIgbm90IGluIGluaXRpYWwgc3RhdGUuXG4iKTsKKyAgICAg IGVycisrOworICAgIH0KKyAgLyogQ29uc3VtZSB0aGUgdGhpcmQgYnl0ZTsgZXhwZWN0IHRo ZSB0aGlyZCB3Y2hhcl90LiAgKi8KKyAgcmV0ID0gbWJydG93YyAoJndjLCBtYnMsIDEsICZz dCk7CisgIGlmIChyZXQgIT0gMSkKKyAgICB7CisgICAgICBwcmludGYgKCJlcnJvcjogVGhp cmQgYnl0ZSBjb252ZXJzaW9uIHJldHVybmVkICV6ZC5cbiIsIHJldCk7CisgICAgICBlcnIr KzsKKyAgICB9CisgIC8qIEFkdmFuY2UgcGFzdCB0aGUgdGhpcmQgY29uc3VtZWQgYnl0ZS4g ICovCisgIG1icyArPSAxOworICBpZiAod2MgIT0gdGVzdC5leHBlY3RlZFsyXSkKKyAgICB7 CisgICAgICBwcmludGYgKCJlcnJvcjogUmVzdWx0IG9mIHRoaXJkIHdjaGFyX3QgY29udmVy c2lvbiB3YXMgd3JvbmcuXG4iKTsKKyAgICAgIGVycisrOworICAgIH0KKwogICAvKiBSZXR1 cm4gMCBpZiB3ZSBzYXcgbm8gZXJyb3JzLiAgKi8KICAgcmV0dXJuIGVycjsKIH0K --------------gxsxyxD5czdfVvyVhg94qHQo--