From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 91874 invoked by alias); 1 Apr 2015 13:34:05 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 91856 invoked by uid 89); 1 Apr 2015 13:34:04 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-4.9 required=5.0 tests=AWL,BAYES_00,LIKELY_SPAM_SUBJECT autolearn=no version=3.3.2 X-HELO: calimero.vinschen.de Received: from aquarius.hirmke.de (HELO calimero.vinschen.de) (217.91.18.234) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 01 Apr 2015 13:34:03 +0000 Received: by calimero.vinschen.de (Postfix, from userid 500) id 107ADA8096E; Wed, 1 Apr 2015 15:34:01 +0200 (CEST) Date: Wed, 01 Apr 2015 13:34:00 -0000 From: Corinna Vinschen To: cygwin@cygwin.com Subject: Re: With bad UTF-8, cygwin can create files it can't read Message-ID: <20150401133401.GV13285@calimero.vinschen.de> Reply-To: cygwin@cygwin.com Mail-Followup-To: cygwin@cygwin.com References: <20150330110446.GK29875@calimero.vinschen.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MRBOAUz+O/XNC2GI" Content-Disposition: inline In-Reply-To: <20150330110446.GK29875@calimero.vinschen.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-SW-Source: 2015-04/txt/msg00023.txt.bz2 --MRBOAUz+O/XNC2GI Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-length: 1748 Hi Stuart, On Mar 30 13:04, Corinna Vinschen wrote: > On Mar 25 14:34, Kyzer wrote: > > Hello, > >=20 > > I've found that if you use cygwin to create a file with badly-encoded > > UTF-8, readdir() gives out an entry with a name that cygwin won't > > subsequently accept. > >=20 > > * create a file using filename with hex bytes F4 8F BF BF > > * readdir() reports the filename as hex bytes E2 8E B3 ED BF BF > > * attempting to open or unlink the filename E2 8E B3 ED BF BF fails > > * attempting to open or unlink the filename F4 8F BF BF succeeds >=20 > Thanks for the testcase. I'll have a look later this week (I hope). Wow. Just wow. You found a long-standing bug in the wctomb conversion from UTF-16 to UTF-8. As you probably know, Unicode values beyond the base plane (that is, everything > 0xffff in UTF-32 and > ef bf bf in UTF-8 notation) are represented as so-called surrogate pairs in UTF-16, two UTF-16 values in the 0xd800 - 0xdfff range. While the conversion from UTF-8 f4 8f Bf Bf to UTF-16 dbff dfff worked fine, the conversion back to UTF-8 has a subtil bug. There's a test for a lone high surrogate pair in the underlying conversion function. This tests the next UTF-16 value like this: if (wchar < 0xdc00 || wchar >=3D 0xdfff) /* Handle lone high surrogate */ Notice the >=3D 0xdfff? That should have been > 0xdfff. Duh. This bug is only a bit over 5 years old... Fixed in the git repo. I'l regenerate the today's fool..., erm, the today's developer snapshot on https://cygwin.com/snapshots/ later today. Thanks, especially for the simple testcase, Corinna --=20 Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat --MRBOAUz+O/XNC2GI Content-Type: application/pgp-signature Content-length: 819 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJVG/PIAAoJEPU2Bp2uRE+giAwP/3GnsohzpFcdr9R81vt2PbgE p3RaBDuqLhA/TMGlG6immpqfthPrIFR1L6ncfatZgPI3yNpDp/4eutKxBTKCkfVD wKFyEH4CPOikCqLAfPUKcIsktfVxm4vdwBp6CUYXYoBbTgCQt9Tc6KgwqgJ9YsJG QaB/IdEas2LS/DURviZr9sDYqtvDcfpA2aaD12mFiKPWYGGgM2m3lgRUbwwsXrL1 w9DPhWwaudYW+xz0aIBwx1XdPKZn3cWFcEl5fzFMmrWJoRRkHz1GIiWsyMDiH1GF OVDxTOD4Cpxsw+5JRYluosW0a4fV5BJ+G4EuJGoDGkijBdOOmaJoA2RXvcuE+7SD nNYioDjE6qWvo7gWZ9SOoXusi5x4H9rS8Vvn3129pgCS1eNafGXqSeuXvduhZ6q2 SCs8Oz2WCNN/4vrqc+vLxVJxuC2pO5I8Kwz5mMyBmiUf3yW0BLoCCIkLNUoXpfTw 80BiS5osKSaSByPCI5ektDp0o7LrvR9PGVK995sVAl0lMVnhaf08/jo++ovY0+fA DZRqye0fb06jYX7whaPLJDNszcTjVziADT7Mqy7yBQcAJ+o6USJ/QR+jTobzBBUE EfeigO5XnGT8sTCYrNMJalwKzGzjPzSD7TBlros/r6Gdku+uVL5ZXPx3Wy1Cyo6f OwuL6SlzCo6lXC0etELU =hFLj -----END PGP SIGNATURE----- --MRBOAUz+O/XNC2GI--