From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.17.13]) by sourceware.org (Postfix) with ESMTPS id C5628385B834 for ; Tue, 24 Mar 2020 09:07:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org C5628385B834 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=towo.net Authentication-Results: sourceware.org; spf=none smtp.mailfrom=towo@towo.net Received: from [192.168.178.45] ([95.90.246.248]) by mrelayeu.kundenserver.de (mreue109 [212.227.15.183]) with ESMTPSA (Nemesis) id 1N8GIg-1jKugU2LJI-0148eM for ; Tue, 24 Mar 2020 10:07:22 +0100 Subject: Re: bug report: shell expansion in argv[] processing sensitive to LANG, e.g. "ls: cannot access '*.pdf': No such file or directory", but works okay in bash To: cygwin@cygwin.com References: From: Thomas Wolff X-Tagtoolbar-Keys: D20200324100722391 Message-ID: Date: Tue, 24 Mar 2020 10:07:22 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K1:K1KvxFU6dgC9y26UtvVF+5I3gY3d2v7ek3s3G07kYFqop4AXe8O L6anyQzikdzpOUF80joYcIa6KVpNoxDmMy8BSu7ZgGKjAZ2lilDdt8ej5YiU9J4d01SAGhg wMD0LlEJsl5k/+RapEagSRDyU5g03ewhQd4Oisx5o6CGz/uLmx2e8mf5Fczq9WRrkwQrgsy pWhSFpoM8lEFppKYBI2Uw== X-UI-Out-Filterresults: notjunk:1;V03:K0:ktZiQCcHjFI=:NvJmTHfheQT6TfTk7S+QSa IQM451ZGhRDR0MLyVg4wASM8Mq7pGCbRswB4RXOxGYyjOAA1fMp+C1+pzC5GLhwRv6BjKSdG7 NYW7tdUvnyx4mYxGEHaTHixNstjibkc+sjOCIGTTdk7jfwxdZeWBAF7tCwwzO5qpH01Di/g7K DGcAzSw0AJIQYGIyECLSl1SucADFhLaRALq1XVVI2Z7ppKNADmPJtm91iUse1Q6s8zEuJF7un /858tdJbXj/wx4w9IW0861zLBA36io4oOnLcaC+cJSmb59VxXaab/N+yyWIx2CE6jmqZkmRZH bMjdBY98Oofo0/P8fn/zcLUJCLlgsIgtQBoF/9MZMWlT4+AtXCLZv3gzvJcp1HOn+IQZEgYJO GKRKyvvvskKERToT1v3tmRUWnFKEgWr9QYx1AQEP6T6W0nPyLKrmuTlOv+TMWNl+1/w4Z7D1y IUrOtrsuC5rr3PNwzPRUk+ZmO+GC1jJYDcU6cxP03Eth8QvgK1SsWKmpCrru7As54hbyC1ccW mSfwFgwMZpy8DCIsILnTCRHuEuOSTchqt1g/WIM+uzIiIhM57V1726J2YQC26wMrDBTfnsltL TUfcixzJHrCCzuW/SCojT/ZkfnaqoJe2n2FF9tP5oVxoGg/BVrY3mjzCQf4frdUgVEJddaZl3 AqBT/DzjVP17vDiJdZCpzjFiLu5ATe2KJR3zKKqxeqxM949XowXAvZe9JlpaCO07mIKpgmH1r 33MFM2p+GGn3NYQ+aMEsaaTJQeFmwm+vUXn2K9dlTBmKWLMnQ1aJrEEeG+F7U3sovuq1XRELx +mDUwAsv1c0RJy7AuNARbvKoOVCqbKJknn7ksvNGbgkYVawLSU= X-Spam-Status: No, score=1.4 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 09:07:25 -0000 Am 24.03.2020 um 08:18 schrieb Jay Libove via Cygwin: > Hi Cygwin team, > Here is a consolidated bug report based on the discussion in recent days which I'd started under the subject " shell expansion produces e.g. "ls: cannot access '*.pdf': No such file or directory" in Windows CMD shell, but works okay in bash " (thread starter https://cygwin.com/pipermail/cygwin/2020-March/244161.html ) > Many thanks to Paul, Andrey, and others for helping me nail down where and how it seems to be happening. > My apologies in advance that my coding days are long behind me, so I'm not in a position to include a proposed code fix. > > cygcheck output attached (lightly modified to redact a couple of personal items). > > Problem: > Under certain circumstances (see Steps to Reproduce, below) Cygwin programs' built-in argv[] globbing will produce unexpected: > "{programName}: cannot access '{glob pattern}: No such file or directory" > e.g. > "ls: cannot access '*.pdf': No such file or directory" > .. despite the fact that e.g. *.pdf definitely exists. > > Steps to Reproduce: > * Have some files in the local director with accented characters in the names, e.g.: > C:> mkdir c:\temp\test > C:> cd c:\temp\test > C:> touch héllo.pdf > C:> touch gòodbye.pdf > C:> touch normal.pdf > * DON'T have the LANG= environment variable set to anything > * NOT in bash or Cygwin Terminal, but rather within Windows CMD.exe, execute a Cygwin command which needs to do file name globbing because the Windows CMD.exe shells does not do so for it, e.g. > C:> ls *.pdf > C:> cat *.pdf > These will produce "ls: cannot access '*.pdf': No such file or directory" > Although, curiously, > C:> ls *or* > does correctly produce: > normal.pdf > > Also, display output of the áccènted characters is incomplete: > C:> ls > 'g'$'\303\262''odbye.pdf' 'h'$'\303\251''llo.pdf' normal.pdf > C:> bash > jay_l@DESKTOP-I9MRIE3 /cygdrive/c/Temp > $ ls > 'g'$'\303\262''odbye.pdf' 'h'$'\303\251''llo.pdf' normal.pdf > > > Analysis: > I've verified that it's not about case sensitivity. That is, it's not a matter of ls *.pdf vs. ls *.PDF. > If these test commands are run either under bash.exe or within a Cygwin Terminal window, the problem does not occur. > I've verified that the Windows system locale (per Windows' Region setting) actually doesn't matter. (I've reproduced this both on systems in Region Spain with language English-International and English-Ireland, and in a VM with a bog standard vanilla US English Windows). > > Credits to Paul for suggesting deleting files one by one until the problem goes away, and to Andrey for pointing out `locale` and the LANG= setting. > > Set LANG=en_US.UTF-8, e.g. > C:> set LANG=en_US.UTF-8 > .. and the problem goes away. > C:> ls *.pdf > gòodbye.pdf > héllo.pdf > normal.pdf > C:> ls > gòodbye.pdf > héllo.pdf > normal.pdf > > Interestingly, Andrey mentioned that he sets LANG=ru_RU.CP866 and he doesn't see the problem. When I tried that exact setting, I still had the problem. > So it's maybe not just that LANG must be set to *something*, but that somehow LANG must be set to something that matches something in Windows? (Sorry, I know that's nearly uselessly vague). > > > In summary, it appears that the way that the argv[] globbing code which gets compiled in to Cygwin programs functions a bit differently than the way the shell globbing code works within bash.exe. > And this produces unexpected globbing failures. (As commented in the other thread already:) Maybe it can simply be fixed by changing the order of setting up locale stuff and applying the expansion in cygwin? (I would look into the code if I had a clue where to find the respective things.) Thomas > Thanks to all the Cygwin maintainers for this amazing software, for so many years! > -Jay