From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 70979 invoked by alias); 24 May 2017 11:34:20 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 70951 invoked by uid 89); 24 May 2017 11:34:19 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=2.8 required=5.0 tests=AWL,BAYES_50,FREEMAIL_FROM,LIKELY_SPAM_SUBJECT,RCVD_IN_DNSWL_NONE,RCVD_IN_SORBS_SPAM,SPF_PASS autolearn=no version=3.3.2 spammy=german, German, ronald, Ronald X-HELO: mail-yw0-f181.google.com Received: from mail-yw0-f181.google.com (HELO mail-yw0-f181.google.com) (209.85.161.181) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 24 May 2017 11:34:18 +0000 Received: by mail-yw0-f181.google.com with SMTP id l14so87695236ywk.1 for ; Wed, 24 May 2017 04:34:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=diiXbv+TNDxOVPvQRyzXtQU7/OIwsM522sCxNTiAxtY=; b=O0J7LfL+Ym+bpizgB/YvrbCRCHFZGYo0DZgEDFg2nUqFhzB6I4/9sJdWypzuh4ZBiu PhFmSngZLod90cuZCtX5Rn5BwNP3qcpMCBAhYeiyyal0UQYaVlv5wA4JewKq2iHvv2qC EFsJ8IftJU8cWkFEgUumXt2SeOEBvCHts2q0tm35jr+1AoIhmZU8XC14pe8pc4Y7b5fJ i3CMyN/t/hU6kADVu3NITRJx438XfHQ/d7ot0pXDvsL2NxeyY8JIEB/8VuJkMIbdlbf2 oEo7Vcerj/SR18WW1vKgsl/1sMsNjVoQ9G6s5APCiyLWwH5eji6McHlDakpEovRqSxcE fWXg== X-Gm-Message-State: AODbwcAtQVisHxWDKcW4kNEdHDuNtorqJks1+JKQPeQDkVRwIuIKT+W3 lUgQtTjlE7V7zOOciRO+wetsHbzqsHyqkrc= X-Received: by 10.129.175.91 with SMTP id x27mr28642122ywj.192.1495625659881; Wed, 24 May 2017 04:34:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.164.34 with HTTP; Wed, 24 May 2017 04:34:19 -0700 (PDT) In-Reply-To: <1495612367.2760331.986814392.79C77EB2@webmail.messagingengine.com> References: <1495612367.2760331.986814392.79C77EB2@webmail.messagingengine.com> From: Erik Bray Date: Wed, 24 May 2017 11:42:00 -0000 Message-ID: Subject: Re: Bug: grep behaves incorrectly under the locale C.UTF-8, if a file contains Umlaut characters To: cygwin@cygwin.com Content-Type: text/plain; charset="UTF-8" X-IsSubscribed: yes X-SW-Source: 2017-05/txt/msg00385.txt.bz2 On Wed, May 24, 2017 at 9:52 AM, Ronald Fischer wrote: > I have a file X which contains ASCII text, but also in some lines German > umlaut characters. The file is classified as: > > $ file X > X: ISO-8859 text, with CRLF line terminators > > If I grep the file using, say, > > $ grep . X >Y > > (i.e. select every non-empty line and write the result to Y), this works > fine, if LANG is set to one of: UTF-8, C, C.de_DE, C.en_EN, en_EN, > de_DE. > > However, if LANG is set to C.UTF-8, two things happen: > > - grep classifies the file as binary file and produces the error message > "Binary file X matches" > > - Both the grepped lines (i.e. in our example the non-empty lines) AND > the error message end up in the standard output (i.e. in file Y). > > IMO, there are several problems with this: > > 1. It's hard to see, why an umlaut character makes the file X binary > under encoding C.UTF-8, but not under encoding UTF-8 or C.en_EN Only one of these, "UTF-8", specifies an *encoding*. Further, I don't think just "UTF-8" or "C.en_EN" are valid locale specifiers. You can read more about how Cygwin handles locales here: https://cygwin.com/cygwin-ug-net/setup-locale.html Certainly, if you set something like LANG="C.UTF-8" it will use UTF-8 to decode the text in the file and, failing that, treat it as binary. If you know it's ISO-8859-1 you can use either LANG= or LC_CTYPE="C.ISO-8859-1". Or, if you know the language, you should be able to use LC_CTYPE="de_DE". Each language+territory has a default encoding associated with it. For German I think it is one of the ISO-8859-* variants. Or you can use the variant "de_DE@euro" which forces ISO-8859-15 which includes the euro symbol, among other possibilities. So I think your main problem here is just not specifying your locale correctly. > 2. If grep classifies a file as binary, I think the desired behaviour > would be to NOT produce any output, unless the -a flag has been > supplied. > > 3. If grep writes a message "Binary file ... matches", this message > should go to stderr, not stdout. The stdout is supposed to contain only > a subset of the input lines. I would tend to agree with this, but this is normal behavior of grep (on Linux too), so I would take it up with the authors of grep. Best, Erik -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple