--line-regexp option with null data

--line-regexp option with null data

[Steven Penny]

Consider this command:

    printf 'alpha\nbravo\ncharlie\n' | grep --line-regexp --quiet bravo

grep sees 3 lines separated by newline, and matches the bravo line. Now consider
this command:

    printf 'alpha\0bravo\0charlie\0' | grep --line-regexp --quiet bravo

My thinking tells me that because I have not used `--null-data`, grep should see
1 or even 0 lines separated by newline, and fail to match a `bravo` followed by
newline. However it does not, it succeeds just like the first command, why is
this?

Note I also tried this on Debian with Grep 2.2 and it works as expected.

http://stackoverflow.com/q/31467045

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: --line-regexp option with null data

[John Hein]

Steven Penny svnpenn-at-gmail.com |cygwin_ml_nodigest| wrote at 20:29 -0500 on Jul 16, 2015:
 > Consider this command:
 >
 >     printf 'alpha\nbravo\ncharlie\n' | grep --line-regexp --quiet bravo
 >
 > grep sees 3 lines separated by newline, and matches the bravo line. Now consider
 > this command:
 >
 >     printf 'alpha\0bravo\0charlie\0' | grep --line-regexp --quiet bravo
 >
 > My thinking tells me that because I have not used `--null-data`, grep should see
 > 1 or even 0 lines separated by newline, and fail to match a `bravo` followed by
 > newline. However it does not, it succeeds just like the first command, why is
 > this?
 >
 > Note I also tried this on Debian with Grep 2.2 and it works as expected.
 >
 > http://stackoverflow.com/q/31467045

cygwin grep is detecting the input as binary which seems to be
overriding the 'match the whole line' behavior of --line-regexp.  Get
rid of --quiet to see that.

That does seem like a bug in the cygwin implementation of grep to me.

As a workaround for this simple example, you can add -a (aka --text)
to force it to treat the input as text.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: --line-regexp option with null data

[Andrey Repin]

Greetings, John Hein!

> cygwin grep is detecting the input as binary which seems to be
> overriding the 'match the whole line' behavior of --line-regexp.  Get
> rid of --quiet to see that.

> That does seem like a bug in the cygwin implementation of grep to me.

Linux grep will do the same.
null byte = not a text.
Wrong encoding, not matching locale = not a text.

This is an upstream decision. It is arguable consistent… on Linux.
On Windows, of course, this is not the case.

> As a workaround for this simple example, you can add -a (aka --text)
> to force it to treat the input as text.


-- 
With best regards,
Andrey Repin
Friday, July 17, 2015 05:29:11

Sorry for my terrible english...

Re: --line-regexp option with null data

[Steven Penny]

On Thu, Jul 16, 2015 at 9:30 PM, Andrey Repin wrote:
> Linux grep will do the same.
> null byte = not a text.
> Wrong encoding, not matching locale = not a text.

I have repeatedly asked you to stay out of my threads. My experience is you
typically misread, misunderstand or misrepresent most or all of the threads you
comment on. I will ask again, please stop.

As I have said, I have already tested this today on Debian Linux 8, With
Grep 2.2 and it works as expected, and differently from the Cygwin version as I
have demonstrated.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: --line-regexp option with null data

[Vince Rice]

> On Jul 16, 2015, at 10:04 PM, Steven Penny <svnpenn@gmail.com> wrote:
> 
> On Thu, Jul 16, 2015 at 9:30 PM, Andrey Repin wrote:
>> Linux grep will do the same.
>> null byte = not a text.
>> Wrong encoding, not matching locale = not a text.
> 
> I have repeatedly asked you to stay out of my threads. My experience is you
> typically misread, misunderstand or misrepresent most or all of the threads you
> comment on. I will ask again, please stop.
> …

And you have been just as repeatedly told, by the people in charge of this list, that’s not your call, and to stop speaking in such a manner to other people on this list.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: --line-regexp option with null data

[Steven Penny]

On Thu, Jul 16, 2015 at 9:04 PM, John Hein wrote:
> cygwin grep is detecting the input as binary which seems to be
> overriding the 'match the whole line' behavior of --line-regexp.  Get
> rid of --quiet to see that.

It appears to be intended behavior starting with version 2.21:

> If a file contains data improperly encoded for the current locale,
> and this is discovered before any of the file's contents are output,
> grep now treats the file as binary.

http://savannah.gnu.org/forum/forum.php?forum_id=8152

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: --line-regexp option with null data

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

On 07/16/2015 08:04 PM, John Hein wrote:

>  >     printf 'alpha\0bravo\0charlie\0' | grep --line-regexp --quiet bravo
>  >
>  > My thinking tells me that because I have not used `--null-data`, grep should see
>  > 1 or even 0 lines separated by newline, and fail to match a `bravo` followed by
>  > newline. However it does not, it succeeds just like the first command, why is
>  > this?
>  >
>  > Note I also tried this on Debian with Grep 2.2 and it works as expected.
>  >
>  > http://stackoverflow.com/q/31467045
> 
> cygwin grep is detecting the input as binary which seems to be
> overriding the 'match the whole line' behavior of --line-regexp.  Get
> rid of --quiet to see that.

The behavior on Linux is the same.  See the NEWS for grep 2.21:

  When searching binary data, grep now may treat non-text bytes as
  line terminators.  This can boost performance significantly.

> 
> That does seem like a bug in the cygwin implementation of grep to me.

No, it is intentional upstream behavior.

> 
> As a workaround for this simple example, you can add -a (aka --text)
> to force it to treat the input as text.

Yes, that IS the correct solution.  You must TELL grep to not treat \0
as a line terminator.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]