Broken since 1.3.10, or earlier

Broken since 1.3.10, or earlier

[Dan Higgins]

Greetings,

If I want to recursively find all files that contain some text, I use, for
example:

find . -name '*.java' | while read l; do grep 'Copyright' "$l" && echo "$l";
done

The output from this pipeline is inconsistent. Mostly I see bogus pathnames
that should not be there, or missing pathnames that should be there, or
duplicate lines from the grep, that sometimes even wind up overlapped by the
command prompt upon completion.

1. Don't suggest another syntax, unless it can handle paths with
   spaces in it.
2. I have already tried adding "2>&1" in various places
   throughout the line.
3. This works fine on Unix and MKS.
4. My system is Windows 2000 SP2, 1.3GHz Athlon.
5. I had Cygwin 1.3.10. I just upgraded to 1.3.12, including
   all freshly downloaded packages. No change.
6. I have already searched Google and the cygwin ml archives.
7. It has caused me much pain between the ears.

I have not been able to narrow it down to a particular program, but I
suspect it has to do with piping, stdin, stdout, and Windows' general lack
of those concepts. Whatever it is, it seems pretty fundamental to me.

Any info or advice could end the pain. That would be greatly appreciated.

Regards,
-Dan


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Randall R Schulz]

Dan,

I take it that by "inconsistent" you mean the relative ordering of the 
output of the "grep" processes and of the "echo" commands is not the strict 
alternation you'd expect.

That's what I see, anyway. I even saw two lines of grep output that follow 
the shell prompt printed after the command "completes."

It seems there's some asynchrony in the processing of the output and that 
somehow, in effect, there's a race condition.

I believe we've seen other reports of similar problems.

Someone who knows about the internal architecture of I/O processing in 
Cygwin might be able to shed some light on this. If, for example, there's 
some kind of queuing of I/O operations in Cygwin1.dll between the 
application code (grep or a shell, in this case) and the Windows I/O 
primitives, then there might be an opportunity for this kind of asynchrony.

It's too bad strace cannot solve this one, but since there's the shell plus 
one grep for each file to be examined, it's not going to help resolve 
relative timing issues.


Lest you think this is just idle speculation ... Well it is ... But here's 
a concrete suggestion on a work-around:

grep -R --include='*.java' Copyright .


If you need those other lines interleaved with the just file names, you can 
do this:

(grep    -R --include='*.java' Copyright tau rrs;
  grep -l -R --include='*.java' Copyright tau rrs) |sort

This may do everything twice, but on Cygwin, anyway, it's distinctly faster 
than your original, what with it's repeated invocation of grep.

I hope you find that helpful.


Randall Schulz
Mountain View, CA USA


At 16:31 2002-07-16, Dan Higgins wrote:
>Greetings,
>
>If I want to recursively find all files that contain some text, I use, for
>example:
>
>find . -name '*.java' | while read l; do grep 'Copyright' "$l" && echo "$l";
>done
>
>The output from this pipeline is inconsistent. Mostly I see bogus pathnames
>that should not be there, or missing pathnames that should be there, or
>duplicate lines from the grep, that sometimes even wind up overlapped by the
>command prompt upon completion.
>
>1. Don't suggest another syntax, unless it can handle paths with
>    spaces in it.
>2. I have already tried adding "2>&1" in various places
>    throughout the line.
>3. This works fine on Unix and MKS.
>4. My system is Windows 2000 SP2, 1.3GHz Athlon.
>5. I had Cygwin 1.3.10. I just upgraded to 1.3.12, including
>    all freshly downloaded packages. No change.
>6. I have already searched Google and the cygwin ml archives.
>7. It has caused me much pain between the ears.
>
>I have not been able to narrow it down to a particular program, but I
>suspect it has to do with piping, stdin, stdout, and Windows' general lack
>of those concepts. Whatever it is, it seems pretty fundamental to me.
>
>Any info or advice could end the pain. That would be greatly appreciated.
>
>Regards,
>-Dan


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Broken since 1.3.10, or earlier

[Dan Higgins]

Ah, it has been a few *years* since I checked the grep man page. Who
would've thought, someone has been improving it all that time...  :)

Thanks for the excellent response.

Cheers,
-Dan
Austin, TX, USA


> -----Original Message-----
> From: Randall R Schulz [mailto:rrschulz@cris.com]
> Sent: Tuesday, July 16, 2002 7:01 PM
> To: dan@danamis.com; cygwin@cygwin.com
> Subject: Re: Broken since 1.3.10, or earlier
>
>
> Dan,
>
> I take it that by "inconsistent" you mean the relative ordering of the
> output of the "grep" processes and of the "echo" commands is not
> the strict
> alternation you'd expect.
>
> That's what I see, anyway. I even saw two lines of grep output
> that follow
> the shell prompt printed after the command "completes."
>
> It seems there's some asynchrony in the processing of the output and that
> somehow, in effect, there's a race condition.
>
> I believe we've seen other reports of similar problems.
>
> Someone who knows about the internal architecture of I/O processing in
> Cygwin might be able to shed some light on this. If, for example, there's
> some kind of queuing of I/O operations in Cygwin1.dll between the
> application code (grep or a shell, in this case) and the Windows I/O
> primitives, then there might be an opportunity for this kind of
> asynchrony.
>
> It's too bad strace cannot solve this one, but since there's the
> shell plus
> one grep for each file to be examined, it's not going to help resolve
> relative timing issues.
>
>
> Lest you think this is just idle speculation ... Well it is ...
> But here's
> a concrete suggestion on a work-around:
>
> grep -R --include='*.java' Copyright .
>
>
> If you need those other lines interleaved with the just file
> names, you can
> do this:
>
> (grep    -R --include='*.java' Copyright tau rrs;
>   grep -l -R --include='*.java' Copyright tau rrs) |sort
>
> This may do everything twice, but on Cygwin, anyway, it's
> distinctly faster
> than your original, what with it's repeated invocation of grep.
>
> I hope you find that helpful.
>
>
> Randall Schulz
> Mountain View, CA USA
[...]


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Dario Alcocer]

On Tue, Jul 16, 2002 at 06:31:53PM -0500, Dan Higgins wrote:
> Greetings,
> 
> If I want to recursively find all files that contain some text, I use, for
> example:
> 
> find . -name '*.java' | while read l; do grep 'Copyright' "$l" && echo "$l";
> done
> 
> The output from this pipeline is inconsistent. Mostly I see bogus pathnames
> that should not be there, or missing pathnames that should be there, or
> duplicate lines from the grep, that sometimes even wind up overlapped by the
> command prompt upon completion.
> 
> 1. Don't suggest another syntax, unless it can handle paths with
>    spaces in it.

Well, the following will handle spaces just fine:

    $ find . -name '*.java' -print0 | xargs -0 grep -l 'Copyright'

This will actually be faster, since it will greatly minimize the
number of fork/exec pairs (your command line does a grep for *each*
file, whereas the find / xargs pipeline will do a *single* grep for
a bunch of files.) BTW, this isn't really a Cygwin-specific question,
rather a somewhat common Unix question.

-- 
Dario Alcocer -- Sr. Software Developer, Helix Digital Inc.
alcocer@helixdigital.com -- http://www.helixdigital.com

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Broken since 1.3.10, or earlier

[Dan Higgins]

> -----Original Message-----
> From: Randall R Schulz [mailto:rrschulz@cris.com]
> Sent: Tuesday, July 16, 2002 7:01 PM
> To: dan@danamis.com; cygwin@cygwin.com
> Subject: Re: Broken since 1.3.10, or earlier
>
>
> Dan,
>
> I take it that by "inconsistent" you mean the relative ordering of the
> output of the "grep" processes and of the "echo" commands is not
> the strict
> alternation you'd expect.
>
> That's what I see, anyway. I even saw two lines of grep output
> that follow
> the shell prompt printed after the command "completes."

For the record, if you run the problematic command multiple times, you'll
see that the number of lines returned is not always the same. This is what I
meant by inconsistent. Also, if I first redir the find to a file, then do
   while read l; do ... done < theFile
it seems to work fine.

> It seems there's some asynchrony in the processing of the output and that
> somehow, in effect, there's a race condition.
>
> I believe we've seen other reports of similar problems.
>
> Someone who knows about the internal architecture of I/O processing in
> Cygwin might be able to shed some light on this. If, for example, there's
> some kind of queuing of I/O operations in Cygwin1.dll between the
> application code (grep or a shell, in this case) and the Windows I/O
> primitives, then there might be an opportunity for this kind of
> asynchrony.
[...]

I thought I smelled something deep down under the hood on this one. Your
guess certainly sounds more scientific than mine would...

Regards,
-Dan


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Christopher Faylor]

On Tue, Jul 16, 2002 at 05:00:38PM -0700, Randall R Schulz wrote:
>Someone who knows about the internal architecture of I/O processing in 
>Cygwin might be able to shed some light on this. If, for example, there's 
>some kind of queuing of I/O operations in Cygwin1.dll between the 
>application code (grep or a shell, in this case) and the Windows I/O 
>primitives, then there might be an opportunity for this kind of asynchrony.

Actually, AFAICT, it is a bash bug and one that I have reported earlier.

bash makes assumptions that pids grow monotonically but that is not the
case on windows.  It's possible that you can run a program twice and get
the same pid twice in a row -- especially on Windows 9x.  I try to work
around this in cygwin by keeping a certain number of process handles
open, so that the pids won't be reused, but that still causes problems
when you are fork/execing processes quickly.

What happens is that sometimes bash ends up not waiting for a process when
it should, starting the next process too early.

For instance, look at the following from an strace that shows the problem:

  881 11112361 [main] bash 1304 fork: *1288* = fork()
  979 11854415 [main] bash 1304 fork: 1120 = fork()
  895 12550685 [main] bash 1304 fork: 1648 = fork()
  947 13188885 [main] bash 1304 fork: 340 = fork()
 1103 13843029 [main] bash 1304 fork: 1484 = fork()
  950 14440217 [main] bash 1304 fork: 1560 = fork()
  880 15097448 [main] bash 1304 fork: 1144 = fork()
  931 15761062 [main] bash 1304 fork: 1620 = fork()
 1191 16484221 [main] bash 1304 fork: 864 = fork()
  902 17124502 [main] bash 1304 fork: 1548 = fork()
  825 17815402 [main] bash 1304 fork: 1280 = fork()
  968 18421987 [main] bash 1304 fork: *1288* = fork()

Guess where the output goes funny?  Right after the second fork which
results in a pid of 1288.

I "fixed" this problem in my version of cygwin by upping the number of
process handles that fork keeps open from 8 to 128.  This reduces, but
doesn't eliminate, the likelihood that bash will be confused by
repeating pids.  When I did this, I get the expected output.

This isn't really a solution, however.  It wastes resources by keeping
*a lot* of handles around so I'm not comfortable with upping the number
like this.

If you want to use this method, then run it under /bin/sh or /bin/zsh.
Neither of these programs seem to have this bug.

Oh, and as far as the "or earlier" part of cygwin goes, I tried this
back to 1.1.8.  It behaves the same way.  Of course, this wouldn't have
been a problem with B20.1 since that version of cygwin invented its own
pids rather than using windows pids.  Back then people were asking for
cygwin to use actual windows pids, of course.

>It's too bad strace cannot solve this one,

Actually strace did solve this one.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Jon Cast]

Christopher Faylor <cgf@redhat.com> wrote:
<snip>
> bash makes assumptions that pids grow monotonically but that is not
> the case on windows.  It's possible that you can run a program twice
> and get the same pid twice in a row -- especially on Windows 9x.  I
> try to work around this in cygwin by keeping a certain number of
> process handles open, so that the pids won't be reused, but that
> still causes problems when you are fork/execing processes quickly.

Just out of curiosity, why would bash care if pids grow monotonically?
(I know I can check the sources, but I'm lazy.)

Jon Cast


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Christopher Faylor]

On Tue, Jul 16, 2002 at 09:45:16PM -0500, Jon Cast wrote:
>Christopher Faylor <cgf@redhat.com> wrote:
><snip>
>>bash makes assumptions that pids grow monotonically but that is not the
>>case on windows.  It's possible that you can run a program twice and
>>get the same pid twice in a row -- especially on Windows 9x.  I try to
>>work around this in cygwin by keeping a certain number of process
>>handles open, so that the pids won't be reused, but that still causes
>>problems when you are fork/execing processes quickly.
>
>Just out of curiosity, why would bash care if pids grow monotonically?
>(I know I can check the sources, but I'm lazy.)

I assume that the bash developer is just mean.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Jon Cast]

Christopher Faylor <cgf@redhat.com> wrote:
<snip>
> I assume that the bash developer is just mean.

> cgf

Alright, I'll get the sources :)

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Andrew DeFaria]

Dan Higgins wrote:

> Greetings,
>
> If I want to recursively find all files that contain some text, I use, 
> for example:
>
> find . -name '*.java' | while read l; do grep 'Copyright' "$l" && echo 
> "$l";
> done

Just as an alternative, why not:

$ find . -name '*.java' -exec grep 'Copyright' {}\;




--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[egor duda]

Hi!

Wednesday, 17 July, 2002 Christopher Faylor cgf@redhat.com wrote:

CF> On Tue, Jul 16, 2002 at 05:00:38PM -0700, Randall R Schulz wrote:
>>Someone who knows about the internal architecture of I/O processing in 
>>Cygwin might be able to shed some light on this. If, for example, there's 
>>some kind of queuing of I/O operations in Cygwin1.dll between the 
>>application code (grep or a shell, in this case) and the Windows I/O 
>>primitives, then there might be an opportunity for this kind of asynchrony.

Seen this too.

CF> Actually, AFAICT, it is a bash bug and one that I have reported earlier.

CF> bash makes assumptions that pids grow monotonically but that is not the
CF> case on windows.

But that can't be the case on any unix too! Sooner or later system
reach maximum possible value of pid and will have to turn it over. Can
you point where bash makes such assumptions, and probably this can be
fixed in bash?

Egor.            mailto:deo@logos-m.ru ICQ 5165414 FidoNet 2:5020/496.19


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Broken since 1.3.10, or earlier

[Christopher Faylor]

On Wed, Jul 17, 2002 at 02:35:04PM +0400, egor duda wrote:
>But that can't be the case on any unix too! Sooner or later system
>reach maximum possible value of pid and will have to turn it over.

It is very unlikely to happen on unix within the timeframe where it
will bother bash.

>Can you point where bash makes such assumptions, and probably this can
>be fixed in bash?

I have traced this through bash in the past enough to convince myself
that it isn't cygwin's fault.  However I don't remember details.  If it
was a simple fix I would have offered a patch.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Broken since 1.3.10, or earlier

[Dan Higgins]

> -----Original Message-----
> From: Andrew DeFaria [mailto:Andrew@DeFaria.com]
> Sent: Tuesday, July 16, 2002 10:12 PM
> To: cygwin@cygwin.com
> Subject: Re: Broken since 1.3.10, or earlier
>
> Dan Higgins wrote:
> > Greetings,
> >
> > If I want to recursively find all files that contain some text, I use,
> > for example:
> >
> > find . -name '*.java' | while read l; do grep 'Copyright' "$l" && echo
> > "$l";
> > done
>
> Just as an alternative, why not:
>
> $ find . -name '*.java' -exec grep 'Copyright' {}\;

I know that one, but it doesn't give me the file path. My idea behind the
syntax is that I have a bit more flexibility to do some other stuff, since I
know both the line and the path to the file. Of course, if I didn't need to
worry about spaces in pathnames, I'd just use a for loop.

But forget about the syntax for a moment. The real issue here is what *bash*
is doing.

Yep, thanks to Mr. Faylor, I'm convinced that it's a bug in bash. If I run
my syntax through Cygwin's sh it does work perfectly (as it does on any
other Unix bash and MKS sh).

In my experience, issues like this one don't usually mean that there is a
flawed design, but that there may be some "optimization" that requires some
assumptions that aren't always correct. I mention this because it could be
an easy fix to let us "turn off" the optimization.

But, I haven't looked at the source for bash at all, so I could be way off.

Regards,
-Dan


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Broken since 1.3.10, or earlier

[Bernard A Badger]

Another off-topic find/grep idea:

$ find . -name '*.java' -print0 | xargs -0 grep 'Copyright' /dev/null

The find -print0 goes with the xargs -0 option to use NUL-terminated strings.
The /dev/null is included in grep to force the name to print.
(Grep prints the name if there's more than one file input.)

A sample of the output might look like this:
glut-3.7.3/progs/advanced/shadowfun.c:/* Copyright (c) Mark J. Kilgard, 1997.
*/
glut-3.7.3/progs/advanced/texwinalign.c:/* Copyright (c) Mark J. Kilgard, 1998.
*/
glut-3.7.3/progs/advanced/videoresize.c:/* Copyright (c) Mark J. Kilgard, 1996.
*/

Is that any help?
>
> I know that one, but it doesn't give me the file path. My idea behind the
> syntax is that I have a bit more flexibility to do some other stuff, since I
> know both the line and the path to the file. Of course, if I didn't need to
> worry about spaces in pathnames, I'd just use a for loop.
>
> But forget about the syntax for a moment. The real issue here is what *bash*
> is doing.
Well, I can't help there.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/