echo vs. ls race condition?

echo vs. ls race condition?

[Henry S. Thompson]

Apologies if this is a known bug, or actually a bash problem, but I
can only reproduce it under cygwin (not linux or solaris, same bash
version (2.05b0) on linux, different (2.02.0) on solaris).

Hard to know what to search for, but nothing obvious in archives.

Short summary:  a loop with echo and ls does not produce well-ordered
output

Recipe to reproduce:

 > mkdir test
 > cd test
 > x=1
 > while [ $x -lt 200 ]; do mkdir $x; x=`expr $x + 1`; done
 > for d in *; do touch $d/aaa; touch $d/bbb; done
 > ls -d * | while read d; do echo $d; ls $d ; done > raceLog

Looking at raceLog will show e.g.

105
aaa
bbb
106
107
108
aaa
bbb
aaa
bbb
aaa
bbb

which is not what it should be. . .

Running up-to-date installations on XP and ME (cygcheck for the latter
[removed -- mail not getting through?]).

ht
-- 
 Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh
                     Half-time member of W3C Team
    2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440
            Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                   URL: http://www.ltg.ed.ac.uk/~ht/
[mail really from me _always_ has this .sig -- mail without it is forged spam]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: echo vs. ls race condition?

[Fred Kulack]

> Short summary:  a loop with echo and ls does not produce well-ordered
> output

For what it's worth, it works fine on my system. 
        XP Professional
        bash                    2.05b-16 

I always try an update and see if its still happens when I find something
not quite right.


"The stuff we call "software" is not like anything that human society 
  is used to thinking about. Software is something like a machine, and 
  something like mathematics, and something like language, and 
  something like thought, and art, and information... 
  but software is not in fact any of those other things."
Bruce Sterling - The Hacker Crackdown

Fred A. Kulack - IBM eServer iSeries - Enterprise Application Solutions
ERP, Java DB2 access, Jdbc, JTA, etc...
IBM in Rochester, MN  (Phone: 507.253.5982   T/L 553-5982)
mailto:kulack/us.ibm.com   Personal: mailto:kulack/magnaspeed.net
AIM Home:FKulack  AIM Work:FKulackWrk 
MSN Work: fakulack/hotmail.com (replace email / with @)

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: echo vs. ls race condition?

[Henry S. Thompson]

Fred Kulack <kulack@us.ibm.com> writes:

>> Short summary:  a loop with echo and ls does not produce well-ordered
>> output
>
> For what it's worth, it works fine on my system. 
>         XP Professional
>         bash                    2.05b-16 
>
> I always try an update and see if its still happens when I find something
> not quite right.

Thanks for (failing to) reproduce, but I did a complete update before
reporting, and indeed that _is_ the bash I'm using -- sorry that in my
first message, I reported what _bash_ itself thinks, as opposed to
what cygwin thinks.

Can't figure out why I can't get my mail with cygcheck output attached
through to the mailing list.

A few more volunteers before I assume I'm just cursed . . .?

Thanks

ht
-- 
 Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh
                     Half-time member of W3C Team
    2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440
            Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                   URL: http://www.ltg.ed.ac.uk/~ht/
[mail really from me _always_ has this .sig -- mail without it is forged spam]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: echo vs. ls race condition?

[Christopher Faylor]

On Mon, May 24, 2004 at 05:31:35PM +0100, Henry S. Thompson wrote:
>Can't figure out why I can't get my mail with cygcheck output attached
>through to the mailing list.

If I had to guess why you couldn't get any cygcheck output through the list,
it would be...

Hmm.  Come to think of it, why would we have to bother guessing?  Wouldn't it
make sense to report what your specific problem is?  Are you getting a bounce?
Is it just disappearing?  Etc.?

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: echo vs. ls race condition?

[Hannu E K Nevalainen]

> From: Henry S. Thompson
> Sent: Monday, May 24, 2004 6:32 PM

> Fred Kulack <kulack@we.bigblue.xxx> writes:
>
> >> Short summary:  a loop with echo and ls does not produce well-ordered
> >> output
> >
> > For what it's worth, it works fine on my system.
> >         XP Professional
> >         bash                    2.05b-16
> >
> > I always try an update and see if its still happens when I find
> something
> > not quite right.
>
> Thanks for (failing to) reproduce, but I did a complete update before
> reporting, and indeed that _is_ the bash I'm using -- sorry that in my
> first message, I reported what _bash_ itself thinks, as opposed to
> what cygwin thinks.

FWIW; I can reproduce it. It seems to me that it is some kind of problem
with bash's use of subshells in loops (synchronizing output). I have no
knowledge of the internals here, so bear with me.

I've changed your script to be little less (runs shorter time), show better
what happens... and also adjusted for the fact that I have a bash function
for "ls" that got in the way. (cause: --classify)

$ md zz && cd zz
$ cat ../zz.sh
x=10
while [ $x -lt 20 ]; do echo -en "\r$x  ";mkdir $x; x=`expr $x + 1`; done
for d in *; do echo -en "\r$d  ";touch $d/aaa-$d; touch $d/bbb-$d; done
command ls -d * | while read d; do echo -e "\n-$d-"; ls $d ; done

NOTE: For me, often the FIRST run after a 'rm -rf *' the script succeeds -
then if you do the following, it fails;

$ command ls -d * | while read d; do echo -e "\n-$d-"; ls $d ; done

-10-
aaa-10  bbb-10

-11-
aaa-11  bbb-11

-12-
aaa-12  bbb-12

-13-
aaa-13  bbb-13

-14-
aaa-14  bbb-14

-15-
aaa-15  bbb-15

-16-
aaa-16  bbb-16

-17-

-18-
aaa-17  bbb-17
aaa-18  bbb-18

-19-
aaa-19  bbb-19


As the following succeeds, I'm lead to believe what I said earlier - not
that it is equivalent though:

$ command ls -d * | while read d; do echo -en "\n-$d-\n$(ls $d)" ; done

-10-
aaa-10
bbb-10

-11-
aaa-11
bbb-11

-12-
aaa-12
bbb-12

-13-
aaa-13
bbb-13

-14-
aaa-14
bbb-14

-15-
aaa-15
bbb-15

-16-
aaa-16
bbb-16

-17-
aaa-17
bbb-17

-18-
aaa-18
bbb-18

-19-
aaa-19
bbb-19


All this seems somehow be triggered by the fact that diskcaching plays a
role here: i.e. the first run has this "time" output;
	real    0m5.385s
	user    0m2.040s
	sys     0m3.670s
the second run and forth;
	real    0m1.233s
	user    0m0.420s
	sys     0m0.850s
(w some tiny differences).

 I have tried to use { _list_of_commands_ } as described in the bash manual;
i.e, this runs, but has the same problem:

$ command ls -d * | while read d; do { echo -e "\n-$d-"; ls $d ; } ; done

The idea is to force 'do _list_ ;done' to be executed in the "local" shell,
not a subshell. I've tried other variants of it, but have failed to make it
work;
 Here I wonder: Is that yet another of those irritating
documentation-failure(-to-understand) issues that I stumble on quite often?


$ uname -a
CYGWIN_NT-5.0 P450 1.5.10s(0.115/4/2) 20040519 13:51:37 i686 unknown unknown
Cygwin

$ cygcheck -cd bash
Cygwin Package Information
Package              Version
bash                 2.05b-16

$ echo $CYGWIN
notitle glob check_case:strict server=1

Tests run on a FAT32 partition on a P2/450Mhz, W2K adv server, SP4++ - if
that has any influence.


/Hannu E K Nevalainen, B.Sc. EE - 59+16.37'N, 17+12.60'E

** on a mailing list; please keep replies on that particular list **

-- printf("LocalTime: UTC+%02d\n",(DST)? 2:1); --
--END OF MESSAGE--


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: echo vs. ls race condition?

[Henry S. Thompson]

Christopher Faylor <cgf-no-personal-reply-please@cygwin.com> writes:

> On Mon, May 24, 2004 at 05:31:35PM +0100, Henry S. Thompson wrote:
>>Can't figure out why I can't get my mail with cygcheck output attached
>>through to the mailing list.
>
> If I had to guess why you couldn't get any cygcheck output through the list,
> it would be...
>
> Hmm.  Come to think of it, why would we have to bother guessing?
> Wouldn't it make sense to report what your specific problem is?  Are
> you getting a bounce?  Is it just disappearing?  Etc.?

Sorry for the waste of your bandwidth, I was focussed on the primary
problem, not the mailing list problem.  Two attempts at sending mail
with cygcheck output attached (by gnus/xemacs) as a text/plain
multipart attachment simply disappeared, one sent on Friday 21st May
and the next on Saturday 22nd May.

The attachment began as follows:

> --=-=-=
> Content-Disposition: attachment; filename=csc
> Content-Description: cygcheck -s

Is the problem possibly because it wasn't encoded?

ht
-- 
 Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh
                     Half-time member of W3C Team
    2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440
            Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                   URL: http://www.ltg.ed.ac.uk/~ht/
[mail really from me _always_ has this .sig -- mail without it is forged spam]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: echo vs. ls race condition?

[Henry S. Thompson]

"Hannu E K Nevalainen" <garbage_collector@telia.com> writes:

>> From: Henry S. Thompson
>> Sent: Monday, May 24, 2004 6:32 PM
>
>> Fred Kulack <kulack@we.bigblue.xxx> writes:
>>
>> >> Short summary:  a loop with echo and ls does not produce well-ordered
>> >> output
>> >

<snip/>

> FWIW; I can reproduce it. It seems to me that it is some kind of problem
> with bash's use of subshells in loops (synchronizing output). I have no
> knowledge of the internals here, so bear with me.

That sounds plausible, but I too am incompetent to delve much
deeper. . .  There's the old issue of process id re-use, perhaps?

Thanks for trying it, and simplifying.

ht
-- 
 Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh
                     Half-time member of W3C Team
    2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440
            Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                   URL: http://www.ltg.ed.ac.uk/~ht/
[mail really from me _always_ has this .sig -- mail without it is forged spam]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: echo vs. ls race condition?

[Fred Kulack]

Interesting tests. I still can't get it to fail...

What terminal are you using? Perhaps there's something weird going on with 

flow control as descriptors are dup'd for the utilities there? 
I'm using plain old winblows console.
Did you try the regular echo instead of the builtin? It may have different 
characteristics
with regard to this.

> uname -a
CYGWIN_NT-5.1 kulack 1.5.9(0.112/4/2) 2004-03-18 23:05 i686 unknown 
unknown Cygwin

Bash /kulack/test/test
> cygcheck -cd bash
Cygwin Package Information
Package              Version
bash                 2.05b-16

Bash /kulack/test/test
> echo $CYGWIN


Also, the zz.sh script wouldn't run correctly for me unless I made it used 
the magic number #!/bin/bash,
it was running sh instead of bash and didn't support some of the 
constructs used. 
Some weirdness with sh versus bash mode... I never had to pay much 
attention
to the relationships there, but perhaps that's related to the cause?

Bash /kulack/test/test
> echo $SHELL
/bin/bash



"The stuff we call "software" is not like anything that human society 
  is used to thinking about. Software is something like a machine, and 
  something like mathematics, and something like language, and 
  something like thought, and art, and information... 
  but software is not in fact any of those other things."
Bruce Sterling - The Hacker Crackdown

Fred A. Kulack - IBM eServer iSeries - Enterprise Application Solutions
ERP, Java DB2 access, Jdbc, JTA, etc...
IBM in Rochester, MN  (Phone: 507.253.5982   T/L 553-5982)
mailto:kulack/us.ibm.com   Personal: mailto:kulack/magnaspeed.net
AIM Home:FKulack  AIM Work:FKulackWrk 
MSN Work: fakulack/hotmail.com (replace email / with @)



"Hannu E K Nevalainen" <garbage_collector@telia.com> 
Sent by: cygwin-owner@cygwin.com
05/24/2004 03:27 PM

To
<cygwin@cygwin.com>
cc

Subject
RE: echo vs. ls race condition?






> From: Henry S. Thompson
> Sent: Monday, May 24, 2004 6:32 PM

> Fred Kulack <kulack@we.bigblue.xxx> writes:
>
> >> Short summary:  a loop with echo and ls does not produce well-ordered
> >> output
> >
> > For what it's worth, it works fine on my system.
> >         XP Professional
> >         bash                    2.05b-16
> >
> > I always try an update and see if its still happens when I find
> something
> > not quite right.
>
> Thanks for (failing to) reproduce, but I did a complete update before
> reporting, and indeed that _is_ the bash I'm using -- sorry that in my
> first message, I reported what _bash_ itself thinks, as opposed to
> what cygwin thinks.

FWIW; I can reproduce it. It seems to me that it is some kind of problem
with bash's use of subshells in loops (synchronizing output). I have no
knowledge of the internals here, so bear with me.

I've changed your script to be little less (runs shorter time), show 
better
what happens... and also adjusted for the fact that I have a bash function
for "ls" that got in the way. (cause: --classify)

$ md zz && cd zz
$ cat ../zz.sh
x=10
while [ $x -lt 20 ]; do echo -en "\r$x  ";mkdir $x; x=`expr $x + 1`; done
for d in *; do echo -en "\r$d  ";touch $d/aaa-$d; touch $d/bbb-$d; done
command ls -d * | while read d; do echo -e "\n-$d-"; ls $d ; done

NOTE: For me, often the FIRST run after a 'rm -rf *' the script succeeds -
then if you do the following, it fails;

$ command ls -d * | while read d; do echo -e "\n-$d-"; ls $d ; done

-10-
aaa-10  bbb-10

-11-
aaa-11  bbb-11

-12-
aaa-12  bbb-12

-13-
aaa-13  bbb-13

-14-
aaa-14  bbb-14

-15-
aaa-15  bbb-15

-16-
aaa-16  bbb-16

-17-

-18-
aaa-17  bbb-17
aaa-18  bbb-18

-19-
aaa-19  bbb-19


As the following succeeds, I'm lead to believe what I said earlier - not
that it is equivalent though:

$ command ls -d * | while read d; do echo -en "\n-$d-\n$(ls $d)" ; done

-10-
aaa-10
bbb-10

-11-
aaa-11
bbb-11

-12-
aaa-12
bbb-12

-13-
aaa-13
bbb-13

-14-
aaa-14
bbb-14

-15-
aaa-15
bbb-15

-16-
aaa-16
bbb-16

-17-
aaa-17
bbb-17

-18-
aaa-18
bbb-18

-19-
aaa-19
bbb-19


All this seems somehow be triggered by the fact that diskcaching plays a
role here: i.e. the first run has this "time" output;
                 real    0m5.385s
                 user    0m2.040s
                 sys     0m3.670s
the second run and forth;
                 real    0m1.233s
                 user    0m0.420s
                 sys     0m0.850s
(w some tiny differences).

 I have tried to use { _list_of_commands_ } as described in the bash 
manual;
i.e, this runs, but has the same problem:

$ command ls -d * | while read d; do { echo -e "\n-$d-"; ls $d ; } ; done

The idea is to force 'do _list_ ;done' to be executed in the "local" 
shell,
not a subshell. I've tried other variants of it, but have failed to make 
it
work;
 Here I wonder: Is that yet another of those irritating
documentation-failure(-to-understand) issues that I stumble on quite 
often?


$ uname -a
CYGWIN_NT-5.0 P450 1.5.10s(0.115/4/2) 20040519 13:51:37 i686 unknown 
unknown
Cygwin

$ cygcheck -cd bash
Cygwin Package Information
Package              Version
bash                 2.05b-16

$ echo $CYGWIN
notitle glob check_case:strict server=1

Tests run on a FAT32 partition on a P2/450Mhz, W2K adv server, SP4++ - if
that has any influence.


/Hannu E K Nevalainen, B.Sc. EE - 59+16.37'N, 17+12.60'E

** on a mailing list; please keep replies on that particular list **

-- printf("LocalTime: UTC+%02d\n",(DST)? 2:1); --
--END OF MESSAGE--


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/




--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: echo vs. ls race condition?

[Hannu E K Nevalainen]

> From: Fred Kulack
> Sent: Tuesday, May 25, 2004 4:17 PM

> Interesting tests. I still can't get it to fail...
>
> What terminal are you using? Perhaps there's something weird
> going on with

It is as always started with:
	set CYGWIN=notitle glob check_case:strict server=1
	bash -c 'rxvt -ls -e /usr/bin/bash --login -i &'
from within cygwin.bat

$ cat .Xdefaults
# XTerm == rxvt as rxvt simulates xterm

rxvt.background:        black
rxvt.foreground:        grey
rxvt.visualBell:        true

rxvt.keysym.7e37:       ^A
rxvt.keysym.7e38:       ^E

rxvt.cutchars: "/`"'&()*,;<=>?@[]{|}"
rxvt.meta8:             true
rxvt.backspacekey: ^H

# Don't juggle display when 'doing' output
rxvt.scrollTtyOutput:   false
rxvt.scrollWithBuffer: false
rxvt.scrollTtyKeypress: true

# Local preferences
rxvt.saveLines: 6000
rxvt.scrollBar_right:   true
rxvt.scrollColor:       #c5c0a5


# Allow four columns of ls output
# ...nicely positioned on 1024x768 pixels
rxvt.geometry:          112x56+20+20
rxvt.font: Lucida Console-12

# Peculiar char spacing
# rxvt.font: Verdana-13


> flow control as descriptors are dup'd for the utilities there?
> I'm using plain old winblows console.

 As you can see above, I'm using rxvt - I've found it superior in many ways.
NOTE: You can run it _without_ X, that is what I do.

> Did you try the regular echo instead of the builtin? It may have
> different characteristics with regard to this.

Hmm... not as it seems from a quick test (read below).

> Also, the zz.sh script wouldn't run correctly for me unless I
> made it used
> the magic number #!/bin/bash,
> it was running sh instead of bash and didn't support some of the
> constructs used.
> Some weirdness with sh versus bash mode... I never had to pay much
> attention
> to the relationships there, but perhaps that's related to the cause?

Nah..? I didn't show it (sorry), but I ran the zz.sh script as in:
	$ . ../zz.sh
from the bash prompt.

/Hannu E K Nevalainen, B.Sc. EE - 59+16.37'N, 17+12.60'E


- test log -

$ md zz && cd zz
$ ls -l
total 0

$ cat ../zz.sh
x=10
while [ $x -lt 20 ]; do echo -en "\r$x  ";mkdir $x; x=`expr $x + 1`; done
for d in *; do echo -en "\r$d  ";touch $d/aaa-$d; touch $d/bbb-$d; done
command ls -d * | while read d; do /bin/echo -e "\n-$d-"; ls $d ; done

$ . ../zz.sh
19
-10-
aaa-10  bbb-10

-11-
aaa-11  bbb-11

-12-
aaa-12  bbb-12

-13-
aaa-13  bbb-13

-14-
aaa-14  bbb-14

-15-
aaa-15  bbb-15

-16-
aaa-16  bbb-16

-17-
aaa-17  bbb-17

-18-
aaa-18  bbb-18

-19-
aaa-19  bbb-19

$ command ls -d * | while read d; do /bin/echo -e "\n-$d-"; ls $d ; done

-10-
aaa-10  bbb-10

-11-
aaa-11  bbb-11

-12-
aaa-12  bbb-12

-13-

-14-
aaa-13  bbb-13
aaa-14  bbb-14

-15-
aaa-15  bbb-15

-16-
aaa-16  bbb-16

-17-

-18-

-19-
aaa-17  bbb-17
aaa-18  bbb-18
$ aaa-19  bbb-19   <I hit enter once here>

$


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: echo vs. ls race condition?

[Henry S. Thompson]

Fred Kulack <kulack@us.ibm.com> writes:

> Interesting tests. I still can't get it to fail...

Note _I_ can't get it to fail with only 10 subdirs -- 100 or 200 are
much more reliable at producing the problem. . .

> What terminal are you using? Perhaps there's something weird going on with 
> flow control as descriptors are dup'd for the utilities there? 
> I'm using plain old winblows console.

I can reproduce using windows console plus bash

> Did you try the regular echo instead of the builtin? It may have different 
> characteristics
> with regard to this.

OK, so _that's_ interesting -- using /bin/echo _and_ running under
windows console I _cannot_ reproduce.  Either running under rxvt _or_
using builtin echo produces the problem.  Also, using /bin/sh to run
the loop does _not_ reproduce the problem, even with builtin echo.

Here's a hack way to test for the bug automatically, by the way:

 > command ls -d * | while read d; do /bin/echo -e "\n-$d-"; ls $d ; done > xxx
 > egrep -A2 'aaa' xxx| tr -d '\012' | sed 's/--/\
/g' | egrep 'aaa.*aaa'

 > uname -a
   CYGWIN_NT-5.1 francis 1.5.9(0.112/4/2) 2004-03-18 23:05 i686 unknown
   unknown Cygwin
 > cygcheck -cd bash
   Cygwin Package Information
   Package              Version        
   bash                 2.05b-16

ht
-- 
 Henry S. Thompson, HCRC Language Technology Group, University of Edinburgh
                     Half-time member of W3C Team
    2 Buccleuch Place, Edinburgh EH8 9LW, SCOTLAND -- (44) 131 650-4440
            Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                   URL: http://www.ltg.ed.ac.uk/~ht/
[mail really from me _always_ has this .sig -- mail without it is forged spam]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/