Reliable old script loses data on new Cygwin installation

Reliable old script loses data on new Cygwin installation

[Thomas Baker]

Dear all,

I have been using a Korn shell script twenty times per day
for more than ten years -- I published an earlier version
in UnixWorld in 1994 and the latest version last year in
lifehacker.com [1].

When I now run the script on brand-new Cygwin installations,
it loses data.  I have tested the script on fresh data many
times and find that it may run correctly twenty times and
then fail.  Then it will run correctly ten more times and fail,
but at a different point in processing the data.

The line of the script that is causing the failure is:

	gawk '$'$Searchfield' !~ /'$Searchkey'/' $3 >|$Tmpfile; mv $Tmpfile $3

(it gets STDOUT from a previous line), where:

    Tmpdir=/tmp             
    Tmpfile=$Tmpdir/shf$$.tmp    (i.e., /tmp/shf2080.tmp)

i.e.:

	gawk '$1 !~ /^LINX/ $3' >|/tmp/sht2080.tmp; mv /tmp/sht2080.tmp huh2

What I get is error messages like the following:

    mv: cannot create regular file `huh2': Permission denied
    gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)
    gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)

What I then find is that data has been lost.  If I interrupt
the script right after the error message I find files 
(such as "huh2") that have a length of zero -- OR I find a file
listed with a correct-looking length but garbage contents.
For example, the text file (before running the script):

    - 2007-10-28  20:25       4010  german        

comes out the other end looking like

    - 2007-10-28  20:31       4010  german        

but "od german" shows the _entire_ contents of the file to be:

    0000000 000000 000000 000000 000000 000000 000000 000000 000000
    *
    0007640 000000 000000 000000 000000 000000
    0007652

I have used this script on many machines and platforms over
the years (Linux, Solaris, DOS 3.3 with MKS Toolkit, Win95,
Win2000, XP).  It is now failing on two different WinXP
computers on which I have just installed Cygwin from the Web.
-- one brand-new (with an English-language XP and a SATA hard
disk) and one two years old (with a German-language-XP and
an ATA hard disk).

One of the machines with the new Cygwin installation also lost
lots of data when I scheduled it to move a large subdirectory
(using "mv") in the middle of the night.  The message I found
on my return was:

   "Windows - Device TimeOut"
   The specified I/O operation on \Device\Harddisk7\DR10 was not completed before the 
   time-out period expired.

Comparing the packages installed on my laptop (which has no
problems with "mv") with the packages on the two computers
with the new Cygwin installation, I see:

            Older (works)     Newer installation (broken)
coreutils   6.4-1             6.9-5
gawk        3.1.5-4           3.1.6-1
pdksh       5.2.14-3          5.2.14-3

The two computers on which I am getting errors have the
following in common:

-- Their processors and hard disks are faster than *most*
   of the computers I have been using (both are generic
   home-office PCs from Siemens-Fujitsu).

-- Both now have the latest version of Cygwin from the Web.

I have searched FAQs and mailing lists for problems with
"timeout" and the like but find nothing obviously relevant.
A colleague of mine who knows alot more about these things is
scratching his head on this.  Is it possible that something
in the more recent versions of Cygwin packages is causing
this problem?

Many thanks,
Tom Baker

[1] http://lifehacker.com/software/command-line/getting-things-done-with-rulebased-list-processing-217063.php



-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Reliable old script loses data on new Cygwin installation

[Thomas Baker]

Dear all,

I have been using a Korn shell script twenty times per day
for more than ten years -- I published an earlier version
in UnixWorld in 1994 and the latest version last year in
lifehacker.com [1].

When I now run the script on brand-new Cygwin installations,
it loses data.  I have tested the script on fresh data many
times and find that it may run correctly twenty times and
then fail.  Then it will run correctly ten more times and fail,
but at a different point in processing the data.

The line of the script that is causing the failure is:

	gawk '$'$Searchfield' !~ /'$Searchkey'/' $3 >|$Tmpfile; mv $Tmpfile $3

(it gets STDOUT from a previous line), where:

    Tmpdir=/tmp             
    Tmpfile=$Tmpdir/shf$$.tmp    (i.e., /tmp/shf2080.tmp)

i.e.:

	gawk '$1 !~ /^LINX/ $3' >|/tmp/sht2080.tmp; mv /tmp/sht2080.tmp huh2

What I get is error messages like the following:

    mv: cannot create regular file `huh2': Permission denied
    gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)
    gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)

What I then find is that data has been lost.  If I interrupt
the script right after the error message I find files 
(such as "huh2") that have a length of zero -- OR I find a file
listed with a correct-looking length but garbage contents.
For example, the text file (before running the script):

    - 2007-10-28  20:25       4010  german        

comes out the other end looking like

    - 2007-10-28  20:31       4010  german        

but "od german" shows the _entire_ contents of the file to be:

    0000000 000000 000000 000000 000000 000000 000000 000000 000000
    *
    0007640 000000 000000 000000 000000 000000
    0007652

I have used this script on many machines and platforms over
the years (Linux, Solaris, DOS 3.3 with MKS Toolkit, Win95,
Win2000, XP).  It is now failing on two different WinXP
computers on which I have just installed Cygwin from the Web.
-- one brand-new (with an English-language XP and a SATA hard
disk) and one two years old (with a German-language-XP and
an ATA hard disk).

One of the machines with the new Cygwin installation also lost
lots of data when I scheduled it to move a large subdirectory
(using "mv") in the middle of the night.  The message I found
on my return was:

   "Windows - Device TimeOut"
   The specified I/O operation on \Device\Harddisk7\DR10 was not completed before the 
   time-out period expired.

Comparing the packages installed on my laptop (which has no
problems with "mv") with the packages on the two computers
with the new Cygwin installation, I see:

            Older (works)     Newer installation (broken)
coreutils   6.4-1             6.9-5
gawk        3.1.5-4           3.1.6-1
pdksh       5.2.14-3          5.2.14-3

The two computers on which I am getting errors have the
following in common:

-- Their processors and hard disks are faster than *most*
   of the computers I have been using (both are generic
   home-office PCs from Siemens-Fujitsu).

-- Both now have the latest version of Cygwin from the Web.

I have searched FAQs and mailing lists for problems with
"timeout" and the like but find nothing obviously relevant.
A colleague of mine who knows alot more about these things is
scratching his head on this.  Is it possible that something
in the more recent versions of Cygwin packages is causing
this problem?

Many thanks,
Tom Baker

[1] http://lifehacker.com/software/command-line/getting-things-done-with-rulebased-list-processing-217063.php

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[René Berber]

Thomas Baker wrote:

[snip]
> I have searched FAQs and mailing lists for problems with
> "timeout" and the like but find nothing obviously relevant.
[snip]

I have seen that problem and it has nothing to do with Cygwin.  The
problem is with SATA drives and Window's asynchronous unbuffered disk
I/O, and my Adaptec 1210SA SATA card and driver (actually, I know the
driver is the culprit, but newer drivers are so bad that they can't even
be installed).

With application that use asynchronous unbuffered disk I/O the disk
stops responding after a while, and Windows pops up an error panel that
shows the "timeout" message.

How does is happen in your setting?  I don't know.  Notice that I said
"applications", Windows doesn't use the problematic mode, I've only seen
a couple of applications using it when you configure them to use the
fastest disk I/O possible, one used it all the time (Azureus?) so it was
unusable with that disk.  Could the problem be caused by something else
that runs at the same time as your script?

Hope this helps.
-- 
René Berber


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

René Berber wrote:
> [snip]
> > I have searched FAQs and mailing lists for problems with
> > "timeout" and the like but find nothing obviously relevant.
> [snip]
> 
> I have seen that problem and it has nothing to do with Cygwin.  The
> problem is with SATA drives and Window's asynchronous unbuffered disk
> I/O, and my Adaptec 1210SA SATA card and driver (actually, I know the
> driver is the culprit, but newer drivers are so bad that they can't even
> be installed).
> 
> With application that use asynchronous unbuffered disk I/O the disk
> stops responding after a while, and Windows pops up an error panel that
> shows the "timeout" message.
> 
> How does is happen in your setting?  I don't know.  Notice that I said
> "applications", Windows doesn't use the problematic mode, I've only seen
> a couple of applications using it when you configure them to use the
> fastest disk I/O possible, one used it all the time (Azureus?) so it was
> unusable with that disk.  Could the problem be caused by something else
> that runs at the same time as your script?

Thank you, René.  

At first we did think it was a problem with the SATA disk --
and it does seem like a plausible explanation for the error
message:

     "Windows - Device TimeOut"
     The specified I/O operation on \Device\Harddisk7\DR10 was not completed before the 
     time-out period expired.

However, the other problem (see below) has occurred --
sporadically -- on three different machines, all running
German or English-language versions of XP, two with SATA
disks and one with an ATA disk, all with freshly downloaded
installations of cygwin.  The line that causes the problem is:

> 	gawk '$1 !~ /^LINX/ $3' >|/tmp/sht2080.tmp; mv /tmp/sht2080.tmp huh2
...
> What I get is error messages like the following:
> 
>     mv: cannot create regular file `huh2': Permission denied
>     gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)
>     gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)
> 
> What I then find is that data has been lost.  If I interrupt
> the script right after the error message I find files 
> (such as "huh2") that have a length of zero -- OR I find a file
> listed with a correct-looking length but garbage contents.
> For example, the text file (before running the script):
> 
>     - 2007-10-28  20:25       4010  german        
> 
> comes out the other end looking like
> 
>     - 2007-10-28  20:31       4010  german        
> 
> but "od german" shows the _entire_ contents of the file to be:
> 
>     0000000 000000 000000 000000 000000 000000 000000 000000 000000
>     *
>     0007640 000000 000000 000000 000000 000000
>     0007652

It seems plausible (to me as a non-expert) that an asynchronous
unbuffered disk could be responsible for this problem too.
However, I am getting this error on _also_ on an older machine
with an ATA disk.

The three test machines on which the problem is occurring have
two things in common:

-- They all have some version of XP with the most recent Cygwin
   installation plus Firefox, OpenOffice, and Java and nothing else.

-- They are all faster than the machines I have been using over the
   years.

A colleague of mine suspects that the Korn shell script on
Cygwin is running so fast that occasionally the next command
is being executed before the buffer is written to disk.  Is it
possible that the shell is somehow creating the file "german"
(above), with its file name and length, a split second before
the contents are written to disk, then the next command is
being run too quickly, the script gets tripped up but keeps
running, and data is lost?

As this is happening both on a SATA disk and an ATA disk, I
can't help wondering whether cygwin is perhaps too efficient
for the faster hardware.  My colleague suggests I modify the
script to add 500 milliseconds of wait time between

    gawk '$1 !~ /^LINX/ $3' >|/tmp/sht2080.tmp
    
and

    mv /tmp/sht2080.tmp huh2

However, he says that this could conceivably solve the problem
for this script, but if the problem is that Cygwin is too fast
for the hardware I could still get this problem while using,
say, "mv".  Can this explanation be ruled out?

Tom

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[René Berber]

Thomas Baker wrote:

[snip]
> However, the other problem (see below) has occurred --
> sporadically -- on three different machines, all running
> German or English-language versions of XP, two with SATA
> disks and one with an ATA disk, all with freshly downloaded
> installations of cygwin.[snip]

OK, that's interesting, 3 machines one using non SATA disk.  Same
irregular pattern of 10 to 20 runs without problem?

> A colleague of mine suspects that the Korn shell script on
> Cygwin is running so fast that occasionally the next command
> is being executed before the buffer is written to disk.  Is it
> possible that the shell is somehow creating the file "german"
> (above), with its file name and length, a split second before
> the contents are written to disk, then the next command is
> being run too quickly, the script gets tripped up but keeps
> running, and data is lost?

No.  The actual storage of data is done by the operating system, not by
Cygwin, the operating system provides the data even if it hasn't been
stored on disk (it could still be in a memory buffer).

> As this is happening both on a SATA disk and an ATA disk, I
> can't help wondering whether cygwin is perhaps too efficient
> for the faster hardware.[snip]
> However, he says that this could conceivably solve the problem
> for this script, but if the problem is that Cygwin is too fast
> for the hardware I could still get this problem while using,
> say, "mv".  Can this explanation be ruled out?

Move is different, it usually only renames the file, but does a copy if
necessary.  But again, it is the operating system the one that does the
actual change, what any program does is usually to a buffer in memory
which is marked to be stored by the operating system.

The speed is not the problem, it could be the usual suspect: an
anti-virus, unlikely because the data written is not executable but it
could be adding an extraneous delay between data written and data read.

The only solid evidence is the error message from Windows, and it says
"device", that means that the hard disk is having problems (it could be
the driver) but not your script or any program.  Did you do anything
special to the hard disks on all 3 machines? something like run an
optimizer or tune parameters?

The other problem could be related to the first, if the disk is
"failing" then creating/moving a file will fail, I'm not sure if
"permission denied" will be the error message but I could test that
later (I can make my SATA disk fail using a program that allows the
async unbuffered I/O).
-- 
René Berber


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Morgan Gangwere]

/!\ WOOGA WOOGA /!\
 Possible Stupid human Error!
/!\ OOOPS OOOPS /!\

if the copy of xp is new enough, its got a funky little tool called
the Windows Malware Detection And removal Tool (a freind of mine who
is an MS MVP has had the same problem as you just under a different
circumstance.)

I took an sp1 machine, created some text files that i wanted to search
for and got no problems on an athlon machine (2.4Ghz)
I then upgraded to sp2 -- same machine, same 200gb sata hdd and everything

no problems

then i let it run its updates

BAM

trace shows:

its the windows malware detector (aka windows defender or whatever you
want to call it.)

also, avg can do this too

try moving 200 files with mv -r /tmp/std-output/* /logs/$$/, each
about 30kb each

-- 
Morgan gangwere

Please Excuse TOFU. Gmail/Mobile has no Power.

"Space does not reflect society, it expresses it." -- Castells, M.,
Space of Flows, Space of Places: Materials for a Theory of Urbanism in
the Information Age, in The Cybercities Reader, S. Graham, Editor.
2004, Routledge: London. p. 82-93.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Robert Pendell]

Morgan Gangwere wrote:
> /!\ WOOGA WOOGA /!\
>  Possible Stupid human Error!
> /!\ OOOPS OOOPS /!\
> 
> if the copy of xp is new enough, its got a funky little tool called
> the Windows Malware Detection And removal Tool (a freind of mine who
> is an MS MVP has had the same problem as you just under a different
> circumstance.)
> 
> I took an sp1 machine, created some text files that i wanted to search
> for and got no problems on an athlon machine (2.4Ghz)
> I then upgraded to sp2 -- same machine, same 200gb sata hdd and everything
> 
> no problems
> 
> then i let it run its updates
> 
> BAM
> 
> trace shows:
> 
> its the windows malware detector (aka windows defender or whatever you
> want to call it.)
> 
> also, avg can do this too
> 
> try moving 200 files with mv -r /tmp/std-output/* /logs/$$/, each
> about 30kb each
> 

One other thing I wanted to add to this was make sure the drives are not 
overheating.  Some drives will actually shut down or begin to act 
erratically if they get too hot.  I discovered this when doing work with 
my external drive (notebook ide using usb adapter).

P.S. - The reason for this issue was because I had the drive near an 
exhaust vent for my laptop -- mistake :(  It is normally fast enough to 
handle high speed burns or file transfers as necessary.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Mon, Nov 12, 2007 at 04:03:14AM -0600, René Berber wrote:
> [snip]
> > However, the other problem (see below) has occurred --
> > sporadically -- on three different machines, all running
> > German or English-language versions of XP, two with SATA
> > disks and one with an ATA disk, all with freshly downloaded
> > installations of cygwin.[snip]
> 
> OK, that's interesting, 3 machines one using non SATA disk.  Same
> irregular pattern of 10 to 20 runs without problem?

I haven't run it enough to see clear patterns, but yes,
the error occurs in an irregular manner.  Since then I have
tested this on a fourth machine -- my old machine, with newly
installed XP and freshly downloaded Cywin but a slower hard
disk -- and I get the same irregular errors there too!

> No.  The actual storage of data is done by the operating system, not by
> Cygwin, the operating system provides the data even if it hasn't been
> stored on disk (it could still be in a memory buffer).

Okay, thank you for the explanation.

> The speed is not the problem, it could be the usual suspect: an
> anti-virus, unlikely because the data written is not executable but it
> could be adding an extraneous delay between data written and data read.

I'll ask the guy who installed the operating systems whether this
looks possible.

> The only solid evidence is the error message from Windows, and it says
> "device", that means that the hard disk is having problems (it could be
> the driver) but not your script or any program.  Did you do anything
> special to the hard disks on all 3 machines? something like run an
> optimizer or tune parameters?

I'll find out.

> The other problem could be related to the first, if the disk is
> "failing" then creating/moving a file will fail, I'm not sure if
> "permission denied" will be the error message but I could test that
> later (I can make my SATA disk fail using a program that allows the
> async unbuffered I/O).

The /bin/pdksh script sequence that is causing problems is:

   1. gawk '$1 !~ /blahblah/' huh2 >|/tmp/shf2080.tmp
      (no error messages)

   2. mv /tmp/shf2080.tmp huh2
      mv: cannot create regular file `huh2': Permission denied

   3. (some other commands in a looping construct)

   4. gawk '$1 !~ /blahblah/' huh2 >|/tmp/shf2080.tmp
      gawk: cmd. line:1: fatal: cannot open file `huh2' for reading (No such file or directory)

Sometimes the sequence runs correct.  Sometimes it results in a
file "huh2" with length "0".  But the sequence has _sometimes_
resulted in the creation of files such as:

   - 2007-10-28  20:31       4010  german

which _appear_ to have a length of 4010, but where "od german"
shows:

    0000000 000000 000000 000000 000000 000000 000000 000000 000000
    *
    0007640 000000 000000 000000 000000 000000
    0007652

I'll get answers to the questions above...

Tom Baker

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Mon, Nov 12, 2007 at 09:38:03PM -0500, Morgan Gangwere wrote:
> if the copy of xp is new enough, its got a funky little tool called
> the Windows Malware Detection And removal Tool (a freind of mine who
> is an MS MVP has had the same problem as you just under a different
> circumstance.)

I'll find out - thanks.

Tom Baker

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Tue, Nov 13, 2007 at 09:58:31AM -0500, Robert Pendell wrote:
> One other thing I wanted to add to this was make sure the drives are not 
> overheating.  Some drives will actually shut down or begin to act 
> erratically if they get too hot.  I discovered this when doing work with 
> my external drive (notebook ide using usb adapter).
> 
> P.S. - The reason for this issue was because I had the drive near an 
> exhaust vent for my laptop -- mistake :(  It is normally fast enough to 
> handle high speed burns or file transfers as necessary.

Thank you for the suggestion, but little script does its
work in 30 seconds or so, so the problem is occurring when
the computer has been moving a few little files around for
just a few minutes at most.

Tom Baker

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Tue, Nov 13, 2007 at 06:06:21PM +0100, Thomas Baker wrote:
> > if the copy of xp is new enough, its got a funky little tool called
> > the Windows Malware Detection And removal Tool (a freind of mine who
> > is an MS MVP has had the same problem as you just under a different
> > circumstance.)

I confirmed that there is no Malware tool on these machines,
so that cannot be the explanation.

Tom

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Tue, Nov 13, 2007 at 05:59:01PM +0100, Thomas Baker wrote:
> > The speed is not the problem, it could be the usual suspect: an
> > anti-virus, unlikely because the data written is not executable but it
> > could be adding an extraneous delay between data written and data read.
> 
> I'll ask the guy who installed the operating systems whether this
> looks possible.

These machines have an anti-virus program, but the same one I have 
been using for the past two or three years.  The filenames either have no 
extension, or ".txt".

> > The only solid evidence is the error message from Windows, and it says
> > "device", that means that the hard disk is having problems (it could be
> > the driver) but not your script or any program.  Did you do anything
> > special to the hard disks on all 3 machines? something like run an
> > optimizer or tune parameters?
> 
> I'll find out.

The guy who set up the machines did not fiddle with the hard
disk at all.  Two of the machines were brand-new.  On these,
he did not touch the hardware - only swapped the installed
German-language version of XP for my own version of XP
in English.  Then I installed Cygwin from the net.

I'm a bit at a loss as to what to do next.  I use this one 
script so extensively (50 times per day?) that I would sooner
move to a Linux or Macintosh system than continue on Windows
without Cygwin - my whole working style depends on it.

I work on a laptop with Cygwin and an up-to-date version of
XP, and thankfully the script continues to run fine there.
Having (seemingly) eliminated SATA/ATA, fast/slow, and
old-XP/up-to-date-XMP as possible explanations for the errors,
the only really obvious remaining difference between my laptop
(where the script works) and the four desktop computers
(where the script does not work reliably) is that the four
desktop computers have the latest version of Cygwin and the
laptop has a version that I have not updated in awhile (and
now I do not dare!).

I would be grateful if someone could suggest a way to test
this.  Would it make sense (and is it possible) to replace
the Cygwin kernel or the "coreutils" package (because of
"mv") with earlier versions and see if the script works then?
The version of "pdksh" is the same between the laptop and the
desktops, so I assume this can be eliminated as the cause?
The script itself is not long, and the data sets are also
just a few pages long, so I could prepare a tar file with a
simple test installation if anyone would like to try it on
their machines.

Tom

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Corinna Vinschen]

On Nov 14 08:22, Thomas Baker wrote:
> I would be grateful if someone could suggest a way to test
> this.  Would it make sense (and is it possible) to replace
> the Cygwin kernel or the "coreutils" package (because of
> "mv") with earlier versions and see if the script works then?

You should first create a minimal, self-contained script which allows to
reproduce the effect reliably on your machines.  A really minimal
script.  Self-contained.  So that everybody can run it to try to
reproduce the effect without relying on your environment.

Then you could try to change the versions of coreutils and gawk.  At no
point in this thread did you mention different Cygwin versions.  Are you
running 1.5.24 on all of them including your laptop?  If so, there's no
point in experimenting with different Cygwin DLLs.

If you found the difference, you could send the observation here
together with your really minimal, self-contained script, and a quick
description what to do to reproduce the effect and how it can be
observed.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Wed, Nov 14, 2007 at 10:07:09AM +0100, Corinna Vinschen wrote:
> > I would be grateful if someone could suggest a way to test
> > this.  Would it make sense (and is it possible) to replace
> > the Cygwin kernel or the "coreutils" package (because of
> > "mv") with earlier versions and see if the script works then?
> 
> You should first create a minimal, self-contained script which allows to
> reproduce the effect reliably on your machines.  A really minimal
> script.  Self-contained.  So that everybody can run it to try to
> reproduce the effect without relying on your environment.

Okay.

> Then you could try to change the versions of coreutils and gawk.  At no
> point in this thread did you mention different Cygwin versions.  Are you
> running 1.5.24 on all of them including your laptop?  If so, there's no
> point in experimenting with different Cygwin DLLs.

On my laptop, I am using (uname -srv):
CYGWIN_NT-5.1 1.5.22(0.156/4/2) 2006-11-13 17:01

(FWIW, I agree with Igor Pechtchanski's suggestion [1]
that the Cygwin FAQ include more detail on how to
check version [2].)

[1] http://www.cygwin.com/ml/cygwin-xfree/2004-02/msg00121.html
[2] http://www.cygwin.com/faq/faq.html#faq.what.version

> If you found the difference, you could send the observation here
> together with your really minimal, self-contained script, and a quick
> description what to do to reproduce the effect and how it can be
> observed.

Thanks, understood - I'll give this a try.

Tom Baker

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Lewis Hyatt]

> These machines have an anti-virus program, but the same one I have 
> been using for the past two or three years.  The filenames either have no 
> extension, or ".txt".

That's most likely the problem anyway, what happens if you turn it off?


-Lewis


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Larry Hall (Cygwin)]

Lewis Hyatt wrote:
>> These machines have an anti-virus program, but the same one I have 
>> been using for the past two or three years.  The filenames either have 
>> no extension, or ".txt".
> 
> That's most likely the problem anyway, what happens if you turn it off?


This is possible, particularly if the AV program has been updated.  If so,
turning it off is typically not enough to avoid the problems introduced by
it though.  You may need to uninstall it (and reboot) to completely rule it
out.


-- 
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      (508) 893-9779 - RFK Office
216 Dalton Rd.                          (508) 893-9889 - FAX
Holliston, MA 01746

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Wed, Nov 14, 2007 at 06:03:42PM -0500, Larry Hall (Cygwin) wrote:
> Lewis Hyatt wrote:
> >>These machines have an anti-virus program, but the same one I have 
> >>been using for the past two or three years.  The filenames either have 
> >>no extension, or ".txt".
> >
> >That's most likely the problem anyway, what happens if you turn it off?
> 
> This is possible, particularly if the AV program has been updated.  If so,
> turning it off is typically not enough to avoid the problems introduced by
> it though.  You may need to uninstall it (and reboot) to completely rule it
> out.

Okay - I'll try this too.

Thank you,
Tom Baker

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script... - seems to be an AVS program

[Thomas Baker]

On Wed, Nov 14, 2007 at 06:03:42PM -0500, Larry Hall (Cygwin) wrote:
> >>These machines have an anti-virus program, but the same one I have 
> >>been using for the past two or three years.  The filenames either have 
> >>no extension, or ".txt".
> >
> >That's most likely the problem anyway, what happens if you turn it off?
> 
> This is possible, particularly if the AV program has been updated.  If so,
> turning it off is typically not enough to avoid the problems introduced by
> it though.  You may need to uninstall it (and reboot) to completely rule it
> out.

I uninstalled Anti-Vir [1], and the script has run hundreds
of times on several different data sources with no errors yet,
so this may indeed be the problem!

One way for me to confirm that the problem is _not_ the kernel would
be to update Cygwin on my laptop, which currently uses

CYGWIN_NT-5.1 1.5.22(0.156/4/2) 2006-11-13 17:01

However, the laptop is currently my only reliably functioning
machine, so I'm deeply reluctant to do so unless I have an
easy and reliable way to go back to the kernel 1.5.22 if
a problem occurs.  

Question: Is the kernel itself part of one of the packages
installed by setup.exe, and is it possible to use setup.exe
to go back to a previous kernel (i.e., to 1.5.22) after one 
has updated to the most recent version?

I see there was a thread last month about AVS programs [2]
but see no clear recommendations except one for Anti-Vir [3]
-- the problem that is causing me problems now! -- and one
recommendation for AVG [4,5].

Tom

[1] http://www.free-av.com/
[2] http://sourceware.org/ml/cygwin/2007-10/msg00341.html
[3] http://sourceware.org/ml/cygwin/2007-10/msg00342.html
[4] http://sourceware.org/ml/cygwin/2007-10/msg00348.html
[5] http://free.grisoft.com/

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Reliable old script... - seems to be an AVS program

[Dave Korn]

On 16 November 2007 09:45, Thomas Baker wrote:

> I uninstalled Anti-Vir [1], and the script has run hundreds
> of times on several different data sources with no errors yet,
> so this may indeed be the problem!

  There's another one for the list then I guess.  Ouch.  Nothing's worse than
data loss.

> Question: Is the kernel itself part of one of the packages
> installed by setup.exe, and is it possible to use setup.exe
> to go back to a previous kernel (i.e., to 1.5.22) after one
> has updated to the most recent version?

  Yep.  The "kernel" of cygwin is implemented as a DLL, /bin/cygwin1.dll, and
is shipped in the "cygwin" setup package (along with a few essential related
files).  You can use setup.exe to wind it forward or back.

> I see there was a thread last month about AVS programs [2]
> but see no clear recommendations except one for Anti-Vir [3]
> -- the problem that is causing me problems now! -- and one
> recommendation for AVG [4,5].

  Been running it for several years now, both free and for the past couple
paid-for versions, on many and varied cygwin hosts.  No problems.  Did once
get a false positive on a couple of dlls, but <shrugs> can happen to anybody;
notified grisoft, fetched them back from the virus vault, next update fixed
the bad detect.  Thoroughly impressed by the reliability and low overhead.

    cheers,
      DaveK
[*] - 
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script... AVS problem - RESOLUTION

[Thomas Baker]

On Fri, Nov 16, 2007 at 10:40:27AM -0000, Dave Korn wrote:
> On 16 November 2007 09:45, Thomas Baker wrote:
> > I uninstalled Anti-Vir [1], and the script has run hundreds
> > of times on several different data sources with no errors yet,
> > so this may indeed be the problem!
> 
>   There's another one for the list then I guess.  Ouch.  Nothing's worse than
> data loss.

Further testing strongly suggests (so far) that the problem
is indeed caused by Anti-Vir.  The solution, however, appears
to be as simple as configuring Anti-Vir to check files only
upon reading, _not_ upon writing.  (I was going to try AVG,
but that no longer seems necessary.)

Four weeks and three machines later, at least I know now.
Can anyone explain why check-on-write would cause this problem
-- maybe because Anti-Vir was holding up writing to disk
until after the next command in my script had been executed
and had tried to read the file that was not yet written?

Though not a "Cygwin problem" per se, I'm guessing that
other people must have this problem as well - maybe a line
of explanation could be added to future BLODA lists and/or
the Cygwin FAQ.

Many thanks to everyone who took the time to respond to my 
postings.

Tom Baker

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Igor Peshansky]

On Tue, 13 Nov 2007, Thomas Baker wrote:

> The /bin/pdksh script sequence that is causing problems is:
> [snip]

The obvious question is: do you still get the error if you replace pdksh
with bash, or is the problem pdksh-specific?

When you reduce this to a minimal testcase, are there bits of that
testcase that require pdksh to be used?
	Igor
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_	    pechtcha@cs.nyu.edu | igor@watson.ibm.com
ZZZzz /,`.-'`'    -.  ;-;;,_		Igor Peshansky, Ph.D. (name changed!)
     |,4-  ) )-,_. ,\ (  `'-'		old name: Igor Pechtchanski
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

Belief can be manipulated.  Only knowledge is dangerous.  -- Frank Herbert

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Reliable old script loses data on new Cygwin installation

[Thomas Baker]

On Thu, Nov 22, 2007 at 12:56:33PM -0500, Igor Peshansky wrote:
> > The /bin/pdksh script sequence that is causing problems is:
> > [snip]
> 
> The obvious question is: do you still get the error if you replace pdksh
> with bash, or is the problem pdksh-specific?
> 
> When you reduce this to a minimal testcase, are there bits of that
> testcase that require pdksh to be used?

It appears that pdksh (or Cygwin) per se had nothing to do
with the problem.  As I reported in [1], the problem was
resolved when I discovered that the anti-virus software
(Anti-Vir) was checking each time a file is written to disk.

When I switched off this extra check (so that Anti-Vir now
checks only when files are read), the script ran perceptibly
faster and data is no longer lost.

I originally wrote the script using the MKS Toolkit Korn shell
with DOS 3.3, and it uses some features which may be Korn
shell-specific (e.g., redirecting to numbered file descriptors
with constructs such as "3>|") -- I have not really tried to
port it to bash.

Tom

[1] http://cygwin.com/ml/cygwin/2007-11/msg00313.html

-- 
Tom Baker - tbaker@tbaker.de - baker@sub.uni-goettingen.de

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/