Losing data with routine cp and mv -- "cannot create hard link"

Losing data with routine cp and mv -- "cannot create hard link"

[Thomas Baker]

I use Cygwin 1.3.17, NTFS file systems, and Win2000 (see
excerpt from cygcheck -s below).  Both with cp and mv, I am
getting error messages when copying or moving rather large
directories (1.4 GB, 89,000 files).  For example, the command:

    $ cp -Rip p:/rchive q:/

starts off fine, but half-way through the job, error messages such
as the following start to appear:

    cp: will not create hard link `q:/rchive/foo/bar' to directory `q:/rchive/fli/bers.gif'

..where the both filename A is always a directory name
and filename B (called a "directory" in the error message)
is usually a file, not a directory.

While showing these messages, cp and mv continue to execute
normally -- with error messages appearing intermittently --
except the directories named by filenames A are not copied,
i.e. the data is lost.  (Good thing I was paying attention
and not running these in batch mode!)

I could not find anything relevant in the mailing list archive
or FAQ.  Can anyone advise?  Is this an inherent limit of
these commands?  Should I perhaps be using xargs?

Tom



Cygwin Win95/NT Configuration Diagnostics
Current System Time: Sun Jan 12 18:41:03 2003

Windows 2000 Professional Ver 5.0 Build 2195 Service Pack 1

SysDir: C:\WINNT\System32
WinDir: C:\WINNT

    Cygwin DLL version info:
        DLL version: 1.3.17
        DLL epoch: 19
        DLL bad signal mask: 19005
        DLL old termios: 5
        DLL malloc env: 28
        API major: 0
        API minor: 67
        Shared data: 3
        DLL identifier: cygwin1
        Mount registry: 2
        Cygnus registry name: Cygnus Solutions
        Cygwin registry name: Cygwin
        Program options name: Program Options
        Cygwin mount registry name: mounts v2
        Cygdrive flags: cygdrive flags
        Cygdrive prefix: cygdrive prefix
        Cygdrive default prefix: 
        Build date: Wed Nov 27 18:54:29 EST 2002
        Shared id: cygwin1S3

-- 
Dr. Thomas Baker                                Thomas.Baker@bi.fhg.de
Institutszentrum Schloss Birlinghoven          mobile +49-171-408-5784
Fraunhofer-Gesellschaft                          work +49-30-8109-9027
53754 Sankt Augustin, Germany                    fax +49-2241-144-2352

Re: Losing data with routine cp and mv -- "cannot create hard link"

[Thomas Baker]

In the absence of responses to my earlier note (below), and
having made a second fruitless search of FAQs and archives,
I'd like to make a second and final attempt with a simpler
question:

   If cp and mv are not reliably copying all of the contents
   of an (apparently) normal directory tree with 89,000 normal
   data files, of 1.4 GB total size, using WIN2000 and NTFS,
   is it most likely due to inherent size limits of cygwin?

If the problems are due to inherent limits, then I can
adjust by copying such big directories in smaller chunks,
as I have already done successfully.  I just want to make
sure that this is in fact the problem.

Thanks,
Tom Baker


On Sun, Jan 12, 2003 at 08:17:32PM +0100, Thomas Baker wrote:
> I use Cygwin 1.3.17, NTFS file systems, and Win2000 (see
> excerpt from cygcheck -s below).  Both with cp and mv, I am
> getting error messages when copying or moving rather large
> directories (1.4 GB, 89,000 files).  For example, the command:
> 
>     $ cp -Rip p:/rchive q:/
> 
> starts off fine, but half-way through the job, error messages such
> as the following start to appear:
> 
>     cp: will not create hard link `q:/rchive/foo/bar' to directory `q:/rchive/fli/bers.gif'
> 
> ..where the both filename A is always a directory name
> and filename B (called a "directory" in the error message)
> is usually a file, not a directory.
> 
> While showing these messages, cp and mv continue to execute
> normally -- with error messages appearing intermittently --
> except the directories named by filenames A are not copied,
> i.e. the data is lost.  (Good thing I was paying attention
> and not running these in batch mode!)
> 
> I could not find anything relevant in the mailing list archive
> or FAQ.  Can anyone advise?  Is this an inherent limit of
> these commands?  Should I perhaps be using xargs?
> 
> Tom
> 
> 
> 
> Cygwin Win95/NT Configuration Diagnostics
> Current System Time: Sun Jan 12 18:41:03 2003
> 
> Windows 2000 Professional Ver 5.0 Build 2195 Service Pack 1
> 
> SysDir: C:\WINNT\System32
> WinDir: C:\WINNT
> 
>     Cygwin DLL version info:
>         DLL version: 1.3.17
>         DLL epoch: 19
>         DLL bad signal mask: 19005
>         DLL old termios: 5
>         DLL malloc env: 28
>         API major: 0
>         API minor: 67
>         Shared data: 3
>         DLL identifier: cygwin1
>         Mount registry: 2
>         Cygnus registry name: Cygnus Solutions
>         Cygwin registry name: Cygwin
>         Program options name: Program Options
>         Cygwin mount registry name: mounts v2
>         Cygdrive flags: cygdrive flags
>         Cygdrive prefix: cygdrive prefix
>         Cygdrive default prefix: 
>         Build date: Wed Nov 27 18:54:29 EST 2002
>         Shared id: cygwin1S3
> 
> -- 
> Dr. Thomas Baker                                Thomas.Baker@bi.fhg.de
> Institutszentrum Schloss Birlinghoven          mobile +49-171-408-5784
> Fraunhofer-Gesellschaft                          work +49-30-8109-9027
> 53754 Sankt Augustin, Germany                    fax +49-2241-144-2352
> 
> --
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
> Bug reporting:         http://cygwin.com/bugs.html
> Documentation:         http://cygwin.com/docs.html
> FAQ:                   http://cygwin.com/faq/
> 

-- 
Dr. Thomas Baker                                Thomas.Baker@bi.fhg.de
Institutszentrum Schloss Birlinghoven          mobile +49-171-408-5784
Fraunhofer-Gesellschaft                          work +49-30-8109-9027
53754 Sankt Augustin, Germany                    fax +49-2241-144-2352

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Losing data with routine cp and mv -- "cannot create hard link"

[Bjoern Kahl AG Resy]

 Hallo!

 I do not know what is going on really, but I have seen something
 like that before.

On Thu, 16 Jan 2003, Thomas Baker wrote:

>    If cp and mv are not reliably copying all of the contents
>    of an (apparently) normal directory tree with 89,000 normal
>    data files, of 1.4 GB total size, using WIN2000 and NTFS,
>    is it most likely due to inherent size limits of cygwin?

  <wild guessing mode>

 There seems to be a random delay in the NT-Filesystem when renaming
 files. This can be easily triggert on smb-shares, but also on
 "normal" drives.

 If renaming (that is: moving around) files, there is a short,
 load-dependend delay between removing the old direktory entry and
 creating the new one. This can even be observed in the windows
 explorer, and I *think* that is not a slow-gui-issue, but the file
 is *really* *not* *there* form some time.

 </wild guessing mode>

 So dont think there is any thing cygwin can do.

> If the problems are due to inherent limits, then I can
> adjust by copying such big directories in smaller chunks,
> as I have already done successfully.  I just want to make
> sure that this is in fact the problem.

 If I move/copy some files over the net, I add sleep instructions
 ("for i in * ; do mv $i $i.bak ; sleep 1 ; sed <$i.bak >$i ... ; done")
 slow, but works. On *my* system, the magic number is around 50 files.
 Less than 50 files works without sleep, more files require the
 sleep. (Else I get a lot random "No such file xxx.bak".)


   Bjoern

-- 
+---------------------------------------------------------------------+
| Dipl.-Phys. Bjoern Kahl +++ AG Embedded Systems and Robotics (RESY) |
| Informatics Faculty +++ Building 48 +++ University of Kaiserslautern|
| phone: +49-631-205-2654 +++ www: http://resy.informatik.uni-kl.de   |
+---------------------------------------------------------------------+


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Losing data with routine cp and mv -- "cannot create hard link"

[Larry Hall (RFK Partners, Inc)]

At 04:55 AM 1/16/2003, Thomas Baker wrote:
>In the absence of responses to my earlier note (below), and
>having made a second fruitless search of FAQs and archives,
>I'd like to make a second and final attempt with a simpler
>question:
>
>    If cp and mv are not reliably copying all of the contents
>    of an (apparently) normal directory tree with 89,000 normal
>    data files, of 1.4 GB total size, using WIN2000 and NTFS,
>    is it most likely due to inherent size limits of cygwin?
>
>If the problems are due to inherent limits, then I can
>adjust by copying such big directories in smaller chunks,
>as I have already done successfully.  I just want to make
>sure that this is in fact the problem.


If you're looking for some 'official' validation of what you see, 
I'm not sure that you'll find it here.  It's not that we wouldn't like
to give it (or even refute your findings ;-) ).  It's just that I don't 
believe anyone has done as much analysis of this issue as you have.  
Certainly, the values you report (89000 files occupying 1.4GB of space) are
not, in and of themselves, an obvious red-flag.  If you'd like to get a 
better handle on the situation (and help the list understand your problem as
well), it would be worthwhile to run strace on this process and look
for any suspect results.  Be warned.  This will generate a huge file
with lots of output, most of which is not going to indicate any problems.
However, if you can help isolate a problem area, it may be easier for 
someone here to diagnose the problem and offer a fix... or at least an
explanation.

Sorry I don't have the magic bullet for you.

  
Larry Hall                              lhall@rfk.com
RFK Partners, Inc.                      http://www.rfk.com
838 Washington Street                   (508) 893-9779 - RFK Office
Holliston, MA 01746                     (508) 893-9889 - FAX


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Losing data with routine cp and mv -- "cannot create hard link"

[Thomas Baker]

Dear Bjoern,

This sounds very plausible and fits with my impression that
the files which cause problems are really random, with no
obvious pattern or periodicity, and with my observation that
this problem seems to come up even more frequently when I move
files from one machine to another (notably slow) machine than
when I just copy them between partitions on the same machine.

Many thanks for sharing your observation.
Tom

On Thu, Jan 16, 2003 at 11:19:14AM +0100, Bjoern Kahl AG Resy wrote:
>  I do not know what is going on really, but I have seen something
>  like that before.
> 
> On Thu, 16 Jan 2003, Thomas Baker wrote:
> 
> >    If cp and mv are not reliably copying all of the contents
> >    of an (apparently) normal directory tree with 89,000 normal
> >    data files, of 1.4 GB total size, using WIN2000 and NTFS,
> >    is it most likely due to inherent size limits of cygwin?
> 
>   <wild guessing mode>
> 
>  There seems to be a random delay in the NT-Filesystem when renaming
>  files. This can be easily triggert on smb-shares, but also on
>  "normal" drives.
> 
>  If renaming (that is: moving around) files, there is a short,
>  load-dependend delay between removing the old direktory entry and
>  creating the new one. This can even be observed in the windows
>  explorer, and I *think* that is not a slow-gui-issue, but the file
>  is *really* *not* *there* form some time.
> 
>  </wild guessing mode>
> 
>  So dont think there is any thing cygwin can do.
> 
> > If the problems are due to inherent limits, then I can
> > adjust by copying such big directories in smaller chunks,
> > as I have already done successfully.  I just want to make
> > sure that this is in fact the problem.
> 
>  If I move/copy some files over the net, I add sleep instructions
>  ("for i in * ; do mv $i $i.bak ; sleep 1 ; sed <$i.bak >$i ... ; done")
>  slow, but works. On *my* system, the magic number is around 50 files.
>  Less than 50 files works without sleep, more files require the
>  sleep. (Else I get a lot random "No such file xxx.bak".)
> 
> 
>    Bjoern
> 
> -- 
> +---------------------------------------------------------------------+
> | Dipl.-Phys. Bjoern Kahl +++ AG Embedded Systems and Robotics (RESY) |
> | Informatics Faculty +++ Building 48 +++ University of Kaiserslautern|
> | phone: +49-631-205-2654 +++ www: http://resy.informatik.uni-kl.de   |
> +---------------------------------------------------------------------+
> 

-- 
Dr. Thomas Baker                                Thomas.Baker@bi.fhg.de
Institutszentrum Schloss Birlinghoven          mobile +49-171-408-5784
Fraunhofer-Gesellschaft                          work +49-30-8109-9027
53754 Sankt Augustin, Germany                    fax +49-2241-144-2352

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Losing data with routine cp and mv -- "cannot create hard link"

[Thomas Baker]

Simply knowing that this is not a frequently reported problem
is helpful.  I'm inclined to suspect now that Bjoern (previous
post) is on the right track by looking at the timing of the
deleting and creating that goes on under Win2000 (in this case)
in order to "mv" something.

Many thanks,
Tom Baker

On Thu, Jan 16, 2003 at 09:55:01AM -0500, Larry Hall (RFK Partners, Inc) wrote:
> At 04:55 AM 1/16/2003, Thomas Baker wrote:
> >In the absence of responses to my earlier note (below), and
> >having made a second fruitless search of FAQs and archives,
> >I'd like to make a second and final attempt with a simpler
> >question:
> >
> >    If cp and mv are not reliably copying all of the contents
> >    of an (apparently) normal directory tree with 89,000 normal
> >    data files, of 1.4 GB total size, using WIN2000 and NTFS,
> >    is it most likely due to inherent size limits of cygwin?
> >
> >If the problems are due to inherent limits, then I can
> >adjust by copying such big directories in smaller chunks,
> >as I have already done successfully.  I just want to make
> >sure that this is in fact the problem.
> 
> If you're looking for some 'official' validation of what you see, 
> I'm not sure that you'll find it here.  It's not that we wouldn't like
> to give it (or even refute your findings ;-) ).  It's just that I don't 
> believe anyone has done as much analysis of this issue as you have.  
> Certainly, the values you report (89000 files occupying 1.4GB of space) are
> not, in and of themselves, an obvious red-flag.  If you'd like to get a 
> better handle on the situation (and help the list understand your problem as
> well), it would be worthwhile to run strace on this process and look
> for any suspect results.  Be warned.  This will generate a huge file
> with lots of output, most of which is not going to indicate any problems.
> However, if you can help isolate a problem area, it may be easier for 
> someone here to diagnose the problem and offer a fix... or at least an
> explanation.
> 
> Sorry I don't have the magic bullet for you.

-- 
Dr. Thomas Baker                                Thomas.Baker@bi.fhg.de
Institutszentrum Schloss Birlinghoven          mobile +49-171-408-5784
Fraunhofer-Gesellschaft                          work +49-30-8109-9027
53754 Sankt Augustin, Germany                    fax +49-2241-144-2352

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/