Limitation in SCP?

Limitation in SCP?

[Bob Smart]

I've discovered that in some circumstances, scp seems to get tired early when transferring a largish file (around 200MB).  It copies diligently for the first 20-40MB (exactly where it stops varies from run to run), then CPU usage drops to zero, the file stops growing on the receiving end, there's no apparent network traffic, and everything just pretty much halts.  A ps shows the scp task running, but not doing anything.

I've only seen this when copying between two W2K machines (both Cygwin, both using the Cygwin SSH package, both of pretty recent vintage with DLL 1.5.11).  I did my development testing between Linux and NT, and I never saw this behavior in that environment.  It always fails with the W2K machines.

I also never see this happen with tar.  If I do a tar-to-tar pipe via SSH, everything works even in the W2K-to-W2K environment.  However, I'd like to understand why I can't use scp for this.

-- 
GPG public key <bsmart@blorch.org> available from public key server
network or from www.blorch.org/bob

Fingerprint BA4A 552C BE3D 8C40 ED76 F372 DF9E 320D 37FA 16AC

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Michael Uman]

Hello,
I had a similar problem copying a file from RH9.0 linux to WinXP Cygwin
where the copy stopped at 128MB every time I tried to copy the file... I
ended up bzipping the file and scping it...

I can provide more information later {I am away from that particular
machine}...

Michael

On Mon, 2004-12-13 at 12:37, Bob Smart wrote:
> I've discovered that in some circumstances, scp seems to get tired early when transferring a largish file (around 200MB).  It copies diligently for the first 20-40MB (exactly where it stops varies from run to run), then CPU usage drops to zero, the file stops growing on the receiving end, there's no apparent network traffic, and everything just pretty much halts.  A ps shows the scp task running, but not doing anything.
> 
> I've only seen this when copying between two W2K machines (both Cygwin, both using the Cygwin SSH package, both of pretty recent vintage with DLL 1.5.11).  I did my development testing between Linux and NT, and I never saw this behavior in that environment.  It always fails with the W2K machines.
> 
> I also never see this happen with tar.  If I do a tar-to-tar pipe via SSH, everything works even in the W2K-to-W2K environment.  However, I'd like to understand why I can't use scp for this.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Francis Litterio]

Bob Smart wrote:

> I've discovered that in some circumstances, scp seems to get tired early when
> transferring a largish file (around 200MB).  It copies diligently for the first
> 20-40MB (exactly where it stops varies from run to run), then CPU usage drops to
> zero, the file stops growing on the receiving end, there's no apparent network
> traffic, and everything just pretty much halts.  A ps shows the scp task
> running, but not doing anything.

I've seen the same thing, except I see scp stop transferring data _very_
close to the end of the large file (i.e., within 10000 bytes of the
end).  I've only seen this when scp'ing files that are over 100 MB.

I've found that this equivalent command works when scp fails:

	ssh remotehost cat remotefile > localfile

> I've only seen this when copying between two W2K machines

In my case it's only between two XP machines.
--
Francis Litterio
franl@world.std.com
http://world.std.com/~franl/
GPG and PGP public keys available on keyservers.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Corinna Vinschen]

On Dec 13 14:37, Bob Smart wrote:
> I've discovered that in some circumstances, scp seems to get tired early when transferring a largish file (around 200MB).  It copies diligently for the first 20-40MB (exactly where it stops varies from run to run), then CPU usage drops to zero, the file stops growing on the receiving end, there's no apparent network traffic, and everything just pretty much halts.  A ps shows the scp task running, but not doing anything.
> 
> I've only seen this when copying between two W2K machines (both Cygwin, both using the Cygwin SSH package, both of pretty recent vintage with DLL 1.5.11).  I did my development testing between Linux and NT, and I never saw this behavior in that environment.  It always fails with the W2K machines.
> 
> I also never see this happen with tar.  If I do a tar-to-tar pipe via SSH, everything works even in the W2K-to-W2K environment.  However, I'd like to understand why I can't use scp for this.

Hmm, that doesn't happen for me.  I tried to copy around a 200 Megs file
between WinXP and XP<->Linux multiple times and don't see any hang.  Did
you try with Cygwin 1.5.12, too?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          mailto:cygwin@cygwin.com
Red Hat, Inc.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Jim Kleckner]

Corinna Vinschen wrote:

> On Dec 13 14:37, Bob Smart wrote:
> 
>> I've discovered that in some circumstances, scp seems to get tired
>> early when transferring a largish file (around 200MB).  It copies
>> diligently for the first 20-40MB (exactly where it stops varies
>> from run to run), then CPU usage drops to zero, the file stops
>> growing on the receiving end, there's no apparent network traffic,
>> and everything just pretty much halts.  A ps shows the scp task
>> running, but not doing anything.
>> 
>> I've only seen this when copying between two W2K machines (both
>> Cygwin, both using the Cygwin SSH package, both of pretty recent
>> vintage with DLL 1.5.11).  I did my development testing between
>> Linux and NT, and I never saw this behavior in that environment.
>> It always fails with the W2K machines.
>> 
>> I also never see this happen with tar.  If I do a tar-to-tar pipe
>> via SSH, everything works even in the W2K-to-W2K environment.
>> However, I'd like to understand why I can't use scp for this.
> 
> 
> Hmm, that doesn't happen for me.  I tried to copy around a 200 Megs
> file between WinXP and XP<->Linux multiple times and don't see any
> hang.  Did you try with Cygwin 1.5.12, too?

Is this possibly related to the pipe hang issue that some
apps have with ssh such as mentioned here and that
Bob Byrnes is looking at?:
  http://sources.redhat.com/ml/cygwin/2004-11/msg01162.html

Bob, are you "pushing" or "pulling" the files?  It has
been noted that rsync works to push files to windows
but hangs in a similar way when pulling files from a
remote machine.  Again, it may or may not be related.

Jim

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Christian Weinberger]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:

> Hmm, that doesn't happen for me.  I tried to copy around a 200 Megs file
> between WinXP and XP<->Linux multiple times and don't see any hang.  Did
> you try with Cygwin 1.5.12, too?
> 
> Corinna

Is it possible the cygwin has a more generic problem here?

This may be off-topic (excuses for that case), but I saw some really strage 
things recently when I configured UW-IMAP.

To prevent quite frequent hangs, I used a wrapper script

#!/bin/sh
tee NUL | /usr/sbin/imapd

and everything was perfect. 
See http://sources.redhat.com/ml/cygwin/2004-11/msg01137.html for details.

PS: I´m also using XP (with SP2)




--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Bob Smart]

I suppose it could be related to the XP piping problem, but I only have NT and W2K.  All I can really say is that it never happens (to me, anyway) on NT.  It certainly sounds like what others have described on XP.

The transfer that fails is a "pull" whether I'm using scp (which fails) or the tar-to-tar method (which doesn't).  It kind of has to be that way, too, because I can't be sure that sshd will always be running on the destination end to support a "push" mode of operation.

-- 
GPG public key <bsmart@blorch.org> available from public key server
network or from www.blorch.org/bob

Fingerprint BA4A 552C BE3D 8C40 ED76 F372 DF9E 320D 37FA 16AC

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Limitation in SCP?

[Bob Smart]

I haven't yet had an opportunity to try 1.5.12, and it may be a while (just because I'm responsible for keeping the databases on these machines running doesn't mean I get regular access to them...) but I'll keep that in mind next time I get a shot at them.

I'd really like to blame Winduhs networking or the drivers for these particular NICs with this particular version of W2K, but then why would tar and cat work any better than scp?

Shrug.  I have a couple of viable workarounds, so it's primarily a matter of idle curiosity for me at this point.

-- 
GPG public key <bsmart@blorch.org> available from public key server
network or from www.blorch.org/bob

Fingerprint BA4A 552C BE3D 8C40 ED76 F372 DF9E 320D 37FA 16AC

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/