From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 26172 invoked by alias); 25 Jun 2009 14:37:11 -0000 Received: (qmail 26162 invoked by uid 22791); 25 Jun 2009 14:37:10 -0000 X-Spam-Check-By: sourceware.org Received: from aquarius.hirmke.de (HELO calimero.vinschen.de) (217.91.18.234) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 25 Jun 2009 14:37:03 +0000 Received: by calimero.vinschen.de (Postfix, from userid 500) id 3A1FF6D5598; Thu, 25 Jun 2009 16:36:51 +0200 (CEST) Date: Thu, 25 Jun 2009 15:21:00 -0000 From: Corinna Vinschen To: cygwin@cygwin.com Subject: Re: [1.7.0-50] scp progress counter flies through first 175 MB or so Message-ID: <20090625143651.GR7289@calimero.vinschen.de> Reply-To: cygwin@cygwin.com Mail-Followup-To: cygwin@cygwin.com References: <2ul445d2cfjj1q2t2viropiikoj70slglb@4ax.com> <20090625101039.GP7289@calimero.vinschen.de> <20090625140508.GA17040@ednor.casa.cgf.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090625140508.GA17040@ednor.casa.cgf.cx> User-Agent: Mutt/1.5.19 (2009-02-20) Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com X-SW-Source: 2009-06/txt/msg00890.txt.bz2 On Jun 25 10:05, Christopher Faylor wrote: > On Thu, Jun 25, 2009 at 12:10:39PM +0200, Corinna Vinschen wrote: > >On Jun 24 13:17, Andrew Schulman wrote: > >> Here's an odd one. > >> > >> Using openssh 5.2p1-2 with Cygwin 1.7.0-50, when I scp any file, the > >> progress counter appears to show ridiculously fast transfer rates, e.g. > >> about 50 MB/s over a 750 KB/s connection, for the first 175 MB or so. After > >> that the counter settles down to normal speed. Then when the counter > >> reaches the end, it "hangs" at 100% for the remaining time while the copy > >> finishes. > >> > >> At first I thought that the copy itself was being corrupted in the first > >> 175 MB, but I'm no longer able to reproduce that. I believe now that the > >> copy is good and it's only the progress counter that's wrong. > >> > >> When I revert to Cygwin 1.7.0-49, this problem doesn't occur. > > > >I can reproduce that copying a file via scp from a Windows machine to > >a Linux box. > > > >It looks like the pipes between the local scp and the local ssh are now > >exchanging the data quicker at the start than the ssh socket can send > >them to the remote machine. On my XP machine, scp advances quickly by > >about 260 Megs (hard to tell, maybe it's exaclty 256 Megs for some > >reason?), then keeps the advance roughly at that value until scp > >finished. At the end scp is just waiting for ssh which still has to > >send the 256/260 Megs of data. > > > >This is really weird, given that Cygwin does not create such a big > >buffer for the pipe. Consequentially Task Manager claims that the > >memory is neither taken by scp, nor by ssh. Both processes have normal > >VM sizes < 10 Megs. Per Task manager the memory is paged Kernel Memory. > >A strange side effect is that the entire time taken by the data > >transmission is longer than with -49, by almost exactly the time it > >takes to empty the big kernel cache. > > > >Puzzeling. > > Is ssh using non-blocking pipes opened for write? Until a week or two > ago, Cygwin didn't support those and treated the non-blocking write as a > blocking write. scp switches the pipes to non-blocking and then tries to do blocking io on its own, using the poll() function. It calls a function called scpio which basically work like this: scpio (io_function, fd, buf, size) { for (offset = 0; offset < size;) { r = io_function (fd, buf + offset, size - offset); [...] if (r < 0) { if (errno == EINTR) continue; if (errno == EAGAIN || errno == EWOULDBLOCK) { poll (fd, 1, -1) /* Use poll() for blocking */ continue; } [...] } offset += r; } } Looks like scp now stumbles over the pipe select() implementation. Btw., scp knows a `-l bandwidth' parameter to allow to specify a maximum throughput. If you set this to a value which corresponds to the maximum throughput of the sending socket in ssh, then you can "optimize" the problem away and you're back to the old timings. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple