SSH/SFTP Network Performance?

SSH/SFTP Network Performance?

[Jordan Geoghegan]

Hi folks,

First time poster here. I was hoping others could weigh in with their 
experiences running an ssh/sftp server from Cygwin.

tl;dr

Has anyone ever successfully transferred 150Mbps or more over the 
network using scp/sftp/rsync etc on Cygwin?

For a more in depth explanation, see below:


I have been experimenting with transferring data from my Windows 
machines through various methods, and was hoping to use rsync as that is 
what I use for backing up my other machines. During my testing, I have 
unfortunately been unable to push more than ~8MB/s (~60 megabits per 
second) regardless of the cipher (aes-gcm, aes-ctr, chacha20 etc) or 
transfer method (scp, sftp or rsync). All of them seem to be hitting 
some sort of bottleneck, with sftp being marginally faster than the others.

I am backing up to an OpenBSD file server on the local LAN. Using 
OpenBSD ---> OpenBSD (or Linux, MacOS etc) I can saturate the 1Gbit 
interface no problem. When using the built in Windows file sharing, I 
can also saturate the interface.

The fastest ssh transfer speed I have managed from Windows has been 
using various proprietary offerings such as the Bitvise ssh server or 
the SSH NetDrive sftp network drive mounter utility. Both of these tools 
also seem to hit some sort of bottlneck, though around ~18MB/s (~150 
megabit max as per Task Manager) for Bitvise, and around ~24MB/s (~200 
megabit max) with SSH netdrive.

Task Manager shows less than 5% cpu and disk usage, with only a slight 
bump up to 7% when using chacha20, I presume due to the lack of AES-NI 
for chacha. The machine is an i3 with an SSD and an Intel NIC so I am 
scratching my head as to what the issue could be. Cygwin will saturate 
my my 150Mbit WAN link in any direction over https no problem as well, 
which is interesting.

So, that's my story. Has anyone had any luck pushing beyond 150 megabits 
via ssh from/to Cygwin?



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: SSH/SFTP Network Performance?

[L A Walsh]

Jordan Geoghegan wrote:
> 
> Has anyone ever successfully transferred 150Mbps or more over the 
> network using scp/sftp/rsync etc on Cygwin?
----
My experience was to use the cifs protocol as easiest (over local
network).  With a 10Gb (limited to 8Gb due to running on older
PCIe interface) card, I'm getting:

/h> bin/iotest
Using bs=16.0M, count=64, iosize=1.0G
R:1073741824 bytes (1.0GB) copied, 1.67486 s, 611MB/s
W:1073741824 bytes (1.0GB) copied, 4.01592 s, 255MB/s

That's on a Win7 client to a linux-samba server.

If you want to get better results over SSH, you need to 
have a custom ssh that allows a null encryption for block
transfers.  See https://www.psc.edu/hpn-ssh.

Encryption really slows things down.

Also, on a local net, using Jumbo packets also helps
(I have a 9000 mtu on my local net).

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: SSH/SFTP Network Performance?

[Jordan Geoghegan]

Hi Elliot,

Thanks for the reply.

What was your setup that allowed you to reach 500Mbit? How were you 
copying the files? Were you using SFTP, SCP or rsync?

Was it a Cygwin to Cygwin transfer, or was there another OS involved?

Sorry for all the questions, I am just trying to rule out all possible 
variables.

Cheers,

Jordan Geoghegan


On 03/23/18 18:17, Eliot Moss wrote:
> Yes - I have gotten on the order of 500 Mbps, sometimes ranging
> a bit higher, on a (shared) 1 Gbps Ethernet LAN.  Wireless is
> not so good (in terms of reaching the max, sustained) - there
> is some evidence that in that case the Windows driver on my
> Surface Book is at least partly to blame.  Anyway, I can't say
> whether Cygwin is what prevent me from getting 1 Gbps sustained
> or not, since I didn't try it.
>
> Regards - Eliot Moss


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: SSH/SFTP Network Performance?

[Eliot Moss]

On 3/23/2018 9:54 PM, Jordan Geoghegan wrote:
> Hi Elliot,

It's Eliot, but no matter ...

> What was your setup that allowed you to reach 500Mbit? How were you copying the files? Were you 
> using SFTP, SCP or rsync?

I think I said rsync.  It was over ssh.  I disable compression since I use this
to send large files that are already compressed.

> Was it a Cygwin to Cygwin transfer, or was there another OS involved?

Sender: Surface Book Windows 10 (very up to date).  Microsoft 1 Gb USB Ethernet adapter.

Other end: RedHat kernel 3.10.0-693.17.1.el7.x86_64 on a recently purchased Xeon
server made by Dell.

As I think I mentioned before, I found that for wireless transfer (which this wasn't),
to get better performance than recent drivers, I had to go to an somewhat old driver,
and also disable 5G, using only the slower rate.  Given that when I use wireless I am
often at home and have only 12 Mbps uplink from Comcase, using the slower wireless is
not a bandwidth problem.  However, something about the drivers still limits me to about
half the theoretical rate.  I get a very regular square-wave sort of pattern when
watching the transfer rate, with a period of almost exactly 10 secs.  Again, that is
only for wireless.  Ethernet, except at the very highest rates, hits the max and generally
stays there, e.g., 150 Mbps on the slower net from my office.

Regards - Eliot

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Network Performance?

[L A Walsh]

L A Walsh wrote:
> Jordan Geoghegan wrote:
>> Has anyone ever successfully transferred 150Mbps or more over the 
>> network using scp/sftp/rsync etc on Cygwin?
> ----
What is more important? testing cygwin's scp/sftp/rsync, 
or using 'ssh'
or running under cygwin, 
or transferring the file to a local backup server from a windows
client as fast as possible?

Some test notes below(been benching my win<->server 
speeds since Win98 days)...



> Using bs=16.0M, count=64, iosize=1.0G
> R:1073741824 bytes (1.0GB) copied, 1.66724 s, 614MB/s
> W:1073741824 bytes (1.0GB) copied, 3.48363 s, 294MB/s
---
Some additional notes..."/h" is my home directory on the
linux server, so 'bin/iotest is a script in my linux home dir.

The speeds / values using 'Bytes', (2**3) use the base2
prefixes for consistency.  To get values in mbits, multiple
by 8.388608.

The above test only tests transfer speed -- not file i/o --
it uses /dev/zero for a source and /dev/null for a target.

To simplify testing, I created devices 'zero' and 'null'
in my home directory:
(on linux):
> ll zero null
crwxrw-rw- 1 1, 3 May 20  2016 null
crw-rw-rw- 1 1, 5 Jun 15  2015 zero
(on cygwin):
/h> ll zero null
-rwxrw-rw- 1 0 May 20  2016 null*
-rw-rw-rw- 1 0 Jun 15  2015 zero

For write, I used cygwin's 'dd' with if=/dev/zero and of=/h/null.
For read, I used if=/h/zero and of=/dev/null.

Flags for reading: iflag=fullblock conv=nocreat
  and for writing: oflag=direct iflag=fullblock conv=nocreat,notrunc

To test the cpu-bounding, I just tried limiting the clock speeds
on the client and server.
Limiting the linux-server's max cpu-speed had
the most affect on performance: 
(limited to 1.6GHz instead of 2.4GHz) (33% limitation)

Using bs=16.0M, count=256, iosize=4.0G (~35% slowdown)
R:4294967296 bytes (4.0GB) copied, 10.4467 s, 392MB/s
W:4294967296 bytes (4.0GB) copied, 21.5026 s, 190MB/s

Limiting the client (cygwin-win7sp1x64): (~7-13% slowdown)
(clock limited to 1.16GHz instead of 3.2)

Using bs=16.0M, count=256, iosize=4.0G
R:4294967296 bytes (4.0GB) copied, 7.14355 s, 573MB/s
W:4294967296 bytes (4.0GB) copied, 15.9781 s, 256MB/s

This would indicate that even in the unencrypted case
network transfer speed is cpu bound (and this is using
Jumbo packets).





--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Network Performance?

[Jordan Geoghegan]

On 03/24/18 17:04, L A Walsh wrote:
> L A Walsh wrote:
>> Jordan Geoghegan wrote:
>>> Has anyone ever successfully transferred 150Mbps or more over the 
>>> network using scp/sftp/rsync etc on Cygwin?
>> ----
> What is more important? testing cygwin's scp/sftp/rsync, or using 'ssh'
> or running under cygwin, or transferring the file to a local backup 
> server from a windows
> client as fast as possible?
What is most important is getting reasonable performance when moving 
data via ssh, be it sftp, rsync or some other sort of data tunnelling 
via ssh. I need to pull backups from a Windows box, and am only able to 
get ~8MB/s from it. All my other machines can saturate the line, even 
this Windows box when it isn't using Cygwin. I will be testing WSL 
(Windows Subsytem for Linux) to see how its "native" rsync fares in 
comparison.
>
> Some test notes below(been benching my win<->server speeds since Win98 
> days)...
>
>> Using bs=16.0M, count=64, iosize=1.0G
>> R:1073741824 bytes (1.0GB) copied, 1.66724 s, 614MB/s
>> W:1073741824 bytes (1.0GB) copied, 3.48363 s, 294MB/s
> ---

>
> The above test only tests transfer speed -- not file i/o --
> it uses /dev/zero for a source and /dev/null for a target.

> For write, I used cygwin's 'dd' with if=/dev/zero and of=/h/null.
> For read, I used if=/h/zero and of=/dev/null.
>
Thanks for the info. Would you be able to test file performance over the 
network? If you could just try copying a ~1GB file or what have you via 
sftp or rsync with actual writes to disk, I would be very interested to 
see how the numbers change.

Jordan Geoghegan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Network Performance?

[L A Walsh]

Jordan Geoghegan wrote:
> Thanks for the info. Would you be able to test file performance over the 
> network? If you could just try copying a ~1GB file or what have you via 
> sftp or rsync with actual writes to disk, I would be very interested to 
> see how the numbers change.
---
	Here is scp and rsync, using a tar, and a 1g file filled
with zeros.   Note, my env RSYNC_RSH=ssh

> time scp default-nlaw.tar.7z ishtar:/tmp  
default-nlaw.tar.7z                           100%   62MB  83.1MB/s   00:00    
0.91sec 0.04usr 0.01sys (6.65% cpu)
> ssh ishtar rm /tmp/default-nlaw.tar.7z
> time rsync -v default-nlaw.tar.7z ishtar:/tmp/
default-nlaw.tar.7z
sent 64,588,267 bytes  received 35 bytes  43,058,868.00 bytes/sec
total size is 64,572,413  speedup is 1.00
0.73sec 0.37usr 0.17sys (74.32% cpu)
time scp /tmp/1g ishtar:/tmp/ 
1g                                            100% 1014MB  72.3MB/s   00:14    
15.00sec 0.04usr 0.07sys (0.81% cpu)
> time rsync -v /tmp/1g ishtar:/tmp/
1g
sent 1,063,515,721 bytes  received 35 bytes  125,119,500.71 bytes/sec
total size is 1,063,256,064  speedup is 1.00
7.57sec 5.95usr 3.40sys (123.54% cpu)

> uname -a
CYGWIN_NT-6.1 Athenae 2.10.0(0.325/5/3) 2018-02-02 15:16 x86_64 Cygwin


I generally don't use them to transfer files over a closed
subnet as the performance is sub-optimal.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Network Performance?

[Jordan Geoghegan]

Thanks for testing that for me! I had feared I was having filesytem 
speed issues as seen in WSL, but alas I was wrong. It turns out my issue 
with network performance bottnecks is due to the TCP window size not 
adjusting. Looking at the debug output of ssh, it seems the window size 
adjusts aren't being performed. Strangely enough, when I grabbed a fresh 
machine to test Cygwin on, I am able to get near wire speeds. Very strange.


On 04/08/18 16:08, L A Walsh wrote:
> Jordan Geoghegan wrote:
>> Thanks for the info. Would you be able to test file performance over 
>> the network? If you could just try copying a ~1GB file or what have 
>> you via sftp or rsync with actual writes to disk, I would be very 
>> interested to see how the numbers change.
> ---
>     Here is scp and rsync, using a tar, and a 1g file filled
> with zeros.   Note, my env RSYNC_RSH=ssh
>
>> time scp default-nlaw.tar.7z ishtar:/tmp 
> default-nlaw.tar.7z                           100%   62MB 83.1MB/s   
> 00:00    0.91sec 0.04usr 0.01sys (6.65% cpu)
>> ssh ishtar rm /tmp/default-nlaw.tar.7z
>> time rsync -v default-nlaw.tar.7z ishtar:/tmp/
> default-nlaw.tar.7z
> sent 64,588,267 bytes  received 35 bytes  43,058,868.00 bytes/sec
> total size is 64,572,413  speedup is 1.00
> 0.73sec 0.37usr 0.17sys (74.32% cpu)
> time scp /tmp/1g ishtar:/tmp/ 
> 1g                                            100% 1014MB 72.3MB/s   
> 00:14    15.00sec 0.04usr 0.07sys (0.81% cpu)
>> time rsync -v /tmp/1g ishtar:/tmp/
> 1g
> sent 1,063,515,721 bytes  received 35 bytes  125,119,500.71 bytes/sec
> total size is 1,063,256,064  speedup is 1.00
> 7.57sec 5.95usr 3.40sys (123.54% cpu)
>
>> uname -a
> CYGWIN_NT-6.1 Athenae 2.10.0(0.325/5/3) 2018-02-02 15:16 x86_64 Cygwin
>
>
> I generally don't use them to transfer files over a closed
> subnet as the performance is sub-optimal.
>
> -- 
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple