* SSH/SFTP Network Performance? @ 2018-03-23 22:40 Jordan Geoghegan 2018-03-24 0:57 ` L A Walsh [not found] ` <d1353aaf-271a-ba2f-7de0-80ca15b18e01@cs.umass.edu> 0 siblings, 2 replies; 8+ messages in thread From: Jordan Geoghegan @ 2018-03-23 22:40 UTC (permalink / raw) To: cygwin Hi folks, First time poster here. I was hoping others could weigh in with their experiences running an ssh/sftp server from Cygwin. tl;dr Has anyone ever successfully transferred 150Mbps or more over the network using scp/sftp/rsync etc on Cygwin? For a more in depth explanation, see below: I have been experimenting with transferring data from my Windows machines through various methods, and was hoping to use rsync as that is what I use for backing up my other machines. During my testing, I have unfortunately been unable to push more than ~8MB/s (~60 megabits per second) regardless of the cipher (aes-gcm, aes-ctr, chacha20 etc) or transfer method (scp, sftp or rsync). All of them seem to be hitting some sort of bottleneck, with sftp being marginally faster than the others. I am backing up to an OpenBSD file server on the local LAN. Using OpenBSD ---> OpenBSD (or Linux, MacOS etc) I can saturate the 1Gbit interface no problem. When using the built in Windows file sharing, I can also saturate the interface. The fastest ssh transfer speed I have managed from Windows has been using various proprietary offerings such as the Bitvise ssh server or the SSH NetDrive sftp network drive mounter utility. Both of these tools also seem to hit some sort of bottlneck, though around ~18MB/s (~150 megabit max as per Task Manager) for Bitvise, and around ~24MB/s (~200 megabit max) with SSH netdrive. Task Manager shows less than 5% cpu and disk usage, with only a slight bump up to 7% when using chacha20, I presume due to the lack of AES-NI for chacha. The machine is an i3 with an SSD and an Intel NIC so I am scratching my head as to what the issue could be. Cygwin will saturate my my 150Mbit WAN link in any direction over https no problem as well, which is interesting. So, that's my story. Has anyone had any luck pushing beyond 150 megabits via ssh from/to Cygwin? -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SSH/SFTP Network Performance? 2018-03-23 22:40 SSH/SFTP Network Performance? Jordan Geoghegan @ 2018-03-24 0:57 ` L A Walsh 2018-03-25 10:24 ` L A Walsh [not found] ` <d1353aaf-271a-ba2f-7de0-80ca15b18e01@cs.umass.edu> 1 sibling, 1 reply; 8+ messages in thread From: L A Walsh @ 2018-03-24 0:57 UTC (permalink / raw) To: cygwin Jordan Geoghegan wrote: > > Has anyone ever successfully transferred 150Mbps or more over the > network using scp/sftp/rsync etc on Cygwin? ---- My experience was to use the cifs protocol as easiest (over local network). With a 10Gb (limited to 8Gb due to running on older PCIe interface) card, I'm getting: /h> bin/iotest Using bs=16.0M, count=64, iosize=1.0G R:1073741824 bytes (1.0GB) copied, 1.67486 s, 611MB/s W:1073741824 bytes (1.0GB) copied, 4.01592 s, 255MB/s That's on a Win7 client to a linux-samba server. If you want to get better results over SSH, you need to have a custom ssh that allows a null encryption for block transfers. See https://www.psc.edu/hpn-ssh. Encryption really slows things down. Also, on a local net, using Jumbo packets also helps (I have a 9000 mtu on my local net). -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Network Performance? 2018-03-24 0:57 ` L A Walsh @ 2018-03-25 10:24 ` L A Walsh 2018-03-27 3:54 ` Jordan Geoghegan 0 siblings, 1 reply; 8+ messages in thread From: L A Walsh @ 2018-03-25 10:24 UTC (permalink / raw) To: cygwin; +Cc: jgeoghegan60 L A Walsh wrote: > Jordan Geoghegan wrote: >> Has anyone ever successfully transferred 150Mbps or more over the >> network using scp/sftp/rsync etc on Cygwin? > ---- What is more important? testing cygwin's scp/sftp/rsync, or using 'ssh' or running under cygwin, or transferring the file to a local backup server from a windows client as fast as possible? Some test notes below(been benching my win<->server speeds since Win98 days)... > Using bs=16.0M, count=64, iosize=1.0G > R:1073741824 bytes (1.0GB) copied, 1.66724 s, 614MB/s > W:1073741824 bytes (1.0GB) copied, 3.48363 s, 294MB/s --- Some additional notes..."/h" is my home directory on the linux server, so 'bin/iotest is a script in my linux home dir. The speeds / values using 'Bytes', (2**3) use the base2 prefixes for consistency. To get values in mbits, multiple by 8.388608. The above test only tests transfer speed -- not file i/o -- it uses /dev/zero for a source and /dev/null for a target. To simplify testing, I created devices 'zero' and 'null' in my home directory: (on linux): > ll zero null crwxrw-rw- 1 1, 3 May 20 2016 null crw-rw-rw- 1 1, 5 Jun 15 2015 zero (on cygwin): /h> ll zero null -rwxrw-rw- 1 0 May 20 2016 null* -rw-rw-rw- 1 0 Jun 15 2015 zero For write, I used cygwin's 'dd' with if=/dev/zero and of=/h/null. For read, I used if=/h/zero and of=/dev/null. Flags for reading: iflag=fullblock conv=nocreat and for writing: oflag=direct iflag=fullblock conv=nocreat,notrunc To test the cpu-bounding, I just tried limiting the clock speeds on the client and server. Limiting the linux-server's max cpu-speed had the most affect on performance: (limited to 1.6GHz instead of 2.4GHz) (33% limitation) Using bs=16.0M, count=256, iosize=4.0G (~35% slowdown) R:4294967296 bytes (4.0GB) copied, 10.4467 s, 392MB/s W:4294967296 bytes (4.0GB) copied, 21.5026 s, 190MB/s Limiting the client (cygwin-win7sp1x64): (~7-13% slowdown) (clock limited to 1.16GHz instead of 3.2) Using bs=16.0M, count=256, iosize=4.0G R:4294967296 bytes (4.0GB) copied, 7.14355 s, 573MB/s W:4294967296 bytes (4.0GB) copied, 15.9781 s, 256MB/s This would indicate that even in the unencrypted case network transfer speed is cpu bound (and this is using Jumbo packets). -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Network Performance? 2018-03-25 10:24 ` L A Walsh @ 2018-03-27 3:54 ` Jordan Geoghegan 2018-04-08 23:08 ` L A Walsh 0 siblings, 1 reply; 8+ messages in thread From: Jordan Geoghegan @ 2018-03-27 3:54 UTC (permalink / raw) To: L A Walsh, cygwin On 03/24/18 17:04, L A Walsh wrote: > L A Walsh wrote: >> Jordan Geoghegan wrote: >>> Has anyone ever successfully transferred 150Mbps or more over the >>> network using scp/sftp/rsync etc on Cygwin? >> ---- > What is more important? testing cygwin's scp/sftp/rsync, or using 'ssh' > or running under cygwin, or transferring the file to a local backup > server from a windows > client as fast as possible? What is most important is getting reasonable performance when moving data via ssh, be it sftp, rsync or some other sort of data tunnelling via ssh. I need to pull backups from a Windows box, and am only able to get ~8MB/s from it. All my other machines can saturate the line, even this Windows box when it isn't using Cygwin. I will be testing WSL (Windows Subsytem for Linux) to see how its "native" rsync fares in comparison. > > Some test notes below(been benching my win<->server speeds since Win98 > days)... > >> Using bs=16.0M, count=64, iosize=1.0G >> R:1073741824 bytes (1.0GB) copied, 1.66724 s, 614MB/s >> W:1073741824 bytes (1.0GB) copied, 3.48363 s, 294MB/s > --- > > The above test only tests transfer speed -- not file i/o -- > it uses /dev/zero for a source and /dev/null for a target. > For write, I used cygwin's 'dd' with if=/dev/zero and of=/h/null. > For read, I used if=/h/zero and of=/dev/null. > Thanks for the info. Would you be able to test file performance over the network? If you could just try copying a ~1GB file or what have you via sftp or rsync with actual writes to disk, I would be very interested to see how the numbers change. Jordan Geoghegan -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Network Performance? 2018-03-27 3:54 ` Jordan Geoghegan @ 2018-04-08 23:08 ` L A Walsh 2018-04-09 18:30 ` Jordan Geoghegan 0 siblings, 1 reply; 8+ messages in thread From: L A Walsh @ 2018-04-08 23:08 UTC (permalink / raw) To: cygwin Jordan Geoghegan wrote: > Thanks for the info. Would you be able to test file performance over the > network? If you could just try copying a ~1GB file or what have you via > sftp or rsync with actual writes to disk, I would be very interested to > see how the numbers change. --- Here is scp and rsync, using a tar, and a 1g file filled with zeros. Note, my env RSYNC_RSH=ssh > time scp default-nlaw.tar.7z ishtar:/tmp default-nlaw.tar.7z 100% 62MB 83.1MB/s 00:00 0.91sec 0.04usr 0.01sys (6.65% cpu) > ssh ishtar rm /tmp/default-nlaw.tar.7z > time rsync -v default-nlaw.tar.7z ishtar:/tmp/ default-nlaw.tar.7z sent 64,588,267 bytes received 35 bytes 43,058,868.00 bytes/sec total size is 64,572,413 speedup is 1.00 0.73sec 0.37usr 0.17sys (74.32% cpu) time scp /tmp/1g ishtar:/tmp/ 1g 100% 1014MB 72.3MB/s 00:14 15.00sec 0.04usr 0.07sys (0.81% cpu) > time rsync -v /tmp/1g ishtar:/tmp/ 1g sent 1,063,515,721 bytes received 35 bytes 125,119,500.71 bytes/sec total size is 1,063,256,064 speedup is 1.00 7.57sec 5.95usr 3.40sys (123.54% cpu) > uname -a CYGWIN_NT-6.1 Athenae 2.10.0(0.325/5/3) 2018-02-02 15:16 x86_64 Cygwin I generally don't use them to transfer files over a closed subnet as the performance is sub-optimal. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Network Performance? 2018-04-08 23:08 ` L A Walsh @ 2018-04-09 18:30 ` Jordan Geoghegan 0 siblings, 0 replies; 8+ messages in thread From: Jordan Geoghegan @ 2018-04-09 18:30 UTC (permalink / raw) To: L A Walsh; +Cc: cygwin Thanks for testing that for me! I had feared I was having filesytem speed issues as seen in WSL, but alas I was wrong. It turns out my issue with network performance bottnecks is due to the TCP window size not adjusting. Looking at the debug output of ssh, it seems the window size adjusts aren't being performed. Strangely enough, when I grabbed a fresh machine to test Cygwin on, I am able to get near wire speeds. Very strange. On 04/08/18 16:08, L A Walsh wrote: > Jordan Geoghegan wrote: >> Thanks for the info. Would you be able to test file performance over >> the network? If you could just try copying a ~1GB file or what have >> you via sftp or rsync with actual writes to disk, I would be very >> interested to see how the numbers change. > --- >     Here is scp and rsync, using a tar, and a 1g file filled > with zeros.  Note, my env RSYNC_RSH=ssh > >> time scp default-nlaw.tar.7z ishtar:/tmp > default-nlaw.tar.7z                          100%  62MB 83.1MB/s  > 00:00   0.91sec 0.04usr 0.01sys (6.65% cpu) >> ssh ishtar rm /tmp/default-nlaw.tar.7z >> time rsync -v default-nlaw.tar.7z ishtar:/tmp/ > default-nlaw.tar.7z > sent 64,588,267 bytes received 35 bytes 43,058,868.00 bytes/sec > total size is 64,572,413 speedup is 1.00 > 0.73sec 0.37usr 0.17sys (74.32% cpu) > time scp /tmp/1g ishtar:/tmp/ > 1g                                           100% 1014MB 72.3MB/s  > 00:14   15.00sec 0.04usr 0.07sys (0.81% cpu) >> time rsync -v /tmp/1g ishtar:/tmp/ > 1g > sent 1,063,515,721 bytes received 35 bytes 125,119,500.71 bytes/sec > total size is 1,063,256,064 speedup is 1.00 > 7.57sec 5.95usr 3.40sys (123.54% cpu) > >> uname -a > CYGWIN_NT-6.1 Athenae 2.10.0(0.325/5/3) 2018-02-02 15:16 x86_64 Cygwin > > > I generally don't use them to transfer files over a closed > subnet as the performance is sub-optimal. > > -- > Problem reports:      http://cygwin.com/problems.html > FAQ:                  http://cygwin.com/faq/ > Documentation:        http://cygwin.com/docs.html > Unsubscribe info:     http://cygwin.com/ml/#unsubscribe-simple > -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <d1353aaf-271a-ba2f-7de0-80ca15b18e01@cs.umass.edu>]
* Re: SSH/SFTP Network Performance? [not found] ` <d1353aaf-271a-ba2f-7de0-80ca15b18e01@cs.umass.edu> @ 2018-03-24 2:06 ` Jordan Geoghegan 2018-03-24 5:07 ` Eliot Moss 0 siblings, 1 reply; 8+ messages in thread From: Jordan Geoghegan @ 2018-03-24 2:06 UTC (permalink / raw) To: moss; +Cc: cygwin Hi Elliot, Thanks for the reply. What was your setup that allowed you to reach 500Mbit? How were you copying the files? Were you using SFTP, SCP or rsync? Was it a Cygwin to Cygwin transfer, or was there another OS involved? Sorry for all the questions, I am just trying to rule out all possible variables. Cheers, Jordan Geoghegan On 03/23/18 18:17, Eliot Moss wrote: > Yes - I have gotten on the order of 500 Mbps, sometimes ranging > a bit higher, on a (shared) 1 Gbps Ethernet LAN. Wireless is > not so good (in terms of reaching the max, sustained) - there > is some evidence that in that case the Windows driver on my > Surface Book is at least partly to blame. Anyway, I can't say > whether Cygwin is what prevent me from getting 1 Gbps sustained > or not, since I didn't try it. > > Regards - Eliot Moss -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: SSH/SFTP Network Performance? 2018-03-24 2:06 ` SSH/SFTP " Jordan Geoghegan @ 2018-03-24 5:07 ` Eliot Moss 0 siblings, 0 replies; 8+ messages in thread From: Eliot Moss @ 2018-03-24 5:07 UTC (permalink / raw) To: Jordan Geoghegan; +Cc: cygwin On 3/23/2018 9:54 PM, Jordan Geoghegan wrote: > Hi Elliot, It's Eliot, but no matter ... > What was your setup that allowed you to reach 500Mbit? How were you copying the files? Were you > using SFTP, SCP or rsync? I think I said rsync. It was over ssh. I disable compression since I use this to send large files that are already compressed. > Was it a Cygwin to Cygwin transfer, or was there another OS involved? Sender: Surface Book Windows 10 (very up to date). Microsoft 1 Gb USB Ethernet adapter. Other end: RedHat kernel 3.10.0-693.17.1.el7.x86_64 on a recently purchased Xeon server made by Dell. As I think I mentioned before, I found that for wireless transfer (which this wasn't), to get better performance than recent drivers, I had to go to an somewhat old driver, and also disable 5G, using only the slower rate. Given that when I use wireless I am often at home and have only 12 Mbps uplink from Comcase, using the slower wireless is not a bandwidth problem. However, something about the drivers still limits me to about half the theoretical rate. I get a very regular square-wave sort of pattern when watching the transfer rate, with a period of almost exactly 10 secs. Again, that is only for wireless. Ethernet, except at the very highest rates, hits the max and generally stays there, e.g., 150 Mbps on the slower net from my office. Regards - Eliot -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-04-09 18:30 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-03-23 22:40 SSH/SFTP Network Performance? Jordan Geoghegan 2018-03-24 0:57 ` L A Walsh 2018-03-25 10:24 ` L A Walsh 2018-03-27 3:54 ` Jordan Geoghegan 2018-04-08 23:08 ` L A Walsh 2018-04-09 18:30 ` Jordan Geoghegan [not found] ` <d1353aaf-271a-ba2f-7de0-80ca15b18e01@cs.umass.edu> 2018-03-24 2:06 ` SSH/SFTP " Jordan Geoghegan 2018-03-24 5:07 ` Eliot Moss
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).