Faster rsync?

Faster rsync?

[Adam Kessel]

I've found rsync to be painfully slow on large folders -- hours to sync 
thousands of files, even when they already match size and --size-only is 
used. It's much faster between native Linux boxes. Is there any trick to 
improving performance? I've been looking for a native version for 
Windows, but it doesn't appear to exist -- as best I can tell, 
https://acrosync.com/windows.html is just the cygwin version packaged up.

 From past experience, WSL rsync is much faster, but I don't want to 
install WSL on this box.

Any ideas?

Re: Faster rsync?

[Mario Emmenlauer]

On 29.08.23 14:32, Adam Kessel via Cygwin wrote:
> I've found rsync to be painfully slow on large folders -- hours to sync thousands of files, even when they already match size and --size-only is used. It's much 
> faster between native Linux boxes.

I've been using rsync, unison and similar tools on Windows and Linux since basically forever. In my humble opinion, the problem is the Windows file system 
performance, not the synchronization tools. As a separate example, try to download the boost source code, and extract the archive. I can do the extraction in 
way under a minute on Linux, but have to wait many many minutes on a similarly equipped Windows machine.

Just my two cents.

     Mario

Re: Faster rsync?

[Eliot Moss]

On 8/29/2023 9:17 AM, Mario Emmenlauer via Cygwin wrote:
> On 29.08.23 14:32, Adam Kessel via Cygwin wrote:
>> I've found rsync to be painfully slow on large folders -- hours to sync thousands of files, even 
>> when they already match size and --size-only is used. It's much faster between native Linux boxes.

I've been told (on this list) that how you mount the Windows drive in Cygwin matters.

I use this in my /etc/fstab:

d:/ /cygdrive/d ntfs binary,posix=0,user,auto,notexec 0 0

Unfortunately, it's still not as quick as I think it ought to be.
(If someone out there can recommend better settings, I'm open!)

I regularly rsync a collection of hundreds of files, each about 2Gb in size.

Windows describes the partition as an NTFS Basic Data Partition.  It takes
rsync a long time to get started (i.e., to figure out which files need
transferring).

My rsync uses these flags:

rsync -avuP --partial --partial-dir=.rsync-partial --log-file=... --log-file-format=...

> I've been using rsync, unison and similar tools on Windows and Linux since basically forever. In my 
> humble opinion, the problem is the Windows file system performance, not the synchronization tools. 
> As a separate example, try to download the boost source code, and extract the archive. I can do the 
> extraction in way under a minute on Linux, but have to wait many many minutes on a similarly 
> equipped Windows machine.
> 
> Just my two cents.
> 
>      Mario
> 

Regards - Eliot Moss

Re: Faster rsync?

[Eliot Moss]

On 8/29/2023 9:43 AM, Eliot Moss wrote:
> On 8/29/2023 9:17 AM, Mario Emmenlauer via Cygwin wrote:
>> On 29.08.23 14:32, Adam Kessel via Cygwin wrote:
>>> I've found rsync to be painfully slow on large folders -- hours to sync thousands of files, even 
>>> when they already match size and --size-only is used. It's much faster between native Linux boxes.
> 
> I've been told (on this list) that how you mount the Windows drive in Cygwin matters.
> 
> I use this in my /etc/fstab:
> 
> d:/ /cygdrive/d ntfs binary,posix=0,user,auto,notexec 0 0
> 
> Unfortunately, it's still not as quick as I think it ought to be.
> (If someone out there can recommend better settings, I'm open!)
> 
> I regularly rsync a collection of hundreds of files, each about 2Gb in size.
> 
> Windows describes the partition as an NTFS Basic Data Partition.  It takes
> rsync a long time to get started (i.e., to figure out which files need
> transferring).
> 
> My rsync uses these flags:
> 
> rsync -avuP --partial --partial-dir=.rsync-partial --log-file=... --log-file-format=...

These flags seems to give speedy directory scanning:

rsync -r --chmod=D2700,F600 -t -z -p -v --size-only --partial-dir=.rsync-partial

Regards - EM