rsync and ls -lR slow for directories with many files

rsync and ls -lR slow for directories with many files

[muell]

I am running rsync on a small linux server to synchronize files in one directory and its subdirectories from Windows (using sshd from Cygwin) to this server for backup purposes. The directory contains almost 1 TB of images and videos in about 160k files on a slow disk (Seagate Archive 8TB with SMR) with NTFS.

Even if there are no changes and whith whole file transfers rsync takes about 45 minutes to come to this conclusion.
I am using the following command line on the linux server:

rsync -avx --stats --whole-file --no-perms --no-owner --no-group <user>@<server>:<source directory> <local destination directory>

As rsync was only transferring a small number of bytes and gave no clue to the cause for being so slow and as rsync should only need filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the linux server this took about 1 minute (only slightly faster magnetic disk, empty read cache at start) and doing the same on cygwin took almost as long as rsync (over 40 minutes). Using Windows Explorer (after a reboot to guarantee that the cache is empty) to get the total number of files and the total size took only a few seconds. Reading all file sizes with Treesize also took less than one minute. As ls -lR needs the same information I would have expected it to take the same time.

Runnin "ls -lR" a second time on Cygwin is fast as lightning as it only takes less than 30s.

Is there any way to get ls -lR or better rsync as fast as listing the directory with Windows tools?

Frank

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync and ls -lR slow for directories with many files

[Stephen John Smoogen]

On Sat, 4 Jan 2020 at 17:16, <muell@plutonium24.de> wrote:
>
> I am running rsync on a small linux server to synchronize files in one directory and its subdirectories from Windows (using sshd from Cygwin) to this server for backup purposes. The directory contains almost 1 TB of images and videos in about 160k files on a slow disk (Seagate Archive 8TB with SMR) with NTFS.

I am not sure if the Linux box has the slow disk or the Windows box
has the slow disk.

> Even if there are no changes and whith whole file transfers rsync takes about 45 minutes to come to this conclusion.
> I am using the following command line on the linux server:
>
> rsync -avx --stats --whole-file --no-perms --no-owner --no-group <user>@<server>:<source directory> <local destination directory>
>
> As rsync was only transferring a small number of bytes and gave no clue to the cause for being so slow and as rsync should only need filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the linux server this took about 1 minute (only slightly faster magnetic disk, empty read cache at start) and doing the same on cygwin took almost as long as rsync (over 40 minutes). Using Windows Explorer (after a reboot to guarantee that the cache is empty) to get the total number of files and the total size took only a few seconds. Reading all file sizes with Treesize also took less than one minute. As ls -lR needs the same information I would have expected it to take the same time.


I would add a bunch of verbose to the rsync to see what it is doing.
(I don't recommend sending that to the list as it will be a lot of
data.. but maybe an excerpt) I am expecting it is spending a lot of
time getting the metadata off of one of the disks and mapping it to
Unix permissions then comparing if those items are the same on the
other side. Each one of those is going to be a separate action which
on a slow drive may be a spinup/get-data/spindown cycle to make it
even slower.

I would then check to see if perms and metadata on that directory
'look sane' (this is highly dependent on your environment.. if you
have an AD server giving out perms it will look different from other
things.) If the lookups for mapping metadata permissions is having to
ping an AD server or some sort of other network lookup that is going
to also slow down things.

Sorry I don't have any 'fixes'. I have always found large rsync
between Windows and Unix to be slow.

> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it only takes less than 30s.
>
> Is there any way to get ls -lR or better rsync as fast as listing the directory with Windows tools?
>
> Frank
>
> --
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
> --
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>


-- 
Stephen J Smoogen.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync and ls -lR slow for directories with many files

[Frank-Ulrich Sommer]

Thank you for the suggestions. The Linux disk is newer and does not use SMR, but for reading there should not be a big difference.

The windows box is running standalone without AD or anything similar. Concerning perms and metadata I must admit that I do not know what to look for.

As I did not know how to capture output of rsync on the remote side I did strace the "ls -lR", but I could not detect anything that looked funny. I fear if something creates more disk accesses than necessary it must be within the system calls.

Then I tried to disable as many features of rsync as possible and used the following options:
-avx
--stats
--whole-file
--no-perms
--no-owner
--no-group
--no-devices
--no-specials
--no-acls
--no-xattrs

but rsync did not get faster.

I'm sorry to admit that the ultimate solution does not use Cygwin any more. I'm now using a Windows share and connect to that share from my Linux server with cifs and autofs. rsync then runs on the linux machine and accesses that share (due to --whole-file this should not cause problems).

rsync without any changed files directly after booting both systems (both caches are empty) now takes 91s instead of 42m.





Am 5. Januar 2020 22:22:35 MEZ schrieb Stephen John Smoogen <smooge@gmail.com>:
>On Sat, 4 Jan 2020 at 17:16, <muell@plutonium24.de> wrote:
>>
>> I am running rsync on a small linux server to synchronize files in
>one directory and its subdirectories from Windows (using sshd from
>Cygwin) to this server for backup purposes. The directory contains
>almost 1 TB of images and videos in about 160k files on a slow disk
>(Seagate Archive 8TB with SMR) with NTFS.
>
>I am not sure if the Linux box has the slow disk or the Windows box
>has the slow disk.
>
>> Even if there are no changes and whith whole file transfers rsync
>takes about 45 minutes to come to this conclusion.
>> I am using the following command line on the linux server:
>>
>> rsync -avx --stats --whole-file --no-perms --no-owner --no-group
><user>@<server>:<source directory> <local destination directory>
>>
>> As rsync was only transferring a small number of bytes and gave no
>clue to the cause for being so slow and as rsync should only need
>filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the
>linux server this took about 1 minute (only slightly faster magnetic
>disk, empty read cache at start) and doing the same on cygwin took
>almost as long as rsync (over 40 minutes). Using Windows Explorer
>(after a reboot to guarantee that the cache is empty) to get the total
>number of files and the total size took only a few seconds. Reading all
>file sizes with Treesize also took less than one minute. As ls -lR
>needs the same information I would have expected it to take the same
>time.
>
>
>I would add a bunch of verbose to the rsync to see what it is doing.
>(I don't recommend sending that to the list as it will be a lot of
>data.. but maybe an excerpt) I am expecting it is spending a lot of
>time getting the metadata off of one of the disks and mapping it to
>Unix permissions then comparing if those items are the same on the
>other side. Each one of those is going to be a separate action which
>on a slow drive may be a spinup/get-data/spindown cycle to make it
>even slower.
>
>I would then check to see if perms and metadata on that directory
>'look sane' (this is highly dependent on your environment.. if you
>have an AD server giving out perms it will look different from other
>things.) If the lookups for mapping metadata permissions is having to
>ping an AD server or some sort of other network lookup that is going
>to also slow down things.
>
>Sorry I don't have any 'fixes'. I have always found large rsync
>between Windows and Unix to be slow.
>
>> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it
>only takes less than 30s.
>>
>> Is there any way to get ls -lR or better rsync as fast as listing the
>directory with Windows tools?
>>
>> Frank
>>
>> --
>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>gesendet.
>> --
>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>gesendet.
>> --
>> Problem reports:       http://cygwin.com/problems.html
>> FAQ:                   http://cygwin.com/faq/
>> Documentation:         http://cygwin.com/docs.html
>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>
>
>
>-- 
>Stephen J Smoogen.
>
>--
>Problem reports:       http://cygwin.com/problems.html
>FAQ:                   http://cygwin.com/faq/
>Documentation:         http://cygwin.com/docs.html
>Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync and ls -lR slow for directories with many files

[L A Walsh]

On 2020/01/08 08:43, Frank-Ulrich Sommer wrote:
> but rsync did not get faster.
>
> I'm sorry to admit that the ultimate solution does not use Cygwin any more. I'm now using a Windows share and connect to that share from my Linux server with cifs and autofs. rsync then runs on the linux machine and accesses that share (due to --whole-file this should not cause problems).
>
> rsync without any changed files directly after booting both systems (both caches are empty) now takes 91s instead of 42m.
>   
---
    I won't go into the many reasons I know of (which are very 
incomplete compared
to those who actually created cygwin), but in situations like you describe,
I almost never use rsync unless I don't care about time and desperately 
need a
feature it has.  Instead I'll find it's faster to create an uncompressed tar
on windows, copy it to linux and expand it there to work locally with it.

    Any compression/signing/encrypting of data will slow down data 
transfer on
my home network where max CIFS transfer speeds are in the 300-700MB/s range.
>
>
>
>
> Am 5. Januar 2020 22:22:35 MEZ schrieb Stephen John Smoogen <smooge@gmail.com>:
>   
>> On Sat, 4 Jan 2020 at 17:16, <muell@plutonium24.de> wrote:
>>     
>>> I am running rsync on a small linux server to synchronize files in
>>>       
>> one directory and its subdirectories from Windows (using sshd from
>> Cygwin) to this server for backup purposes. The directory contains
>> almost 1 TB of images and videos in about 160k files on a slow disk
>> (Seagate Archive 8TB with SMR) with NTFS.
>>
>> I am not sure if the Linux box has the slow disk or the Windows box
>> has the slow disk.
>>
>>     
>>> Even if there are no changes and whith whole file transfers rsync
>>>       
>> takes about 45 minutes to come to this conclusion.
>>     
>>> I am using the following command line on the linux server:
>>>
>>> rsync -avx --stats --whole-file --no-perms --no-owner --no-group
>>>       
>> <user>@<server>:<source directory> <local destination directory>
>>     
>>> As rsync was only transferring a small number of bytes and gave no
>>>       
>> clue to the cause for being so slow and as rsync should only need
>> filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the
>> linux server this took about 1 minute (only slightly faster magnetic
>> disk, empty read cache at start) and doing the same on cygwin took
>> almost as long as rsync (over 40 minutes). Using Windows Explorer
>> (after a reboot to guarantee that the cache is empty) to get the total
>> number of files and the total size took only a few seconds. Reading all
>> file sizes with Treesize also took less than one minute. As ls -lR
>> needs the same information I would have expected it to take the same
>> time.
>>
>>
>> I would add a bunch of verbose to the rsync to see what it is doing.
>> (I don't recommend sending that to the list as it will be a lot of
>> data.. but maybe an excerpt) I am expecting it is spending a lot of
>> time getting the metadata off of one of the disks and mapping it to
>> Unix permissions then comparing if those items are the same on the
>> other side. Each one of those is going to be a separate action which
>> on a slow drive may be a spinup/get-data/spindown cycle to make it
>> even slower.
>>
>> I would then check to see if perms and metadata on that directory
>> 'look sane' (this is highly dependent on your environment.. if you
>> have an AD server giving out perms it will look different from other
>> things.) If the lookups for mapping metadata permissions is having to
>> ping an AD server or some sort of other network lookup that is going
>> to also slow down things.
>>
>> Sorry I don't have any 'fixes'. I have always found large rsync
>> between Windows and Unix to be slow.
>>
>>     
>>> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it
>>>       
>> only takes less than 30s.
>>     
>>> Is there any way to get ls -lR or better rsync as fast as listing the
>>>       
>> directory with Windows tools?
>>     
>>> Frank
>>>
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>       
>> gesendet.
>>     
>>> --
>>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>>>       
>> gesendet.
>>     
>>> --
>>> Problem reports:       http://cygwin.com/problems.html
>>> FAQ:                   http://cygwin.com/faq/
>>> Documentation:         http://cygwin.com/docs.html
>>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>>
>>>       
>> -- 
>> Stephen J Smoogen.
>>
>> --
>> Problem reports:       http://cygwin.com/problems.html
>> FAQ:                   http://cygwin.com/faq/
>> Documentation:         http://cygwin.com/docs.html
>> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>>     
>
>   

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple