rsync performance degradation? Could be a windows issue?

rsync performance degradation? Could be a windows issue?

[Greg Freemyer]

All,

I'm not sure this is on-topic, but at least I'm in cygwin when I
notice the below:

I do a lot of large data transfers between USB drives.  Often I get
great speeds (70MB/sec or more).

Sometimes it falls way off to closer to 20MB/sec with the same class
of hardware.

I experienced the "slow" transfer speed today, so I thought I'd ask if
anyone knew a way to resolve it?

The PC I'm running on is spec'ed for data processing so a 6 core (12
thread) setup with 128GB or RAM.

No significant data processing is going on.  Basically the machine is
currently dedicated to running rsync (as below).

I have 2 USB drives hooked up.  Both are USB-3 and should be able to
do 70MB/sec with no issue.

Neither drive has any unrelated I/O going on that I know of.  (I don't
see any in the Windows Performance Monitor.)

I have a folder with almost 200 4GB files ( 750GB total and all the
files are large).

I'm using "rsync -av" to copy files from the source to the destination
and the folder on the destination started out empty

Using Windows "Resource Monitor" I normally would see both drives
hitting over 70 MB/sec.  Today I'm seeing the source drive jump all
the way up to 125 MB/sec or so, but the destination is down around 20
MB/sec

When the 4GB source file completes reading, the source drive basically
goes idle for a while until the file is written out.

Then the source drive jumps back to 125 MB/sec until it finishes
reading the next file.  The destination drive on the other hand is
stuck down around 20 MB/sec.

I really don't think this is caused by slow hardware.  Could the
destination drive be dropping down to USB-2 mode?

===
And now that I've written up the above, the system has returned to a
more normal 75 MB/sec consistently on the output drive.

I'm still curious what might be going on.  As I say I move a lot of
data around as part of my work, and much of the time it is big groups
of either 1.5GB or 4GB files.

Thanks
Greg
--
Greg Freemyer
www.IntelligentAvatar.net

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync performance degradation? Could be a windows issue?

[Greg Freemyer]

On Thu, Mar 31, 2016 at 4:48 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
> All,
>
> I'm not sure this is on-topic, but at least I'm in cygwin when I
> notice the below:
>
> I do a lot of large data transfers between USB drives.  Often I get
> great speeds (70MB/sec or more).
>
> Sometimes it falls way off to closer to 20MB/sec with the same class
> of hardware.
>
> I experienced the "slow" transfer speed today, so I thought I'd ask if
> anyone knew a way to resolve it?

I'm trying rsync of a bunch of large files again today.

Initially I saw the same slow copy speed (about 20 MB/sec).  My
destination was connected to a USB 3.1 port, but it seemed to be the
bottleneck so I moved it to a USB 3 port (3.1 should be 2x the speed
of 3.0, so it should not have been the bottleneck).

After getting my throughput up above 50 MB/sec I notice in the windows
"performance monitor" that my source disk is hitting 100% utilization,
then a few seconds later my destination disk is.  And back and forth.
It seems I'm only reading or writing for a few seconds, then
alternating.

I assume the issue is that too much data is being read / cached by
rsync prior to it being written out so I'm getting no advantage of
reading and writing in parallel.

(I gather iostat isn't available for cygwin?)

Is there some rsync (or cygwin) option / feature that would encourage
parallel reading/writing?

fyi: I did some linux testing with "dd" over the weekend and I hit 140
MB/sec if I used a 100MB blocksize.  That was to / from the raw disk
(/dev/sda => /dev/sdb).

If I bumped my blocksize to 1GB for dd in linux, my throughput dropped
to 70 MB/sec just as I see right now with rsync in cygwin.

Thanks
Greg

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync performance degradation? Could be a windows issue?

[Marco Atzeri]

On 11/04/2016 18:57, Greg Freemyer wrote:
> On Thu, Mar 31, 2016 at 4:48 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
>> All,
>>
>> I'm not sure this is on-topic, but at least I'm in cygwin when I
>> notice the below:
>>
>> I do a lot of large data transfers between USB drives.  Often I get
>> great speeds (70MB/sec or more).
>>
>> Sometimes it falls way off to closer to 20MB/sec with the same class
>> of hardware.
>>
>> I experienced the "slow" transfer speed today, so I thought I'd ask if
>> anyone knew a way to resolve it?
>
> I'm trying rsync of a bunch of large files again today.
>
> Initially I saw the same slow copy speed (about 20 MB/sec).  My
> destination was connected to a USB 3.1 port, but it seemed to be the
> bottleneck so I moved it to a USB 3 port (3.1 should be 2x the speed
> of 3.0, so it should not have been the bottleneck).
>
> After getting my throughput up above 50 MB/sec I notice in the windows
> "performance monitor" that my source disk is hitting 100% utilization,
> then a few seconds later my destination disk is.  And back and forth.
> It seems I'm only reading or writing for a few seconds, then
> alternating.
>
> I assume the issue is that too much data is being read / cached by
> rsync prior to it being written out so I'm getting no advantage of
> reading and writing in parallel.
>
> (I gather iostat isn't available for cygwin?)
>
> Is there some rsync (or cygwin) option / feature that would encourage
> parallel reading/writing?
>
> fyi: I did some linux testing with "dd" over the weekend and I hit 140
> MB/sec if I used a 100MB blocksize.  That was to / from the raw disk
> (/dev/sda => /dev/sdb).
>
> If I bumped my blocksize to 1GB for dd in linux, my throughput dropped
> to 70 MB/sec just as I see right now with rsync in cygwin.
>
> Thanks
> Greg
>


Is windows robocopy faster ?



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync performance degradation? Could be a windows issue?

[Greg Freemyer]

On Mon, Apr 11, 2016 at 1:34 PM, Marco Atzeri <marco.atzeri@gmail.com> wrote:
>
>
> On 11/04/2016 18:57, Greg Freemyer wrote:
>>
>> On Thu, Mar 31, 2016 at 4:48 PM, Greg Freemyer <greg.freemyer@gmail.com>
>> wrote:
>>>
>>> All,
>>>
>>> I'm not sure this is on-topic, but at least I'm in cygwin when I
>>> notice the below:
>>>
>>> I do a lot of large data transfers between USB drives.  Often I get
>>> great speeds (70MB/sec or more).
>>>
>>> Sometimes it falls way off to closer to 20MB/sec with the same class
>>> of hardware.
>>>
>>> I experienced the "slow" transfer speed today, so I thought I'd ask if
>>> anyone knew a way to resolve it?
>>
>>
>> I'm trying rsync of a bunch of large files again today.
>>
>> Initially I saw the same slow copy speed (about 20 MB/sec).  My
>> destination was connected to a USB 3.1 port, but it seemed to be the
>> bottleneck so I moved it to a USB 3 port (3.1 should be 2x the speed
>> of 3.0, so it should not have been the bottleneck).
>>
>> After getting my throughput up above 50 MB/sec I notice in the windows
>> "performance monitor" that my source disk is hitting 100% utilization,
>> then a few seconds later my destination disk is.  And back and forth.
>> It seems I'm only reading or writing for a few seconds, then
>> alternating.
>>
>> I assume the issue is that too much data is being read / cached by
>> rsync prior to it being written out so I'm getting no advantage of
>> reading and writing in parallel.
>>
>> (I gather iostat isn't available for cygwin?)
>>
>> Is there some rsync (or cygwin) option / feature that would encourage
>> parallel reading/writing?
>>
>> fyi: I did some linux testing with "dd" over the weekend and I hit 140
>> MB/sec if I used a 100MB blocksize.  That was to / from the raw disk
>> (/dev/sda => /dev/sdb).
>>
>> If I bumped my blocksize to 1GB for dd in linux, my throughput dropped
>> to 70 MB/sec just as I see right now with rsync in cygwin.
>>
>> Thanks
>> Greg
>>
>
>
> Is windows robocopy faster ?

As I test, I just copied 30 GB of 1.5 GB files via robocopy.  None of
the files had been accessed since a reboot, so none should have been
in cache.

According to Resource Monitor, yes.

I'm getting about 105 MB/sec for read and 105 MB/sec for write vs  70
MB/sec for rsync.

Significantly, with robocopy Resource Monitor is showing a consistent
read/write speed.  With rsync read and write fluctuate back and forth.

As I stated before, I believe I should be able to get 140 MB/sec, but
I wouldn't complain about 105 MB/sec.  70 MB/sec just seems to
highlight a flaw in how rsync manages the data flow since it is 50% of
theoretical max.

ie.  this seems to be what rsync in cygwin is doing:

while (files)  {
    read 1.5 GB file to ram
    write 1.5 GB file from ram
    fsync()  ensure 1.5 GB file is on disk
} endwhile

I haven't tested in linux.  Maybe rsync just isn't as efficient as I expect?

There is also the problem that rsync often slows down to below 50
MB/sec   I've tried it with no other activity on the PC.  Today I was
only getting 20 MB/sec at first.  (I moved the USB cable and it came
up to 70 MB/sec).

Thanks
Greg

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync performance degradation? Could be a windows issue?

[Greg Freemyer]

On Mon, Apr 11, 2016 at 2:20 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
> As I test, I just copied 30 GB of 1.5 GB files via robocopy.  None of
> the files had been accessed since a reboot, so none should have been
> in cache.
>
> According to Resource Monitor, yes.
>
> I'm getting about 105 MB/sec for read and 105 MB/sec for write vs  70
> MB/sec for rsync.

I just moved both drives to a Linux box.  Using the same rsync command
I'm getting 85 - 90 MB/sec.

That linux box is a 5-year old laptop, where as the windows box is a
pretty beefy box.  (6 cores, 64 GB ram)   Clearly the WIndows box
should be better performing and when running robocopy it is.

So it seems something about the combination of a Windows kernel,
cygwin, and rsync is dropping my performance at least a factor 20% and
maybe as much as 35%.

If I were to run rsync in strace, would anyone want to try and work on
the performance issue?  Or should it just be chalked up to rsync being
optimized for a linux kernel?

Thanks
Greg
--
Greg Freemyer
www.IntelligentAvatar.net

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync performance degradation? Could be a windows issue?

[Vincent Rivière]

On 31/03/2016 22:48, Greg Freemyer wrote:
> I experienced the "slow" transfer speed today,

I also experience similar slowness when using the current Cygwin rsync to 
backup my internal hard drives to external USB3 hard drive.

Because of that, I continue using the old official Cygwin rsync 3.0.8 
binary. I get correct speed with this one.

If that could matter, I use Windows XP and Cygwin 32.

-- 
Vincent Rivière

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple