From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 87278 invoked by alias); 30 Jan 2020 03:52:09 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 87267 invoked by uid 89); 30 Jan 2020 03:52:08 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00,FOREIGN_BODY,GIT_PATCH_2,SPF_PASS autolearn=no version=3.3.1 spammy=von, diese, signing, wurde X-HELO: Ishtar.sc.tlinx.org Received: from ishtar.tlinx.org (HELO Ishtar.sc.tlinx.org) (173.164.175.65) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 30 Jan 2020 03:52:06 +0000 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.sc.tlinx.org (8.14.7/8.14.4/SuSE Linux 0.8) with ESMTP id 00U3q20e043751; Wed, 29 Jan 2020 19:52:04 -0800 Message-ID: <5E3252E2.7070104@tlinx.org> Date: Thu, 30 Jan 2020 03:52:00 -0000 From: L A Walsh User-Agent: Thunderbird MIME-Version: 1.0 To: cygwin@cygwin.com, Frank-Ulrich Sommer Subject: Re: rsync and ls -lR slow for directories with many files References: <8582CD6F-C872-41FB-9425-2CBD1126AE33@plutonium24.de> <30EE28DF-7A24-4665-9C62-A36C16F9D285@gmx.net> In-Reply-To: <30EE28DF-7A24-4665-9C62-A36C16F9D285@gmx.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2020-01/txt/msg00306.txt.bz2 On 2020/01/08 08:43, Frank-Ulrich Sommer wrote: > but rsync did not get faster. > > I'm sorry to admit that the ultimate solution does not use Cygwin any more. I'm now using a Windows share and connect to that share from my Linux server with cifs and autofs. rsync then runs on the linux machine and accesses that share (due to --whole-file this should not cause problems). > > rsync without any changed files directly after booting both systems (both caches are empty) now takes 91s instead of 42m. > --- I won't go into the many reasons I know of (which are very incomplete compared to those who actually created cygwin), but in situations like you describe, I almost never use rsync unless I don't care about time and desperately need a feature it has. Instead I'll find it's faster to create an uncompressed tar on windows, copy it to linux and expand it there to work locally with it. Any compression/signing/encrypting of data will slow down data transfer on my home network where max CIFS transfer speeds are in the 300-700MB/s range. > > > > > Am 5. Januar 2020 22:22:35 MEZ schrieb Stephen John Smoogen : > >> On Sat, 4 Jan 2020 at 17:16, wrote: >> >>> I am running rsync on a small linux server to synchronize files in >>> >> one directory and its subdirectories from Windows (using sshd from >> Cygwin) to this server for backup purposes. The directory contains >> almost 1 TB of images and videos in about 160k files on a slow disk >> (Seagate Archive 8TB with SMR) with NTFS. >> >> I am not sure if the Linux box has the slow disk or the Windows box >> has the slow disk. >> >> >>> Even if there are no changes and whith whole file transfers rsync >>> >> takes about 45 minutes to come to this conclusion. >> >>> I am using the following command line on the linux server: >>> >>> rsync -avx --stats --whole-file --no-perms --no-owner --no-group >>> >> @: >> >>> As rsync was only transferring a small number of bytes and gave no >>> >> clue to the cause for being so slow and as rsync should only need >> filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the >> linux server this took about 1 minute (only slightly faster magnetic >> disk, empty read cache at start) and doing the same on cygwin took >> almost as long as rsync (over 40 minutes). Using Windows Explorer >> (after a reboot to guarantee that the cache is empty) to get the total >> number of files and the total size took only a few seconds. Reading all >> file sizes with Treesize also took less than one minute. As ls -lR >> needs the same information I would have expected it to take the same >> time. >> >> >> I would add a bunch of verbose to the rsync to see what it is doing. >> (I don't recommend sending that to the list as it will be a lot of >> data.. but maybe an excerpt) I am expecting it is spending a lot of >> time getting the metadata off of one of the disks and mapping it to >> Unix permissions then comparing if those items are the same on the >> other side. Each one of those is going to be a separate action which >> on a slow drive may be a spinup/get-data/spindown cycle to make it >> even slower. >> >> I would then check to see if perms and metadata on that directory >> 'look sane' (this is highly dependent on your environment.. if you >> have an AD server giving out perms it will look different from other >> things.) If the lookups for mapping metadata permissions is having to >> ping an AD server or some sort of other network lookup that is going >> to also slow down things. >> >> Sorry I don't have any 'fixes'. I have always found large rsync >> between Windows and Unix to be slow. >> >> >>> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it >>> >> only takes less than 30s. >> >>> Is there any way to get ls -lR or better rsync as fast as listing the >>> >> directory with Windows tools? >> >>> Frank >>> >>> -- >>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail >>> >> gesendet. >> >>> -- >>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail >>> >> gesendet. >> >>> -- >>> Problem reports: http://cygwin.com/problems.html >>> FAQ: http://cygwin.com/faq/ >>> Documentation: http://cygwin.com/docs.html >>> Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple >>> >>> >> -- >> Stephen J Smoogen. >> >> -- >> Problem reports: http://cygwin.com/problems.html >> FAQ: http://cygwin.com/faq/ >> Documentation: http://cygwin.com/docs.html >> Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple >> > > -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple