From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 127440 invoked by alias); 8 Jan 2020 16:43:20 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 127423 invoked by uid 89); 8 Jan 2020 16:43:20 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=3.3 required=5.0 tests=AWL,BAYES_00,FOREIGN_BODY,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=no version=3.3.1 spammy=magnetic, rsync, sshd, spending X-HELO: mout.gmx.net Received: from mout.gmx.net (HELO mout.gmx.net) (212.227.17.20) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 08 Jan 2020 16:43:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1578501794; bh=NAYNc7kotqJezu99YTDzgTEhDCXL26Oczxk6gUpbQq4=; h=X-UI-Sender-Class:Date:In-Reply-To:References:Subject:To:From; b=AjB189q/k4tD8DIgORHXgYpXEju2MTAQDZznVOP9dVJ/8l8VhYV56uz0WTVK8jiCd n0GnQjhbqcISECCD/feLAV9RA6Rtig1HdlCOE0tS+WljwsagkmNJffvNbKqc4bMlc+ Bq5FsM1NstqkfsPF90b//OHtHCDSuVwaFRA6dwtQ= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [10.0.8.1] ([109.42.0.68]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MVeMG-1jEd3m1t2K-00RVeR for ; Wed, 08 Jan 2020 17:43:14 +0100 Date: Wed, 08 Jan 2020 16:43:00 -0000 User-Agent: K-9 Mail for Android In-Reply-To: References: <8582CD6F-C872-41FB-9425-2CBD1126AE33@plutonium24.de> MIME-Version: 1.0 Subject: Re: rsync and ls -lR slow for directories with many files To: cygwin@cygwin.com From: Frank-Ulrich Sommer Message-ID: <30EE28DF-7A24-4665-9C62-A36C16F9D285@gmx.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SW-Source: 2020-01/txt/msg00079.txt.bz2 Thank you for the suggestions. The Linux disk is newer and does not use SMR= , but for reading there should not be a big difference. The windows box is running standalone without AD or anything similar. Conce= rning perms and metadata I must admit that I do not know what to look for. As I did not know how to capture output of rsync on the remote side I did s= trace the "ls -lR", but I could not detect anything that looked funny. I fe= ar if something creates more disk accesses than necessary it must be within= the system calls. Then I tried to disable as many features of rsync as possible and used the = following options: -avx --stats --whole-file --no-perms --no-owner --no-group --no-devices --no-specials --no-acls --no-xattrs but rsync did not get faster. I'm sorry to admit that the ultimate solution does not use Cygwin any more.= I'm now using a Windows share and connect to that share from my Linux serv= er with cifs and autofs. rsync then runs on the linux machine and accesses = that share (due to --whole-file this should not cause problems). rsync without any changed files directly after booting both systems (both c= aches are empty) now takes 91s instead of 42m. Am 5. Januar 2020 22:22:35 MEZ schrieb Stephen John Smoogen : >On Sat, 4 Jan 2020 at 17:16, wrote: >> >> I am running rsync on a small linux server to synchronize files in >one directory and its subdirectories from Windows (using sshd from >Cygwin) to this server for backup purposes. The directory contains >almost 1 TB of images and videos in about 160k files on a slow disk >(Seagate Archive 8TB with SMR) with NTFS. > >I am not sure if the Linux box has the slow disk or the Windows box >has the slow disk. > >> Even if there are no changes and whith whole file transfers rsync >takes about 45 minutes to come to this conclusion. >> I am using the following command line on the linux server: >> >> rsync -avx --stats --whole-file --no-perms --no-owner --no-group >@: >> >> As rsync was only transferring a small number of bytes and gave no >clue to the cause for being so slow and as rsync should only need >filenames, dates and sizes I did a "ls -lR|wc" on both systems. On the >linux server this took about 1 minute (only slightly faster magnetic >disk, empty read cache at start) and doing the same on cygwin took >almost as long as rsync (over 40 minutes). Using Windows Explorer >(after a reboot to guarantee that the cache is empty) to get the total >number of files and the total size took only a few seconds. Reading all >file sizes with Treesize also took less than one minute. As ls -lR >needs the same information I would have expected it to take the same >time. > > >I would add a bunch of verbose to the rsync to see what it is doing. >(I don't recommend sending that to the list as it will be a lot of >data.. but maybe an excerpt) I am expecting it is spending a lot of >time getting the metadata off of one of the disks and mapping it to >Unix permissions then comparing if those items are the same on the >other side. Each one of those is going to be a separate action which >on a slow drive may be a spinup/get-data/spindown cycle to make it >even slower. > >I would then check to see if perms and metadata on that directory >'look sane' (this is highly dependent on your environment.. if you >have an AD server giving out perms it will look different from other >things.) If the lookups for mapping metadata permissions is having to >ping an AD server or some sort of other network lookup that is going >to also slow down things. > >Sorry I don't have any 'fixes'. I have always found large rsync >between Windows and Unix to be slow. > >> Runnin "ls -lR" a second time on Cygwin is fast as lightning as it >only takes less than 30s. >> >> Is there any way to get ls -lR or better rsync as fast as listing the >directory with Windows tools? >> >> Frank >> >> -- >> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail >gesendet. >> -- >> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail >gesendet. >> -- >> Problem reports: http://cygwin.com/problems.html >> FAQ: http://cygwin.com/faq/ >> Documentation: http://cygwin.com/docs.html >> Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple >> > > >--=20 >Stephen J Smoogen. > >-- >Problem reports: http://cygwin.com/problems.html >FAQ: http://cygwin.com/faq/ >Documentation: http://cygwin.com/docs.html >Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple --=20 Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple