From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 57519 invoked by alias); 5 Jan 2020 21:22:49 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 57510 invoked by uid 89); 5 Jan 2020 21:22:49 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: Yes, score=5.5 required=5.0 tests=AWL,BAYES_50,FOREIGN_BODY,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=no version=3.3.1 spammy=videos, mit, minute, magnetic X-HELO: mail-pg1-f180.google.com Received: from mail-pg1-f180.google.com (HELO mail-pg1-f180.google.com) (209.85.215.180) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sun, 05 Jan 2020 21:22:48 +0000 Received: by mail-pg1-f180.google.com with SMTP id z124so25907479pgb.13 for ; Sun, 05 Jan 2020 13:22:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=/TLiu4Wr5EDkLdgKTu4CB1PvUlBxZrH9BA9KtC64wss=; b=CjEBXHZSOHDYBcjh/3uHYvC98SxM2Xi12MRUGrI2w/6Wl4589y2N5Z3iqzQgvWmFT8 Ggxvt14xNrzIdnMK2NZi/abi1lgeNBFY4RXD2v0nC5VZbsMUnS2jk4ArB6iNohqgw+ZQ +T6FCSItlREv9N3Wxqi1xAQLCgfqEsebcuvSK20wJaT2Hw0xlizVFaXSHhBkOt/W0XHc kxU0hj9PQQmEXUKZzuEFQje6kSXvcmJc0ISnE3ARl0EdIyItCpitBVNcfG9IZMekEz8f 0l3a2pxa+v4puMKbWxYeB78e1yeopkx6ql5eZUTqebdHgfOlJqRjbn2MXFCExWZgG+xw GmeA== MIME-Version: 1.0 References: <8582CD6F-C872-41FB-9425-2CBD1126AE33@plutonium24.de> In-Reply-To: <8582CD6F-C872-41FB-9425-2CBD1126AE33@plutonium24.de> From: Stephen John Smoogen Date: Sun, 05 Jan 2020 21:22:00 -0000 Message-ID: Subject: Re: rsync and ls -lR slow for directories with many files To: cygwin@cygwin.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes X-SW-Source: 2020-01/txt/msg00042.txt.bz2 On Sat, 4 Jan 2020 at 17:16, wrote: > > I am running rsync on a small linux server to synchronize files in one di= rectory and its subdirectories from Windows (using sshd from Cygwin) to thi= s server for backup purposes. The directory contains almost 1 TB of images = and videos in about 160k files on a slow disk (Seagate Archive 8TB with SMR= ) with NTFS. I am not sure if the Linux box has the slow disk or the Windows box has the slow disk. > Even if there are no changes and whith whole file transfers rsync takes a= bout 45 minutes to come to this conclusion. > I am using the following command line on the linux server: > > rsync -avx --stats --whole-file --no-perms --no-owner --no-group @<= server>: > > As rsync was only transferring a small number of bytes and gave no clue t= o the cause for being so slow and as rsync should only need filenames, date= s and sizes I did a "ls -lR|wc" on both systems. On the linux server this t= ook about 1 minute (only slightly faster magnetic disk, empty read cache at= start) and doing the same on cygwin took almost as long as rsync (over 40 = minutes). Using Windows Explorer (after a reboot to guarantee that the cach= e is empty) to get the total number of files and the total size took only a= few seconds. Reading all file sizes with Treesize also took less than one = minute. As ls -lR needs the same information I would have expected it to ta= ke the same time. I would add a bunch of verbose to the rsync to see what it is doing. (I don't recommend sending that to the list as it will be a lot of data.. but maybe an excerpt) I am expecting it is spending a lot of time getting the metadata off of one of the disks and mapping it to Unix permissions then comparing if those items are the same on the other side. Each one of those is going to be a separate action which on a slow drive may be a spinup/get-data/spindown cycle to make it even slower. I would then check to see if perms and metadata on that directory 'look sane' (this is highly dependent on your environment.. if you have an AD server giving out perms it will look different from other things.) If the lookups for mapping metadata permissions is having to ping an AD server or some sort of other network lookup that is going to also slow down things. Sorry I don't have any 'fixes'. I have always found large rsync between Windows and Unix to be slow. > Runnin "ls -lR" a second time on Cygwin is fast as lightning as it only t= akes less than 30s. > > Is there any way to get ls -lR or better rsync as fast as listing the dir= ectory with Windows tools? > > Frank > > -- > Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesend= et. > -- > Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesend= et. > -- > Problem reports: http://cygwin.com/problems.html > FAQ: http://cygwin.com/faq/ > Documentation: http://cygwin.com/docs.html > Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple > --=20 Stephen J Smoogen. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple