From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout10.t-online.de (mailout10.t-online.de [194.25.134.21]) by sourceware.org (Postfix) with ESMTPS id 6D4D93858D28 for ; Tue, 17 Jan 2023 15:21:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6D4D93858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=t-online.de Authentication-Results: sourceware.org; spf=none smtp.mailfrom=t-online.de Received: from fwd88.dcpf.telekom.de (fwd88.aul.t-online.de [10.223.144.114]) by mailout10.t-online.de (Postfix) with SMTP id BAB46161E7 for ; Tue, 17 Jan 2023 16:21:50 +0100 (CET) Received: from [192.168.2.101] ([87.187.47.96]) by fwd88.t-online.de with (TLSv1.3:TLS_AES_256_GCM_SHA384 encrypted) esmtp id 1pHnmY-1g4BYu0; Tue, 17 Jan 2023 16:21:50 +0100 Subject: Re: Question about slow access to file information To: cygwin References: <797a8935-e38b-0c0f-87d8-b8df1e9fd76f@cs.umass.edu> <0c9c111e-9e63-bf8c-8049-06fd23f66351@t-online.de> From: Christian Franke Message-ID: <08d1d7ae-32b7-45b3-595a-ae92eff9f8e1@t-online.de> Date: Tue, 17 Jan 2023 16:21:51 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 SeaMonkey/2.53.14 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TOI-EXPURGATEID: 150726::1673968910-736BB7D8-D99EF2BD/0/0 CLEAN NORMAL X-TOI-MSGID: 099f1691-f176-4445-91c5-67537fa662ae X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,FREEMAIL_FROM,KAM_DMARC_STATUS,KAM_LAZY_DOMAIN_SECURITY,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Eliot Moss via Cygwin wrote: > On 1/15/2023 3:38 AM, Christian Franke via Cygwin wrote: >> Eliot Moss via Cygwin wrote: >>> I have a separate drive mounted this way: >>> >>> d:/ /cygdrive/d ntfs binary,posix=0,user,noacl,auto 0 0 >>> >>> One thing I use it for is to store backup files.  These tend to be 2 Gb >>> chunks, and there can be hundreds of them in the backup directory. >>> (The drive >>> is 5Tb.)  The Windows Disk Management tool describes it as NTFS, >>> Basic Data >>> Partition. >>> >>> Doing ls (for example) takes a very perceptible numbers of seconds >>> (though >>> whatever takes a long time seems to be cached, at least for a while, >>> since a >>> second ls soon after is fast). >> >> The problem is the 'noacl' mount option and the fact that POSIX only >> offers the *stat*() functions to retrieve file information. These >> functions always need to provide the full file information, even if >> only a small subset is needed. >> >> To determine the 'x'-permission bits in the 'stat.st_mode' field on a >> 'noacl'-mount, Cygwin reads the first bytes of most files (all except >> *.exe, *.lnk, *.com). The 'x' bits are set if the file starts with >> "#!" (script), ":\n" (?) or "MZ" (Windows executable). >> >> On 'noacl' mounts, this behavior could be suppressed by 'exec' or >> 'noexec' mount options. > > Interesting.  I removed the noacl from /etc/fstab and restarted all > Cygwin processes. > The mount program now shows that drive without noacl.  It still takes > surprisingly > long to ls if I have not done so recently.  The directory contains > ~1200 files. This depends on storage device, sometimes (HDD) on filesystem fragmentation and always on 'ls' options. Plain '/bin/ls' without any arguments does not call stat(). 'ls -s' or 'ls --color=yes' call stat() for each file. 'ls -l' additionally calls getfacl() for each file if on an 'acl' mount. The latter is apparently slower than expected, see below. Here a quick test on a directory with 10000 ~3KB files on a NTFS USB drive connected via USB-2 (~28MB/s raw read speed). The first test of each mount variant was done immediately after connecting the drive: $ TIMEFORMAT='%R' 1. mount [-o acl] $ time ls -l > /dev/null 4.282 $ time ls -l > /dev/null 1.322 $ time ls -s > /dev/null 0.404 $ time ls > /dev/null 0.032 2. mount -o noacl $ time ls -l > /dev/null 13.452 $ time ls -l > /dev/null 0.789 $ time ls -s > /dev/null 0.764 $ time ls > /dev/null 0.033 3. mount -o noacl,noexec $ time ls -l > /dev/null 3.215 $ time ls -l > /dev/null 0.368 $ time ls -s > /dev/null 0.355 $ time ls > /dev/null 0.032 -- Regards, Christian