From: Christian Franke <Christian.Franke@t-online.de>
To: cygwin <cygwin@cygwin.com>
Subject: Re: Question about slow access to file information
Date: Tue, 17 Jan 2023 16:21:51 +0100 [thread overview]
Message-ID: <08d1d7ae-32b7-45b3-595a-ae92eff9f8e1@t-online.de> (raw)
In-Reply-To: <cbc3fce3-64a5-47af-0cf1-5ca55daf176a@cs.umass.edu>
Eliot Moss via Cygwin wrote:
> On 1/15/2023 3:38 AM, Christian Franke via Cygwin wrote:
>> Eliot Moss via Cygwin wrote:
>>> I have a separate drive mounted this way:
>>>
>>> d:/ /cygdrive/d ntfs binary,posix=0,user,noacl,auto 0 0
>>>
>>> One thing I use it for is to store backup files. These tend to be 2 Gb
>>> chunks, and there can be hundreds of them in the backup directory.
>>> (The drive
>>> is 5Tb.) The Windows Disk Management tool describes it as NTFS,
>>> Basic Data
>>> Partition.
>>>
>>> Doing ls (for example) takes a very perceptible numbers of seconds
>>> (though
>>> whatever takes a long time seems to be cached, at least for a while,
>>> since a
>>> second ls soon after is fast).
>>
>> The problem is the 'noacl' mount option and the fact that POSIX only
>> offers the *stat*() functions to retrieve file information. These
>> functions always need to provide the full file information, even if
>> only a small subset is needed.
>>
>> To determine the 'x'-permission bits in the 'stat.st_mode' field on a
>> 'noacl'-mount, Cygwin reads the first bytes of most files (all except
>> *.exe, *.lnk, *.com). The 'x' bits are set if the file starts with
>> "#!" (script), ":\n" (?) or "MZ" (Windows executable).
>>
>> On 'noacl' mounts, this behavior could be suppressed by 'exec' or
>> 'noexec' mount options.
>
> Interesting. I removed the noacl from /etc/fstab and restarted all
> Cygwin processes.
> The mount program now shows that drive without noacl. It still takes
> surprisingly
> long to ls if I have not done so recently. The directory contains
> ~1200 files.
This depends on storage device, sometimes (HDD) on filesystem
fragmentation and always on 'ls' options. Plain '/bin/ls' without any
arguments does not call stat(). 'ls -s' or 'ls --color=yes' call stat()
for each file. 'ls -l' additionally calls getfacl() for each file if on
an 'acl' mount. The latter is apparently slower than expected, see below.
Here a quick test on a directory with 10000 ~3KB files on a NTFS USB
drive connected via USB-2 (~28MB/s raw read speed). The first test of
each mount variant was done immediately after connecting the drive:
$ TIMEFORMAT='%R'
1. mount [-o acl]
$ time ls -l > /dev/null
4.282
$ time ls -l > /dev/null
1.322
$ time ls -s > /dev/null
0.404
$ time ls > /dev/null
0.032
2. mount -o noacl
$ time ls -l > /dev/null
13.452
$ time ls -l > /dev/null
0.789
$ time ls -s > /dev/null
0.764
$ time ls > /dev/null
0.033
3. mount -o noacl,noexec
$ time ls -l > /dev/null
3.215
$ time ls -l > /dev/null
0.368
$ time ls -s > /dev/null
0.355
$ time ls > /dev/null
0.032
--
Regards,
Christian
prev parent reply other threads:[~2023-01-17 15:21 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-14 0:42 Eliot Moss
2023-01-14 13:45 ` Adam Dinwoodie
2023-01-14 16:38 ` Christian Franke
2023-01-15 1:05 ` Eliot Moss
2023-01-15 3:24 ` gs-cygwin.com
2023-01-17 15:21 ` Christian Franke [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=08d1d7ae-32b7-45b3-595a-ae92eff9f8e1@t-online.de \
--to=christian.franke@t-online.de \
--cc=cygwin@cygwin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).