Performance of "ls -F"

Performance of "ls -F"

[Achim Gratz]

I am finding a large performance gap between plain "ls" and "ls -F" in a
directory with many files on a network share (NetApp disguised as NTFS if
that matters).  This has been there for quite a while, I've just now
realized what the reason was (I have "ls -F" as an alias for "ls" in my
interactive shells).  In a directory with 1300 files, a plain "ls" completes
in 0.3s, while "ls -F" requires about 95s.  Determining the file class seems
to require around 70...90ms per file, which I can confirm also for
directories with a lot less files.  What's involved in that determination
that takes such a long time?

Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[William M. (Mike) Miller]

On Thu, Jan 21, 2016 at 10:44 AM, Achim Gratz <Stromeko@nexgo.de> wrote:
> I am finding a large performance gap between plain "ls" and "ls -F" in a
> directory with many files on a network share (NetApp disguised as NTFS if
> that matters).  This has been there for quite a while, I've just now
> realized what the reason was (I have "ls -F" as an alias for "ls" in my
> interactive shells).  In a directory with 1300 files, a plain "ls" completes
> in 0.3s, while "ls -F" requires about 95s.  Determining the file class seems
> to require around 70...90ms per file, which I can confirm also for
> directories with a lot less files.  What's involved in that determination
> that takes such a long time?

The overhead appears to be in checking for executable files; using
--file-type instead of -F, which just omits the '*' category, reduces
the time for ls in one of my (local) large directories from over one
second to 0.04 seconds.

-- 
William M. (Mike) Miller | Edison Design Group
william.m.miller@gmail.com

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[Achim Gratz]

William M. (Mike) Miller <william.m.miller <at> gmail.com> writes:
> The overhead appears to be in checking for executable files; using
> --file-type instead of -F, which just omits the '*' category, reduces
> the time for ls in one of my (local) large directories from over one
> second to 0.04 seconds.

Indeed.  Thanks for letting me know!


Regards,
Achim.





--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Performance of "ls -F"

[Bill Smith]

Hi Achim,
I'm also having this issue but my investigation has found it to be a behavior specific to C-DOT.  This doesn't happen with 7mode.  I currently have a support case open with NetApp to get to the bottom of this behavior.  It could be a Cygwin bug.

> -----Original Message-----
> From: cygwin-owner@cygwin.com [mailto:cygwin-owner@cygwin.com] On
> Behalf Of Achim Gratz
> Sent: Thursday, January 21, 2016 10:45 AM
> To: cygwin@cygwin.com
> Subject: Performance of "ls -F"
> 
> I am finding a large performance gap between plain "ls" and "ls -F" in a
> directory with many files on a network share (NetApp disguised as NTFS if
> that matters).  This has been there for quite a while, I've just now realized
> what the reason was (I have "ls -F" as an alias for "ls" in my interactive shells).
> In a directory with 1300 files, a plain "ls" completes in 0.3s, while "ls -F"
> requires about 95s.  Determining the file class seems to require around
> 70...90ms per file, which I can confirm also for directories with a lot less files.
> What's involved in that determination that takes such a long time?
> 
> Regards,
> Achim.
> 
> 
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Performance of "ls -F"

[Bill Smith]

In my particular case, we're seeing this behavior:

7-mode: (//devnas04/largedisk/bsmith/netapp)

:$ //devnas04/largedisk/bsmith/netapp>time ls -ld struct5*
-rw-r--r-- 1 bsmith Domain Users 0 Nov  5 10:25 struct51.log
[snipped]
-rw-r--r-- 1 bsmith Domain Users 0 Nov  5 10:26 struct5z.prf

real    0m1.308s
user    0m0.031s
sys     0m0.125s


cdot-mode: (//rdlserv/testdata/rdl117_nt/test)

:$ //rdlserv/testdata/rdl117_nt/test>time ls -ld struct5*
-rwxrwx---+ 1 Unknown+User Unknown+Group 23047 Nov  4 21:47 struct51.log
[snipped]
-rwxr-x---+ 1 Unknown+User Unknown+Group   595 Oct 31 23:53 struct5z.prf

real    1m7.698s
user    0m0.249s
sys     0m11.484s

The difference is 1.3 seconds versus 1 minute 7 seconds.  The directory is identical on the two NetApps and they both contain ~29K files.  C-dot (Cluster Data On Tap) is the newest operating system for the NetApp.  It also supports the newer SMB protocols.

I also tried the experiment with MKS Toolkit 8.6 and in both cases, it takes around .1 seconds.



> -----Original Message-----
> From: cygwin-owner@cygwin.com [mailto:cygwin-owner@cygwin.com] On
> Behalf Of William M. (Mike) Miller
> Sent: Thursday, January 21, 2016 10:53 AM
> To: cygwin@cygwin.com
> Subject: Re: Performance of "ls -F"
> 
> On Thu, Jan 21, 2016 at 10:44 AM, Achim Gratz <Stromeko@nexgo.de>
> wrote:
> > I am finding a large performance gap between plain "ls" and "ls -F" in
> > a directory with many files on a network share (NetApp disguised as
> > NTFS if that matters).  This has been there for quite a while, I've
> > just now realized what the reason was (I have "ls -F" as an alias for
> > "ls" in my interactive shells).  In a directory with 1300 files, a
> > plain "ls" completes in 0.3s, while "ls -F" requires about 95s.
> > Determining the file class seems to require around 70...90ms per file,
> > which I can confirm also for directories with a lot less files.
> > What's involved in that determination that takes such a long time?
> 
> The overhead appears to be in checking for executable files; using --file-type
> instead of -F, which just omits the '*' category, reduces the time for ls in one
> of my (local) large directories from over one second to 0.04 seconds.
> 
> --
> William M. (Mike) Miller | Edison Design Group william.m.miller@gmail.com
> 
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[Achim Gratz]

Bill Smith writes:
> Hi Achim,
> I'm also having this issue but my investigation has found it to be a
> behavior specific to C-DOT.  This doesn't happen with 7mode.  I
> currently have a support case open with NetApp to get to the bottom of
> this behavior.  It could be a Cygwin bug.

Hmm.  I have no idea what C-DOT and 7mode are or how to check for that.
But since you are able to open support cases with NetApp, any chance you
can ask them for a fix of those unstable inode numbers?  :-)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[Achim Gratz]

Bill Smith <bsmith <at> progress.com> writes:
> The difference is 1.3 seconds versus 1 minute 7 seconds.  The directory is
identical on the two NetApps and
> they both contain ~29K files.  C-dot (Cluster Data On Tap) is the newest
operating system for the NetApp.  It
> also supports the newer SMB protocols.
> 
> I also tried the experiment with MKS Toolkit 8.6 and in both cases, it
takes around .1 seconds.

Could you please show the result for running /usr/lib/csih/getVolInfo on the
two directories?


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On Jan 22 07:30, Achim Gratz wrote:
> Bill Smith <bsmith <at> progress.com> writes:
> > The difference is 1.3 seconds versus 1 minute 7 seconds.  The directory is
> identical on the two NetApps and
> > they both contain ~29K files.  C-dot (Cluster Data On Tap) is the newest
> operating system for the NetApp.  It
> > also supports the newer SMB protocols.
> > 
> > I also tried the experiment with MKS Toolkit 8.6 and in both cases, it
> takes around .1 seconds.
> 
> Could you please show the result for running /usr/lib/csih/getVolInfo on the
> two directories?

Just a hint:  ls -F requires to stat every single file.  stat in turn
requires to load not only the usual metadata but also to fetch the ACL
and convert it to POSIX permissions.  If the ACL hasn't been created
with Cygwin 2.4.0, an additional number of AuthZ calls might be
performed with the number of calls == the number of different user
accounts in the ACLs of all files.  Which means, there are quite a few
packages going over the network.  Assuming this slow access only occurs
under 2.4.0, this might be one of the reasons to allow the user to
disable Authz processing (you have been warned, yada yada).  If that's
*not* the problem, we certainly need more details.  Like debugging
results to find out where the time is spent.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Performance of "ls -F"

[Achim Gratz]

Corinna Vinschen writes:
> Just a hint:  ls -F requires to stat every single file.  stat in turn
> requires to load not only the usual metadata but also to fetch the ACL
> and convert it to POSIX permissions.

The timings are from shares mounted with the noacl option, so that bit
of code shouldn't be involved.  I will try the same operations on an ACL
enabled mount of the same share later and I also need to find a share
that is exported from a true NTFS server for comparison.

> Assuming this slow access only occurs under 2.4.0,

No, the behaviour is quite a bit older, although I didn't time it since
I didn't recognize the connection to the aliased ls.  In any case, it
really is just the determination of whether or not the file is
executable that takes up all that time.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1497 bytes --]

On Jan 22 22:04, Achim Gratz wrote:
> Corinna Vinschen writes:
> > Just a hint:  ls -F requires to stat every single file.  stat in turn
> > requires to load not only the usual metadata but also to fetch the ACL
> > and convert it to POSIX permissions.
> 
> The timings are from shares mounted with the noacl option, so that bit
> of code shouldn't be involved.  I will try the same operations on an ACL
> enabled mount of the same share later and I also need to find a share
> that is exported from a true NTFS server for comparison.
> 
> > Assuming this slow access only occurs under 2.4.0,
> 
> No, the behaviour is quite a bit older, although I didn't time it since
> I didn't recognize the connection to the aliased ls.  In any case, it
> really is just the determination of whether or not the file is
> executable that takes up all that time.

In the noacl case, Cygwin tries to find out if files are scripts.  It
opens the file and checks the first two bytes in the file for a shebang
(and other stuff).  This may take a lot of time, more so on network
drives.  Can you try adding the "notexec" mount option to the "noacl"
share and see if that helps?

This test is done for a looong time to accommodate FAT filesystems in
the first place.  It might be prudent to disable it by default these
days...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Performance of "ls -F"

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> In the noacl case, Cygwin tries to find out if files are scripts.  It
> opens the file and checks the first two bytes in the file for a shebang
> (and other stuff).

acl > time ls --file-type > /dev/null
0.015u 0.015s 0:00.42 4.7%      0+0k 0+0io 2473pf+0w
acl > time ls -F > /dev/null
0.078u 0.858s 1:10.69 1.3%      0+0k 0+0io 5243pf+0w

noacl > time ls --file-type > /dev/null
0.015u 0.015s 0:00.37 5.4%      0+0k 0+0io 2391pf+0w
noacl > time ls -F > /dev/null
0.093u 1.327s 1:38.90 1.4%      0+0k 0+0io 6309pf+0w

>  This may take a lot of time, more so on network
> drives.  Can you try adding the "notexec" mount option to the "noacl"
> share and see if that helps?

acl,notexec > time ls --file-type > /dev/null
0.015u 0.030s 0:00.41 9.7%      0+0k 0+0io 2471pf+0w
acl,noexec > time ls -F > /dev/null
0.062u 0.811s 1:10.31 1.2%      0+0k 0+0io 5240pf+0w

noacl,notexec > time ls --file-type > /dev/null
0.031u 0.030s 0:00.41 14.6%     0+0k 0+0io 2389pf+0w
noacl,notexec > time ls -F > /dev/null
0.046u 0.718s 0:56.23 1.3%      0+0k 0+0io 4994pf+0w

> This test is done for a looong time to accommodate FAT filesystems in
> the first place.  It might be prudent to disable it by default these
> days...

Looks like that's not the main reason for the extra time spent.


Here's another NetApp share, but this time there are about half as many
files with only two of them in each sub-directory.

(1046)/mnt/upload/install > time ls --file-type x86*/patches/*/* > /dev/null
0.155u 1.358s 0:09.42 15.9%     0+0k 0+0io 10555pf+0w
(1047)/mnt/upload/install > time ls -F x86*/patches/*/* > /dev/null
0.109u 1.046s 0:08.20 13.9%     0+0k 0+0io 9817pf+0w

Somehow that takes a lot less time and there's no difference between the two
invocations (or actually a bit less time for -F).  The getVolInfo helper
sees these two shares with the same settings.  Not sure what to make of that...


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Performance of "ls -F"

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2758 bytes --]

On Jan 25 11:02, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > In the noacl case, Cygwin tries to find out if files are scripts.  It
> > opens the file and checks the first two bytes in the file for a shebang
> > (and other stuff).
> 
> acl > time ls --file-type > /dev/null
> 0.015u 0.015s 0:00.42 4.7%      0+0k 0+0io 2473pf+0w
> acl > time ls -F > /dev/null
> 0.078u 0.858s 1:10.69 1.3%      0+0k 0+0io 5243pf+0w
> 
> noacl > time ls --file-type > /dev/null
> 0.015u 0.015s 0:00.37 5.4%      0+0k 0+0io 2391pf+0w
> noacl > time ls -F > /dev/null
> 0.093u 1.327s 1:38.90 1.4%      0+0k 0+0io 6309pf+0w
> 
> >  This may take a lot of time, more so on network
> > drives.  Can you try adding the "notexec" mount option to the "noacl"
> > share and see if that helps?
> 
> acl,notexec > time ls --file-type > /dev/null
> 0.015u 0.030s 0:00.41 9.7%      0+0k 0+0io 2471pf+0w
> acl,noexec > time ls -F > /dev/null
> 0.062u 0.811s 1:10.31 1.2%      0+0k 0+0io 5240pf+0w
> 
> noacl,notexec > time ls --file-type > /dev/null
> 0.031u 0.030s 0:00.41 14.6%     0+0k 0+0io 2389pf+0w
> noacl,notexec > time ls -F > /dev/null
> 0.046u 0.718s 0:56.23 1.3%      0+0k 0+0io 4994pf+0w
> 
> > This test is done for a looong time to accommodate FAT filesystems in
> > the first place.  It might be prudent to disable it by default these
> > days...
> 
> Looks like that's not the main reason for the extra time spent.

Off the top of my head, the common denominator seems to be that on
directories returning d_type information, --file-type doesn't have to
call stat().  That makes it very fast since only a directory enumeration
is done.  Calling stat on the other hand is time consuming since on
Windows it requires to open the file and read meta information (including
the ACL) or data (noacl exe recognition) from it.

> Here's another NetApp share, but this time there are about half as many
> files with only two of them in each sub-directory.
> 
> (1046)/mnt/upload/install > time ls --file-type x86*/patches/*/* > /dev/null
> 0.155u 1.358s 0:09.42 15.9%     0+0k 0+0io 10555pf+0w
> (1047)/mnt/upload/install > time ls -F x86*/patches/*/* > /dev/null
> 0.109u 1.046s 0:08.20 13.9%     0+0k 0+0io 9817pf+0w
> 
> Somehow that takes a lot less time and there's no difference between the two
> invocations (or actually a bit less time for -F).  The getVolInfo helper
> sees these two shares with the same settings.  Not sure what to make of that...

Hmm, no, sorry, I have no idea.  Somebody will have to debug this.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Performance of "ls -F"

[Lemke, Michael ST/HZA-ZIC2]

I just want to add an observation and stay out of this
conversation otherwise: ls -F is comparably slow on a
network drive here but as noted by someone else in this
thread ls --file-type is fast.  However, the combination
ls --file-type --color is as slow as ls -F here.  Maybe
a hint for you to find out what is going on.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple