bad performance when opening many files on Win7

bad performance when opening many files on Win7

[Achim Gratz]

I've seen major performance regressions on Win7 compared to an pre-1.7
Cygwin version on Win2000.  I had initially suspected the virus scanner
and disabling the real-time scan for the whole Cygwin folder did improve
things appreciably.  However the real performance hit when an
application opens or even just stats many files in succession (like Perl
is prone to at startup when it scans @INC) is still there.

Superficially it looks like you get 100% load on one CPU and the
application is busy, but perfmon tells you that the application sits on
its thumbs and the CPU is eaten by lsass.exe (which doesn't have any
file open or any network connection).  Eventually the application starts
doing what it is supposed to do, but lsass continues to eat CPU for a
few minutes.  I'm not even sure there's if it only happens for Cygwin
applications, but for those I can reliably reproduce it.  When I time
the application in Cygwin, I get something like less than a second user
time and over two minutes core time.

If someone has seen something like that and maybe there is some setup
that I can tweak to avoid this it would be great to know.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Larry Hall (Cygwin)]

On 4/26/2012 4:47 PM, Achim Gratz wrote:
>
> I've seen major performance regressions on Win7 compared to an pre-1.7
> Cygwin version on Win2000.  I had initially suspected the virus scanner
> and disabling the real-time scan for the whole Cygwin folder did improve
> things appreciably.  However the real performance hit when an
> application opens or even just stats many files in succession (like Perl
> is prone to at startup when it scans @INC) is still there.
>
> Superficially it looks like you get 100% load on one CPU and the
> application is busy, but perfmon tells you that the application sits on
> its thumbs and the CPU is eaten by lsass.exe (which doesn't have any
> file open or any network connection).  Eventually the application starts
> doing what it is supposed to do, but lsass continues to eat CPU for a
> few minutes.  I'm not even sure there's if it only happens for Cygwin
> applications, but for those I can reliably reproduce it.  When I time
> the application in Cygwin, I get something like less than a second user
> time and over two minutes core time.
>
> If someone has seen something like that and maybe there is some setup
> that I can tweak to avoid this it would be great to know.

Here's some background.

<http://www.suacommunity.com/forum/tm.aspx?m=1697&mpage=1>

Anything there describe your setup?


-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Larry Hall (Cygwin) writes:
> Here's some background.
>
> <http://www.suacommunity.com/forum/tm.aspx?m=1697&mpage=1>
>
> Anything there describe your setup?

The part about "groups with many members" looks oddly familiar.
Incidentally I've never had enough patience for makepasswd to finish
while scanning the AD... so, given that I don't think I could change all
that code, I guess that all the ids/groups used in the ACL should be in
passwd/groups?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Achim Gratz <Stromeko <at> nexgo.de> writes:
> The part about "groups with many members" looks oddly familiar.
> Incidentally I've never had enough patience for makepasswd to finish
> while scanning the AD... so, given that I don't think I could change all
> that code, I guess that all the ids/groups used in the ACL should be in
> passwd/groups?

Fetching all groups from the DC gave me some 60000 entries in a 4MiB group file.
 And if anything, things became even slower when using that file.  :-(

Regards,
Achim


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Larry Hall (Cygwin)]

On 4/27/2012 4:41 AM, Achim Gratz wrote:
> Achim Gratz<Stromeko<at>  nexgo.de>  writes:
>> The part about "groups with many members" looks oddly familiar.
>> Incidentally I've never had enough patience for makepasswd to finish
>> while scanning the AD... so, given that I don't think I could change all
>> that code, I guess that all the ids/groups used in the ACL should be in
>> passwd/groups?
>
> Fetching all groups from the DC gave me some 60000 entries in a 4MiB group file.
>   And if anything, things became even slower when using that file.  :-(

If you only care about the group for the current user, use the '-c' option
for 'mkgroup'.


-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Achim Gratz <Stromeko <at> NexGo.DE> writes:
> Fetching all groups from the DC gave me some 60000 entries in a 4MiB group
> file.  And if anything, things became even slower when using that file.  

I finally found some reproduceable case where the lsass.exe process will go
berserk.  It's all programs that scan all files in /usr/bin... there are 3800
and there seems to be no caching of any sort going on, so that always takes
about 20 seconds. :-(

I've scanned the ACL in that directory and these are the principals used:

BUILTIN\Administrators
BUILTIN\Administrators
BUILTIN\Users
BUILTIN\Users
AD\Domain Users
AD\Domain Users
NT AUTHORITY\SYSTEM
NT AUTHORITY\SYSTEM

I have these in /etc/passwd and/or /etc/group.  I'm probably missing out on:

Everyone
Everyone
NT AUTHORITY\Authenticated Users
NT AUTHORITY\Authenticated Users

How does one get suitable entries for /etc/group on these (I have
Authentificated Users:S-1-5-11:11: in /etc/group, but nothing that I could map
to "Everyone")?


Regards,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Achim Gratz <Stromeko <at> NexGo.DE> writes:
> How does one get suitable entries for /etc/group on these?

As per the description in the Cygwin UG "ntsec" chapter, I've created an entry
for "Everyone", which incidentally revealed what Cygwin associated with that
pesky "????????" user that was shown in ~-expansion.  Unfortunately, performance
did not improve.


Regards,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Corinna Vinschen]

On May  8 13:19, Achim Gratz wrote:
> Achim Gratz <Stromeko <at> NexGo.DE> writes:
> > Fetching all groups from the DC gave me some 60000 entries in a 4MiB group
> > file.  And if anything, things became even slower when using that file.  
> 
> I finally found some reproduceable case where the lsass.exe process will go
> berserk.  It's all programs that scan all files in /usr/bin... there are 3800
> and there seems to be no caching of any sort going on, so that always takes
> about 20 seconds. :-(
> 
> I've scanned the ACL in that directory and these are the principals used:
> 
> BUILTIN\Administrators
> BUILTIN\Administrators
> BUILTIN\Users
> BUILTIN\Users
> AD\Domain Users
> AD\Domain Users
> NT AUTHORITY\SYSTEM
> NT AUTHORITY\SYSTEM

How did you do that?  The ACLs in the Cygwin default dirs are supposed
to contain only three ACEs, one for the installing user, one for the
local admin group, and one for everyone.  At least that's how setup.exe
installs them.

> I have these in /etc/passwd and/or /etc/group.  I'm probably missing out on:
> 
> Everyone
> Everyone
> NT AUTHORITY\Authenticated Users
> NT AUTHORITY\Authenticated Users
> 
> How does one get suitable entries for /etc/group on these (I have
> Authentificated Users:S-1-5-11:11: in /etc/group, but nothing that I could map
> to "Everyone")?

You don't.  Everyone is the entry corresponding to POSIX "other".  It's
not actually a group, even if it has a SID.

Btw., my machine and my account are domain members as well.  Obviously
all entries in /usr/bin have an Everyone ACE.  My ls -l takes only about
1.2 secs with > 2200 entries in /usr/bin.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Corinna Vinschen writes:
> How did you do that?  The ACLs in the Cygwin default dirs are supposed
> to contain only three ACEs, one for the installing user, one for the
> local admin group, and one for everyone.  At least that's how setup.exe
> installs them.

These are inherited from the settings in C:\, AFAIK (the list is
extracted from the output of icacls for each file in /usr/bin).  I have
another test installation that may differ in minor details since it
hasn't been in use for many years, but in principle the output of icacls
looks the same for both.  I can remove the test installation and install
it completely fresh if there's hope to get this problem resolved.

> Btw., my machine and my account are domain members as well.  Obviously
> all entries in /usr/bin have an Everyone ACE.  My ls -l takes only about
> 1.2 secs with > 2200 entries in /usr/bin.

Running ls is considerably faster on my machine as well (but probably
not under 3 seconds, that machine has a slow disk and that it is almost
full doesn't really help, even with regular defragmentation).  One
application that shows the behviour I'm talking(*) about is pod2html,
and it finishes in almost no time when I simply remove "bin" from the
podpath argument.  If it makes a difference, I am running this as
administrator under Win7 Pro/32 (but presumably the same or a very
similar problem happens when a colleague of mine runs some scripts on
his machine as "normal" user).  The whole hierarchy that Cygwin is
installed into is excluded from being real-time-checked by the virus
scanner and that seems to be working as it gets another 2-3 times slower
if I remove that exclusion.

(*) lsass.exe consumes one of the two cores completely, there is
absolutely no network activity and not much disk activity except at the
very beginning (which I assume is where the stat is collected).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Corinna Vinschen]

On May  8 18:38, Achim Gratz wrote:
> Corinna Vinschen writes:
> > How did you do that?  The ACLs in the Cygwin default dirs are supposed
> > to contain only three ACEs, one for the installing user, one for the
> > local admin group, and one for everyone.  At least that's how setup.exe
> > installs them.
> 
> These are inherited from the settings in C:\, AFAIK (the list is
> extracted from the output of icacls for each file in /usr/bin).  I have
> another test installation that may differ in minor details since it
> hasn't been in use for many years, but in principle the output of icacls
> looks the same for both.  I can remove the test installation and install
> it completely fresh if there's hope to get this problem resolved.

No idea if that helps, actually I doubt it.  I was just wondering since
this is not how setup creates the permissions.

> > Btw., my machine and my account are domain members as well.  Obviously
> > all entries in /usr/bin have an Everyone ACE.  My ls -l takes only about
> > 1.2 secs with > 2200 entries in /usr/bin.
> 
> Running ls is considerably faster on my machine as well (but probably
> not under 3 seconds, that machine has a slow disk and that it is almost
> full doesn't really help, even with regular defragmentation).  One
> application that shows the behviour I'm talking(*) about is pod2html,
> and it finishes in almost no time when I simply remove "bin" from the
> podpath argument.

I don't know how to use pod2html.  Please send a simple usage example
which works out of the box, so that I can see if I can reproduce this
behaviour.

> If it makes a difference, I am running this as
> administrator under Win7 Pro/32 (but presumably the same or a very

Metoo, with UAC enabled.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> I don't know how to use pod2html.  Please send a simple usage example
> which works out of the box, so that I can see if I can reproduce this
> behaviour.

(1004)~ > time pod2html --flush --title=DateTime::Locale::ml \
--podpath=lib/perl5/5.14:bin:lib/perl5/site_perl/5.14 \
--infile=/usr/lib/perl5/5.14/version.pm \
--outfile=/dev/null --podroot=/usr --htmlroot=../../../.. \
--header --backlink=Back\ to\ Top --verbose
Flushing item and directory caches
Scanning for sections in input file(s)
No headings in /usr/lib/perl5/5.14/version.pm
scanning directories in pod-path
caching items for later use
caching directories for later use
Converting input file /usr/lib/perl5/5.14/version.pm
Finished
1.014u 3.743s 1:17.94 6.0%      0+0k 0+0io 5781pf+0w

The second run simply drops /usr/bin from the podpath:

(1005)~ > time pod2html --flush --title=DateTime::Locale::ml \
--podpath=lib/perl5/5.14:lib/perl5/site_perl/5.14 \
--infile=/usr/lib/perl5/5.14/version.pm \
--outfile=/dev/null --podroot=/usr --htmlroot=../../../.. \
--header --backlink=Back\ to\ Top --verbose
Flushing item and directory caches
Scanning for sections in input file(s)
No headings in /usr/lib/perl5/5.14/version.pm
scanning directories in pod-path
caching items for later use
caching directories for later use
Converting input file /usr/lib/perl5/5.14/version.pm
Finished
0.405u 2.074s 0:02.94 84.0%     0+0k 0+0io 5667pf+0w


HTH,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Corinna Vinschen]

On May  9 06:57, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > I don't know how to use pod2html.  Please send a simple usage example
> > which works out of the box, so that I can see if I can reproduce this
> > behaviour.
> 
> (1004)~ > time pod2html --flush --title=DateTime::Locale::ml \
> --podpath=lib/perl5/5.14:bin:lib/perl5/site_perl/5.14 \
> --infile=/usr/lib/perl5/5.14/version.pm \
> --outfile=/dev/null --podroot=/usr --htmlroot=../../../.. \
> --header --backlink=Back\ to\ Top --verbose
> Flushing item and directory caches
> Scanning for sections in input file(s)
> No headings in /usr/lib/perl5/5.14/version.pm
> scanning directories in pod-path
> caching items for later use
> caching directories for later use
> Converting input file /usr/lib/perl5/5.14/version.pm
> Finished
> 1.014u 3.743s 1:17.94 6.0%      0+0k 0+0io 5781pf+0w
> 
> The second run simply drops /usr/bin from the podpath:
> 
> (1005)~ > time pod2html --flush --title=DateTime::Locale::ml \
> --podpath=lib/perl5/5.14:lib/perl5/site_perl/5.14 \
> --infile=/usr/lib/perl5/5.14/version.pm \
> --outfile=/dev/null --podroot=/usr --htmlroot=../../../.. \
> --header --backlink=Back\ to\ Top --verbose
> Flushing item and directory caches
> Scanning for sections in input file(s)
> No headings in /usr/lib/perl5/5.14/version.pm
> scanning directories in pod-path
> caching items for later use
> caching directories for later use
> Converting input file /usr/lib/perl5/5.14/version.pm
> Finished
> 0.405u 2.074s 0:02.94 84.0%     0+0k 0+0io 5667pf+0w

The first run of the first command takes 15 secs on my machine.

Note especially that lsass takes 0 CPU all the time!

The second run of the first command takes 1.35 secs since caching kicked
in.  Subsequently running the second command takes about 0.8 secs.

There's something fishy on your machine, and it's not Cygwin, afaics.
I'd still blame some firewall/virus stuff.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Corinna Vinschen writes:
> There's something fishy on your machine, and it's not Cygwin, afaics.
> I'd still blame some firewall/virus stuff.

Thank you for looking into this.  I suspect the configuration of our
machines as well (since it is not just mine), since it only seems to
happen when executable files (as Windows sees them, dll and exe) are
involved — I suspect one of those group policies that IT bestows upon
us, although I can't rule out the virus scanner just yet.

I can't waste more time on this at the moment, so I've patched
Module::Build::Base to simply not send pod2html down that path and that
shortened the creation of one particularly POD-heavy perl module down
from almost a day to just over an hour.

I will have to revisit that problem again, but I have no idea how to
drill down to the core of it really...


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: bad performance when opening many files on Win7

[Achim Gratz]

Corinna Vinschen writes:
> There's something fishy on your machine, and it's not Cygwin, afaics.
> I'd still blame some firewall/virus stuff.

Just to put some closure on this: after the latest bout of security
patches distributed by our IT the issue has vanished completely.  I
suspect that it had been a group policy that was installed to check for
some known vulnerability before it got fixed via one of the patches.
Unfortunately I've not been able to find out what that was.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple