last snapshot is not better than previous version

last snapshot is not better than previous version

[Denis Excoffier]

Hello,

I tried the snapshot dated 2016-05-20. Okay, this one was not announced.

Functionally it seems ok. But some simple tasks now take a lot of time compared to the last non-snapshot version (2.5.1):

1) searching into a small text file
A simple text file (created with "seq 10000") takes several seconds to load in vim. Lookup for the string "9999" then takes 5 seconds. Previously, searching "9999999" in a bigger file (created with "seq 10000000") used to take 2 seconds only.

Note: The "seq 10000000 > myfile" itself makes no notable difference.

2) ls in a large directory
My /usr/bin contains 8098 files. Previously, a plain "ls -F" in this folder took some initialization time (2 seconds), then the display of the files was quite instantaneous. With the new snapshot, the initialization (stat and sort) takes more or less the same time, but afterwards, each single line is displayed at a visible pace, a clear screen is executed after each set of ~24 lines and the whole directory is listed in about 10 minutes.

Note: "ls | wc -l" itself makes no notable difference.

This occurs under X-Windows (xterm) or not (login window). Under x86, Windows 7. I switched back to 2.5.1.

This message just in case it could be useful.

Regards,

Denis Excoffier.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: last snapshot is not better than previous version

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2405 bytes --]

Hi Denis,

On May 26 19:21, Denis Excoffier wrote:
> Hello,
> 
> I tried the snapshot dated 2016-05-20. Okay, this one was not announced.
> 
> Functionally it seems ok. But some simple tasks now take a lot of time compared to the last non-snapshot version (2.5.1):
> 
> 1) searching into a small text file
> A simple text file (created with "seq 10000") takes several seconds to load in vim. Lookup for the string "9999" then takes 5 seconds. Previously, searching "9999999" in a bigger file (created with "seq 10000000") used to take 2 seconds only.

I can't reproduce this.

> Note: The "seq 10000000 > myfile" itself makes no notable difference.
> 
> 2) ls in a large directory
> My /usr/bin contains 8098 files. Previously, a plain "ls -F" in this folder took some initialization time (2 seconds), then the display of the files was quite instantaneous. With the new snapshot, the initialization (stat and sort) takes more or less the same time, but afterwards, each single line is displayed at a visible pace, a clear screen is executed after each set of ~24 lines and the whole directory is listed in about 10 minutes.

I can't reproduce this either.

> Note: "ls | wc -l" itself makes no notable difference.
> 
> This occurs under X-Windows (xterm) or not (login window). Under x86, Windows 7. I switched back to 2.5.1.
> 
> This message just in case it could be useful.

The problem is, there's no change at all in the snapshot which would
explain a slowdown of such basic functions:

- Assorted select(2) improvements:

  commits a23e6a35d896a075640db714b28ce74bb6b8d7ff
          e5665d8c930485d5ac6d8913573e27b9e5043d92

- There's a change to the sched_* functions, which is used only rarely
  and which doesn't introduce any waits or loops.

  commits 450f557feee5dd4900e3fd4f16cbb6cabe89fbc6
          4b51e4c142445c133d6075d2a68ce98e8ec6058b

- Fix a size restriction to 2 GB areas

  commit 0aa738220bb9dea2ad479e484560767b36701947

- Replace toupper with explicit tests of character values in the rare
  case TZ isn't set

  commit 030d86d04dbd6ae878ff894256c5a465890d5970

And that's it.  All other changes since 2.5.1 only affect non-Cygwin
targets in newlib and libgloss.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: last snapshot is not better than previous version

[Denis Excoffier]

> On 2016-05-30 10:25, Corinna Vinschen wrote:
> 
> 
> The problem is, there's no change at all in the snapshot which would
> explain a slowdown of such basic functions:
> 
> - Assorted select(2) improvements:
> 
>  commits a23e6a35d896a075640db714b28ce74bb6b8d7ff
>          e5665d8c930485d5ac6d8913573e27b9e5043d92

Thank you for your answer. However, i do insist.
I use again the file created with "seq 10000 > 10k" and still
look-up the string "9999" inside it. 

I perform:
strace -o vi.out /usr/bin/vim 10k 
and inside vi: 
:0
/9999
:q

Without snapshot, vi.out contains 15133 lines
With snapshot, vi.out contains 426553 lines. Huh?

Without snapshot, the 6 lines:
------------------------------
select: sel.always_ready 0
select: sel.wait returns 0
select_stuff::cleanup: calling cleanup routines
select_stuff::destroy: deleting select records
select: recalculating ms
select: ms now 0
------------------------------
occur about 450 times

With snapshot, the 6 lines:
------------------------------
select: sel.always_ready 0
select: sel.wait returns 0
select_stuff::cleanup: calling cleanup routines
select_stuff::destroy: deleting select records
select: recalculating us
select: us now 0
------------------------------
occur more than 69000 times

Something is probably wrong near here. Please tell me what i should experiment next.

Al least i have found a wrong line in select.cc: line 4, the copyright year is missing. ;-)


Regards,

Denis Excoffier.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: last snapshot is not better than previous version

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2291 bytes --]

On May 30 19:00, Denis Excoffier wrote:
> 
> > On 2016-05-30 10:25, Corinna Vinschen wrote:
> > 
> > 
> > The problem is, there's no change at all in the snapshot which would
> > explain a slowdown of such basic functions:
> > 
> > - Assorted select(2) improvements:
> > 
> >  commits a23e6a35d896a075640db714b28ce74bb6b8d7ff
> >          e5665d8c930485d5ac6d8913573e27b9e5043d92
> 
> Thank you for your answer. However, i do insist.
> I use again the file created with "seq 10000 > 10k" and still
> look-up the string "9999" inside it. 
> 
> I perform:
> strace -o vi.out /usr/bin/vim 10k 
> and inside vi: 
> :0
> /9999
> :q
> 
> Without snapshot, vi.out contains 15133 lines
> With snapshot, vi.out contains 426553 lines. Huh?

I tested with 2.5.1, with the latest snapshot and with the latest from
git master, on 32 and 64 bit.  In all cases, the above results in about
1000 'select_stuff::cleanup: calling cleanup routines' lines in vi.out,
give or take, depending on my typing speed.

> Without snapshot, the 6 lines:
> ------------------------------
> select: sel.always_ready 0
> select: sel.wait returns 0
> select_stuff::cleanup: calling cleanup routines
> select_stuff::destroy: deleting select records
> select: recalculating ms
> select: ms now 0
> ------------------------------
> occur about 450 times
> 
> With snapshot, the 6 lines:
> ------------------------------
> select: sel.always_ready 0
> select: sel.wait returns 0
> select_stuff::cleanup: calling cleanup routines
> select_stuff::destroy: deleting select records
> select: recalculating us
> select: us now 0
> ------------------------------
> occur more than 69000 times
> 
> Something is probably wrong near here. Please tell me what i should experiment next.
> 
> Al least i have found a wrong line in select.cc: line 4, the copyright year is missing. ;-)

Yeah, no worries.

Again, I can *not* reproduce this.  Maybe we should start at the basics,
a cygcheck output.  What's your OS?  I'm testing on a 64 bit W10.
What's your terminal?  I tried from Mintty as well as from a console.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: last snapshot is not better than previous version

[John Hood]

On 5/31/2016 5:16 AM, Corinna Vinschen wrote:
> On May 30 19:00, Denis Excoffier wrote:
>>> On 2016-05-30 10:25, Corinna Vinschen wrote:
>>>
>>>
>>> The problem is, there's no change at all in the snapshot which would
>>> explain a slowdown of such basic functions:
>>>
>>> - Assorted select(2) improvements:
>>>
>>>   commits a23e6a35d896a075640db714b28ce74bb6b8d7ff
>>>           e5665d8c930485d5ac6d8913573e27b9e5043d92
>> Thank you for your answer. However, i do insist.
>> I use again the file created with "seq 10000 > 10k" and still
>> look-up the string "9999" inside it.
>>
>> I perform:
>> strace -o vi.out /usr/bin/vim 10k
>> and inside vi:
>> :0
>> /9999
>> :q
>>
>> Without snapshot, vi.out contains 15133 lines
>> With snapshot, vi.out contains 426553 lines. Huh?
> I tested with 2.5.1, with the latest snapshot and with the latest from
> git master, on 32 and 64 bit.  In all cases, the above results in about
> 1000 'select_stuff::cleanup: calling cleanup routines' lines in vi.out,
> give or take, depending on my typing speed.
>
>> Without snapshot, the 6 lines:
>> ------------------------------
>> select: sel.always_ready 0
>> select: sel.wait returns 0
>> select_stuff::cleanup: calling cleanup routines
>> select_stuff::destroy: deleting select records
>> select: recalculating ms
>> select: ms now 0
>> ------------------------------
>> occur about 450 times
>>
>> With snapshot, the 6 lines:
>> ------------------------------
>> select: sel.always_ready 0
>> select: sel.wait returns 0
>> select_stuff::cleanup: calling cleanup routines
>> select_stuff::destroy: deleting select records
>> select: recalculating us
>> select: us now 0
>> ------------------------------
>> occur more than 69000 times
>>
>> Something is probably wrong near here. Please tell me what i should experiment next.
>>
>> Al least i have found a wrong line in select.cc: line 4, the copyright year is missing. ;-)
> Yeah, no worries.
>
> Again, I can *not* reproduce this.  Maybe we should start at the basics,
> a cygcheck output.  What's your OS?  I'm testing on a 64 bit W10.
> What's your terminal?  I tried from Mintty as well as from a console.
Corinna asked me to look at this since I coded the select()-related 
changes she mentioned.  I can't reproduce either of your cases.

Some things I'll mention:

1) Your Cygwin install seems to be in your Windows home directory. 
There's nothing wrong with that but it is unusual.  Is there some reason 
the usual location doesn't work?

2) The cygcheck output shows you being a member of over 50 groups, is 
this system in a domain?  Are any network shares involved?  (I tried 
mounting a local network share, then using ls.  It was 10 times slower-- 
but that was still only 4 seconds to list 11000 files.)

3) You have many thousands of packages installed.  It's not the complete 
Cygwin set but it is a substantial fraction.  It might be worth trying 
an install with a smaller number of packages.  Are you running these 
tests on the same Cygwin install but with different DLLs installed, or 
on different Cygwin installs?

4) You've given an interpretation of the strace output that you see, but 
it might be useful to see the actual strace output for your two test 
cases.  Perhaps the noisy logging for select() is an effect of the 
problem, rather than its cause.  Could you run them again (also with the 
-d flag, which will help us see where the time goes), and send them in 
email?

Regards,

   --jh

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: last snapshot is not better than previous version

[Denis Excoffier]

> On 2016-06-04 05:52, John Hood wrote:
> 
> Corinna asked me to look at this since I coded the select()-related changes she mentioned.  I can't reproduce either of your cases.
Thank you for your help.
> 
> Some things I'll mention:
> 
> 1) Your Cygwin install seems to be in your Windows home directory. There's nothing wrong with that but it is unusual.  Is there some reason the usual location doesn't work?
This PC belongs to my company, and i don't know where to write if not in my home directory. C:\CYGWIN is not an option for me (permission denied). I have no special/administrator/whatever right, only basic user. Cygwin is installed with the "Just Me" option.
> 
> 2) The cygcheck output shows you being a member of over 50 groups, is this system in a domain?  Are any network shares involved?  (I tried mounting a local network share, then using ls.  It was 10 times slower-- but that was still only 4 seconds to list 11000 files.)
Yes this system is in a domain. No network shares are involved. For these tests, i'm not connected to anything, only local drives.
> 
> 3) You have many thousands of packages installed.  It's not the complete Cygwin set but it is a substantial fraction.  It might be worth trying an install with a smaller number of packages.  Are you running these tests on the same Cygwin install but with different DLLs installed, or on different Cygwin installs?
I usually install all the packages (but not the test ones), except the Debug category for which i keep the default. I usually also install each test Cygwin package as soon as it is announced, including the cygwin-devel package (currently there are none). I also install each new snapshot as soon i can see it.
For each cygwin version installed (regular package, test package, snapshot) and immediately after installation, i compile a fresh cygwin1.dll from the corresponding source package (modifying 'define SYMLOOP_MAX' from 10 to 32 since 10 is definitely too small for me). If everything goes well, i switch to the fresh cygwin1.dll and perform my usual business with it.
For this particular 20160520 snapshot (now 20160531) i've exercised this whole process, then observed the abovementioned slowness, then switched back to the cygwin1.dll from 2.5.1-1, keeping the other files from cygwin-20160520.tar.xz (then installed cygwin-20160531.tar.xz, then switched back again to the cygwin1.dll from 2.5.1-1).
In other terms, "with snapshot" means "with the snapshot fully installed" (all files from cygwin-yyyymmdd.tar.xz). "without snashot" means the same, except that the cygwin1.dll is replaced by the one from 2.5.1-1.
> 
> 4) You've given an interpretation of the strace output that you see, but it might be useful to see the actual strace output for your two test cases.  Perhaps the noisy logging for select() is an effect of the problem, rather than its cause.  Could you run them again (also with the -d flag, which will help us see where the time goes), and send them in email?
I'll send this by private email. Please wait a little.

Regards,

Denis Excoffier.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: last snapshot is not better than previous version

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

On Jun  4 21:14, Denis Excoffier wrote:
> 
> > On 2016-06-04 05:52, John Hood wrote:
> > 
> > Corinna asked me to look at this since I coded the select()-related
> > changes she mentioned.  I can't reproduce either of your cases.
> Thank you for your help.
> > 
> > Some things I'll mention:
> > 
> > 1) Your Cygwin install seems to be in your Windows home directory.
> > There's nothing wrong with that but it is unusual.  Is there some
> > reason the usual location doesn't work?
> This PC belongs to my company, and i don't know where to write if not
> in my home directory. C:\CYGWIN is not an option for me (permission
> denied). I have no special/administrator/whatever right, only basic
> user. Cygwin is installed with the "Just Me" option.

Guys, I think I found the problem, see
https://cygwin.com/ml/cygwin/2016-06/msg00066.html and
https://cygwin.com/ml/cygwin-cvs/2016-q2/msg00069.html

I uploaded a new snapshot to https://cygwin.com/snapshots/  Please
give it a good testing.


Thanks guys,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]