* Slowness of Cygwin commands on Windows Server 2016
@ 2020-09-21 14:53 Praveen Kumar Chandrakar
2020-09-21 15:20 ` Ken Brown
2020-09-21 21:24 ` bzs
0 siblings, 2 replies; 4+ messages in thread
From: Praveen Kumar Chandrakar @ 2020-09-21 14:53 UTC (permalink / raw)
To: cygwin; +Cc: Jayakrishna Karicharla, Narayanarao Yenduri
[-- Attachment #1: Type: text/plain, Size: 4861 bytes --]
Hello,
We are experiencing some issues with the Cygwin commands (ie. from bash
-login) taking several minutes on three of our Windows 2016 servers.
ls -al -- 1min 32sec
find -- 4min 46sec
When you search for a specific file, such as ls -al ./ABC/XYZ.txt, it is
very quick.
We ran strace on the machines where it is quick and where it slow.
Here is some annotation for not working and working excerpted from the
files:
*=====> Not working Scenario*
Command run: strace -m all -o does-not-work.txt find . -name VNM.OUT
===> Initially it hangs here for 15-25 seconds: samlib.dll
214 995796 [main] find 7768 symlink_info::check: 0 =
symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
--- Process 7768 loaded C:\Windows\SysWOW64\samcli.dll at 73080000
--- Process 7768 loaded C:\Windows\SysWOW64\samlib.dll at 72c90000
5435 423059 [main] find 7768 pwdgrp::fetch_account_from_windows: line:
<WMW2CAOC01+TempUser Users:S-1-5-21-858922594-1104986591-3574882633-1
000:197608:>
===> Then the rest of the almost 4 minutes, it hangs at FWPUCLNT.DLL
214 995796 [main] find 7768 symlink_info::check: 0 =
symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
--- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
--- Process 7768 thread 4596 exited with status 0x0
--- Process 7768 thread 6568 exited with status 0x0
--- Process 7768 thread 7040 exited with status 0x0
82877524 83873956 [main] find 7768 pwdgrp::fetch_account_from_windows:
line: <sa-ca:*:1050640:1049089:U-TAS\sa-ca,S-1-5-21-3577821319-10804827
17-3061384290-2064:/home/sa-ca:/bin/bash>
===> You see the process threads created, and created again, then they
close after FWPUCLNT.DLL unfreezes (see above)
drwx------ 1 CKimball UsersGrp 0 Jul 28 23:13 ..
303 995341 [main] find 7768 symlink_info::check: 0x0 = NtCreateFile
(\??\D:\Abc\Xyz\conf.txt)
241 995582 [main] find 7768 symlink_info::check: not a symlink
214 995796 [main] find 7768 symlink_info::check: 0 =
symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
--- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
--- Process 7768 thread 4596 created
--- Process 7768 thread 6568 created
--- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
--- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
--- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
6 6 [main] find (7768)
**********************************************
505 511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
(windows pid 7768)
203 714 [main] find (7768) OS version: Windows NT-10.0
183 897 [main] find (7768)
**********************************************
--- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
--- Process 7768 loaded C:\Windows\SysWOW64\msvcrt.dll at 753d0000
--- Process 7768 loaded C:\Windows\SysWOW64\sechost.dll at 75490000
212 996008 [main] find 7768 path_conv::check:
this->path(D:\Abc\Xyz\conf.txt), has_acls(1)
--- Process 7768 thread 7040 created
--- Process 7768 thread 4596 created
--- Process 7768 loaded D:\Cygwin\bin\cyggcc_s-1.dll at 6c100000
--- Process 7768 thread 6568 created
--- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
--- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
--- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
6 6 [main] find (7768)
**********************************************
505 511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
(windows pid 7768)
203 714 [main] find (7768) OS version: Windows NT-10.0
183 897 [main] find (7768)
**********************************************
--- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
*=====> Working Scenario*
===> Runs straight through to completion, no pauses/freezes
215 696807 [main] find 9456 build_fh_pc: fh 0x612AFFD0, dev 000000C3
235 697042 [main] find 9456 stat_worker: (\??\F:\Abc\Xyz\conf.txt,
0x200749F8, 0x612AFFD0), file_attributes 16
--- Process 9456 loaded C:\Windows\SysWOW64\rasadhlp.dll at 73190000
--- Process 9456 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 72fa0000
161759 858801 [main] find 9456 pwdgrp::fetch_account_from_windows: line:
<_svcCASpecLAN:*:1319973:1049089:U-TSYS\_svcCASpecLAN,S-1-5-21-21232
42984-354184167-320618023-271397:/home/_svcCASpecLAN:/bin/bash>
1034 859835 [main] find 9456 fhandler_base::fstat_helper: 0 = fstat
(\??\F:\Abc\Xyz\conf.txt, 0x200749F8) st_size=0, st_mod
e=040775, st_ino=1688849860295204st_atim=5F210CE0.25975E6C
st_ctim=5F210CE0.25975E6C st_mtim=5F210CE0.25975E6C
st_birthtim=5F210CE0.16B102A4
195 860030 [main] find 9456 stat_worker: 0 =
(\??\F:\Abc\Xyz\conf.txt,0x200749F8)
All our other Windows 2016 servers do not experience any issues.
Do you have any suggestions on finding out what is causing this?
Regards,
Praveen
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4200 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Slowness of Cygwin commands on Windows Server 2016
2020-09-21 14:53 Slowness of Cygwin commands on Windows Server 2016 Praveen Kumar Chandrakar
@ 2020-09-21 15:20 ` Ken Brown
2020-09-21 16:23 ` Brian Inglis
2020-09-21 21:24 ` bzs
1 sibling, 1 reply; 4+ messages in thread
From: Ken Brown @ 2020-09-21 15:20 UTC (permalink / raw)
To: Praveen Kumar Chandrakar, cygwin
Cc: Jayakrishna Karicharla, Narayanarao Yenduri
On 9/21/2020 10:53 AM, Praveen Kumar Chandrakar via Cygwin wrote:
> Hello,
>
> We are experiencing some issues with the Cygwin commands (ie. from bash
> -login) taking several minutes on three of our Windows 2016 servers.
>
> ls -al -- 1min 32sec
>
> find -- 4min 46sec
>
> When you search for a specific file, such as ls -al ./ABC/XYZ.txt, it is
> very quick.
>
> We ran strace on the machines where it is quick and where it slow.
>
> Here is some annotation for not working and working excerpted from the
> files:
>
> *=====> Not working Scenario*
>
> Command run: strace -m all -o does-not-work.txt find . -name VNM.OUT
>
> ===> Initially it hangs here for 15-25 seconds: samlib.dll
>
> 214 995796 [main] find 7768 symlink_info::check: 0 =
> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
> --- Process 7768 loaded C:\Windows\SysWOW64\samcli.dll at 73080000
> --- Process 7768 loaded C:\Windows\SysWOW64\samlib.dll at 72c90000
> 5435 423059 [main] find 7768 pwdgrp::fetch_account_from_windows: line:
> <WMW2CAOC01+TempUser Users:S-1-5-21-858922594-1104986591-3574882633-1
> 000:197608:>
>
> ===> Then the rest of the almost 4 minutes, it hangs at FWPUCLNT.DLL
>
> 214 995796 [main] find 7768 symlink_info::check: 0 =
> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
> --- Process 7768 thread 4596 exited with status 0x0
> --- Process 7768 thread 6568 exited with status 0x0
> --- Process 7768 thread 7040 exited with status 0x0
> 82877524 83873956 [main] find 7768 pwdgrp::fetch_account_from_windows:
> line: <sa-ca:*:1050640:1049089:U-TAS\sa-ca,S-1-5-21-3577821319-10804827
> 17-3061384290-2064:/home/sa-ca:/bin/bash>
>
>
> ===> You see the process threads created, and created again, then they
> close after FWPUCLNT.DLL unfreezes (see above)
>
> drwx------ 1 CKimball UsersGrp 0 Jul 28 23:13 ..
> 303 995341 [main] find 7768 symlink_info::check: 0x0 = NtCreateFile
> (\??\D:\Abc\Xyz\conf.txt)
> 241 995582 [main] find 7768 symlink_info::check: not a symlink
> 214 995796 [main] find 7768 symlink_info::check: 0 =
> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
> --- Process 7768 thread 4596 created
> --- Process 7768 thread 6568 created
> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
> 6 6 [main] find (7768)
> **********************************************
> 505 511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
> (windows pid 7768)
> 203 714 [main] find (7768) OS version: Windows NT-10.0
> 183 897 [main] find (7768)
> **********************************************
> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
> --- Process 7768 loaded C:\Windows\SysWOW64\msvcrt.dll at 753d0000
> --- Process 7768 loaded C:\Windows\SysWOW64\sechost.dll at 75490000
> 212 996008 [main] find 7768 path_conv::check:
> this->path(D:\Abc\Xyz\conf.txt), has_acls(1)
> --- Process 7768 thread 7040 created
> --- Process 7768 thread 4596 created
> --- Process 7768 loaded D:\Cygwin\bin\cyggcc_s-1.dll at 6c100000
> --- Process 7768 thread 6568 created
> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
> 6 6 [main] find (7768)
> **********************************************
> 505 511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
> (windows pid 7768)
> 203 714 [main] find (7768) OS version: Windows NT-10.0
> 183 897 [main] find (7768)
> **********************************************
> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
>
>
> *=====> Working Scenario*
>
> ===> Runs straight through to completion, no pauses/freezes
>
> 215 696807 [main] find 9456 build_fh_pc: fh 0x612AFFD0, dev 000000C3
> 235 697042 [main] find 9456 stat_worker: (\??\F:\Abc\Xyz\conf.txt,
> 0x200749F8, 0x612AFFD0), file_attributes 16
> --- Process 9456 loaded C:\Windows\SysWOW64\rasadhlp.dll at 73190000
> --- Process 9456 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 72fa0000
> 161759 858801 [main] find 9456 pwdgrp::fetch_account_from_windows: line:
> <_svcCASpecLAN:*:1319973:1049089:U-TSYS\_svcCASpecLAN,S-1-5-21-21232
> 42984-354184167-320618023-271397:/home/_svcCASpecLAN:/bin/bash>
> 1034 859835 [main] find 9456 fhandler_base::fstat_helper: 0 = fstat
> (\??\F:\Abc\Xyz\conf.txt, 0x200749F8) st_size=0, st_mod
> e=040775, st_ino=1688849860295204st_atim=5F210CE0.25975E6C
> st_ctim=5F210CE0.25975E6C st_mtim=5F210CE0.25975E6C
> st_birthtim=5F210CE0.16B102A4
> 195 860030 [main] find 9456 stat_worker: 0 =
> (\??\F:\Abc\Xyz\conf.txt,0x200749F8)
>
> All our other Windows 2016 servers do not experience any issues.
>
> Do you have any suggestions on finding out what is causing this?
I can't give you a definitive answer, but it looks like you're running 32-bit
Cygwin. I suggest switching to 64-bit Cygwin. Here's a quote from
https://cygwin.com/ that might be relevant:
-----------------------------------------------------------------------------
32 bit Cygwin
Address space is a very limiting factor for Cygwin. These days, a full 32 bit
Cygwin distro is not feasible anymore, and will in all likelihood fail in random
places due to an issue with the fork(2) system call.
Therefore we recommend using 32 bit Cygwin only in limited scenarios, with only
a minimum of necessary packages installed, and only if there's no way to run 64
bit Cygwin instead.
-----------------------------------------------------------------------------
Ken
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Slowness of Cygwin commands on Windows Server 2016
2020-09-21 15:20 ` Ken Brown
@ 2020-09-21 16:23 ` Brian Inglis
0 siblings, 0 replies; 4+ messages in thread
From: Brian Inglis @ 2020-09-21 16:23 UTC (permalink / raw)
To: cygwin
On 2020-09-21 09:20, Ken Brown via Cygwin wrote:
> On 9/21/2020 10:53 AM, Praveen Kumar Chandrakar via Cygwin wrote:
>> Hello,
>>
>> We are experiencing some issues with the Cygwin commands (ie. from bash
>> -login) taking several minutes on three of our Windows 2016 servers.
>>
>> ls -al -- 1min 32sec
>>
>> find -- 4min 46sec
>>
>> When you search for a specific file, such as ls -al ./ABC/XYZ.txt, it is
>> very quick.
>>
>> We ran strace on the machines where it is quick and where it slow.
>>
>> Here is some annotation for not working and working excerpted from the
>> files:
>>
>> *=====> Not working Scenario*
>>
>> Command run: strace -m all -o does-not-work.txt find . -name VNM.OUT
>>
>> ===> Initially it hangs here for 15-25 seconds: samlib.dll
>>
>> 214 995796 [main] find 7768 symlink_info::check: 0 =
>> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
>> --- Process 7768 loaded C:\Windows\SysWOW64\samcli.dll at 73080000
>> --- Process 7768 loaded C:\Windows\SysWOW64\samlib.dll at 72c90000
>> 5435 423059 [main] find 7768 pwdgrp::fetch_account_from_windows: line:
>> <WMW2CAOC01+TempUser Users:S-1-5-21-858922594-1104986591-3574882633-1
>> 000:197608:>
>>
>> ===> Then the rest of the almost 4 minutes, it hangs at FWPUCLNT.DLL
>>
>> 214 995796 [main] find 7768 symlink_info::check: 0 =
>> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
>> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
>> --- Process 7768 thread 4596 exited with status 0x0
>> --- Process 7768 thread 6568 exited with status 0x0
>> --- Process 7768 thread 7040 exited with status 0x0
>> 82877524 83873956 [main] find 7768 pwdgrp::fetch_account_from_windows:
>> line: <sa-ca:*:1050640:1049089:U-TAS\sa-ca,S-1-5-21-3577821319-10804827
>> 17-3061384290-2064:/home/sa-ca:/bin/bash>
>>
>>
>> ===> You see the process threads created, and created again, then they
>> close after FWPUCLNT.DLL unfreezes (see above)
>>
>> drwx------ 1 CKimball UsersGrp 0 Jul 28 23:13 ..
>> 303 995341 [main] find 7768 symlink_info::check: 0x0 = NtCreateFile
>> (\??\D:\Abc\Xyz\conf.txt)
>> 241 995582 [main] find 7768 symlink_info::check: not a symlink
>> 214 995796 [main] find 7768 symlink_info::check: 0 =
>> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022)
>> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000
>> --- Process 7768 thread 4596 created
>> --- Process 7768 thread 6568 created
>> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
>> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
>> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
>> 6 6 [main] find (7768)
>> **********************************************
>> 505 511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
>> (windows pid 7768)
>> 203 714 [main] find (7768) OS version: Windows NT-10.0
>> 183 897 [main] find (7768)
>> **********************************************
>> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
>> --- Process 7768 loaded C:\Windows\SysWOW64\msvcrt.dll at 753d0000
>> --- Process 7768 loaded C:\Windows\SysWOW64\sechost.dll at 75490000
>> 212 996008 [main] find 7768 path_conv::check:
>> this->path(D:\Abc\Xyz\conf.txt), has_acls(1)
>> --- Process 7768 thread 7040 created
>> --- Process 7768 thread 4596 created
>> --- Process 7768 loaded D:\Cygwin\bin\cyggcc_s-1.dll at 6c100000
>> --- Process 7768 thread 6568 created
>> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000
>> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000
>> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000
>> 6 6 [main] find (7768)
>> **********************************************
>> 505 511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe
>> (windows pid 7768)
>> 203 714 [main] find (7768) OS version: Windows NT-10.0
>> 183 897 [main] find (7768)
>> **********************************************
>> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000
>>
>>
>> *=====> Working Scenario*
>>
>> ===> Runs straight through to completion, no pauses/freezes
>>
>> 215 696807 [main] find 9456 build_fh_pc: fh 0x612AFFD0, dev 000000C3
>> 235 697042 [main] find 9456 stat_worker: (\??\F:\Abc\Xyz\conf.txt,
>> 0x200749F8, 0x612AFFD0), file_attributes 16
>> --- Process 9456 loaded C:\Windows\SysWOW64\rasadhlp.dll at 73190000
>> --- Process 9456 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 72fa0000
>> 161759 858801 [main] find 9456 pwdgrp::fetch_account_from_windows: line:
>> <_svcCASpecLAN:*:1319973:1049089:U-TSYS\_svcCASpecLAN,S-1-5-21-21232
>> 42984-354184167-320618023-271397:/home/_svcCASpecLAN:/bin/bash>
>> 1034 859835 [main] find 9456 fhandler_base::fstat_helper: 0 = fstat
>> (\??\F:\Abc\Xyz\conf.txt, 0x200749F8) st_size=0, st_mod
>> e=040775, st_ino=1688849860295204st_atim=5F210CE0.25975E6C
>> st_ctim=5F210CE0.25975E6C st_mtim=5F210CE0.25975E6C
>> st_birthtim=5F210CE0.16B102A4
>> 195 860030 [main] find 9456 stat_worker: 0 =
>> (\??\F:\Abc\Xyz\conf.txt,0x200749F8)
>>
>> All our other Windows 2016 servers do not experience any issues.
>>
>> Do you have any suggestions on finding out what is causing this?
May not be anything to do with Cygwin.
This DLL is part of the Windows Filtering Platform:
https://docs.microsoft.com/en-us/windows/win32/fwp/windows-filtering-platform-start-page
If your org is doing network content or traffic filtering, have those folks look
into what is happening here, what filters are being applied where, whether this
is hooked into your AV.
Also maybe check how your network interface(s) and subnet(s) hardware are
performing for these servers.
Cygwin may be able to help caching domain information to reduce network and
process load by installing and running cygserver with only default settings and
options:
https://cygwin.com/cygwin-ug-net/using-cygserver.html
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Slowness of Cygwin commands on Windows Server 2016
2020-09-21 14:53 Slowness of Cygwin commands on Windows Server 2016 Praveen Kumar Chandrakar
2020-09-21 15:20 ` Ken Brown
@ 2020-09-21 21:24 ` bzs
1 sibling, 0 replies; 4+ messages in thread
From: bzs @ 2020-09-21 21:24 UTC (permalink / raw)
To: Praveen Kumar Chandrakar
Cc: cygwin, Jayakrishna Karicharla, Narayanarao Yenduri
> ls -al -- 1min 32sec
>
> find -- 4min 46sec
Probably security/anti-virus software, I had this problem for a long
time with Rapport until I'd stop it in taskmgr > services tho I
believe newer versions of Rapport improved this somewhat.
I doubt 1.5 minutes for a simple ls is just caused by simple system
performance limitations unless everything runs this slow. Something is
intercepting file opens or similar.
--
-Barry Shein
Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD
The World: Since 1989 | A Public Information Utility | *oo*
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-09-21 21:25 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-21 14:53 Slowness of Cygwin commands on Windows Server 2016 Praveen Kumar Chandrakar
2020-09-21 15:20 ` Ken Brown
2020-09-21 16:23 ` Brian Inglis
2020-09-21 21:24 ` bzs
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).