From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) by sourceware.org (Postfix) with ESMTPS id 490F2385EC4E for ; Mon, 21 Sep 2020 16:23:24 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 490F2385EC4E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=SystematicSw.ab.ca Authentication-Results: sourceware.org; spf=none smtp.mailfrom=brian.inglis@systematicsw.ab.ca Received: from [192.168.1.104] ([24.64.172.44]) by shaw.ca with ESMTP id KOb3ksItHs3D6KOb4kfZ7m; Mon, 21 Sep 2020 10:23:22 -0600 X-Authority-Analysis: v=2.4 cv=bZHV7MDB c=1 sm=1 tr=0 ts=5f68d37a a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17 a=IkcTkHD0fZMA:10 a=yMhMjlubAAAA:8 a=w_pzkKWiAAAA:8 a=Maudkj2rSRX00NJfr2cA:9 a=QEXdDO2ut3YA:10 a=KyLo1vIQnU4A:10 a=sRI3_1zDfAgwuvI8zelB:22 Reply-To: cygwin@cygwin.com Subject: Re: Slowness of Cygwin commands on Windows Server 2016 To: cygwin@cygwin.com References: <0f041eee-0219-eb16-8c8b-1eb02c57a469@cornell.edu> From: Brian Inglis Autocrypt: addr=Brian.Inglis@SystematicSw.ab.ca; prefer-encrypt=mutual; keydata= mDMEXopx8xYJKwYBBAHaRw8BAQdAnCK0qv/xwUCCZQoA9BHRYpstERrspfT0NkUWQVuoePa0 LkJyaWFuIEluZ2xpcyA8QnJpYW4uSW5nbGlzQFN5c3RlbWF0aWNTdy5hYi5jYT6IlgQTFggA PhYhBMM5/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsDBQkJZgGABQsJCAcCBhUKCQgLAgQW AgMBAh4BAheAAAoJEB62lxu92I8Y0ioBAI8xrggNxziAVmr+Xm6nnyjoujMqWcq3oEhlYGAO WacZAQDFtdDx2koSVSoOmfaOyRTbIWSf9/Cjai29060fsmdsDLg4BF6KcfMSCisGAQQBl1UB BQEBB0Awv8kHI2PaEgViDqzbnoe8B9KMHoBZLS92HdC7ZPh8HQMBCAeIfgQYFggAJhYhBMM5 /lbU970GBS2bZB62lxu92I8YBQJeinHzAhsMBQkJZgGAAAoJEB62lxu92I8YZwUBAJw/74rF IyaSsGI7ewCdCy88Lce/kdwX7zGwid+f8NZ3AQC/ezTFFi5obXnyMxZJN464nPXiggtT9gN5 RSyTY8X+AQ== Organization: Systematic Software Message-ID: Date: Mon, 21 Sep 2020 10:23:21 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <0f041eee-0219-eb16-8c8b-1eb02c57a469@cornell.edu> Content-Type: text/plain; charset=utf-8 Content-Language: en-CA Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4xfH6nLayY4kaq5efL4z2n3F2SeB5XatRAKBi7VU+oVlpNdCMrDpVSPFUWJKHpVme7ixf+Z7DEBLZq2uRDSb7DCYUMeoq0HRGOfw80894NdSnQPpnnK+oy CCZT1xeJGRAsP+DzezVcMN+mzuL0URF2uHHA6qYSX8+6Q2ZFpVFCLwsv0jjsO4glArH/TmflIcrM2ahtKjU0pzT1E5Ab3VcJuf0= X-Spam-Status: No, score=-5.1 required=5.0 tests=BAYES_00, BODY_8BITS, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Sep 2020 16:23:44 -0000 On 2020-09-21 09:20, Ken Brown via Cygwin wrote: > On 9/21/2020 10:53 AM, Praveen Kumar Chandrakar via Cygwin wrote: >> Hello, >> >> We are experiencing some issues with the Cygwin commands (ie. from bash >> -login) taking several minutes on three of our Windows 2016 servers. >> >>      ls -al -- 1min 32sec >> >>      find  -- 4min 46sec >> >> When you search for a specific file, such as ls -al ./ABC/XYZ.txt, it is >> very quick. >> >> We ran strace on the machines where it is quick and where it slow. >> >> Here is some annotation for not working and working excerpted from the >> files: >> >> *=====> Not working Scenario* >> >> Command run: strace -m all -o does-not-work.txt find . -name VNM.OUT >> >> ===> Initially it hangs here for 15-25 seconds: samlib.dll >> >>    214  995796 [main] find 7768 symlink_info::check: 0 = >> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022) >> --- Process 7768 loaded C:\Windows\SysWOW64\samcli.dll at 73080000 >> --- Process 7768 loaded C:\Windows\SysWOW64\samlib.dll at 72c90000 >>   5435  423059 [main] find 7768 pwdgrp::fetch_account_from_windows: line: >> > 000:197608:> >> >> ===> Then the rest of the almost 4 minutes, it hangs at FWPUCLNT.DLL >> >>    214  995796 [main] find 7768 symlink_info::check: 0 = >> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022) >> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000 >> --- Process 7768 thread 4596 exited with status 0x0 >> --- Process 7768 thread 6568 exited with status 0x0 >> --- Process 7768 thread 7040 exited with status 0x0 >> 82877524 83873956 [main] find 7768 pwdgrp::fetch_account_from_windows: >> line: > 17-3061384290-2064:/home/sa-ca:/bin/bash> >> >> >> ===> You see the process threads created, and created again, then they >> close after FWPUCLNT.DLL unfreezes (see above) >> >> drwx------    1 CKimball UsersGrp         0 Jul 28 23:13 .. >>    303  995341 [main] find 7768 symlink_info::check: 0x0 = NtCreateFile >> (\??\D:\Abc\Xyz\conf.txt) >>    241  995582 [main] find 7768 symlink_info::check: not a symlink >>    214  995796 [main] find 7768 symlink_info::check: 0 = >> symlink.check(D:\Abc\Xyz\conf.txt, 0x64B818) (0x404022) >> --- Process 7768 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 73950000 >> --- Process 7768 thread 4596 created >> --- Process 7768 thread 6568 created >> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000 >> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000 >> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000 >>      6       6 [main] find (7768) >> ********************************************** >>    505     511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe >> (windows pid 7768) >>    203     714 [main] find (7768) OS version:   Windows NT-10.0 >>    183     897 [main] find (7768) >> ********************************************** >> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000 >> --- Process 7768 loaded C:\Windows\SysWOW64\msvcrt.dll at 753d0000 >> --- Process 7768 loaded C:\Windows\SysWOW64\sechost.dll at 75490000 >>    212  996008 [main] find 7768 path_conv::check: >> this->path(D:\Abc\Xyz\conf.txt), has_acls(1) >> --- Process 7768 thread 7040 created >> --- Process 7768 thread 4596 created >> --- Process 7768 loaded D:\Cygwin\bin\cyggcc_s-1.dll at 6c100000 >> --- Process 7768 thread 6568 created >> --- Process 7768 loaded D:\Cygwin\bin\cygintl-8.dll at 645e0000 >> --- Process 7768 loaded D:\Cygwin\bin\cygwin1.dll at 61000000 >> --- Process 7768 loaded D:\Cygwin\bin\cygiconv-2.dll at 6f150000 >>      6       6 [main] find (7768) >> ********************************************** >>    505     511 [main] find (7768) Program name: D:\Cygwin\bin\find.exe >> (windows pid 7768) >>    203     714 [main] find (7768) OS version:   Windows NT-10.0 >>    183     897 [main] find (7768) >> ********************************************** >> --- Process 7768 loaded C:\Windows\SysWOW64\advapi32.dll at 754e0000 >> >> >> *=====> Working Scenario* >> >> ===> Runs straight through to completion, no pauses/freezes >> >>    215  696807 [main] find 9456 build_fh_pc: fh 0x612AFFD0, dev 000000C3 >>    235  697042 [main] find 9456 stat_worker: (\??\F:\Abc\Xyz\conf.txt, >> 0x200749F8, 0x612AFFD0), file_attributes 16 >> --- Process 9456 loaded C:\Windows\SysWOW64\rasadhlp.dll at 73190000 >> --- Process 9456 loaded C:\Windows\SysWOW64\FWPUCLNT.DLL at 72fa0000 >> 161759  858801 [main] find 9456 pwdgrp::fetch_account_from_windows: line: >> <_svcCASpecLAN:*:1319973:1049089:U-TSYS\_svcCASpecLAN,S-1-5-21-21232 >> 42984-354184167-320618023-271397:/home/_svcCASpecLAN:/bin/bash> >>   1034  859835 [main] find 9456 fhandler_base::fstat_helper: 0 = fstat >> (\??\F:\Abc\Xyz\conf.txt, 0x200749F8) st_size=0, st_mod >> e=040775, st_ino=1688849860295204st_atim=5F210CE0.25975E6C >> st_ctim=5F210CE0.25975E6C st_mtim=5F210CE0.25975E6C >> st_birthtim=5F210CE0.16B102A4 >>    195  860030 [main] find 9456 stat_worker: 0 = >> (\??\F:\Abc\Xyz\conf.txt,0x200749F8) >> >> All our other Windows 2016 servers do not experience any issues. >> >> Do you have any suggestions on finding out what is causing this? May not be anything to do with Cygwin. This DLL is part of the Windows Filtering Platform: https://docs.microsoft.com/en-us/windows/win32/fwp/windows-filtering-platform-start-page If your org is doing network content or traffic filtering, have those folks look into what is happening here, what filters are being applied where, whether this is hooked into your AV. Also maybe check how your network interface(s) and subnet(s) hardware are performing for these servers. Cygwin may be able to help caching domain information to reduce network and process load by installing and running cygserver with only default settings and options: https://cygwin.com/cygwin-ug-net/using-cygserver.html -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in IEC units and prefixes, physical quantities in SI.]