Cygwin commands are intermittently hanging on Windows

Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

[-- Attachment #1: Type: text/plain, Size: 2463 bytes --]

We've been encountering Cygwin commands that are intermittently hanging on Windows. We don't have any predictable way to reproduce it besides running commands in a loop until we encounter a hang. So, we've created a script that will loop and execute 5 different Cygwin commands (touch, cp, mv, which, and rm) and log any commands that take longer than a second. The script is set to loop 10,000 times which gives a handful of instances of hangs in the log that have taken anywhere from 10-200 seconds to complete. Has anyone else encountered hangs from Cygwin commands?

I've printed the script below for reference:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
use strict;
use Cwd;

my $TESTDIR = getcwd() . "\\test<file://test>";
my $MKDIRCMD = "mkdir test";
my $TOUCHCMD = 'touch test.txt';
my $CPCMD = 'cp test.txt test2.txt';
my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>";
my $WHICHCMD = 'which explorer.exe';
my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";
my $count = 0;
my $fh;
my $LOG = open $fh, '>', 'cygwinTest.log';

if ( ! -e $TESTDIR ) {
                runCMD($MKDIRCMD, 0);
}

while ($count < 10000) {
                print "Loop $count:\n";
                runCMD($TOUCHCMD, $count);
                runCMD($CPCMD, $count);
                runCMD($MVCMD, $count);
                runCMD($WHICHCMD, $count);
                runCMD($RMCMD, $count);
                $count++;
}

sub runCMD {
                my $CMD = $_[0];
                my $COUNT = $_[1];
                my $retCode;
                my $startTime;
                my $endTime;
                my $totalTime;

                $startTime = time();
                $CMD = "C:\\Windows\\system32\\cmd.exe /c $CMD";
                $retCode = system($CMD);
                $endTime = time();
                $totalTime = $endTime - $startTime;
                if ($totalTime > 1 ) {
                                print $fh "Loop #$COUNT:\n";
                                if ($retCode == 0) {
                                                print $fh "It took $totalTime seconds to run [$CMD]\n";
                                } else {
                                                print $fh "It took $totalTime seconds to unsuccesfully run [$CMD]\n";
                                }
                }
}

close $fh;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Re: Cygwin commands are intermittently hanging on Windows

[Takashi Yano]

On Mon, 7 Nov 2022 22:06:46 +0000
Derek Pagel wrote:
> I've printed the script below for reference:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> use strict;
> use Cwd;
> 
> my $TESTDIR = getcwd() . "\\test<file://test>";
> my $MKDIRCMD = "mkdir test";
> my $TOUCHCMD = 'touch test.txt';
> my $CPCMD = 'cp test.txt test2.txt';
> my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>";
> my $WHICHCMD = 'which explorer.exe';
> my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";
> my $count = 0;
> my $fh;
> my $LOG = open $fh, '>', 'cygwinTest.log';
> 
> if ( ! -e $TESTDIR ) {
>                 runCMD($MKDIRCMD, 0);
> }
> 
> while ($count < 10000) {
>                 print "Loop $count:\n";
>                 runCMD($TOUCHCMD, $count);
>                 runCMD($CPCMD, $count);
>                 runCMD($MVCMD, $count);
>                 runCMD($WHICHCMD, $count);
>                 runCMD($RMCMD, $count);
>                 $count++;
> }
> 
> sub runCMD {
>                 my $CMD = $_[0];
>                 my $COUNT = $_[1];
>                 my $retCode;
>                 my $startTime;
>                 my $endTime;
>                 my $totalTime;
> 
>                 $startTime = time();
>                 $CMD = "C:\\Windows\\system32\\cmd.exe /c $CMD";
>                 $retCode = system($CMD);
>                 $endTime = time();
>                 $totalTime = $endTime - $startTime;
>                 if ($totalTime > 1 ) {
>                                 print $fh "Loop #$COUNT:\n";
>                                 if ($retCode == 0) {
>                                                 print $fh "It took $totalTime seconds to run [$CMD]\n";
>                                 } else {
>                                                 print $fh "It took $totalTime seconds to unsuccesfully run [$CMD]\n";
>                                 }
>                 }
> }
> 
> close $fh;
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Which script language do you use for this script?
It looks like perl script, however, cygwin perl
cannot run this script.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

RE: Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

The script was written using Strawberry Perl with Cygwin in the %PATH% variable. Then the script makes system calls using that Cygwin.


On Mon, 7 Nov 2022 22:06:46 +0000
Derek Pagel wrote:
> I've printed the script below for reference:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> use strict;
> use Cwd;
> 
> my $TESTDIR = getcwd() . "\\test<file://test>"; my $MKDIRCMD = "mkdir 
> test"; my $TOUCHCMD = 'touch test.txt'; my $CPCMD = 'cp test.txt 
> test2.txt'; my $MVCMD = "mv test.txt " . $TESTDIR . 
> "\\test.txt<file://test.txt>"; my $WHICHCMD = 'which explorer.exe'; my 
> $RMCMD = "rm " . $TESTDIR . "\\test.txt 
> test2.txt<file://test.txt%20test2.txt>";
> my $count = 0;
> my $fh;
> my $LOG = open $fh, '>', 'cygwinTest.log';
> 
> if ( ! -e $TESTDIR ) {
>                 runCMD($MKDIRCMD, 0);
> }
> 
> while ($count < 10000) {
>                 print "Loop $count:\n";
>                 runCMD($TOUCHCMD, $count);
>                 runCMD($CPCMD, $count);
>                 runCMD($MVCMD, $count);
>                 runCMD($WHICHCMD, $count);
>                 runCMD($RMCMD, $count);
>                 $count++;
> }
> 
> sub runCMD {
>                 my $CMD = $_[0];
>                 my $COUNT = $_[1];
>                 my $retCode;
>                 my $startTime;
>                 my $endTime;
>                 my $totalTime;
> 
>                 $startTime = time();
>                 $CMD = "C:\\Windows\\system32\\cmd.exe /c $CMD";
>                 $retCode = system($CMD);
>                 $endTime = time();
>                 $totalTime = $endTime - $startTime;
>                 if ($totalTime > 1 ) {
>                                 print $fh "Loop #$COUNT:\n";
>                                 if ($retCode == 0) {
>                                                 print $fh "It took $totalTime seconds to run [$CMD]\n";
>                                 } else {
>                                                 print $fh "It took $totalTime seconds to unsuccesfully run [$CMD]\n";
>                                 }
>                 }
> }
> 
> close $fh;
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Which script language do you use for this script?
It looks like perl script, however, cygwin perl cannot run this script.

--
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Cygwin commands are intermittently hanging on Windows

[Takashi Yano]

On Tue, 8 Nov 2022 13:25:05 +0000
Derek Pagel wrote:
> The script was written using Strawberry Perl with Cygwin in the %PATH% variable. Then the script makes system calls using that Cygwin.

my $TESTDIR = getcwd() . "\\test<file://test>";
my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>";
my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";

Even using Strawberry Perl, "<file://xxx>" in above lines cause error.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

RE: Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

my $TESTDIR = getcwd() . "\\test<file://test>"; my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>"; my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";

Even using Strawberry Perl, "<file://xxx>" in above lines cause error.


Oh, I see, when I copy-pasted the script into the thread, it added a few code changes, and I missed cleaning up those ones. The script should be able to run if you remove " <file://test>", "<file://test.txt>" and " <file://test.txt%20test2.txt>" out of the above lines.

Re: Cygwin commands are intermittently hanging on Windows

[Takashi Yano]

On Tue, 8 Nov 2022 16:00:52 +0000
Derek Pagel wrote:
> Even using Strawberry Perl, "<file://xxx>" in above lines cause error.
> 
> Oh, I see, when I copy-pasted the script into the thread, it added a few code changes, and I missed cleaning up those ones. The script should be able to run if you remove " <file://test>", "<file://test.txt>" and " <file://test.txt%20test2.txt>" out of the above lines.

I ran your script twice, once in command prompt and once in mintty.
Both attempts finished without any problems.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

RE: Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

I ran your script twice, once in command prompt and once in mintty.
Both attempts finished without any problems.

Thanks for running the script and trying it. What OS and Cygwin version were you using?

Re: Cygwin commands are intermittently hanging on Windows

[Takashi Yano]

On Tue, 8 Nov 2022 17:17:50 +0000
Derek Pagel wrote:
> Thanks for running the script and trying it. What OS and Cygwin version were you using?

Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin

And you?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

RE: Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin

And you?

I've seen the issue on Windows Server 2012 R2 Version 6.3 (Build 9600) and Windows Server 2019 Version 1809 (OS Build 17763.3532) with all different versions of Cygwin 3.x.x.

RE: Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin

And you?

*************************************

We've recreated the hanging issue on Windows 2019 and were able to capture and attach a DMP file of a 'cp' command that was hanging. One other thing that we noticed is that the Cygwin commands each tend to hang about 42 or 43 seconds.


[-- Attachment #2: cp.DMP --]
[-- Type: application/octet-stream, Size: 20206092 bytes --]

RE: Cygwin commands are intermittently hanging on Windows

[Derek Pagel]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin

And you?

I've seen the issue on Windows Server 2012 R2 Version 6.3 (Build 9600) and Windows Server 2019 Version 1809 (OS Build 17763.3532) with all different versions of Cygwin 3.x.x.

***********************************************

We've recreated the hanging issue on Windows 2019 and were able to capture and attach a DMP file of a 'cp' command that was hanging. One other thing that we noticed is that the Cygwin commands each tend to hang about 42 or 43 seconds.

[-- Attachment #2: cp.DMP --]
[-- Type: application/octet-stream, Size: 20206092 bytes --]

RE: Cygwin commands are intermittently hanging on Windows

[Brian Inglis]

On Wed, 16 Nov 2022 17:19:22 +0000, Derek Pagel wrote:
> We've recreated the hanging issue on Windows 2019 and were able to capture 
> and attach a DMP file of a 'cp' command that was hanging. One other thing 
> that we noticed is that the Cygwin commands each tend to hang about 42 or 43
> seconds.
So the commands are not hanging, they are slowed down, or delayed by 40 seconds.
That sounds like the common symptoms of long/large queries/responses to/from a 
network server, usually a DC, often for AD info, or a slow network connection to 
a DC or AD server, or possibly a slow SMB or NFS file server.

If you are exec-ing each Cygwin command from a non-Cygwin process, that will 
require setting up the Cygwin environment, including possibly long searches 
through your AD each time for all group memberships.

See:	https://cygwin.com/cygwin-ug-net/ntsec.html

The solution is usually to run the Cygwin cygserver daemon at system startup to 
preload, cache, and share AD entries and other info across Cygwin processes.

See:	https://cygwin.com/faq/faq.html#faq.using.startup-slow

You can provide confirmation or more information on the low level causes by 
running the slow command from strace e.g. "strace -o cp.strace cp ..." and 
providing the cp.strace output as a *TEXT* attachment to a reply post.

Long PATH lists, especially with network shares, can also slow down searches for 
executable binaries, as can home directories or configuration files on shares.
Setting Cygwin PATH to minimal $HOME/bin:/usr/local/bin:/usr/bin:/bin assuming 
those exist and are not on the network reduces searches.

It is also possible that a less optimal AV is doing something on every Cygwin 
process start, or some process is intercepting DLLs, which may slow down processes.
Run cygcheck cp to see if anything other than cygwin and standard Windows base 
system32 DLLs from standard locations are being loaded.
Check to see if any AV product you are running is chewing up a lot of time on 
the system while you are running Cygwin commands.

See:	https://cygwin.com/faq/faq.html#faq.using.bloda

Windows system DMPs are not useful for diagnosing problems with Cygwin 
multithreaded processes.
You could see if you can find anyone who could diagnose what Windows was doing 
during those 40 second periods from the DMP as a cross-check.

-- 
Take care. Thanks, Brian Inglis			Calgary, Alberta, Canada

La perfection est atteinte			Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter	not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer	but when there is no more to cut
			-- Antoine de Saint-Exupéry