* Cygwin commands are intermittently hanging on Windows
@ 2022-11-07 22:06 Derek Pagel
2022-11-08 0:24 ` Takashi Yano
0 siblings, 1 reply; 12+ messages in thread
From: Derek Pagel @ 2022-11-07 22:06 UTC (permalink / raw)
To: cygwin
[-- Attachment #1: Type: text/plain, Size: 2463 bytes --]
We've been encountering Cygwin commands that are intermittently hanging on Windows. We don't have any predictable way to reproduce it besides running commands in a loop until we encounter a hang. So, we've created a script that will loop and execute 5 different Cygwin commands (touch, cp, mv, which, and rm) and log any commands that take longer than a second. The script is set to loop 10,000 times which gives a handful of instances of hangs in the log that have taken anywhere from 10-200 seconds to complete. Has anyone else encountered hangs from Cygwin commands?
I've printed the script below for reference:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
use strict;
use Cwd;
my $TESTDIR = getcwd() . "\\test<file://test>";
my $MKDIRCMD = "mkdir test";
my $TOUCHCMD = 'touch test.txt';
my $CPCMD = 'cp test.txt test2.txt';
my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>";
my $WHICHCMD = 'which explorer.exe';
my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";
my $count = 0;
my $fh;
my $LOG = open $fh, '>', 'cygwinTest.log';
if ( ! -e $TESTDIR ) {
runCMD($MKDIRCMD, 0);
}
while ($count < 10000) {
print "Loop $count:\n";
runCMD($TOUCHCMD, $count);
runCMD($CPCMD, $count);
runCMD($MVCMD, $count);
runCMD($WHICHCMD, $count);
runCMD($RMCMD, $count);
$count++;
}
sub runCMD {
my $CMD = $_[0];
my $COUNT = $_[1];
my $retCode;
my $startTime;
my $endTime;
my $totalTime;
$startTime = time();
$CMD = "C:\\Windows\\system32\\cmd.exe /c $CMD";
$retCode = system($CMD);
$endTime = time();
$totalTime = $endTime - $startTime;
if ($totalTime > 1 ) {
print $fh "Loop #$COUNT:\n";
if ($retCode == 0) {
print $fh "It took $totalTime seconds to run [$CMD]\n";
} else {
print $fh "It took $totalTime seconds to unsuccesfully run [$CMD]\n";
}
}
}
close $fh;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Cygwin commands are intermittently hanging on Windows
2022-11-07 22:06 Cygwin commands are intermittently hanging on Windows Derek Pagel
@ 2022-11-08 0:24 ` Takashi Yano
2022-11-08 13:25 ` Derek Pagel
0 siblings, 1 reply; 12+ messages in thread
From: Takashi Yano @ 2022-11-08 0:24 UTC (permalink / raw)
To: cygwin; +Cc: Derek Pagel
On Mon, 7 Nov 2022 22:06:46 +0000
Derek Pagel wrote:
> I've printed the script below for reference:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> use strict;
> use Cwd;
>
> my $TESTDIR = getcwd() . "\\test<file://test>";
> my $MKDIRCMD = "mkdir test";
> my $TOUCHCMD = 'touch test.txt';
> my $CPCMD = 'cp test.txt test2.txt';
> my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>";
> my $WHICHCMD = 'which explorer.exe';
> my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";
> my $count = 0;
> my $fh;
> my $LOG = open $fh, '>', 'cygwinTest.log';
>
> if ( ! -e $TESTDIR ) {
> runCMD($MKDIRCMD, 0);
> }
>
> while ($count < 10000) {
> print "Loop $count:\n";
> runCMD($TOUCHCMD, $count);
> runCMD($CPCMD, $count);
> runCMD($MVCMD, $count);
> runCMD($WHICHCMD, $count);
> runCMD($RMCMD, $count);
> $count++;
> }
>
> sub runCMD {
> my $CMD = $_[0];
> my $COUNT = $_[1];
> my $retCode;
> my $startTime;
> my $endTime;
> my $totalTime;
>
> $startTime = time();
> $CMD = "C:\\Windows\\system32\\cmd.exe /c $CMD";
> $retCode = system($CMD);
> $endTime = time();
> $totalTime = $endTime - $startTime;
> if ($totalTime > 1 ) {
> print $fh "Loop #$COUNT:\n";
> if ($retCode == 0) {
> print $fh "It took $totalTime seconds to run [$CMD]\n";
> } else {
> print $fh "It took $totalTime seconds to unsuccesfully run [$CMD]\n";
> }
> }
> }
>
> close $fh;
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Which script language do you use for this script?
It looks like perl script, however, cygwin perl
cannot run this script.
--
Takashi Yano <takashi.yano@nifty.ne.jp>
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-08 0:24 ` Takashi Yano
@ 2022-11-08 13:25 ` Derek Pagel
2022-11-08 15:41 ` Takashi Yano
0 siblings, 1 reply; 12+ messages in thread
From: Derek Pagel @ 2022-11-08 13:25 UTC (permalink / raw)
To: cygwin; +Cc: Takashi Yano
The script was written using Strawberry Perl with Cygwin in the %PATH% variable. Then the script makes system calls using that Cygwin.
On Mon, 7 Nov 2022 22:06:46 +0000
Derek Pagel wrote:
> I've printed the script below for reference:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> use strict;
> use Cwd;
>
> my $TESTDIR = getcwd() . "\\test<file://test>"; my $MKDIRCMD = "mkdir
> test"; my $TOUCHCMD = 'touch test.txt'; my $CPCMD = 'cp test.txt
> test2.txt'; my $MVCMD = "mv test.txt " . $TESTDIR .
> "\\test.txt<file://test.txt>"; my $WHICHCMD = 'which explorer.exe'; my
> $RMCMD = "rm " . $TESTDIR . "\\test.txt
> test2.txt<file://test.txt%20test2.txt>";
> my $count = 0;
> my $fh;
> my $LOG = open $fh, '>', 'cygwinTest.log';
>
> if ( ! -e $TESTDIR ) {
> runCMD($MKDIRCMD, 0);
> }
>
> while ($count < 10000) {
> print "Loop $count:\n";
> runCMD($TOUCHCMD, $count);
> runCMD($CPCMD, $count);
> runCMD($MVCMD, $count);
> runCMD($WHICHCMD, $count);
> runCMD($RMCMD, $count);
> $count++;
> }
>
> sub runCMD {
> my $CMD = $_[0];
> my $COUNT = $_[1];
> my $retCode;
> my $startTime;
> my $endTime;
> my $totalTime;
>
> $startTime = time();
> $CMD = "C:\\Windows\\system32\\cmd.exe /c $CMD";
> $retCode = system($CMD);
> $endTime = time();
> $totalTime = $endTime - $startTime;
> if ($totalTime > 1 ) {
> print $fh "Loop #$COUNT:\n";
> if ($retCode == 0) {
> print $fh "It took $totalTime seconds to run [$CMD]\n";
> } else {
> print $fh "It took $totalTime seconds to unsuccesfully run [$CMD]\n";
> }
> }
> }
>
> close $fh;
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Which script language do you use for this script?
It looks like perl script, however, cygwin perl cannot run this script.
--
Takashi Yano <takashi.yano@nifty.ne.jp>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Cygwin commands are intermittently hanging on Windows
2022-11-08 13:25 ` Derek Pagel
@ 2022-11-08 15:41 ` Takashi Yano
2022-11-08 16:00 ` Derek Pagel
0 siblings, 1 reply; 12+ messages in thread
From: Takashi Yano @ 2022-11-08 15:41 UTC (permalink / raw)
To: cygwin; +Cc: Derek Pagel
On Tue, 8 Nov 2022 13:25:05 +0000
Derek Pagel wrote:
> The script was written using Strawberry Perl with Cygwin in the %PATH% variable. Then the script makes system calls using that Cygwin.
my $TESTDIR = getcwd() . "\\test<file://test>";
my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>";
my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";
Even using Strawberry Perl, "<file://xxx>" in above lines cause error.
--
Takashi Yano <takashi.yano@nifty.ne.jp>
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-08 15:41 ` Takashi Yano
@ 2022-11-08 16:00 ` Derek Pagel
2022-11-08 16:50 ` Takashi Yano
0 siblings, 1 reply; 12+ messages in thread
From: Derek Pagel @ 2022-11-08 16:00 UTC (permalink / raw)
To: cygwin; +Cc: Takashi Yano
my $TESTDIR = getcwd() . "\\test<file://test>"; my $MVCMD = "mv test.txt " . $TESTDIR . "\\test.txt<file://test.txt>"; my $RMCMD = "rm " . $TESTDIR . "\\test.txt test2.txt<file://test.txt%20test2.txt>";
Even using Strawberry Perl, "<file://xxx>" in above lines cause error.
Oh, I see, when I copy-pasted the script into the thread, it added a few code changes, and I missed cleaning up those ones. The script should be able to run if you remove " <file://test>", "<file://test.txt>" and " <file://test.txt%20test2.txt>" out of the above lines.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Cygwin commands are intermittently hanging on Windows
2022-11-08 16:00 ` Derek Pagel
@ 2022-11-08 16:50 ` Takashi Yano
2022-11-08 17:17 ` Derek Pagel
0 siblings, 1 reply; 12+ messages in thread
From: Takashi Yano @ 2022-11-08 16:50 UTC (permalink / raw)
To: cygwin; +Cc: Derek Pagel
On Tue, 8 Nov 2022 16:00:52 +0000
Derek Pagel wrote:
> Even using Strawberry Perl, "<file://xxx>" in above lines cause error.
>
> Oh, I see, when I copy-pasted the script into the thread, it added a few code changes, and I missed cleaning up those ones. The script should be able to run if you remove " <file://test>", "<file://test.txt>" and " <file://test.txt%20test2.txt>" out of the above lines.
I ran your script twice, once in command prompt and once in mintty.
Both attempts finished without any problems.
--
Takashi Yano <takashi.yano@nifty.ne.jp>
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-08 16:50 ` Takashi Yano
@ 2022-11-08 17:17 ` Derek Pagel
2022-11-08 17:24 ` Takashi Yano
0 siblings, 1 reply; 12+ messages in thread
From: Derek Pagel @ 2022-11-08 17:17 UTC (permalink / raw)
To: Takashi Yano, cygwin
I ran your script twice, once in command prompt and once in mintty.
Both attempts finished without any problems.
Thanks for running the script and trying it. What OS and Cygwin version were you using?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Cygwin commands are intermittently hanging on Windows
2022-11-08 17:17 ` Derek Pagel
@ 2022-11-08 17:24 ` Takashi Yano
2022-11-08 18:11 ` Derek Pagel
2022-11-16 17:19 ` Derek Pagel
0 siblings, 2 replies; 12+ messages in thread
From: Takashi Yano @ 2022-11-08 17:24 UTC (permalink / raw)
To: cygwin; +Cc: Derek Pagel
On Tue, 8 Nov 2022 17:17:50 +0000
Derek Pagel wrote:
> Thanks for running the script and trying it. What OS and Cygwin version were you using?
Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin
And you?
--
Takashi Yano <takashi.yano@nifty.ne.jp>
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-08 17:24 ` Takashi Yano
@ 2022-11-08 18:11 ` Derek Pagel
2022-11-22 13:23 ` Derek Pagel
2022-11-16 17:19 ` Derek Pagel
1 sibling, 1 reply; 12+ messages in thread
From: Derek Pagel @ 2022-11-08 18:11 UTC (permalink / raw)
To: cygwin; +Cc: Takashi Yano
Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin
And you?
I've seen the issue on Windows Server 2012 R2 Version 6.3 (Build 9600) and Windows Server 2019 Version 1809 (OS Build 17763.3532) with all different versions of Cygwin 3.x.x.
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-08 17:24 ` Takashi Yano
2022-11-08 18:11 ` Derek Pagel
@ 2022-11-16 17:19 ` Derek Pagel
2022-11-25 21:54 ` Brian Inglis
1 sibling, 1 reply; 12+ messages in thread
From: Derek Pagel @ 2022-11-16 17:19 UTC (permalink / raw)
To: cygwin; +Cc: Takashi Yano
[-- Attachment #1: Type: text/plain, Size: 446 bytes --]
Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin
And you?
*************************************
We've recreated the hanging issue on Windows 2019 and were able to capture and attach a DMP file of a 'cp' command that was hanging. One other thing that we noticed is that the Cygwin commands each tend to hang about 42 or 43 seconds.
[-- Attachment #2: cp.DMP --]
[-- Type: application/octet-stream, Size: 20206092 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-08 18:11 ` Derek Pagel
@ 2022-11-22 13:23 ` Derek Pagel
0 siblings, 0 replies; 12+ messages in thread
From: Derek Pagel @ 2022-11-22 13:23 UTC (permalink / raw)
To: cygwin; +Cc: Takashi Yano
[-- Attachment #1: Type: text/plain, Size: 632 bytes --]
Windows 10 Professional (64 bit) 22H2 Version 10.0.19045.2130
Cygwin: CYGWIN_NT-10.0-19045 HP-Z230 3.3.6-341.x86_64 2022-09-05 11:15 UTC x86_64 Cygwin
And you?
I've seen the issue on Windows Server 2012 R2 Version 6.3 (Build 9600) and Windows Server 2019 Version 1809 (OS Build 17763.3532) with all different versions of Cygwin 3.x.x.
***********************************************
We've recreated the hanging issue on Windows 2019 and were able to capture and attach a DMP file of a 'cp' command that was hanging. One other thing that we noticed is that the Cygwin commands each tend to hang about 42 or 43 seconds.
[-- Attachment #2: cp.DMP --]
[-- Type: application/octet-stream, Size: 20206092 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Cygwin commands are intermittently hanging on Windows
2022-11-16 17:19 ` Derek Pagel
@ 2022-11-25 21:54 ` Brian Inglis
0 siblings, 0 replies; 12+ messages in thread
From: Brian Inglis @ 2022-11-25 21:54 UTC (permalink / raw)
To: cygwin; +Cc: derek.pagel
On Wed, 16 Nov 2022 17:19:22 +0000, Derek Pagel wrote:
> We've recreated the hanging issue on Windows 2019 and were able to capture
> and attach a DMP file of a 'cp' command that was hanging. One other thing
> that we noticed is that the Cygwin commands each tend to hang about 42 or 43
> seconds.
So the commands are not hanging, they are slowed down, or delayed by 40 seconds.
That sounds like the common symptoms of long/large queries/responses to/from a
network server, usually a DC, often for AD info, or a slow network connection to
a DC or AD server, or possibly a slow SMB or NFS file server.
If you are exec-ing each Cygwin command from a non-Cygwin process, that will
require setting up the Cygwin environment, including possibly long searches
through your AD each time for all group memberships.
See: https://cygwin.com/cygwin-ug-net/ntsec.html
The solution is usually to run the Cygwin cygserver daemon at system startup to
preload, cache, and share AD entries and other info across Cygwin processes.
See: https://cygwin.com/faq/faq.html#faq.using.startup-slow
You can provide confirmation or more information on the low level causes by
running the slow command from strace e.g. "strace -o cp.strace cp ..." and
providing the cp.strace output as a *TEXT* attachment to a reply post.
Long PATH lists, especially with network shares, can also slow down searches for
executable binaries, as can home directories or configuration files on shares.
Setting Cygwin PATH to minimal $HOME/bin:/usr/local/bin:/usr/bin:/bin assuming
those exist and are not on the network reduces searches.
It is also possible that a less optimal AV is doing something on every Cygwin
process start, or some process is intercepting DLLs, which may slow down processes.
Run cygcheck cp to see if anything other than cygwin and standard Windows base
system32 DLLs from standard locations are being loaded.
Check to see if any AV product you are running is chewing up a lot of time on
the system while you are running Cygwin commands.
See: https://cygwin.com/faq/faq.html#faq.using.bloda
Windows system DMPs are not useful for diagnosing problems with Cygwin
multithreaded processes.
You could see if you can find anyone who could diagnose what Windows was doing
during those 40 second periods from the DMP as a cross-check.
--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-11-25 21:54 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-07 22:06 Cygwin commands are intermittently hanging on Windows Derek Pagel
2022-11-08 0:24 ` Takashi Yano
2022-11-08 13:25 ` Derek Pagel
2022-11-08 15:41 ` Takashi Yano
2022-11-08 16:00 ` Derek Pagel
2022-11-08 16:50 ` Takashi Yano
2022-11-08 17:17 ` Derek Pagel
2022-11-08 17:24 ` Takashi Yano
2022-11-08 18:11 ` Derek Pagel
2022-11-22 13:23 ` Derek Pagel
2022-11-16 17:19 ` Derek Pagel
2022-11-25 21:54 ` Brian Inglis
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).