Hanging cygwin processes in Windows Server 2019

Hanging cygwin processes in Windows Server 2019

[chris.hardison]

My cygwin install looks good and things seem to work perfectly for some time (hours or days), then a cygwin process started by a windows process that is usually a child process of a windows service written in perl will hang and then most all other cygwin processes will hang or take a very long time to return.  The simplest example is a call to \cygwin\bin\ps from powershell. That normally returns with sub-second response time.  Once the problem occurs that call to \cygwin\bin\ps will often take several minutes to return.  I've found that killing all cygwin processes will temporarily resolve the problem.
I've read about redirecting NUL to stdin when making the call to cygwin processes from windows and that didn't seem to help.
I've tried "set-processmitigation -name \cygwin\bin\ps.exe -disable ForceRelocateImages" with no real benefit.
I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup before the sshd service starts and that didn't seem to help.
Any suggestions on how to prevent this problem will be greatly appreciated.
Thanks!chris


$ cygcheck -v /usr/bin/perl
C:\Windows\Cygwin\bin\perl.exe - os=4.0 img=1.0 sys=4.0
  "perl.exe" v0.0 ts=2021-02-06 11:46
  C:\Windows\Cygwin\bin\cygperl5_32.dll - os=4.0 img=1.0 sys=4.0
    "cygperl5_32.dll" v0.0 ts=2021-02-06 11:44
    C:\Windows\Cygwin\bin\cygcrypt-2.dll - os=4.0 img=1.0 sys=4.0
      "cygcrypt-2.dll" v0.0 ts=2021-05-20 09:20
      C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
        "cygwin1.dll" v0.0 ts=2021-03-29 09:37
        C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
          "KERNEL32.dll" v0.0
          C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
            "ntdll.dll" v0.0 ts=2010-05-08 14:49
          C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
            "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
            C:\Windows\system32\ntdll.dll (already done)
        C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\Cygwin\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
        "cyggcc_s-1.dll" v0.0 ts=2020-09-12 12:49
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cygwin1.dll (already done)
    C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\Cygwin\bin\cygwin1.dll (already done)
  C:\Windows\system32\KERNEL32.dll (already done)

$ cygcheck -v /usr/bin/ps
C:\Windows\Cygwin\bin\ps.exe - os=4.0 img=1.0 sys=4.0
  C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
    "cygwin1.dll" v0.0 ts=2021-03-29 09:37
    C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
      "KERNEL32.dll" v0.0
      C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
        "ntdll.dll" v0.0 ts=2010-05-08 14:49
      C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
        "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
        C:\Windows\system32\ntdll.dll (already done)
    C:\Windows\system32\ntdll.dll (already done)
  C:\Windows\system32\ADVAPI32.dll - os=10.0 img=10.0 sys=10.0
    "ADVAPI32.dll" v0.0
    C:\Windows\system32\msvcrt.dll - os=10.0 img=10.0 sys=10.0
      "msvcrt.dll" v0.0
      C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\system32\KERNELBASE.dll (already done)
    C:\Windows\system32\ntdll.dll (already done)
    C:\Windows\system32\KERNELBASE.dll (already done)
    C:\Windows\system32\SECHOST.dll - os=10.0 img=10.0 sys=10.0
      "SECHOST.dll" v0.0
      C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\system32\RPCRT4.dll - os=10.0 img=10.0 sys=10.0
        "RPCRT4.dll" v0.0
        C:\Windows\system32\ntdll.dll (already done)
        C:\Windows\system32\SspiCli.dll - os=10.0 img=10.0 sys=10.0
          "SspiCli.dll" v0.0 ts=2036-05-08 21:49
          C:\Windows\system32\ntdll.dll (already done)
          C:\Windows\system32\RPCRT4.dll (recursive)
          C:\Windows\system32\CRYPTBASE.dll - os=10.0 img=10.0 sys=10.0
            "CRYPTBASE.dll" v0.0
            C:\Windows\system32\ntdll.dll (already done)
            C:\Windows\system32\bcryptPrimitives.dll - os=10.0 img=10.0 sys=10.0
              "bcryptPrimitives.dll" v0.0
              C:\Windows\system32\ntdll.dll (already done)
        C:\Windows\system32\KERNELBASE.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\system32\RPCRT4.dll (already done)
  C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\system32\ntdll.dll (already done)

$ cygcheck -v /usr/bin/ssh
C:\Windows\Cygwin\bin\ssh.exe - os=4.0 img=1.0 sys=4.0
  C:\Windows\Cygwin\bin\cygcrypto-1.1.dll - os=4.0 img=1.0 sys=4.0
    "cygcrypto-1.1.dll" v0.0 ts=2020-04-08 19:02
    C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
      "cygwin1.dll" v0.0 ts=2021-03-29 09:37
      C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
        "KERNEL32.dll" v0.0
        C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
          "ntdll.dll" v0.0 ts=2010-05-08 14:49
        C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
          "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
          C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\system32\ntdll.dll (already done)
    C:\Windows\Cygwin\bin\cygz.dll - os=4.0 img=1.0 sys=4.0
      "cygz.dll" v0.0 ts=2017-03-03 21:51
      C:\Windows\Cygwin\bin\cygwin1.dll (already done)
      C:\Windows\Cygwin\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
        "cyggcc_s-1.dll" v0.0 ts=2020-09-12 12:49
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\Cygwin\bin\cygwin1.dll (already done)
  C:\Windows\Cygwin\bin\cyggssapi_krb5-2.dll - os=4.0 img=1.0 sys=4.0
    "cyggssapi_krb5-2.dll" v0.0 ts=2018-03-16 03:46
    C:\Windows\Cygwin\bin\cygk5crypto-3.dll - os=4.0 img=1.0 sys=4.0
      "cygk5crypto-3.dll" v0.0 ts=2018-03-16 03:45
      C:\Windows\Cygwin\bin\cygkrb5support-0.dll - os=4.0 img=1.0 sys=4.0
        "cygkrb5support-0.dll" v0.0 ts=2018-03-16 03:44
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\Cygwin\bin\cygintl-8.dll - os=4.0 img=1.0 sys=4.0
          "cygintl-8.dll" v0.0 ts=2021-07-29 03:43
          C:\Windows\Cygwin\bin\cygwin1.dll (already done)
          C:\Windows\Cygwin\bin\cygiconv-2.dll - os=4.0 img=1.0 sys=4.0
            "cygiconv-2.dll" v0.0 ts=2020-07-13 13:57
            C:\Windows\Cygwin\bin\cygwin1.dll (already done)
            C:\Windows\system32\KERNEL32.dll (already done)
          C:\Windows\system32\KERNEL32.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\Cygwin\bin\cygwin1.dll (already done)
      C:\Windows\Cygwin\bin\cygintl-8.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cygkrb5-3.dll - os=4.0 img=1.0 sys=4.0
      "cygkrb5-3.dll" v0.0 ts=2018-03-16 03:45
      C:\Windows\Cygwin\bin\cygk5crypto-3.dll (already done)
      C:\Windows\Cygwin\bin\cygkrb5support-0.dll (already done)
      C:\Windows\Cygwin\bin\cygcom_err-2.dll - os=4.0 img=1.0 sys=4.0
        "cygcom_err-2.dll" v0.0 ts=2019-01-02 01:21
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\Cygwin\bin\cygwin1.dll (already done)
      C:\Windows\Cygwin\bin\cygintl-8.dll (already done)
      C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cygkrb5support-0.dll (already done)
    C:\Windows\Cygwin\bin\cygcom_err-2.dll (already done)
    C:\Windows\Cygwin\bin\cygwin1.dll (already done)
    C:\Windows\Cygwin\bin\cygintl-8.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\Cygwin\bin\cygz.dll (already done)
  C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
  C:\Windows\system32\KERNEL32.dll (already done)

Re: Hanging cygwin processes in Windows Server 2019

[Ken Brown]

On 9/30/2021 1:22 PM, chris.hardison--- via Cygwin wrote:
> My cygwin install looks good and things seem to work perfectly for some time (hours or days), then a cygwin process started by a windows process that is usually a child process of a windows service written in perl will hang and then most all other cygwin processes will hang or take a very long time to return.

If there are pipes involved, it's possible that this works better in the test 
release of cygwin-3.3.0:

   https://cygwin.com/pipermail/cygwin/2021-September/249475.html

Could you try that and see if it helps?

Ken

Re: Hanging cygwin processes in Windows Server 2019

[Takashi Yano]

On Thu, 30 Sep 2021 17:22:25 +0000 (UTC)
"chris.hardison--- via Cygwin" wrote:
> My cygwin install looks good and things seem to work perfectly for some time (hours or days), then a cygwin process started by a windows process that is usually a child process of a windows service written in perl will hang and then most all other cygwin processes will hang or take a very long time to return.

The relationship of the processes is not clear.
How many processes are there regarding windows service?
"a cygwin process" :  A (hangs)
"a window process" :  B (starts A)
"a child process"  :  C
"a windows service":  D (starts C)
perl program       :  E

A = C and B = D = E? (two processes)
or
B = C and D = E? (three processes)
or other than that?

If you can provide simple test case, it will help much.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Hanging cygwin processes in Windows Server 2019

[chris.hardison]

 The problem seems to be that my cygwin processes that are always the child or grandchild of a windows service use a console device that is block buffered rather than character buffered.  So using mintty to start those processes forces character buffered which solves the problem.  I'm looking for cleaner way to force character buffered IO.  Here's an example of the change:
From 
chomp(@output = `/cygwin/bin/ps -W`);TO
chomp(@output = `/cygwin/bin/mintty.exe -w hide /bin/dash -c "/bin/ps -W > /tmp/${PID}.out"; sleep 1; type /cygwin/tmp/${PID}.out; del /cygwin/tmp/${PID}.out`);
Hopefully someone can suggest a simpler solution.
Thanks!chris



    On Thursday, September 30, 2021, 01:26:37 PM EDT, chris.hardison--- via Cygwin <cygwin@cygwin.com> wrote:  
 
 My cygwin install looks good and things seem to work perfectly for some time (hours or days), then a cygwin process started by a windows process that is usually a child process of a windows service written in perl will hang and then most all other cygwin processes will hang or take a very long time to return.  The simplest example is a call to \cygwin\bin\ps from powershell. That normally returns with sub-second response time.  Once the problem occurs that call to \cygwin\bin\ps will often take several minutes to return.  I've found that killing all cygwin processes will temporarily resolve the problem.
I've read about redirecting NUL to stdin when making the call to cygwin processes from windows and that didn't seem to help.
I've tried "set-processmitigation -name \cygwin\bin\ps.exe -disable ForceRelocateImages" with no real benefit.
I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup before the sshd service starts and that didn't seem to help.
Any suggestions on how to prevent this problem will be greatly appreciated.
Thanks!chris


$ cygcheck -v /usr/bin/perl
C:\Windows\Cygwin\bin\perl.exe - os=4.0 img=1.0 sys=4.0
  "perl.exe" v0.0 ts=2021-02-06 11:46
  C:\Windows\Cygwin\bin\cygperl5_32.dll - os=4.0 img=1.0 sys=4.0
    "cygperl5_32.dll" v0.0 ts=2021-02-06 11:44
    C:\Windows\Cygwin\bin\cygcrypt-2.dll - os=4.0 img=1.0 sys=4.0
      "cygcrypt-2.dll" v0.0 ts=2021-05-20 09:20
      C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
        "cygwin1.dll" v0.0 ts=2021-03-29 09:37
        C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
          "KERNEL32.dll" v0.0
          C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
            "ntdll.dll" v0.0 ts=2010-05-08 14:49
          C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
            "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
            C:\Windows\system32\ntdll.dll (already done)
        C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\Cygwin\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
        "cyggcc_s-1.dll" v0.0 ts=2020-09-12 12:49
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cygwin1.dll (already done)
    C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\Cygwin\bin\cygwin1.dll (already done)
  C:\Windows\system32\KERNEL32.dll (already done)

$ cygcheck -v /usr/bin/ps
C:\Windows\Cygwin\bin\ps.exe - os=4.0 img=1.0 sys=4.0
  C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
    "cygwin1.dll" v0.0 ts=2021-03-29 09:37
    C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
      "KERNEL32.dll" v0.0
      C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
        "ntdll.dll" v0.0 ts=2010-05-08 14:49
      C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
        "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
        C:\Windows\system32\ntdll.dll (already done)
    C:\Windows\system32\ntdll.dll (already done)
  C:\Windows\system32\ADVAPI32.dll - os=10.0 img=10.0 sys=10.0
    "ADVAPI32.dll" v0.0
    C:\Windows\system32\msvcrt.dll - os=10.0 img=10.0 sys=10.0
      "msvcrt.dll" v0.0
      C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\system32\KERNELBASE.dll (already done)
    C:\Windows\system32\ntdll.dll (already done)
    C:\Windows\system32\KERNELBASE.dll (already done)
    C:\Windows\system32\SECHOST.dll - os=10.0 img=10.0 sys=10.0
      "SECHOST.dll" v0.0
      C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\system32\RPCRT4.dll - os=10.0 img=10.0 sys=10.0
        "RPCRT4.dll" v0.0
        C:\Windows\system32\ntdll.dll (already done)
        C:\Windows\system32\SspiCli.dll - os=10.0 img=10.0 sys=10.0
          "SspiCli.dll" v0.0 ts=2036-05-08 21:49
          C:\Windows\system32\ntdll.dll (already done)
          C:\Windows\system32\RPCRT4.dll (recursive)
          C:\Windows\system32\CRYPTBASE.dll - os=10.0 img=10.0 sys=10.0
            "CRYPTBASE.dll" v0.0
            C:\Windows\system32\ntdll.dll (already done)
            C:\Windows\system32\bcryptPrimitives.dll - os=10.0 img=10.0 sys=10.0
              "bcryptPrimitives.dll" v0.0
              C:\Windows\system32\ntdll.dll (already done)
        C:\Windows\system32\KERNELBASE.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\system32\RPCRT4.dll (already done)
  C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\system32\ntdll.dll (already done)

$ cygcheck -v /usr/bin/ssh
C:\Windows\Cygwin\bin\ssh.exe - os=4.0 img=1.0 sys=4.0
  C:\Windows\Cygwin\bin\cygcrypto-1.1.dll - os=4.0 img=1.0 sys=4.0
    "cygcrypto-1.1.dll" v0.0 ts=2020-04-08 19:02
    C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
      "cygwin1.dll" v0.0 ts=2021-03-29 09:37
      C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
        "KERNEL32.dll" v0.0
        C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
          "ntdll.dll" v0.0 ts=2010-05-08 14:49
        C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
          "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
          C:\Windows\system32\ntdll.dll (already done)
      C:\Windows\system32\ntdll.dll (already done)
    C:\Windows\Cygwin\bin\cygz.dll - os=4.0 img=1.0 sys=4.0
      "cygz.dll" v0.0 ts=2017-03-03 21:51
      C:\Windows\Cygwin\bin\cygwin1.dll (already done)
      C:\Windows\Cygwin\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
        "cyggcc_s-1.dll" v0.0 ts=2020-09-12 12:49
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\Cygwin\bin\cygwin1.dll (already done)
  C:\Windows\Cygwin\bin\cyggssapi_krb5-2.dll - os=4.0 img=1.0 sys=4.0
    "cyggssapi_krb5-2.dll" v0.0 ts=2018-03-16 03:46
    C:\Windows\Cygwin\bin\cygk5crypto-3.dll - os=4.0 img=1.0 sys=4.0
      "cygk5crypto-3.dll" v0.0 ts=2018-03-16 03:45
      C:\Windows\Cygwin\bin\cygkrb5support-0.dll - os=4.0 img=1.0 sys=4.0
        "cygkrb5support-0.dll" v0.0 ts=2018-03-16 03:44
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\Cygwin\bin\cygintl-8.dll - os=4.0 img=1.0 sys=4.0
          "cygintl-8.dll" v0.0 ts=2021-07-29 03:43
          C:\Windows\Cygwin\bin\cygwin1.dll (already done)
          C:\Windows\Cygwin\bin\cygiconv-2.dll - os=4.0 img=1.0 sys=4.0
            "cygiconv-2.dll" v0.0 ts=2020-07-13 13:57
            C:\Windows\Cygwin\bin\cygwin1.dll (already done)
            C:\Windows\system32\KERNEL32.dll (already done)
          C:\Windows\system32\KERNEL32.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\Cygwin\bin\cygwin1.dll (already done)
      C:\Windows\Cygwin\bin\cygintl-8.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cygkrb5-3.dll - os=4.0 img=1.0 sys=4.0
      "cygkrb5-3.dll" v0.0 ts=2018-03-16 03:45
      C:\Windows\Cygwin\bin\cygk5crypto-3.dll (already done)
      C:\Windows\Cygwin\bin\cygkrb5support-0.dll (already done)
      C:\Windows\Cygwin\bin\cygcom_err-2.dll - os=4.0 img=1.0 sys=4.0
        "cygcom_err-2.dll" v0.0 ts=2019-01-02 01:21
        C:\Windows\Cygwin\bin\cygwin1.dll (already done)
        C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
        C:\Windows\system32\KERNEL32.dll (already done)
      C:\Windows\Cygwin\bin\cygwin1.dll (already done)
      C:\Windows\Cygwin\bin\cygintl-8.dll (already done)
      C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
      C:\Windows\system32\KERNEL32.dll (already done)
    C:\Windows\Cygwin\bin\cygkrb5support-0.dll (already done)
    C:\Windows\Cygwin\bin\cygcom_err-2.dll (already done)
    C:\Windows\Cygwin\bin\cygwin1.dll (already done)
    C:\Windows\Cygwin\bin\cygintl-8.dll (already done)
    C:\Windows\system32\KERNEL32.dll (already done)
  C:\Windows\Cygwin\bin\cygz.dll (already done)
  C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
  C:\Windows\system32\KERNEL32.dll (already done)



-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:    https://cygwin.com/ml/#unsubscribe-simple
  

Re: Hanging cygwin processes in Windows Server 2019

[Brian Inglis]

On 2021-10-18 06:01, chris.hardison wrote:
> On Thursday, September 30, 2021, 01:26:37 PM EDT, chris.hardison wrote:
>> My cygwin install looks good and things seem to work perfectly for
some time (hours or days), then a cygwin process started by a windows
process that is usually a child process of a windows service written in
perl will hang and then most all other cygwin processes will hang or
take a very long time to return.  The simplest example is a call to
\cygwin\bin\ps from powershell. That normally returns with sub-second
response time.  Once the problem occurs that call to \cygwin\bin\ps will
often take several minutes to return.  I've found that killing all
cygwin processes will temporarily resolve the problem.
>> I've read about redirecting NUL to stdin when making the call to
cygwin processes from windows and that didn't seem to help.
>> I've tried "set-processmitigation -name \cygwin\bin\ps.exe
>> -disable ForceRelocateImages" with no real benefit.
>> I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup
before the sshd service starts and that didn't seem to help.
>> Any suggestions on how to prevent this problem will be greatly
appreciated.

> The problem seems to be that my cygwin processes that are always the
child or grandchild of a windows service use a console device that is
block buffered rather than character buffered. So using mintty to start
those processes forces character buffered which solves the problem. I'm
looking for cleaner way to force character buffered IO. Here's an
example of the change:
> From
> chomp(@output = `/cygwin/bin/ps -W`);TO
> chomp(@output = `/cygwin/bin/mintty.exe -w hide /bin/dash -c "/bin/ps -W > /tmp/${PID}.out"; sleep 1; type /cygwin/tmp/${PID}.out; del /cygwin/tmp/${PID}.out`);
> Hopefully someone can suggest a simpler solution.

Most Cygwin processes don't care about buffering, as they may be line or 
character buffered from the terminal but block buffered when used with 
pipe or file I/O.
Try using stdbuf(1) instead of mintty to change buffering.
You use it as a command prefix like nohup or time.
Perl may also have ways of changing I/O buffering internally.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Hanging cygwin processes in Windows Server 2019

[chris.hardison]

 Brian,
Thanks for the feedback.  I came across stdbuf earlier today and thought for sure that was the answer but when I showed it to my co-worker he said he had already tied it with no luck, both with -i0 -e0 -o0 and with -oL.
Actually, what is working is:`\\cygwin\\bin\\mintty.exe -w hide /bin/dash -c "/bin/ps -W > /tmp/${PID}.out"`;
chomp(@output = `type \\cygwin\\tmp\\${PID}.out && del \\cygwin\\tmp\\${PID}.out`);
I'm still hoping for a better solution.

Thanks!
chris

    On Tuesday, October 19, 2021, 12:11:42 PM EDT, Brian Inglis <brian.inglis@systematicsw.ab.ca> wrote:  
 
 On 2021-10-18 06:01, chris.hardison wrote:
> On Thursday, September 30, 2021, 01:26:37 PM EDT, chris.hardison wrote:
>> My cygwin install looks good and things seem to work perfectly for
some time (hours or days), then a cygwin process started by a windows
process that is usually a child process of a windows service written in
perl will hang and then most all other cygwin processes will hang or
take a very long time to return.  The simplest example is a call to
\cygwin\bin\ps from powershell. That normally returns with sub-second
response time.  Once the problem occurs that call to \cygwin\bin\ps will
often take several minutes to return.  I've found that killing all
cygwin processes will temporarily resolve the problem.
>> I've read about redirecting NUL to stdin when making the call to
cygwin processes from windows and that didn't seem to help.
>> I've tried "set-processmitigation -name \cygwin\bin\ps.exe
>> -disable ForceRelocateImages" with no real benefit.
>> I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup
before the sshd service starts and that didn't seem to help.
>> Any suggestions on how to prevent this problem will be greatly
appreciated.

> The problem seems to be that my cygwin processes that are always the
child or grandchild of a windows service use a console device that is
block buffered rather than character buffered. So using mintty to start
those processes forces character buffered which solves the problem. I'm
looking for cleaner way to force character buffered IO. Here's an
example of the change:
> From
> chomp(@output = `/cygwin/bin/ps -W`);TO
> chomp(@output = `/cygwin/bin/mintty.exe -w hide /bin/dash -c "/bin/ps -W > /tmp/${PID}.out"; sleep 1; type /cygwin/tmp/${PID}.out; del /cygwin/tmp/${PID}.out`);
> Hopefully someone can suggest a simpler solution.

Most Cygwin processes don't care about buffering, as they may be line or 
character buffered from the terminal but block buffered when used with 
pipe or file I/O.
Try using stdbuf(1) instead of mintty to change buffering.
You use it as a command prefix like nohup or time.
Perl may also have ways of changing I/O buffering internally.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:    https://cygwin.com/ml/#unsubscribe-simple
  

Re: Hanging cygwin processes in Windows Server 2019

[Brian Inglis]

On 2021-10-19 11:56, chris.hardison wrote:
> On Tuesday, October 19, 2021, 12:11:42 PM EDT, Brian Inglis wrote:
> On 2021-10-18 06:01, chris.hardison wrote:
>> On Thursday, September 30, 2021, 01:26:37 PM EDT, chris.hardison wrote:
>>> My cygwin install looks good and things seem to work perfectly
>>> for some time (hours or days), then a cygwin process started by a
>>> windows process that is usually a child process of a windows
>>> service written in perl will hang and then most all other cygwin
>>> processes will hang or take a very long time to return. The
>>> simplest example is a call to \cygwin\bin\ps from powershell.
>>> That normally returns with sub-second response time. Once the
>>> problem occurs that call to \cygwin\bin\ps will often take
>>> several minutes to return. I've found that killing all cygwin
>>> processes will temporarily resolve the problem. >>> I've read about redirecting NUL to stdin when making the call to
>>> cygwin processes from windows and that didn't seem to help.
>>> I've tried "set-processmitigation -name \cygwin\bin\ps.exe 
>>> -disable ForceRelocateImages" with no real benefit.
>>> I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup 
>>> before the sshd service starts and that didn't seem to help.
>>> Any suggestions on how to prevent this problem will be greatly 
>>> appreciated.

>> The problem seems to be that my cygwin processes that are always
>> the child or grandchild of a windows service use a console device
>> that is block buffered rather than character buffered. So using
>> mintty to start those processes forces character buffered which
>> solves the problem. I'm looking for cleaner way to force character
>> buffered IO. Here's an example of the change:
>> From
>> chomp(@output = `/cygwin/bin/ps -W`);TO
>> chomp(@output = `/cygwin/bin/mintty.exe -w hide /bin/dash -c 
>> "/bin/ps -W > /tmp/${PID}.out"; sleep 1; type 
>> /cygwin/tmp/${PID}.out; del /cygwin/tmp/${PID}.out`);
>> Hopefully someone can suggest a simpler solution.

> Most Cygwin processes don't care about buffering, as they may be line
> or character buffered from the terminal but block buffered when used
> with pipe or file I/O.
> Try using stdbuf(1) instead of mintty to change buffering.
> You use it as a command prefix like nohup or time.
> Perl may also have ways of changing I/O buffering internally.

 > Thanks for the feedback.  I came across stdbuf earlier today and 
thought for sure that was the answer but when I showed it to my 
co-worker he said he had already tied it with no luck, both with -i0 -e0 
-o0 and with -oL.
 > Actually, what is working is:`\\cygwin\\bin\\mintty.exe -w hide 
/bin/dash -c "/bin/ps -W > /tmp/${PID}.out"`;
 > chomp(@output = `type \\cygwin\\tmp\\${PID}.out && del 
\\cygwin\\tmp\\${PID}.out`);
 > I'm still hoping for a better solution.

You use perl and Cygwin has package perl-IO-Tty which provides perl 
modules IO:Tty and IO::Pty.
You may be able to build a solution equivalent to mintty using one or 
both of those modules.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]