RE: [cygwin] Re: EXTERNAL: Re: sshd high cpu load

["Jason Pyeron" <jpyeron@pdinc.us>]

> -----Original Message-----
> From: Andrey Repin
> Sent: Thursday, May 20, 2021 4:33 PM
> To: Wells, Roger K. <ROGER.K.WELLS@leidos.com>; cygwin@cygwin.com
> Cc: Andrey Repin <anrdaemon@yandex.ru>
> Subject: [cygwin] Re: EXTERNAL: Re: sshd high cpu load
> 
> Greetings, Wells, Roger K.!
> 
> >> On 5/19/2021 12:48 AM, A. Doggy wrote:
> >>>
> >>>
> >>> I am running cygwin openssh as a windows service. I have been doing
> >>> so for many years with out issue. Recently, I have been running into
> >>> an issue where it maxes out my cpu on any version newer than 8.4p1-1.
> >>> The solution is to downgrade to 8.4p1-1. My server machine is a dell
> >>> t330 running windows 10. I am not a business despite using business
> >>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are
> >>> no users signed in when the issues occur and occurs within minutes of
> >>> booting up. The only change from the default config is I have it
> >>> running on a nonstandard port. Any advice is welcome as I really
> >>> would like to upgrade to a newer version. Thanks
> 
> > I noticed your initial contact and tried to duplicate what you observed
> > to no avail.
> 
> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
> 
> > I set up cygwin openssh as a windows service as you described and also
> > have been doing it this way for many years.
> > sshd.exe doesn't show any cpu load on task manager even after days (yes
> > it still works when I log in from another machine)
> > My system is a Lenovo Thinkpad-x240 running updated W10. Cygwin is at
> > 3.2.0(0.340/5/3)
> > and ssh is at OpenSSH_8.5p1, OpenSSL 1.1.1f  31 Mar 2020.
> > Let me know if you would like me to try something else.
> 
> Connect from remote machine to the usual shell prompt and force kill remote
> ssh process.
> The hung SSH session will cause full core CPU load.

If this test case works, I will debug the sshd.exe Sunday!!!! My team will be so happy.


This has been our workaround:

$FROM="XXXXX@$([System.Net.Dns]::GetHostByName($env:computerName).HostName)"
$TO="ZZZZ@WWWW"
$process="sshd"
$cpu=90
$maindelay=60
$checkdelay=5
$logging=$false
$LOGFILE="D:\INC000008507666\INC000008507666.log"

while($true)
{
    $killedany=$false
    if ($logging) { echo "$(Get-Date):loop" >> $LOGFILE }
    $x=get-wmiobject Win32_PerfFormattedData_PerfProc_Process | Where-Object -FilterScript { $_.PercentProcessorTime -gt $cpu -and $_.Name.startswith($process) } | foreach { $_.IDProcess }

    if ($x -and $x.Count -gt 0)
    {
        if ($logging) { echo "$(Get-Date):now sleep, found: $x" >> $LOGFILE }
        for ($retry=$checkdelay; $retry -ge 1; $retry=$retry/2)
        {
            if ($logging) { echo "$(Get-Date):check again: $retry" >> $LOGFILE }
            Start-Sleep -s $retry;
            if ($logging) { echo "$(Get-Date):woke" >> $LOGFILE }
            $y=get-wmiobject Win32_PerfFormattedData_PerfProc_Process | Where-Object -FilterScript { $_.PercentProcessorTime -gt $cpu -and $_.Name.startswith($process) } | foreach { $_.IDProcess }
            if ($y -and $y.Count -gt 0)
            {
                if ($logging) { echo "$(Get-Date):still found: $y" >> $LOGFILE }
                $y | foreach {
                    $badpid=$_
                    if ($logging) { echo "$(Get-Date):to kill: $badpid" >> $LOGFILE }
                    $p = Get-Process -Id $badpid
                    try
                    {
                        Stop-Process -Id $badpid -Force -ErrorAction Stop
                        $killedany=$true
                        Send-MailMessage -Subject "INC000008507666: $($env:COMPUTERNAME) killed $($p.Path) with pid $($badpid)" -From "$FROM" -To "$TO"
                    }
                    catch
                    {
                        Get-Date >> "$LOGFILE"
                        $_.Exception.Message >> "$LOGFILE"
                        $_.Exception.ItemName >> "$LOGFILE"
                        $_ >> "$LOGFILE"
                        Get-Date >> "$LOGFILE"

                        Send-MailMessage -Subject "INC000008507666: $($env:COMPUTERNAME) broke, check logs" -From "$FROM" -To "$TO"
                    }
                }
            }
            else
            {
                if ($logging) { echo "$(Get-Date):oh well: $y" >> $LOGFILE }
            }
        }
        if ($killedany -and -not (Test-NetConnection -Port 22 -InformationLevel Quiet -ComputerName 127.0.0.1))
        {
            Start-Service sshd
        }
    }
    Start-Sleep $maindelay
}


--
Jason Pyeron  | Architect
PD Inc        |
10 w 24th St  |
Baltimore, MD |
 
.mil: jason.j.pyeron.ctr@mail.mil
.com: jpyeron@pdinc.us
tel : 202-741-9397