sshd high cpu load

sshd high cpu load

[A. Doggy]

To Cygwin,


I am running cygwin openssh as a windows service. I have been doing so 
for many years with out issue. Recently, I have been running into an 
issue where it maxes out my cpu on any version newer than 8.4p1-1. The 
solution is to downgrade to 8.4p1-1. My server machine is a dell t330 
running windows 10. I am not a business despite using business grade 
hardware.I have tried both 20h2 and 21h1 but no luck. There are no users 
signed in when the issues occur and occurs within minutes of booting up. 
The only change from the default config is I have it running on a 
nonstandard port. Any advice is welcome as I really would like to 
upgrade to a newer version. Thanks

Re: sshd high cpu load

[A. Doggy]

Anyone?

On 5/19/2021 12:48 AM, A. Doggy wrote:
> To Cygwin,
>
>
> I am running cygwin openssh as a windows service. I have been doing so 
> for many years with out issue. Recently, I have been running into an 
> issue where it maxes out my cpu on any version newer than 8.4p1-1. The 
> solution is to downgrade to 8.4p1-1. My server machine is a dell t330 
> running windows 10. I am not a business despite using business grade 
> hardware.I have tried both 20h2 and 21h1 but no luck. There are no 
> users signed in when the issues occur and occurs within minutes of 
> booting up. The only change from the default config is I have it 
> running on a nonstandard port. Any advice is welcome as I really would 
> like to upgrade to a newer version. Thanks
>

RE: [cygwin] Re: sshd high cpu load

[Jason Pyeron]

> -----Original Message-----
> From: A. Doggy via Cygwin
> Sent: Thursday, May 20, 2021 12:03 PM
> 
> Anyone?

Psst, put at bottom.

> 
> On 5/19/2021 12:48 AM, A. Doggy wrote:
> > To Cygwin,
> >
> >
> > I am running cygwin openssh as a windows service. I have been doing so
> > for many years with out issue. Recently, I have been running into an
> > issue where it maxes out my cpu on any version newer than 8.4p1-1.

Observed 100% CPU on server 2012 r2 x64 joined to AD, not internet access. Issue intermittent. Unable to confirm if same issue.

openssh                     8.5p1-1

> > The
> > solution is to downgrade to 8.4p1-1.

Have not had enough time to test - planning on Sunday.

> > My server machine is a dell t330
> > running windows 10. I am not a business despite using business grade
> > hardware.I have tried both 20h2 and 21h1 but no luck. There are no
> > users signed in when the issues occur and occurs within minutes of
> > booting up. The only change from the default config is I have it
> > running on a nonstandard port. Any advice is welcome as I really would
> > like to upgrade to a newer version. Thanks

Our environment is different in almost every metric, testing is planned.

Re: EXTERNAL: Re: sshd high cpu load

[Wells, Roger K.]

On 5/20/21 12:02 PM, A. Doggy via Cygwin wrote:
> Anyone?

Sorry,
I noticed your initial contact and tried to duplicate what you observed
to no avail.
I set up cygwin openssh as a windows service as you described and also
have been doing it this way for many years.
sshd.exe doesn't show any cpu load on task manager even after days (yes
it still works when I log in from another machine)
My system is a Lenovo Thinkpad-x240 running updated W10. Cygwin is at
3.2.0(0.340/5/3)
and ssh is at OpenSSH_8.5p1, OpenSSL 1.1.1f  31 Mar 2020.
Let me know if you would like me to try something else.

>
> On 5/19/2021 12:48 AM, A. Doggy wrote:
>> To Cygwin,
>>
>>
>> I am running cygwin openssh as a windows service. I have been doing
>> so for many years with out issue. Recently, I have been running into
>> an issue where it maxes out my cpu on any version newer than 8.4p1-1.
>> The solution is to downgrade to 8.4p1-1. My server machine is a dell
>> t330 running windows 10. I am not a business despite using business
>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are
>> no users signed in when the issues occur and occurs within minutes of
>> booting up. The only change from the default config is I have it
>> running on a nonstandard port. Any advice is welcome as I really
>> would like to upgrade to a newer version. Thanks
>>
>

-- 
Roger Wells, P.E.
leidos
221 Third St
Newport, RI 02840
401-847-4210 (voice)
401-849-1585 (fax)
roger.k.wells@leidos.com

Re: EXTERNAL: Re: sshd high cpu load

[Andrey Repin]

Greetings, Wells, Roger K.!

>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>>
>>>
>>> I am running cygwin openssh as a windows service. I have been doing
>>> so for many years with out issue. Recently, I have been running into
>>> an issue where it maxes out my cpu on any version newer than 8.4p1-1.
>>> The solution is to downgrade to 8.4p1-1. My server machine is a dell
>>> t330 running windows 10. I am not a business despite using business
>>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are
>>> no users signed in when the issues occur and occurs within minutes of
>>> booting up. The only change from the default config is I have it
>>> running on a nonstandard port. Any advice is welcome as I really
>>> would like to upgrade to a newer version. Thanks

> I noticed your initial contact and tried to duplicate what you observed
> to no avail.

https://cygwin.com/pipermail/cygwin/2021-April/248299.html

> I set up cygwin openssh as a windows service as you described and also
> have been doing it this way for many years.
> sshd.exe doesn't show any cpu load on task manager even after days (yes
> it still works when I log in from another machine)
> My system is a Lenovo Thinkpad-x240 running updated W10. Cygwin is at
> 3.2.0(0.340/5/3)
> and ssh is at OpenSSH_8.5p1, OpenSSL 1.1.1f  31 Mar 2020.
> Let me know if you would like me to try something else.

Connect from remote machine to the usual shell prompt and force kill remote
ssh process.
The hung SSH session will cause full core CPU load.


-- 
With best regards,
Andrey Repin
Thursday, May 20, 2021 23:31:39

Sorry for my terrible english...

RE: [cygwin] Re: EXTERNAL: Re: sshd high cpu load

[Jason Pyeron]

> -----Original Message-----
> From: Andrey Repin
> Sent: Thursday, May 20, 2021 4:33 PM
> To: Wells, Roger K. <ROGER.K.WELLS@leidos.com>; cygwin@cygwin.com
> Cc: Andrey Repin <anrdaemon@yandex.ru>
> Subject: [cygwin] Re: EXTERNAL: Re: sshd high cpu load
> 
> Greetings, Wells, Roger K.!
> 
> >> On 5/19/2021 12:48 AM, A. Doggy wrote:
> >>>
> >>>
> >>> I am running cygwin openssh as a windows service. I have been doing
> >>> so for many years with out issue. Recently, I have been running into
> >>> an issue where it maxes out my cpu on any version newer than 8.4p1-1.
> >>> The solution is to downgrade to 8.4p1-1. My server machine is a dell
> >>> t330 running windows 10. I am not a business despite using business
> >>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are
> >>> no users signed in when the issues occur and occurs within minutes of
> >>> booting up. The only change from the default config is I have it
> >>> running on a nonstandard port. Any advice is welcome as I really
> >>> would like to upgrade to a newer version. Thanks
> 
> > I noticed your initial contact and tried to duplicate what you observed
> > to no avail.
> 
> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
> 
> > I set up cygwin openssh as a windows service as you described and also
> > have been doing it this way for many years.
> > sshd.exe doesn't show any cpu load on task manager even after days (yes
> > it still works when I log in from another machine)
> > My system is a Lenovo Thinkpad-x240 running updated W10. Cygwin is at
> > 3.2.0(0.340/5/3)
> > and ssh is at OpenSSH_8.5p1, OpenSSL 1.1.1f  31 Mar 2020.
> > Let me know if you would like me to try something else.
> 
> Connect from remote machine to the usual shell prompt and force kill remote
> ssh process.
> The hung SSH session will cause full core CPU load.

If this test case works, I will debug the sshd.exe Sunday!!!! My team will be so happy.


This has been our workaround:

$FROM="XXXXX@$([System.Net.Dns]::GetHostByName($env:computerName).HostName)"
$TO="ZZZZ@WWWW"
$process="sshd"
$cpu=90
$maindelay=60
$checkdelay=5
$logging=$false
$LOGFILE="D:\INC000008507666\INC000008507666.log"

while($true)
{
    $killedany=$false
    if ($logging) { echo "$(Get-Date):loop" >> $LOGFILE }
    $x=get-wmiobject Win32_PerfFormattedData_PerfProc_Process | Where-Object -FilterScript { $_.PercentProcessorTime -gt $cpu -and $_.Name.startswith($process) } | foreach { $_.IDProcess }

    if ($x -and $x.Count -gt 0)
    {
        if ($logging) { echo "$(Get-Date):now sleep, found: $x" >> $LOGFILE }
        for ($retry=$checkdelay; $retry -ge 1; $retry=$retry/2)
        {
            if ($logging) { echo "$(Get-Date):check again: $retry" >> $LOGFILE }
            Start-Sleep -s $retry;
            if ($logging) { echo "$(Get-Date):woke" >> $LOGFILE }
            $y=get-wmiobject Win32_PerfFormattedData_PerfProc_Process | Where-Object -FilterScript { $_.PercentProcessorTime -gt $cpu -and $_.Name.startswith($process) } | foreach { $_.IDProcess }
            if ($y -and $y.Count -gt 0)
            {
                if ($logging) { echo "$(Get-Date):still found: $y" >> $LOGFILE }
                $y | foreach {
                    $badpid=$_
                    if ($logging) { echo "$(Get-Date):to kill: $badpid" >> $LOGFILE }
                    $p = Get-Process -Id $badpid
                    try
                    {
                        Stop-Process -Id $badpid -Force -ErrorAction Stop
                        $killedany=$true
                        Send-MailMessage -Subject "INC000008507666: $($env:COMPUTERNAME) killed $($p.Path) with pid $($badpid)" -From "$FROM" -To "$TO"
                    }
                    catch
                    {
                        Get-Date >> "$LOGFILE"
                        $_.Exception.Message >> "$LOGFILE"
                        $_.Exception.ItemName >> "$LOGFILE"
                        $_ >> "$LOGFILE"
                        Get-Date >> "$LOGFILE"

                        Send-MailMessage -Subject "INC000008507666: $($env:COMPUTERNAME) broke, check logs" -From "$FROM" -To "$TO"
                    }
                }
            }
            else
            {
                if ($logging) { echo "$(Get-Date):oh well: $y" >> $LOGFILE }
            }
        }
        if ($killedany -and -not (Test-NetConnection -Port 22 -InformationLevel Quiet -ComputerName 127.0.0.1))
        {
            Start-Service sshd
        }
    }
    Start-Sleep $maindelay
}


--
Jason Pyeron  | Architect
PD Inc        |
10 w 24th St  |
Baltimore, MD |
 
.mil: jason.j.pyeron.ctr@mail.mil
.com: jpyeron@pdinc.us
tel : 202-741-9397

Re: EXTERNAL: Re: sshd high cpu load

[Wells, Roger K.]

On 5/20/21 4:35 PM, Andrey Repin wrote:
> CAUTION: This email originated from outside of Leidos. Be cautious when clicking or opening content.
>
> Greetings, Wells, Roger K.!
>
>>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>>>
>>>> I am running cygwin openssh as a windows service. I have been doing
>>>> so for many years with out issue. Recently, I have been running into
>>>> an issue where it maxes out my cpu on any version newer than 8.4p1-1.
>>>> The solution is to downgrade to 8.4p1-1. My server machine is a dell
>>>> t330 running windows 10. I am not a business despite using business
>>>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are
>>>> no users signed in when the issues occur and occurs within minutes of
>>>> booting up. The only change from the default config is I have it
>>>> running on a nonstandard port. Any advice is welcome as I really
>>>> would like to upgrade to a newer version. Thanks
>> I noticed your initial contact and tried to duplicate what you observed
>> to no avail.
> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
>
>> I set up cygwin openssh as a windows service as you described and also
>> have been doing it this way for many years.
>> sshd.exe doesn't show any cpu load on task manager even after days (yes
>> it still works when I log in from another machine)
>> My system is a Lenovo Thinkpad-x240 running updated W10. Cygwin is at
>> 3.2.0(0.340/5/3)
>> and ssh is at OpenSSH_8.5p1, OpenSSL 1.1.1f  31 Mar 2020.
>> Let me know if you would like me to try something else.
> Connect from remote machine to the usual shell prompt and force kill remote
> ssh process.
> The hung SSH session will cause full core CPU load.
will do & report back
>
>
> --
> With best regards,
> Andrey Repin
> Thursday, May 20, 2021 23:31:39
>
> Sorry for my terrible english...
>

-- 
Roger Wells, P.E.
leidos
221 Third St
Newport, RI 02840
401-847-4210 (voice)
401-849-1585 (fax)
roger.k.wells@leidos.com

Re: sshd high cpu load

[A. Doggy]

To cygwin.


A Little more info: I actually have 1 client that connects regularly to 
the server. That client uses a program called goodsync. When I block 
goodsync from connecting, the problem goes away and I only have 1 
sshd.exe process running. When I unblock goodsync, I have multiple 
sshd.exe processes running and my cpu gets maxed out. Please test it 
like this. Thanks.

On 5/20/2021 12:02 PM, A. Doggy wrote:
> Anyone?
>
> On 5/19/2021 12:48 AM, A. Doggy wrote:
>> To Cygwin,
>>
>>
>> I am running cygwin openssh as a windows service. I have been doing 
>> so for many years with out issue. Recently, I have been running into 
>> an issue where it maxes out my cpu on any version newer than 8.4p1-1. 
>> The solution is to downgrade to 8.4p1-1. My server machine is a dell 
>> t330 running windows 10. I am not a business despite using business 
>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are 
>> no users signed in when the issues occur and occurs within minutes of 
>> booting up. The only change from the default config is I have it 
>> running on a nonstandard port. Any advice is welcome as I really 
>> would like to upgrade to a newer version. Thanks
>>

Re: sshd high cpu load

[Andrey Repin]

Greetings, A. Doggy!

Bottom post please.

> On 5/20/2021 12:02 PM, A. Doggy wrote:
>> Anyone?
>>
>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>> To Cygwin,
>>>
>>>
>>> I am running cygwin openssh as a windows service. I have been doing 
>>> so for many years with out issue. Recently, I have been running into 
>>> an issue where it maxes out my cpu on any version newer than 8.4p1-1. 
>>> The solution is to downgrade to 8.4p1-1. My server machine is a dell 
>>> t330 running windows 10. I am not a business despite using business 
>>> grade hardware.I have tried both 20h2 and 21h1 but no luck. There are 
>>> no users signed in when the issues occur and occurs within minutes of 
>>> booting up. The only change from the default config is I have it 
>>> running on a nonstandard port. Any advice is welcome as I really 
>>> would like to upgrade to a newer version. Thanks
>>>

> A Little more info: I actually have 1 client that connects regularly to
> the server. That client uses a program called goodsync. When I block 
> goodsync from connecting, the problem goes away and I only have 1 
> sshd.exe process running. When I unblock goodsync, I have multiple 
> sshd.exe processes running and my cpu gets maxed out. Please test it 
> like this. Thanks.

That means the program does not terminate SSH session cleanly.


-- 
With best regards,
Andrey Repin
Saturday, May 22, 2021 18:47:32

Sorry for my terrible english...

Re: sshd high cpu load

[A. Doggy]

On 5/20/2021 9:31 PM, A. Doggy wrote:
> To cygwin.
>
>
> A Little more info: I actually have 1 client that connects regularly 
> to the server. That client uses a program called goodsync. When I 
> block goodsync from connecting, the problem goes away and I only have 
> 1 sshd.exe process running. When I unblock goodsync, I have multiple 
> sshd.exe processes running and my cpu gets maxed out. Please test it 
> like this. Thanks.
>
> On 5/20/2021 12:02 PM, A. Doggy wrote:
>> Anyone?
>>
>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>> To Cygwin,
>>>
>>>
>>> I am running cygwin openssh as a windows service. I have been doing 
>>> so for many years with out issue. Recently, I have been running into 
>>> an issue where it maxes out my cpu on any version newer than 
>>> 8.4p1-1. The solution is to downgrade to 8.4p1-1. My server machine 
>>> is a dell t330 running windows 10. I am not a business despite using 
>>> business grade hardware.I have tried both 20h2 and 21h1 but no luck. 
>>> There are no users signed in when the issues occur and occurs within 
>>> minutes of booting up. The only change from the default config is I 
>>> have it running on a nonstandard port. Any advice is welcome as I 
>>> really would like to upgrade to a newer version. Thanks
>>>



To Cygwin,


I remember seeing someone was going to work on this on Sunday. I am 
checking if this is resolved for other users as I do not want to update 
and be stuck. Thanks.

Re: sshd high cpu load

[A. Doggy]

On 5/25/2021 8:23 PM, A. Doggy wrote:
>
> On 5/20/2021 9:31 PM, A. Doggy wrote:
>> To cygwin.
>>
>>
>> A Little more info: I actually have 1 client that connects regularly 
>> to the server. That client uses a program called goodsync. When I 
>> block goodsync from connecting, the problem goes away and I only have 
>> 1 sshd.exe process running. When I unblock goodsync, I have multiple 
>> sshd.exe processes running and my cpu gets maxed out. Please test it 
>> like this. Thanks.
>>
>> On 5/20/2021 12:02 PM, A. Doggy wrote:
>>> Anyone?
>>>
>>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>>> To Cygwin,
>>>>
>>>>
>>>> I am running cygwin openssh as a windows service. I have been doing 
>>>> so for many years with out issue. Recently, I have been running 
>>>> into an issue where it maxes out my cpu on any version newer than 
>>>> 8.4p1-1. The solution is to downgrade to 8.4p1-1. My server machine 
>>>> is a dell t330 running windows 10. I am not a business despite 
>>>> using business grade hardware.I have tried both 20h2 and 21h1 but 
>>>> no luck. There are no users signed in when the issues occur and 
>>>> occurs within minutes of booting up. The only change from the 
>>>> default config is I have it running on a nonstandard port. Any 
>>>> advice is welcome as I really would like to upgrade to a newer 
>>>> version. Thanks
>>>>
>
>
>
> To Cygwin,
>
>
> I remember seeing someone was going to work on this on Sunday. I am 
> checking if this is resolved for other users as I do not want to 
> update and be stuck. Thanks.
>

bump


To Cygwin,


I remember seeing someone was going to work on this on Sunday. I am 
checking if this is resolved for other users as I do not want to update 
and be stuck. Thanks.

Re: sshd high cpu load

[A. Doggy]

On 5/27/2021 7:51 AM, A. Doggy wrote:
>
> On 5/25/2021 8:23 PM, A. Doggy wrote:
>>
>> On 5/20/2021 9:31 PM, A. Doggy wrote:
>>> To cygwin.
>>>
>>>
>>> A Little more info: I actually have 1 client that connects regularly 
>>> to the server. That client uses a program called goodsync. When I 
>>> block goodsync from connecting, the problem goes away and I only 
>>> have 1 sshd.exe process running. When I unblock goodsync, I have 
>>> multiple sshd.exe processes running and my cpu gets maxed out. 
>>> Please test it like this. Thanks.
>>>
>>> On 5/20/2021 12:02 PM, A. Doggy wrote:
>>>> Anyone?
>>>>
>>>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>>>> To Cygwin,
>>>>>
>>>>>
>>>>> I am running cygwin openssh as a windows service. I have been 
>>>>> doing so for many years with out issue. Recently, I have been 
>>>>> running into an issue where it maxes out my cpu on any version 
>>>>> newer than 8.4p1-1. The solution is to downgrade to 8.4p1-1. My 
>>>>> server machine is a dell t330 running windows 10. I am not a 
>>>>> business despite using business grade hardware.I have tried both 
>>>>> 20h2 and 21h1 but no luck. There are no users signed in when the 
>>>>> issues occur and occurs within minutes of booting up. The only 
>>>>> change from the default config is I have it running on a 
>>>>> nonstandard port. Any advice is welcome as I really would like to 
>>>>> upgrade to a newer version. Thanks
>>>>>
>>
>>
>>
>> To Cygwin,
>>
>>
>> I remember seeing someone was going to work on this on Sunday. I am 
>> checking if this is resolved for other users as I do not want to 
>> update and be stuck. Thanks.
>> bump
>
>
> To Cygwin,
>
>
> I remember seeing someone was going to work on this on Sunday. I am 
> checking if this is resolved for other users as I do not want to 
> update and be stuck. Thanks.





double bump


To Cygwin,


I remember seeing someone was going to work on this on Sunday. I am 
checking if this is resolved for other users as I do not want to update 
and be stuck. Thanks.

Re: sshd high cpu load

[A. Doggy]

On 5/28/2021 7:06 PM, A. Doggy wrote:
>
> On 5/27/2021 7:51 AM, A. Doggy wrote:
>>
>> On 5/25/2021 8:23 PM, A. Doggy wrote:
>>>
>>> On 5/20/2021 9:31 PM, A. Doggy wrote:
>>>> To cygwin.
>>>>
>>>>
>>>> A Little more info: I actually have 1 client that connects 
>>>> regularly to the server. That client uses a program called 
>>>> goodsync. When I block goodsync from connecting, the problem goes 
>>>> away and I only have 1 sshd.exe process running. When I unblock 
>>>> goodsync, I have multiple sshd.exe processes running and my cpu 
>>>> gets maxed out. Please test it like this. Thanks.
>>>>
>>>> On 5/20/2021 12:02 PM, A. Doggy wrote:
>>>>> Anyone?
>>>>>
>>>>> On 5/19/2021 12:48 AM, A. Doggy wrote:
>>>>>> To Cygwin,
>>>>>>
>>>>>>
>>>>>> I am running cygwin openssh as a windows service. I have been 
>>>>>> doing so for many years with out issue. Recently, I have been 
>>>>>> running into an issue where it maxes out my cpu on any version 
>>>>>> newer than 8.4p1-1. The solution is to downgrade to 8.4p1-1. My 
>>>>>> server machine is a dell t330 running windows 10. I am not a 
>>>>>> business despite using business grade hardware.I have tried both 
>>>>>> 20h2 and 21h1 but no luck. There are no users signed in when the 
>>>>>> issues occur and occurs within minutes of booting up. The only 
>>>>>> change from the default config is I have it running on a 
>>>>>> nonstandard port. Any advice is welcome as I really would like to 
>>>>>> upgrade to a newer version. Thanks
>>>>>>
>>>
>>>
>>>
>>> To Cygwin,
>>>
>>>
>>> I remember seeing someone was going to work on this on Sunday. I am 
>>> checking if this is resolved for other users as I do not want to 
>>> update and be stuck. Thanks.
>>> bump
>>
>>
>> To Cygwin,
>>
>>
>> I remember seeing someone was going to work on this on Sunday. I am 
>> checking if this is resolved for other users as I do not want to 
>> update and be stuck. Thanks.
>
>
>
>
>
> double bump
>
>
> To Cygwin,
>
>
> I remember seeing someone was going to work on this on Sunday. I am 
> checking if this is resolved for other users as I do not want to 
> update and be stuck. Thanks.
>


To Cygwin,


The problem still exits. Thanks.