SSHD causes 100% CPU load

SSHD causes 100% CPU load

[Christoph Lüders]

Hi,

I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
openssh is 8.5p1-1.

I use sshd to access the machine from afar.  I notice (often multiple)
processes of sshd.exe with 100% CPU load.

A stack shown with procexp has function names like
"cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".

Can anyone give me some advice how to proceed further?  I have updated
to the latest cygwin multiple times; reboot doesn't help either.

Best,
Chris

Re: SSHD causes 100% CPU load

[Andrey Repin]

Greetings, Christoph Lüders!

> I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
> openssh is 8.5p1-1.

> I use sshd to access the machine from afar.  I notice (often multiple)
> processes of sshd.exe with 100% CPU load.

https://cygwin.com/pipermail/cygwin/2021-April/248299.html
https://cygwin.com/pipermail/cygwin/2021-April/248309.html

> A stack shown with procexp has function names like
> "cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".

> Can anyone give me some advice how to proceed further?  I have updated
> to the latest cygwin multiple times; reboot doesn't help either.


-- 
With best regards,
Andrey Repin
Wednesday, April 28, 2021 7:21:53

Sorry for my terrible english...

SSHD causes 100% CPU load

[Christoph Lüders]

Hi Andrey,

thanks for the link.  Am I mistaken or was there no solution
mentioned?  Do you have any advice how to fix this problem?

Best,
Chris


Wednesday, April 28, 2021, 06:23:02, "Andrey Repin" wrote:

> Greetings, Christoph Lüders!

>> I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
>> openssh is 8.5p1-1.

>> I use sshd to access the machine from afar.  I notice (often multiple)
>> processes of sshd.exe with 100% CPU load.

> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
> https://cygwin.com/pipermail/cygwin/2021-April/248309.html

>> A stack shown with procexp has function names like
>> "cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".

>> Can anyone give me some advice how to proceed further?  I have updated
>> to the latest cygwin multiple times; reboot doesn't help either.




-- 
* https://cfos.de -- High speed & low ping through traffic shaping
* cFosSpeed 11.10 ==> https://cfos.de/dl
*
* cFos Software GmbH -- Amtsgericht Bonn, HRB 9150
* Geschäftsführer: Christoph Lüders, Martin Winkler

Re: SSHD causes 100% CPU load

[Andrey Repin]

Greetings, Christoph Lüders!

This list is in "no top posting, please" mode.


> Wednesday, April 28, 2021, 06:23:02, "Andrey Repin" wrote:

>> Greetings, Christoph Lüders!

>>> I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
>>> openssh is 8.5p1-1.

>>> I use sshd to access the machine from afar.  I notice (often multiple)
>>> processes of sshd.exe with 100% CPU load.

>> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
>> https://cygwin.com/pipermail/cygwin/2021-April/248309.html

> thanks for the link.  Am I mistaken or was there no solution
> mentioned?  Do you have any advice how to fix this problem?

No, you are not. There were multiple reports (I only listed recent ones), but
no solution yet.

>>> A stack shown with procexp has function names like
>>> "cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".

>>> Can anyone give me some advice how to proceed further?  I have updated
>>> to the latest cygwin multiple times; reboot doesn't help either.


-- 
With best regards,
Andrey Repin
Tuesday, May 4, 2021 18:13:40

Sorry for my terrible english...

Re: SSHD causes 100% CPU load

[Brian Inglis]

On 2021-05-04 09:14, Andrey Repin via Cygwin wrote:
>> Wednesday, April 28, 2021, 06:23:02, "Andrey Repin" wrote:
>>>> I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
>>>> openssh is 8.5p1-1.
>>>> I use sshd to access the machine from afar.  I notice (often multiple)
>>>> processes of sshd.exe with 100% CPU load.

>>> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
>>> https://cygwin.com/pipermail/cygwin/2021-April/248309.html

>> thanks for the link.  Am I mistaken or was there no solution
>> mentioned?  Do you have any advice how to fix this problem?

> No, you are not. There were multiple reports (I only listed recent ones), but
> no solution yet.

>>>> A stack shown with procexp has function names like 
>>>> "cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".

>>>> Can anyone give me some advice how to proceed further?  I have updated 
>>>> to the latest cygwin multiple times; reboot doesn't help either.

If relevant changes are made, they are normally deployed as snapshots: see the 
Cygwin home page link.

Are you using:
* SAM with only local Windows accounts?
* MS accounts?
* AD accounts, without cygserver, and without preloading the cygserver group and 
passwd caches?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: SSHD causes 100% CPU load

[Takashi Yano]

Hi everyone,

On Wed, 28 Apr 2021 07:23:02 +0300
Andrey Repin <anrdaemon@yandex.ru> wrote:

> Greetings, Christoph Lüders!
> 
> > I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
> > openssh is 8.5p1-1.
> 
> > I use sshd to access the machine from afar.  I notice (often multiple)
> > processes of sshd.exe with 100% CPU load.
> 
> https://cygwin.com/pipermail/cygwin/2021-April/248299.html
> https://cygwin.com/pipermail/cygwin/2021-April/248309.html
> 
> > A stack shown with procexp has function names like
> > "cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".
> 
> > Can anyone give me some advice how to proceed further?  I have updated
> > to the latest cygwin multiple times; reboot doesn't help either.

I have finally found how to reproduce this problem in my
environment. The reproduce steps are as follows.

1. Install cygwin sshd as a service and start it.
2. Connect to sshd by 'ssh user@localhost ls' using **Windows** ssh.exe.
3. Check if sshd remains with high CPU load.

I have confirmed that this issue happens in a few my PCs.

Corinna, could you please have a look?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: SSHD causes 100% CPU load

[Corinna Vinschen]

On Aug  5 19:12, Takashi Yano via Cygwin wrote:
> Hi everyone,
> 
> On Wed, 28 Apr 2021 07:23:02 +0300
> Andrey Repin <anrdaemon@yandex.ru> wrote:
> 
> > Greetings, Christoph Lüders!
> > 
> > > I run cygwin 32-bit on Windows 10 Pro Build 19042.  Cygwin is 3.2.0-1,
> > > openssh is 8.5p1-1.
> > 
> > > I use sshd to access the machine from afar.  I notice (often multiple)
> > > processes of sshd.exe with 100% CPU load.
> > 
> > https://cygwin.com/pipermail/cygwin/2021-April/248299.html
> > https://cygwin.com/pipermail/cygwin/2021-April/248309.html
> > 
> > > A stack shown with procexp has function names like
> > > "cygwin1.dll!acl_get_perm" and "cygwin1.dll!_assert".
> > 
> > > Can anyone give me some advice how to proceed further?  I have updated
> > > to the latest cygwin multiple times; reboot doesn't help either.
> 
> I have finally found how to reproduce this problem in my
> environment. The reproduce steps are as follows.
> 
> 1. Install cygwin sshd as a service and start it.
> 2. Connect to sshd by 'ssh user@localhost ls' using **Windows** ssh.exe.
> 3. Check if sshd remains with high CPU load.
> 
> I have confirmed that this issue happens in a few my PCs.
> 
> Corinna, could you please have a look?

Could you perhaps bisect the issue on your machine?

The fact that using Windows ssh is an issue makes me wonder if this is
some kind of pty problem.

The stack info seems fishy to me.  sshd doesn't use any ACL functions.


Corinna

Re: SSHD causes 100% CPU load

[Takashi Yano]

Hi Corinna,

On Thu, 5 Aug 2021 17:07:24 +0200
Corinna Vinschen wrote:
> Could you perhaps bisect the issue on your machine?

In my environment, this does not happen with 8.4p1-1 and 8.4p1-2.
However it happens with 8.5p1-1.

> The fact that using Windows ssh is an issue makes me wonder if this is
> some kind of pty problem.

With command 'ssh user@localhost ls', pty does not allocated.
It seems that this issue does not happen if -t option (allocate
tty) is specified. So I think this is not a pty problem.

In order to look into this problem, I tried to build openssh-8.5p1-1
from source, however it cause the error in configure stage:

...
checking for pkg-config... (cached) /usr/bin/pkg-config
checking if /usr/bin/pkg-config knows about libfido2... no
checking for fido_init in -lfido2... no
configure: error: no usable libfido2 found
*** ERROR: configure failed

despite libfido2-devel had been installed.

$ cygcheck -c libfido2-devel
Cygwin Package Information
Package              Version        Status
libfido2-devel       1.5.0-1        OK

How can I solve this error?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: SSHD causes 100% CPU load

[Takashi Yano]

Hi Corinna,

On Fri, 6 Aug 2021 01:43:31 +0900
Takashi Yano wrote:
> Hi Corinna,
> 
> On Thu, 5 Aug 2021 17:07:24 +0200
> Corinna Vinschen wrote:
> > Could you perhaps bisect the issue on your machine?
> 
> In my environment, this does not happen with 8.4p1-1 and 8.4p1-2.
> However it happens with 8.5p1-1.
> 
> > The fact that using Windows ssh is an issue makes me wonder if this is
> > some kind of pty problem.
> 
> With command 'ssh user@localhost ls', pty does not allocated.
> It seems that this issue does not happen if -t option (allocate
> tty) is specified. So I think this is not a pty problem.
> 
> In order to look into this problem, I tried to build openssh-8.5p1-1
> from source, however it cause the error in configure stage:
> 
> ...
> checking for pkg-config... (cached) /usr/bin/pkg-config
> checking if /usr/bin/pkg-config knows about libfido2... no
> checking for fido_init in -lfido2... no
> configure: error: no usable libfido2 found
> *** ERROR: configure failed
> 
> despite libfido2-devel had been installed.
> 
> $ cygcheck -c libfido2-devel
> Cygwin Package Information
> Package              Version        Status
> libfido2-devel       1.5.0-1        OK
> 
> How can I solve this error?

I looked into the openssh code and might find a calprit.
I guess the following patch solves the issue.

--- serverloop.c.orig	2021-03-02 19:31:47.000000000 +0900
+++ serverloop.c	2021-08-06 10:41:55.439357200 +0900
@@ -333,7 +333,7 @@
 			return -1;
 		} else if (len == -1) {
 			if (errno == EINTR || errno == EAGAIN ||
-			    errno != EWOULDBLOCK)
+			    errno == EWOULDBLOCK)
 				return 0;
 			verbose("Read error from remote host %s port %d: %s",
 			    ssh_remote_ipaddr(ssh), ssh_remote_port(ssh),


This seems to be a upstream bug which has been already fixed.

Please refer to:
https://github.com/openssh/openssh-portable/commit/2dc328023f60212cd29504fc05d849133ae47355

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: SSHD causes 100% CPU load

[Corinna Vinschen]

On Aug  6 10:55, Takashi Yano via Cygwin wrote:
> Hi Corinna,
> 
> On Fri, 6 Aug 2021 01:43:31 +0900
> Takashi Yano wrote:
> > Hi Corinna,
> > 
> > On Thu, 5 Aug 2021 17:07:24 +0200
> > Corinna Vinschen wrote:
> > > Could you perhaps bisect the issue on your machine?
> > 
> > In my environment, this does not happen with 8.4p1-1 and 8.4p1-2.
> > However it happens with 8.5p1-1.
> > [...]
> I looked into the openssh code and might find a calprit.
> I guess the following patch solves the issue.
> 
> --- serverloop.c.orig	2021-03-02 19:31:47.000000000 +0900
> +++ serverloop.c	2021-08-06 10:41:55.439357200 +0900
> @@ -333,7 +333,7 @@
>  			return -1;
>  		} else if (len == -1) {
>  			if (errno == EINTR || errno == EAGAIN ||
> -			    errno != EWOULDBLOCK)
> +			    errno == EWOULDBLOCK)
>  				return 0;
>  			verbose("Read error from remote host %s port %d: %s",
>  			    ssh_remote_ipaddr(ssh), ssh_remote_port(ssh),
> 
> 
> This seems to be a upstream bug which has been already fixed.
> 
> Please refer to:
> https://github.com/openssh/openssh-portable/commit/2dc328023f60212cd29504fc05d849133ae47355

Gosh, looks like I totally missed the 8.6p1 release!

I'll build and upload 8.6p1 ASAP.


Thanks,
Corinna

Re: SSHD causes 100% CPU load

[Takashi Yano]

On Fri, 6 Aug 2021 11:11:05 +0200
Corinna Vinschen wrote:
> On Aug  6 10:55, Takashi Yano via Cygwin wrote:
> > Hi Corinna,
> > 
> > On Fri, 6 Aug 2021 01:43:31 +0900
> > Takashi Yano wrote:
> > > Hi Corinna,
> > > 
> > > On Thu, 5 Aug 2021 17:07:24 +0200
> > > Corinna Vinschen wrote:
> > > > Could you perhaps bisect the issue on your machine?
> > > 
> > > In my environment, this does not happen with 8.4p1-1 and 8.4p1-2.
> > > However it happens with 8.5p1-1.
> > > [...]
> > I looked into the openssh code and might find a calprit.
> > I guess the following patch solves the issue.
> > 
> > --- serverloop.c.orig	2021-03-02 19:31:47.000000000 +0900
> > +++ serverloop.c	2021-08-06 10:41:55.439357200 +0900
> > @@ -333,7 +333,7 @@
> >  			return -1;
> >  		} else if (len == -1) {
> >  			if (errno == EINTR || errno == EAGAIN ||
> > -			    errno != EWOULDBLOCK)
> > +			    errno == EWOULDBLOCK)
> >  				return 0;
> >  			verbose("Read error from remote host %s port %d: %s",
> >  			    ssh_remote_ipaddr(ssh), ssh_remote_port(ssh),
> > 
> > 
> > This seems to be a upstream bug which has been already fixed.
> > 
> > Please refer to:
> > https://github.com/openssh/openssh-portable/commit/2dc328023f60212cd29504fc05d849133ae47355
> 
> Gosh, looks like I totally missed the 8.6p1 release!
> 
> I'll build and upload 8.6p1 ASAP.

I have confirmed that the issue has been fixed in 8.6p1-1.

Thansk.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: SSHD causes 100% CPU load

[Takashi Yano]

On Fri, 6 Aug 2021 01:43:31 +0900
Takashi Yano wrote:
> In order to look into this problem, I tried to build openssh-8.5p1-1
> from source, however it cause the error in configure stage:
> 
> ...
> checking for pkg-config... (cached) /usr/bin/pkg-config
> checking if /usr/bin/pkg-config knows about libfido2... no
> checking for fido_init in -lfido2... no
> configure: error: no usable libfido2 found
> *** ERROR: configure failed
> 
> despite libfido2-devel had been installed.

BTW, does anyone have suggestion for this error?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: SSHD causes 100% CPU load

[Takashi Yano]

On Fri, 6 Aug 2021 20:30:58 +0900
Takashi Yano wrote:
> On Fri, 6 Aug 2021 01:43:31 +0900
> Takashi Yano wrote:
> > In order to look into this problem, I tried to build openssh-8.5p1-1
> > from source, however it cause the error in configure stage:
> > 
> > ...
> > checking for pkg-config... (cached) /usr/bin/pkg-config
> > checking if /usr/bin/pkg-config knows about libfido2... no
> > checking for fido_init in -lfido2... no
> > configure: error: no usable libfido2 found
> > *** ERROR: configure failed
> > 
> > despite libfido2-devel had been installed.
> 
> BTW, does anyone have suggestion for this error?

I found the solution by myself. Installing libcbor-devel package
fixes this error.

Thanks.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: SSHD causes 100% CPU load

[Corinna Vinschen]

On Aug  6 21:35, Takashi Yano via Cygwin wrote:
> On Fri, 6 Aug 2021 20:30:58 +0900
> Takashi Yano wrote:
> > On Fri, 6 Aug 2021 01:43:31 +0900
> > Takashi Yano wrote:
> > > In order to look into this problem, I tried to build openssh-8.5p1-1
> > > from source, however it cause the error in configure stage:
> > > 
> > > ...
> > > checking for pkg-config... (cached) /usr/bin/pkg-config
> > > checking if /usr/bin/pkg-config knows about libfido2... no
> > > checking for fido_init in -lfido2... no
> > > configure: error: no usable libfido2 found
> > > *** ERROR: configure failed
> > > 
> > > despite libfido2-devel had been installed.
> > 
> > BTW, does anyone have suggestion for this error?
> 
> I found the solution by myself. Installing libcbor-devel package
> fixes this error.

I just uploaded libfido2-1.5.0-2, which just adds a dependency from
libfido2-devel to libcbor-devel.


HTH,
Corinna

Re: SSHD causes 100% CPU load

[Corinna Vinschen]

On Aug  6 20:14, Takashi Yano via Cygwin wrote:
> On Fri, 6 Aug 2021 11:11:05 +0200
> Corinna Vinschen wrote:
> > On Aug  6 10:55, Takashi Yano via Cygwin wrote:
> > > Hi Corinna,
> > > 
> > > On Fri, 6 Aug 2021 01:43:31 +0900
> > > Takashi Yano wrote:
> > > > Hi Corinna,
> > > > 
> > > > On Thu, 5 Aug 2021 17:07:24 +0200
> > > > Corinna Vinschen wrote:
> > > > > Could you perhaps bisect the issue on your machine?
> > > > 
> > > > In my environment, this does not happen with 8.4p1-1 and 8.4p1-2.
> > > > However it happens with 8.5p1-1.
> > > > [...]
> > > I looked into the openssh code and might find a calprit.
> > > I guess the following patch solves the issue.
> > > 
> > > --- serverloop.c.orig	2021-03-02 19:31:47.000000000 +0900
> > > +++ serverloop.c	2021-08-06 10:41:55.439357200 +0900
> > > @@ -333,7 +333,7 @@
> > >  			return -1;
> > >  		} else if (len == -1) {
> > >  			if (errno == EINTR || errno == EAGAIN ||
> > > -			    errno != EWOULDBLOCK)
> > > +			    errno == EWOULDBLOCK)
> > >  				return 0;
> > >  			verbose("Read error from remote host %s port %d: %s",
> > >  			    ssh_remote_ipaddr(ssh), ssh_remote_port(ssh),
> > > 
> > > 
> > > This seems to be a upstream bug which has been already fixed.
> > > 
> > > Please refer to:
> > > https://github.com/openssh/openssh-portable/commit/2dc328023f60212cd29504fc05d849133ae47355
> > 
> > Gosh, looks like I totally missed the 8.6p1 release!
> > 
> > I'll build and upload 8.6p1 ASAP.
> 
> I have confirmed that the issue has been fixed in 8.6p1-1.

Great, thanks for testing!


Corinna

Re: SSHD causes 100% CPU load

[ASSI]

Corinna Vinschen via Cygwin writes:
>> I found the solution by myself. Installing libcbor-devel package
>> fixes this error.
>
> I just uploaded libfido2-1.5.0-2, which just adds a dependency from
> libfido2-devel to libcbor-devel.

It would have sufficed to just upload a new hint file…
:-)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: SSHD causes 100% CPU load

[Corinna Vinschen]

On Aug  6 16:16, ASSI wrote:
> Corinna Vinschen via Cygwin writes:
> >> I found the solution by myself. Installing libcbor-devel package
> >> fixes this error.
> >
> > I just uploaded libfido2-1.5.0-2, which just adds a dependency from
> > libfido2-devel to libcbor-devel.
> 
> It would have sufficed to just upload a new hint file…
> :-)

That wouldn't have helped those who already have libfido2-devel
installed ;)


Corinna