Issue with seteuid and openssh

Issue with seteuid and openssh

[Dale Lobb]

Greetings All,

  Has anyone seen an issue similar to this?

  I have a VMWare virtual machine loaded with Windows Server 2016 OS and a Cygwin installation.  Cygwin runs an installed SSHD service via cygrunsrv.exe.  A data gateway engine on a different machine makes regular programmatic connections via SFTP to the server throughout the day.  This setup was established in 2021 and has run without issue for almost a year.

  Last night, the server rebooted automatically after windows updates.  After the reboot, the data gateway was then no longer able to connect to the server.  This condition persisted until I was informed of the issue this morning and connected to the Windows server using RDP to take a look at the issue, at which point the SSH connection suddenly started working.  Further tests showed this to be entirely repeatable.  After rebooting the server, the SSHD daemon does not allow connections, neither with password nor public key authorization, until someone connects to the server via RDP, at which time the SSH connections suddenly starts working again.

  The server's Windows application event log shows numerous errors from the SSHD daemon stating "sshd: PID <####>: fatal: seteuid 197108: No such device or address" during the time frame when SSH connection were not working.  The errors stop immediately when the RDP connection is recorded in the same event log.

  A google search for the error message turned up something somewhat similar from this mailing list back in March of 2019, bit there is no mention of RDP in that exchange.  Also, the advice given, to convert the SSHD service from running under the cyg_server account to LocalSystem, does not apply here, because the Cygwin installation is recent enough that it is already running under LocalSystem.

  When this issue started, the server was running openssh-8.7p1-1.  The server was subsequently updated to the latest, openssh-9.0p1-1, but there has been no change in the observed behavior.

Best Regards,

Dale Lobb



________________________________

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipients and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

Re: Issue with seteuid and openssh

[Achim Gratz]

Dale Lobb via Cygwin writes:
>   Last night, the server rebooted automatically after windows updates.
> After the reboot, the data gateway was then no longer able to connect
> to the server.  This condition persisted until I was informed of the
> issue this morning and connected to the Windows server using RDP to
> take a look at the issue, at which point the SSH connection suddenly
> started working.  Further tests showed this to be entirely repeatable.
> After rebooting the server, the SSHD daemon does not allow
> connections, neither with password nor public key authorization, until
> someone connects to the server via RDP, at which time the SSH
> connections suddenly starts working again.

Check the dependencies on both the cygserver and sshd service
definitions.  You must start them after the network is up (make them
both depend on tcp_ip and sshd depend on cygserver) or they won't work
correctly.  On domain machines that sometimes still isn't quite enough,
so I've set them to "delayed start" in those instances and that seems to
have fixed it.

Also check that the firewall allows connections through whatever port
you've configured sshd to use.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Issue with seteuid and openssh

[Stephen Carrier]

On Wed, May 25, 2022 at 07:51:14PM +0200, Achim Gratz wrote:
> Check the dependencies on both the cygserver and sshd service
> definitions.  You must start them after the network is up (make them
> both depend on tcp_ip and sshd depend on cygserver) or they won't work
> correctly.  On domain machines that sometimes still isn't quite enough,
> so I've set them to "delayed start" in those instances and that seems to
> have fixed it.

In the 2019 occurence of a similar problem, cygserver wasn't being used,
so I would be interested and happy to hear if running it and waiting for
it are enough to fix this.

--Stephen

RE: EXTERNAL SENDER: Re: Issue with seteuid and openssh

[Dale Lobb]

[-- Attachment #1: Type: text/plain, Size: 4597 bytes --]

Greetings All,

  I got the chance to work on this issue some more today.  Unfortunately, none of the suggestions helped.  What I tried:

  1.  I verified that that the windows firewall is allowing port 22 inbound for the cygsshd service.  This has not changed since system inception last year.

  2.  I verified that the cygsshd service has a dependency on the tcpip service and I set cygsshd to delayed automatic start:  no change in behavior after reboot.

  3.  I installed the cygserver service and set it to delayed automatic start.  I changed the cygsshd service back to normal automatic start and added a dependency on the cygserver service: again, no change in behavior after reboot.

  I also verified that a windows console login will end the embargo on cygsshd after reboot, just as an RDP session will.

  Any other ideas?

Best Regards,

Dale Lobb


> -----Original Message-----
> From: Cygwin <cygwin-bounces+dale.lobb=bryanhealth.org@cygwin.com>
> On Behalf Of Stephen Carrier
> Sent: Thursday, May 26, 2022 6:43 PM
> To: Achim Gratz <Stromeko@nexgo.de>
> Cc: cygwin@cygwin.com
> Subject: EXTERNAL SENDER: Re: Issue with seteuid and openssh
>
> In the 2019 occurence of a similar problem, cygserver wasn't being used,
> so I would be interested and happy to hear if running it and waiting for it
> are enough to fix this.
>
> --Stephen
>
>
> On Wed, May 25, 2022 at 07:51:14PM +0200, Achim Gratz wrote:
>>Check the dependencies on both the cygserver and sshd service definitions.
>> You must start them after the network is up (make them both depend
>> on tcp_ip and sshd depend on cygserver) or they won't work correctly.
>>  On domain machines that sometimes still isn't quite enough, so I've set
>> them to "delayed start" in those instances and that seems to have fixed it.
>>
>>Also check that the firewall allows connections through whatever port you've
>> configured sshd to use.
>>
>>Regards,
>>Achim.
>>
>>Dale Lobb wrote on 5/24/2022:
>>>Greetings All,
>>>
>>>  Has anyone seen an issue similar to this?
>>>
>>>  I have a VMWare virtual machine loaded with Windows Server 2016 OS and
>>>a Cygwin installation.  Cygwin runs an installed SSHD service via
>>>cygrunsrv.exe.  A data gateway engine on a different machine makes regular
>>>programmatic connections via SFTP to the server throughout the day.  This
>>>setup was established in 2021 and has run without issue for almost a year.
>>>
>>>  Last night, the server rebooted automatically after windows updates.
>>>After the reboot, the data gateway was then no longer able to connect to
>>>the server.  This condition persisted until I was informed of the issue
>>>this morning and connected to the Windows server using RDP to take a look
>>>at the issue, at which point the SSH connection suddenly started working.
>>>Further tests showed this to be entirely repeatable.  After rebooting the
>>>server, the SSHD daemon does not allow connections, neither with password
>>>nor public key authorization, until someone connects to the server via RDP,
>>>at which time the SSH connections suddenly starts working again.
>>>
>>>  The server's Windows application event log shows numerous errors from
>>>the SSHD daemon stating "sshd: PID <####>: fatal: seteuid 197108: No such
>>>device or address" during the time frame when SSH connection were not
>>>working.  The errors stop immediately when the RDP connection is recorded
>>>in the same event log.
>>>
>>>  A google search for the error message turned up something somewhat
>>>similar from this mailing list back in March of 2019, but there is no
>>>mention of RDP in that exchange.  Also, the advice given, to convert the
>>>SSHD service from running under the cyg_server account to LocalSystem, does
>>>not apply here, because the Cygwin installation is recent enough that it is
>>>already running under LocalSystem.
>>>
>>>  When this issue started, the server was running openssh-8.7p1-1.  The
>>>server was subsequently updated to the latest, openssh-9.0p1-1, but there
>>>has been no change in the observed behavior.
>>>
>>>Best Regards,
>>>
>>>Dale Lobb


________________________________

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipients and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Dale Lobb.vcf --]
[-- Type: text/x-vcard; name="Dale Lobb.vcf", Size: 4581 bytes --]

BEGIN:VCARD
VERSION:2.1
X-MS-SIGNATURE:YES
N;LANGUAGE=en-us:Lobb;Dale
FN:Dale Lobb
ORG:Bryan Health;Information Technology
TITLE:Systems Programmer
TEL;WORK;VOICE:(402) 481-2109
TEL;CELL;VOICE:(402) 730-1050
TEL;WORK;FAX:(402) 481-8354
ADR;WORK;PREF:;;1600 South 48th Street;Lincoln;NE;68506-1299;United States of America
LABEL;WORK;PREF;ENCODING=QUOTED-PRINTABLE:1600 South 48th Street=0D=0A=
Lincoln, NE 68506-1299
X-MS-OL-DEFAULT-POSTAL-ADDRESS:2
URL;WORK:www.bryanhealth.org
EMAIL;PREF;INTERNET:dale.lobb@bryanhealth.org
X-MS-CARDPICTURE;TYPE=JPEG;ENCODING=BASE64:
 /9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAcFBQYFBAcGBQYIBwcIChELCgkJChUPEAwRGBUa
 GRgVGBcbHichGx0lHRcYIi4iJSgpKywrGiAvMy8qMicqKyr/2wBDAQcICAoJChQLCxQqHBgc
 KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKir/wAAR
 CAA1AGoDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAA
 AgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkK
 FhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWG
 h4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl
 5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREA
 AgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYk
 NOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOE
 hYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk
 5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD6RooooAKKM0UAFFFFABSGlpDQBl65rlno
 Gmm+1BisAdUZgM4ycVZsNRtNTtluLCeOeFhkMjZrG8cafZan4YmtNRkkihdl+eMZKnPB/OvH
 2sPE3gKUX+jXJubAnPmw/MhHo69jXbQw0a1Pe0vwZ5tfFyoVbWvH8UfQYpc1534V+K+m615d
 tq22wvCQMsf3bn2Pb8a9BRwwyDkHoa5qtGpRlyzVjso16daPNB3JKKQEHpS1mbAa4o+IPEDe
 NpNAii03Cxed5zq/3ewxnrXa159cfbf+FwT/ANnGHzfsC584HGPwoA0tY8R6v4aktptZtbWe
 wmlELz2rMrRE9CVbORxXWhgcHPWvOvGsWrRxW154iWC40a2nSSaG0JVic4BOeoyaZdWwv/iT
 YQ2uoXiWOoWLTlUmIBBBPHpkYoA9I3qRkMMfWl3D1Fec+LfC39jeCLh9Nu7x5LaUTbmnbdtJ
 wRwelaPiGW3v/BOnR6e0ge+eFLQpIwYFiM8g54Gc0AdoWVRksAPc0pI9a4eSy0601q4h1m/a
 6j2KtrZRF2aJcckhe+e5qLwUz6tp+uabNcXJt4Lto4WZyJEXAIGetAep12qafBqenyW0/CN3
 /unsa4S70jVfDsrS27ma3P3mUZVh/tLT/AWkDW/Cnm6vd3VyBPIio0zADpzxyTWl8PpZXtNV
 sriV54rS9aKLzDuIX0zW9KtKnp0OOvhY1ve2l3OD1Twto3iDdLYFdI1E8mMk+RKf/ZM+1UtL
 8U+KPh/eJY6tG81iDjypTuUL6ow/lyPavQvFtj4fsWV57+DTriU4VCeGJ9hyPrWJM01rbC01
 a3jvtOk+6G+ZSPVW7V60akalPVXXZ/ozxHTlQqb2fdbfNHpunXkd/p8N3Cf3cyB1+hq1iqGj
 QwQaPbRWmRAsYCAnnFX68OWkmfSQu4psU9K4WTTPEUfj2TXbfTbd7dofIETXQViB/FnHH0ru
 qKRZxev6X4h8VWg0ye3tdL0+R1NxIJ/OkdQcgKAoA5A61C2havb+NbHUbTT4vsFjbfZUU3A3
 FMYz+Xau6oNAFe5iiurOWG6UGKRCkin0I5FcB4E0m6bUH+0XC3Ol6PNLFp7gcOzHlvfAyPqT
 XV634Zi1y6ie5vLqOFF2vBFKVWQZ70/UtJdtGhsdI22whliKBeAqq4J/QfjQBgaXpWuaDrWq
 PBZ2t8t9N5kd09xsMf8AssMEnHtRoWl674fu9XaS2t777bKZ1eOYIASOhDdqsf2Bqa3EDzOb
 hY7uSRvnAypxjj8OlLc6LrUtxqsyyR4v7aaFYw5HlnbiM/4/WgBngfTNW8P6DLYajaxGVZHl
 jMc4w5bHHtSeDNK1fSbjU/7TtYY47yczq8cwfbnsRWjJpM93qEF5LAsLpbSxsPMyd527Dx7A
 0aDpOoafPG17cmZRaLFyfusDz9frTA8y1nwWbjxXqOpeJLwxWz3LtBbwtvmlXdx3wq4//VXS
 aJNLcyQ6ZpmnRx6Wh2mDbuG31LHvW0PBrXus3N3qMmInmZkRTyRnjntXT2tnDZQiK1iWNB2A
 r0J4mPIo7v8ABHi08JUdRyeiv82OtoEtbZIYx8qDAqXH1pwzSba85nspK2w6iiigYUUUUAFB
 6UUUAJS4oooAKKKKACiiigAooooA/9k=

X-MS-OL-DESIGN;CHARSET=utf-8:<card xmlns="http://schemas.microsoft.com/office/outlook/12/electronicbusinesscards" ver="1.0" layout="left" bgcolor="ffffff"><img xmlns="" align="mid" area="46" use="cardpicture"/><fld xmlns="" prop="name" align="left" dir="ltr" style="b" color="000000" size="10"/><fld xmlns="" prop="title" align="left" dir="ltr" color="000000" size="8"/><fld xmlns="" prop="dept" align="left" dir="ltr" color="000000" size="8"/><fld xmlns="" prop="org" align="left" dir="ltr" color="000000" size="8"/><fld xmlns="" prop="blank" size="4"/><fld xmlns="" prop="addrwork" align="left" dir="ltr" color="000000" size="7"/><fld xmlns="" prop="telwork" align="left" dir="ltr" color="000000" size="7"><label align="right" color="626262">(Work)</label></fld><fld xmlns="" prop="faxwork" align="left" dir="ltr" color="000000" size="7"><label align="right" color="626262">(Fax)</label></fld><fld xmlns="" prop="telcell" align="left" dir="ltr" color="000000" size="7"><label align="right" color="626262">(Mobile)</label></fld><fld xmlns="" prop="email" align="left" dir="ltr" color="000000" size="7"/><fld xmlns="" prop="webwork" align="left" dir="ltr" color="000000" size="7"/><fld xmlns="" prop="blank" size="8"/><fld xmlns="" prop="blank" size="8"/><fld xmlns="" prop="blank" size="8"/><fld xmlns="" prop="blank" size="8"/><fld xmlns="" prop="blank" size="8"/></card>
REV:20220607T162238Z
END:VCARD

Re: Issue with seteuid and openssh

[Stephen Carrier]

On Tue, May 24, 2022 at 10:15:05PM +0000, Dale Lobb via Cygwin wrote:
> Greetings All,
> 
>   Has anyone seen an issue similar to this?
> 
>   I have a VMWare virtual machine loaded with Windows Server 2016 OS and a Cygwin installation.  Cygwin runs an installed SSHD service via cygrunsrv.exe.  A data gateway engine on a different machine makes regular programmatic connections via SFTP to the server throughout the day.  This setup was established in 2021 and has run without issue for almost a year.
> 
>   Last night, the server rebooted automatically after windows updates.  After the reboot, the data gateway was then no longer able to connect to the server.  This condition persisted until I was informed of the issue this morning and connected to the Windows server using RDP to take a look at the issue, at which point the SSH connection suddenly started working.  Further tests showed this to be entirely repeatable.  After rebooting the server, the SSHD daemon does not allow connections, neither with password nor public key authorization, until someone connects to the server via RDP, at which time the SSH connections suddenly starts working again.
> 
>   The server's Windows application event log shows numerous errors from the SSHD daemon stating "sshd: PID <####>: fatal: seteuid 197108: No such device or address" during the time frame when SSH connection were not working.  The errors stop immediately when the RDP connection is recorded in the same event log.
> 
>   A google search for the error message turned up something somewhat similar from this mailing list back in March of 2019, bit there is no mention of RDP in that exchange.  Also, the advice given, to convert the SSHD service from running under the cyg_server account to LocalSystem, does not apply here, because the Cygwin installation is recent enough that it is already running under LocalSystem.

Do you mean the thread started by this message:

https://cygwin.com/pipermail/cygwin/2019-March/240389.html

which describes a nearly identical problem.  The main difference
is that the problem occored for Windows Server 2008R2 and 2012 but was
not confirmed on Windows Server 2016.  This looks like regression in
Windows so that now the problem occurs in Windows Server 2016 too.

This underlying issue was never addressed or fully understood because
the affected systems were EOL or nearly so.  (and there are awkward
workarounds for making do.)  Looks like WS2016 has been EOL since January,
so maybe no help this time either.

The thread does mention RDP, and sshd service was already running as Local
System, so I wonder if you read a different thread also from March 2019.

2019's problem occured for local accounts only.  Is the new problem
occuring for local accounts only?

2019's problem affected cron similarly to sshd so was a seteuid()
problem and not a sshd problem.  You might check if cron service is
similarly affected.

Hope this helps.

Stephen Carrier
BEAR Center
UC Berkeley