Windows Server 2012R2 64bit and 32bit Cygwin sshd

Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

I'm not making progress on this, but I've come to the conclusion that it
is highly unlikely that the existence of the second (64bit) Cygwin
installation has anything to do with the problem, first reported here:

http://thread.gmane.org/gmane.os.cygwin/147823

Meanwhile I've tried to make sure that the two installations don't
interact by creating a separate privileged user for both of them, but
that's not made a difference.

So let me re-phrase the question, given:

- Windows Server 2012R2 64bit running on hardware
- Cygwin 32bit (latest snapshot from 2014-08-18 w/ AD integration)

can anybody confirm they have gotten sshd successfully running?  As far
as I can debug this, the sshd I have installed goes through all the
motions of checking and setting up the environment, but any process it
then spawns to actually run my command or shell immediately exits.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Achim Gratz <Stromeko <at> nexgo.de> writes:
> I'm not making progress on this, but I've come to the conclusion that it
> is highly unlikely that the existence of the second (64bit) Cygwin
> installation has anything to do with the problem, first reported here:
> 
> http://thread.gmane.org/gmane.os.cygwin/147823
> 
> Meanwhile I've tried to make sure that the two installations don't
> interact by creating a separate privileged user for both of them, but
> that's not made a difference.
> 
> So let me re-phrase the question, given:
> 
> - Windows Server 2012R2 64bit running on hardware
> - Cygwin 32bit (latest snapshot from 2014-08-18 w/ AD integration)
> 
> can anybody confirm they have gotten sshd successfully running?  As far
> as I can debug this, the sshd I have installed goes through all the
> motions of checking and setting up the environment, but any process it
> then spawns to actually run my command or shell immediately exits.

I've run strace on both the 32bit and the 64bit sshd and here's the
difference between the two, right after sshd gets an exception 5 from the DC:

32bit
--- Process 5820, exception 00000005 at 75BF1D4D
43239 4372793 [waitproc] sshd 2400 pinfo::maybe_set_exit_code_from_windows:
pid 5820, exit value - old 0x0, windows 0x5, cygwin 0x8000500

64bit
--- Process 484, exception 00000005 at 00007FFA4F5D5BF8
 9907 4522411 [main] sshd 484 seterrno_from_win_error:
.../cygwin-snapshot-20140819-1/winsup/cygwin/sec_auth.cc:263
windows error 5
   36 4522447 [main] sshd 484 geterrno_from_win_error: windows error 5 ==
errno 13

The 64bit sshd then proceeds to fetch my credentials from the registry and
forks the command, while the 32bit winds down and does an exit(5).

Ideas?  I have done a passwd -R in both environments just in case these
would be stored in different registry keys.


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

On Aug 19 13:37, Achim Gratz wrote:
> I've run strace on both the 32bit and the 64bit sshd and here's the
> difference between the two, right after sshd gets an exception 5 from the DC:
> 
> 32bit
> --- Process 5820, exception 00000005 at 75BF1D4D
> 43239 4372793 [waitproc] sshd 2400 pinfo::maybe_set_exit_code_from_windows:
> pid 5820, exit value - old 0x0, windows 0x5, cygwin 0x8000500
> 
> 64bit
> --- Process 484, exception 00000005 at 00007FFA4F5D5BF8
>  9907 4522411 [main] sshd 484 seterrno_from_win_error:
> .../cygwin-snapshot-20140819-1/winsup/cygwin/sec_auth.cc:263
> windows error 5

This is a call to the Win32 API call NetUserGetGroups creating an
exception due to an "access denied".  The difference between 32 and 64
bit is probably a result of the differences in exception handling, and
given that an ExceptionCode 5 will be ignored by Cygwin's exception
handler, it's probably the next handler in the chain.

Are you running sshd under a local cyg_server account or under a domain
cyg_server account?  Is it possible that this is a local cyg_server
account, and as a non-domain account actually gets an "access denied"
when trying to request AD user information?

If your cyg_server is a local account I'd suggest to try with an
AD account per https://cygwin.com/faq/faq.html#faq.using.sshd-in-domain

Other than that I have a funny idea how to workaround this problem from
inside Cygwin.  If you want to give it a try, I'll send you a pointer
to a 32 bit DLL via PM.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen writes:
> This is a call to the Win32 API call NetUserGetGroups creating an
> exception due to an "access denied".  The difference between 32 and 64
> bit is probably a result of the differences in exception handling, and
> given that an ExceptionCode 5 will be ignored by Cygwin's exception
> handler, it's probably the next handler in the chain.

Well, as I said, Cygwin64 dutifully ignores the exception and then goes
to fetch my password from the registry and forks off the command given
to ssh.

> Are you running sshd under a local cyg_server account or under a domain
> cyg_server account?  Is it possible that this is a local cyg_server
> account, and as a non-domain account actually gets an "access denied"
> when trying to request AD user information?

It's a local account, actually I've got a separate account cyg_server32
for that.  I've double checked that this is set up the same way as the
original cyg_server account except for the home directory.

> If your cyg_server is a local account I'd suggest to try with an
> AD account per https://cygwin.com/faq/faq.html#faq.using.sshd-in-domain

I can try but the AD admins are still sitting on their thumbs for the
sshd domain account, so I don't have high hopes for fast enough
turnaround.  Plus, they have new rules that forbid administrative
accounts that have unlimited activation, so I don't know how much use
this is.

> Other than that I have a funny idea how to workaround this problem from
> inside Cygwin.  If you want to give it a try, I'll send you a pointer
> to a 32 bit DLL via PM.

Yes please.  As long as I don't get sshd to work, I can't use this
installation for its intended purpose anyway.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2492 bytes --]

On Aug 19 19:21, Achim Gratz wrote:
> Corinna Vinschen writes:
> > This is a call to the Win32 API call NetUserGetGroups creating an
> > exception due to an "access denied".  The difference between 32 and 64
> > bit is probably a result of the differences in exception handling, and
> > given that an ExceptionCode 5 will be ignored by Cygwin's exception
> > handler, it's probably the next handler in the chain.
> 
> Well, as I said, Cygwin64 dutifully ignores the exception and then goes
> to fetch my password from the registry and forks off the command given
> to ssh.

Yeah, as I said.

> > Are you running sshd under a local cyg_server account or under a domain
> > cyg_server account?  Is it possible that this is a local cyg_server
> > account, and as a non-domain account actually gets an "access denied"
> > when trying to request AD user information?
> 
> It's a local account, actually I've got a separate account cyg_server32
> for that.  I've double checked that this is set up the same way as the
> original cyg_server account except for the home directory.

It doesn't matter if it's the same.  An exception is generated and 32
and 64 bit versions react differently for whatever reason.  It's also
really unnecessary to generate two accounts for this.

> > If your cyg_server is a local account I'd suggest to try with an
> > AD account per https://cygwin.com/faq/faq.html#faq.using.sshd-in-domain
> 
> I can try but the AD admins are still sitting on their thumbs for the
> sshd domain account, so I don't have high hopes for fast enough
> turnaround.  Plus, they have new rules that forbid administrative
> accounts that have unlimited activation, so I don't know how much use
> this is.

Setting up sshd should be the job of an admin in the first place.  The
domain account is a service starter account, not just some arbitrary
administrative account.  That, and much easier maintainance domain-wide
should be blatantly obvious to the admins.

> > Other than that I have a funny idea how to workaround this problem from
> > inside Cygwin.  If you want to give it a try, I'll send you a pointer
> > to a 32 bit DLL via PM.
> 
> Yes please.  As long as I don't get sshd to work, I can't use this
> installation for its intended purpose anyway.

Thanks, coming...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen writes:
> It doesn't matter if it's the same.  An exception is generated and 32
> and 64 bit versions react differently for whatever reason.  It's also
> really unnecessary to generate two accounts for this.

I know that now.  But at one point I was wondering if somehow the fact
that certain settings in the cyg_server user account point to the 64bit
installation was responsible for my problem as it looked like there was
some interaction.

> Setting up sshd should be the job of an admin in the first place.  The
> domain account is a service starter account, not just some arbitrary
> administrative account.  That, and much easier maintainance domain-wide
> should be blatantly obvious to the admins.

I am the admin, but just of that single server and not all the time.  As
I said I can and will ask, but even if the guy I'm talking to agrees it
makes sense that still doesn't mean it's going to happen once it's been
filtered through their set of rules that don't have provisions for
Cygwin in them.  They've already asked why I don't convert that box to
Linux (different rules again which at least know of sshd) and I actually
might end up having to do so in the long run.

> Thanks, coming...

Appreciated.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Achim Gratz <Stromeko <at> nexgo.de> writes:
> - Windows Server 2012R2 64bit running on hardware
> - domain member
> - Cygwin 32bit (latest snapshot from 2014-08-31 w/ AD integration)
> 
> can anybody confirm they have gotten sshd successfully running?  As far
> as I can debug this, the sshd I have installed goes through all the
> motions of checking and setting up the environment, but any process it
> then spawns to actually run my command or shell immediately exits.

The server I was testing this on had to be re-installed for reasons
unrelated to Cygwin, so I've had a fresh start.  The issue remains the same
however: I have successfully set up an sshd for both a productive 64bit
Cygwin and a separate 64bit build environment and can use them both in
parallel.  The 32bit build environment works just fine in an RDP session,
but I can't make the 32bit sshd actually start programs.  As before, it
validates the user logging in successfully and then fails to start the
program requested.  The failure is the same for a local account and a domain
account and regardless of whether I try to log in from the outside or via
localhost.


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Sep  2 11:51, Achim Gratz wrote:
> Achim Gratz <Stromeko <at> nexgo.de> writes:
> > - Windows Server 2012R2 64bit running on hardware
> > - domain member
> > - Cygwin 32bit (latest snapshot from 2014-08-31 w/ AD integration)
> > 
> > can anybody confirm they have gotten sshd successfully running?  As far
> > as I can debug this, the sshd I have installed goes through all the
> > motions of checking and setting up the environment, but any process it
> > then spawns to actually run my command or shell immediately exits.
> 
> The server I was testing this on had to be re-installed for reasons
> unrelated to Cygwin, so I've had a fresh start.  The issue remains the same
> however: I have successfully set up an sshd for both a productive 64bit
> Cygwin and a separate 64bit build environment and can use them both in
> parallel.  The 32bit build environment works just fine in an RDP session,
> but I can't make the 32bit sshd actually start programs.  As before, it
> validates the user logging in successfully and then fails to start the
> program requested.  The failure is the same for a local account and a domain
> account and regardless of whether I try to log in from the outside or via
> localhost.

This is strange, really.  I just set up a Windows 8.1 machine with
two services, one called sshd64 in 64 bit Cygwin, running under my
usual AD account called cyg_server, listening on port 22 and one called
sshd in 32 bit Cygwin, running under the same cyg_server account,
listening on port 20022.  Both services work fine in parallel, and
password as wll as pubkey auth work as expected.

I have no idea what could be wrong in your environment, sorry.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> I have no idea what could be wrong in your environment, sorry.

Me neither.  I've set all three Cygwin environments up exactly the same way,
using identical users and command lines (just different service names,
obviously).

I still have to use a local cyg_server account and a local sshd account, the
latter of which gets mapped from MACHINE+sshd to plain sshd via /etc/passwd
to get sshd to start.  Did the strace I sent you give any clue (I can make
one again if that helps) because it seems that everything just works out as
it should until the fork that should start the command given to ssh.


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

On Sep  2 15:16, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > I have no idea what could be wrong in your environment, sorry.
> 
> Me neither.  I've set all three Cygwin environments up exactly the same way,
> using identical users and command lines (just different service names,
> obviously).
> 
> I still have to use a local cyg_server account and a local sshd account, the
> latter of which gets mapped from MACHINE+sshd to plain sshd via /etc/passwd
> to get sshd to start.

Don't use privilege separation, then the non-privileged sshd user won't
matter at all.  Privsep on Cygwin is only half-useful on Cygwin anyway,
if at all.  As for the local cyg_server account, I'm not sure.  Usually,
a local machine account has no or only limited access to AD information.
As an account which needs AD to get user information it's a bit
unfortunate if it doesn't have access.

>   Did the strace I sent you give any clue (I can make
> one again if that helps) because it seems that everything just works out as
> it should until the fork that should start the command given to ssh.

The strace shows that it doesn't even *try* to start bash, but it's
entirely unclear why.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> I have no idea what could be wrong in your environment, sorry.

Some more information: up until the check for /etc/nologin everything works.
 Things fall apart when sshd tries to switch to the user logging on.  I've
traced this again, sshd apparently then tries to use the user token to set
the primary group and gets the predictable error "0xC000005B --
STATUS_INVALID_PRIMARY_GROUP".  The 64bit sshd catches the resulting
exception in that same pid, sets errno and then goes on to switch the uid. 
In 32bit the exception winds up in the parent process instead and leads to
the termination of the connection.  The seteuid32 call is never initiated.



Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen writes:
> Don't use privilege separation, then the non-privileged sshd user won't
> matter at all.  Privsep on Cygwin is only half-useful on Cygwin anyway,
> if at all.  As for the local cyg_server account, I'm not sure.  Usually,
> a local machine account has no or only limited access to AD information.
> As an account which needs AD to get user information it's a bit
> unfortunate if it doesn't have access.

Unless it's possible to set these rights depending on whether a 32bit or
a 64bit process is run by the user I don't see how this would come into
play.  As I said, it's the exact same user and the exact same setup for
both 64bit and 32bit.

>>   Did the strace I sent you give any clue (I can make
>> one again if that helps) because it seems that everything just works out as
>> it should until the fork that should start the command given to ssh.
>
> The strace shows that it doesn't even *try* to start bash, but it's
> entirely unclear why.

See my other mail, not that it elucidates the reason, but I think I've
pinpointed where it happens now.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]

On Sep  2 16:24, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > I have no idea what could be wrong in your environment, sorry.
> 
> Some more information: up until the check for /etc/nologin everything works.
>  Things fall apart when sshd tries to switch to the user logging on.  I've
> traced this again, sshd apparently then tries to use the user token to set
> the primary group and gets the predictable error "0xC000005B --
> STATUS_INVALID_PRIMARY_GROUP".

This is a red herring.  The 0xC000005B status code returned by
NtSetInformationToken is not at all fatal, nor is it an exception.

The only exceptions in this code occur right after figuring
out the DC, when calling functions to request user or group information
from the DC, NetUserGetGroups and NetUserGetLocalGroups.  But these
are first chance exceptions for the debugger.  They don't occur if
no debugger is present.

However, for some reason, when this exception occurs in the sshd grand
child which is (probably) supposed to execve the shell, this exception
appears to be fatal suddenly.  There's no hint at all why, given that
it's a simple "access denied".

This is really unexpected.  We can try to workaround that, but it's
puzzeling that this should be required at all.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> Don't use privilege separation, then the non-privileged sshd user won't
> matter at all.  Privsep on Cygwin is only half-useful on Cygwin anyway,
> if at all.

I've switched privilege separateion off completely, but no dice.  The Access
Denied comes from trying to switch from primary group "MACHINE+None" to
"Domain Users".  That is expected to happen, what I still don't get is why
the parent process winds up with the exception instead of the originating
process as on 64bit.

> As for the local cyg_server account, I'm not sure.  Usually,
> a local machine account has no or only limited access to AD information.
> As an account which needs AD to get user information it's a bit
> unfortunate if it doesn't have access.

When the process comes to this point it has already verified the user via AD.

> The strace shows that it doesn't even *try* to start bash, but it's
> entirely unclear why.

Is it possible to run sshd in gdb?


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Achim Gratz <Stromeko <at> NexGo.DE> writes:
> > The strace shows that it doesn't even *try* to start bash, but it's
> > entirely unclear why.
> 
> Is it possible to run sshd in gdb?

I can attach the debugger but I didn't manage to break into something useful
(or anything at all,  really).  However, the bug must be triggered by a race
because with the debugger attached once in a blue moon I'll get the command
actually executed correctly.  I'm running this on an 8-core machine, so it
is very likely that each new thread hits a different core...


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

On Sep  3 07:17, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > Don't use privilege separation, then the non-privileged sshd user won't
> > matter at all.  Privsep on Cygwin is only half-useful on Cygwin anyway,
> > if at all.
> 
> I've switched privilege separateion off completely, but no dice.  The Access
> Denied comes from trying to switch from primary group "MACHINE+None" to
> "Domain Users".  That is expected to happen, what I still don't get is why
> the parent process winds up with the exception instead of the originating
> process as on 64bit.

As I wrote, this is a red herring.  A failing setgid is no error at
all.  It has nothing to do with the exception, except for the debug
output preceeding the exception occurance.  Note the

  get_logon_server: DC: server: \\SC301

between them.  This, and the subsequent seterrno_from_win_error in the
parent sshd are a pretty sure sign that the exception is triggered by
the NetUserXXX calls.

> > As for the local cyg_server account, I'm not sure.  Usually,
> > a local machine account has no or only limited access to AD information.
> > As an account which needs AD to get user information it's a bit
> > unfortunate if it doesn't have access.
> 
> When the process comes to this point it has already verified the user via AD.

Yes, but this has nothing to do with it.  Before calling setuid (which
it calls a couple of times during login), sshd calls initgroups for the
new user, POSIX-like.  Initgroups in turn has to call NetUserGetGroups
and NetUserGetLocalGroups on the DC to fetch the full list of groups for
a user.  From the strace it *seems* that the call to NetUserGetGroups in
the grand child sshd process results in simply terminating the process.
The fact that there's no more output *at all* from the grand child
points to Windows killing the process hard.  Lacking any hint why this
occurs, it's just an assumption, of course.

> > The strace shows that it doesn't even *try* to start bash, but it's
> > entirely unclear why.
> 
> Is it possible to run sshd in gdb?

Yes, but Windows/Cygwin gdb don't allow to follow the child process
so it's very tricky.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]

On Sep  3 13:02, Achim Gratz wrote:
> Achim Gratz <Stromeko <at> NexGo.DE> writes:
> > > The strace shows that it doesn't even *try* to start bash, but it's
> > > entirely unclear why.
> > 
> > Is it possible to run sshd in gdb?
> 
> I can attach the debugger but I didn't manage to break into something useful
> (or anything at all,  really).  However, the bug must be triggered by a race
> because with the debugger attached once in a blue moon I'll get the command
> actually executed correctly.  I'm running this on an 8-core machine, so it
> is very likely that each new thread hits a different core...

You already built your own Cygwin DLL, right?  What you could do is to
do some good old printf debugging.  First let's try to find out if it's
really one of the NetUser calls:

Index: sec_auth.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v
retrieving revision 1.57
diff -u -p -r1.57 sec_auth.cc
--- sec_auth.cc	22 May 2014 16:40:13 -0000	1.57
+++ sec_auth.cc	3 Sep 2014 13:37:11 -0000
@@ -256,8 +256,10 @@ get_user_groups (WCHAR *logonserver, cyg
   NET_API_STATUS ret;
 
   /* Look only on logonserver */
+  debug_printf ("Before NetUserGetGroups");
   ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf,
 			  MAX_PREFERRED_LENGTH, &cnt, &tot);
+  debug_printf ("After NetUserGetGroups");
   if (ret)
     {
       __seterrno_from_win_error (ret);
@@ -306,9 +308,11 @@ get_user_local_groups (PWCHAR logonserve
   DWORD cnt, tot;
   NET_API_STATUS ret;
 
+  debug_printf ("Before NetUserGetLocalGroups");
   ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT,
 			       (LPBYTE *) &buf, MAX_PREFERRED_LENGTH,
 			       &cnt, &tot);
+  debug_printf ("After NetUserGetLocalGroups");
   if (ret)
     {
       __seterrno_from_win_error (ret);


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> You already built your own Cygwin DLL, right?  What you could do is to
> do some good old printf debugging.  First let's try to find out if it's
> really one of the NetUser calls:

It looks like I need to install more than the DLL to make this work?  I
couldn't start cygserver as a service with (just) the built DLL in place. 
So I started it in debug mode from the command line (which makes it have
less rights than it needs) and started the sshd in debug mode also.  Due to
presumably the missing rights mentioned I could only log in with an
administrative account (domain account, but restricted to run on  the server
only).  I didn't get any failure from the debug_printf instrumented
functions.  With my normal user account I got a "/bin/bash: Operation not
permitted".  The cygserver debug output also showed unfettered access to the
AD.  With the sshd running without privilege separation I've noticed some
requests to the cygserver that seemed to indicate memory corruption:  Early
on in starting the daemon it would normally try to get account information
for Administrators:544, but the debug output from cygserver was showing
sshdrs as the account name being asked for.  Also there are (probably
unrelated since they are also present on x86_64) complaints about requests
of illegal length (11).

Going back to the original snapshot and using the same debugging setup the
behaviour was still the same.  Since I could now start the services again, I
did that and am back to the original behaviour.  I've asked our IT if there
are restrictions specifically targetting 32bit services or processes, but
got no answer so far (I'm not even sure this is possible).

I'm not sure what to make of these results, but at the moment I've ran out
of time anyway.


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

On Sep  4 11:23, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > You already built your own Cygwin DLL, right?  What you could do is to
> > do some good old printf debugging.  First let's try to find out if it's
> > really one of the NetUser calls:
> 
> It looks like I need to install more than the DLL to make this work?

No.

> I couldn't start cygserver as a service with (just) the built DLL in place.

No idea why.  The patch just adds debug output to strace ouptput, nothing 
else.

> So I started it in debug mode from the command line (which makes it have
> less rights than it needs) and started the sshd in debug mode also.

In a cyg_server GUI session?  If so, you should have all rights required
when starting this in an elevated shell.

> Due to
> presumably the missing rights mentioned I could only log in with an
> administrative account (domain account, but restricted to run on  the server
> only).  I didn't get any failure from the debug_printf instrumented
> functions.  With my normal user account I got a "/bin/bash: Operation not
> permitted".  The cygserver debug output also showed unfettered access to the
> AD.  With the sshd running without privilege separation I've noticed some
> requests to the cygserver that seemed to indicate memory corruption:  Early
> on in starting the daemon it would normally try to get account information
> for Administrators:544, but the debug output from cygserver was showing
> sshdrs as the account name being asked for.  Also there are (probably
> unrelated since they are also present on x86_64) complaints about requests
> of illegal length (11).

Ok, I don't grok this.  If you have trouble with cygserver, which is
completely unrelated to this issue, please discuss this in a new thread
and please describe detailed what you did to provoke the problem and
paste what you see.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > I couldn't start cygserver as a service with (just) the built DLL in place.
> 
> No idea why.  The patch just adds debug output to strace ouptput, nothing 
> else.

Whatever.  I've installed all the binaries from that build and things work
normally now.

> > So I started it in debug mode from the command line (which makes it have
> > less rights than it needs) and started the sshd in debug mode also.
> 
> In a cyg_server GUI session?  If so, you should have all rights required
> when starting this in an elevated shell.

Not the token privileges, I don't think so.  But I'm not sure how to check.

Here's the salient parts from the strace (attaching to the sshd running as a
service in sandbox mode, running with no privilege separation produces a
slightly different trace, but the events leading up to the error are the same):

 262 1161585 [main] sshd 2044 getpid: 2044 = getpid()
10593 1172178 [main] sshd 2044 get_logon_server: DC: server: \\SC301
   58 1172236 [main] sshd 2044 get_user_groups: Before NetUserGetGroups
--- Process 560, exception 00000005 at 75511D4D
 6543 1178779 [main] sshd 2044 get_user_groups: After NetUserGetGroups ret = 5
   56 1178835 [main] sshd 2044 seterrno_from_win_error:
../../../../source/cygwin-snapshot-20140903-1/winsup/cygwin/sec_auth.cc:265
windows error 5
   36 1178871 [main] sshd 2044 geterrno_from_win_error: windows error 5 ==
errno 13
   33 1178904 [main] sshd 2044 get_user_local_groups: Before
NetUserGetLocalGroups
--- Process 560, exception 00000005 at 75511D4D
 7964 1186868 [main] sshd 2044 get_user_local_groups: After
NetUserGetLocalGroups ret = 5
   50 1186918 [main] sshd 2044 seterrno_from_win_error:
../../../../source/cygwin-snapshot-20140903-1/winsup/cygwin/sec_auth.cc:318
windows error 5
   38 1186956 [main] sshd 2044 geterrno_from_win_error: windows error 5 ==
errno 13
   37 1186993 [main] sshd 2044 initgroups32: 0 = initgroups(gratz, 1049089)

It then proceeds to log on via the token and mounts the entries from my
personal fstab (that should fail if it was running as a different user for
some of the entries).  After checking for /etc/nologin this happens:

   35 5023308 [main] sshd 2248 setegid32: new egid: 1049089 current: 197121
   41 5023349 [main] sshd 2248 setegid32: NtSetInformationToken (hProcToken,
TokenPrimaryGroup), 0xC000005B
 3105 5026454 [main] sshd 2248 get_logon_server: DC: server: \\SC301
   44 5026498 [main] sshd 2248 get_user_groups: Before NetUserGetGroups
--- Process 2248, exception 00000005 at 75511D4D

The process apparently gets killed while in the NetUserGetGroups call (much
as you suspected).  I'm not sure this tells us anything new, though. :-(


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Achim Gratz <Stromeko <at> NexGo.DE> writes:
> Not the token privileges, I don't think so.  But I'm not sure how to check.

If I run the sshd as the user trying to log in, then it works; regardless of
whether the user has administrative privileges or not or with or without
privilege separation enabled for the sshd.


Regards,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3115 bytes --]

On Sep  4 14:12, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > > I couldn't start cygserver as a service with (just) the built DLL in place.
> > 
> > No idea why.  The patch just adds debug output to strace ouptput, nothing 
> > else.
> 
> Whatever.  I've installed all the binaries from that build and things work
> normally now.
> 
> > > So I started it in debug mode from the command line (which makes it have
> > > less rights than it needs) and started the sshd in debug mode also.
> > 
> > In a cyg_server GUI session?  If so, you should have all rights required
> > when starting this in an elevated shell.
> 
> Not the token privileges, I don't think so.  But I'm not sure how to check.

Windows whoami /all

> Here's the salient parts from the strace (attaching to the sshd running as a
> service in sandbox mode, running with no privilege separation produces a
> slightly different trace, but the events leading up to the error are the same):
> [...]
>    44 5026498 [main] sshd 2248 get_user_groups: Before NetUserGetGroups
> --- Process 2248, exception 00000005 at 75511D4D
> 
> The process apparently gets killed while in the NetUserGetGroups call (much
> as you suspected).  I'm not sure this tells us anything new, though. :-(

No, it just confirms it.

Please remove the debug_printf patch and try this instead:

Index: sec_auth.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v
retrieving revision 1.57
diff -u -p -r1.57 sec_auth.cc
--- sec_auth.cc	22 May 2014 16:40:13 -0000	1.57
+++ sec_auth.cc	5 Sep 2014 11:12:01 -0000
@@ -20,6 +20,7 @@ details. */
 #include "fhandler.h"
 #include "dtable.h"
 #include "cygheap.h"
+#include "exception.h"
 #include "ntdll.h"
 #include "tls_pbuf.h"
 #include <lm.h>
@@ -255,9 +256,13 @@ get_user_groups (WCHAR *logonserver, cyg
   DWORD cnt, tot, len;
   NET_API_STATUS ret;
 
-  /* Look only on logonserver */
-  ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf,
-			  MAX_PREFERRED_LENGTH, &cnt, &tot);
+  {
+    /* Experimental SEH */
+    exception protect;
+    /* Look only on logonserver */
+    ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf,
+			    MAX_PREFERRED_LENGTH, &cnt, &tot);
+  }
   if (ret)
     {
       __seterrno_from_win_error (ret);
@@ -306,9 +311,14 @@ get_user_local_groups (PWCHAR logonserve
   DWORD cnt, tot;
   NET_API_STATUS ret;
 
-  ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT,
-			       (LPBYTE *) &buf, MAX_PREFERRED_LENGTH,
-			       &cnt, &tot);
+  {
+    /* Experimental SEH */
+    exception protect;
+
+    ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT,
+				 (LPBYTE *) &buf, MAX_PREFERRED_LENGTH,
+				 &cnt, &tot);
+  }
   if (ret)
     {
       __seterrno_from_win_error (ret);


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

On Sep  5 13:16, Corinna Vinschen wrote:
> On Sep  4 14:12, Achim Gratz wrote:
> > Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > > > I couldn't start cygserver as a service with (just) the built DLL in place.
> > > 
> > > No idea why.  The patch just adds debug output to strace ouptput, nothing 
> > > else.
> > 
> > Whatever.  I've installed all the binaries from that build and things work
> > normally now.
> > 
> > > > So I started it in debug mode from the command line (which makes it have
> > > > less rights than it needs) and started the sshd in debug mode also.
> > > 
> > > In a cyg_server GUI session?  If so, you should have all rights required
> > > when starting this in an elevated shell.
> > 
> > Not the token privileges, I don't think so.  But I'm not sure how to check.
> 
> Windows whoami /all
> 
> > Here's the salient parts from the strace (attaching to the sshd running as a
> > service in sandbox mode, running with no privilege separation produces a
> > slightly different trace, but the events leading up to the error are the same):
> > [...]
> >    44 5026498 [main] sshd 2248 get_user_groups: Before NetUserGetGroups
> > --- Process 2248, exception 00000005 at 75511D4D
> > 
> > The process apparently gets killed while in the NetUserGetGroups call (much
> > as you suspected).  I'm not sure this tells us anything new, though. :-(
> 
> No, it just confirms it.
> 
> Please remove the debug_printf patch and try this instead:
> 
> Index: sec_auth.cc
> [...]

For simplicity I added this experimental patch to the snapshot I just
uploaded to https://cygwin.com/snapshots/.

Please give it a try.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Windows Server 2012R2 64bit and 32bit Cygwin sshd

[Achim Gratz]

Corinna Vinschen writes:
> For simplicity I added this experimental patch to the snapshot I just
> uploaded to https://cygwin.com/snapshots/.

I'm ooO so no updates from me on this until I get back.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple