DS_FORCE_REDISCOVERY lookup slows ssh logon

DS_FORCE_REDISCOVERY lookup slows ssh logon

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2300 bytes --]

In sec_auth.cc, get_server_groups contains this clause:

  if (get_logon_server (domain, server, false)
      && !get_user_groups (server, grp_list, user, domain)
      && get_logon_server (domain, server, true))
    get_user_groups (server, grp_list, user, domain);

The first call to get_logon_server retrieves cached domain information. We try
to look up user groups based on this information, and if fail to find this group
information (for any reason), we re-query the AD domain, get a new server, and
try again.

get_logon_server is a thin wrapper around DsGetDcName; get_logon_server's third
parameter determines whether we pass the DS_FORCE_REDISCOVERY flag to
DsGetDcName. DsGetDcName's documentation suggests that when doing AD operations,
we first retrieve cached information (by omitting DS_FORCE_REDISCOVERY), try
doing whatever it is that we're going to do, and if we can't reach the domain
controller, ask for another DC name, this time with DS_FORCE_REDISCOVERY, and
having found a better DC, try the operation again.

The problem I'm having is that this rediscover-and-retry step is slowing down my
ssh logons by about three seconds. The DCs on my network (for reasons I don't
understand) reject anonymous connections to PIPE\SAMR, making NetUserGetGroups
fail with ERROR_ACCESS_DENIED. The first call to get_user_groups fails almost
instantaneously, but there's a delay of about three seconds querying the second
server, the one found when we call get_logon_server (domain, server, true), and
this second call also eventually fails with ERROR_ACCESS_DENIED, probably
because the failure is a matter of policy, not of connectivity.

Would it be possible not to make the second call to NetUserGetGroups if the
first fails for a reason that doesn't have anything to do with network
connectivity? The purpose of DS_FORCE_REDISCOVERY seems to be to support
failover, and it doesn't seem useful to try a different server if we
successfully asked the first server and it just happened to say "no".

(By the way: how on earth does logon eventually succeed if group enumeration
fails? I'm using the stored-password authentication method, and when sshd
eventually connects, my user (according to whoami.exe /priv) is a member of the
groups I expect.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: DS_FORCE_REDISCOVERY lookup slows ssh logon

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 3965 bytes --]

On 6/7/2013 11:55 PM, Daniel Colascione wrote:
> (By the way: how on earth does logon eventually succeed if group enumeration
> fails? I'm using the stored-password authentication method, and when sshd
> eventually connects, my user (according to whoami.exe /priv) is a member of the
> groups I expect.)

Ah, I found http://cygwin.com/ml/cygwin/2009-06/msg00828.html. sshd is just
getting a truncated group list from initgroups while checking ~/.ssh
permissions, which still happens to work fine in my case, the logon delay aside.

Changing openssh to call setgroups only after calling seteuid might help (so
we'd retrieve the group list in the context of our new user), but because
get_groups calls deimpersonate before talking to the server, that wouldn't
actually work.

What about something like this?

Index: sec_auth.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v
retrieving revision 1.47
diff -u -r1.47 sec_auth.cc
--- sec_auth.cc	23 Apr 2013 09:44:33 -0000	1.47
+++ sec_auth.cc	8 Jun 2013 08:31:16 -0000
@@ -246,7 +246,8 @@

 static bool
 get_user_groups (WCHAR *logonserver, cygsidlist &grp_list,
-		 PWCHAR user, PWCHAR domain)
+		 PWCHAR user, PWCHAR domain,
+		 struct passwd *pw)
 {
   WCHAR dgroup[MAX_DOMAIN_NAME_LEN + GNLEN + 2];
   LPGROUP_USERS_INFO_0 buf;
@@ -256,6 +257,33 @@
   /* Look only on logonserver */
   ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf,
 			  MAX_PREFERRED_LENGTH, &cnt, &tot);
+
+  if (ret == ERROR_ACCESS_DENIED)
+    {
+      /* If we can't list the user's groups as ourselves, try
+	 impersonating the user and trying again.  If the user is a
+	 domain account and we're just a privileged local account, the
+	 user might have more access than we do. Only try
+	 lsaprivkeyauth because other methods for creating user tokens
+	 don't give us network credentials anyway.
+      */
+
+      HANDLE user_token = lsaprivkeyauth (pw);
+
+      if (user_token)
+	{
+	  if (ImpersonateLoggedOnUser (user_token))
+	    {
+	      ret = NetUserGetGroups (logonserver, user, 0, (LPBYTE *) &buf,
+				      MAX_PREFERRED_LENGTH, &cnt, &tot);
+
+	      RevertToSelf ();
+	    }
+
+	  CloseHandle (user_token);
+	}
+    }
+
   if (ret)
     {
       __seterrno_from_win_error (ret);
@@ -292,7 +320,8 @@

 static bool
 get_user_local_groups (PWCHAR logonserver, PWCHAR domain,
-		       cygsidlist &grp_list, PWCHAR user)
+		       cygsidlist &grp_list, PWCHAR user,
+		       struct passwd *pw)
 {
   LPLOCALGROUP_INFO_0 buf;
   DWORD cnt, tot;
@@ -301,6 +330,29 @@
   ret = NetUserGetLocalGroups (logonserver, user, 0, LG_INCLUDE_INDIRECT,
 			       (LPBYTE *) &buf, MAX_PREFERRED_LENGTH,
 			       &cnt, &tot);
+
+  if (ret == ERROR_ACCESS_DENIED)
+    {
+      /* See the ERROR_ACCESS_DENIED comment in get_user_groups */
+
+      HANDLE user_token = lsaprivkeyauth (pw);
+
+      if (user_token)
+	{
+	  if (ImpersonateLoggedOnUser (user_token))
+	    {
+	      ret = NetUserGetLocalGroups (
+		logonserver, user, 0, LG_INCLUDE_INDIRECT,
+		(LPBYTE *) &buf, MAX_PREFERRED_LENGTH,
+		&cnt, &tot);
+
+	      RevertToSelf ();
+	    }
+
+	  CloseHandle (user_token);
+	}
+    }
+
   if (ret)
     {
       __seterrno_from_win_error (ret);
@@ -482,10 +534,10 @@
       return false;
     }
   if (get_logon_server (domain, server, false)
-      && !get_user_groups (server, grp_list, user, domain)
+      && !get_user_groups (server, grp_list, user, domain, pw)
       && get_logon_server (domain, server, true))
-    get_user_groups (server, grp_list, user, domain);
-  get_user_local_groups (server, domain, grp_list, user);
+    get_user_groups (server, grp_list, user, domain, pw);
+  get_user_local_groups (server, domain, grp_list, user, pw);
   get_unix_group_sidlist (pw, grp_list);
   return true;
 }





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: DS_FORCE_REDISCOVERY lookup slows ssh logon

[Corinna Vinschen]

On Jun  8 01:33, Daniel Colascione wrote:
> On 6/7/2013 11:55 PM, Daniel Colascione wrote:
> > (By the way: how on earth does logon eventually succeed if group enumeration
> > fails? I'm using the stored-password authentication method, and when sshd
> > eventually connects, my user (according to whoami.exe /priv) is a member of the
> > groups I expect.)
> 
> Ah, I found http://cygwin.com/ml/cygwin/2009-06/msg00828.html. sshd is just
> getting a truncated group list from initgroups while checking ~/.ssh
> permissions, which still happens to work fine in my case, the logon delay aside.
> 
> Changing openssh to call setgroups only after calling seteuid might help (so
> we'd retrieve the group list in the context of our new user), but because
> get_groups calls deimpersonate before talking to the server, that wouldn't
> actually work.
> 
> What about something like this?

Hmm.  I'm not so sure.  I think it's a bit of a hack to depend on the
availability of the LSA private key entry for this part of the code.

Actually, the problem you have is based on the fact that you're using a
machine-local cyg_server account to run sshd.  In domain environments
it's prudent to create such an account in AD and add a matching group
policy to make sure that account has the required rights on the machines
which are supposed to run sshd.  I created a short FAQ entry once,
http://cygwin.com/faq.html#faq.using.sshd-in-domain

What probably *does* make sense is not to call get_logon_server twice
if the first call returned with ERROR_ACCESS_DENIED.  That requires 
only a bit of minor code rearranging.  I'll prepare something today
or tomorrow.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: DS_FORCE_REDISCOVERY lookup slows ssh logon

[Corinna Vinschen]

On Jun  8 20:47, Corinna Vinschen wrote:
> On Jun  8 01:33, Daniel Colascione wrote:
> > On 6/7/2013 11:55 PM, Daniel Colascione wrote:
> > > (By the way: how on earth does logon eventually succeed if group enumeration
> > > fails? I'm using the stored-password authentication method, and when sshd
> > > eventually connects, my user (according to whoami.exe /priv) is a member of the
> > > groups I expect.)
> > 
> > Ah, I found http://cygwin.com/ml/cygwin/2009-06/msg00828.html. sshd is just
> > getting a truncated group list from initgroups while checking ~/.ssh
> > permissions, which still happens to work fine in my case, the logon delay aside.
> > 
> > Changing openssh to call setgroups only after calling seteuid might help (so
> > we'd retrieve the group list in the context of our new user), but because
> > get_groups calls deimpersonate before talking to the server, that wouldn't
> > actually work.
> > 
> > What about something like this?
> 
> Hmm.  I'm not so sure.  I think it's a bit of a hack to depend on the
> availability of the LSA private key entry for this part of the code.
> 
> Actually, the problem you have is based on the fact that you're using a
> machine-local cyg_server account to run sshd.  In domain environments
> it's prudent to create such an account in AD and add a matching group
> policy to make sure that account has the required rights on the machines
> which are supposed to run sshd.  I created a short FAQ entry once,
> http://cygwin.com/faq.html#faq.using.sshd-in-domain
> 
> What probably *does* make sense is not to call get_logon_server twice
> if the first call returned with ERROR_ACCESS_DENIED.  That requires 
> only a bit of minor code rearranging.  I'll prepare something today
> or tomorrow.

In facxt, this tiny patch should fix the 3 second timeout:

Index: sec_auth.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v
retrieving revision 1.47
diff -u -p -r1.47 sec_auth.cc
--- sec_auth.cc	23 Apr 2013 09:44:33 -0000	1.47
+++ sec_auth.cc	8 Jun 2013 19:00:46 -0000
@@ -259,8 +259,14 @@ get_user_groups (WCHAR *logonserver, cyg
   if (ret)
     {
       __seterrno_from_win_error (ret);
-      /* It's no error when the user name can't be found. */
-      return ret == NERR_UserNotFound;
+      /* It's no error when the user name can't be found.
+	 It's also no error if access has been denied.  Yes, sounds weird, but
+	 keep in mind that ERROR_ACCESS_DENIED means the current user has no
+	 permission to access the AD user information.  However, if we return
+	 an error, Cygwin will call DsGetDcName with DS_FORCE_REDISCOVERY set
+	 to ask for another server.  This is not only time consuming, it's also
+	 useless; the next server will return access denied again. */
+      return ret == NERR_UserNotFound || ret == ERROR_ACCESS_DENIED;
     }
 
   len = wcslen (domain);

Would you mind to give it a try in your environment?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: DS_FORCE_REDISCOVERY lookup slows ssh logon

[Corinna Vinschen]

Daniel?  Ping?

On Jun  8 21:02, Corinna Vinschen wrote:
> On Jun  8 20:47, Corinna Vinschen wrote:
> > Actually, the problem you have is based on the fact that you're using a
> > machine-local cyg_server account to run sshd.  In domain environments
> > it's prudent to create such an account in AD and add a matching group
> > policy to make sure that account has the required rights on the machines
> > which are supposed to run sshd.  I created a short FAQ entry once,
> > http://cygwin.com/faq.html#faq.using.sshd-in-domain
> > 
> > What probably *does* make sense is not to call get_logon_server twice
> > if the first call returned with ERROR_ACCESS_DENIED.  That requires 
> > only a bit of minor code rearranging.  I'll prepare something today
> > or tomorrow.
> 
> In facxt, this tiny patch should fix the 3 second timeout:
> 
> Index: sec_auth.cc
> ===================================================================
> RCS file: /cvs/src/src/winsup/cygwin/sec_auth.cc,v
> retrieving revision 1.47
> diff -u -p -r1.47 sec_auth.cc
> --- sec_auth.cc	23 Apr 2013 09:44:33 -0000	1.47
> +++ sec_auth.cc	8 Jun 2013 19:00:46 -0000
> @@ -259,8 +259,14 @@ get_user_groups (WCHAR *logonserver, cyg
>    if (ret)
>      {
>        __seterrno_from_win_error (ret);
> -      /* It's no error when the user name can't be found. */
> -      return ret == NERR_UserNotFound;
> +      /* It's no error when the user name can't be found.
> +	 It's also no error if access has been denied.  Yes, sounds weird, but
> +	 keep in mind that ERROR_ACCESS_DENIED means the current user has no
> +	 permission to access the AD user information.  However, if we return
> +	 an error, Cygwin will call DsGetDcName with DS_FORCE_REDISCOVERY set
> +	 to ask for another server.  This is not only time consuming, it's also
> +	 useless; the next server will return access denied again. */
> +      return ret == NERR_UserNotFound || ret == ERROR_ACCESS_DENIED;
>      }
>  
>    len = wcslen (domain);
> 
> Would you mind to give it a try in your environment?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple