Testers needed: New passwd/group handling in Cygwin

Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 27960 bytes --]

Hi folks,


this week I applied the first incarnation of the new passwd/group
handling code to the Cygwin repository and after fixing a crash which
manifested in Denis Excoffier's network, I think we're at a point
which allows to push this forward.

This is a pretty intrusive change, in need of some serious testing, so
I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
http://cygwin.com/snapshots/ contains the changes, including the latest
bugfix.

If you're interested in helping, please read on.  It's a long text, but
I feel it's necessary to explain how this works, so you get a grasp of
what to look out for.  In the long run, this is the basis for the
documentation to go into the User's Guide.

----------------------------------------------------------------------
Please note:

If you have questions or comments, please do *NOT* full quote this mail!
Just quote the important text snippet and ask away.

Thanks for considering.
----------------------------------------------------------------------

Did I mention that this mail is *really* long?  Sorry about that, but I
don't see how to explain all the details otherwise.  Fetch yourself some
drink, preferredly some strong coffee or tea.

Oh, and there be footnotes[0].

===========================
So, what is this all about?
===========================

For as long as Cygwin has existed, it has stored user and group
information in /etc/passwd and /etc/group files.  Under the assumption
that these files would never be too large, the first process in a
process tree, as well as every execing process within the tree would
parse them into structures in memory.  Thus every Cygwin process would
contain an expanded copy of the full information from /etc/passwd and
/etc/group.

This approach has a few downsides.  One of them is that the idea to have
always small files is flawed.  Another one is that reading the entire
file is most of the time entirely useless, since most processes only
need information on their own user and the primary group.  Last but not
least, the passwd and group files have to be maintained separately from
the already existing Windows user databases, the local SAM and Active
Directory.

On the other hand, we have to have a mapping between Windows SIDs and
POSIX uid/gid values (for how this works in Cygwin so far, see [1]), so
we rely on some mechanism to convert SIDs to uid/gid values and vice
versa.

Microsoft's "Services for UNIX" (SFU) (which are deprecated since
Windows 8/Server 2012) never used passwd/group files.  Rather, SFU used
a fixed, computational mapping between SIDs and POSIX uid/gid.  It
allows to generate uid/gid values from SIDs and vice versa.  The
mechanism is documented, albeit in a confusing way and spread over
multiple MSDN articles.  For my changes, I took the liberty to clone the
mapping, with a tiny difference for backward compatibility with existing
Cygwin applications.

===================================
Yes, yes, but how does it work now?
===================================

You *are* comfortable with the concept of SIDs and RIDs, right?  If not,
please read [1] again.  I'll wait...  Finished?  Ok, let's start.

Cygwin's new mapping between SIDs and uid/gid values works in two ways.

- Read /etc/passwd and /etc/group files, like before, mainly for
  backward compatibility.

- If no files are present, or if an entry is missing in the files,
  ask Windows.

At least, that's the default behaviour now.  It will be configurable
using a file /etc/nsswitch.conf, but I'm getting ahead of myself.
Let's stick to the default for now.

If files are present, they will be scanned on demand as soon as a
mapping from SIDs to uid/gid or user names is required.  The new
mechanism will never read the entire file into memory, but only scan
for the requested entry and cache this one in memory[2].

If no entry is found, or no passwd or group file was present, Cygwin
will ask the OS.

  Note:  If the first process in a Cygwin process tree determines
         that no passwd or group file is present, no other process
	 in the entire process tree will try to read the files later
	 on.  This is done for self-preservation.  It's kind of bad
	 if the uid or gid of a user changes during the ride.

So if we've drawn a blank reading the files, we're going to ask the OS.
First thing, we ask the local machine for the SID or the user name. 
The OS functions (LookupAccountSid/LookupAccountName[3]) are kind of
intelligent.  They have all the stuff built in to ask for any account
of the local machine, the Active Directory domain of the machine,
the Global Catalog of the forest of the domain, as well as any
trusted domain of our forest for the information.  One OS call and we're
practically done.  Except...

...the calls only return the mapping between SID, account name and the
account's domain.  We don't have a mapping to POSIX uid/gid and we're
missing information on the user's home dir and login shell.  Let's
discuss the SID<=>uid/gid mapping first.

Here's how the SID<=>uid/gid mapping works.  If you're fuzzy on the
existing well-known SIDs, see [4].

- Well-known SIDs in the NT_AUTHORITY domain of the S-1-5-RID type,
  or aliases of the S-1-5-32-RID type are mapped to the uid/gid value
  RID[5].
  
  Examples:

    "SYSTEM" S-1-5-18     <=> uid/gid: 18
    "Users"  S-1-5-32-545 <=> uid/gid: 545

- Other well-known SIDs in the NT_AUTHORITY domain (S-1-5-X-RID):

    S-1-5-X-RID           <=> uid/gid: 0x1000 * X + RID

  Example:

    "NTLM Authentication"
    S-1-5-64-10           <=> uid/gid: 0x4000A == 262154

- Other well-known SIDs:

    S-1-X-Y               <=> uid/gid: 0x10000 + 0x100 * X + Y

  Example:

    "LOCAL" S-1-2-0       <=> uid/gid: 0x10200 == 66048
    "Creator Group"
    S-1-3-1               <=> uid/gid: 0x10301 == 66305

- Logon SIDs:

    The own LogonSid is converted to the fixed uid 0xfff == 4095 and
    named "CurrentSession".
    Any other LogonSid is converted to the fixed uid 0xffe == 4094
    and named "OtherSession".

- Mandatory Labels:

    S-1-16-RID            <=> uid/gid: 0x60000 + RID

  Example:

    "Medium Mandatory Level"
    S-1-16-8192           <=> uid/gid: 0x62000 == 401408

- Accounts from the local machine's user DB (SAM):

    S-1-5-21-X-Y-Z-RID    <=> 0x30000 + RID

  Example:

    "Administrator"
    S-1-5-X-Y-Z-500       <=> uid/gid: 0x301f4 == 197108

- Accounts from the machine's primary domain:
  
    S-1-5-21-X-Y-Z-RID    <=> 0x100000 + RID

  Example:

    "Domain Users"
    S-1-5-X-Y-Z-513       <=> 0x100201 == 1049089

- Accounts from a trusted domain of the machine's primary domain:

    S-1-5-21-X-Y-Z-RID    <=> trustPosixOffset(domain) + RID

  Hang on... "trustPosixOffset"?  Uh, yes.  This value exists in Windows
  domains already since before Active Directory days.  What happens is
  this.  If you create a domain trust between two domains, a
  trustedDomain entry will be added to both databases.  It describes
  how *this* domain trusts the *other* domain.  One attribute of a trust
  is a 32 bit value called "trustPosixOffset"  For each new trust,
  trustPosixOffset will get some automatic value.  In recent AD domain
  implementations, the first trusted domain will get trustPosixOffset
  set to 0x80000000.  Following domains will get lower values.  Also,
  the domain admins are allowed to set the trustPosixOffset value for
  each trusted domain to some arbitrary 32 bit value, thus allowing any
  kind of collisions between the trustPosixOffsets of domains.  Nice,
  isn't it?  Anyway, as the user of this value, we have to *trust*
  the domain admins to set trustPosixOffset to sensible values, or
  to leave it alone.

  So, for the first (or only) trusted domain of your domain, the
  automatic offset is 0x80000000.  An example for a user of that trusted
  domain is:

    S-1-5-X-Y-Z-1234         <=> uid/gid 0x800004d2 == 2147484882

  There's only one problem with this approach.  Assuming you're
  running in the context of a local user on a domain member machine.
  Local users don't have the right to fetch this kind of domain
  information from the DC, they'll get permission denied.  In this
  case Cygwin will fake a, mostly, sensible trustPosixOffset value
  for this session.

- Local accounts from another machine in the network:

  There's no SID<=>uid/gid mapping implemented for this case.  The
  problem is, there's no way to generate a bijective mapping.  There's
  no central place which keeps an analogue value of the trustPosixOffset,
  and there's the additional problem that the LookupAccountSid and
  LookupAccountName functions cannnot resolve the SIDs, unless they know
  the name of the machine this SID comes from.  And even then it will
  probably suffer a "Permission denied" when trying to ask the machine
  for its local account.

  SFU just prints the account RID in this case, Cygwin maps the account
  to the fake accounts "Unknown+User"/"Unknown+Group" with uid/gid -1.

Ok, now we have a semi-bijective mapping between SIDs and POSIX
uid/gid values, but, given that we have potentially users and groups in
different domains having the same name, how do we uniquely differ
between them by name?  Well, we can do that by making their names
unique in a per-machine way.  Dependent on the domain membership of
the account, and dependent of the machine being a domain member or not,
the user and group names will be generated using a domain prefix and
a separator character between domain and account name.  The default
separator character is the plus sign, '+', as in SFU.

- Well-known SIDs will have the separator character prepended:

    "+SYSTEM", "+LOCAL", "+Medium Mandatory Level", ...

- If the machine is no domain member machine, only local accounts
  can be resolved into names, so for ease of use, just the account names
  are used as Cygwin user/group names:

    "corinna", "bigfoot", "None", ...

- If the machine is a domain member machine, all accounts from
  the primary domain of the machine are mapped to Cygwin names
  without domain prefix:

    "corinna", "bigfoot", "Domain Users", ...

  while accounts from other domains are prepended by their domain:

    "DOMAIN1+corinna", "DOMAIN2+bigfoot", "DOMAIN3+Domain Users", ...

- Local machine accounts of a domain member machine get a Cygwin user
  name the same way as accounts from another domain:  The local machine
  name gets prepended:

    "MYMACHINE+corinna", "MYMACHINE+bigfoot", "MYMACHINE+None", ...

- If LookupAccountSid fails, Cygwin checks the accounts against the
  known trusted domains.  If the account is from one of the trusted
  domains, an artificial account name is created.  It consists of
  the domain name, and a special name created from the account RID:

    "MY_DOM+User(1234)", "MY_DOM+Group(5678)"

  Otherwise we know nothing about this SID, so it will be mapped to
  the fake accounts "Unknown+User"/"Unknown+Group" with uid/gid -1.

Any questions?  No?  Ok, let's have a short break and then, next point.

[...5 minutes commercials...]

==========================================
Cygwin user names, home dirs, login shells
==========================================

"That's all very interesting" you think (if you're still awake, that
is), "but passwd entries consist of more than just uids.  What's up
with that?"

Obviously, if you don't maintain passwd and group files, you need
to have a way to maintain the other fields of a passwd entry as well.
Three things come to mind:

- You want to use a Cygwin user name different from your Windows
  user name.

- You want a home dir different from the default /home/$USER.

- You want to specify a different login shell than /bin/sh.

How this is done depends on your account being a domain account or a
local account.  Let's start with the default.  Assuming your Windows
account name is "bigfoot" and your domain is "MY_DOM".  Your default
passwd entry in absence of anything I'll describe below looks like this:

  bigfoot:*:<uid>:<gid>:U-MY_DOM\bigfoot,S-1-5-....:/home/bigfoot:/bin/sh

or, if your account is from a different domain than the primary domain of
the machine:

  MY_DOM+bigfoot:*:<uid>:<gid>:U-MY_DOM\bigfoot,S-1-5-....:/home/bigfoot:/bin/sh

Yes, the default homedir is still /home/bigfoot.

If your account is a domain account:

  Either create an /etc/passwd and/or /etc/group file with entries for
  your account and use that, just as before.

  Or, Cygwin will utilize the posixAccount/posixGroup attributes per RFC
  2307[6].  These attributes are by default available in Active Directory
  since Windows Server 2003 R2.  They are "not set", unless utilized by
  the (deprecated since Server 2012 R2) Active Directory "Server for
  NIS" feature.  The user attributes utilized by Cygwin are:

    uid                If set, will be used as Cygwin user name.
    uidNumber          See next section.
    gecos              Content will be added to the pw_gecos field.
    unixHomeDirectory  If set, will be used as Cygwin home directory.
    loginShell         If set, will be used as Cygwin login shell.

  The group attributes utilized by Cygwin are:

    cn                 If set, will be used as Cygwin group name.
    gidNumber          See next section.

  Apart from power shell scripting or inventing new CLI tools, these
  attributes can be changed using the "Attribute Editor" tab in the user
  properties dialog of the "Active Directory Users and Computers"
  MMC snap-in.  Alternatively, if the "Server for NIS" administration
  feature has been installed, ther will be a "UNIX Attributes" tab which
  contains the required fields, except for the gecos field, which isn't
  really important anyway.  Last resort is "ADSI Edit".

  The primary group of a user is always the Windows primary group set in
  Active Directory and can't be changed.

If your machine is not a domain member machine or your account is a
local account for some reason:

  Either create an /etc/passwd and/or /etc/group file with entries for
  your account and use that, just as before.

  Or enter the information into the "Comment" field of your local user
  entry.  In the "Local Users and Groups" MMC snap-in it's called
  "Description".

  You can utilze this field even if you're running a "home edition" of
  Windows, using the command line.  The "net user" command allows to set
  all values in the SAM, even if the GUI is crippled.

  A Cygwin SAM comment entry looks like this:

    <cygwin key="value" key="value" [...] />

  The supported keys are

    name="value"      Sets the Cygwin user name to value.

    home="value"      Sets the Cygwin home dir to value.

    shell="value"     Sets the Cygwin login shell to value.

    group="value"     Sets the Cygwin primary group of the account
                      to value, provided that the user *is* already
                      a member of that group.  This allows to
                      override the default "None" primary group for
                      local accounts.
		      One nice idea here is, for instance group="Users".
		      This is the *Windows* name of the group, not the
		      Cygwin name, assuming they differ.

    unix="value"      Sets the NFS/Samba uid of the user to the decimal
                      value.  See the next chapter.

  The <cygwin .../> string can start at any point in the comment, but
  you have to follow the rules:

  - It starts with "<cygwin " and ends with "/>".
  - The "cygwin" string and the key names have to be lowercase.
  - No spaces between key and "value", just the equal sign.
  - The value must be placed within double quotes and it must not
    contain a double quote itself.  The double quotes are required
    for the decimal values as well!

  CMD example:

    net use corinna /comment:"<cygwin home=\"/home/foo\"/>"

  Bash example (use single quotes):

    net use corinna /comment:'<cygwin home="/home/foo"/>'

  For changing group comments, use the `net localgroup' command.  The
  supported key/value pair for groups are

    name="value"      Sets the Cygwin group name to value.

    unix="value"      Sets the NFS/Samba gid of the group to the
                      decimal value.  See the next chapter.


=============================
NFS and Samba account mapping
=============================

If you're using Microsoft's NFS client to access UNIX shares, like me,
you might have been bothered by each file on the share seemingly being
yours.  The reason for this is that Microsoft's NFS client does not map
the uid/gid values on the NFS shares to SIDs.  There's no such thing as
a (fake) security descriptor returned to the application.  Rather, via
an undocumented API you can fetch RFC 1813 compatible NFSv3 stat
information from the share[7].  This is what Cygwin is using to show
stat information for files on NFS shares.

The problem is this.  While all other information in this stat record,
like timestamps, or file size etc., can be used by Cygwin, Cygwin had no
way to map the values of the st_uid and st_gid members to a Windows SID.
So it just faked the file owner info and claimed that it's you.

However, SFU has, over time, developed multiple methods to map UNIX
uid/gid values on NFS shares to Windows SIDs.  You'll find the full
documentation of the mapping methods in [8].

Cygwin now utilizes the RFC 2307 mapping for this purpose.  This is most
of the time provided by an AD domain, but it could also be a standalone
LDAP mapping server.  Per RFC 2307, the uid is in the attribute
uidNumber.  For groups, the gid is in the gidNumber attribute.

When Cygwin stat's files on an NFS share, it asks the mapping server via
LDAP in two different ways, depending on the role of the mapping server.

- If the server is an AD domain controller, it asks for an account with
  uidNumber attribute == st_uid field of the stat record returned by
  NFS.  If an account matches, AD returns the Windows SID, so we have an
  immediate mapping from UNIX uid to a Windows SID, if the user account
  has a valid uidNumber attribute.  For groups, the method is the same,
  just that Cygwin asks for a group with gidNumber attribute == st_gid
  field of the stat record.

- If the server is a standalone LDAP mapping server Cygwin asks for
  the same uidNumber/gidNumber attributes, but it can't expect that
  the LDAP server knows anything about Windows SIDs.  Rather, the
  mapping server returns the account name.  Cygwin then asks the DC for
  an account with this name, and if that succeeds, we have a mapping
  between UNIX uid/gid and Windows SIDs.
  
The mapping will be cached for the lifetime of the process, and inherited
by child processes.

And what about Samba?

A fully set up Samba with domain integration is running winbindd to
map Window SIDs to artificially created UNIX uids and gids, and this
mapping is transparent within the domain, so Cygwin doesn't have to do
anything special.

However, setting up winbindd isn't for everybody, and it fails to map
Windows accounts to already existing UNIX users or groups.  In contrast
to NFS, Samba returns security descriptors, but unmapped UNIX accounts
get special SIDs:

- A UNIX user account with uid X is mapped to the Windows SID S-1-22-1-X.

- A UNIX group account with gid X is mapped to SID S-1-22-2-X.

As you can see, even though we have SIDs, they just reflect the actual
uid/gid values on the UNIX box in the RID value.  It's only marginally
different from the NFS method, so... why not just do the same rote as
for NFS?

That's what Cygwin will do.  If it encounters a S-1-22-x-y SID, it
will perform the same RFC 2307 mapping as for NFS shares.

Hang on.  What about home users without any Windows domain or LDAP
server per RFC 2307, but with a Linux machine running Samba?

Not all is lost.  If you're a user on a standalone machine, you
can add this information to your SAM account.  Assuming the uid of
your Linux user account is 505 and the gid of your primary group is,
say, 100, just add the values to your SAM user and group accounts.

The following example assumes you didn't already add something else
to the comment field.

To your user's SAM comment (remember: called "Description" in the GUI),
add:

  <cygwin group="Users" unix="505"/>

To the user's group SAM comment add:

  <cygwin unix="100"/>

This should be sufficient to work on your Samba share and to see
all files owned by your Linux user account as your files.


======================================================================
The /etc/nsswitch.conf file
===========================

Last, but not least, let's talk about a way to configure how the
stuff works on your machine.  On Linux and some other UNIXy OSes,
we have a file called /etc/nsswitch.conf[9].  One part of it is to
specify how the passwd and group entries are generated.  That's
what Cygwin now provides as well.

The /etc/nsswitch.conf file is optional.  If you don't have one,
Cygwin uses sensible defaults.

  Note:  The /etc/nsswitch.conf file is read exactly once by the first
         process of a Cygwin process tree.  If there was no
         /etc/nsswitch.conf file when this first process started, then
         no other process in the running Cygwin process trees will try
         to read the file.  If you create or change /etc/nsswitch.conf,
         make sure to stop and restart all Cygwin processes to pick up
         the change.

So, what mischief can we perform with /etc/nsswitch.conf?  To explain,
lets have a look into an /etc/nsswitch.conf file set up to all default
values:

  # /etc/nsswitch.conf
  passwd: files db
  group:  files db

  db_prefix:    auto
  db_separator: +
  db_cache:     yes

The first line, starting with a hash '#' is a comment.  The hash
sign starts a comment, just as in shell scripts.  Everything up to
the end of the line is ignored.  So this:

  foo:  bar # baz

means, for the entry foo, do bar, ignore everything after the hash sign,
so baz is only a comment.

The other lines define the available settings.  The first word up to a
colon is a keyword.  Note that the colon *must* follow immediately after
the keyword.  This is a valid line:

  foo: bar

This is not valid:

  foo  :  bar

Apart from this restriction, the reminder of the line can have as
may spaces and TABs as you like.  This is a valid line:

        foo:                       bar                baz

Now let's have a look at the available keywords and settings.

The two lines starting with the keywords "passwd" and "group" define
where Cygwin gets its passwd and group information from.  "files" means,
fetch the information from the corresponding file in the /etc directory.
"db" means, fetch the information from the Windows account databases,
the SAM for local accounts, Active Directory for domain account.
Examples:

  passwd: files

Read passwd entries only from /etc/passwd.

  group: db

Read group entries only from SAM/AD.

  group: files # db

Read group entries only from /etc/group ("db" is ignored due to the
preceding hash sign).

  passwd: files db

Read passwd entries from /etc/passwd.  If a user account isn't found,
try to find it in SAM or AD.  This is the default for both, passwd
and group information.

  group: db files

This is a valid entry, but the order will be ignored by Cygwin.  If
both, files and db are specified, Cygwin will always try the files
first, then the db.

The remaining entries define certain aspects of the Windows account
database search.  "db_prefix" determines how the Cygwin user or group
name is created:

  db_prefix: auto

    This is the default.  If your account is from the primary domain of
    your machine, or if your machine is a standalone machine (not a domain
    member), your Cygwin account name is just the Windows account
    name.
    If your account is from another domain, or if you're logged in as
    local user on a domain machine, the Cygwin username will be the
    combination of Windows domainname and username, with the separator
    char in between:

      MY_DOM+username      (foreign domain)
      MACHINE+username     (local account)
    
    Builtin accounts have just the separator char prepended:

      +LOCAL
      +Users

    Unknown accounts on NFS or Samba shares (that is, accounts which
    cannot be mapped to Windows user accounts via RFC 2307) get a
    Cygwin account name consisting of the artificial domains "UNIX_User"
    or "UNIX_Group" and the uid/gid value, for instance:

      UNIX_User+0          (root)
      UNIX_Group+10        (wheel)

  db_prefx: primary

    Like "auto", but primary domain accounts will be prepended by
    the domainname as well.

  db_prefix: always

    All accounts, even the builtin accounts, will have the domain
    name prepended:

      BUILTIN+Users

    As a special case, if the Cygwin account name differs from the
    Windows account name, it will be prepended by the artificial domain
    name "Posix_User" or "Posix_Group" if db_prefix is set to "always":

      Posix_User+cygwin_user_name
      Posix_Group+cygwin_group_name

"db_separator" defines the spearator char used to prepend the domain
name to the user or group name.  The default is '+':

    MY_DOM+username

With "db_separator", you can define any ASCII char except space,
tab, carriage return, line feed, and the colon, as separator char.
Example:

  db_separator: \

    MY_DOM\username

"db_cache" allows to specify if user and group data should be cached in
the process at all or not.  Default is "yes".  "no" is an experimental
feature.  The idea is this.  Assuming you're working in a volatile
domain environment, and you have long-running processes like sshd.
You might want sshd to recognize changes in the existing user entries
as soon as they occur.  Caching user data in sshd *might* not be helpful
in such a scenario.  However, the user data which gets changed a lot
is typically *not* the data required for login purposes, except for
the password, and that doesn't matter in Cygwin's case.  So take this
feature with a grain of salt.

==========
Footnotes:
==========

[0] Not dragons.  Just a test.

[1] http://cygwin.com/cygwin-ug-net/ntsec.html

[2] This may change.  Right now the file is read in 32K chunks, but
    we could easily read the file in 64K chunks and, if we find the
    file is < 64K anyway, just cache the entire bunch, like before.
    Not implemented yet, but something to keep in mind.

[3] http://msdn.microsoft.com/en-us/library/windows/desktop/aa379166%28v=vs.85%29.aspx
    http://msdn.microsoft.com/en-us/library/windows/desktop/aa379159%28v=vs.85%29.aspx

[4] http://support.microsoft.com/kb/243330

[5] This is where Cygwin differs from SFU.  The reason is that we need
    the old uid/gid values for backward compatibility.  There are Cygwin
    packages (cron, for instance) who rely on the fact that the uid of
    SYSTEM is 18.  In SFU, these accounts get mapped like the other
    built in SIDs.

[6] https://tools.ietf.org/html/rfc2307

[7] https://tools.ietf.org/html/rfc1813

[8] http://msdn.microsoft.com/en-us/library/cc980032.aspx

[9] http://linux.die.net/man/5/nsswitch.conf

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Steven Penny]

On Thu, Feb 13, 2014 at 8:38 AM, Corinna Vinschen wrote:
> For as long as Cygwin has existed, it has stored user and group
> information in /etc/passwd and /etc/group files.  Under the assumption
> that these files would never be too large, the first process in a
> process tree, as well as every execing process within the tree would
> parse them into structures in memory.  Thus every Cygwin process would
> contain an expanded copy of the full information from /etc/passwd and
> /etc/group.

Stellar writeup! I read the whole post. I am happy to help, but I have couple of
questions

- How will this affect "normal" users, that is to say one admin user on one
  computer with no domain or networking? Will it be better to use this new
  system or keep /etc/passwd?

- Do you have any benchmarks available? Or instructions on how we could test the
  speed of the new system?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1650 bytes --]

On Feb 13 09:48, Steven Penny wrote:
> On Thu, Feb 13, 2014 at 8:38 AM, Corinna Vinschen wrote:
> > For as long as Cygwin has existed, it has stored user and group
> > information in /etc/passwd and /etc/group files.  Under the assumption
> > that these files would never be too large, the first process in a
> > process tree, as well as every execing process within the tree would
> > parse them into structures in memory.  Thus every Cygwin process would
> > contain an expanded copy of the full information from /etc/passwd and
> > /etc/group.
> 
> Stellar writeup! I read the whole post. I am happy to help, but I have couple of
> questions
> 
> - How will this affect "normal" users, that is to say one admin user on one
>   computer with no domain or networking? Will it be better to use this new
>   system or keep /etc/passwd?

That should have been clear from the writeup.  Just continue to use
/etc/passwd and /etc/group if you're not comfortable to change your
local SAM.  But my mail also contains examples how to change your SAM
entry from the CMD or bach command line.

> - Do you have any benchmarks available? Or instructions on how we could test the
>   speed of the new system?

Nope.  Try something time-consuming you're doing every day under
time(1), I guess.  Building some project or so.  But first get
comfortable with the new output of `id' and `ls -l' in some
environments.  I'm all for performance, but functionality first, please.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3432 bytes --]

On Feb 13 17:09, Corinna Vinschen wrote:
> On Feb 13 09:48, Steven Penny wrote:
> > On Thu, Feb 13, 2014 at 8:38 AM, Corinna Vinschen wrote:
> > > For as long as Cygwin has existed, it has stored user and group
> > > information in /etc/passwd and /etc/group files.  Under the assumption
> > > that these files would never be too large, the first process in a
> > > process tree, as well as every execing process within the tree would
> > > parse them into structures in memory.  Thus every Cygwin process would
> > > contain an expanded copy of the full information from /etc/passwd and
> > > /etc/group.
> > 
> > Stellar writeup! I read the whole post. I am happy to help, but I have couple of
> > questions
> > 
> > - How will this affect "normal" users, that is to say one admin user on one
> >   computer with no domain or networking? Will it be better to use this new
> >   system or keep /etc/passwd?
> 
> That should have been clear from the writeup.  Just continue to use
> /etc/passwd and /etc/group if you're not comfortable to change your
> local SAM.  But my mail also contains examples how to change your SAM
> entry from the CMD or bach command line.
> 
> > - Do you have any benchmarks available? Or instructions on how we could test the
> >   speed of the new system?
> 
> Nope.  Try something time-consuming you're doing every day under
> time(1), I guess.  Building some project or so.  But first get
> comfortable with the new output of `id' and `ls -l' in some
> environments.  I'm all for performance, but functionality first, please.

Btw., for completeness, here's a /etc/nsswitch.conf file with comments,
with everything set to the default values.  Again, note that the file is
optional and only needed if you want to diverge from the defaults:

========= SNIP ==========
# /etc/nsswitch.conf
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.
#
# passwd:
# group:
#
#    "files"	only use /etc/passwd or /etc/group file.
#    "db"	only use SAM/AD retrieval.
#    "files db"	both, files preferred.  This is the default.
#
#    "db files"	does not make any sense
#
passwd: files db
group:  files db
#
# Configuration of "db" style passwd/group handling:
#
# db_prefix:
#
#    "auto" 	If "auto", prepend domain to account name if the account
#		is not a member of the machine's primary domain.  Prepend
#		just the separator char if the account is a well-known
#		or builtin group.
#
#    "primary" 	"primary" is like "auto", but prepend domain to account name
#		as well, if the account is a member of the machine's primary
#		domain.
#
#    "always" 	If set to "always", always prepend domain, even for
#		well-known and builtin accounts.
#
db_prefix: auto
#
# db_cache:
#
#    "yes" 	If yes, cache once retrieved DB values in local process,
#		hand cache down to child processes.  This is the default.
#
#    "no"	If no, fetch passwd or group entries anew, every time an
#		entry is requested.
#
db_cache: yes
#
#  db_separator:
#
#		Set separator character between domain and account name to
#		the ASCII char X.  Default is '+'.
#
db_separator: +
========= SNAP ==========


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Steven Penny!

>> For as long as Cygwin has existed, it has stored user and group
>> information in /etc/passwd and /etc/group files.  Under the assumption
>> that these files would never be too large, the first process in a
>> process tree, as well as every execing process within the tree would
>> parse them into structures in memory.  Thus every Cygwin process would
>> contain an expanded copy of the full information from /etc/passwd and
>> /etc/group.

> Stellar writeup! I read the whole post. I am happy to help, but I have couple of
> questions

> - How will this affect "normal" users, that is to say one admin user on one
>   computer with no domain or networking? Will it be better to use this new
>   system or keep /etc/passwd?

Largely doesn't matter. Would probably simplify new installs once this change
is tested and deemed stable enough to be the default way of going with it.

> - Do you have any benchmarks available? Or instructions on how we could test the
>   speed of the new system?

In the end of the day, this should affect program startup times. But in home
environment, the difference will be negligible.
The main point of testing is consistency and general usability of a new
feature, as I see it.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 13.02.2014, <20:03>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Lord Laraby]

Subject: Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header
declaration changes ?)

This would be a nice mechanism for typical windows home systems except for
the format of the cygwin tag that goes in the comment field. Home users
that utilize 'net user USERNAME "/comment:yadda yadda"' command cannot use
equal signs on the command line, and the entire string has to be quoted to
prevent angle brackets from messing with redirection. Equal signs are
stripped by CMD.exe and the nexted quotes are stripped as well.

Example: I entered the following at elevated cmd prompt,

C:\WINDOWS\system32>net user TestUser "/comment:<cygwin check="maybe"
then="okay>"
The command completed successfullly.

C:\WINDOWS\system32>net user TestUser
User name                   TestUser
Full name                   Test User
Comment                     <cygwin check=maybe then=okay>
...
The command completed successfullly.

So, I do not know how to get the information into the field in the proper
format.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Larry Hall (Cygwin)]

On 2/13/2014 1:33 PM, Lord Laraby wrote:
> Subject: Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header
> declaration changes ?)
>
> This would be a nice mechanism for typical windows home systems except for
> the format of the cygwin tag that goes in the comment field. Home users
> that utilize 'net user USERNAME "/comment:yadda yadda"' command cannot use
> equal signs on the command line, and the entire string has to be quoted to
> prevent angle brackets from messing with redirection. Equal signs are
> stripped by CMD.exe and the nexted quotes are stripped as well.
>
> Example: I entered the following at elevated cmd prompt,
>
> C:\WINDOWS\system32>net user TestUser "/comment:<cygwin check="maybe"
> then="okay>"
> The command completed successfullly.
>
> C:\WINDOWS\system32>net user TestUser
> User name                   TestUser
> Full name                   Test User
> Comment                     <cygwin check=maybe then=okay>
> ...
> The command completed successfullly.
>

Corinna's email with a description of this entire mechanism recommends this
syntax:

>    CMD example:
>
>      net use corinna /comment:"<cygwin home=\"/home/foo\"/>"

Does that work for you?

-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Lord Laraby]

Yes, thank you.
That form of the syntax seems to work (at least on windows 8.1).

Thanks again.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>   The group attributes utilized by Cygwin are:

>     cn                 If set, will be used as Cygwin group name.

sambaPrimaryGroupSID
?
(Yeah, I have Samba3/LDAP domain...)


>   The <cygwin .../> string can start at any point in the comment, but
>   you have to follow the rules:

>   - It starts with "<cygwin " and ends with "/>".
>   - The "cygwin" string and the key names have to be lowercase.
>   - No spaces between key and "value", just the equal sign.
>   - The value must be placed within double quotes and it must not
>     contain a double quote itself.  The double quotes are required
>     for the decimal values as well!

>   CMD example:

>     net use corinna /comment:"<cygwin home=\"/home/foo\"/>"

  net user corinna /comment:"<cygwin home=\"/home/foo\"/>"
---------^

>   Bash example (use single quotes):

>     net use corinna /comment:'<cygwin home="/home/foo"/>'

Same as above. (I'm actually surprised, it works, though.)

> However, setting up winbindd isn't for everybody, and it fails to map
> Windows accounts to already existing UNIX users or groups.

Oh?... it really depends on the curvature of admin's hands.
Properly setup Samba accounting transparently map users through PAM.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 13.02.2014, <21:17>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

On Feb 13 13:40, Larry Hall (Cygwin) wrote:
> On 2/13/2014 1:33 PM, Lord Laraby wrote:
> >Subject: Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header
> >declaration changes ?)
> >
> >This would be a nice mechanism for typical windows home systems except for
> >the format of the cygwin tag that goes in the comment field. Home users
> >that utilize 'net user USERNAME "/comment:yadda yadda"' command cannot use
> >equal signs on the command line, and the entire string has to be quoted to
> >prevent angle brackets from messing with redirection. Equal signs are
> >stripped by CMD.exe and the nexted quotes are stripped as well.
> >
> >Example: I entered the following at elevated cmd prompt,
> >
> >C:\WINDOWS\system32>net user TestUser "/comment:<cygwin check="maybe"
> >then="okay>"
> >The command completed successfullly.
> >
> >C:\WINDOWS\system32>net user TestUser
> >User name                   TestUser
> >Full name                   Test User
> >Comment                     <cygwin check=maybe then=okay>
> >...
> >The command completed successfullly.
> >
> 
> Corinna's email with a description of this entire mechanism recommends this
> syntax:
> 
> >   CMD example:
> >
> >     net use corinna /comment:"<cygwin home=\"/home/foo\"/>"
> 
> Does that work for you?

Sigh.  I read my mail at least 6 or 7 times, but never once it occured
to me that I mistyped the commands.

It's `net user', of course, not `net use'.  Oh holy naïveté.

So again, either use

  net user corinna /comment:"<cygwin home=\"/home/foo\"/>"

from CMD (double quotes, escaping double quotes in the string) or

  net user corinna /comment:'<cygwin home="/home/foo"/>'

from bash or tcsh (single quotes around the string, no escaping
of double quotes inside the string).

You are aware that you can start a Cygwin shell elevated as well, right?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[m0viefreak]

Hello,

I have a a question about how this change is going to affect
third-party utilities. Especially in the case once chooses to
use the PAM method and get rid of any /etc/{passwd,group}
files completely.

There seem to be programs (mostly scripts) that make use of
/etc/passwd as a file directly.

One of them is for example "ssh-host-config".

The shell script:

- works with the file directly:
  ...
  run_service_as=$(/usr/bin/grep -Fi "${accnt_name}" /etc/passwd | /usr/bin/awk -F: '{print $1;}') 
  ...

  I assume this would have to be changed in the sshd package?

- After the script is run it sets various entries in the
  /etc/passwd file (sshd shell to /bin/false and home to /var/empty).
  I assume I would have to migrate these changes into the <cygwin />
  comment if I were to remove the passwd file now.
  But more importantly, if I were to run sshd-host-config with
  no passwd file present at all, would it correctly set up the
  <cygwin /> comment entry in the PAM?

Grepping through /bin I found at least one other package
that makes use of /etc/passwd as a file directly (cvsbug), but 
since I don't have everything installed I can only assume there
are more cygwin-packages and other programs someone might build
from source.


An idea that might sound a bit far-fetched: In case the above
is actually a problem, would it be possible to populate a virtual
/etc/passwd file on-the-fly the same way /dev/ and /proc/ are?
This way programs that rely on reading the passwd file direclty
would just keep working (writing would be more complicated
of course).


--
m0vie

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

On Feb 13 22:59, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >   The group attributes utilized by Cygwin are:
> 
> >     cn                 If set, will be used as Cygwin group name.
> 
> sambaPrimaryGroupSID

Apples/Oranges.  "cn" is used as the Cygwin group name for an existing
Windows group.  It's just a name change.   I don;'t know if anybody is
interested in this at all, but it was possible with /etc/group.

For instance, you just hate the group name "Domain Users", so
you set the cn attribute to "HiveAnts", and Cygwin will use the
name "HiveAnts" as Cygwin user name.

> ?
> (Yeah, I have Samba3/LDAP domain...)
> 
> 
> >   The <cygwin .../> string can start at any point in the comment, but
> >   you have to follow the rules:
> 
> >   - It starts with "<cygwin " and ends with "/>".
> >   - The "cygwin" string and the key names have to be lowercase.
> >   - No spaces between key and "value", just the equal sign.
> >   - The value must be placed within double quotes and it must not
> >     contain a double quote itself.  The double quotes are required
> >     for the decimal values as well!
> 
> >   CMD example:
> 
> >     net use corinna /comment:"<cygwin home=\"/home/foo\"/>"
> 
>   net user corinna /comment:"<cygwin home=\"/home/foo\"/>"
> ---------^

Yup, shame on me.

> >   Bash example (use single quotes):
> 
> >     net use corinna /comment:'<cygwin home="/home/foo"/>'
> 
> Same as above. (I'm actually surprised, it works, though.)
> 
> > However, setting up winbindd isn't for everybody, and it fails to map
> > Windows accounts to already existing UNIX users or groups.
> 
> Oh?... it really depends on the curvature of admin's hands.
> Properly setup Samba accounting transparently map users through PAM.

As I wrote.  It's not for everybody.  I never managed it :}


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Lord Laraby!

> This would be a nice mechanism for typical windows home systems except for
> the format of the cygwin tag that goes in the comment field.

???...

> Home users that utilize 'net user USERNAME "/comment:yadda yadda"' command
> cannot use equal signs on the command line,

That's just not true, unless someone holding their hands away from keyboard,
mouse and every other input device.

> and the entire string has to be quoted to prevent angle brackets from
> messing with redirection.

Well, see the statement above. Simply not true.

> Equal signs are stripped by CMD.exe and the nexted quotes are stripped as well.

> Example: I entered the following at elevated cmd prompt,

> C:\WINDOWS\system32>net user TestUser "/comment:<cygwin check="maybe"
> then="okay>"

But Corinna did not said to enter this.
She told you to enter (in your case):

net user TestUser /comment:"<cygwin check=\"maybe\" then=\"okay\"/>"

Or, using Cygwin shell (console or mintty):

net user TestUser /comment:'<cygwin check="maybe" then="okay"/>'

> The command completed successfullly.

> C:\WINDOWS\system32>net user TestUser
> User name                   TestUser
> Full name                   Test User
> Comment                     <cygwin check=maybe then=okay>
> ...
> The command completed successfullly.

Of course, it did. Computers are very stupid. They do what you tell them to
do, no more, no less.

> So, I do not know how to get the information into the field in the proper
> format.

Exactly as Corinna told you to.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 13.02.2014, <23:01>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

On Feb 13 20:36, m0viefreak wrote:
> Hello,
> 
> I have a a question about how this change is going to affect
> third-party utilities. Especially in the case once chooses to
> use the PAM method and get rid of any /etc/{passwd,group}
> files completely.
> 
> There seem to be programs (mostly scripts) that make use of
> /etc/passwd as a file directly.
> 
> One of them is for example "ssh-host-config".
> 
> The shell script:
> 
> - works with the file directly:
>   ...
>   run_service_as=$(/usr/bin/grep -Fi "${accnt_name}" /etc/passwd | /usr/bin/awk -F: '{print $1;}') 
>   ...
> 
>   I assume this would have to be changed in the sshd package?

Indeed.  There's also the csih package which needs to adapt.

> - After the script is run it sets various entries in the
>   /etc/passwd file (sshd shell to /bin/false and home to /var/empty).
>   I assume I would have to migrate these changes into the <cygwin />
>   comment if I were to remove the passwd file now.

Yes and no.  In theory, yes, but in fact the settings for the
sshd account don't really matter.  It's the account used for
privilege separation, and the part of privilege separation which
actually needs the sshd account doesn't work on Cygwin due to missing
sendmsg/recvmsg descriptor passing.

>   But more importantly, if I were to run sshd-host-config with
>   no passwd file present at all, would it correctly set up the
>   <cygwin /> comment entry in the PAM?

Uh... there is no PAM here, just local SAM or AD :)

But the anwser to your question is "no" at the moment.  It's probably
advisable to keep the necessary entries for services in /etc/passwd for
now.  I'll experiment with this, too, in the next couple of days.

> Grepping through /bin I found at least one other package
> that makes use of /etc/passwd as a file directly (cvsbug), but 
> since I don't have everything installed I can only assume there
> are more cygwin-packages and other programs someone might build
> from source.

If the package isn't very explicitely a Cygwin-only package, it has to
be treated as broken since direct access to /etc/passwd and /etc/group
files is a no-no.

Yes, the change is a big one, and we will encounter the occasional
fallout in the next weeks or months.  I'm pretty confident that moving
to SAM/AD is the right way to go, but I'm also sure that a a few
surprises are still waiting for us.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[David Stacey]

On 13/02/2014 19:36, m0viefreak wrote:
> Grepping through /bin I found at least one other package
> that makes use of /etc/passwd as a file directly (cvsbug), but
> since I don't have everything installed I can only assume there
> are more cygwin-packages and other programs someone might build
> from source.

My complete install is proving quite useful this week:

     chere
     cron_diagnose.sh
     cron-config
     cvsbug
     exim
     exim-config
     mailq
     mysqlbug
     newaliases
     nfs-server-config
     rcs2log
     runq
     ssh-host-config
     xhere

All of these reference /etc/passwd; some of the above also use /etc/group.

Dave.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

New passwd/group handling in Cygwin - test results and observations

[Andrey Repin]

Greetings, Corinna Vinschen!

> This week I applied the first incarnation of the new passwd/group
> handling code to the Cygwin repository and after fixing a crash which
> manifested in Denis Excoffier's network, I think we're at a point
> which allows to push this forward.

Oookaaaay... What to say about it?...
First impression is "oh my god, did I bought a new rig?"

$ uname -a
CYGWIN_NT-5.1 daemon2 1.7.29s(0.271/5/3) 20140213 14:06:26 i686 Cygwin

mintty startup is almost instant. Same with diff in native console.
I have no idea, what you did, I don't even have a good explanation of why it
happened. My /etc/passwd is about 1.5kb, half that - /etc/group.

I only dumped new cygwin1.dll into place and not changed anything else.
I'm going to run unattended tests just to prove I'm not dreaming.

HOWEVER, here's a first bug. Actually, it was a long standing issue, that
I've been discarding as nonessential, but since you are here already, can we
have a bit of attention?

The issue can be observed when you have a user or group name containing
characters outside basic ASCII character set. Even western diacritics will
suffice.

Add somewhere in your startup files an equivalent of the following block:
(I have it in private .profile)

---->8-------->8-------->8-------->8-------->8-------->8-------->8----
case "$TERM" in
  xterm*)
    LANG=ru_RU.UTF-8
    ;;
  *)
    LANG=ru_RU.CP866
    ;;
esac

export PATH HISTCONTROL LANG
----8<--------8<--------8<--------8<--------8<--------8<--------8<----

restart your shell, and try to ls -l a directory, where you have files owned
by abovementioned user/group.

Try it in mintty(the encoding will be UTF-8 and names will show up readable)
and in native console (with appropriate single-byte encoding, the names will
still be printed in unicode, means, raw byte sequences will be dumped to
terminal).
I though it could be affected by the fact I'm changing LANG on the fly, but
starting bash in a console that initially have correct LANG= variable doesn't
change observed results.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 14.02.2014, <01:50>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 2/13/2014 09:07, Andrey Repin wrote:
> Greetings, Steven Penny!
>
>> - Do you have any benchmarks available? Or instructions on how we could test the
>>    speed of the new system?
>
> In the end of the day, this should affect program startup times. But in home
> environment, the difference will be negligible.

Whose home are we talking about here? :)

In my home, we type commands like ./configure && make && make install

That sort of thing may launch and kill hundreds of programs in a short 
period of time.

If we save only 1/20 second (say 0.1 sec down to 0.05) per process 
launch, we can save tens of seconds on such a command sequence.

I look forward to testing this.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 2/13/2014 07:38, Corinna Vinschen wrote:
>
> The new
> mechanism will never read the entire file into memory, but only scan
> for the requested entry and cache this one in memory[2].

Does this feature avoid the stale cache problem?

For instance, do you check the mtime on /etc/{passwd,group} before 
checking the cache, then dump the whole cache if the file was changed 
since the last file scan?

>    Apart from power shell scripting or inventing new CLI tools, these
>    attributes can be changed using the "Attribute Editor" tab in the user
>    properties dialog of the "Active Directory Users and Computers"
>    MMC snap-in.

A week ago, we were talking about possible Cygwin {user,group}{add,mod} 
programs, modeled on Linux's.  Was that simply shelved once "net user" 
and MMC were found to be sufficient?

If such programs existed, they could abstract away the difference 
between /etc/passwd, SAM and AD.  Plus, net.exe is a hairball.

If, magically, such programs were to appear from outside the Cygwin core 
dev group, would that be a good thing or a bad thing?

>     unix="value"      Sets the NFS/Samba uid of the user to the decimal
>                       value.  See the next chapter.

I know I'm bikeshedding, but "unix" seems like a pretty vague attribute 
name here.

     unix="good" ?
     unix="linux" ?
     unix="yesplease" ?

I'd be happier with "uid" or similar.

Write it in a sentence:

a. My uid is 502; vs
b. My unix is 502.

>           If you create or change /etc/nsswitch.conf,
>           make sure to stop and restart all Cygwin processes to pick up
>           the change.

"All" processes?

If I have cron running, then exit the one instance of MinTTY after 
vim'ing /etc/nsswitch.conf, will the file be read when I re-open MinTTY?

cron.exe is running in a different process group, isn't it?

If true, I realize cron.exe and any programs it runs will continue to 
believe there is no /etc/nsswitch.conf until *it* restarts.

> the order [in nsswitch.conf] will be ignored by Cygwin.

Hmmm...different from Linux.

The inability to say "db files" as distinct from "files db" means you 
can't set up a SAM-only machine with SAM as a fast primary source of 
truth and "files" as a fallback.

That seems like a sensible configuration to me, since SAM should always 
be more trustworthy than /etc/passwd.

Personally, I'm going to try to get away with "db" only, but for systems 
that really require "files", I'd like the choice to make it secondary to 
"db".

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Warren Young!

> On 2/13/2014 09:07, Andrey Repin wrote:
>> Greetings, Steven Penny!
>>
>>> - Do you have any benchmarks available? Or instructions on how we could test the
>>>    speed of the new system?
>>
>> In the end of the day, this should affect program startup times. But in home
>> environment, the difference will be negligible.

> Whose home are we talking about here? :)

<3

> In my home, we type commands like ./configure && make && make install

> That sort of thing may launch and kill hundreds of programs in a short 
> period of time.

> If we save only 1/20 second (say 0.1 sec down to 0.05) per process 
> launch, we can save tens of seconds on such a command sequence.

> I look forward to testing this.

I'm, already. It seems, I'm ought to retract my "negligible" statement.
Even simple diff'ing is way faster now.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 14.02.2014, <10:39>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On Feb 13, 2014, at 4:11 PM, Warren Young <warren@etr-usa.com> wrote:

> If we save only 1/20 second (say 0.1 sec down to 0.05) per process launch, we can save tens of seconds on such a command sequence.
> 
> I look forward to testing this.

I decided to build R, since it’s big but fairly self-contained:

With a 4-processor XP VM, I can’t tell the difference between 1.7.28-2 and 1.7.29s (20140213) in terms of build time.  +/- a second or two over about 4.3 CPU-minutes of build time.  Configure time doesn’t change much, either.

My nsswitch.conf:

passwd: db
group:  db

I’ve tried it with /etc/files present and removed.

Waah.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Dave Kilroy]

On 13/02/2014 21:48, David Stacey wrote:
> On 13/02/2014 19:36, m0viefreak wrote:
>> Grepping through /bin I found at least one other package
>> that makes use of /etc/passwd as a file directly (cvsbug), but
>> since I don't have everything installed I can only assume there
>> are more cygwin-packages and other programs someone might build
>> from source.
>
> My complete install is proving quite useful this week:
>
>     chere
>     xhere
>
> All of these reference /etc/passwd; some of the above also use 
> /etc/group.
chere (with the -s passwd) reads etc/passwd and starts the default shell 
of the current user from the explorer context menu. For now, I will 
expect users wanting this functionality to keep using/etc/passwd. If we 
grow a mechanism where the login shell can be queried easily, I'll 
update chere to use that.


Regards,

Dave.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

On Feb 14 00:36, Warren Young wrote:
> On Feb 13, 2014, at 4:11 PM, Warren Young <warren@etr-usa.com> wrote:
> 
> > If we save only 1/20 second (say 0.1 sec down to 0.05) per process launch, we can save tens of seconds on such a command sequence.
> > 
> > I look forward to testing this.
> 
> I decided to build R, since it’s big but fairly self-contained:
> 
> With a 4-processor XP VM, I can’t tell the difference between 1.7.28-2 and 1.7.29s (20140213) in terms of build time.  +/- a second or two over about 4.3 CPU-minutes of build time.  Configure time doesn’t change much, either.
> 
> My nsswitch.conf:
> 
> passwd: db
> group:  db
> 
> I’ve tried it with /etc/files present and removed.
> 
> Waah.

Don't you worry.  This may be a bad example.  Most of the time the build
spends with actual compiling and stuff, so the time spent in calling
getpwXXX/getgrXXX functions is probably very, very minor.  At least
it didn't get slower :)


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

On Feb 13 21:48, David Stacey wrote:
> On 13/02/2014 19:36, m0viefreak wrote:
> >Grepping through /bin I found at least one other package
> >that makes use of /etc/passwd as a file directly (cvsbug), but
> >since I don't have everything installed I can only assume there
> >are more cygwin-packages and other programs someone might build
> >from source.
> 
> My complete install is proving quite useful this week:
> 
>     chere
>     cron_diagnose.sh
>     cron-config
>     cvsbug
>     exim
>     exim-config
>     mailq
>     mysqlbug
>     newaliases
>     nfs-server-config
>     rcs2log
>     runq
>     ssh-host-config
>     xhere
> 
> All of these reference /etc/passwd; some of the above also use /etc/group.

Wow, thank you.  It's not as bad as I feared.  Most of the stuff is
actually Cygwin-specific.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

On Feb 14 10:02, Dave Kilroy wrote:
> On 13/02/2014 21:48, David Stacey wrote:
> >On 13/02/2014 19:36, m0viefreak wrote:
> >>Grepping through /bin I found at least one other package
> >>that makes use of /etc/passwd as a file directly (cvsbug), but
> >>since I don't have everything installed I can only assume there
> >>are more cygwin-packages and other programs someone might build
> >>from source.
> >
> >My complete install is proving quite useful this week:
> >
> >    chere
> >    xhere
> >
> >All of these reference /etc/passwd; some of the above also use
> >/etc/group.
> chere (with the -s passwd) reads etc/passwd and starts the default
> shell of the current user from the explorer context menu. For now, I
> will expect users wanting this functionality to keep
> using/etc/passwd. If we grow a mechanism where the login shell can
> be queried easily, I'll update chere to use that.

Indeed.  After the discussion yesterday I already realized that we will
need some commandline tool to request passwd and group entries.

There is a function called getpw on Linux which allows to fetch a passwd
entry in the form it has in /etc/passwd:

  http://linux.die.net/man/3/getpw

But I can't find any tool which actually utilizes the function.  There's
also no group equivalent.

Does anybody know an existing CLI tool with the whole purpose of
requesting user or group information?

In the next few weeks I will change mkpasswd/mkgroup to create passwd
and group entries which are equivalent to the entires created by Cygwin
itself.  So, if we don't find any tool, maybe we can utilize the
`mkpasswd -u' or `mkgroup -g' options at least.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-02-13 15:38 GMT+01:00 Corinna Vinschen:
> This is a pretty intrusive change, in need of some serious testing, so
> I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> http://cygwin.com/snapshots/ contains the changes, including the latest
> bugfix.

I've installed that snapshot on my work laptop, part of AD domain. I
moved passwd and group out of the way and noticed my .bashrc is not
executed.

With 1.7.28:

Frank@PC205 ~
$ uname -a
CYGWIN_NT-6.1-WOW64 PC205 1.7.28(0.271/5/3) 2014-02-09 21:06 i686 Cygwin

Frank@PC205 ~
$ ls -l
total 86
-rw-------  1 Frank          Domeingebruikers  6759 2014-02-14 09:30
.bash_history
-rwxr-xr-x  1 Frank          Domeingebruikers  1150 2010-11-17 18:58
.bash_profile*
-rwxr-xr-x  1 Frank          Domeingebruikers  3814 2013-10-18 12:02 .bashrc*
<snip list of 26 files in total>

With snapshot without passwd/group:

Frank@PC205 ~
$ uname -a
CYGWIN_NT-6.1-WOW64 PC205 1.7.29s(0.271/5/3) 20140213 14:06:26 i686 Cygwin

Frank@PC205 ~
$ ls -l
totaal 19
-rwxr-xr-x  1 Frank           Domeingebruikers  98 16 mei  2012 mail-ek.sh
-rw-r--r--  1 Frank           Domeingebruikers 811 16 mei  2012 mail-ek.txt
-rwxr-xr-x  1 Frank           Domeingebruikers 332 12 mei  2013 makeepub.sh
drwxrwxrwt+ 1 +Administrators Domeingebruikers   0 24 okt 15:22 python
drwxrwxrwt+ 1 +Administrators Domeingebruikers   0 13 feb 17:05 rsync
drwxr-xr-x+ 1 Frank           Domeingebruikers   0 13 feb 16:44 shutdown-package
-rwxr-xr-x  1 Frank           Domeingebruikers  38 13 feb 13:57 test.sh
<this is the complete list of files>

Most noticeable difference between the two are in the dates of the
files (The months are in Dutch, and not in "long-iso") and not showing
the dot-files. Among others, I have set an export LANG=en_US and an
alias for "ls" in my bashrc for that.

Reverting to 1.7.28 (with passwd/group) solved the problem.

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: New passwd/group handling in Cygwin - test results and observations

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3560 bytes --]

On Feb 14 02:11, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> > This week I applied the first incarnation of the new passwd/group
> > handling code to the Cygwin repository and after fixing a crash which
> > manifested in Denis Excoffier's network, I think we're at a point
> > which allows to push this forward.
> 
> Oookaaaay... What to say about it?...
> First impression is "oh my god, did I bought a new rig?"
> 
> $ uname -a
> CYGWIN_NT-5.1 daemon2 1.7.29s(0.271/5/3) 20140213 14:06:26 i686 Cygwin
> 
> mintty startup is almost instant. Same with diff in native console.
> I have no idea, what you did, I don't even have a good explanation of why it
> happened. My /etc/passwd is about 1.5kb, half that - /etc/group.
> 
> I only dumped new cygwin1.dll into place and not changed anything else.
> I'm going to run unattended tests just to prove I'm not dreaming.
> 
> HOWEVER, here's a first bug. Actually, it was a long standing issue, that
> I've been discarding as nonessential, but since you are here already, can we
> have a bit of attention?
> 
> The issue can be observed when you have a user or group name containing
> characters outside basic ASCII character set. Even western diacritics will
> suffice.
> 
> Add somewhere in your startup files an equivalent of the following block:
> (I have it in private .profile)
> 
> ---->8-------->8-------->8-------->8-------->8-------->8-------->8----
> case "$TERM" in
>   xterm*)
>     LANG=ru_RU.UTF-8
>     ;;
>   *)
>     LANG=ru_RU.CP866
>     ;;
> esac
> 
> export PATH HISTCONTROL LANG
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<----
> 
> restart your shell, and try to ls -l a directory, where you have files owned
> by abovementioned user/group.
> 
> Try it in mintty(the encoding will be UTF-8 and names will show up readable)
> and in native console (with appropriate single-byte encoding, the names will
> still be printed in unicode, means, raw byte sequences will be dumped to
> terminal).
> I though it could be affected by the fact I'm changing LANG on the fly, but
> starting bash in a console that initially have correct LANG= variable doesn't
> change observed results.

Yes, this is a problem, and I'm not sure how to fix it, if at all.

The problem is hopefully obvious.  We have to initialize things in some
order.  For instance, to read /etc/fstab.d/$USER, we need the username.
And since the Cygwin username can be different from the Windows username
(I guess I should have never added this functionality in the first
place), we have to read the user's passwd before we read the fstabs.

Same for the initialization of $LANG and friends.  That occurs pretty
late in the process initialization.  You know that Windows uses UTF-16
under the hood, so a lot of stuff gets read and converted to UTF-8
before we even care for the environment.  And if you set the codeset in
the application only, all the relevant information has already been read
long ago, of course.

But this is a problem not different from Linux.  If you have a username
with non-ASCII chars, it will use *some* encoding in the passwd DB,
usually UTF-8 these days.  If you then change the codeset in your
application, you will still get your username in UTF-8.  It won't be
changed on the fly, just because your application calls setlocale.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3817 bytes --]

On Feb 13 17:30, Warren Young wrote:
> On 2/13/2014 07:38, Corinna Vinschen wrote:
> >
> >The new
> >mechanism will never read the entire file into memory, but only scan
> >for the requested entry and cache this one in memory[2].
> 
> Does this feature avoid the stale cache problem?
> 
> For instance, do you check the mtime on /etc/{passwd,group} before
> checking the cache, then dump the whole cache if the file was
> changed since the last file scan?

Yes.

> >   Apart from power shell scripting or inventing new CLI tools, these
> >   attributes can be changed using the "Attribute Editor" tab in the user
> >   properties dialog of the "Active Directory Users and Computers"
> >   MMC snap-in.
> 
> A week ago, we were talking about possible Cygwin
> {user,group}{add,mod} programs, modeled on Linux's.  Was that simply
> shelved once "net user" and MMC were found to be sufficient?

Huh?  "Apart from [...] or inventing new CLI tools, [...]"
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^

> If such programs existed, they could abstract away the difference
> between /etc/passwd, SAM and AD.  Plus, net.exe is a hairball.

Well, "net user" works.  New tools still have to catch up...

> If, magically, such programs were to appear from outside the Cygwin
> core dev group, would that be a good thing or a bad thing?

It would be a really great thing!

> >    unix="value"      Sets the NFS/Samba uid of the user to the decimal
> >                      value.  See the next chapter.
> 
> I know I'm bikeshedding, but "unix" seems like a pretty vague
> attribute name here.
> 
>     unix="good" ?
>     unix="linux" ?
>     unix="yesplease" ?
> 
> I'd be happier with "uid" or similar.
> 
> Write it in a sentence:
> 
> a. My uid is 502; vs
> b. My unix is 502.

Would you believe me that I sweated about this, too?  The important
thing to consider is, the keyword should not give the impression, that
the uid or gid set at this point is the Cygwin uid or gid.  It's only
the uid/gid of your NFS or Samba account on that weird Linux box.

Really, I'm open to suggestions to have a better keyword, but it
should make very clear that this is not your Cygwin uid/gid.

> >          If you create or change /etc/nsswitch.conf,
> >          make sure to stop and restart all Cygwin processes to pick up
> >          the change.
> 
> "All" processes?
> 
> If I have cron running, then exit the one instance of MinTTY after
> vim'ing /etc/nsswitch.conf, will the file be read when I re-open
> MinTTY?
> 
> cron.exe is running in a different process group, isn't it?
> 
> If true, I realize cron.exe and any programs it runs will continue
> to believe there is no /etc/nsswitch.conf until *it* restarts.

You are absolutely right, but, please, suggest a better wording.
This is what I'm trying to explain in fact, but everything I came
up with sounded like beating around the bush, more confusing than
helping.

> >the order [in nsswitch.conf] will be ignored by Cygwin.
> 
> Hmmm...different from Linux.

Yes, that's how it's implemented.  It's just a bitmask and the
order inside Cygwin is fixed.

> The inability to say "db files" as distinct from "files db" means
> you can't set up a SAM-only machine with SAM as a fast primary
> source of truth and "files" as a fallback.

What for?

> That seems like a sensible configuration to me, since SAM should
> always be more trustworthy than /etc/passwd.

Again, what for?  What entry would you find in passwd which you
didn't already find in SAM or via the implemented automatisms
for unknown SIDs?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2800 bytes --]

On Feb 14 11:16, Frank Fesevur wrote:
> 2014-02-13 15:38 GMT+01:00 Corinna Vinschen:
> > This is a pretty intrusive change, in need of some serious testing, so
> > I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> > http://cygwin.com/snapshots/ contains the changes, including the latest
> > bugfix.
> 
> I've installed that snapshot on my work laptop, part of AD domain. I
> moved passwd and group out of the way and noticed my .bashrc is not
> executed.
> 
> With 1.7.28:
> 
> Frank@PC205 ~
> $ uname -a
> CYGWIN_NT-6.1-WOW64 PC205 1.7.28(0.271/5/3) 2014-02-09 21:06 i686 Cygwin
> 
> Frank@PC205 ~
> $ ls -l
> total 86
> -rw-------  1 Frank          Domeingebruikers  6759 2014-02-14 09:30
> .bash_history
> -rwxr-xr-x  1 Frank          Domeingebruikers  1150 2010-11-17 18:58
> .bash_profile*
> -rwxr-xr-x  1 Frank          Domeingebruikers  3814 2013-10-18 12:02 .bashrc*
> <snip list of 26 files in total>
> 
> With snapshot without passwd/group:
> 
> Frank@PC205 ~
> $ uname -a
> CYGWIN_NT-6.1-WOW64 PC205 1.7.29s(0.271/5/3) 20140213 14:06:26 i686 Cygwin
> 
> Frank@PC205 ~
> $ ls -l
> totaal 19
> -rwxr-xr-x  1 Frank           Domeingebruikers  98 16 mei  2012 mail-ek.sh
> -rw-r--r--  1 Frank           Domeingebruikers 811 16 mei  2012 mail-ek.txt
> -rwxr-xr-x  1 Frank           Domeingebruikers 332 12 mei  2013 makeepub.sh
> drwxrwxrwt+ 1 +Administrators Domeingebruikers   0 24 okt 15:22 python
> drwxrwxrwt+ 1 +Administrators Domeingebruikers   0 13 feb 17:05 rsync
> drwxr-xr-x+ 1 Frank           Domeingebruikers   0 13 feb 16:44 shutdown-package
> -rwxr-xr-x  1 Frank           Domeingebruikers  38 13 feb 13:57 test.sh
> <this is the complete list of files>
> 
> Most noticeable difference between the two are in the dates of the
> files (The months are in Dutch, and not in "long-iso") and not showing
> the dot-files. Among others, I have set an export LANG=en_US

...hopefully LANG='en_US.utf8"...

> and an
> alias for "ls" in my bashrc for that.

Hmm, off the top of my head I can't explain this.  Is that
via mintty or a Windows console?  What does `id' print?  Is there a
difference in upper/lower case of your name?

[...time passes...]

Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
a bit fuzzy on bash, but does bash read the .bachrc file only if it's
called bash and not if it's called sh?  What happens if you symlink
your .bashrc to .profile (Not that I suggest this as the ultimate
solution, this is just for testing)?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

On Feb 14 11:10, Corinna Vinschen wrote:
> On Feb 14 10:02, Dave Kilroy wrote:
> > On 13/02/2014 21:48, David Stacey wrote:
> > >On 13/02/2014 19:36, m0viefreak wrote:
> > >>Grepping through /bin I found at least one other package
> > >>that makes use of /etc/passwd as a file directly (cvsbug), but
> > >>since I don't have everything installed I can only assume there
> > >>are more cygwin-packages and other programs someone might build
> > >>from source.
> > >
> > >My complete install is proving quite useful this week:
> > >
> > >    chere
> > >    xhere
> > >
> > >All of these reference /etc/passwd; some of the above also use
> > >/etc/group.
> > chere (with the -s passwd) reads etc/passwd and starts the default
> > shell of the current user from the explorer context menu. For now, I
> > will expect users wanting this functionality to keep
> > using/etc/passwd. If we grow a mechanism where the login shell can
> > be queried easily, I'll update chere to use that.
> 
> Indeed.  After the discussion yesterday I already realized that we will
> need some commandline tool to request passwd and group entries.
> 
> There is a function called getpw on Linux which allows to fetch a passwd
> entry in the form it has in /etc/passwd:
> 
>   http://linux.die.net/man/3/getpw

Gosh, please ignore this function.  It's old and dangerous.  Treat this
mail as if I never mentioned this function, ok?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

On Feb 14 11:48, Corinna Vinschen wrote:
> On Feb 14 11:16, Frank Fesevur wrote:
> > I've installed that snapshot on my work laptop, part of AD domain. I
> > moved passwd and group out of the way and noticed my .bashrc is not
> > executed.
> > [...]
> > Most noticeable difference between the two are in the dates of the
> > files (The months are in Dutch, and not in "long-iso") and not showing
> > the dot-files. Among others, I have set an export LANG=en_US
> 
> ...hopefully LANG='en_US.utf8"...
> 
> > and an
> > alias for "ls" in my bashrc for that.
> 
> Hmm, off the top of my head I can't explain this.  Is that
> via mintty or a Windows console?  What does `id' print?  Is there a
> difference in upper/lower case of your name?
> 
> [...time passes...]
> 
> Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
> However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
> a bit fuzzy on bash, but does bash read the .bachrc file only if it's
> called bash and not if it's called sh?  What happens if you symlink
> your .bashrc to .profile (Not that I suggest this as the ultimate
> solution, this is just for testing)?

Ultimately, the right thing to do is either:

- We switch Cygwin to use /bin/bash as default, instead of /bin/sh,

- or, you ask your admin to set the "loginShell" attribute of your
  AD entry to "/bin/bash".


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: New passwd/group handling in Cygwin - test results and observations

[Andrey Repin]

Greetings, Corinna Vinschen!

>> The issue can be observed when you have a user or group name containing
>> characters outside basic ASCII character set. Even western diacritics will
>> suffice.
>> 
>> Add somewhere in your startup files an equivalent of the following block:
>> (I have it in private .profile)
>> 
>> ---->8-------->8-------->8-------->8-------->8-------->8-------->8----
>> case "$TERM" in
>>   xterm*)
>>     LANG=ru_RU.UTF-8
>>     ;;
>>   *)
>>     LANG=ru_RU.CP866
>>     ;;
>> esac
>> 
>> export PATH HISTCONTROL LANG
>> ----8<--------8<--------8<--------8<--------8<--------8<--------8<----
>> 
>> restart your shell, and try to ls -l a directory, where you have files owned
>> by abovementioned user/group.
>> 
>> Try it in mintty(the encoding will be UTF-8 and names will show up readable)
>> and in native console (with appropriate single-byte encoding, the names will
>> still be printed in unicode, means, raw byte sequences will be dumped to
>> terminal).
>> I though it could be affected by the fact I'm changing LANG on the fly, but
>> starting bash in a console that initially have correct LANG= variable doesn't
>> change observed results.

> Yes, this is a problem, and I'm not sure how to fix it, if at all.

> The problem is hopefully obvious.  We have to initialize things in some
> order.  For instance, to read /etc/fstab.d/$USER, we need the username.
> And since the Cygwin username can be different from the Windows username
> (I guess I should have never added this functionality in the first
> place),

I feel your pain...

> we have to read the user's passwd before we read the fstabs.

> Same for the initialization of $LANG and friends.  That occurs pretty
> late in the process initialization. You know that Windows uses UTF-16
> under the hood, so a lot of stuff gets read and converted to UTF-8
> before we even care for the environment. And if you set the codeset in
> the application only, all the relevant information has already been read
> long ago, of course.

> But this is a problem not different from Linux.  If you have a username
> with non-ASCII chars, it will use *some* encoding in the passwd DB,
> usually UTF-8 these days.  If you then change the codeset in your
> application, you will still get your username in UTF-8.  It won't be
> changed on the fly, just because your application calls setlocale.

I understand it (mostly), but there's actually two issues, not one.
One issue is the display part, where names are output for user consumption.
Another can be observed in, i.e., rsync, and file access in general (remember
the discussion about accessing long directory names in unicode).
Changing LANG variable DO matter for the latter, and you may only hope that
whatever is output in the former case is actually printable (thank God, most
of the time it actually is, in case of UTF-8).
It is getting even more complicated, when you consider the fact, that in
Windows you have 2 different single-byte encodings, so-called ANSI (for GUI
applications) and OEM (for console). And alot of stuff making assumptions
without consulting with current status of things.
As convoluted the problem is, I think, we need some sort of solution, or at
the very least - documentation.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 14.02.2014, <15:15>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> >    unix="value"      Sets the NFS/Samba uid of the user to the decimal
>> >                      value.  See the next chapter.
>> 
>> I know I'm bikeshedding, but "unix" seems like a pretty vague
>> attribute name here.
>> 
>>     unix="good" ?
>>     unix="linux" ?
>>     unix="yesplease" ?
>> 
>> I'd be happier with "uid" or similar.
>> 
>> Write it in a sentence:
>> 
>> a. My uid is 502; vs
>> b. My unix is 502.

> Would you believe me that I sweated about this, too?  The important
> thing to consider is, the keyword should not give the impression, that
> the uid or gid set at this point is the Cygwin uid or gid.  It's only
> the uid/gid of your NFS or Samba account on that weird Linux box.

> Really, I'm open to suggestions to have a better keyword, but it
> should make very clear that this is not your Cygwin uid/gid.

nfsUid/-Gid
Since it is almost exclusively used to talk to NFS servers.

>> >          If you create or change /etc/nsswitch.conf,
>> >          make sure to stop and restart all Cygwin processes to pick up
>> >          the change.
>> 
>> "All" processes?
>> 
>> If I have cron running, then exit the one instance of MinTTY after
>> vim'ing /etc/nsswitch.conf, will the file be read when I re-open
>> MinTTY?
>> 
>> cron.exe is running in a different process group, isn't it?
>> 
>> If true, I realize cron.exe and any programs it runs will continue
>> to believe there is no /etc/nsswitch.conf until *it* restarts.

> You are absolutely right, but, please, suggest a better wording.
> This is what I'm trying to explain in fact, but everything I came
> up with sounded like beating around the bush, more confusing than
> helping.

"...each relevant Cygwin process tree needs to be restarted to catch up"?
Something along these lines. Since these tokens maintained on per-tree basis,
inherited from the very first Cygwin process started.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 14.02.2014, <16:19>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Kurt Franke]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:

> > Indeed.  After the discussion yesterday I already realized that we will
> > need some commandline tool to request passwd and group entries.
> > 
> > There is a function called getpw on Linux which allows to fetch a passwd
> > entry in the form it has in /etc/passwd:
> > 
> >   http://linux.die.net/man/3/getpw
> 
> Gosh, please ignore this function.  It's old and dangerous.  Treat this
> mail as if I never mentioned this function, ok?
> 
> Corinna
> 

Hi Corinna,

I think the function family

getpwent()
getgrent()
...

would be good for this if you add checking in SAM for asked information.

And of course to small executables utilizing it for use in shell scripts


Regards

kf




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Ken Brown]

On 2/14/2014 6:00 AM, Corinna Vinschen wrote:
> On Feb 14 11:48, Corinna Vinschen wrote:
>> On Feb 14 11:16, Frank Fesevur wrote:
>>> I've installed that snapshot on my work laptop, part of AD domain. I
>>> moved passwd and group out of the way and noticed my .bashrc is not
>>> executed.
>>> [...]
>>> Most noticeable difference between the two are in the dates of the
>>> files (The months are in Dutch, and not in "long-iso") and not showing
>>> the dot-files. Among others, I have set an export LANG=en_US
>>
>> ...hopefully LANG='en_US.utf8"...
>>
>>> and an
>>> alias for "ls" in my bashrc for that.
>>
>> Hmm, off the top of my head I can't explain this.  Is that
>> via mintty or a Windows console?  What does `id' print?  Is there a
>> difference in upper/lower case of your name?
>>
>> [...time passes...]
>>
>> Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
>> However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
>> a bit fuzzy on bash, but does bash read the .bachrc file only if it's
>> called bash and not if it's called sh?  What happens if you symlink
>> your .bashrc to .profile (Not that I suggest this as the ultimate
>> solution, this is just for testing)?
>
> Ultimately, the right thing to do is either:
>
> - We switch Cygwin to use /bin/bash as default, instead of /bin/sh,

Isn't /bin/bash already the default?  The output of `mkpasswd -l' seems 
to confirm this.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

On Feb 14 12:55, Kurt Franke wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> 
> > > Indeed.  After the discussion yesterday I already realized that we will
> > > need some commandline tool to request passwd and group entries.
> > > 
> > > There is a function called getpw on Linux which allows to fetch a passwd
> > > entry in the form it has in /etc/passwd:
> > > 
> > >   http://linux.die.net/man/3/getpw
> > 
> > Gosh, please ignore this function.  It's old and dangerous.  Treat this
> > mail as if I never mentioned this function, ok?
> > 
> > Corinna
> > 
> 
> Hi Corinna,
> 
> I think the function family
> 
> getpwent()
> getgrent()

Sure, but the problem is not to have functions, the problem is to have a
tool which can be used from shell scripts.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: New passwd/group handling in Cygwin - test results and observations

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Feb 14 15:56, Andrey Repin wrote:
> > But this is a problem not different from Linux.  If you have a username
> > with non-ASCII chars, it will use *some* encoding in the passwd DB,
> > usually UTF-8 these days.  If you then change the codeset in your
> > application, you will still get your username in UTF-8.  It won't be
> > changed on the fly, just because your application calls setlocale.
> 
> I understand it (mostly), but there's actually two issues, not one.
> One issue is the display part, where names are output for user consumption.
> Another can be observed in, i.e., rsync, and file access in general (remember
> the discussion about accessing long directory names in unicode).
> Changing LANG variable DO matter for the latter, and you may only hope that
> whatever is output in the former case is actually printable (thank God, most
> of the time it actually is, in case of UTF-8).
> It is getting even more complicated, when you consider the fact, that in
> Windows you have 2 different single-byte encodings, so-called ANSI (for GUI
> applications) and OEM (for console). And alot of stuff making assumptions
> without consulting with current status of things.
> As convoluted the problem is, I think, we need some sort of solution, or at
> the very least - documentation.

Sorry, I can't provide an easy solution, and afaik this is documented.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2453 bytes --]

On Feb 14 16:23, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> >    unix="value"      Sets the NFS/Samba uid of the user to the decimal
> >> >                      value.  See the next chapter.
> >> 
> >> I know I'm bikeshedding, but "unix" seems like a pretty vague
> >> attribute name here.
> >> 
> >>     unix="good" ?
> >>     unix="linux" ?
> >>     unix="yesplease" ?
> >> 
> >> I'd be happier with "uid" or similar.
> >> 
> >> Write it in a sentence:
> >> 
> >> a. My uid is 502; vs
> >> b. My unix is 502.
> 
> > Would you believe me that I sweated about this, too?  The important
> > thing to consider is, the keyword should not give the impression, that
> > the uid or gid set at this point is the Cygwin uid or gid.  It's only
> > the uid/gid of your NFS or Samba account on that weird Linux box.
> 
> > Really, I'm open to suggestions to have a better keyword, but it
> > should make very clear that this is not your Cygwin uid/gid.
> 
> nfsUid/-Gid
> Since it is almost exclusively used to talk to NFS servers.

Except when it's used for Samba shares, which is more likely than NFS,
given that the NFS client is restricted to the Enterprise version of
Windows since Windows 8.  I asked twice to change that *at least* to the
professional version, but Microsoft simply doesn't care, despite NFS
being much faster than SMB.  Grrrrrrrrrrrr.

I would prefer a keyword in all lowercase, btw.  I would also prefer a
keyword without underscore.  And I would prefer a keyword which can be
used for user and group entries because that simplifies the code.  And
I would prefer golden or platinum framed letters...

  nfsuid/nfsgid
  nfsid
  unixuid/unixgid
  unixid
  unix_id
  [...to be continued...]

> >> If true, I realize cron.exe and any programs it runs will continue
> >> to believe there is no /etc/nsswitch.conf until *it* restarts.
> 
> > You are absolutely right, but, please, suggest a better wording.
> > This is what I'm trying to explain in fact, but everything I came
> > up with sounded like beating around the bush, more confusing than
> > helping.
> 
> "...each relevant Cygwin process tree needs to be restarted to catch up"?
> Something along these lines.

Yes, something along these lines.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On Feb 14 08:50, Ken Brown wrote:
> On 2/14/2014 6:00 AM, Corinna Vinschen wrote:
> >On Feb 14 11:48, Corinna Vinschen wrote:
> >>On Feb 14 11:16, Frank Fesevur wrote:
> >>>I've installed that snapshot on my work laptop, part of AD domain. I
> >>>moved passwd and group out of the way and noticed my .bashrc is not
> >>>executed.
> >>>[...]
> >>>Most noticeable difference between the two are in the dates of the
> >>>files (The months are in Dutch, and not in "long-iso") and not showing
> >>>the dot-files. Among others, I have set an export LANG=en_US
> >>
> >>...hopefully LANG='en_US.utf8"...
> >>
> >>>and an
> >>>alias for "ls" in my bashrc for that.
> >>
> >>Hmm, off the top of my head I can't explain this.  Is that
> >>via mintty or a Windows console?  What does `id' print?  Is there a
> >>difference in upper/lower case of your name?
> >>
> >>[...time passes...]
> >>
> >>Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
> >>However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
> >>a bit fuzzy on bash, but does bash read the .bachrc file only if it's
> >>called bash and not if it's called sh?  What happens if you symlink
> >>your .bashrc to .profile (Not that I suggest this as the ultimate
> >>solution, this is just for testing)?
> >
> >Ultimately, the right thing to do is either:
> >
> >- We switch Cygwin to use /bin/bash as default, instead of /bin/sh,
> 
> Isn't /bin/bash already the default?  The output of `mkpasswd -l'
> seems to confirm this.

Oh, right.  I didn't remember that.  So I guess I should do the same
for automatically created entries, right?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Ken Brown]

On 2/14/2014 9:14 AM, Corinna Vinschen wrote:
> On Feb 14 08:50, Ken Brown wrote:
>> On 2/14/2014 6:00 AM, Corinna Vinschen wrote:
>>> On Feb 14 11:48, Corinna Vinschen wrote:
>>>> On Feb 14 11:16, Frank Fesevur wrote:
>>>>> I've installed that snapshot on my work laptop, part of AD domain. I
>>>>> moved passwd and group out of the way and noticed my .bashrc is not
>>>>> executed.
>>>>> [...]
>>>>> Most noticeable difference between the two are in the dates of the
>>>>> files (The months are in Dutch, and not in "long-iso") and not showing
>>>>> the dot-files. Among others, I have set an export LANG=en_US
>>>>
>>>> ...hopefully LANG='en_US.utf8"...
>>>>
>>>>> and an
>>>>> alias for "ls" in my bashrc for that.
>>>>
>>>> Hmm, off the top of my head I can't explain this.  Is that
>>>> via mintty or a Windows console?  What does `id' print?  Is there a
>>>> difference in upper/lower case of your name?
>>>>
>>>> [...time passes...]
>>>>
>>>> Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
>>>> However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
>>>> a bit fuzzy on bash, but does bash read the .bachrc file only if it's
>>>> called bash and not if it's called sh?  What happens if you symlink
>>>> your .bashrc to .profile (Not that I suggest this as the ultimate
>>>> solution, this is just for testing)?
>>>
>>> Ultimately, the right thing to do is either:
>>>
>>> - We switch Cygwin to use /bin/bash as default, instead of /bin/sh,
>>
>> Isn't /bin/bash already the default?  The output of `mkpasswd -l'
>> seems to confirm this.
>
> Oh, right.  I didn't remember that.  So I guess I should do the same
> for automatically created entries, right?

Right.

Ken


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 2/14/2014 03:42, Corinna Vinschen wrote:
> On Feb 13 17:30, Warren Young wrote:
>> On 2/13/2014 07:38, Corinna Vinschen wrote:
>
>>>    Apart from power shell scripting or inventing new CLI tools, these
>>>    attributes can be changed using the "Attribute Editor" tab in the user
>>>    properties dialog of the "Active Directory Users and Computers"
>>>    MMC snap-in.
>>
>> A week ago, we were talking about possible Cygwin
>> {user,group}{add,mod} programs, modeled on Linux's.  Was that simply
>> shelved once "net user" and MMC were found to be sufficient?
>
> Huh?  "Apart from [...] or inventing new CLI tools, [...]"
>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^

I wasn't sure how to interpret that.  It could be read as an unfulfilled 
possibility or as dismissing a ridiculous idea.  i.e. "Apart from 
rewriting Cygwin in Erlang..."

>> If, magically, such programs were to appear from outside the Cygwin
>> core dev group, would that be a good thing or a bad thing?
>
> It would be a really great thing!

Okay.  I thought you might feel proprietary about such tools.  "I know 
how it needs to be written, so I'm going to be the one to write it, 
right after the other 59 bazillion things on my wishlist."

>> I know I'm bikeshedding, but "unix" seems like a pretty vague
>> attribute name here.
>
> Really, I'm open to suggestions to have a better keyword, but it
> should make very clear that this is not your Cygwin uid/gid.

Okay, netfsuid, then.

>> "All" processes?
>
> You are absolutely right, but, please, suggest a better wording.

"If you create or change /etc/nsswitch.conf, you need to restart all 
Cygwin processes that need to see the change.  If the process you want 
to see the change is a child of another process, you need to restart all 
of that process's parents, too."

"For example, if you run Vim inside the default Cygwin Terminal, Vim is 
a child of your shell, which is a child of mintty.exe.  If you edit 
/etc/nsswitch.conf in that Vim instance, your shell won't immediately 
see the change, nor will Vim if you restart it from that same shell 
instance.  This is because both are getting their nsswitch information 
from their ancestor, mintty.exe.  You need to start a fresh terminal 
window for the change to take effect."

"By contrast, if you leave that Cygwin Terminal window open after making 
the change to /etc/nsswitch.conf, then restart a Cygwin service like 
cron, cron will see the change, because it is not a child of mintty.exe 
or any other Cygwin process. (Technically, it is a child of cygrunsrv, 
but that instance also restarts when you restart the service.)"

"The reason we point all this out is that the requirements for 
restarting things are not quite as stringent as when you replace 
cygwin1.dll. If you have three process trees, you have three independent 
copies of the nsswitch information.  You do not need to restart any of 
them for a new process tree to see the changes, but until each restarts, 
they will not see the change."



Better?

You could stop with the first paragraph, but I think the following 
exposition helps.

> What entry would you find in passwd which you
> didn't already find in SAM or via the implemented automatisms
> for unknown SIDs?

That makes sense.

Is nsswitch.conf the right thing, then?  Are we borrowing that mechanism 
just because it exists and looks close enough?

It seems to me that we really only need a single Boolean setting:

     ignore_db=true

If this is true, it uses files only.  If false, DB is the sole source of 
truth if /etc/{passwd,group} are missing, or it is a fallback source of 
truth if those files are present.

Does this help us get to a world where we configure this in nscd.conf, 
as cgf proposed?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 2/14/2014 13:11, Warren Young wrote:
> "The reason we point all this out is that the requirements for
> restarting things are not quite as stringent as when you replace
> cygwin1.dll. If you have three process trees, you have three independent
> copies of the nsswitch information.  You do not need to restart any of
> them for a new process tree to see the changes, but until each restarts,
> they will not see the change."

EDIT, replace last sentence:

"If you start a fresh process tree, it will see the changes.  As long as 
any process in an existing process tree remains running, all processes 
in that tree will continue to use the old information."

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-02-14 11:48 GMT+01:00 Corinna Vinschen:
> Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
> However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
> a bit fuzzy on bash, but does bash read the .bachrc file only if it's
> called bash and not if it's called sh?  What happens if you symlink
> your .bashrc to .profile (Not that I suggest this as the ultimate
> solution, this is just for testing)?

Just tested and copied .bashrc to .profile and that solved the problem.

I vote[1] for /bin/bash being the default shell.

Regards,
Frank

[1] not that there is any voting going on ;-)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3765 bytes --]

On Feb 14 13:11, Warren Young wrote:
> On 2/14/2014 03:42, Corinna Vinschen wrote:
> >On Feb 13 17:30, Warren Young wrote:
> >>On 2/13/2014 07:38, Corinna Vinschen wrote:
> >
> >>>   Apart from power shell scripting or inventing new CLI tools, these
> >>>   attributes can be changed using the "Attribute Editor" tab in the user
> >>>   properties dialog of the "Active Directory Users and Computers"
> >>>   MMC snap-in.
> >>
> >>A week ago, we were talking about possible Cygwin
> >>{user,group}{add,mod} programs, modeled on Linux's.  Was that simply
> >>shelved once "net user" and MMC were found to be sufficient?
> >
> >Huh?  "Apart from [...] or inventing new CLI tools, [...]"
> >                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> I wasn't sure how to interpret that.  It could be read as an
> unfulfilled possibility or as dismissing a ridiculous idea.  i.e.
> "Apart from rewriting Cygwin in Erlang..."

Nah, I'm not *always* cynical.

> >>If, magically, such programs were to appear from outside the Cygwin
> >>core dev group, would that be a good thing or a bad thing?
> >
> >It would be a really great thing!
> 
> Okay.  I thought you might feel proprietary about such tools.  "I
> know how it needs to be written, so I'm going to be the one to write
> it, right after the other 59 bazillion things on my wishlist."

Actually, high on my wishlist is more active maintainers for the distro.
We have a couple of maintainers of very important packages which only
show up very sporadically lately, which is pretty frustrating.


> >>I know I'm bikeshedding, but "unix" seems like a pretty vague
> >>attribute name here.
> >
> >Really, I'm open to suggestions to have a better keyword, but it
> >should make very clear that this is not your Cygwin uid/gid.
> 
> Okay, netfsuid, then.

Hmm.

> >>"All" processes?
> >
> >You are absolutely right, but, please, suggest a better wording.
> 
> "If you create or change /etc/nsswitch.conf, you need to restart all
> Cygwin processes that need to see the change.  If the process you
> want to see the change is a child of another process, you need to
> restart all of that process's parents, too."
> [...]
> Better?

Yes.  Thanks a lot.  I grabbed all of this including your followup
change shamelessly and added it to the text.

> >What entry would you find in passwd which you
> >didn't already find in SAM or via the implemented automatisms
> >for unknown SIDs?
> 
> That makes sense.
> 
> Is nsswitch.conf the right thing, then?  Are we borrowing that
> mechanism just because it exists and looks close enough?

Perhaps.

> It seems to me that we really only need a single Boolean setting:
> 
>     ignore_db=true

No, that's not right.  We have two mechanisms implemented you can
choose three out of four possible combinations:

  files only
  db only
  files, then db

> If this is true, it uses files only.  If false, DB is the sole
> source of truth if /etc/{passwd,group} are missing, or it is a
> fallback source of truth if those files are present.

The third combination is to prevent Cygwin from reading /etc/passwd
and /etc/group at all.  It drops any check for existence, too, which
is one code point less which has to run for each getpwXXX/getgrXXX
invocation.

> Does this help us get to a world where we configure this in
> nscd.conf, as cgf proposed?

I'm open to discuss this.  We can switch from nssswitch.conf to
nscd.conf, but our settings will still not match the role-model,
so it's kind of a name-reuse only, either way.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

On Feb 14 22:48, Frank Fesevur wrote:
> 2014-02-14 11:48 GMT+01:00 Corinna Vinschen:
> > Oh, hang on.  I'm a tcsh gal, so I set /bin/tcsh in my AD domain entry.
> > However, the default is *not* /bin/bash, but /bin/sh at the moment.  I'm
> > a bit fuzzy on bash, but does bash read the .bachrc file only if it's
> > called bash and not if it's called sh?  What happens if you symlink
> > your .bashrc to .profile (Not that I suggest this as the ultimate
> > solution, this is just for testing)?
> 
> Just tested and copied .bashrc to .profile and that solved the problem.

Good, so it was really just a shell thingy.

> I vote[1] for /bin/bash being the default shell.

Done in CVS.

> Regards,
> Frank
> 
> [1] not that there is any voting going on ;-)

Not quite.  This *is* a big change, so if I can smooth the ride for
the users, I'm open to change.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On Feb 15, 2014, at 5:50 AM, Corinna Vinschen wrote:

> I grabbed all of this including your followup
> change shamelessly and added it to the text.

Happy to provide it.

>> It seems to me that we really only need a single Boolean setting:
>> 
>>   ignore_db=true
> 
> No, that's not right.  We have two mechanisms implemented you can
> choose three out of four possible combinations:
> 
> files only
> db only
> files, then db

Yes, I realize that.

While composing the previous email, I considered a 2-Boolean design:

   ignore_db=false
   ignore_files=false

I rejected that design when I realized that ignore_files is redundant with respect to /etc/foo file existence.  If the files are present, use them.  If not, you have a DB-only configuration.

I consider it an advantage of this design that it is not possible for the files to be present but ignored.  It avoids confusion, both human and software.

A person modifying /etc/passwd expects the change to take immediate effect.  (Obviously SAM or AD could mask it, but…)

Software that scans /etc/passwd expects that the data found within have relevance to system operation.

The nonsense configuration — ignore files *and* ignore DB — is still possible to achieve with the 1-Boolean design.  Set ignore_db=true and remove the /etc files.  Cygwin should simply treat this as a “DB-only” configuration, since that’s the only place it *can* get answers.

> It drops any check for existence, too, which
> is one code point less which has to run for each getpwXXX/getgrXXX
> invocation.

Cygwin should check for file existence at the process tree start, where it reads /etc/nsswitch.conf.  If the files are missing at that time, set the “ignore_files” flag.

Cygwin should use that flag instead of explicit file existence tests for the same reason it doesn’t keep re-reading nsswitch.conf.  If the files are missing at process tree start, then later created, you have to create a new process tree for the new files to be used.

This is no sacrifice.  If you use a 2-Boolean design, you only end up in the code fork where file checking is done if ignore_files == false.  That means there is no point to checking for file existence explicitly. Just blindly try to read the files; if the file is missing, the open will fail, so your check is implicit in the attempt.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

On Feb 16 03:30, Warren Young wrote:
> On Feb 15, 2014, at 5:50 AM, Corinna Vinschen wrote:
> > No, that's not right.  We have two mechanisms implemented you can
> > choose three out of four possible combinations:
> > 
> > files only
> > db only
> > files, then db
> 
> Yes, I realize that.
> 
> While composing the previous email, I considered a 2-Boolean design:
> 
>    ignore_db=false
>    ignore_files=false
> 
> I rejected that design when I realized that ignore_files is redundant with respect to /etc/foo file existence.  If the files are present, use them.  If not, you have a DB-only configuration.

What if an admin wants to avoid that files are read at all, even if a
user manages to generate them?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: New passwd/group handling in Cygwin - test results and observations

[Andrey Repin]

Greetings, Corinna Vinschen!

>> > But this is a problem not different from Linux.  If you have a username
>> > with non-ASCII chars, it will use *some* encoding in the passwd DB,
>> > usually UTF-8 these days.  If you then change the codeset in your
>> > application, you will still get your username in UTF-8.  It won't be
>> > changed on the fly, just because your application calls setlocale.
>> 
>> I understand it (mostly), but there's actually two issues, not one.
>> One issue is the display part, where names are output for user consumption.
>> Another can be observed in, i.e., rsync, and file access in general (remember
>> the discussion about accessing long directory names in unicode).
>> Changing LANG variable DO matter for the latter, and you may only hope that
>> whatever is output in the former case is actually printable (thank God, most
>> of the time it actually is, in case of UTF-8).
>> It is getting even more complicated, when you consider the fact, that in
>> Windows you have 2 different single-byte encodings, so-called ANSI (for GUI
>> applications) and OEM (for console). And alot of stuff making assumptions
>> without consulting with current status of things.
>> As convoluted the problem is, I think, we need some sort of solution, or at
>> the very least - documentation.

> Sorry, I can't provide an easy solution, and afaik this is documented.

I don't know, how it did not hit my head earlier, but now that I think about
it...
The fact `ls' does not print correct group names, but correctly translate
dates... eww, wait. There's something bad happens under the hood.

Please check this image: http://img513.imageshack.us/img513/43/jn7n.png
LANG is set (pre-set, before Cygwin tree start) to ru_RU.UTF-8 for mintty and
ru_RU.CP866 for console.
As you can see, existing group (i.e. - "Administrators") is properly
translated to chosen locale, which is not the case for group "Отсутствует".

The SID of the group "DAEMON2\Отсутствует" is S-1-5-21-1801674531-1644491937-1606980848-513

Cygwin show it as
-rwxr-xr-x    1 197612 197121    45597 мар 22  2013 xargs.exe

which matching your description for "Accounts from the local machine's user DB
(SAM)" (0x30000+513 =197121) (and yes, the leading part of the SID is matching
the one of the other users on this system), but the group is missing from
`wmic GROUP LIST' output. I presume, it was deleted at one point, or it is
remnants of the previous OS installation, transferred over with the disk to
this system. Either case explains the name of the group - "Missing" or
"Nonexistent". Means, it's name does not exist in current SAM database.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 14.02.2014, <21:19>

Sorry for my terrible english...

Re: New passwd/group handling in Cygwin - test results and observations

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3602 bytes --]

On Feb 16 17:52, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> > But this is a problem not different from Linux.  If you have a username
> >> > with non-ASCII chars, it will use *some* encoding in the passwd DB,
> >> > usually UTF-8 these days.  If you then change the codeset in your
> >> > application, you will still get your username in UTF-8.  It won't be
> >> > changed on the fly, just because your application calls setlocale.
> >> 
> >> I understand it (mostly), but there's actually two issues, not one.
> >> One issue is the display part, where names are output for user consumption.
> >> Another can be observed in, i.e., rsync, and file access in general (remember
> >> the discussion about accessing long directory names in unicode).
> >> Changing LANG variable DO matter for the latter, and you may only hope that
> >> whatever is output in the former case is actually printable (thank God, most
> >> of the time it actually is, in case of UTF-8).
> >> It is getting even more complicated, when you consider the fact, that in
> >> Windows you have 2 different single-byte encodings, so-called ANSI (for GUI
> >> applications) and OEM (for console). And alot of stuff making assumptions
> >> without consulting with current status of things.
> >> As convoluted the problem is, I think, we need some sort of solution, or at
> >> the very least - documentation.
> 
> > Sorry, I can't provide an easy solution, and afaik this is documented.
> 
> I don't know, how it did not hit my head earlier, but now that I think about
> it...
> The fact `ls' does not print correct group names, but correctly translate
> dates... eww, wait. There's something bad happens under the hood.
> 
> Please check this image: http://img513.imageshack.us/img513/43/jn7n.png
> LANG is set (pre-set, before Cygwin tree start) to ru_RU.UTF-8 for mintty and
> ru_RU.CP866 for console.
> As you can see, existing group (i.e. - "Administrators") is properly
> translated to chosen locale, which is not the case for group "Отсутствует".
> 
> The SID of the group "DAEMON2\Отсутствует" is S-1-5-21-1801674531-1644491937-1606980848-513
> 
> Cygwin show it as
> -rwxr-xr-x    1 197612 197121    45597 мар 22  2013 xargs.exe
> 
> which matching your description for "Accounts from the local machine's user DB
> (SAM)" (0x30000+513 =197121) (and yes, the leading part of the SID is matching
> the one of the other users on this system), but the group is missing from
> `wmic GROUP LIST' output. I presume, it was deleted at one point, or it is
> remnants of the previous OS installation, transferred over with the disk to
> this system. Either case explains the name of the group - "Missing" or
> "Nonexistent". Means, it's name does not exist in current SAM database.

No, it's the local group with the RID 513, which exists on *all*
machines, but is invisible, unless you call NetUserEnum, which
enumerates it, or you call NetUserGetInfo with the localized name.

This group is called "None" in the english OS versions, "Kein" in
the german version, "Aucun" in the french version, etc.  All local
users accounts have this group invariably set as their primary group.

For some reason Microsoft decided ages ago, that this group does
not show up in the GUI, and that it's not possible to change the primary
group outside of domains.  Why that is, I have no idea.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2135 bytes --]

On Feb 16 12:56, Corinna Vinschen wrote:
> On Feb 16 03:30, Warren Young wrote:
> > On Feb 15, 2014, at 5:50 AM, Corinna Vinschen wrote:
> > > No, that's not right.  We have two mechanisms implemented you can
> > > choose three out of four possible combinations:
> > > 
> > > files only
> > > db only
> > > files, then db
> > 
> > Yes, I realize that.
> > 
> > While composing the previous email, I considered a 2-Boolean design:
> > 
> >    ignore_db=false
> >    ignore_files=false
> > 
> > I rejected that design when I realized that ignore_files is redundant with respect to /etc/foo file existence.  If the files are present, use them.  If not, you have a DB-only configuration.
> 
> What if an admin wants to avoid that files are read at all, even if a
> user manages to generate them?

Anyway, I'm willing to switch from /etc/nsswitch.conf to something else.

Let's assume we go with /etc/cygwin.conf instead.

This wouldn't set a bad precedent in terms of syntax and keyword
differences to configuration files existing on Linux, and it would allow
to define more settings which have nothing to do with passwd/group stuff
in the long run.

So, how should this look like?  Let's do some free floating, 

  # /etc/cygwin.conf
  pwdgrp="files db"

  db_separator="+"
  db_cache="yes"
  db_prefix="auto"

And now the new one.  I have getpwent/getgrent practically ready to go.
It's just missing the setting to restrict the enumeration.  It occured
to me that an admin or a user might want to restrict the enumeration to
a specific set of domains.  What about something like this?

  db_enum="local primary trusted_dom1 trusted_dom2"

"local" and "primary" being fixed keywords for the local machine
accounts and the accounts from the primary domain.  Everything else is
treated as a domain name of a trusted domain, and the name can be given
as flat domain name as well as as dns domain name.

Does that make sense?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: New passwd/group handling in Cygwin - test results and observations

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3338 bytes --]

On Feb 16 15:34, Corinna Vinschen wrote:
> On Feb 16 17:52, Andrey Repin wrote:
> > > Sorry, I can't provide an easy solution, and afaik this is documented.
> > 
> > I don't know, how it did not hit my head earlier, but now that I think about
> > it...
> > The fact `ls' does not print correct group names, but correctly translate
> > dates... eww, wait. There's something bad happens under the hood.
> > 
> > Please check this image: http://img513.imageshack.us/img513/43/jn7n.png
> > LANG is set (pre-set, before Cygwin tree start) to ru_RU.UTF-8 for mintty and
> > ru_RU.CP866 for console.
> > As you can see, existing group (i.e. - "Administrators") is properly
> > translated to chosen locale, which is not the case for group "Отсутствует".
> > 
> > The SID of the group "DAEMON2\Отсутствует" is S-1-5-21-1801674531-1644491937-1606980848-513
> > 
> > Cygwin show it as
> > -rwxr-xr-x    1 197612 197121    45597 мар 22  2013 xargs.exe
> > 
> > which matching your description for "Accounts from the local machine's user DB
> > (SAM)" (0x30000+513 =197121) (and yes, the leading part of the SID is matching
> > the one of the other users on this system), but the group is missing from
> > `wmic GROUP LIST' output. I presume, it was deleted at one point, or it is
> > remnants of the previous OS installation, transferred over with the disk to
> > this system. Either case explains the name of the group - "Missing" or
> > "Nonexistent". Means, it's name does not exist in current SAM database.
> 
> No, it's the local group with the RID 513, which exists on *all*
> machines, but is invisible, unless you call NetUserEnum, which
> enumerates it, or you call NetUserGetInfo with the localized name.
> 
> This group is called "None" in the english OS versions, "Kein" in
> the german version, "Aucun" in the french version, etc.  All local
> users accounts have this group invariably set as their primary group.
> 
> For some reason Microsoft decided ages ago, that this group does
> not show up in the GUI, and that it's not possible to change the primary
> group outside of domains.  Why that is, I have no idea.

Maybe I have a simple solution.  From your other mails I gather you
suceeded building your own Cygwin DLL.

If you apply this patch:

Index: dcrt0.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/dcrt0.cc,v
retrieving revision 1.453
diff -u -p -r1.453 dcrt0.cc
--- dcrt0.cc	10 Feb 2014 10:45:50 -0000	1.453
+++ dcrt0.cc	16 Feb 2014 15:15:44 -0000
@@ -929,14 +929,14 @@ dll_crt0_1 (void *)
   /* Allocate cygheap->fdtab */
   dtable_init ();
 
+  /* Set internal locale to the environment settings. */
+  initial_setlocale ();
+
   uinfo_init ();	/* initialize user info */
 
   /* Connect to tty. */
   tty::init_session ();
 
-  /* Set internal locale to the environment settings. */
-  initial_setlocale ();
-
   if (!__argc)
     {
       PWCHAR wline = GetCommandLineW ();

and rebuild your Cygwin DLL, does it make a difference?  In theory
it should work as expected now even with ru_RU.CP866.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: New passwd/group handling in Cygwin - test results and observations

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]

On Feb 16 16:16, Corinna Vinschen wrote:
> On Feb 16 15:34, Corinna Vinschen wrote:
> > On Feb 16 17:52, Andrey Repin wrote:
> > > > Sorry, I can't provide an easy solution, and afaik this is documented.
> > > [...]
> Maybe I have a simple solution.  From your other mails I gather you
> suceeded building your own Cygwin DLL.
> 
> If you apply this patch:
> 
> Index: dcrt0.cc
> ===================================================================
> RCS file: /cvs/src/src/winsup/cygwin/dcrt0.cc,v
> retrieving revision 1.453
> diff -u -p -r1.453 dcrt0.cc
> --- dcrt0.cc	10 Feb 2014 10:45:50 -0000	1.453
> +++ dcrt0.cc	16 Feb 2014 15:15:44 -0000
> @@ -929,14 +929,14 @@ dll_crt0_1 (void *)
>    /* Allocate cygheap->fdtab */
>    dtable_init ();
>  
> +  /* Set internal locale to the environment settings. */
> +  initial_setlocale ();
> +
>    uinfo_init ();	/* initialize user info */
>  
>    /* Connect to tty. */
>    tty::init_session ();
>  
> -  /* Set internal locale to the environment settings. */
> -  initial_setlocale ();
> -
>    if (!__argc)
>      {
>        PWCHAR wline = GetCommandLineW ();
> 
> and rebuild your Cygwin DLL, does it make a difference?  In theory
> it should work as expected now even with ru_RU.CP866.

It works for me when using german umlauts with LANG=de_DE.cp1252, while
it didn't work before, so I applied this path to CVS.


HTH,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: New passwd/group handling in Cygwin - test results and observations

[Andrey Repin]

Greetings, Corinna Vinschen!

>> > > > Sorry, I can't provide an easy solution, and afaik this is documented.
>> > > [...]
>> Maybe I have a simple solution.  From your other mails I gather you
>> suceeded building your own Cygwin DLL.

I wasn't building cygwin1.dll specifically, but it appears it was actually
doing so as a by-product... :)

>> 
>> If you apply this patch:
>> 
>> Index: dcrt0.cc
>> ===================================================================
>> RCS file: /cvs/src/src/winsup/cygwin/dcrt0.cc,v
>> retrieving revision 1.453
>> diff -u -p -r1.453 dcrt0.cc
>> --- dcrt0.cc  10 Feb 2014 10:45:50 -0000      1.453
>> +++ dcrt0.cc  16 Feb 2014 15:15:44 -0000
>> @@ -929,14 +929,14 @@ dll_crt0_1 (void *)
>>    /* Allocate cygheap->fdtab */
>>    dtable_init ();
>>  
>> +  /* Set internal locale to the environment settings. */
>> +  initial_setlocale ();
>> +
>>    uinfo_init ();     /* initialize user info */
>>  
>>    /* Connect to tty. */
>>    tty::init_session ();
>>  
>> -  /* Set internal locale to the environment settings. */
>> -  initial_setlocale ();
>> -
>>    if (!__argc)
>>      {
>>        PWCHAR wline = GetCommandLineW ();
>> 
>> and rebuild your Cygwin DLL, does it make a difference?  In theory
>> it should work as expected now even with ru_RU.CP866.

> It works for me when using german umlauts with LANG=de_DE.cp1252, while
> it didn't work before, so I applied this path to CVS.

IC.
Picked the changes from CVS and rebuilding it right now.

...
Done and installed new library. All seems to be in order. Thanks for support!
And patch makes some sense to my untrained eye, too.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 16.02.2014, <20:11>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On Feb 16, 2014, at 4:56 AM, Corinna Vinschen wrote:

> What if an admin wants to avoid that files are read at all, even if a
> user manages to generate them?

Such an admin should adjust the permissions on /etc so normal users can’t create files there, as on a “real” *ix.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Warren Young!

>> What if an admin wants to avoid that files are read at all, even if a
>> user manages to generate them?

> Such an admin should adjust the permissions on /etc so normal users can’t
> create files there, as on a “real” *ix.

I approve of this product and/or service.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 16.02.2014, <20:24>

Sorry for my terrible english...

Re: Testers needed: New passwd/group handling in Cygwin

[Christopher Faylor]

On Sun, Feb 16, 2014 at 03:48:35PM +0100, Corinna Vinschen wrote:
>On Feb 16 12:56, Corinna Vinschen wrote:
>>On Feb 16 03:30, Warren Young wrote:
>>>On Feb 15, 2014, at 5:50 AM, Corinna Vinschen wrote:
>>>>No, that's not right.  We have two mechanisms implemented you can
>>>>choose three out of four possible combinations:
>>>>
>>>>files only db only files, then db
>>>
>>>Yes, I realize that.
>>>
>>>While composing the previous email, I considered a 2-Boolean design:
>>>
>>>ignore_db=false ignore_files=false
>>>
>>>I rejected that design when I realized that ignore_files is redundant
>>>with respect to /etc/foo file existence.  If the files are present, use
>>>them.  If not, you have a DB-only configuration.
>>
>>What if an admin wants to avoid that files are read at all, even if a
>>user manages to generate them?
>
>Anyway, I'm willing to switch from /etc/nsswitch.conf to something
>else.

Can you summarize why this is necessary?  I haven't really followed how
we got to the point where documented nscd.conf/nsswitch.conf
functionality wouldn't be sufficient.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

On Feb 16 12:09, Christopher Faylor wrote:
> On Sun, Feb 16, 2014 at 03:48:35PM +0100, Corinna Vinschen wrote:
> >On Feb 16 12:56, Corinna Vinschen wrote:
> >>On Feb 16 03:30, Warren Young wrote:
> >>>On Feb 15, 2014, at 5:50 AM, Corinna Vinschen wrote:
> >>>>No, that's not right.  We have two mechanisms implemented you can
> >>>>choose three out of four possible combinations:
> >>>>
> >>>>files only db only files, then db
> >>>
> >>>Yes, I realize that.
> >>>
> >>>While composing the previous email, I considered a 2-Boolean design:
> >>>
> >>>ignore_db=false ignore_files=false
> >>>
> >>>I rejected that design when I realized that ignore_files is redundant
> >>>with respect to /etc/foo file existence.  If the files are present, use
> >>>them.  If not, you have a DB-only configuration.
> >>
> >>What if an admin wants to avoid that files are read at all, even if a
> >>user manages to generate them?
> >
> >Anyway, I'm willing to switch from /etc/nsswitch.conf to something
> >else.
> 
> Can you summarize why this is necessary?  I haven't really followed how
> we got to the point where documented nscd.conf/nsswitch.conf
> functionality wouldn't be sufficient.

Documented for Linux or documented for Cygwin?

Most of the settings now done in nsswitch.conf have nothing to do with
what is done in nsswitch.conf on other OSes.  These settings have no
equivalent to the settings done in nscd.conf either.  Except for the
passwd: and group: settings, they are plain Cygwin only.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1.1: Type: text/plain, Size: 4052 bytes --]

On Feb 13 15:38, Corinna Vinschen wrote:
> Hi folks,
> 
> 
> this week I applied the first incarnation of the new passwd/group
> handling code to the Cygwin repository and after fixing a crash which
> manifested in Denis Excoffier's network, I think we're at a point
> which allows to push this forward.
> [...]

Ok guys, I just applied a patch implementing getpwent and getgrent,
and the way to configure the output is probably more detailed than
you ever wanted:

The default output of both functions consists of the so far cached
accounts, plus a fixed set of builtin accounts for backward
compatibility with mkpasswd/mkgroup output.  For instance, getpwent
output looks like this:

  corinna:*:1049577:1049701:Corinna Vinschen,U-VINSCHEN\corinna,S-1-5-21-2913048732-1697188782-3448811101-1001:/home/corinna:/bin/tcsh
  +SYSTEM:*:18:18:U-NT AUTHORITY\SYSTEM,S-1-5-18:/home/SYSTEM:/bin/bash
  +LOCAL SERVICE:*:19:19:U-NT AUTHORITY\LOCAL SERVICE,S-1-5-19:/home/LOCAL SERVICE:/bin/bash
  +NETWORK SERVICE:*:20:20:U-NT AUTHORITY\NETWORK SERVICE,S-1-5-20:/home/NETWORK SERVICE:/bin/bash
  +Administrators:*:544:513:U-BUILTIN\Administrators,S-1-5-32-544:/home/Administrators:/bin/bash
  +TrustedInstaller:*:328384:328384:U-NT SERVICE\TrustedInstaller,S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464:/home/TrustedInstaller:/bin/bash

and getgrent like this:

  vinschen:S-1-5-21-2913048732-1697188782-3448811101-1125:1049701:
  +Administrators:S-1-5-32-544:544:
  +SYSTEM:S-1-5-18:18:
  +TrustedInstaller:S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464:328384:

As long as we didn't decide to do it differently, the configuration
takes place in /etc/nsswitch.conf.  The new keyword is "db_enum", and
the value is a list of sources:

  db_enum: source...

A source can be

  none             No output from getpwent/getgrent at all.

  all              The opposite.  Output accounts from all known
                   sources, including all trusted domains.

  cache            List all accounts currently cached in memory.

  builtin          List the predefined builtin accounts for backward
                   compatibility.

  files            List the accounts from /etc/passwd or /etc/group.

  local            List all accounts from the local SAM.

  primary          List all accounts from the primary domain.

  alltrusted       List all accounts from all trusted domains.

  some.domain      List all accounts from the trusted domain some.domain.
                   The trusted domain can be given as Netbios flat name
                   (MY_DOMAIN) or as dns domain name (my_domain.corp).
                   In contrast to the aforementioned fixed source keywords,
                   distinct domain names are caseinsensitive.  Only domains
                   which are actually trusted domains are enumerated.
                   Unknown domains are simply ignored.

Please note that this functionality does *not* test if an account was
already listed from another source, so an account can easily show up
twice or three times.  Such a test would be rather tricky, nor does the
Linux implementation perform such test.

Here are a few examples for /etc/nsswitch.conf, which are hopefully
self-explaining:

  db_enum: none

  db_enum: all

  db_enum: cache files

  db_enum: cache local primary

  db_enum: local primary alltrusted

  db_enum: primary domain1.corp sub.domain.corp domain2.net

If those examples are *not* self-explaining, don't hesitate to ask.

Please fetch the latest snapshot from http://cygwin.com/snapshots/
It contains this change.  I attached two simple test applications
to this mail.

If it works nicely, I will rewrite mkpasswd and mkgroup to use the
Cygwin implementation as well, rather than implementing their own
enumerations.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #1.2: getpwent.c --]
[-- Type: text/plain, Size: 321 bytes --]

#include <stdio.h>
#include <pwd.h>

int
main ()
{
  struct passwd *pwd;

  setpwent ();
  while ((pwd = getpwent ()))
    printf ("%s:%s:%u:%u:%s:%s:%s\n",
	    pwd->pw_name,
	    pwd->pw_passwd,
	    pwd->pw_uid,
	    pwd->pw_gid,
	    pwd->pw_gecos,
	    pwd->pw_dir,
	    pwd->pw_shell);
  endpwent ();
  return 0;
}

[-- Attachment #1.3: getgrent.c --]
[-- Type: text/plain, Size: 233 bytes --]

#include <stdio.h>
#include <grp.h>

int
main ()
{
  struct group *grp;

  setgrent ();
  while ((grp = getgrent ()))
    printf ("%s:%s:%u:\n",
	    grp->gr_name,
	    grp->gr_passwd,
	    grp->gr_gid);
  endgrent ();
  return 0;
}

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-02-15 13:52 GMT+01:00 Corinna Vinschen:
>> Just tested and copied .bashrc to .profile and that solved the problem.
>
> Good, so it was really just a shell thingy.
>
>> I vote[1] for /bin/bash being the default shell.
>
> Done in CVS.

Just downloaded the 2014-02-18 snapshot, deleted that .profile and
things works as expected!

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> this week I applied the first incarnation of the new passwd/group
>> handling code to the Cygwin repository and after fixing a crash which
>> manifested in Denis Excoffier's network, I think we're at a point
>> which allows to push this forward.
>> [...]

> Ok guys, I just applied a patch implementing getpwent and getgrent,
> and the way to configure the output is probably more detailed than
> you ever wanted:

>   db_enum: source...

Setting

db_enum: local

seems to be really destructive.
I'm not good at reading straces, so... Can I make a useful trace for your
inspection?
So far, I'm unable to start anything with freshly rebuilt Cygwin1.dll
if db_enum set to local. Even uname -a crashes.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 19.02.2014, <18:42>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Feb 19 18:56, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> this week I applied the first incarnation of the new passwd/group
> >> handling code to the Cygwin repository and after fixing a crash which
> >> manifested in Denis Excoffier's network, I think we're at a point
> >> which allows to push this forward.
> >> [...]
> 
> > Ok guys, I just applied a patch implementing getpwent and getgrent,
> > and the way to configure the output is probably more detailed than
> > you ever wanted:
> 
> >   db_enum: source...
> 
> Setting
> 
> db_enum: local
> 
> seems to be really destructive.
> I'm not good at reading straces, so... Can I make a useful trace for your
> inspection?
> So far, I'm unable to start anything with freshly rebuilt Cygwin1.dll
> if db_enum set to local. Even uname -a crashes.

The last one is weird since uname doesn't call getpwent.

Anyway, with more help from Denis Excoffier I think I got it.  I'm
just building a new snapshot which should be up in half an hour or so.
Please give it a try.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> Setting
>> 
>> db_enum: local
>> 
>> seems to be really destructive.
>> I'm not good at reading straces, so... Can I make a useful trace for your
>> inspection?
>> So far, I'm unable to start anything with freshly rebuilt Cygwin1.dll
>> if db_enum set to local. Even uname -a crashes.

> The last one is weird since uname doesn't call getpwent.

That's what was my thought, too. Because (un)setting it back cures the crash.

> Anyway, with more help from Denis Excoffier I think I got it.  I'm
> just building a new snapshot which should be up in half an hour or so.
> Please give it a try.

Looking into it now.
Doesn't crash or anything so far. (With `db_enum: local' set.)
Tried a few utilities, as well as your example get*ent code.
BTW, can we have it as one "getent"[1] tool for the next release?

Going to play with different settings, I will post if I find anything
unexpected.

[1] http://linux.die.net/man/1/getent


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 19.02.2014, <23:11>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

On Feb 19 23:19, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> Setting
> >> 
> >> db_enum: local
> >> 
> >> seems to be really destructive.
> >> I'm not good at reading straces, so... Can I make a useful trace for your
> >> inspection?
> >> So far, I'm unable to start anything with freshly rebuilt Cygwin1.dll
> >> if db_enum set to local. Even uname -a crashes.
> 
> > The last one is weird since uname doesn't call getpwent.
> 
> That's what was my thought, too. Because (un)setting it back cures the crash.
> 
> > Anyway, with more help from Denis Excoffier I think I got it.  I'm
> > just building a new snapshot which should be up in half an hour or so.
> > Please give it a try.
> 
> Looking into it now.
> Doesn't crash or anything so far. (With `db_enum: local' set.)
> Tried a few utilities, as well as your example get*ent code.
> BTW, can we have it as one "getent"[1] tool for the next release?
> 
> Going to play with different settings, I will post if I find anything
> unexpected.
> 
> [1] http://linux.die.net/man/1/getent

That sounds pretty much like the tool we need to be able to get rid of
the `grep /etc/passwd' stuff we have in some of the service configuration
helper scripts.  Unfortunately it's a glibc tool so we probably have to
create our own.  For a start it would suffice to support only passwd
and group databases.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> [1] http://linux.die.net/man/1/getent

> That sounds pretty much like the tool we need to be able to get rid of
> the `grep /etc/passwd' stuff we have in some of the service configuration
> helper scripts.  Unfortunately it's a glibc tool so we probably have to
> create our own.

> For a start it would suffice to support only passwd and group databases.

Yup, was exactly what I thought.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 19.02.2014, <23:56>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Feb 19 23:57, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> [1] http://linux.die.net/man/1/getent
> 
> > That sounds pretty much like the tool we need to be able to get rid of
> > the `grep /etc/passwd' stuff we have in some of the service configuration
> > helper scripts.  Unfortunately it's a glibc tool so we probably have to
> > create our own.
> 
> > For a start it would suffice to support only passwd and group databases.
> 
> Yup, was exactly what I thought.

Just FYI, I succeeded in porting the Glibc getent.  I will provide this
as a standalone package ASAP, so we can start converting the service
configuration helper scripts even before we release the new passwd/group
code.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> >> [1] http://linux.die.net/man/1/getent
>> 
>> > That sounds pretty much like the tool we need to be able to get rid of
>> > the `grep /etc/passwd' stuff we have in some of the service configuration
>> > helper scripts.  Unfortunately it's a glibc tool so we probably have to
>> > create our own.
>> 
>> > For a start it would suffice to support only passwd and group databases.
>> 
>> Yup, was exactly what I thought.

> Just FYI, I succeeded in porting the Glibc getent.  I will provide this
> as a standalone package ASAP, so we can start converting the service
> configuration helper scripts even before we release the new passwd/group
> code.

O.o thursday of miracles.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 20.02.2014, <21:13>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 32654 bytes --]

On Feb 17 17:21, Corinna Vinschen wrote:
> On Feb 13 15:38, Corinna Vinschen wrote:
> > Hi folks,
> > 
> > 
> > this week I applied the first incarnation of the new passwd/group
> > handling code to the Cygwin repository and after fixing a crash which
> > manifested in Denis Excoffier's network, I think we're at a point
> > which allows to push this forward.
> > [...]
> 
> Ok guys, I just applied a patch implementing getpwent and getgrent,
> and the way to configure the output is probably more detailed than
> you ever wanted
> [...]

I just added the latest changes to my preliminary documentation, see
below.  Please have a look.  I'll rework this into our documetation at
one point, and it would be nice if it's more or less bugfree and,
hopefully, comprehensible at this point.


Thanks,
Corinna


=======
History
=======

For as long as Cygwin has existed, it has stored user and group
information in /etc/passwd and /etc/group files.  Under the assumption
that these files would never be too large, the first process in a
process tree, as well as every execing process within the tree would
parse them into structures in memory.  Thus every Cygwin process would
contain an expanded copy of the full information from /etc/passwd and
/etc/group.

This approach has a few downsides.  One of them is that the idea to have
always small files is flawed.  Another one is that reading the entire
file is most of the time entirely useless, since most processes only
need information on their own user and the primary group.  Last but not
least, the passwd and group files have to be maintained separately from
the already existing Windows user databases, the local SAM and Active
Directory.

On the other hand, we have to have a mapping between Windows SIDs and
POSIX uid/gid values (see [1]), so we rely on some mechanism to convert
SIDs to uid/gid values and vice versa.

Microsoft "Services for UNIX" (SFU) (which are unfortunately deprecated
since Windows 8/Server 2012) never used passwd/group files.  Rather, SFU
used a fixed, computational mapping between SIDs and POSIX uid/gid.  It
allows to generate uid/gid values from SIDs and vice versa.  The
mechanism is documented, albeit in a confusing way and spread over
multiple MSDN articles.  The Cygwin approach clones the mapping, with
just tiny differences for backward compatibility.


=================
How does it work?
=================

The following description assumes you're comfortable with the concept of
Windows SIDs and RIDs.  For a brief introduction, please read [1].

Cygwin's new mapping between SIDs and uid/gid values works in two ways.

- Read /etc/passwd and /etc/group files, like before, mainly for
  backward compatibility.

- If no files are present, or if an entry is missing in the files, ask
  Windows.

At least, that's the default behaviour now.  It will be configurable
using a file /etc/nsswitch.conf, which is discussed in a later section.
Let's explore the default for now.

If files are present, they will be scanned on demand as soon as a
mapping from SIDs to uid/gid or user names is required.  The new
mechanism will never read the entire file into memory, but only scan for
the requested entry and cache this one in memory[2].

If no entry is found, or no passwd or group file was present, Cygwin
will ask the OS.

Note:  If the first process in a Cygwin process tree determines that no
       /etc/passwd or /etc/group file is present, no other process in
       the entire process tree will try to read the files later on.
       This is done for self-preservation.  It's rather bad if the uid
       or gid of a user changes during the lifetime of a process tree.

       For the same reason, if you delete the /etc/passwd or /etc/group
       file, this will be ignored.  The passwd and group records read
       from the files will persist in memory until either a new
       /etc/passwd or /etc/group files is created, or you exit all
       processes in the current process tree.

       See the note in the section on /etc/nsswitch.conf for some
       comprehensive examples.

So if we've drawn a blank reading the files, we're going to ask the OS.
First thing, we ask the local machine for the SID or the user name.  The
OS functions LookupAccountSid and LookupAccountName[3] are pretty
intelligent.  They have all the stuff built in to ask for any account of
the local machine, the Active Directory domain of the machine, the
Global Catalog of the forest of the domain, as well as any trusted
domain of our forest for the information.  One OS call and we're
practically done...

Except, the calls only return the mapping between SID, account name and
the account's domain.  We don't have a mapping to POSIX uid/gid and
we're missing information on the user's home dir and login shell.

Let's discuss the SID<=>uid/gid mapping first.  Here's how it works.

- Well-known SIDs in the NT_AUTHORITY domain of the S-1-5-RID type, or
  aliases of the S-1-5-32-RID type are mapped to the uid/gid value
  RID[4].  For an overview of well-known SIDs, see [5].

  Examples:

    "SYSTEM" S-1-5-18     <=> uid/gid: 18
    "Users"  S-1-5-32-545 <=> uid/gid: 545

- Other well-known SIDs in the NT_AUTHORITY domain (S-1-5-X-RID):

    S-1-5-X-RID           <=> uid/gid: 0x1000 * X + RID

  Example:

    "NTLM Authentication"
    S-1-5-64-10           <=> uid/gid: 0x4000A == 262154

- Other well-known SIDs:

    S-1-X-Y               <=> uid/gid: 0x10000 + 0x100 * X + Y

  Example:

    "LOCAL" S-1-2-0       <=> uid/gid: 0x10200 == 66048
    "Creator Group"
    S-1-3-1               <=> uid/gid: 0x10301 == 66305

- Logon SIDs:

    The own LogonSid is converted to the fixed uid 0xfff == 4095 and
    named "CurrentSession".  Any other LogonSid is converted to the
    fixed uid 0xffe == 4094 and named "OtherSession".

- Mandatory Labels:

    S-1-16-RID            <=> uid/gid: 0x60000 + RID

  Example:

    "Medium Mandatory Level"
    S-1-16-8192           <=> uid/gid: 0x62000 == 401408

- Accounts from the local machine's user DB (SAM):

    S-1-5-21-X-Y-Z-RID    <=> 0x30000 + RID

  Example:

    "Administrator"
    S-1-5-X-Y-Z-500       <=> uid/gid: 0x301f4 == 197108

- Accounts from the machine's primary domain:
  
    S-1-5-21-X-Y-Z-RID    <=> 0x100000 + RID

  Example:

    "Domain Users"
    S-1-5-X-Y-Z-513       <=> 0x100201 == 1049089

- Accounts from a trusted domain of the machine's primary domain:

    S-1-5-21-X-Y-Z-RID    <=> trustPosixOffset(domain) + RID

  "trustPosixOffset"?  This needs a bit of explanation.  This value
  exists in Windows domains already since before Active Directory days.
  What happens is this.  If you create a domain trust between two
  domains, a trustedDomain entry will be added to both databases.  It
  describes how *this* domain trusts the *other* domain.  One attribute
  of a trust is a 32 bit value called "trustPosixOffset"  For each new
  trust, trustPosixOffset will get some automatic value.  In recent AD
  domain implementations, the first trusted domain will get
  trustPosixOffset set to 0x80000000.  Following domains will get lower
  values.  Unfortunately the domain admins are allowed to set the
  trustPosixOffset value for each trusted domain to some arbitrary 32
  bit value, no matter what the other trustPosixOffsets are seet to,
  thus allowing any kind of collisions between the trustPosixOffsets of
  domains.  That's not exactly helpful, but as the user of this value,
  we have to *trust* the domain admins to set trustPosixOffset to
  sensible values, or to keep it at the system chosen values.

  So, for the first (or only) trusted domain of your domain, the
  automatic offset is 0x80000000.  An example for a user of that trusted
  domain is:

    S-1-5-X-Y-Z-1234         <=> uid/gid 0x800004d2 == 2147484882

  There's only one problem with this approach.  Assuming you're running
  in the context of a local SAM user on a domain member machine.  Local
  users don't have the right to fetch this kind of domain information
  from the DC, they'll get permission denied.  In this case Cygwin will
  fake a, mostly, sensible trustPosixOffset value for this session.

- Local accounts from another machine in the network:

  There's no SID<=>uid/gid mapping implemented for this case.  The
  problem is, there's no way to generate a bijective mapping.  There's
  no central place which keeps an analogue value of the
  trustPosixOffset, and there's the additional problem that the
  LookupAccountSid and LookupAccountName functions cannnot resolve the
  SIDs, unless they know the name of the machine this SID comes from.
  And even then it will probably suffer a "Permission denied" when
  trying to ask the machine for its local account.

  SFU just prints the account RID in this case, Cygwin maps the account
  to the fake accounts "Unknown+User"/"Unknown+Group" with uid/gid -1.

Now we have a semi-bijective mapping between SIDs and POSIX uid/gid
values, but, given that we have potentially users and groups in
different domains having the same name, how do we uniquely differ
between them by name?  Well, we can do that by making their names unique
in a per-machine way.  Dependent on the domain membership of the
account, and dependent of the machine being a domain member or not, the
user and group names will be generated using a domain prefix and a
separator character between domain and account name.  The default
separator character is the plus sign, '+', as in SFU.

- Well-known SIDs will have the separator character prepended:

    "+SYSTEM", "+LOCAL", "+Medium Mandatory Level", ...

- If the machine is no domain member machine, only local accounts can be
  resolved into names, so for ease of use, just the account names are
  used as Cygwin user/group names:

    "corinna", "bigfoot", "None", ...

- If the machine is a domain member machine, all accounts from the
  primary domain of the machine are mapped to Cygwin names without
  domain prefix:

    "corinna", "bigfoot", "Domain Users", ...

  while accounts from other domains are prepended by their domain:

    "DOMAIN1+corinna", "DOMAIN2+bigfoot", "DOMAIN3+Domain Users", ...

- Local machine accounts of a domain member machine get a Cygwin user
  name the same way as accounts from another domain:  The local machine
  name gets prepended:

    "MYMACHINE+corinna", "MYMACHINE+bigfoot", "MYMACHINE+None", ...

- If LookupAccountSid fails, Cygwin checks the accounts against the
  known trusted domains.  If the account is from one of the trusted
  domains, an artificial account name is created.  It consists of the
  domain name, and a special name created from the account RID:

    "MY_DOM+User(1234)", "MY_DOM+Group(5678)"

  Otherwise we know nothing about this SID, so it will be mapped to the
  fake accounts "Unknown+User"/"Unknown+Group" with uid/gid -1.


==========================================
Cygwin user names, home dirs, login shells
==========================================

Obviously, if you don't maintain passwd and group files, you need to
have a way to maintain the other fields of a passwd entry as well.
Three things come to mind:

- You want to use a Cygwin user name different from your Windows
  user name.

- You want a home dir different from the default /home/$USER.

- You want to specify a different login shell than /bin/sh.

How this is done depends on your account being a domain account or a
local account.  Let's start with the default.  Assuming your Windows
account name is "bigfoot" and your domain is "MY_DOM".  Your default
passwd entry in absence of anything I'll describe below looks like this:

  bigfoot:*:<uid>:<gid>:U-MY_DOM\bigfoot,S-1-5-....:/home/bigfoot:/bin/bash

or, if your account is from a different domain than the primary domain of
the machine:

  MY_DOM+bigfoot:*:<uid>:<gid>:U-MY_DOM\bigfoot,S-1-5-....:/home/bigfoot:/bin/bash

Yes, the default homedir is still /home/bigfoot.

If your account is a domain account:

  Either create an /etc/passwd and/or /etc/group file with entries for
  your account and use that, just as before.

  Or, Cygwin will utilize the posixAccount/posixGroup attributes per RFC
  2307[6].  These attributes are by default available in Active Directory
  since Windows Server 2003 R2.  They are "not set", unless utilized by
  the (deprecated since Server 2012 R2) Active Directory "Server for
  NIS" feature.  The user attributes utilized by Cygwin are:

    uid                If set, will be used as Cygwin user name.
    uidNumber          See next section.
    gecos              Content will be added to the pw_gecos field.
    unixHomeDirectory  If set, will be used as Cygwin home directory.
    loginShell         If set, will be used as Cygwin login shell.

  The group attributes utilized by Cygwin are:

    cn                 If set, will be used as Cygwin group name.
    gidNumber          See next section.

  Apart from power shell scripting or inventing new CLI tools, these
  attributes can be changed using the "Attribute Editor" tab in the user
  properties dialog of the "Active Directory Users and Computers"
  MMC snap-in.  Alternatively, if the "Server for NIS" administration
  feature has been installed, there will be a "UNIX Attributes" tab which
  contains the required fields, except for the gecos field, which isn't
  really important anyway.  Last resort is "ADSI Edit".

  The primary group of a user is always the Windows primary group set in
  Active Directory and can't be changed.

If your machine is not a domain member machine or your account is a
local account for some reason:

  Either create an /etc/passwd and/or /etc/group file with entries for
  your account and use that, just as before.

  Or enter the information into the "Comment" field of your local user
  entry.  In the "Local Users and Groups" MMC snap-in it's called
  "Description".

  You can utilze this field even if you're running a "home edition" of
  Windows, using the command line.  The "net user" command allows to set
  all values in the SAM, even if the GUI is crippled.

  A Cygwin SAM comment entry looks like this:

    <cygwin key="value" key="value" [...] />

  The supported keys are

    name="value"      Sets the Cygwin user name to value.

    home="value"      Sets the Cygwin home dir to value.

    shell="value"     Sets the Cygwin login shell to value.

    group="value"     Sets the Cygwin primary group of the account
                      to value, provided that the user *is* already
                      a member of that group.  This allows to
                      override the default "None" primary group for
                      local accounts.
		      One nice idea here is, for instance group="Users".
		      This is the *Windows* name of the group, not the
		      Cygwin name, assuming they differ.

    unix="value"      Sets the NFS/Samba uid of the user to the decimal
                      value.  See the next chapter.

  The <cygwin .../> string can start at any point in the comment, but
  you have to follow the rules:

  - It starts with "<cygwin " and ends with "/>".
  - The "cygwin" string and the key names have to be lowercase.
  - No spaces between key and "value", just the equal sign.
  - The value must be placed within double quotes and it must not
    contain a double quote itself.  The double quotes are required
    for the decimal values as well!

  CMD example:

    net user corinna /comment:"<cygwin home=\"/home/foo\"/>"

  Bash example (use single quotes):

    net user corinna /comment:'<cygwin home="/home/foo"/>'

  For changing group comments, use the `net localgroup' command.  The
  supported key/value pair for groups are

    name="value"      Sets the Cygwin group name to value.

    unix="value"      Sets the NFS/Samba gid of the group to the
                      decimal value.  See the next chapter.


===================
NFS account mapping
===================

Microsoft's NFS client does not map the uid/gid values on the NFS shares
to SIDs.  There's no such thing as a (fake) security descriptor returned
to the application.  Rather, via an undocumented API an applications can
fetch RFC 1813 compatible NFSv3 stat information from the share[7].
This is what Cygwin is using to show stat information for files on NFS
shares.

The problem is, while all other information in this stat record, like
timestamps, file size etc., can be used by Cygwin, Cygwin had no way to
map the values of the st_uid and st_gid members to a Windows SID for a
long time.  So it just faked the file owner info and claimed that it's
you.

However, SFU has, over time, developed multiple methods to map UNIX
uid/gid values on NFS shares to Windows SIDs.  You'll find the full
documentation of the mapping methods in [8].

Cygwin now utilizes the RFC 2307 mapping for this purpose.  This is most
of the time provided by an AD domain, but it could also be a standalone
LDAP mapping server.  Per RFC 2307, the uid is in the attribute
uidNumber.  For groups, the gid is in the gidNumber attribute.

When Cygwin stat's files on an NFS share, it asks the mapping server via
LDAP in two different ways, depending on the role of the mapping server.

- If the server is an AD domain controller, it asks for an account with
  uidNumber attribute == st_uid field of the stat record returned by
  NFS.  If an account matches, AD returns the Windows SID, so we have an
  immediate mapping from UNIX uid to a Windows SID, if the user account
  has a valid uidNumber attribute.  For groups, the method is the same,
  just that Cygwin asks for a group with gidNumber attribute == st_gid
  field of the stat record.

- If the server is a standalone LDAP mapping server Cygwin asks for the
  same uidNumber/gidNumber attributes, but it can't expect that the LDAP
  server knows anything about Windows SIDs.  Rather, the mapping server
  returns the account name.  Cygwin then asks the DC for an account with
  this name, and if that succeeds, we have a mapping between UNIX
  uid/gid and Windows SIDs.

The mapping will be cached for the lifetime of the process, and inherited
by child processes.


=====================
Samba account mapping
=====================

A fully set up Samba with domain integration is running winbindd to
map Window SIDs to artificially created UNIX uids and gids, and this
mapping is transparent within the domain, so Cygwin doesn't have to do
anything special.

However, setting up winbindd isn't for everybody, and it fails to map
Windows accounts to already existing UNIX users or groups.  In contrast
to NFS, Samba returns security descriptors, but unmapped UNIX accounts
get special SIDs:

- A UNIX user account with uid X is mapped to the Windows SID S-1-22-1-X.

- A UNIX group account with gid X is mapped to SID S-1-22-2-X.

As you can see, even though we have SIDs, they just reflect the actual
uid/gid values on the UNIX box in the RID value.  It's only marginally
different from the NFS method, so why not just use the same method as
for NFS?

That's what Cygwin will do.  If it encounters a S-1-22-x-y SID, it
will perform the same RFC 2307 mapping as for NFS shares.

For home users without any Windows domain or LDAP server per RFC 2307,
but with a Linux machine running Samba, just add this information to
your SAM account.  Assuming the uid of your Linux user account is 505
and the gid of your primary group is, say, 100, just add the values to
your SAM user and group accounts.  The following example assumes you
didn't already add something else to the comment field.

To your user's SAM comment (remember: called "Description" in the GUI),
add:

  <cygwin group="Users" unix="505"/>

To the user's group SAM comment add:

  <cygwin unix="100"/>

This should be sufficient to work on your Samba share and to see
all files owned by your Linux user account as your files.


===========================
The /etc/nsswitch.conf file
===========================

Last, but not least, let's talk about the way to configure how the
mapping works on your machine.  On Linux and some other UNIXy OSes, we
have a file called /etc/nsswitch.conf[9].  One part of it is to specify
how the passwd and group entries are generated.  That's what Cygwin now
provides as well.

The /etc/nsswitch.conf file is optional.  If you don't have one, Cygwin
uses sensible defaults.

Note:  The /etc/nsswitch.conf file is read exactly once by the first
       process of a Cygwin process tree.  If there was no /etc/nsswitch.conf
       file when this first process started, then no other process in
       the running Cygwin process trees will try to read the file.

       If you create or change /etc/nsswitch.conf, you need to restart
       all Cygwin processes that need to see the change.  If the process
       you want to see the change is a child of another process, you
       need to restart all of that process's parents, too.

       For example, if you run Vim inside the default Cygwin Terminal,
       Vim is a child of your shell, which is a child of mintty.exe.  If
       you edit /etc/nsswitch.conf in that Vim instance, your shell
       won't immediately see the change, nor will Vim if you restart it
       from that same shell instance.  This is because both are getting
       their nsswitch information from their ancestor, mintty.exe.  You
       need to start a fresh terminal window for the change to take
       effect.

       By contrast, if you leave that Cygwin Terminal window open after
       making the change to /etc/nsswitch.conf, then restart a Cygwin
       service like cron, cron will see the change, because it is not a
       child of mintty.exe or any other Cygwin process. (Technically, it
       is a child of cygrunsrv, but that instance also restarts when you
       restart the service.)

       The reason we point all this out is that the requirements for
       restarting things are not quite as stringent as when you replace
       cygwin1.dll. If you have three process trees, you have three
       independent copies of the nsswitch information.  If you start a
       fresh process tree, it will see the changes.  As long as any
       process in an existing process tree remains running, all
       processes in that tree will continue to use the old information.

So, what mischief can we perform with /etc/nsswitch.conf?  To explain,
lets have a look into an /etc/nsswitch.conf file set up to all default
values:

  # /etc/nsswitch.conf
  passwd: files db
  group:  files db

  db_prefix:    auto
  db_separator: +
  db_enum:      cache builtin
  db_cache:     yes

The first line, starting with a hash '#' is a comment.  The hash
character starts a comment, just as in shell scripts.  Everything up to
the end of the line is ignored.  So this:

  foo:  bar # baz

means, for the entry "foo", do "bar", ignore everything after the hash
sign.  "baz" is only a comment.

The other lines define the available settings.  The first word up to a
colon is a keyword.  Note that the colon *must* follow immediately after
the keyword.  This is a valid line:

  foo: bar

This is not valid:

  foo  :  bar

Apart from this restriction, the reminder of the line can have as
may spaces and TABs as you like.  This is a valid line:

        foo:                       bar                baz

Now let's have a look at the available keywords and settings.

The two lines starting with the keywords "passwd" and "group" define
where Cygwin gets its passwd and group information from.  "files" means,
fetch the information from the corresponding file in the /etc directory.
"db" means, fetch the information from the Windows account databases,
the SAM for local accounts, Active Directory for domain account.
Examples:

  passwd: files

Read passwd entries only from /etc/passwd.

  group: db

Read group entries only from SAM/AD.

  group: files # db

Read group entries only from /etc/group ("db" is ignored due to the
preceding hash sign).

  passwd: files db

Read passwd entries from /etc/passwd.  If a user account isn't found,
try to find it in SAM or AD.  This is the default for both, passwd
and group information.

  group: db files

This is a valid entry, but the order will be ignored by Cygwin.  If
both, files and db are specified, Cygwin will always try the files
first, then the db.

The remaining entries define certain aspects of the Windows account
database search.  "db_prefix" determines how the Cygwin user or group
name is created:

  db_prefix: auto

    This is the default.  If your account is from the primary domain of
    your machine, or if your machine is a standalone machine (not a domain
    member), your Cygwin account name is just the Windows account
    name.
    If your account is from another domain, or if you're logged in as
    local user on a domain machine, the Cygwin username will be the
    combination of Windows domainname and username, with the separator
    char in between:

      MY_DOM+username      (foreign domain)
      MACHINE+username     (local account)
    
    Builtin accounts have just the separator char prepended:

      +LOCAL
      +Users

    Unknown accounts on NFS or Samba shares (that is, accounts which
    cannot be mapped to Windows user accounts via RFC 2307) get a
    Cygwin account name consisting of the artificial domains "UNIX_User"
    or "UNIX_Group" and the uid/gid value, for instance:

      UNIX_User+0          (root)
      UNIX_Group+10        (wheel)

  db_prefx: primary

    Like "auto", but primary domain accounts will be prepended by
    the domainname as well.

  db_prefix: always

    All accounts, even the builtin accounts, will have the domain
    name prepended:

      BUILTIN+Users

    As a special case, if the Cygwin account name differs from the
    Windows account name, it will be prepended by the artificial domain
    name "Posix_User" or "Posix_Group" if db_prefix is set to "always":

      Posix_User+cygwin_user_name
      Posix_Group+cygwin_group_name

"db_separator" defines the spearator char used to prepend the domain
name to the user or group name.  The default is '+':

    MY_DOM+username

With "db_separator", you can define any ASCII char except space,
tab, carriage return, line feed, and the colon, as separator char.
Example:

  db_separator: \

    MY_DOM\username

"db_enum" defines the depth of a database search, if an application
calls one of the enumeration functions getpwent[10] or getgrent[11].
The problem with these functions is, they neither allow to define how many
entries will be enumerated when calling them in a loop, nor do they
allow to add some filter criteria.  They were designed back in the days,
when only /etc/passwd and /etc/group files existed and the number of
user accounts on a typical UNIX system was seldomly a three-digit
number.

These days, with user and group databases sometimes going in the
six-digit range, they are a potential burden.  For that reason, Cygwin
does not enumerate all user or group accounts by default, but rather
just a very small list, consisting only of the accounts cached in memory
by the current process, as well as a handful of predefined builtin
accounts.

"db_enum" allows to specify the accounts to enumerate in a fine-grained
way.  It takes a list of sources as argument:

  db_enum:  source1 source2 ...

The recognized sources are the following:

  none             No output from getpwent/getgrent at all.

  all              The opposite.  Enumerates accounts from all known
                   sources, including all trusted domains.

  cache            Enumerate all accounts currently cached in memory.

  builtin          Enumerate the predefined builtin accounts for backward
                   compatibility.  These are five passwd accounts
                   (SYSTEM, LocalService, NetworkService, Administrators,
                   TrustedInstaller) and two group accounts (SYSTEM and
                   TrustedInstaller).

  files            Enumerate the accounts from /etc/passwd or /etc/group.

  local            Enumerate all accounts from the local SAM.

  primary          Enumerate all accounts from the primary domain.

  alltrusted	   Enumerate all accounts from all trusted domains.

  some.domain	   Enumerate all accounts from the trusted domain some.domain.
                   The trusted domain can be given as Netbios flat name
                   (MY_DOMAIN) or as dns domain name (my_domain.corp).
                   In contrast to the aforementioned fixed source keywords,
                   distinct domain names are caseinsensitive.  Only domains
                   which are actually trusted domains are enumerated.
                   Unknown domains are simply ignored.

Please note that getpwent/getgrent do *not* test if an account was
already listed from another source, so an account can easily show up
twice or three times.  Such a test would be rather tricky, nor does the
Linux implementation perform such test.  Here are a few examples for
/etc/nsswitch.conf:

  db_enum: none

    No output from getpwent/getgrent at all.  The first call to the
    function immediately returns a NULL pointer.

  db_enum: cache files

    Enumerate all accounts cached by the current process, plus all entries
    from either the /etc/passwd or /etc/group file.

  db_enum: cache local primary

    Enumerate all accounts cached by the current process, all accounts
    from the SAM of the local machine, and all accounts from the
    primary domain of the machine.

  db_enum: local primary alltrusted

    Enumerate the accounts from the machine's SAM, from the primary domain
    of the machine, and from all trusted domains.

  db_enum: primary domain1.corp sub.domain.corp domain2.net

    Enumerate the accounts from the primary domain and from the domains
    domain1.corp, sub.domain.corp and domain2.net.

  db_enum: all

    Enumerate everything and the kitchen sink.

"db_cache" allows to specify if user and group data should be cached in
the process at all or not.  Default is "yes".  "no" is an experimental
feature.  The idea is this.  Assuming you're working in a volatile
domain environment, and you have long-running processes like sshd.
You might want sshd to recognize changes in the existing user entries
as soon as they occur.  Caching user data in sshd *might* not be helpful
in such a scenario.  However, the user data which gets changed a lot
is typically *not* the data required for login purposes, except for
the password, and that doesn't matter in Cygwin's case.  So take this
feature with a grain of salt.


==========
Footnotes:
==========

[1] http://cygwin.com/cygwin-ug-net/ntsec.html

[2] This may change.  Right now the file is read in 32K chunks, but
    we could easily read the file in 64K chunks and, if we find the
    file is < 64K anyway, just cache the entire bunch, like before.
    Not implemented yet, but something to keep in mind.

[3] http://msdn.microsoft.com/en-us/library/windows/desktop/aa379166%28v=vs.85%29.aspx
    http://msdn.microsoft.com/en-us/library/windows/desktop/aa379159%28v=vs.85%29.aspx

[4] This is where Cygwin differs from SFU.  The reason is that we need
    the old uid/gid values for backward compatibility.  There are Cygwin
    packages (cron, for instance) who rely on the fact that the uid of
    SYSTEM is 18.  In SFU, these accounts get mapped like the other
    built in SIDs.

[5] http://support.microsoft.com/kb/243330

[6] https://tools.ietf.org/html/rfc2307

[7] https://tools.ietf.org/html/rfc1813

[8] http://msdn.microsoft.com/en-us/library/cc980032.aspx

[9] http://linux.die.net/man/5/nsswitch.conf

[10] http://linux.die.net/man/3/getpwent

[11] http://linux.die.net/man/3/getgrent



-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Ken Brown]

On 2/21/2014 9:55 AM, Corinna Vinschen wrote:
> - You want to specify a different login shell than /bin/sh.

Typo: You forgot to change /bin/sh to /bin/bash.

Ken


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

On Feb 21 11:11, Ken Brown wrote:
> On 2/21/2014 9:55 AM, Corinna Vinschen wrote:
> >- You want to specify a different login shell than /bin/sh.
> 
> Typo: You forgot to change /bin/sh to /bin/bash.
> 
> Ken

Fixed, thank you.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> This is a pretty intrusive change, in need of some serious testing, so
> I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> http://cygwin.com/snapshots/ contains the changes, including the latest
> bugfix.

I've tested the 2014-02-19 snapshot at work.  Two problems:

1. Running "id" takes 13 seconds to fetch my 440 group memberships
(possibly there are some users that would be in ~10x as many groups).
Caching doesn't seem to be effective for this since the next several
invocations of "id" take the same time.  During most of that time you
actually can't ^C the process, lsass is growing a few dozen threads and
seems to be talking to the DC.  Falling back to use just the /etc files
makes this work really fast (much faster than without the snapshot).

2. I use a few volumes on NetApp filers that have security set up so
that you can't change attributes.  That means POSIX permissions are
always listed as "0000".  I uausally mount these noacl, but when I
access them via their UNC path (for instance when Windows runs a script
from a CWD on that volume, then Perl reports false for file test
operators (-x, -w) other than existence.  Backing out the snapshot
reverts to the previous behaviour of these test operators correctly
determining that my effective rights (via normal and extended security
attributes tied to a group memberships) are sufficient.  The shell
(bash, tcsh) test operators work correctly, but I don't know what Perl
is doing differently.  This is independent of whether I use just the
/etc files or full AD lookup (switched via /etc/nsswitch.conf) and
currently a showstopper for me.  I've booby-trapped a few checks to
always return true and the actual creation of directories and files then
succeeds (as it has always done), but I can't just pull out every one of
these from all scripts.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Achim Gratz!

> 2. I use a few volumes on NetApp filers that have security set up so
> that you can't change attributes.  That means POSIX permissions are
> always listed as "0000".

That seems like a bug elsewhere. Being able to change permissions shouldn't
restrict you from requesting them.

> I uausally mount these noacl, but when I access them via their UNC path (for
> instance when Windows runs a script from a CWD on that volume, then Perl
> reports false for file test operators (-x, -w) other than existence.

You can change that with cygdrive prefix options.
Adjust your fstab line for it to include noacl, and operations on UNC paths
should run better, according to your setup.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 25.02.2014, <22:45>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Andrey Repin writes:
> That seems like a bug elsewhere. Being able to change permissions shouldn't
> restrict you from requesting them.

There is no bug, not in Cygwin nor anywhere else.  The standard file
attributes are all cleared and since I can't set any of them (a policy
which gets inherited) that stays for all files and directories that are
created.  Before you ask, I can't change any of that.

> You can change that with cygdrive prefix options.

That'd be a possible workaround, thanks.

> Adjust your fstab line for it to include noacl, and operations on UNC paths
> should run better, according to your setup.

I'd rather want to know why the snapshot triggers such a change in
behaviour and why the notion of the shells and Perl don't match.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2877 bytes --]

On Feb 25 19:14, Achim Gratz wrote:
> Corinna Vinschen writes:
> > This is a pretty intrusive change, in need of some serious testing, so
> > I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> > http://cygwin.com/snapshots/ contains the changes, including the latest
> > bugfix.
> 
> I've tested the 2014-02-19 snapshot at work.  Two problems:
> 
> 1. Running "id" takes 13 seconds to fetch my 440 group memberships
> (possibly there are some users that would be in ~10x as many groups).
> Caching doesn't seem to be effective for this since the next several

The stuff in the `id' application is not cached at all.  Caching is
inherited from the parent process, but the parent never asked for all
your groups so it hasn't cached this information.  Every invocation of
id has to request the group info anew.

> invocations of "id" take the same time.  During most of that time you
> actually can't ^C the process, lsass is growing a few dozen threads and
> seems to be talking to the DC.  Falling back to use just the /etc files
> makes this work really fast (much faster than without the snapshot).

Do you have a very slow connection to your DC by any chance?  I admit
that I never tested with 440 groups, only with about 30 or so, but 13
seconds sounds *very* lame.

OTOH, this isn't *quite* unexpected.  Right now, the LDAP connection to
the DC is opened and closed for every single account request.  I wasn't
sure yet if the ldap connection should be opened only once per process
and then stay open for the rest of the process lifetime.  This sounds so
much like wasting sockets...

> 2. I use a few volumes on NetApp filers that have security set up so
> that you can't change attributes.  That means POSIX permissions are
> always listed as "0000".  I uausally mount these noacl, but when I
> access them via their UNC path (for instance when Windows runs a script
> from a CWD on that volume, then Perl reports false for file test
> operators (-x, -w) other than existence.  Backing out the snapshot
> reverts to the previous behaviour of these test operators correctly
> determining that my effective rights (via normal and extended security
> attributes tied to a group memberships) are sufficient.  The shell
> (bash, tcsh) test operators work correctly, but I don't know what Perl
> is doing differently.

The fact that the shells are doing it right seems to indicate that this
isn't a generic problem.  I can't debug this, though.  Can you see if
you can figure out what's going on under the hood?  Does strace show
anything of interest?  Can we perhaps set up some joint debugging via
private mail during the next couple of days?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> The stuff in the `id' application is not cached at all.  Caching is
> inherited from the parent process, but the parent never asked for all
> your groups so it hasn't cached this information.  Every invocation of
> id has to request the group info anew.

OK, then I was misunderstanding what caching meant.  I was operating
under the assumption that there'd be something like a session cache that
serves all Cygwin processes in the same process tree.

> Do you have a very slow connection to your DC by any chance?  I admit
> that I never tested with 440 groups, only with about 30 or so, but 13
> seconds sounds *very* lame.

That's around 33 lookups per second; not great, but also not bad (I'm
far from the only user of that DC obviously).  I'll have to check what
our web application server is doing (albeit it uses a connection pool to
amortize the connection overhead), but I don't think it'll do much more
than about 200 peak.  The answers can be insanely large depending on
what and how you ask, too.  In any case, based on the fact that I'm
certainly not in as many groups as some other folks, I don't think I can
drop file-based operation at the moment (which I really hoped I could).

> OTOH, this isn't *quite* unexpected.  Right now, the LDAP connection to
> the DC is opened and closed for every single account request.  I wasn't
> sure yet if the ldap connection should be opened only once per process
> and then stay open for the rest of the process lifetime.  This sounds so
> much like wasting sockets...

See above, there's a timeout on each pooled connection, again I'll have
to ask how long it is (I think somewhere in the single digits seconds
range, we don't want to tie up the DC for too long).

> The fact that the shells are doing it right seems to indicate that this
> isn't a generic problem.  I can't debug this, though.  Can you see if
> you can figure out what's going on under the hood?  Does strace show
> anything of interest?  Can we perhaps set up some joint debugging via
> private mail during the next couple of days?

I can't dig further into this for the next two weeks.  I need to get
some work done, so I have rolled the snapshot back for now.  I'll get
back to you when this is out of the way, thanks for the offer.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3706 bytes --]

On Feb 25 21:25, Achim Gratz wrote:
> Corinna Vinschen writes:
> > The stuff in the `id' application is not cached at all.  Caching is
> > inherited from the parent process, but the parent never asked for all
> > your groups so it hasn't cached this information.  Every invocation of
> > id has to request the group info anew.
> 
> OK, then I was misunderstanding what caching meant.  I was operating
> under the assumption that there'd be something like a session cache that
> serves all Cygwin processes in the same process tree.

Right now the cache is in the cygheap, so it's per-process, with
inheritance from the parent process.  I am mulling over adding other
caching methods, but I'm reluctant to add another shared memory region
of variable and potentially big size to the 32 bit version of Cygwin.
I'm not as reluctant to do this for the 64 bit version, but I *am*
reluctant to add different code for 32 and 64 bit.

> > Do you have a very slow connection to your DC by any chance?  I admit
> > that I never tested with 440 groups, only with about 30 or so, but 13
> > seconds sounds *very* lame.
> 
> That's around 33 lookups per second; not great, but also not bad (I'm
> far from the only user of that DC obviously).  I'll have to check what
> our web application server is doing (albeit it uses a connection pool to

Sorry, I don't grok this.  What has a web application server to do with
asking a DC for user info?

> than about 200 peak.  The answers can be insanely large depending on
> what and how you ask, too.  In any case, based on the fact that I'm
> certainly not in as many groups as some other folks, I don't think I can
> drop file-based operation at the moment (which I really hoped I could).

Erm... how often are you calling id, usually?  Also, we're in the early
stages of testing this change.  The idea is not that you just switch,
the idea is that we *test* this and I get enough feedback to be able to
ease the biggest pains.

Other than that, I just had an in-shower inspiration how to speed up
`id' specificially.  The getgroups(2) call is in the center of this and
I could probably speed up the stuiff a lot by opening the LDAP
connection in getgroups only once. 

Also, more radically, if we drop the functionality to define another
group name for a group, we could drop the requirement to open an LDAP
connection to fetch group information to the DC entirely(*).  This would
only affect domain groups, local groups could still have different
names.  But I'm already wondering for a couple of days if having a
Cygwin group name different from the Windows group name is really
necessary at all.  I added this years ago for fun, but there's no
serious reason I can think of which would require to keep up with this.

(*) Assuming the group info is cached in the local LSA, which is
    pretty likely for the groups of the current user.

> > The fact that the shells are doing it right seems to indicate that this
> > isn't a generic problem.  I can't debug this, though.  Can you see if
> > you can figure out what's going on under the hood?  Does strace show
> > anything of interest?  Can we perhaps set up some joint debugging via
> > private mail during the next couple of days?
> 
> I can't dig further into this for the next two weeks.  I need to get
> some work done, so I have rolled the snapshot back for now.  I'll get
> back to you when this is out of the way, thanks for the offer.

Sigh.  Testing in this tempo will take ages.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

> Also, more radically, if we drop the functionality to define another
> group name for a group, we could drop the requirement to open an LDAP
> connection to fetch group information to the DC entirely(*).  This would
> only affect domain groups, local groups could still have different
> names.  But I'm already wondering for a couple of days if having a
> Cygwin group name different from the Windows group name is really
> necessary at all.

If you ask me, it not only unnecessary, but potentially confusing, if not
dangerous.
Given that you've presumable fixed the group name mangling on locale changes,
I think all the necessity for this functionality is now gone.

> I added this years ago for fun, but there's no
> serious reason I can think of which would require to keep up with this.

> (*) Assuming the group info is cached in the local LSA, which is
>     pretty likely for the groups of the current user.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 26.02.2014, <02:17>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

> Other than that, I just had an in-shower inspiration how to speed up
> `id' specificially.  The getgroups(2) call is in the center of this and
> I could probably speed up the stuiff a lot by opening the LDAP
> connection in getgroups only once. 

Isn't local SAM database augmented with domain entries, when machine is part
of the domain? 


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 26.02.2014, <06:16>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

> Sorry, I don't grok this.  What has a web application server to do with
> asking a DC for user info?

We have one of these that does a lot of DC lookups because it authenticates
all users.  It's also in a much faster network, so I can check there what
the lookup rate could be reasonably expected to be.

> Erm... how often are you calling id, usually?

I'm currently doing this in the login process to figure out whether the
prompt should show "root" powers.  I'll have to figure out something else to
do instead.

> Also, we're in the early
> stages of testing this change.  The idea is not that you just switch,
> the idea is that we *test* this and I get enough feedback to be able to
> ease the biggest pains.

Understood.  Until now I had to generate passwd and group files and I was
hoping that the need for doing that would go away (I'd also need to talk to
our AD folks so they start populating the correct fields).

> Other than that, I just had an in-shower inspiration how to speed up
> `id' specificially.  The getgroups(2) call is in the center of this and
> I could probably speed up the stuiff a lot by opening the LDAP
> connection in getgroups only once. 

Thursday?  :-)

> Also, more radically, if we drop the functionality to define another
> group name for a group, we could drop the requirement to open an LDAP
> connection to fetch group information to the DC entirely(*).  This would
> only affect domain groups, local groups could still have different
> names.  But I'm already wondering for a couple of days if having a
> Cygwin group name different from the Windows group name is really
> necessary at all.  I added this years ago for fun, but there's no
> serious reason I can think of which would require to keep up with this.
> 
> (*) Assuming the group info is cached in the local LSA, which is
>     pretty likely for the groups of the current user.

That would also work for me (I don't think I've ever used that feature
consciously).

> Sigh.  Testing in this tempo will take ages.

Sorry, but that's not my decision to make in this case.  I'll see if I can
sneak in something until the end of the week.


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Feb 26 06:17, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> > Other than that, I just had an in-shower inspiration how to speed up
> > `id' specificially.  The getgroups(2) call is in the center of this and
> > I could probably speed up the stuiff a lot by opening the LDAP
> > connection in getgroups only once. 
> 
> Isn't local SAM database augmented with domain entries, when machine is part
> of the domain? 

No.  The local SAM only ever contains local and well-known accounts.
Side-effect: Even the supplementary well-known accounts defined in AD
are not resolved by LookupAccountSid(NULL, ...).


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2812 bytes --]

On Feb 26 08:09, Achim Gratz wrote:
> > Sorry, I don't grok this.  What has a web application server to do with
> > asking a DC for user info?
> 
> We have one of these that does a lot of DC lookups because it authenticates
> all users.  It's also in a much faster network, so I can check there what
> the lookup rate could be reasonably expected to be.
> 
> > Erm... how often are you calling id, usually?
> 
> I'm currently doing this in the login process to figure out whether the
> prompt should show "root" powers.  I'll have to figure out something else to
> do instead.

No, you don't.  I'm actually doing the same.  Let's keep up with this
and try to make Cygwin faster in the first place.

> > Also, we're in the early
> > stages of testing this change.  The idea is not that you just switch,
> > the idea is that we *test* this and I get enough feedback to be able to
> > ease the biggest pains.
> 
> Understood.  Until now I had to generate passwd and group files and I was
> hoping that the need for doing that would go away (I'd also need to talk to
> our AD folks so they start populating the correct fields).

Yup.  Feedback from AD admins is greatly appreciated.

> > Other than that, I just had an in-shower inspiration how to speed up
> > `id' specificially.  The getgroups(2) call is in the center of this and
> > I could probably speed up the stuiff a lot by opening the LDAP
> > connection in getgroups only once. 
> 
> Thursday?  :-)

Hmm, probably.

> > Also, more radically, if we drop the functionality to define another
> > group name for a group, we could drop the requirement to open an LDAP
> > connection to fetch group information to the DC entirely(*).  This would
> > only affect domain groups, local groups could still have different
> > names.  But I'm already wondering for a couple of days if having a
> > Cygwin group name different from the Windows group name is really
> > necessary at all.  I added this years ago for fun, but there's no
> > serious reason I can think of which would require to keep up with this.
> > 
> > (*) Assuming the group info is cached in the local LSA, which is
> >     pretty likely for the groups of the current user.
> 
> That would also work for me (I don't think I've ever used that feature
> consciously).
> 
> > Sigh.  Testing in this tempo will take ages.
> 
> Sorry, but that's not my decision to make in this case.  I'll see if I can
> sneak in something until the end of the week.

That would be nice.  Especially since your mail was the last straw
which keeps me from having to do *shudder* more documentation.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3605 bytes --]

On Feb 26 11:02, Corinna Vinschen wrote:
> On Feb 26 08:09, Achim Gratz wrote:
> > > Sorry, I don't grok this.  What has a web application server to do with
> > > asking a DC for user info?
> > 
> > We have one of these that does a lot of DC lookups because it authenticates
> > all users.  It's also in a much faster network, so I can check there what
> > the lookup rate could be reasonably expected to be.
> > 
> > > Erm... how often are you calling id, usually?
> > 
> > I'm currently doing this in the login process to figure out whether the
> > prompt should show "root" powers.  I'll have to figure out something else to
> > do instead.
> 
> No, you don't.  I'm actually doing the same.  Let's keep up with this
> and try to make Cygwin faster in the first place.

I just created 400 groups in AD, and added myself as member.  An `id' on
a 32 bit Windows 7 domain member machine in my tiny network consisting
only of a handful of Windows VMs and with me as the only real user takes
about 3.6 secs with the latest code from CVS, using a non-optimzed
Cygwin DLL.

> > > Also, we're in the early
> > > stages of testing this change.  The idea is not that you just switch,
> > > the idea is that we *test* this and I get enough feedback to be able to
> > > ease the biggest pains.
> > 
> > Understood.  Until now I had to generate passwd and group files and I was
> > hoping that the need for doing that would go away (I'd also need to talk to
> > our AD folks so they start populating the correct fields).
> 
> Yup.  Feedback from AD admins is greatly appreciated.
> 
> > > Other than that, I just had an in-shower inspiration how to speed up
> > > `id' specificially.  The getgroups(2) call is in the center of this and
> > > I could probably speed up the stuiff a lot by opening the LDAP
> > > connection in getgroups only once. 
> > 
> > Thursday?  :-)
> 
> Hmm, probably.

With this patch applied, the aforementioned `id' now takes about 1.9
secs, in an otherwise identical scenario.

> > > Also, more radically, if we drop the functionality to define another
> > > group name for a group, we could drop the requirement to open an LDAP
> > > connection to fetch group information to the DC entirely(*).  This would
> > > only affect domain groups, local groups could still have different
> > > names.  But I'm already wondering for a couple of days if having a
> > > Cygwin group name different from the Windows group name is really
> > > necessary at all.  I added this years ago for fun, but there's no
> > > serious reason I can think of which would require to keep up with this.
> > > 
> > > (*) Assuming the group info is cached in the local LSA, which is
> > >     pretty likely for the groups of the current user.
> > 
> > That would also work for me (I don't think I've ever used that feature
> > consciously).

With this patch applied as well, `id' now takes constantly 0.4 secs.

Note that this speedup is only possible when fetching lots of group
account information.  For user accounts we still need the info from AD,
but apart from the getpwent functionality, which can be restricted via
the db_enum setting in /etc/nsswitch.conf, there's not very often a good
reason to fetch information for hundreds of user accounts.

Anyway, if I send you the link to two DLLs with these patches, would you
mind to test their speed in your environment?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> I just created 400 groups in AD, and added myself as member.  An `id' on
> a 32 bit Windows 7 domain member machine in my tiny network consisting
> only of a handful of Windows VMs and with me as the only real user takes
> about 3.6 secs with the latest code from CVS, using a non-optimzed
> Cygwin DLL.

OK, so you do less than 200 lu/s even in that favorable case.  Our DC is
hit by some four figure number of clients I suppose.  I've asked my
colleague to check the lookup rate on our test web server and he also
gets around 30 lu/s with caching disabled, just like I did via Cygwin.
So the network speed isn't the limiting factor.

> With this patch applied, the aforementioned `id' now takes about 1.9
> secs, in an otherwise identical scenario.

Sounds interesting.

> With this patch applied as well, `id' now takes constantly 0.4 secs.

Gets even better… :-)

> Note that this speedup is only possible when fetching lots of group
> account information.  For user accounts we still need the info from AD,
> but apart from the getpwent functionality, which can be restricted via
> the db_enum setting in /etc/nsswitch.conf, there's not very often a good
> reason to fetch information for hundreds of user accounts.
>
> Anyway, if I send you the link to two DLLs with these patches, would you
> mind to test their speed in your environment?

Bring it on.  I'd need 32bit DLL since I have to keep the 64bit Cygwin
running on my machine.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
[...]
> With this patch applied, the aforementioned `id' now takes about 1.9
> secs, in an otherwise identical scenario.
[...]
> With this patch applied as well, `id' now takes constantly 0.4 secs.
> 
> Note that this speedup is only possible when fetching lots of group
> account information.  For user accounts we still need the info from AD,
> but apart from the getpwent functionality, which can be restricted via
> the db_enum setting in /etc/nsswitch.conf, there's not very often a good
> reason to fetch information for hundreds of user accounts.
> 
> Anyway, if I send you the link to two DLLs with these patches, would you
> mind to test their speed in your environment?

Thanks.  I'm basically seeing the same with your DLL snapshots, plus the
login times (each with an invocation of "id") are following suit.  It also
seems that our DC is a bit less loaded today.  Here are the results of doing
this:

foreach p ( `seq 1 10` )
 time id >/dev/null
end


"stock"

0.561u 0.514s 0:08.93 11.9%     0+0k 0+0io 7690pf+0w
0.499u 0.499s 0:09.04 10.8%     0+0k 0+0io 7717pf+0w
0.592u 0.686s 0:09.03 14.0%     0+0k 0+0io 7690pf+0w
0.639u 0.576s 0:09.18 13.0%     0+0k 0+0io 7698pf+0w
0.608u 0.390s 0:09.11 10.8%     0+0k 0+0io 7691pf+0w
0.577u 0.514s 0:09.03 11.9%     0+0k 0+0io 7048pf+0w
0.499u 0.468s 0:08.91 10.6%     0+0k 0+0io 7724pf+0w
0.498u 0.561s 0:08.84 11.8%     0+0k 0+0io 7744pf+0w
0.561u 0.405s 0:08.73 10.9%     0+0k 0+0io 6664pf+0w
0.452u 0.545s 0:08.96 11.0%     0+0k 0+0io 6968pf+0w


"getgroups"

0.249u 0.202s 0:03.33 13.2%     0+0k 0+0io 3506pf+0w
0.234u 0.296s 0:03.33 15.6%     0+0k 0+0io 3489pf+0w
0.171u 0.296s 0:03.26 14.1%     0+0k 0+0io 3493pf+0w
0.171u 0.265s 0:03.25 13.2%     0+0k 0+0io 3486pf+0w
0.156u 0.358s 0:03.33 15.0%     0+0k 0+0io 3486pf+0w
0.171u 0.265s 0:03.29 13.0%     0+0k 0+0io 3487pf+0w
0.311u 0.373s 0:03.46 19.6%     0+0k 0+0io 3506pf+0w
0.171u 0.217s 0:03.27 11.6%     0+0k 0+0io 3488pf+0w
0.234u 0.202s 0:03.36 12.7%     0+0k 0+0io 3491pf+0w
0.218u 0.202s 0:03.42 11.9%     0+0k 0+0io 3487pf+0w


"noldap"

0.249u 0.202s 0:03.33 13.2%     0+0k 0+0io 3506pf+0w
0.234u 0.296s 0:03.33 15.6%     0+0k 0+0io 3489pf+0w
0.171u 0.296s 0:03.26 14.1%     0+0k 0+0io 3493pf+0w
0.171u 0.265s 0:03.25 13.2%     0+0k 0+0io 3486pf+0w
0.156u 0.358s 0:03.33 15.0%     0+0k 0+0io 3486pf+0w
0.171u 0.265s 0:03.29 13.0%     0+0k 0+0io 3487pf+0w
0.311u 0.373s 0:03.46 19.6%     0+0k 0+0io 3506pf+0w
0.171u 0.217s 0:03.27 11.6%     0+0k 0+0io 3488pf+0w
0.234u 0.202s 0:03.36 12.7%     0+0k 0+0io 3491pf+0w
0.218u 0.202s 0:03.42 11.9%     0+0k 0+0io 3487pf+0w


Best regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Achim Gratz writes:
> "noldap"

Sorry, copied the "getgroups" data again, this is the data for "noldap" of
course:

0.171u 0.015s 0:01.03 17.4%     0+0k 0+0io 3298pf+0w
0.093u 0.140s 0:00.97 23.7%     0+0k 0+0io 3292pf+0w
0.046u 0.093s 0:00.97 13.4%     0+0k 0+0io 3285pf+0w
0.062u 0.140s 0:00.97 20.6%     0+0k 0+0io 3283pf+0w
0.031u 0.124s 0:00.97 15.4%     0+0k 0+0io 3285pf+0w
0.093u 0.077s 0:00.99 16.1%     0+0k 0+0io 3291pf+0w
0.093u 0.077s 0:00.96 16.6%     0+0k 0+0io 3298pf+0w
0.015u 0.077s 0:00.97 8.2%      0+0k 0+0io 3291pf+0w
0.093u 0.108s 0:00.99 19.1%     0+0k 0+0io 3300pf+0w
0.062u 0.109s 0:00.97 16.4%     0+0k 0+0io 3294pf+0w


Regards,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1363 bytes --]

Hi Achim,


thanks for testing.

On Feb 27 09:07, Achim Gratz wrote:
> Achim Gratz writes:
> > "noldap"
> 
> Sorry, copied the "getgroups" data again, this is the data for "noldap" of
> course:
> 
> 0.171u 0.015s 0:01.03 17.4%     0+0k 0+0io 3298pf+0w
> 0.093u 0.140s 0:00.97 23.7%     0+0k 0+0io 3292pf+0w
> 0.046u 0.093s 0:00.97 13.4%     0+0k 0+0io 3285pf+0w
> 0.062u 0.140s 0:00.97 20.6%     0+0k 0+0io 3283pf+0w
> 0.031u 0.124s 0:00.97 15.4%     0+0k 0+0io 3285pf+0w
> 0.093u 0.077s 0:00.99 16.1%     0+0k 0+0io 3291pf+0w
> 0.093u 0.077s 0:00.96 16.6%     0+0k 0+0io 3298pf+0w
> 0.015u 0.077s 0:00.97 8.2%      0+0k 0+0io 3291pf+0w
> 0.093u 0.108s 0:00.99 19.1%     0+0k 0+0io 3300pf+0w
> 0.062u 0.109s 0:00.97 16.4%     0+0k 0+0io 3294pf+0w

1 second?  That sounds still a bit slow.  Considering that I'm now
member of 414 groups, and you are member of 440 groups, the extra number
of groups cannot account for that.  This sounds surprisingly as if the
names of some of your groups are not cached on your machine.  Or
something.  Or is this a rather slow machine?!?

Still, it seems like the right thing to do to drop the group name
configuration stuff entirely.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> 1 second?  That sounds still a bit slow.  Considering that I'm now
> member of 414 groups, and you are member of 440 groups, the extra number
> of groups cannot account for that.
>
>  This sounds surprisingly as if the
> names of some of your groups are not cached on your machine.  Or
> something.  Or is this a rather slow machine?!?
 
It's not a slow machine by any means, but it certainly gets its fair share
of security policies, so it may have something to do with that.  I don't know.

> Still, it seems like the right thing to do to drop the group name
> configuration stuff entirely.

Yes (unless you'd want to make it configurable like the getpwent stuff).


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Feb 27 12:54, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > 1 second?  That sounds still a bit slow.  Considering that I'm now
> > member of 414 groups, and you are member of 440 groups, the extra number
> > of groups cannot account for that.
> >
> >  This sounds surprisingly as if the
> > names of some of your groups are not cached on your machine.  Or
> > something.  Or is this a rather slow machine?!?
>  
> It's not a slow machine by any means, but it certainly gets its fair share
> of security policies, so it may have something to do with that.  I don't know.
> 
> > Still, it seems like the right thing to do to drop the group name
> > configuration stuff entirely.
> 
> Yes (unless you'd want to make it configurable like the getpwent stuff).

Nah, not really.  As I said, I'm questioning some of the old functionality
anyway, and the less we have to ask AD the better for us.

I applied my patch which removes this group name change facility from AD
and uploaded a new snapshot to http://cygwin.com/snapshots/.

While we're at it, I just had this weird idea.

What if, as soon as the first Cygwin process in a process tree starts,
this process not only caches the primary group info, but also caches all
supplementary groups from the user's token?  This would slow down
startup of the first process slightly, but it would speed up any
subsequent request for group information of a group in the user's token.
An `id' call would be almost instant, and `ls' calls would probably be
faster as well.

As always, there's a trade-off: Users running cygwin processes from CMD
a lot would encounter a slowdown.

What do you think, guys?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-02-13 15:38 GMT+01:00 Corinna Vinschen:
> This is a pretty intrusive change, in need of some serious testing, so
> I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> http://cygwin.com/snapshots/ contains the changes, including the latest
> bugfix.

I noticed another thing. Not sure if I would call it a problem, just
an observation or if it is a feature.

As said before, my laptop is part of a domain and I have moved the
local passwd and group out of the way.

When I am not connected to the domain (on the road) I get this error
every time I start mintty: "id: kan geen naam vinden bij groeps-ID
1049089"
I have an export LANG=en_US.utf8 in my .bashrc so since the text is in
Dutch this happens before the .bashrc is executed. It basically
translates to "id: unable to find name for group ID 1049089".

When I start my vpn and re-open mintty the error doesn't show up
anymore, even if I disconnect the vpn again. After rebooting the error
message occurs again.

The reason for the message seems obvious. The process can't lookup the
information. But it can be confusing to users to see an error at
startup. Is it error useful to the user? Can I suppress this message?

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Frank Fesevur!

>> This is a pretty intrusive change, in need of some serious testing, so
>> I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
>> http://cygwin.com/snapshots/ contains the changes, including the latest
>> bugfix.

> I noticed another thing. Not sure if I would call it a problem, just
> an observation or if it is a feature.

> As said before, my laptop is part of a domain and I have moved the
> local passwd and group out of the way.

> When I am not connected to the domain (on the road) I get this error
> every time I start mintty: "id: kan geen naam vinden bij groeps-ID
> 1049089"
> I have an export LANG=en_US.utf8 in my .bashrc so since the text is in
> Dutch this happens before the .bashrc is executed. It basically
> translates to "id: unable to find name for group ID 1049089".

> When I start my vpn and re-open mintty the error doesn't show up
> anymore, even if I disconnect the vpn again. After rebooting the error
> message occurs again.

> The reason for the message seems obvious. The process can't lookup the
> information. But it can be confusing to users to see an error at
> startup. Is it error useful to the user? Can I suppress this message?

I suppose, with latest Corinna's modifications to the ways `id' lookup the
data, this message would go away naturally.
I don't know, if the changes are available in a public snapshot, though.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 27.02.2014, <23:13>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2139 bytes --]

On Feb 27 20:06, Frank Fesevur wrote:
> 2014-02-13 15:38 GMT+01:00 Corinna Vinschen:
> > This is a pretty intrusive change, in need of some serious testing, so
> > I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> > http://cygwin.com/snapshots/ contains the changes, including the latest
> > bugfix.
> 
> I noticed another thing. Not sure if I would call it a problem, just
> an observation or if it is a feature.
> 
> As said before, my laptop is part of a domain and I have moved the
> local passwd and group out of the way.
> 
> When I am not connected to the domain (on the road) I get this error
> every time I start mintty: "id: kan geen naam vinden bij groeps-ID
> 1049089"

My dutch is a bit rusty but this text is obvious.  And unexpected.
What was supposed to happen is this:

- LookupAccountSid returns with error
  ==> Check if the local machine knows the domain name of the SID
  ==> Yes: Create a fake group name "DOMAIN+RID"
      No:  Create a fake group name "Unknown_Group+GID"

> I have an export LANG=en_US.utf8 in my .bashrc so since the text is in
> Dutch this happens before the .bashrc is executed. It basically
> translates to "id: unable to find name for group ID 1049089".
> When I start my vpn and re-open mintty the error doesn't show up
> anymore, even if I disconnect the vpn again.

At this point all the required information isd apparently cashed
by the local LSA.

> After rebooting the error message occurs again.

Cache is empty again.

> The reason for the message seems obvious. The process can't lookup the
> information. But it can be confusing to users to see an error at
> startup. Is it error useful to the user? Can I suppress this message?

No, because it comes from `id', not from Cygwin, and as I wrote
above, it's in fact unexpected and unwanted.  Could you try to
strace this scenario?  Maybe I can figure out why the account name
faking didn't work.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

On Feb 27 23:15, Andrey Repin wrote:
> Greetings, Frank Fesevur!
> 
> >> This is a pretty intrusive change, in need of some serious testing, so
> >> I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> >> http://cygwin.com/snapshots/ contains the changes, including the latest
> >> bugfix.
> 
> > I noticed another thing. Not sure if I would call it a problem, just
> > an observation or if it is a feature.
> 
> > As said before, my laptop is part of a domain and I have moved the
> > local passwd and group out of the way.
> 
> > When I am not connected to the domain (on the road) I get this error
> > every time I start mintty: "id: kan geen naam vinden bij groeps-ID
> > 1049089"
> > I have an export LANG=en_US.utf8 in my .bashrc so since the text is in
> > Dutch this happens before the .bashrc is executed. It basically
> > translates to "id: unable to find name for group ID 1049089".
> 
> > When I start my vpn and re-open mintty the error doesn't show up
> > anymore, even if I disconnect the vpn again. After rebooting the error
> > message occurs again.
> 
> > The reason for the message seems obvious. The process can't lookup the
> > information. But it can be confusing to users to see an error at
> > startup. Is it error useful to the user? Can I suppress this message?
> 
> I suppose, with latest Corinna's modifications to the ways `id' lookup the
> data, this message would go away naturally.
> I don't know, if the changes are available in a public snapshot, though.

http://cygwin.com/ml/cygwin/2014-02/msg00707.html ?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> > 1 second?  That sounds still a bit slow.  Considering that I'm now
>> > member of 414 groups, and you are member of 440 groups, the extra number
>> > of groups cannot account for that.
>> >
>> >  This sounds surprisingly as if the
>> > names of some of your groups are not cached on your machine.  Or
>> > something.  Or is this a rather slow machine?!?
>>  
>> It's not a slow machine by any means, but it certainly gets its fair share
>> of security policies, so it may have something to do with that.  I don't know.
>> 
>> > Still, it seems like the right thing to do to drop the group name
>> > configuration stuff entirely.
>> 
>> Yes (unless you'd want to make it configurable like the getpwent stuff).

> Nah, not really.  As I said, I'm questioning some of the old functionality
> anyway, and the less we have to ask AD the better for us.

> I applied my patch which removes this group name change facility from AD
> and uploaded a new snapshot to http://cygwin.com/snapshots/.

> While we're at it, I just had this weird idea.

> What if, as soon as the first Cygwin process in a process tree starts,
> this process not only caches the primary group info, but also caches all
> supplementary groups from the user's token?  This would slow down
> startup of the first process slightly, but it would speed up any
> subsequent request for group information of a group in the user's token.
> An `id' call would be almost instant, and `ls' calls would probably be
> faster as well.

Umhm. I'd have to see, how slightly. Because my primary use of Cygwin tools
does not expect to have a parent Cygwin process. (I.e. diff'ing between
file manager panels.)

> As always, there's a trade-off: Users running cygwin processes from CMD
> a lot would encounter a slowdown.

> What do you think, guys?

Needs some tests. I would certainly appreciate faster initial startups, but if
the delay is manageable, I can cope with more long-term friendly approach.
I've got entangled in current work and lost track of snapshots for now.
If there's some DEF's in library, that I can toggle and rebuild it with certain
features enabled and disable for local test, I'd be glad to have a round at it
over weekend.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 28.02.2014, <03:06>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> >> This is a pretty intrusive change, in need of some serious testing, so
>> >> I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
>> >> http://cygwin.com/snapshots/ contains the changes, including the latest
>> >> bugfix.
>> 
>> > I noticed another thing. Not sure if I would call it a problem, just
>> > an observation or if it is a feature.
>> 
>> > As said before, my laptop is part of a domain and I have moved the
>> > local passwd and group out of the way.
>> 
>> > When I am not connected to the domain (on the road) I get this error
>> > every time I start mintty: "id: kan geen naam vinden bij groeps-ID
>> > 1049089"
>> > I have an export LANG=en_US.utf8 in my .bashrc so since the text is in
>> > Dutch this happens before the .bashrc is executed. It basically
>> > translates to "id: unable to find name for group ID 1049089".
>> 
>> > When I start my vpn and re-open mintty the error doesn't show up
>> > anymore, even if I disconnect the vpn again. After rebooting the error
>> > message occurs again.
>> 
>> > The reason for the message seems obvious. The process can't lookup the
>> > information. But it can be confusing to users to see an error at
>> > startup. Is it error useful to the user? Can I suppress this message?
>> 
>> I suppose, with latest Corinna's modifications to the ways `id' lookup the
>> data, this message would go away naturally.
>> I don't know, if the changes are available in a public snapshot, though.

> http://cygwin.com/ml/cygwin/2014-02/msg00707.html ?

Nearly missed it. This was a bit messy thursday for me.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 28.02.2014, <03:06>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

On Feb 28 03:12, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> > While we're at it, I just had this weird idea.
> 
> > What if, as soon as the first Cygwin process in a process tree starts,
> > this process not only caches the primary group info, but also caches all
> > supplementary groups from the user's token?  This would slow down
> > startup of the first process slightly, but it would speed up any
> > subsequent request for group information of a group in the user's token.
> > An `id' call would be almost instant, and `ls' calls would probably be
> > faster as well.
> 
> Umhm. I'd have to see, how slightly. Because my primary use of Cygwin tools
> does not expect to have a parent Cygwin process. (I.e. diff'ing between
> file manager panels.)
> 
> > As always, there's a trade-off: Users running cygwin processes from CMD
> > a lot would encounter a slowdown.
> 
> > What do you think, guys?
> 
> Needs some tests. I would certainly appreciate faster initial startups, but if
> the delay is manageable, I can cope with more long-term friendly approach.
> I've got entangled in current work and lost track of snapshots for now.
> If there's some DEF's in library, that I can toggle and rebuild it with certain
> features enabled and disable for local test, I'd be glad to have a round at it
> over weekend.

I've just uploaded a new snapshot to http://cygwin.com/snapshots/

It allows you to do the following in /etc/nsswitch.conf:

  db_cache: no

    No caching of passwd and group data at all

  db_cache: yes

      Caching of passwd and group data, no initial caching of the user's
      supplementary groups from the user token.

  db_cache: full

      This is the default now.

      Caching of passwd and group data *plus* initial caching of all groups
      from the user token.

Please try it out.  Average timings from your environment (process
startup, and calling `id') would be nice.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2200 bytes --]

On Feb 27 20:38, Corinna Vinschen wrote:
> On Feb 27 20:06, Frank Fesevur wrote:
> > 2014-02-13 15:38 GMT+01:00 Corinna Vinschen:
> > > This is a pretty intrusive change, in need of some serious testing, so
> > > I'd like to ask for volunteers.  The latest 2014-02-13 snapshot from
> > > http://cygwin.com/snapshots/ contains the changes, including the latest
> > > bugfix.
> > 
> > I noticed another thing. Not sure if I would call it a problem, just
> > an observation or if it is a feature.
> > 
> > As said before, my laptop is part of a domain and I have moved the
> > local passwd and group out of the way.
> > 
> > When I am not connected to the domain (on the road) I get this error
> > every time I start mintty: "id: kan geen naam vinden bij groeps-ID
> > 1049089"
> 
> My dutch is a bit rusty but this text is obvious.  And unexpected.
> What was supposed to happen is this:
> 
> - LookupAccountSid returns with error
>   ==> Check if the local machine knows the domain name of the SID
>   ==> Yes: Create a fake group name "DOMAIN+RID"
>       No:  Create a fake group name "Unknown_Group+GID"
> [...]
> > The reason for the message seems obvious. The process can't lookup the
> > information. But it can be confusing to users to see an error at
> > startup. Is it error useful to the user? Can I suppress this message?
> 
> No, because it comes from `id', not from Cygwin, and as I wrote
> above, it's in fact unexpected and unwanted.  Could you try to
> strace this scenario?  Maybe I can figure out why the account name
> faking didn't work.

Never mind that, I could easily reproduce the issue.  Please try the
latest snapshot from http://cygwin.com/snapshots/.  It's supposed to fix
this issue.  It also introduces another new behaviour as outlined in

  http://cygwin.com/ml/cygwin/2014-02/msg00735.html

Note that "db_cache: full" is the default for now so the new feature
gets a bit more testing.  The former default would have been equivalent
to "db_cache: yes".


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> 1 second?  That sounds still a bit slow.

It appears that that there are multiple DC involved, either via
delegation or redirection (as I've managed to get some partial group
resolutions where groups from a particular domain were absent).  So all
this slowness probably has to do with roundtrip times.  Based on that
hypothesis I've done the same test again via DSL/VPN and got this:

1:49 stock-cvs
1:15 getgroups
0:13 noldap

The times don't change all that much whether I've clogged the DSL
connection or not, so the size of the response doesn't seem to be a
major factor here.

> Still, it seems like the right thing to do to drop the group name
> configuration stuff entirely.

Indeed.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

On Feb 28 16:45, Achim Gratz wrote:
> Corinna Vinschen writes:
> > 1 second?  That sounds still a bit slow.
> 
> It appears that that there are multiple DC involved, either via
> delegation or redirection (as I've managed to get some partial group
> resolutions where groups from a particular domain were absent).  So all
> this slowness probably has to do with roundtrip times.  Based on that
> hypothesis I've done the same test again via DSL/VPN and got this:
> 
> 1:49 stock-cvs
> 1:15 getgroups
> 0:13 noldap
> 
> The times don't change all that much whether I've clogged the DSL
> connection or not, so the size of the response doesn't seem to be a
> major factor here.

I made some tests myself today, while debugging Frank's problem.  If I
had no network connection to my DC, the group names couldn't be
resolved.  This is using the stock LookupAccountSid function from
advapi32.dll so that means, the names of the groups are not cached
anywhere in LSA, not even the names of the current user's groups.

Given that, it was pretty surprising that the noldap code is so fast
compared to the getgroups version.  The LDAP connection is opened once
only, so the ldap request should be fast.  Even with a call to
LookupAccountSid and an additional call to ldap_search_st, I would
understand if the getgroups version takes twice as much as the noldap
version, but *8* times?

After some more testing it seems LookupAccountSid is asking the Global
Catalog (GC).  If I switch my LDAP queries to the GC port 3286, it's
getting a *lot* faster.  In fact, it's suddenly not 8 times slower,
but only two times, as expected.

Unfortunately that doesn't help us at all, because the POSIX attributes
are not duplicated to the GC by default, and I guess it's not exactly
helpful to ask administrators to duplicate the POSIX attributes to the
GC :(


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> It allows you to do the following in /etc/nsswitch.conf:
>
>   db_cache: no

Using the 15:31 snapshot DLL again via VPN, id dumps core after about
2:30 minutes.

>     No caching of passwd and group data at all
>
>   db_cache: yes

Startup of a shell in mintty takes about 20 seconds, each id around 10...13s.

>       Caching of passwd and group data, no initial caching of the user's
>       supplementary groups from the user token.
>
>   db_cache: full

Startup in around 10seconds, each id takes a third of a second.

>       This is the default now.

A good choice, I'd say.  :-)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

On Feb 28 20:49, Achim Gratz wrote:
> Corinna Vinschen writes:
> > It allows you to do the following in /etc/nsswitch.conf:
> >
> >   db_cache: no
> 
> Using the 15:31 snapshot DLL again via VPN, id dumps core after about
> 2:30 minutes.

How?  Details?  Stackdump?  It works for me(TM).  The timing only shows
that it's not the right thing for you, or that in the long run the
non-caching option should just go away.  For the time being, though, I
need *details*.

> >     No caching of passwd and group data at all
> >
> >   db_cache: yes
> 
> Startup of a shell in mintty takes about 20 seconds, each id around 10...13s.
> 
> >       Caching of passwd and group data, no initial caching of the user's
> >       supplementary groups from the user token.
> >
> >   db_cache: full
> 
> Startup in around 10seconds, each id takes a third of a second.

Hang on.  Are you telling me this snapshot takes longer than the
snapshot from yesterday?  And you're telling me that normal caching, as
was the default all the time, is slower in startup than with the
additional caching of groups at startup?  Is your startup procedure
calling `id' twice by any chance?

The problem with your numbers is that I have nothing really to compare
it with.  It would be helpful to know what you're doing, the essential
parts of your startup for instance.

Please, please, please, if something doesn't work as expected, please
add meat to your reports.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-02-28 18:14 GMT+01:00 Corinna Vinschen:
> I made some tests myself today, while debugging Frank's problem.  If I

I updated to the snapshot of today (28th) and now the message is not
shown anymore.
Thanks for the fix.

I have yet another observation. When I am not connected with VPN it
takes a while (don't know how to time exactly, around 3 seconds) after
I double click the shortcut for mintty to show up. When connect it
shows up almost instantly. Tested 24th, 27th and 28th snapshots and
see the problem with all of them.

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Feb 28 21:53, Frank Fesevur wrote:
> 2014-02-28 18:14 GMT+01:00 Corinna Vinschen:
> > I made some tests myself today, while debugging Frank's problem.  If I
> 
> I updated to the snapshot of today (28th) and now the message is not
> shown anymore.
> Thanks for the fix.

Thanks for the report.

> I have yet another observation. When I am not connected with VPN it
> takes a while (don't know how to time exactly, around 3 seconds) after
> I double click the shortcut for mintty to show up. When connect it
> shows up almost instantly. Tested 24th, 27th and 28th snapshots and
> see the problem with all of them.

That's not really a problem but a case of "it is as it is".  To get the
user and group info, Cygwin has to contact the DC and/or GC and then
runs into a timeout.  Right now, the LDAP timeout is set to 3 seconds.
I don't know yet if it's such a bright idea to lower this timeout value.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

On Feb 28 22:08, Corinna Vinschen wrote:
> On Feb 28 21:53, Frank Fesevur wrote:
> > 2014-02-28 18:14 GMT+01:00 Corinna Vinschen:
> > > I made some tests myself today, while debugging Frank's problem.  If I
> > 
> > I updated to the snapshot of today (28th) and now the message is not
> > shown anymore.
> > Thanks for the fix.
> 
> Thanks for the report.
> 
> > I have yet another observation. When I am not connected with VPN it
> > takes a while (don't know how to time exactly, around 3 seconds) after
> > I double click the shortcut for mintty to show up. When connect it
> > shows up almost instantly. Tested 24th, 27th and 28th snapshots and
> > see the problem with all of them.
> 
> That's not really a problem but a case of "it is as it is".  To get the
> user and group info, Cygwin has to contact the DC and/or GC and then
> runs into a timeout.  Right now, the LDAP timeout is set to 3 seconds.
> I don't know yet if it's such a bright idea to lower this timeout value.

Btw., if the timeout is unbearable, it might be an interesting idea
to switch on reading passwd and group info from the files when the
VPN is not available.

The last snapshots contain a new version of the mkpasswd and mkgroup
tools which produce passwd and group entries with the same values
(especially uid and gid values!) as the "db" setting in Cygwin does.

The idea is that files and db don't contradict each other by default,
rather they provide the same values.  Therefore it might be a nice
addition to use the files in offline mode.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

> I've just uploaded a new snapshot to http://cygwin.com/snapshots/

I've sent a reply with testing details, but it may have been caught by SPAM
deflection for "excess use of foreign language".
Please disregard this notification, if a message will finally come through.

> Please try it out.  Average timings from your environment (process
> startup, and calling `id') would be nice.

Oh, and I thought of a way to test only invocation times.
I ran /bin/true alot...

db_cache: no
Timer 2  Elapsed: 0:00:00,75
Stable (+-0.2 regardless of current credentials.)

db_cache: yes
Timer 2  Elapsed: 0:00:00,85 (current user)
Timer 2  Elapsed: 0:00:00,75 (different user)
Unstable results. (I.e. vary by ~.10 in both cases.)

Default caching settings.
Timer 2  Elapsed: 0:00:01,15
Unstable results. (I.e. vary by ~.15 or more in all cases.)


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 01.03.2014, <01:51>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Andrey Repin!

> db_cache: no
> Timer 2  Elapsed: 0:00:00,75
> Stable (+-0.02 regardless of current credentials.)

My bad.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 01.03.2014, <02:27>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> How?  Details?  Stackdump?  It works for me(TM).  The timing only shows
> that it's not the right thing for you, or that in the long run the
> non-caching option should just go away.  For the time being, though, I
> need *details*.

Sorry, that's the best I can do at the moment.

> Hang on.  Are you telling me this snapshot takes longer than the
> snapshot from yesterday?  And you're telling me that normal caching, as
> was the default all the time, is slower in startup than with the
> additional caching of groups at startup?  Is your startup procedure
> calling `id' twice by any chance?

I've been using the network via VPN, and measured the numbers with the
previous DLL(s) before testing the snapshot, results are in a different
reply.

> The problem with your numbers is that I have nothing really to compare
> it with.  It would be helpful to know what you're doing, the essential
> parts of your startup for instance.

I'm on my way to catch a plane, sorry this will be in about a week.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-02-28 22:08 GMT+01:00 Corinna Vinschen:
> That's not really a problem but a case of "it is as it is".  To get the
> user and group info, Cygwin has to contact the DC and/or GC and then
> runs into a timeout.  Right now, the LDAP timeout is set to 3 seconds.
> I don't know yet if it's such a bright idea to lower this timeout value.

I understand why it happens. And three seconds are not the problem.

It is just a bit confusing to activate the shortcut and for 3 seconds
nothing seems to happen. First time I double-clicked again because I
though it wasn't working at all. Obviously a bit later two terminal
showed up. If mintty would should up empty and "wait" those three
seconds before something happens it is more clear it just takes some
startup time. But maybe that is more a mintty problem.

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1841 bytes --]

On Mar  2 14:20, Frank Fesevur wrote:
> 2014-02-28 22:08 GMT+01:00 Corinna Vinschen:
> > That's not really a problem but a case of "it is as it is".  To get the
> > user and group info, Cygwin has to contact the DC and/or GC and then
> > runs into a timeout.  Right now, the LDAP timeout is set to 3 seconds.
> > I don't know yet if it's such a bright idea to lower this timeout value.
> 
> I understand why it happens. And three seconds are not the problem.
> 
> It is just a bit confusing to activate the shortcut and for 3 seconds
> nothing seems to happen. First time I double-clicked again because I
> though it wasn't working at all. Obviously a bit later two terminal
> showed up. If mintty would should up empty and "wait" those three
> seconds before something happens it is more clear it just takes some
> startup time. But maybe that is more a mintty problem.

No, this is not mintty's fault.  You have to understand that the
request for information from the DC is called before the actual
application had any chance to initialize,  This happens before
the application has, in fact, been called from the Cygwin DLL.

Therefore, if the three seconds are a burden, there are three choices
going forward:

- We reduce the timeout to 1 second in general, which could be a problem
  when utilizing a slow VPN conncetion to the DC,

- We allow the user to specify a timeout value and set the default to
  1 second.

- We stick to the 3 seconds and users should switch to files (/etc/passwd
  /etc/group) when being offline from their DC.

I'm really not sure what the best way is, I guess we as a community have
to dice this out somehow.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Mar  1 02:08, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> > I've just uploaded a new snapshot to http://cygwin.com/snapshots/
> 
> I've sent a reply with testing details, but it may have been caught by SPAM
> deflection for "excess use of foreign language".
> Please disregard this notification, if a message will finally come through.
> 
> > Please try it out.  Average timings from your environment (process
> > startup, and calling `id') would be nice.
> 
> Oh, and I thought of a way to test only invocation times.
> I ran /bin/true alot...
> 
> db_cache: no
> Timer 2  Elapsed: 0:00:00,75
> Stable (+-0.2 regardless of current credentials.)
> 
> db_cache: yes
> Timer 2  Elapsed: 0:00:00,85 (current user)
> Timer 2  Elapsed: 0:00:00,75 (different user)
> Unstable results. (I.e. vary by ~.10 in both cases.)
> 
> Default caching settings.
> Timer 2  Elapsed: 0:00:01,15
> Unstable results. (I.e. vary by ~.15 or more in all cases.)

So the process start takes 0.3 secs extra for you.  Is that feasible?
Would you rather keep the new "db_cache" setting as an option?  What
do you think is the best default, objectively spoken?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Frank Fesevur]

2014-03-03 10:21 GMT+01:00 Corinna Vinschen:
> No, this is not mintty's fault.  You have to understand that the
> request for information from the DC is called before the actual
> application had any chance to initialize,  This happens before
> the application has, in fact, been called from the Cygwin DLL.

Understood.

> Therefore, if the three seconds are a burden, there are three choices
> going forward:
>
> - We reduce the timeout to 1 second in general, which could be a problem
>   when utilizing a slow VPN conncetion to the DC,
>
> - We allow the user to specify a timeout value and set the default to
>   1 second.
>
> - We stick to the 3 seconds and users should switch to files (/etc/passwd
>   /etc/group) when being offline from their DC.
>
> I'm really not sure what the best way is, I guess we as a community have
> to dice this out somehow.

I think the second one makes most sense. Maybe even keep the default
timeout on 3 seconds. When the user can change the value, it is up to
them to play with it and figure out what works best for them. I
personally would try to set it to 1 second, since I almost never have
a slow VPN.

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Henry S. Thompson]

Frank Fesevur writes:

> 2014-03-03 10:21 GMT+01:00 Corinna Vinschen:
>> - We allow the user to specify a timeout value and set the default to
>>   1 second.

> I think the [above] makes most sense. Maybe even keep the default
> timeout on 3 seconds. When the user can change the value, it is up to
> them to play with it and figure out what works best for them. I
> personally would try to set it to 1 second, since I almost never have
> a slow VPN.

As someone who _never_ cares about group membership at all (as far as
I know), and runs on a network where no domain server will ever
respond, can I request that the default setup be carefully considered,
if the proposal above would mean me always waiting 1 second, to say
nothing of 3 seconds, to launch anything.

Should we be considering some form of configuration tool, similar to
ssh-host-config, which will set appropriate configuration values for a
range of common situations, including the "no domain controller" one?

Not sure I can volunteer, as I haven't been following the details up
to this point because the functionality is not one I use.

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

On Mar  3 12:29, Henry S. Thompson wrote:
> Frank Fesevur writes:
> 
> > 2014-03-03 10:21 GMT+01:00 Corinna Vinschen:
> >> - We allow the user to specify a timeout value and set the default to
> >>   1 second.
> 
> > I think the [above] makes most sense. Maybe even keep the default
> > timeout on 3 seconds. When the user can change the value, it is up to
> > them to play with it and figure out what works best for them. I
> > personally would try to set it to 1 second, since I almost never have
> > a slow VPN.
> 
> As someone who _never_ cares about group membership at all (as far as
> I know), and runs on a network where no domain server will ever
> respond,

If you don't have a domain member machine, you don't have this
problem anyway.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> > I've just uploaded a new snapshot to http://cygwin.com/snapshots/
>> 
>> I've sent a reply with testing details, but it may have been caught by SPAM
>> deflection for "excess use of foreign language".
>> Please disregard this notification, if a message will finally come through.
>> 
>> > Please try it out.  Average timings from your environment (process
>> > startup, and calling `id') would be nice.
>> 
>> Oh, and I thought of a way to test only invocation times.
>> I ran /bin/true alot...
>> 
>> db_cache: no
>> Timer 2  Elapsed: 0:00:00,75
>> Stable (+-0.2 regardless of current credentials.)
>> 
>> db_cache: yes
>> Timer 2  Elapsed: 0:00:00,85 (current user)
>> Timer 2  Elapsed: 0:00:00,75 (different user)
>> Unstable results. (I.e. vary by ~.10 in both cases.)
>> 
>> Default caching settings.
>> Timer 2  Elapsed: 0:00:01,15
>> Unstable results. (I.e. vary by ~.15 or more in all cases.)

> So the process start takes 0.3 secs extra for you.  Is that feasible?
> Would you rather keep the new "db_cache" setting as an option?  What
> do you think is the best default, objectively spoken?

I could never notice it. And it's not 0.3 sec, it's closer to 0.005sec (the
cycle was run about sixty times, so, divide approptely)
I'd need to test it in domain environment, but I don't have access to one with
Cygwin installed right now. Or, perhaps, I do?... let me check something.

In the meantime, I see my previous mail was indeed caught in the middle. I'll
repost it with a link to test results.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 03.03.2014, <18:41>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

On Mar  3 18:47, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> > I've just uploaded a new snapshot to http://cygwin.com/snapshots/
> >> 
> >> I've sent a reply with testing details, but it may have been caught by SPAM
> >> deflection for "excess use of foreign language".
> >> Please disregard this notification, if a message will finally come through.
> >> 
> >> > Please try it out.  Average timings from your environment (process
> >> > startup, and calling `id') would be nice.
> >> 
> >> Oh, and I thought of a way to test only invocation times.
> >> I ran /bin/true alot...
> >> 
> >> db_cache: no
> >> Timer 2  Elapsed: 0:00:00,75
> >> Stable (+-0.2 regardless of current credentials.)
> >> 
> >> db_cache: yes
> >> Timer 2  Elapsed: 0:00:00,85 (current user)
> >> Timer 2  Elapsed: 0:00:00,75 (different user)
> >> Unstable results. (I.e. vary by ~.10 in both cases.)
> >> 
> >> Default caching settings.
> >> Timer 2  Elapsed: 0:00:01,15
> >> Unstable results. (I.e. vary by ~.15 or more in all cases.)
> 
> > So the process start takes 0.3 secs extra for you.  Is that feasible?
> > Would you rather keep the new "db_cache" setting as an option?  What
> > do you think is the best default, objectively spoken?
> 
> I could never notice it. And it's not 0.3 sec, it's closer to 0.005sec (the
> cycle was run about sixty times, so, divide approptely)
> I'd need to test it in domain environment, but I don't have access to one with
> Cygwin installed right now. Or, perhaps, I do?... let me check something.

Sure.

> In the meantime, I see my previous mail was indeed caught in the middle. I'll
> repost it with a link to test results.

Nah, never mind.  I'm more interested of your experiences than in
dry numbers :)


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> In the meantime, I see my previous mail was indeed caught in the middle. I'll
>> repost it with a link to test results.

> Nah, never mind.  I'm more interested of your experiences than in
> dry numbers :)

Oh, don't be so quick, lady...
The results were quite interesting.

http://www.rootdir.org/.offload/MissingMail.eml


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 03.03.2014, <19:20>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> In the meantime, I see my previous mail was indeed caught in the middle. I'll
>> repost it with a link to test results.

> Nah, never mind.  I'm more interested of your experiences than in
> dry numbers :)

Oh, don't be so quick, lady...
The results were quite interesting.

http://www.rootdir.org/.offload/MissingMail.eml


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 03.03.2014, <19:20>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

> No, this is not mintty's fault.  You have to understand that the
> request for information from the DC is called before the actual
> application had any chance to initialize,  This happens before
> the application has, in fact, been called from the Cygwin DLL.

> Therefore, if the three seconds are a burden, there are three choices
> going forward:

> - We reduce the timeout to 1 second in general, which could be a problem
>   when utilizing a slow VPN conncetion to the DC,

> - We allow the user to specify a timeout value and set the default to
>   1 second.

> - We stick to the 3 seconds and users should switch to files (/etc/passwd
>   /etc/group) when being offline from their DC.

> I'm really not sure what the best way is, I guess we as a community have
> to dice this out somehow.

I'd say it again, "sane defaults are better, than alot of options". 1 second
to CONNECT to domain server is more than enough time under normal
circumstances (for comparison, default DNS _roundtrip_ timeout is 2 seconds,
that for a whole conversation sequence).
Leave the connect timeout short, and if certain users are unable to cope with
slow connections to their domain controllers, they can use less pre-caching,
or switch to local files to speed up uid/gid resolution.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 03.03.2014, <18:40>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Testers needed: New passwd/group handling in Cygwin

[Andy Hall]

> On Mar  2 14:20, Frank Fesevur wrote:
> > 2014-02-28 22:08 GMT+01:00 Corinna Vinschen:
> > > That's not really a problem but a case of "it is as it is".  To get the
> > > user and group info, Cygwin has to contact the DC and/or GC and then
> > > runs into a timeout.  Right now, the LDAP timeout is set to 3 seconds.
> > > I don't know yet if it's such a bright idea to lower this timeout value.
> >
> > I understand why it happens. And three seconds are not the problem.
> >
> > It is just a bit confusing to activate the shortcut and for 3 seconds
> > nothing seems to happen. First time I double-clicked again because I
> > though it wasn't working at all. Obviously a bit later two terminal
> > showed up. If mintty would should up empty and "wait" those three
> > seconds before something happens it is more clear it just takes some
> > startup time. But maybe that is more a mintty problem.
> 
> No, this is not mintty's fault.  You have to understand that the
> request for information from the DC is called before the actual
> application had any chance to initialize,  This happens before
> the application has, in fact, been called from the Cygwin DLL.
> 
> Therefore, if the three seconds are a burden, there are three choices
> going forward:
> 
> - We reduce the timeout to 1 second in general, which could be a problem
>   when utilizing a slow VPN conncetion to the DC,
> 
> - We allow the user to specify a timeout value and set the default to
>   1 second.
> 
> - We stick to the 3 seconds and users should switch to files (/etc/passwd
>   /etc/group) when being offline from their DC.
> 
> I'm really not sure what the best way is, I guess we as a community have
> to dice this out somehow.
> 
> 
> Corinna

Why not have a command that allows you choose whether or not you are "on-line", that is, currently connected to a
network with a DC or "off-line" where you are not.  In the former you consult the DC in the latter you use the local
files?  That way you can leave the timeout at a reasonable value.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> I'd need to test it in domain environment, but I don't have access to one with
>> Cygwin installed right now. Or, perhaps, I do?... let me check something.

> Sure.

I just thought of a derelict pair of virtualbox containers, that were set up
to simulate domain environment of my workplace. Glad they are still in a good
shape... for tests. Here are the results:

Round 1: Running `id' command 50 times:

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin                           
db_cache: no                                                                                      
Timer 1  Elapsed: 0:01:48,36                                                                      
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin                           
db_cache: yes                                                                                     
Timer 1  Elapsed: 0:00:36,64                                                                      
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
Default caching settings.                                                                         
Timer 1  Elapsed: 0:00:36,33                                                                      
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]



Round 2: Running /bin/true same 50 times:

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
db_cache: no
Timer 1  Elapsed: 0:00:01,64
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
db_cache: yes
Timer 1  Elapsed: 0:00:01,58
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
db_cache: full
Timer 1  Elapsed: 0:00:35,91
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]


I find it interesting, that running `id' with caching disabled takes about 2
times longer, than when any form of caching is enabled.
But overall, when caching is set to minimum, having more than 200
supplementary groups attached to your profile doesn't impose alot of trouble
on process startup times, as I can see.

The Cygwin installation in this test was a fresh one, augmented with stuff
from x86/cygwin-inst-20140228.tar.xz archive.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 03.03.2014, <23:10>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> I'd need to test it in domain environment, but I don't have access to one with
>> Cygwin installed right now. Or, perhaps, I do?... let me check something.

> Sure.

I just thought of a derelict pair of virtualbox containers, that were set up
to simulate domain environment of my workplace. Glad they are still in a good
shape... for tests. Here are the results:

Round 1: Running `id' command 50 times:

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin                           
db_cache: no                                                                                      
Timer 1  Elapsed: 0:01:48,36                                                                      
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin                           
db_cache: yes                                                                                     
Timer 1  Elapsed: 0:00:36,64                                                                      
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
Default caching settings.                                                                         
Timer 1  Elapsed: 0:00:36,33                                                                      
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]



Round 2: Running /bin/true same 50 times:

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
db_cache: no
Timer 1  Elapsed: 0:00:01,64
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
db_cache: yes
Timer 1  Elapsed: 0:00:01,58
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]

CYGWIN_NT-5.1 station1 1.7.29s(0.271/5/3) 20140228 15:31:43 i686 Cygwin
db_cache: full
Timer 1  Elapsed: 0:00:35,91
uid=1109576(domainuser) gid=1049089(Domain Users) groups=[some 200+ trash groups]


I find it interesting, that running `id' with caching disabled takes about 2
times longer, than when any form of caching is enabled.
But overall, when caching is set to minimum, having more than 200
supplementary groups attached to your profile doesn't impose alot of trouble
on process startup times, as I can see.

The Cygwin installation in this test was a fresh one, augmented with stuff
from x86/cygwin-inst-20140228.tar.xz archive.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 03.03.2014, <23:10>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> I'd need to test it in domain environment,

And with sane number of groups (less than 20), the results are barely
different between different caching strategies (less than 1%, which could
easily be explained by measurement error), except (again) for the case of
runing `id' under "db_cache: no", which takes approximately twice as long, as
any other way. 

If you ask me, the choice is between "db_cahce: yes" and "db_cache: full", and
I don't have solid reason to prefer one over another. Not running with remote
connection to the domain here...


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 04.03.2014, <00:33>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Corinna Vinschen!

>> I'd need to test it in domain environment,

And with sane number of groups (less than 20), the results are barely
different between different caching strategies (less than 1%, which could
easily be explained by measurement error), except (again) for the case of
runing `id' under "db_cache: no", which takes approximately twice as long, as
any other way. 

If you ask me, the choice is between "db_cahce: yes" and "db_cache: full", and
I don't have solid reason to prefer one over another. Not running with remote
connection to the domain here...


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 04.03.2014, <00:33>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 3/3/2014 08:52, Andrey Repin wrote:
>
> I'd say it again, "sane defaults are better, than alot of options".

Agreed in principle.

However, observe that all network stacks have a bunch of built-in 
timeout options.  They're rarely exposed to the user level, but their 
defaults are typically quite high.  (e.g. 60 seconds for connection 
timeout.)  Over the past 3 decades of TCP/IP, we've discovered that 
networks are weird.

> for comparison, default DNS _roundtrip_ timeout is 2 seconds,

The typical DNS transaction is just 2 UDP packets, one each direction: 
query and response.

I tested a simple, unencrypted LDAP login-and-drop-conn transaction here 
against a real production AD server, and it required 8 packets, 5 of 
which were TCP/IP connection establishment and shutdown.

Add in the encryption, authentication, and authorization overheads of a 
"real" LDAP query, and it could go up to dozens of packets.

That said, it only took about 1 ms to my simple test.  The AD server was 
on the other side of a router, on a fast WAN.

Someone testing this new cygwin1.dll in a domain environment[*] should 
do a packet capture of what Windows sends when the DLL does its new thing.

The captured data isn't terribly interesting here.  What we want to know 
is how many packets it takes, and what the timestamps are on the 
captured frames.  Most especially, the delta between the first and last 
packets, but if there are any significant time gaps, that could be 
interesting, too.


[*] Not me.  The only reason we have any AD around here at all is for 
testing software that authenticates users against third party AD servers.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Warren Young!

>> I'd say it again, "sane defaults are better, than alot of options".

> Agreed in principle.

> However, observe that all network stacks have a bunch of built-in 
> timeout options.  They're rarely exposed to the user level, but their 
> defaults are typically quite high.  (e.g. 60 seconds for connection 
> timeout.)

I've tried to rely on some obscure "system timeout"... it didn't worked well.
Socket just doesn't die by itself after hours of waiting.
I had to manually enforce some sane timeout to ensure consistent operation of
the program. Even if it'd be something like "15 minutes", it was still better
than "I don't know, may be next century..."

> Over the past 3 decades of TCP/IP, we've discovered that 
> networks are weird.

I concur.

>> for comparison, default DNS _roundtrip_ timeout is 2 seconds,

> The typical DNS transaction is just 2 UDP packets, one each direction: 
> query and response.

> I tested a simple, unencrypted LDAP login-and-drop-conn transaction here 
> against a real production AD server, and it required 8 packets, 5 of 
> which were TCP/IP connection establishment and shutdown.

> Add in the encryption, authentication, and authorization overheads of a 
> "real" LDAP query, and it could go up to dozens of packets.

> That said, it only took about 1 ms to my simple test.  The AD server was 
> on the other side of a router, on a fast WAN.

Perhaps, you didn't understand, but I specifically mentioned connection time,
which is clearly distinct in case of TCP connection.
Once connection is established, control is returned to application, the
connection timeout is a thing of past, and session can only be terminated by
a request of calling application.

> Someone testing this new cygwin1.dll in a domain environment[*] should 
> do a packet capture of what Windows sends when the DLL does its new thing.

> The captured data isn't terribly interesting here.  What we want to know 
> is how many packets it takes, and what the timestamps are on the 
> captured frames.  Most especially, the delta between the first and last 
> packets, but if there are any significant time gaps, that could be 
> interesting, too.

Amount of packets doesn't matter. Once TCP session is established, it remains,
until closed or dropped on either end of the wire.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 04.03.2014, <05:25>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 3/3/2014 18:36, Andrey Repin wrote:
> Once TCP session is established, it remains,
> until closed or dropped on either end of the wire.

Have you done the packet capture to prove that Windows does in fact keep 
the LDAP connection to the AD server up continually, or are you making 
an assumption?

> Amount of packets doesn't matter

Even if the connection stays up all the time, an LDAP lookup is almost 
certain to take more than one packet each way.

Packet round trip time (RTT) varies by orders of magnitude among 
networks, from sub-ms on a fast, quiet LAN to on the order of a full 
second for a slow WAN.

Since the number of packets is a function of the number of round trips, 
a network with an RTT of 0.9s will take more than the proposed 3 seconds 
to process a query that requires 4 round trips, even if the processing 
delay on either end is essentially 0.

You have to measure it to find out.

And yes, I have personally used networks with an RTT > 1.0s.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

On Mar  3 22:07, Warren Young wrote:
> On 3/3/2014 18:36, Andrey Repin wrote:
> >Once TCP session is established, it remains,
> >until closed or dropped on either end of the wire.
> 
> Have you done the packet capture to prove that Windows does in fact
> keep the LDAP connection to the AD server up continually, or are you
> making an assumption?
> 
> >Amount of packets doesn't matter
> 
> Even if the connection stays up all the time, an LDAP lookup is
> almost certain to take more than one packet each way.
> 
> Packet round trip time (RTT) varies by orders of magnitude among
> networks, from sub-ms on a fast, quiet LAN to on the order of a full
> second for a slow WAN.
> 
> Since the number of packets is a function of the number of round
> trips, a network with an RTT of 0.9s will take more than the
> proposed 3 seconds to process a query that requires 4 round trips,
> even if the processing delay on either end is essentially 0.
> 
> You have to measure it to find out.
> 
> And yes, I have personally used networks with an RTT > 1.0s.

I was inclined to go with Andrey's suggestion for simplicity.  What's
yours?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

On Mar  4 00:44, Andrey Repin wrote:
> Greetings, Corinna Vinschen!
> 
> >> I'd need to test it in domain environment,
> 
> And with sane number of groups (less than 20), the results are barely
> different between different caching strategies (less than 1%, which could
> easily be explained by measurement error), except (again) for the case of
> runing `id' under "db_cache: no", which takes approximately twice as long, as
> any other way. 
> 
> If you ask me, the choice is between "db_cahce: yes" and "db_cache: full", and
> I don't have solid reason to prefer one over another. Not running with remote
> connection to the domain here...

Ok, thanks.  I guess I'll remove the "db_cache: no" option at one
point.  It just doesn't seem to make sense.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Warren Young]

On 3/4/2014 01:07, Corinna Vinschen wrote:
> On Mar  3 22:07, Warren Young wrote:
>>
>> You have to measure it to find out.
>
> I was inclined to go with Andrey's suggestion for simplicity.  What's
> yours?

Science!

Survey actual lookup times, and make a statistical determination from 
that.  e.g. +3 sigma[1] above the normal to capture > 99% of the Cygwin 
user base, or +6 sigma to reduce outliers to the order of 2 ppm.

To collect the data, instrument the DLL and report anonymous statistics 
back to cygwin.com if the user opts in via setup.exe.

Rather than report back continually, just keep a model of the lookup 
time[2], and report the current lookup time average on each setup.exe 
run, such as in the HTTP request for the mirror list:

     GET /mirrors.lst?adluavg=123

That is, setup.exe is reporting that the typical AD lookup time at this 
site is currently 123 ms.

Note that you don't have to write a CGI program to accomplish this. 
Apache will continue to serve static mirror.lst files for you with the 
GET parameter.[3]  It will dutifully log each query to access_log, and 
you can dig the reported lookup time values out of the file when you 
want to compile your statistics.

When calculating statistics, use only the latest value for each 
reporting IP, so that sites that update Cygwin more often don't skew the 
results.

The default timeout should be baked into the DLL, but overridable in 
nsswitch.conf, with ms resolution.

The default shouldn't be in the shipping nsswitch.conf file, because you 
want DLL updates to be able to provide a new default timeout if your 
initial guess turns out to be inappropriate.

You want ms resolution, since my test above shows at least one case 
where the normal lookup time is on the order of 1 ms.  If cygwin1.dll 
defaults to 1 s, I'd want to change my local timeout to 10 or 100 ms.



[1] https://en.wikipedia.org/wiki/Standard_deviation

[2] A decaying average technique should work.  Save the last 10 or so 
lookup times to the registry on DLL shutdown.  If there isn't that much 
lookup data on hand, start with repeats of the last known lookup time, 
or the default value.

[3] Try it: http://cygwin.com/mirrors.lst?foo=bar

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, Warren Young!

> On 3/4/2014 01:07, Corinna Vinschen wrote:
>> On Mar  3 22:07, Warren Young wrote:
>>>
>>> You have to measure it to find out.
>>
>> I was inclined to go with Andrey's suggestion for simplicity.  What's
>> yours?

> Science!

> Survey actual lookup times,

TCP is not UDP.
Lookup time is not connection time.
This is third time I say it. Should be enough.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 05.03.2014, <20:00>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]

On Mar  5 08:34, Warren Young wrote:
> On 3/4/2014 01:07, Corinna Vinschen wrote:
> >On Mar  3 22:07, Warren Young wrote:
> >>
> >>You have to measure it to find out.
> >
> >I was inclined to go with Andrey's suggestion for simplicity.  What's
> >yours?
> 
> Science!
> 
> Survey actual lookup times, and make a statistical determination
> from that.  e.g. +3 sigma[1] above the normal to capture > 99% of
> the Cygwin user base, or +6 sigma to reduce outliers to the order of
> 2 ppm.
> 
> To collect the data, instrument the DLL and report anonymous
> statistics back to cygwin.com if the user opts in via setup.exe.
> 
> Rather than report back continually, just keep a model of the lookup
> time[2], and report the current lookup time average on each
> setup.exe run, such as in the HTTP request for the mirror list:
> 
>     GET /mirrors.lst?adluavg=123
> 
> That is, setup.exe is reporting that the typical AD lookup time at
> this site is currently 123 ms.
> 
> Note that you don't have to write a CGI program to accomplish this.
> Apache will continue to serve static mirror.lst files for you with
> the GET parameter.[3]  It will dutifully log each query to
> access_log, and you can dig the reported lookup time values out of
> the file when you want to compile your statistics.
> 
> When calculating statistics, use only the latest value for each
> reporting IP, so that sites that update Cygwin more often don't skew
> the results.
> 
> The default timeout should be baked into the DLL, but overridable in
> nsswitch.conf, with ms resolution.

You're kidding, aren't you?  Apart from that, ms are not useful since
LDAP_OPT_TIMELIMIT used for ldap_bind_s only takes values in 1 second
resolution.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> The fact that the shells are doing it right seems to indicate that this
> isn't a generic problem.  I can't debug this, though.  Can you see if
> you can figure out what's going on under the hood?  Does strace show
> anything of interest?  Can we perhaps set up some joint debugging via
> private mail during the next couple of days?

Perl goes through stat64 that seems to explicitly check the ACL while sh
uses a different codepath.  I don't see anything obviously wrong with either
trace.  If I set up cygdrive to use the noacl option the problem goes away,
apparently because the ACL check never takes place within stat64.

  869 2501512 [main] perl 2604 stat64: entering
  814 2502326 [main] perl 2604 normalize_posix_path: src x86
  802 2503128 [main] perl 2604 cwdstuff::get: posix /cygdrive/x/install
 1195 2504323 [main] perl 2604 cwdstuff::get: (/cygdrive/x/install) =
cwdstuff::get (0x80000008, 32768, 1, 0), errno 0
  810 2505133 [main] perl 2604 normalize_posix_path: /cygdrive/x/install/x86
= normalize_posix_path (x86)
 1195 2506328 [main] perl 2604 mount_info::conv_to_win32_path:
conv_to_win32_path (/cygdrive/x/install/x86)
  810 2507138 [main] perl 2604 mount_info::cygdrive_win32_path: src
'/cygdrive/x/install/x86', dst 'X:\install\x86'
  837 2507975 [main] perl 2604 set_flags: flags: binary (0x2)
  810 2508785 [main] perl 2604 mount_info::conv_to_win32_path: src_path
/cygdrive/x/install/x86, dst X:\install\x86, flags 0x4022, rc 0
 3329 2512114 [main] perl 2604 symlink_info::check: 0x0 = NtCreateFile
(\??\X:\install\x86)
 2816 2514930 [main] perl 2604 symlink_info::check: not a symlink
  845 2515775 [main] perl 2604 symlink_info::check: 0 =
symlink.check(X:\install\x86, 0x289888) (0x404022)
  839 2516614 [main] perl 2604 path_conv::check: this->path(X:\install\x86),
has_acls(1)
  828 2517442 [main] perl 2604 build_fh_pc: fh 0x612DD5A0, dev 000000C3
  816 2518258 [main] perl 2604 stat_worker: (\??\X:\install\x86, 0x800390D0,
0x612DD5A0), file_attributes 16
 1603 2519861 [main] perl 2604 cygpsid::debug_print: get_sids_info: owner
SID = S-1-5-21-2052111302-842925246-682003330-75441
  818 2520679 [main] perl 2604 cygpsid::debug_print: get_sids_info: group
SID = S-1-5-21-2052111302-842925246-682003330-513
  818 2521497 [main] perl 2604 get_info_from_sd: ACL 0x4000, uid 75441, gid
10513
  875 2522372 [main] perl 2604 fhandler_base::fstat_helper: 0 = fstat
(\??\X:\install\x86, 0x800390D0) st_size=0, st_mode=0x4000, st_ino=-197262732544
4575109st_atim=531DE525.1B5BB150 st_ctim=530C5D84.1F71B690
st_mtim=52D570D0.251FE418 st_birthtim=51EFE5A9.12BDBAC0

 1424 1978515 [main] sh 4736 normalize_posix_path: src x86
  882 1979397 [main] sh 4736 cwdstuff::get: posix /cygdrive/x/install
  918 1980315 [main] sh 4736 cwdstuff::get: (/cygdrive/x/install) =
cwdstuff::get (0x20000008, 32768, 1, 0), errno 0
 1647 1981962 [main] sh 4736 normalize_posix_path: /cygdrive/x/install/x86 =
normalize_posix_path (x86)
 1218 1983180 [main] sh 4736 mount_info::conv_to_win32_path:
conv_to_win32_path (/cygdrive/x/install/x86)
  932 1984112 [main] sh 4736 mount_info::cygdrive_win32_path: src
'/cygdrive/x/install/x86', dst 'X:\install\x86'
 2389 1986501 [main] sh 4736 set_flags: flags: binary (0x2)
 1221 1987722 [main] sh 4736 mount_info::conv_to_win32_path: src_path
/cygdrive/x/install/x86, dst X:\install\x86, flags 0x4022, rc 0
 4069 1991791 [main] sh 4736 symlink_info::check: 0x0 = NtCreateFile
(\??\X:\install\x86)
 2709 1994500 [main] sh 4736 symlink_info::check: not a symlink
  985 1995485 [main] sh 4736 symlink_info::check: 0 =
symlink.check(X:\install\x86, 0x289408) (0x404022)
  946 1996431 [main] sh 4736 path_conv::check: this->path(X:\install\x86),
has_acls(1)
  947 1997378 [main] sh 4736 build_fh_pc: fh 0x612DD8D8, dev 000000C3
 1935 1999313 [main] sh 4736 check_file_access: flags 0x2, ret 0
  971 2000284 [main] sh 4736 fhandler_base::fhaccess: returning 0
 1526 2001810 [main] sh 4736 euidaccess: returning 0

Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

On Mar 10 17:19, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > The fact that the shells are doing it right seems to indicate that this
> > isn't a generic problem.  I can't debug this, though.  Can you see if
> > you can figure out what's going on under the hood?  Does strace show
> > anything of interest?  Can we perhaps set up some joint debugging via
> > private mail during the next couple of days?
> 
> Perl goes through stat64 that seems to explicitly check the ACL while sh
> uses a different codepath.  I don't see anything obviously wrong with either
> trace.  If I set up cygdrive to use the noacl option the problem goes away,
> apparently because the ACL check never takes place within stat64.
> 
>   869 2501512 [main] perl 2604 stat64: entering
>   814 2502326 [main] perl 2604 normalize_posix_path: src x86
>   802 2503128 [main] perl 2604 cwdstuff::get: posix /cygdrive/x/install
>  1195 2504323 [main] perl 2604 cwdstuff::get: (/cygdrive/x/install) =
> cwdstuff::get (0x80000008, 32768, 1, 0), errno 0
>   810 2505133 [main] perl 2604 normalize_posix_path: /cygdrive/x/install/x86
> = normalize_posix_path (x86)
>  1195 2506328 [main] perl 2604 mount_info::conv_to_win32_path:
> conv_to_win32_path (/cygdrive/x/install/x86)
>   810 2507138 [main] perl 2604 mount_info::cygdrive_win32_path: src
> '/cygdrive/x/install/x86', dst 'X:\install\x86'
>   837 2507975 [main] perl 2604 set_flags: flags: binary (0x2)
>   810 2508785 [main] perl 2604 mount_info::conv_to_win32_path: src_path
> /cygdrive/x/install/x86, dst X:\install\x86, flags 0x4022, rc 0
>  3329 2512114 [main] perl 2604 symlink_info::check: 0x0 = NtCreateFile
> (\??\X:\install\x86)
>  2816 2514930 [main] perl 2604 symlink_info::check: not a symlink
>   845 2515775 [main] perl 2604 symlink_info::check: 0 =
> symlink.check(X:\install\x86, 0x289888) (0x404022)
>   839 2516614 [main] perl 2604 path_conv::check: this->path(X:\install\x86),
> has_acls(1)
>   828 2517442 [main] perl 2604 build_fh_pc: fh 0x612DD5A0, dev 000000C3
>   816 2518258 [main] perl 2604 stat_worker: (\??\X:\install\x86, 0x800390D0,
> 0x612DD5A0), file_attributes 16
>  1603 2519861 [main] perl 2604 cygpsid::debug_print: get_sids_info: owner
> SID = S-1-5-21-2052111302-842925246-682003330-75441
>   818 2520679 [main] perl 2604 cygpsid::debug_print: get_sids_info: group
> SID = S-1-5-21-2052111302-842925246-682003330-513
>   818 2521497 [main] perl 2604 get_info_from_sd: ACL 0x4000, uid 75441, gid
> 10513
>   875 2522372 [main] perl 2604 fhandler_base::fstat_helper: 0 = fstat
> (\??\X:\install\x86, 0x800390D0) st_size=0, st_mode=0x4000, st_ino=-197262732544
                                              ^^^^^^^^^^^^^^
This is the important snippet, but I don't see how this could have been
different before my patches.  The mode is S_IFDIR and 000 permissions.

That usually means:

- The owner of the file, here S-1-5-21-2052111302-842925246-682003330-75441,
  has no ACCESS_ALLOWED_ACE in the ACL, or the owner has no FILE_READ_DATA,
  FILE_WRITE_DATA, and FILE_EXECUTE permissions on the file.

- The group of the file, here S-1-5-21-2052111302-842925246-682003330-513
  (Domain Users, apparently) has no ACCESS_ALLOWED_ACE in the ACL, or
  the owner has no FILE_READ_DATA, FILE_WRITE_DATA, and FILE_EXECUTE
  permissions on the file.

- The Everyone group S-1-1-0 has no ACCESS_ALLOWED_ACE in the ACL, or
  the owner has no FILE_READ_DATA, FILE_WRITE_DATA, and FILE_EXECUTE
  permissions on the file.

This stuff is entirely independent of the new passwd/group code, unless
the owner and group are Samba Unix Users/Groups (S-1-22-[...]), in which
case I made some changes in this area on 2014-02-27.

The uid and gid values point to the fact that you're still using a
passwd and group file.  How are your /etc/nsswitch.conf settings and
does switching to db-only make a difference?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
>> (\??\X:\install\x86, 0x800390D0) st_size=0, st_mode=0x4000, st_ino=-197262732544
>                                               ^^^^^^^^^^^^^^
> This is the important snippet, but I don't see how this could have been
> different before my patches.  The mode is S_IFDIR and 000 permissions.

I've run the same on Cygwin64 (where I don't use the snapshot yet) and
it does indeed produce the same line.  It still correctly determines
that I do have permission to change into (and write in) the directory,
but I don't know how.

> That usually means:
>
> - The owner of the file, here S-1-5-21-2052111302-842925246-682003330-75441,
>   has no ACCESS_ALLOWED_ACE in the ACL, or the owner has no FILE_READ_DATA,
>   FILE_WRITE_DATA, and FILE_EXECUTE permissions on the file.
>
> - The group of the file, here S-1-5-21-2052111302-842925246-682003330-513
>   (Domain Users, apparently) has no ACCESS_ALLOWED_ACE in the ACL, or
>   the owner has no FILE_READ_DATA, FILE_WRITE_DATA, and FILE_EXECUTE
>   permissions on the file.
>
> - The Everyone group S-1-1-0 has no ACCESS_ALLOWED_ACE in the ACL, or
>   the owner has no FILE_READ_DATA, FILE_WRITE_DATA, and FILE_EXECUTE
>   permissions on the file.
>
> This stuff is entirely independent of the new passwd/group code, unless
> the owner and group are Samba Unix Users/Groups (S-1-22-[...]), in which
> case I made some changes in this area on 2014-02-27.

The owner is me and the primary group is indeed Domain Users.  As I
said, the whole share (a NetApp filer) is set up to not forbid access to
anyone except via extended security settings that enable access for a
certain AD group (and administrative access for another).  These
settings are forced upon all new files via inheritance, plus if I
managed to change this (there was such a loophole once, but it likely
has been closed) there'd be a script to periodically remove all extra
permissions.

The owner and groups are not Samba Unix as far as I can tell.

> The uid and gid values point to the fact that you're still using a
> passwd and group file.  How are your /etc/nsswitch.conf settings and
> does switching to db-only make a difference?

The same test without an /etc/passwd file produces a different uid (the
original one in /etc/passwd was actually 85441 and I just changed it to
see where it came from), I haven't yet checked if the nsswitch.conf
settings make a difference.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

On Mar 10 19:28, Achim Gratz wrote:
> Corinna Vinschen writes:
> >> (\??\X:\install\x86, 0x800390D0) st_size=0, st_mode=0x4000, st_ino=-197262732544
> >                                               ^^^^^^^^^^^^^^
> > This is the important snippet, but I don't see how this could have been
> > different before my patches.  The mode is S_IFDIR and 000 permissions.
> 
> I've run the same on Cygwin64 (where I don't use the snapshot yet) and
> it does indeed produce the same line.  It still correctly determines
> that I do have permission to change into (and write in) the directory,
> but I don't know how.

In that case the stat call is very likely unrelated.  There must be some
other call involved in perl.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen writes:
> In that case the stat call is very likely unrelated.  There must be some
> other call involved in perl.

IIRC, the next call is the write that prints "no" or "yes"... there may
have been something like "stat_helper" inbetween only on Cygwin64, but
I'll have to check again tomorrow.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Achim Gratz <Stromeko <at> nexgo.de> writes:
> IIRC, the next call is the write that prints "no" or "yes"... there may
> have been something like "stat_helper" inbetween only on Cygwin64, but
> I'll have to check again tomorrow.

It's a call to stat_worker with the UNC file path and the stat_worker
handle, both times returning zero.  There is only one interesting difference
between the release and the snapshot DLL, they are reporting the atime to be
different:

st_atim=531DE525.1B5BB150 (release)
st_atim=531DF887.5D9B9F8  (snapshot)

Whatever differences there are probably inside stat_worker, but I don't
really know.

I've just found out that cygcheck cuts off the group listing after exactly
16kiB.  Maybe there are some other places where it's now possible to overrun
some buffer (pwdgrp::fetch_account_from_windows has got all of them
according top the trace) or maybe some other limit for the number of groups?
 The group that grants access is number 293 in the list as returned by id...

I've also found that two trusted domains aren't returning an entry for
cyg_ldap::fetch_posix_offset_for_domain from the DC I'm assigned to, so that
probably explains the long startup times I've been seeing with earlier
snapshots.  Maybe of interest:

  987 1803457 [main] perl 6856 alloc_sd: uid 1124017, gid 10513, attribute
0x2190
  860 1804317 [main] perl 6856 cygsid::debug_print: alloc_sd: owner SID =
S-1-5-21-2052111302-842925246-682003330-75441 (+)
  877 1805194 [main] perl 6856 cygsid::debug_print: alloc_sd: group SID =
S-1-5-21-2052111302-842925246-682003330-513 (+)
  827 1806021 [main] perl 6856 alloc_sd: ACL-Size: 124
  934 1806955 [main] perl 6856 alloc_sd: Created SD-Size: 200



Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2924 bytes --]

On Mar 11 07:44, Achim Gratz wrote:
> Achim Gratz <Stromeko <at> nexgo.de> writes:
> > IIRC, the next call is the write that prints "no" or "yes"... there may
> > have been something like "stat_helper" inbetween only on Cygwin64, but
> > I'll have to check again tomorrow.
> 
> It's a call to stat_worker with the UNC file path and the stat_worker
> handle, both times returning zero.

zero == success

> There is only one interesting difference
> between the release and the snapshot DLL, they are reporting the atime to be
> different:
> 
> st_atim=531DE525.1B5BB150 (release)
> st_atim=531DF887.5D9B9F8  (snapshot)

Access time.  On Windows it even changes when requesting certain
kinds of metadata :-P

> Whatever differences there are probably inside stat_worker, but I don't
> really know.

stat_worker is just a wrapper.  Nothing happens in there which could
explain any differences.

> I've just found out that cygcheck cuts off the group listing after exactly
> 16kiB.

That's because cygcheck uses a fixed buffer of 16K to collect the
output of the `id' call.

> Maybe there are some other places where it's now possible to overrun
> some buffer (pwdgrp::fetch_account_from_windows has got all of them
> according top the trace) or maybe some other limit for the number of groups?

Not in Cygwin.  The limit is set by the application in the call
to getgroups: http://linux.die.net/man/2/getgroups

What we have is a NGROUPS_MAX define in limits.h and the equivalent
NGROUPS in sys/param.h.  It's defined as 1024, but it's not used
or enforced anywhere in Cygwin.

>  The group that grants access is number 293 in the list as returned by id...

But the question is, does perl actually use that list?  The strace
snippets don't show anything here.  If it doesn't use the access(2)
call, it would have to load the full ACL of the file and match that
against your token group list.  This requires calls to getgroups (which
would create the litany of group SIDs from fetch_account_from_windows)
and acl.  It's pretty unlikely that perl would do this manually.

> I've also found that two trusted domains aren't returning an entry for
> cyg_ldap::fetch_posix_offset_for_domain from the DC I'm assigned to, so that
> probably explains the long startup times I've been seeing with earlier
> snapshots.  Maybe of interest:
> 
>   987 1803457 [main] perl 6856 alloc_sd: uid 1124017, gid 10513, attribute
> 0x2190

You're still using a group file?  

Anyway, I need the full straces.  It's pretty hard to say anything, let
alone isolate at least the most probable cause without them.  Can you
please run the simple examples under the 2014-03-09 snapshot and send
URLs to the snapshots?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > st_atim=531DE525.1B5BB150 (release)
> > st_atim=531DF887.5D9B9F8  (snapshot)
> 
> Access time.  On Windows it even changes when requesting certain
> kinds of metadata :-P

It's been consistent over many days of testing and the only difference
between release and snapshot, so I thought I'd mention it.

> But the question is, does perl actually use that list?  The strace
> snippets don't show anything here.  If it doesn't use the access(2)
> call, it would have to load the full ACL of the file and match that
> against your token group list.  This requires calls to getgroups (which
> would create the litany of group SIDs from fetch_account_from_windows)
> and acl.  It's pretty unlikely that perl would do this manually.

Not that I can see, it simply seems to call stat64, which it did before.  I
don't think it drops any groups, either.  And again, just mounting cygdrive
with "noacl" gets rid of the problem entirely.

> You're still using a group file?  

I think yes, I hadn't moved it away (but no passwd file).

> Anyway, I need the full straces.  It's pretty hard to say anything, let
> alone isolate at least the most probable cause without them.  Can you
> please run the simple examples under the 2014-03-09 snapshot and send
> URLs to the snapshots?

I've just updated to the latest snapshot w/o AD integration for testing,
will revert to the other one for more testing.  Not sure if I can fit that
in today, will send the links via PM when I have the traces uploaded.


Regards,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]

On Mar 11 12:07, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > > st_atim=531DE525.1B5BB150 (release)
> > > st_atim=531DF887.5D9B9F8  (snapshot)
> > 
> > Access time.  On Windows it even changes when requesting certain
> > kinds of metadata :-P
> 
> It's been consistent over many days of testing and the only difference
> between release and snapshot, so I thought I'd mention it.

Can't explain that.  There are no changes at all in this piece of code.

> > But the question is, does perl actually use that list?  The strace
> > snippets don't show anything here.  If it doesn't use the access(2)
> > call, it would have to load the full ACL of the file and match that
> > against your token group list.  This requires calls to getgroups (which
> > would create the litany of group SIDs from fetch_account_from_windows)
> > and acl.  It's pretty unlikely that perl would do this manually.
> 
> Not that I can see, it simply seems to call stat64, which it did before.  I
> don't think it drops any groups, either.  And again, just mounting cygdrive
> with "noacl" gets rid of the problem entirely.

There's a test in perl somewhere, which fails.  "noacl" only fakes
permissions anyway, so it doesn't matter.

> > You're still using a group file?  
> 
> I think yes, I hadn't moved it away (but no passwd file).

You don't have to move them away.  Just set nsswitch.conf.

> > Anyway, I need the full straces.  It's pretty hard to say anything, let
> > alone isolate at least the most probable cause without them.  Can you
> > please run the simple examples under the 2014-03-09 snapshot and send
> > URLs to the snapshots?
> 
> I've just updated to the latest snapshot w/o AD integration for testing,
> will revert to the other one for more testing.  Not sure if I can fit that
> in today, will send the links via PM when I have the traces uploaded.

Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> You don't have to move them away.  Just set nsswitch.conf.

Did that and using the snapshot DLL from 2014-03-05 on top of a full
snapshot install from 2014-03-10.  The ACL is this:

# file: x86
# owner: gratz
# group: Domain Users
user::---
group::---
group:admin-cygwinupload:rwx
group:user-cygwinupload:rwx
mask:rwx
other:---
default:user::---
default:group::---
default:group:admin-cygwinupload:rwx
default:group:user-cygwinupload:rwx
default:mask:rwx
default:other:---

With the original passwd and group file in place and nsswitch.conf set to
either "files" or "files db" the test fails.  With just "files" getfacl
doesn't show the group ACL at all, while with "files db" I see the ACL for
both the admin and the user group (both are not in the group file).  Setting
to just "db" the ACL is shown as before and the test from Perl now succeeds!
 In fact any combination that includes "files" fails.  So, after some head
scratching I changed the uid and gid in the passwd and group files to match
the new mapping scheme and lo and behold the test is now working.  The
getfacl command starts to show the group ACL when I add them to the group
file (with the correct gid mapping), but the test still fails with "files"
only.  With the correct group entries and "files db", the test also works.

So, Perl somehow uses the gid/uid mapping and relies on those to be working,
while bash uses a code path that doesn't and probably just uses the uid/gid
directly.  I guess I could make the "files" only case work by adding some
more groups (no time for checking what that might be at the moment), again
changing the mapping (will mkpasswd do this at some point?).  Do you still
need traces or does get you a test case that works in your environment?


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3235 bytes --]

On Mar 11 15:07, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > You don't have to move them away.  Just set nsswitch.conf.
> 
> Did that and using the snapshot DLL from 2014-03-05 on top of a full
> snapshot install from 2014-03-10.  The ACL is this:
> 
> # file: x86
> # owner: gratz
> # group: Domain Users
> user::---
> group::---
> group:admin-cygwinupload:rwx
> group:user-cygwinupload:rwx
> mask:rwx
> other:---
> default:user::---
> default:group::---
> default:group:admin-cygwinupload:rwx
> default:group:user-cygwinupload:rwx
> default:mask:rwx
> default:other:---
> 
> With the original passwd and group file in place and nsswitch.conf set to
> either "files" or "files db" the test fails.  With just "files" getfacl
> doesn't show the group ACL at all,

How does it look with any non-AD integrated Cygwin?

> while with "files db" I see the ACL for
> both the admin and the user group (both are not in the group file).  Setting
> to just "db" the ACL is shown as before and the test from Perl now succeeds!

Ok.

>  In fact any combination that includes "files" fails.

Hmm.  So you're saying that the groups in question are not in
/etc/groups, but it works with the non-AD Cygwin but not with the
AD-Cygwin?  A group which is not in /etc/groups is, in theory, just not
in the ACL with the old Cygwin.  What's not in Cygwin anymore is the
mapping of a non-existing account to the uid/gid -1, what would have
been printed as "????????" in ls output.  This automatism would have
collided with the DB stuff, but maybe I have to re-introduce it if only
"files" is used.  This could explain what happens in the "files"-only
case...

...but that doesn't explain what happens with "files db".  The uid/gid
values may differ from the DB values, but only if the account actually
exists in the file.  And then the values in the files would have
precedent over the db values.  I'm really wondering what perl is
checking there.

> So, after some head
> scratching I changed the uid and gid in the passwd and group files to match
> the new mapping scheme and lo and behold the test is now working.  The
> getfacl command starts to show the group ACL when I add them to the group
> file (with the correct gid mapping), but the test still fails with "files"
> only.  With the correct group entries and "files db", the test also works.

Erm...

> So, Perl somehow uses the gid/uid mapping and relies on those to be working,

Whatever it's doing there.  That doesn't make sense, unless it calls
getgrent maybe?!?

> while bash uses a code path that doesn't and probably just uses the uid/gid
> directly.

Much easier.  bash just calls access(2).

> I guess I could make the "files" only case work by adding some
> more groups (no time for checking what that might be at the moment), again
> changing the mapping (will mkpasswd do this at some point?).  Do you still
> need traces or does get you a test case that works in your environment?

Yes, please.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > With the original passwd and group file in place and nsswitch.conf set to
> > either "files" or "files db" the test fails.  With just "files" getfacl
> > doesn't show the group ACL at all,
> 
> How does it look with any non-AD integrated Cygwin?

... doesn't show the group ACL until I add them to the group file.  That
part is consistent with the AD enabled snapshot.  Actually... if I create a
group file with those two groups added, the access again doesn't get
granted.  Which finally reveals that I also need to have the administrators
group present in that file (which mkpasswd had been doing) -- then it works.
 I can even leave out the two ACL groups again and it still works.

> Hmm.  So you're saying that the groups in question are not in
> /etc/groups, but it works with the non-AD Cygwin but not with the
> AD-Cygwin?

Exactly.  But as revealed above, what was really missing is the
Administrators group.  Somehow, when "files" is in effect, that mapping
doesn't seem to exist unless it is explicitly listed in the file.  It does
get auto-created when I use _only_ the "db".  I hope that somehow makes sense...

> > So, Perl somehow uses the gid/uid mapping and relies on those to be working,

No, it seems to balk on not being able to map the Administrator group (which
is my egid).


Regards,
Achim.




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Achim Gratz <Stromeko <at> NexGo.DE> writes:
> Exactly.  But as revealed above, what was really missing is the
> Administrators group.  Somehow, when "files" is in effect, that mapping
> doesn't seem to exist unless it is explicitly listed in the file.  It does
> get auto-created when I use _only_ the "db".  I hope that somehow makes
sense...

I guess it does: the mapping that gets created from AD is sometimes 1049120
instead of 544.  That depends on the settings in nsswitch.conf and whether
an /etc/group file exists at all or contains an entry for Administrators.  I
guess that once this behaviour is stable (and maps to 544 in every case) the
problem that Perl is having also goes away.


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2138 bytes --]

On Mar 11 17:05, Achim Gratz wrote:
> Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> > > With the original passwd and group file in place and nsswitch.conf set to
> > > either "files" or "files db" the test fails.  With just "files" getfacl
> > > doesn't show the group ACL at all,
> > 
> > How does it look with any non-AD integrated Cygwin?
> 
> ... doesn't show the group ACL until I add them to the group file.  That
> part is consistent with the AD enabled snapshot.  Actually... if I create a
> group file with those two groups added, the access again doesn't get
> granted.  Which finally reveals that I also need to have the administrators
> group present in that file (which mkpasswd had been doing) -- then it works.
>  I can even leave out the two ACL groups again and it still works.
> 
> > Hmm.  So you're saying that the groups in question are not in
> > /etc/groups, but it works with the non-AD Cygwin but not with the
> > AD-Cygwin?
> 
> Exactly.  But as revealed above, what was really missing is the
> Administrators group.  Somehow, when "files" is in effect, that mapping
> doesn't seem to exist unless it is explicitly listed in the file.  It does
> get auto-created when I use _only_ the "db".  I hope that somehow makes sense...
> 
> > > So, Perl somehow uses the gid/uid mapping and relies on those to be working,
> 
> No, it seems to balk on not being able to map the Administrator group (which
> is my egid).

Eh, what?  You never mentioned that before!  So you changed your passwd
file to have group 544 as primary group?  That usually won't work well
if UAC is enabled.

That's the problem with bug reports which don't provide *details*.  If
you had provided full straces, your passwd and group files and your
nsswitch.conf settings, we may have been a lot further.  As it is now,
we're discussing your perl problem for days and days and *still* don't
really know what's going on in the code :(


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]

On Mar 11 17:12, Achim Gratz wrote:
> Achim Gratz <Stromeko <at> NexGo.DE> writes:
> > Exactly.  But as revealed above, what was really missing is the
> > Administrators group.  Somehow, when "files" is in effect, that mapping
> > doesn't seem to exist unless it is explicitly listed in the file.  It does
> > get auto-created when I use _only_ the "db".  I hope that somehow makes
> sense...
> 
> I guess it does: the mapping that gets created from AD is sometimes 1049120
> instead of 544.  That depends on the settings in nsswitch.conf and whether
> an /etc/group file exists at all or contains an entry for Administrators.

Argh!  What about *details* which allow to reproduce this behaviour.

In my environment I *never* see the admins group being produced with
a GID of 1049120.  And it doesn't make any sense at all.  The UID/GID
values starting with 0x100000 are reserved for accounts of the primary
domain.  The admins account is *never* an account of the primary domain
because its SID doesn't start with S-1-5-21-YOUR-DOMAIN-SID.  Rather
it's a builtin account with SID 1-5-32-544, which always gets converted
to UID/GID 544.

[...time passes...]

Hmm.  The only reason I can see which would allow to generate the GID
1049120 is, if your account has a primaryGroupID of 544 in AD, but that
*should* be impossible.  The AD user properties dialog doesn't allow to
set the primary group to a builtin group... unless you set this in the
attribute editor.  This is totally unexpected by Cygwin.  From AD it
just gets a RID value, and it simply adds this to the primary domain
offset 0x100000.

Ok, so, here's the question.  Is your primaryGroupID in AD 544?  If not,
you will have to explain to me how this happens.  I have found no other
way to reproduce this.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Corinna Vinschen <corinna-cygwin <at> cygwin.com> writes:
> Ok, so, here's the question.  Is your primaryGroupID in AD 544?  If not,
> you will have to explain to me how this happens.  I have found no other
> way to reproduce this.

My primary group ID in AD is 513 (Domain Users), just as shown by id, I am
not member of the Administrators group and I only have effective group
membership when running from an elevated prompt (which I'm allowed to get to
via another group membership and a group policy that's totally unrelated to
the well-known Administrators group).  I have no idea what happens either,
none of the traces even contain a check for Administrators group or a
recognizable ID.   All I know is that when the mapping for Administrators
group is not 544, then the test fails.  Anyway, I've run out of time for the
moment, I'll try to set up a clean test environment later.  This whole
business of various settings in nsswitch.conf and entries in passwd/group is
creating a combinatorial explosion that I'm not sure how to manage yet. 
Then, I'll have to sanitize the traces, which will require a script if it's
more than two or three.


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Achim Gratz]

Achim Gratz <Stromeko <at> NexGo.DE> writes:
> I guess it does: the mapping that gets created from AD is sometimes 1049120
> instead of 544.  That depends on the settings in nsswitch.conf and whether
> an /etc/group file exists at all or contains an entry for Administrators.  I
> guess that once this behaviour is stable (and maps to 544 in every case) the
> problem that Perl is having also goes away.

I've installed todays snapshot and removed passwd, group and nsswitch.conf.
 So far everything looks OK (while I'm wired to the local network, wireless
and VPN testing will have to wait).  The Administrators group maps to 544 as
it should and Perl correctly determines accessibility of files and
directories via the ACL.


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>> >If you only want to use the content of your passwd and group files,
>> >rather than the db (which is pretty much upside down from what this
>> >patch is trying to accomplish, but, anyway), simply change your
>> >/etc/nsswitch.conf file accordingly.  Just as on Linux.
>>
>> :-) My /etc/nsswitch.conf file ALREADY specifies 'db_enum: files'.
>>
>> However, as you probably aware of, the output of 'id' _still_ shows all
>> the (non-file ;-) supplementary groups I do not care about ...
>>
>> That is what I tried to make clear with my message. Nothing more.
>
>Try
>
>  passwd: files
>  group: files
>
>The output of `id' has nothing to do with the getpwent/getgrent
>functionality db_enum is about.

Sorry, sorry, sorry (yes, I tried). How stupid of me. Thank you, thank you!

Apparently, I must learn how to read ... (shame on me!).

Thanks again, Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Feb 20 19:58, J.H. vd Water wrote:
> Hi Corinna,
> 
> >> However, as you know, the supplementary group to which I referred above do NOT show
> >> up in my /etc/group file on Cygwin (they show up in SAM ... somewhere).
> >>
> >> ... and I do not care, as these groups are NOT relevant to me as a Cygwin-only user
> >> of the Windows operating system (using passwd and group files).
> >
> >If you only want to use the content of your passwd and group files,
> >rather than the db (which is pretty much upside down from what this
> >patch is trying to accomplish, but, anyway), simply change your
> >/etc/nsswitch.conf file accordingly.  Just as on Linux.
> 
> :-) My /etc/nsswitch.conf file ALREADY specifies 'db_enum: files'.
> 
> However, as you probably aware of, the output of 'id' _still_ shows all
> the (non-file ;-) supplementary groups I do not care about ...
> 
> That is what I tried to make clear with my message. No more.

Try

  passwd: files
  group: files

The output of `id' has nothing to do with the getpwent/getgrent
functionality db_enum is about.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>> However, as you know, the supplementary group to which I referred above do NOT show
>> up in my /etc/group file on Cygwin (they show up in SAM ... somewhere).
>>
>> ... and I do not care, as these groups are NOT relevant to me as a Cygwin-only user
>> of the Windows operating system (using passwd and group files).
>
>If you only want to use the content of your passwd and group files,
>rather than the db (which is pretty much upside down from what this
>patch is trying to accomplish, but, anyway), simply change your
>/etc/nsswitch.conf file accordingly.  Just as on Linux.

:-) My /etc/nsswitch.conf file ALREADY specifies 'db_enum: files'.

However, as you probably aware of, the output of 'id' _still_ shows all
the (non-file ;-) supplementary groups I do not care about ...

That is what I tried to make clear with my message. No more.

Henri



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On Feb 20 14:28, J.H. vd Water wrote:
> >I don't know about you, but on my Linux box I get something like this:
> >
> >  $ id
> >  uid=500(corinna) gid=11125(vinschen) groups=11125(vinschen),33(tape),
> >  100(users),107(qemu),486(wireshark),1000(cvs),11126(libvirt)
> >
> >Most of the time I don't care about the supplementary groups after my
> >primary group either, but that doesn't mean I don't want id to print
> >them, nor are they unimportant.
> 
> Agreed, the supplementary groups I observe on Unix are relevant to me ... I know
> that out of experience.
> 
> The difference is, that all supplementary groups show up in the /etc/group file on
> my Unix box ...

That's because you're using the files option exclusively on your
Unix box.

> However, as you know, the supplementary group to which I referred above do NOT show
> up in my /etc/group file on Cygwin (they show up in SAM ... somewhere).
> 
> ... and I do not care, as these groups are NOT relevant to me as a Cygwin-only user
> of the Windows operating system (using passwd and group files).

If you only want to use the content of your passwd and group files,
rather than the db (which is pretty much upside down from what this
patch is trying to accomplish, but, anyway), simply change your
/etc/nsswitch.conf file accordingly.  Just as on Linux.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>> Now that the XP bug is out of the way ... I would like come back to the change in
>> the output of 'id'.
>>
>> Obviously, everything after '545(Users),' in the output of 'id' are "well-known"
>> concepts (Windows) to you (but, not to me) ...
>
>I explained that in my mail and I already wrote
>http://cygwin.com/cygwin-ug-net/ntsec.html at one point.
[snip]

Yes, yes, yes :-) I have read all that a long time ago ...

>The group concept in Windows isn't much different from the group concept in Unix,
> except that Windows applications don't really care for the owning group of objects,
> typically.

Agreed.
[snip]

>> But as an Unix adept, I do not really care about the stuff after '545(Users),' in
>> the output of 'id', especially in case of a passwd file and a group file.
>>
>> To me the stuff after '545(Users),' is ... well, superfluous (most certainly, most
>> of the time).

>I don't know about you, but on my Linux box I get something like this:
>
>  $ id
>  uid=500(corinna) gid=11125(vinschen) groups=11125(vinschen),33(tape),
>  100(users),107(qemu),486(wireshark),1000(cvs),11126(libvirt)
>
>Most of the time I don't care about the supplementary groups after my
>primary group either, but that doesn't mean I don't want id to print
>them, nor are they unimportant.

Agreed, the supplementary groups I observe on Unix are relevant to me ... I know
that out of experience.

The difference is, that all supplementary groups show up in the /etc/group file on
my Unix box ...

However, as you know, the supplementary group to which I referred above do NOT show
up in my /etc/group file on Cygwin (they show up in SAM ... somewhere).

... and I do not care, as these groups are NOT relevant to me as a Cygwin-only user
of the Windows operating system (using passwd and group files).

Sorry, for having a different opinion on the matter.

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2241 bytes --]

On Feb 20 12:24, J.H. vd Water wrote:
> Hi Corinna,
> 
> >> What I meant was:
> >>
> >> The output of 'id' in my "regular setup" shows:
> >>
> >> @@# id
> >> uid=1003(Henri) gid=513(None) groups=513(None),0(root),544(Administrators),545(Users)
> >>
> >> The output of 'id' in the "test setup" shows:
> >>
> >> $ id
> >> uid=1003(Henri) gid=513(None) groups=513(None),0(root),545(Users),4(+INTERACTIVE),\
> >> 11(+AuthenticatedUsers),4095(CurrentSession),66048(+LOCAL)
> >>
> >> To me the second output is different - but perhaps it is not.
> >
> >It shows groups which are present in your user token, but which are
> >not in your /etc/group file.
> 
> Now that the XP bug is out of the way ... I would like come back to the change in
> the output of 'id'.
> 
> Obviously, everything after '545(Users),' in the output of 'id' are "well-known"
> concepts (Windows) to you (but, not to me) ...

I explained that in my mail and I already wrote
http://cygwin.com/cygwin-ug-net/ntsec.html at one point.  There are also
links on the MSDN web site which explain how it works.  The group concept
in Windows isn't much different from the group concept in Unix, except
that Windows applications don't really care for the owning group of
objects, typically.

> ... and I know Cygwin is not Linux/Unix ...

We're striving for similarity...

> But as an Unix adept, I do not really care about the stuff after '545(Users),' in
> the output of 'id', especially in case of a passwd file and a group file.
> 
> To me the stuff after '545(Users),' is ... well, superfluous (most certainly, most
> of the time).
> 
> Perhaps other users think likewise ...

I don't know about you, but on my Linux box I get something like this:

  $ id
  uid=500(corinna) gid=11125(vinschen) groups=11125(vinschen),33(tape),
  100(users),107(qemu),486(wireshark),1000(cvs),11126(libvirt)

Most of the time I don't care about the supplementary groups after my
primary group either, but that doesn't mean I don't want id to print
them, nor are they unimportant.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>> What I meant was:
>>
>> The output of 'id' in my "regular setup" shows:
>>
>> @@# id
>> uid=1003(Henri) gid=513(None) groups=513(None),0(root),544(Administrators),545(Users)
>>
>> The output of 'id' in the "test setup" shows:
>>
>> $ id
>> uid=1003(Henri) gid=513(None) groups=513(None),0(root),545(Users),4(+INTERACTIVE),\
>> 11(+AuthenticatedUsers),4095(CurrentSession),66048(+LOCAL)
>>
>> To me the second output is different - but perhaps it is not.
>
>It shows groups which are present in your user token, but which are
>not in your /etc/group file.

Now that the XP bug is out of the way ... I would like come back to the change in
the output of 'id'.

Obviously, everything after '545(Users),' in the output of 'id' are "well-known"
concepts (Windows) to you (but, not to me) ...

... and I know Cygwin is not Linux/Unix ...

But as an Unix adept, I do not really care about the stuff after '545(Users),' in
the output of 'id', especially in case of a passwd file and a group file.

To me the stuff after '545(Users),' is ... well, superfluous (most certainly, most
of the time).

Perhaps other users think likewise ...

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Feb 19 21:58, J.H. vd Water wrote:
> Hi Corinna,
> 
> >Can you please just test the today's snapshot which should show up
> >soon on the http://cygwin.com/snapshots/ page.
> 
> Like Andrey I tested the 19/2 snapshot of today (x86).
> 
> For good measure, I again started afresh ...
> 
> getpwent (db_enum: local) no longer crashes. Splendid!

Thanks for testing!


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>Can you please just test the today's snapshot which should show up
>soon on the http://cygwin.com/snapshots/ page.

Like Andrey I tested the 19/2 snapshot of today (x86).

For good measure, I again started afresh ...

getpwent (db_enum: local) no longer crashes. Splendid!

Henri



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On Feb 19 12:01, J.H. vd Water wrote:
> Hi Corinna,
> 
> (Sorry, have't found time to properly subscribe to the list, yet)
> 
> >> Both cases show the same output:
> >>
> >> $ ./lsa
> >> pdom name: <WORKGROUP> dnsname: <(null)>, sid: 0x0
> >> adom name: <XP> sid: 0x246058
> 
> I "lied" here :-) The non-admin case showed: ... sid: 0x246060
> (So, almost the same)
> 
> >Huh.  This looks entirely normal and expected.  Which makes the
> >SEGV even more weird.  I'm apparently missing something here.
> >
> >> Perhaps, it is best to stop here for the moment ... There must be more pressing
> >> things on your list.
> >
> >No, there aren't.  This testing is very important.  I'd like to
> >have such crashes like yours fixed before this new code gets
> >released.
> 
> >> Moreover, I already decided to use 'db_enum: files' in /etc/nssswitch.conf (and
> >> XP is a thing of the past, is it not?).
> 
> But who will still be using XP within a company after after april 2014?'. So, does
> have XP priority? Your decision, of course ...
> 
> >I'm going to make some tests on XP, but if I can't track this down,
> >would you mind if I send you a test Cygwin DLL with extended debug
> >output to help tracking it down?
> 
> The best thing to do, I believe, if you want to fix 'XP' (my machine may turn out
> to be a "misfit").
> 
> However, I have started afresh with the 18/2 snapshot ...
> 
> Same result: getpwent crashed (db_enum: local) ... getgrent is ok.
> 
> Send me a cygwin1.dll? Ok, if you tell me how to handle the thing (i.e. include
> some pointers to documents that I must read in order to be able to get results).

Can you please just test the today's snapshot which should show up
soon on the http://cygwin.com/snapshots/ page.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Andrey,

>>>I'm going to make some tests on XP, but if I can't track this down,
>>>would you mind if I send you a test Cygwin DLL with extended debug
>>>output to help tracking it down?
>
>> The best thing to do, I believe, if you want to fix 'XP' (my machine may turn out
>> to be a "misfit").
>
>> However, I have started afresh with the 18/2 snapshot ...
>
>> Same result: getpwent crashed (db_enum: local) ... getgrent is ok.
>
>> Send me a cygwin1.dll? Ok, if you tell me how to handle the thing (i.e. include
>> some pointers to documents that I must read in order to be able to get results).
>
> I'm volunteering as well. And I'm a bit more sure of my system conditions >.>

:-)) ... to tell you the truth: a few months ago I completely reinstalled XP from
scratch ... all went well, according to "Redmond".

However I do not have the same expert knowledge of Windows as Corinna or you have;
as result of that, I do not see "the signs on the wall" if something is 'obviously'
wrong.

So, my machine might not turn out to be a misfit, meaning: others, who are still in
favour of using XP, are strongly advised to test Corinna's work as well.

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Andrey Repin]

Greetings, J.H. vd Water!

>>> Both cases show the same output:
>>>
>>> $ ./lsa
>>> pdom name: <WORKGROUP> dnsname: <(null)>, sid: 0x0
>>> adom name: <XP> sid: 0x246058

> I "lied" here :-) The non-admin case showed: ... sid: 0x246060
> (So, almost the same)

Yeah, I understand, they supposed to be different.
Mine's also different, but results show data similar to yours.

> But who will still be using XP within a company after after april 2014?'. So, does
> have XP priority? Your decision, of course ...

We will. Because we're not going to go over a headache of reinstalling
everything just because some jerk out in the world decided we should.
It's just not worth the money spent. Even if you ignore the cost of new OS
itself.

>>I'm going to make some tests on XP, but if I can't track this down,
>>would you mind if I send you a test Cygwin DLL with extended debug
>>output to help tracking it down?

> The best thing to do, I believe, if you want to fix 'XP' (my machine may turn out
> to be a "misfit").

> However, I have started afresh with the 18/2 snapshot ...

> Same result: getpwent crashed (db_enum: local) ... getgrent is ok.

> Send me a cygwin1.dll? Ok, if you tell me how to handle the thing (i.e. include
> some pointers to documents that I must read in order to be able to get results).

I'm volunteering as well. And I'm a bit more sure of my system conditions >.>


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 19.02.2014, <18:58>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

(Sorry, have't found time to properly subscribe to the list, yet)

>> Both cases show the same output:
>>
>> $ ./lsa
>> pdom name: <WORKGROUP> dnsname: <(null)>, sid: 0x0
>> adom name: <XP> sid: 0x246058

I "lied" here :-) The non-admin case showed: ... sid: 0x246060
(So, almost the same)

>Huh.  This looks entirely normal and expected.  Which makes the
>SEGV even more weird.  I'm apparently missing something here.
>
>> Perhaps, it is best to stop here for the moment ... There must be more pressing
>> things on your list.
>
>No, there aren't.  This testing is very important.  I'd like to
>have such crashes like yours fixed before this new code gets
>released.

>> Moreover, I already decided to use 'db_enum: files' in /etc/nssswitch.conf (and
>> XP is a thing of the past, is it not?).

But who will still be using XP within a company after after april 2014?'. So, does
have XP priority? Your decision, of course ...

>I'm going to make some tests on XP, but if I can't track this down,
>would you mind if I send you a test Cygwin DLL with extended debug
>output to help tracking it down?

The best thing to do, I believe, if you want to fix 'XP' (my machine may turn out
to be a "misfit").

However, I have started afresh with the 18/2 snapshot ...

Same result: getpwent crashed (db_enum: local) ... getgrent is ok.

Send me a cygwin1.dll? Ok, if you tell me how to handle the thing (i.e. include
some pointers to documents that I must read in order to be able to get results).

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

On Feb 18 21:02, J.H. vd Water wrote:
> Hi Corinna,
> 
> >The crash looks weird.  It looks like your machine doesn't return
> >the machine sid when it's requested.  Can you please run the following
> >test application as non-admin and as admin and paster the output for
> >both cases into your reply?  Hmm, maybe that's a windows XP thingy?
> >I didn't test this stuff on anything prior to Vista.
> 
> Both cases show the same output:
> 
> $ ./lsa
> pdom name: <WORKGROUP> dnsname: <(null)>, sid: 0x0
> adom name: <XP> sid: 0x246058

Huh.  This looks entirely normal and expected.  Which makes the
SEGV even more weird.  I'm apparently missing something here.

> Perhaps, it is best to stop here for the moment ... There must be more pressing
> things on your list.

No, there aren't.  This testing is very important.  I'd like to
have such crashes like yours fixed before this new code gets
released.

> Moreover, I already decided to use 'db_enum: files' in /etc/nssswitch.conf (and
> XP is a thing of the past, is it not?).

I'm going to make some tests on XP, but if I can't track this down,
would you mind if I send you a test Cygwin DLL with extended debug
output to help tracking it down?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>The crash looks weird.  It looks like your machine doesn't return
>the machine sid when it's requested.  Can you please run the following
>test application as non-admin and as admin and paster the output for
>both cases into your reply?  Hmm, maybe that's a windows XP thingy?
>I didn't test this stuff on anything prior to Vista.

Both cases show the same output:

$ ./lsa
pdom name: <WORKGROUP> dnsname: <(null)>, sid: 0x0
adom name: <XP> sid: 0x246058

Perhaps, it is best to stop here for the moment ... There must be more pressing
things on your list.

Moreover, I already decided to use 'db_enum: files' in /etc/nssswitch.conf (and
XP is a thing of the past, is it not?).

Anyhow, thank you for listening and for your effort.

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Testers needed: New passwd/group handling in Cygwin

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> These functions are not called for domain accounts, only for local
> accounts.

Okay then..

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On Feb 18 19:18, Lavrentiev, Anton (NIH/NLM/NCBI) [C] wrote:
> This will not work (verified!) for code run from under an unmanaged
> Windows service account (NT_Service\...), once the machine changes its
> password per security policy (the access then becomes anonymous and will
> result in only first 100 entries returned):
> 
> winsup\cygwin\passwd.cc:
> 	  else if (group)
> 	    ret = NetGroupEnum (NULL, 2, (PBYTE *) &buf, MAX_PREFERRED_LENGTH,
> 				&max, &total, &resume);
> 	  else
> 	    ret = NetUserEnum (NULL, 20, FILTER_NORMAL_ACCOUNT, (PBYTE *) &buf,
> 			       MAX_PREFERRED_LENGTH, &max, &total,
> 			       (PDWORD) &resume);
> 
> This is what I was trying to point out, in my earlier message...

These functions are not called for domain accounts, only for local
accounts.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Testers needed: New passwd/group handling in Cygwin

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

This will not work (verified!) for code run from under an unmanaged
Windows service account (NT_Service\...), once the machine changes its
password per security policy (the access then becomes anonymous and will
result in only first 100 entries returned):

winsup\cygwin\passwd.cc:
	  else if (group)
	    ret = NetGroupEnum (NULL, 2, (PBYTE *) &buf, MAX_PREFERRED_LENGTH,
				&max, &total, &resume);
	  else
	    ret = NetUserEnum (NULL, 20, FILTER_NORMAL_ACCOUNT, (PBYTE *) &buf,
			       MAX_PREFERRED_LENGTH, &max, &total,
			       (PDWORD) &resume);

This is what I was trying to point out, in my earlier message...

Anton Lavrentiev
Contractor NIH/NLM/NCBI

P.S.  This behavior is obscurely documented in here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa370610(v=vs.85).aspx
(by the virtue of that page is pointed to from NetUserEnum and NetGroupEnum:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa370652(v=vs.85).aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/aa370428(v=vs.85).aspx)

<quote>
The number of entries returned by this function depends on the security descriptor located on the root domain object. The API will return either the first 100 entries or the entire set of entries in the domain, depending on the access privileges of the user. The ACE used to control this behavior is "SAM-Enumerate-Entire-Domain", and is granted to Authenticated Users by default. Administrators can modify this setting to allow users to enumerate the entire domain.
</quote>

The bad thing is that there is error indication at reaching the 100,
so it just looks like your environment has suddenly reduced to exactly 100
users...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1376 bytes --]

On Feb 18 19:44, J.H. vd Water wrote:
> Hi Corinna,
> 
> >> After I had set up /etc/nsswitch.conf as follows
> >>
> >> db_enum: files
> >>
> >> the output of getpwent and getgrent looked "familiar" to me. (though the
> >> output of 'id' is not).
> >
> >In how far?  Did you read my text in terms of how user and group names
> >are created?  The leading separator char for builtin groups is an SFU
> 
> Read and understand your text completely? No.

Uh, well, I graciously allow to ask questions ;)

> >thingy.  If it's too disturbing we can discuss changing that, but right
> >now I'm still looking for some kind of similarity.
> 
> What I meant was:
> 
> The output of 'id' in my "regular setup" shows:
> 
> @@# id
> uid=1003(Henri) gid=513(None) groups=513(None),0(root),544(Administrators),545(Users)
> 
> The output of 'id' in the "test setup" shows:
> 
> $ id
> uid=1003(Henri) gid=513(None) groups=513(None),0(root),545(Users),4(+INTERACTIVE),\
> 11(+AuthenticatedUsers),4095CurrentSession),66048(+LOCAL)
> 
> To me the second output is different - but perhaps it is not.

It shows groups which are present in your user token, but which are
not in your /etc/group file.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2431 bytes --]

On Feb 18 19:14, J.H. vd Water wrote:
> Hi Corinna,
> 
> To answer some of your questions ...
> 
> >> How did you build getpwent?  Did you stop your Cygwin shell and restart
> >> it?  Can you please send the getpwent.exe.stackdump file?
> 
> I used my "regular" setup of Cygwin to compile the sourcefile. Perhaps it is
> there where I went wrong.
> 
> gcc -o getpwent -Wall -std=c99 -fno-builtin -pedantic getpwent.c

No, that's unnecessarily detailed, but certainly not wrong.

> And, yes, I restarted my Cygwin shell.
> 
> >And, is the machine a domain member machine or not?
> 
> No, not a member of any domain ...

The crash looks weird.  It looks like your machine doesn't return
the machine sid when it's requested.  Can you please run the following
test application as non-admin and as admin and paster the output for
both cases into your reply?  Hmm, maybe that's a windows XP thingy?
I didn't test this stuff on anything prior to Vista.

==== SNIP ====
#include <stdio.h>
#include <windows.h>
#include <ntstatus.h>
#include <ntsecapi.h>

int
main ()
{
  static LSA_OBJECT_ATTRIBUTES oa = { 0, 0, 0, 0, 0, 0 };
  HANDLE lsa;
  NTSTATUS status;

  status = LsaOpenPolicy (NULL, &oa, POLICY_VIEW_LOCAL_INFORMATION, &lsa);
  if (status == STATUS_SUCCESS)
    {
      PPOLICY_DNS_DOMAIN_INFO pdom;
      PPOLICY_ACCOUNT_DOMAIN_INFO adom;

      status = LsaQueryInformationPolicy (lsa, PolicyDnsDomainInformation,
                                          (PVOID *) &pdom);
      if (status == STATUS_SUCCESS)
        {
          printf ("pdom name: <%ls> dnsname: <%ls>, sid: %p\n",
                  pdom->Name.Buffer, pdom->DnsDomainName.Buffer, pdom->Sid);
          LsaFreeMemory (pdom);
        }
      else
        fprintf (stderr, "LsaQueryInformationPolicy(Primary) 0x%x", status);
      status = LsaQueryInformationPolicy (lsa, PolicyAccountDomainInformation,
                                            (PVOID *) &adom);
      if (status == STATUS_SUCCESS)
        {
          printf ("adom name: <%ls> sid: %p\n",
                  adom->DomainName.Buffer, adom->DomainSid);
          LsaFreeMemory (adom);
        }
      LsaClose (lsa);
    }
  return 0;
}
==== SNAP ====


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

Hi Corinna,

>> After I had set up /etc/nsswitch.conf as follows
>>
>> db_enum: files
>>
>> the output of getpwent and getgrent looked "familiar" to me. (though the
>> output of 'id' is not).
>
>In how far?  Did you read my text in terms of how user and group names
>are created?  The leading separator char for builtin groups is an SFU

Read and understand your text completely? No.

>thingy.  If it's too disturbing we can discuss changing that, but right
>now I'm still looking for some kind of similarity.

What I meant was:

The output of 'id' in my "regular setup" shows:

@@# id
uid=1003(Henri) gid=513(None) groups=513(None),0(root),544(Administrators),545(Users)

The output of 'id' in the "test setup" shows:

$ id
uid=1003(Henri) gid=513(None) groups=513(None),0(root),545(Users),4(+INTERACTIVE),\
11(+AuthenticatedUsers),4095CurrentSession),66048(+LOCAL)

To me the second output is different - but perhaps it is not.

Henri




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

Hi Corinna,

To answer some of your questions ...

>> How did you build getpwent?  Did you stop your Cygwin shell and restart
>> it?  Can you please send the getpwent.exe.stackdump file?

I used my "regular" setup of Cygwin to compile the sourcefile. Perhaps it is
there where I went wrong.

gcc -o getpwent -Wall -std=c99 -fno-builtin -pedantic getpwent.c

And, yes, I restarted my Cygwin shell.

>And, is the machine a domain member machine or not?

No, not a member of any domain ...

Henri

[-- Attachment #2: getpwent.exe.stackdump --]
[-- Type: application/octet-stream, Size: 886 bytes --]

Exception: STATUS_ACCESS_VIOLATION at eip=610A2418
eax=00000001 ebx=0022AB94 ecx=7C809A0D edx=00000000 esi=61268128 edi=61268120
ebp=0022ABD8 esp=0022AB54 program=E:\Cygwin17\home\Henri\test\getpwent.exe, pid 448, thread main
cs=001B ds=0023 es=0023 fs=003B gs=0000 ss=0023
Stack trace:
Frame     Function  Args
0022ABD8  610A2418 (61268120, 00000000, 00000000, 00000000)
0022AC78  610A32D8 (00000001, 0022AC9C, 20010100, 61007FDA)
0022AD28  61008039 (00000000, 0022CDB4, 610071D0, 00000000)
0022CD88  61005E84 (0022CDB4, 00000000, 00000000, 00000000)
0022FF58  61005FF6 (610071D0, 00000000, 00000003, B7EEDC8C)
0022FF78  61006F54 (00401190, 00000000, 005C0000, E1300C00)
0022FF98  00401252 (00401190, 00000006, B7EEDD08, B7EEDD08)
0022FFC0  00401015 (00000000, 0000006D, 7FFDC000, 805512FA)
0022FFF0  7C81776F (00401000, 00000000, 78746341, 00000020)
End of stack trace

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

On Feb 18 18:48, Corinna Vinschen wrote:
> On Feb 18 18:16, J.H. vd Water wrote:
> > >> this week I applied the first incarnation of the new passwd/group
> > >> handling code to the Cygwin repository and after fixing a crash which
> > >> manifested in Denis Excoffier's network, I think we're at a point
> > >> which allows to push this forward.
> > >> [...]
> > >
> > >Ok guys, I just applied a patch implementing getpwent and getgrent,
> > >and the way to configure the output is probably more detailed than
> > >you ever wanted ...
> > 
> > Hi Corinna,
> > 
> > As I saw no response to your latest call for testers,  I decided to test it
> > myself ... (x86-XP, SP3).
> > 
> >  - setup "Cygwin" (base) in a different directory
> >  - applied tar ball (17/2 snapshot) as instructed at
> > 
> >    http://cygwin.com/faq-nochunks.html#faq.setup.snapshots
> > 
> > After I had set up /etc/nsswitch.conf as follows
> > 
> > db_enum: files
> > 
> > the output of getpwent and getgrent looked "familiar" to me. (though the
> > output of 'id' is not).
> 
> In how far?  Did you read my text in terms of how user and group names
> are created?  The leading separator char for builtin groups is an SFU
> thingy.  If it's too disturbing we can discuss changing that, but right
> now I'm still looking for some kind of similarity.
> 
> > However, after I had modified /etc/nsswitch.conf as follows
> > 
> > db_enum: local
> > 
> > getpwent crashed (segment violation) ...
> 
> Thanks for testing, but I can't reproduce this.  I tried on 32 bit
> Windows 7 and 64 bit Windows 8.1.
> 
> How did you build getpwent?  Did you stop your Cygwin shell and restart
> it?  Can you please send the getpwent.exe.stackdump file?

And, is the machine a domain member machine or not?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

On Feb 18 18:16, J.H. vd Water wrote:
> >> this week I applied the first incarnation of the new passwd/group
> >> handling code to the Cygwin repository and after fixing a crash which
> >> manifested in Denis Excoffier's network, I think we're at a point
> >> which allows to push this forward.
> >> [...]
> >
> >Ok guys, I just applied a patch implementing getpwent and getgrent,
> >and the way to configure the output is probably more detailed than
> >you ever wanted ...
> 
> Hi Corinna,
> 
> As I saw no response to your latest call for testers,  I decided to test it
> myself ... (x86-XP, SP3).
> 
>  - setup "Cygwin" (base) in a different directory
>  - applied tar ball (17/2 snapshot) as instructed at
> 
>    http://cygwin.com/faq-nochunks.html#faq.setup.snapshots
> 
> After I had set up /etc/nsswitch.conf as follows
> 
> db_enum: files
> 
> the output of getpwent and getgrent looked "familiar" to me. (though the
> output of 'id' is not).

In how far?  Did you read my text in terms of how user and group names
are created?  The leading separator char for builtin groups is an SFU
thingy.  If it's too disturbing we can discuss changing that, but right
now I'm still looking for some kind of similarity.

> However, after I had modified /etc/nsswitch.conf as follows
> 
> db_enum: local
> 
> getpwent crashed (segment violation) ...

Thanks for testing, but I can't reproduce this.  I tried on 32 bit
Windows 7 and 64 bit Windows 8.1.

How did you build getpwent?  Did you stop your Cygwin shell and restart
it?  Can you please send the getpwent.exe.stackdump file?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Testers needed: New passwd/group handling in Cygwin

[J.H. vd Water]

>> this week I applied the first incarnation of the new passwd/group
>> handling code to the Cygwin repository and after fixing a crash which
>> manifested in Denis Excoffier's network, I think we're at a point
>> which allows to push this forward.
>> [...]
>
>Ok guys, I just applied a patch implementing getpwent and getgrent,
>and the way to configure the output is probably more detailed than
>you ever wanted ...

Hi Corinna,

As I saw no response to your latest call for testers,  I decided to test it
myself ... (x86-XP, SP3).

 - setup "Cygwin" (base) in a different directory
 - applied tar ball (17/2 snapshot) as instructed at

   http://cygwin.com/faq-nochunks.html#faq.setup.snapshots

After I had set up /etc/nsswitch.conf as follows

db_enum: files

the output of getpwent and getgrent looked "familiar" to me. (though the
output of 'id' is not).

However, after I had modified /etc/nsswitch.conf as follows

db_enum: local

getpwent crashed (segment violation) ...

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple