cygwin-1.7.28 getpwent header declaration changes ?

cygwin-1.7.28 getpwent header declaration changes ?

[Marco Atzeri]

on cygwin-1.7.28 64 bit but not on cygwin-1.7.27,
(only system difference) compiling octave I see:

----------------------------------------------------------------
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
:114:25: error: '::getpwent' has not been declared
    return octave_passwd (::getpwent (), msg);
                          ^
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
: In static member function 'static int 
octave_passwd::setpwent(std::string&)':
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
:171:3: error: '::setpwent' has not been declared
    ::setpwent ();
    ^
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
: In static member function 'static int 
octave_passwd::endpwent(std::string&)':
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
:191:3: error: '::endpwent' has not been declared
    ::endpwent ();
    ^
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
: In static member function 'static octave_passwd 
octave_passwd::getpwent(std::s
tring&)':
/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
:119:1: warning: control reaches end of non-void function [-Wreturn-type]
  }
  ^
Makefile:8702: recipe for target 
'system/system_libsystem_la-oct-passwd.lo' fail
ed
make[4]: *** [system/system_libsystem_la-oct-passwd.lo] Error 1

----------------------------------------------------------------

the piece of code is a very simple:
-----------------------------------------------
octave_passwd
octave_passwd::getpwent (std::string& msg)
{
#if defined HAVE_GETPWENT
   msg = std::string ();
   return octave_passwd (::getpwent (), msg);
#else
   msg = NOT_SUPPORTED ("getpwent");
   return octave_passwd ();
#endif
}

----------------------------------------

Is something slightly changed in the  relative system headers ?

Regards
Marco


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

On Feb  6 08:29, Marco Atzeri wrote:
> on cygwin-1.7.28 64 bit but not on cygwin-1.7.27,
> (only system difference) compiling octave I see:
> 
> ----------------------------------------------------------------
> /pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
> :114:25: error: '::getpwent' has not been declared
>    return octave_passwd (::getpwent (), msg);
>                          ^
> [...]
> Is something slightly changed in the  relative system headers ?

I don't know anything about octave, but what has changed is that newlib
is finally introducing BSD-like visibility macros.  The pwd.h header now
contains this:

  #if __XSI_VISIBLE >= 500
  struct passwd   *getpwent (void);
  void             setpwent (void);
  void             endpwent (void);
  #endif

The default settings per the sys/cdefs.h header are:

  #define __POSIX_VISIBLE         200809
  #define __XSI_VISIBLE           700
  #define __BSD_VISIBLE           1
  #define __ISO_C_VISIBLE         2011

unless your environment defines one of

  _XOPEN_SOURCE
  _POSIX_C_SOURCE
  _ANSI_SOURCE
  _C99_SOURCE
  _C11_SOURCE
  _GNU_SOURCE

Have a look at the end of /usr/include/sys/cdefs.h, it explains things.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Marco Atzeri]

On 06/02/2014 10:01, Corinna Vinschen wrote:
> On Feb  6 08:29, Marco Atzeri wrote:
>> on cygwin-1.7.28 64 bit but not on cygwin-1.7.27,
>> (only system difference) compiling octave I see:
>>
>> ----------------------------------------------------------------
>> /pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
>> :114:25: error: '::getpwent' has not been declared
>>     return octave_passwd (::getpwent (), msg);
>>                           ^
>> [...]
>> Is something slightly changed in the  relative system headers ?
>
> I don't know anything about octave, but what has changed is that newlib
> is finally introducing BSD-like visibility macros.  The pwd.h header now
> contains this:
>
>    #if __XSI_VISIBLE >= 500
>    struct passwd   *getpwent (void);
>    void             setpwent (void);
>    void             endpwent (void);
>    #endif
>
> The default settings per the sys/cdefs.h header are:
>
>    #define __POSIX_VISIBLE         200809
>    #define __XSI_VISIBLE           700
>    #define __BSD_VISIBLE           1
>    #define __ISO_C_VISIBLE         2011
>
> unless your environment defines one of
>
>    _XOPEN_SOURCE
>    _POSIX_C_SOURCE
>    _ANSI_SOURCE
>    _C99_SOURCE
>    _C11_SOURCE
>    _GNU_SOURCE
>
> Have a look at the end of /usr/include/sys/cdefs.h, it explains things.
>
>
> Corinna

thanks for the hints

for what I can see  "_GNU_SOURCE" is defined in both cases at
configure stage and reported in config.status and config.h

I will look to compare the preprocessor output to see why building
on  1.7.28 the "/* Everything and the kitchen sink. */" is not pulled in

Regards
Marco







--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Jan Nijtmans]

On Feb  6 08:29, Marco Atzeri wrote:
> Is something slightly changed in the  relative system headers ?

After upgrading to Cygwin 1.7.28, Sqlite doesn't compile any
more. See log below. When reverting to 1.7.27-2,
everything is back to normal.

The LOCK_EX macro's and friends are supposed to
be defined when including <sys/file.h>:
    <http://man7.org/linux/man-pages/man2/flock.2.html>
but they aren't any more.

I added a:
    #define __BSD_VISIBLE 1
to sqliteInt.h (before all headers are included), then
everything works again.

Regards,
       Jan Nijtmans

==============================================
sqlite3.c: In function 'flockCheckReservedLock':
sqlite3.c:25869:38: error: 'LOCK_EX' undeclared (first use in this function)
     int lrc = robust_flock(pFile->h, LOCK_EX | LOCK_NB);
                                      ^
sqlite3.c:25869:38: note: each undeclared identifier is reported only
once for e         ach function it appears in
sqlite3.c:25869:48: error: 'LOCK_NB' undeclared (first use in this function)
     int lrc = robust_flock(pFile->h, LOCK_EX | LOCK_NB);
                                                ^
sqlite3.c:25872:36: error: 'LOCK_UN' undeclared (first use in this function)
       lrc = robust_flock(pFile->h, LOCK_UN);
                                    ^
sqlite3.c: In function 'flockLock':
sqlite3.c:25949:30: error: 'LOCK_EX' undeclared (first use in this function)
   if (robust_flock(pFile->h, LOCK_EX | LOCK_NB)) {
                              ^
sqlite3.c:25949:40: error: 'LOCK_NB' undeclared (first use in this function)
   if (robust_flock(pFile->h, LOCK_EX | LOCK_NB)) {
                                        ^
sqlite3.c: In function 'flockUnlock':
sqlite3.c:25998:30: error: 'LOCK_UN' undeclared (first use in this function)
   if( robust_flock(pFile->h, LOCK_UN) ){
                              ^
Makefile:563: recipe for target 'sqlite3.lo' failed
make: *** [sqlite3.lo] Error 1

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Feb  6 14:00, Jan Nijtmans wrote:
> On Feb  6 08:29, Marco Atzeri wrote:
> > Is something slightly changed in the  relative system headers ?
> 
> After upgrading to Cygwin 1.7.28, Sqlite doesn't compile any
> more. See log below. When reverting to 1.7.27-2,
> everything is back to normal.
> 
> The LOCK_EX macro's and friends are supposed to
> be defined when including <sys/file.h>:
>     <http://man7.org/linux/man-pages/man2/flock.2.html>
> but they aren't any more.

They are.  There's an #include <fcntl.h> in sys/file.h, which in
turn includes sys/fcntl.h which includes sys/_default_fcntl.h.

In sys/_default_fcntl.h, the LOCK_xx macros are defined if
__BSD_VISIBLE is defined, which *is* defined by default, unless you
define one of

  _ANSI_SOURCE
  _C99_SOURCE
  _C11_SOURCE
  _POSIX_C_SOURCE
  _XOPEN_SOURCE


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Fwd: cygwin-1.7.28 getpwent header declaration changes ?

[Jan Nijtmans]

2014-02-06 15:02 GMT+01:00 Corinna Vinschen:
> In sys/_default_fcntl.h, the LOCK_xx macros are defined if
> __BSD_VISIBLE is defined, which *is* defined by default, unless you
> define one of
>
>   _ANSI_SOURCE
>   _C99_SOURCE
>   _C11_SOURCE
>   _POSIX_C_SOURCE
>   _XOPEN_SOURCE

Yes, SQLite defines _XOPEN_SOURCE in sqliteInt.h:

    <http://www.sqlite.org/src/artifact/fdab572b3567d587?ln=198-200>

I could add a !defined(__CYGWIN__) there, but I prefer
just to define __BSD_VISIBLE.

Thanks!

Regards.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

On Feb  6 15:15, Jan Nijtmans wrote:
> 2014-02-06 15:02 GMT+01:00 Corinna Vinschen:
> > In sys/_default_fcntl.h, the LOCK_xx macros are defined if
> > __BSD_VISIBLE is defined, which *is* defined by default, unless you
> > define one of
> >
> >   _ANSI_SOURCE
> >   _C99_SOURCE
> >   _C11_SOURCE
> >   _POSIX_C_SOURCE
> >   _XOPEN_SOURCE
> 
> Yes, SQLite defines _XOPEN_SOURCE in sqliteInt.h:
> 
>     <http://www.sqlite.org/src/artifact/fdab572b3567d587?ln=198-200>
> 
> I could add a !defined(__CYGWIN__) there, but I prefer
> just to define __BSD_VISIBLE.

On second thought, Linux defines the LOCK_xx macros unconditionally
in sys/file.h.  What we could do is to define the macros in sys/file.h
if they are not already defined because __BSD_VISIBLE wasn't set.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

On Feb  6 15:28, Corinna Vinschen wrote:
> On Feb  6 15:15, Jan Nijtmans wrote:
> > 2014-02-06 15:02 GMT+01:00 Corinna Vinschen:
> > > In sys/_default_fcntl.h, the LOCK_xx macros are defined if
> > > __BSD_VISIBLE is defined, which *is* defined by default, unless you
> > > define one of
> > >
> > >   _ANSI_SOURCE
> > >   _C99_SOURCE
> > >   _C11_SOURCE
> > >   _POSIX_C_SOURCE
> > >   _XOPEN_SOURCE
> > 
> > Yes, SQLite defines _XOPEN_SOURCE in sqliteInt.h:
> > 
> >     <http://www.sqlite.org/src/artifact/fdab572b3567d587?ln=198-200>
> > 
> > I could add a !defined(__CYGWIN__) there, but I prefer
> > just to define __BSD_VISIBLE.
> 
> On second thought, Linux defines the LOCK_xx macros unconditionally
> in sys/file.h.  What we could do is to define the macros in sys/file.h
> if they are not already defined because __BSD_VISIBLE wasn't set.

Jan, can you please check if SQlite builds *without* defining
__BSD_VISIBLE, but adding the following lines to /usr/include/sys/file.h;

Index: sys/file.h
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/include/sys/file.h,v
retrieving revision 1.8
diff -u -p -r1.8 file.h
--- sys/file.h	17 Jan 2014 11:01:46 -0000	1.8
+++ sys/file.h	6 Feb 2014 14:41:23 -0000
@@ -31,4 +31,16 @@
 #define L_INCR		SEEK_CUR
 #define L_XTND		SEEK_END
 
+/* Including <sys/file.h> always defines flock & macros. */
+#if __BSD_VISIBLE - 0  == 0
+
+#define LOCK_SH		0x01		/* shared file lock */
+#define LOCK_EX		0x02		/* exclusive file lock */
+#define LOCK_NB		0x04		/* don't block when locking */
+#define LOCK_UN		0x08		/* unlock file */
+
+extern int flock _PARAMS ((int, int));
+
+#endif
+
 #endif


Thanks in advance,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

On Feb  6 11:19, Marco Atzeri wrote:
> 
> 
> On 06/02/2014 10:01, Corinna Vinschen wrote:
> >On Feb  6 08:29, Marco Atzeri wrote:
> >>on cygwin-1.7.28 64 bit but not on cygwin-1.7.27,
> >>(only system difference) compiling octave I see:
> >>
> >>----------------------------------------------------------------
> >>/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
> >>:114:25: error: '::getpwent' has not been declared
> >>    return octave_passwd (::getpwent (), msg);
> >>                          ^
> >>[...]
> >>Is something slightly changed in the  relative system headers ?
> >
> >I don't know anything about octave, but what has changed is that newlib
> >is finally introducing BSD-like visibility macros.  The pwd.h header now
> >contains this:
> >
> >   #if __XSI_VISIBLE >= 500
> >   struct passwd   *getpwent (void);
> >   void             setpwent (void);
> >   void             endpwent (void);
> >   #endif
> > [...]
> 
> thanks for the hints
> 
> for what I can see  "_GNU_SOURCE" is defined in both cases at
> configure stage and reported in config.status and config.h
> 
> I will look to compare the preprocessor output to see why building
> on  1.7.28 the "/* Everything and the kitchen sink. */" is not pulled in

Btw., it would be a good idea to get rid of calls to getpwent/getgrent
in future.  They *probably* won't do anymore what they were supposed to
do if you don't have passwd/group files.  A full implementation would
require to enumerate the local SAM and, at least, the primary domain
accounts at runtime.  That would be possible, but it comes at a hefty
price in terms of performance.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/6/2014 07:13, Corinna Vinschen wrote:
>
> Btw., it would be a good idea to get rid of calls to getpwent/getgrent
> in future.  They *probably* won't do anymore what they were supposed to
> do if you don't have passwd/group files.

There must be a way to list an executable's DLL imports, and thereby do 
a survey on Cygwin to see which executables currently import those 
functions.  If so, I know a guy who currently has all of Cygwin 
downloaded and ready to re-install, to test this. :)

I tried futzing with objdump, nm, and dumpbin, and couldn't get anything 
useful.  Mainly what I got was either irrelevant or complaints of the 
"no string table" sort.

 > A full implementation would
> require to enumerate the local SAM and, at least, the primary domain
> accounts at runtime.  That would be possible, but it comes at a hefty
> price in terms of performance.

Linux and Cygwin are pretty much the last ones standing when it comes to 
preferentially reading plain text files in /etc for user info.  Big iron 
Unix, the BSDs, and Mac OS X now all treat these files as secondary to 
some behind-the-scenes database.

In some of these systems, you can edit /etc/foo and run a command to 
manually sync that content back to the "real" user info DB.  (e.g. the 
BSDs)  In others, direct edits to these files are ignored, but the OS 
syncs a subset of changes to the user info DB to these files, for the 
benefit of getpwent() and friends.  (e.g. Mac OS X.)

In Cygwin, we have a kind of hybrid of these, owing to the fact that the 
integration between Cygwin and Windows is pretty much one-way.  We have 
mkpasswd/group, which treats the DB as primary, like OS X, but which 
must be run manually to sync changes, like the BSDs.

I don't see a reason for this to change, given that so many other POSIX 
systems share aspects of this behavior.

It would be nicer if Cygwin behaved more like OS X in this regard. 
That is, for mkpasswd/group to be run automatically when the SAM/AD 
changes.  I don't see Microsoft doing that for us, though.

The only way I can think of for Cygwin to do that for itself would be to 
run mkpasswd/group from setup.exe, in the same way that it runs 
autorebase.

I realize the current recommended practice is to keep /etc/foo as small 
as possible, but shouldn't an AD/SAM DB lookup be faster than a linear 
scan of a large /etc/foo file?  Why lament the fact that getpwent() is 
slow, when getpwnam() is its logical replacement, and presumably much 
faster?

(I assume getpwnam() consults SAM/AD in Cygwin, now or in Cygwin.next.)

Here's the Mac OS X passwd(5) man page: http://goo.gl/AwIHku

It's relevant here because Mac OS X uses OpenDirectory, an LDAP 
directory server.  In that way, it is not unlike future-Cygwin+AD.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 4787 bytes --]

On Feb  6 14:43, Warren Young wrote:
> On 2/6/2014 07:13, Corinna Vinschen wrote:
> >
> >Btw., it would be a good idea to get rid of calls to getpwent/getgrent
> >in future.  They *probably* won't do anymore what they were supposed to
> >do if you don't have passwd/group files.
> 
> There must be a way to list an executable's DLL imports, and thereby
> do a survey on Cygwin to see which executables currently import
> those functions.  If so, I know a guy who currently has all of
> Cygwin downloaded and ready to re-install, to test this. :)

Try this:  strings -f /bin/*.exe /bin/*.dll | grep getgrent

> > A full implementation would
> >require to enumerate the local SAM and, at least, the primary domain
> >accounts at runtime.  That would be possible, but it comes at a hefty
> >price in terms of performance.
> 
> Linux and Cygwin are pretty much the last ones standing when it
> comes to preferentially reading plain text files in /etc for user
> info.  Big iron Unix, the BSDs, and Mac OS X now all treat these
> files as secondary to some behind-the-scenes database.

Which would be SAM and AD for us.

> In some of these systems, you can edit /etc/foo and run a command to
> manually sync that content back to the "real" user info DB.  (e.g.
> the BSDs)  In others, direct edits to these files are ignored, but
> the OS syncs a subset of changes to the user info DB to these files,
> for the benefit of getpwent() and friends.  (e.g. Mac OS X.)

That won't make much sense on Cygwin.  The idea is to use the existing
OS tools to maintain the user/group databases in the first place.
Having said that, it would, of course, be possible to implement Cygwin
command line tools along the lines of useradd/usermod/groupdel.  For AD,
they would just have to use LDAP, as Cygwin will do to read the account
info.  It's pretty simple, as far as you view LDAP as simple,

> In Cygwin, we have a kind of hybrid of these, owing to the fact that
> the integration between Cygwin and Windows is pretty much one-way.
> We have mkpasswd/group, which treats the DB as primary, like OS X,
> but which must be run manually to sync changes, like the BSDs.

That won't be necessary anymore.  In fact, I would prefer if we could
do without the passswd and group files, but we have to keep them
for backward compatibility at least.

However, I found that one functionality has to go completely.  It will
not be possible anymore to ssh into your machine and add yourself to
arbitrary groups by adding your user name to groups in /etc/group.  This
is just not feasible anymore.

> I don't see a reason for this to change, given that so many other
> POSIX systems share aspects of this behavior.

Sorry, I lost you there.  What exactly of the current behaviour do
you want to keep?

> It would be nicer if Cygwin behaved more like OS X in this regard.
> That is, for mkpasswd/group to be run automatically when the SAM/AD
> changes.  I don't see Microsoft doing that for us, though.

Cygwin will directly read from SAM/AD.  What's the point of automating
writing passwd and group files?  Also, the usual problem:  What exactly
do you write to passwd/group?  If you write all users of a big corp,
the files get really big.  So far, every exec'ing process is reading
the files anew.  This is *so* 1990ths.

> The only way I can think of for Cygwin to do that for itself would
> be to run mkpasswd/group from setup.exe, in the same way that it
> runs autorebase.

Sorry, but that sounds rather backward to me.  And again, what exactly
do you write to the files?  Only local accunts?  Or local and primary
domain?  Or local, primary domain and all trusted domains?  And what do
you do with local changes to the files?  mkpasswd/mkgroup would
overwrite all changes, which does not sound like a terribly good idea to
me.

> I realize the current recommended practice is to keep /etc/foo as
> small as possible, but shouldn't an AD/SAM DB lookup be faster than
> a linear scan of a large /etc/foo file?  Why lament the fact that
> getpwent() is slow, when getpwnam() is its logical replacement, and
> presumably much faster?

I don't lament the fact.  The problem is that a SAM/AD scan would have
to be implemented in Cygwin in the first place.  And then again, what do
you suppose Cygwin should do in big AD forests?  Enumerate all accounts
of all trusted domains?  As of today, this is an unsolved problem,


Corinna

> (I assume getpwnam() consults SAM/AD in Cygwin, now or in Cygwin.next.)

It checks for the file first, then it asks SAM/AD.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Corinna Vinschen!

>> In some of these systems, you can edit /etc/foo and run a command to
>> manually sync that content back to the "real" user info DB.  (e.g.
>> the BSDs)  In others, direct edits to these files are ignored, but
>> the OS syncs a subset of changes to the user info DB to these files,
>> for the benefit of getpwent() and friends.  (e.g. Mac OS X.)

> That won't make much sense on Cygwin.  The idea is to use the existing
> OS tools to maintain the user/group databases in the first place.
> Having said that, it would, of course, be possible to implement Cygwin
> command line tools along the lines of useradd/usermod/groupdel.  For AD,
> they would just have to use LDAP, as Cygwin will do to read the account
> info.  It's pretty simple, as far as you view LDAP as simple,

LDAP IS simple. It's a shame it is so little known outside specific circles.

> However, I found that one functionality has to go completely.  It will
> not be possible anymore to ssh into your machine and add yourself to
> arbitrary groups by adding your user name to groups in /etc/group.  This
> is just not feasible anymore.

Erm. I call it a security improvement. To put it mildly.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 07.02.2014, <16:43>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/7/2014 05:49, Andrey Repin wrote:
>
> LDAP IS simple.

Anything tied to a PKI is going to be pretty complex, no matter how 
simple the underlying tech is.

Then there's the fact that LDAP derives from X.500, a prototypically 
overengineered OSI emission.  DC=my,DC=sub,DC=domain,DC=com.  P'tui!

> It's a shame it is so little known outside specific circles.

I blame the books.

Several years ago, I bought most of what was available, since we needed 
to do user authentication against an existing AD system in a custom 
software system, to avoid parallel user management.

The most useful book I found was "LDAP: Programming Directory-Enabled 
Applications with Lightweight Directory Access Protocol", by Howes and 
Smith, since it covers things from the C API level upward.  Most other 
LDAP books either don't cover the programming level at all, or mention 
it briefly on their way down from the directory service level.

The thing is, the book was written way back in 1997, before OpenLDAP was 
forked from the umich LDAP reference implementation.  A huge number of 
the things the book says no longer applies because the API has changed 
out from under it.  Many of the examples that do still work give 
deprecation warnings, or need to be compiled with -D flags to re-enable 
APIs that are marked for deprecation.

I did a search on Amazon, sorting all LDAP books by release date.  The 
newest one that covers programing looks to be Howes & Smith's other 
book, "Understanding and Deploying LDAP Directory Services" (coauthored 
with Good) which came out in 2003.[1]  The book primarily covers 
administration of the Netscape (!) Directory Server.  It does cover LDAP 
programming in a couple of chapters spanning 150 pages, but most of the 
examples are in Java.  There is a touch of C and Perl, only.  Even if 
you count all the code samples together and ignore the language and API 
differences, you can't call this anything but a whirlwind overview of 
LDAP programming.

"Mastering OpenLDAP" is billed as "for developers", but there isn't a 
line of C code in the book.  (Or any other programming language, 
really.)  It's a sysadmin book.

It's not even the case that all the teaching material has moved online. 
  The OpenLDAP programming manual is a pile of man pages.[2]  Man pages 
are great, but go toss a neophyte into that pile and see if she doesn't 
come out welted, weary and wincing.

Given this, it's no wonder those who can write LDAP based software are 
seen as a kind of elite.  Acquiring a working knowledge of LDAP 
programming is akin to a baleen whale feeding itself.


----------

[1] Amazon claims it was updated in 2013, but it's just a paperback 
re-issue of their 2003 hardback title.

[2] http://goo.gl/gE2e6P

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Warren Young!

>> LDAP IS simple.

> Anything tied to a PKI is going to be pretty complex, no matter how 
> simple the underlying tech is.

> Then there's the fact that LDAP derives from X.500, a prototypically 
> overengineered OSI emission.  DC=my,DC=sub,DC=domain,DC=com.  P'tui!

Well, well, mom said spitting is not nice.
I agree that certain parts of specification look weird for untrained eye, but
being properly explained, they make sense.
Not to mention, this all could be wrapped up by a proper interface.

>> It's a shame it is so little known outside specific circles.

> I blame the books.

I concur. Was a long time since I started grasping it, but then it was
breakthrough. Much like regexps.

> Several years ago, I bought most of what was available, since we needed 
> to do user authentication against an existing AD system in a custom 
> software system, to avoid parallel user management.

> The most useful book I found was "LDAP: Programming Directory-Enabled 
> Applications with Lightweight Directory Access Protocol", by Howes and 
> Smith, since it covers things from the C API level upward.  Most other 
> LDAP books either don't cover the programming level at all, or mention 
> it briefly on their way down from the directory service level.

I find most books problematic to understand, because they talk too much about
implementation, and too little about actual application of the system in real
world.
As an example: I have little knowledge in TCP/IP internal structure, this
doesn't stop me from troubleshooting network issues... because it is not
really necessary in 90% of real situations. The 90% of the rest is covered by
a quick googling and little extrapolation. The 1% that is left falls under
"Professionalism: An ability to make right decisions in a situation that never
happen again in your life."
Same applies to LDAP. People spoil their \m/ \m/ about how awesome they are
using such awesome technology, but offering little real help.

> It's not even the case that all the teaching material has moved online.
>   The OpenLDAP programming manual is a pile of man pages.[2]  Man pages 
> are great, but go toss a neophyte into that pile and see if she doesn't 
> come out welted, weary and wincing.

Same for ISC-DHCP... :/ The WORST documentation support for (probably) the
MOST used DHCP server.

> Given this, it's no wonder those who can write LDAP based software are 
> seen as a kind of elite.  Acquiring a working knowledge of LDAP 
> programming is akin to a baleen whale feeding itself.

Programming anything is really simple. Or I find it really simple. If you know
how to use it, you can code it. If you don't know how to use it... You are
coding a monster noone will be able to use.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 07.02.2014, <22:02>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[David Stacey]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On 07/02/2014 09:49, Corinna Vinschen wrote:
> On Feb  6 14:43, Warren Young wrote:
>>   On 2/6/2014 07:13, Corinna Vinschen wrote:
>>> Btw., it would be a good idea to get rid of calls to getpwent/getgrent
>>> in future.  They*probably*  won't do anymore what they were supposed to
>>> do if you don't have passwd/group files.
>> There must be a way to list an executable's DLL imports, and thereby
>> do a survey on Cygwin to see which executables currently import
>> those functions.  If so, I know a guy who currently has all of
>> Cygwin downloaded and ready to re-install, to test this.:)
> Try this:  strings -f/bin/*.exe/bin/*.dll | grep getgrent

Let me save you the trouble of reinstalling it - one of my PCs has a 
complete install of both architectures. Results attached.

Cheers,

Dave.


[-- Attachment #2: cygwin32.txt --]
[-- Type: text/plain, Size: 4459 bytes --]

/bin/bash.exe: _getgrent32
/bin/biber.exe: getgrent
/bin/biber.exe: _getgrent32
/bin/biber.exe: __imp___getgrent32
/bin/biber.exe: _getgrent
/bin/cppcheck.exe: fgetgrent
/bin/cppcheck.exe: getgrent
/bin/cygdrop.exe: _getgrent32
/bin/emacs-nox.exe: _getgrent32
/bin/emacs-w32.exe: _getgrent32
/bin/emacs-X11.exe: _getgrent32
/bin/filan.exe: _getgrent32
/bin/mc.exe: _getgrent32
/bin/nedit.exe: 	library functions:"<((?# arithmetic functions)abs|atan2|cos|exp|int|log|rand|sin|sqrt|srand|time|(?# conversion functions)chr|gmtime|hex|localtime|oct|ord|vec|(?# structure conversion)pack|unpack|(?# string functions)chomp|chop|crypt|eval(?=\s*[^{])|index|lc|lcfirst|length|quotemeta|rindex|substr|uc|ucfirst|(?# array and hash functions)delete|each|exists|grep|join|keys|map|pop|push|reverse|scalar|shift|sort|splice|split|unshift|values|(?# search and replace functions)pos|study|(?# file operations)chmod|chown|link|lstat|mkdir|readlink|rename|rmdir|stat|symlink|truncate|unlink|utime|(?# input/output)binmode|close|eof|fcntl|fileno|flock|getc|ioctl|open|pipe|print|printf|read|readline|readpipe|seek|select|sprintf|sysopen|sysread|sysseek|syswrite|tell|(?# formats)formline|write|(?# tying variables)tie|tied|untie|(?# directory reading routines)closedir|opendir|readdir|rewinddir|seekdir|telldir|(?# system interaction)alarm|chdir|chroot|die|exec|exit|fork|getlogin|getpgrp|getppid|getpriority|glob|kill|setpgrp|setpriority|sleep|syscall|system|times|umask|wait|waitpid|warn|(?# networking)accept|bind|connect|getpeername|getsockname|getsockopt|listen|recv|send|setsockopt|shutdown|socket|socketpair|(?# system V ipc)msgctl|msgget|msgrcv|msgsnd|semctl|semget|semop|shmctl|shmget|shmread|shmwrite|(?# miscellaneous)defined|do|dump|eval(?=\s*\{)|local|my|ref|reset|undef|(?# informations from system databases)endpwent|getpwent|getpwnam|getpwuid|setpwent|endgrent|getgrent|getgrgid|getgrnam|setgrent|endnetent|getnetbyaddr|getnetbyname|getnetent|setnetent|endhostend|gethostbyaddr|gethostbyname|gethostent|sethostent|endservent|getservbyname|getservbyport|getservent|setservent|endprotoent|getprotobyname|getprotobynumber|getprotoent|setprotoent)>":::Subroutine::
/bin/procan.exe: _getgrent32
/bin/sh.exe: _getgrent32
/bin/socat.exe: _getgrent32
/bin/tcsh.exe: _getgrent32
/bin/xgettext.exe: getgrent
/bin/ytree.exe: _getgrent32
/bin/cygguile-12.dll: _getgrent32
/bin/cygguile-12.dll: _getgrent32
/bin/cygguile-17.dll: _getgrent32
/bin/cygobt-2.dll: _getgrent32
/bin/cygoctave-1.dll: _ZN12octave_group8getgrentERSs
/bin/cygoctave-1.dll: _ZN12octave_group8getgrentEv
/bin/cygoctave-1.dll: _getgrent32
/bin/cygperl5_14.dll: getgrent
/bin/cygperl5_14.dll: _getgrent32
/bin/cygqscintilla2-9.dll: NULL __FILE__ __LINE__ __PACKAGE__ __DATA__ __END__ AUTOLOAD BEGIN CORE DESTROY END EQ GE GT INIT LE LT NE CHECK abs accept alarm and atan2 bind binmode bless caller chdir chmod chomp chop chown chr chroot close closedir cmp connect continue cos crypt dbmclose dbmopen defined delete die do dump each else elsif endgrent endhostent endnetent endprotoent endpwent endservent eof eq eval exec exists exit exp fcntl fileno flock for foreach fork format formline ge getc getgrent getgrgid getgrnam gethostbyaddr gethostbyname gethostent getlogin getnetbyaddr getnetbyname getnetent getpeername getpgrp getppid getpriority getprotobyname getprotobynumber getprotoent getpwent getpwnam getpwuid getservbyname getservbyport getservent getsockname getsockopt glob gmtime goto grep gt hex if index int ioctl join keys kill last lc lcfirst le length link listen local localtime lock log lstat lt m map mkdir msgctl msgget msgrcv msgsnd my ne next no not oct open opendir or ord our pack package pipe pop pos print printf prototype push q qq qr quotemeta qu qw qx rand read readdir readline readlink readpipe recv redo ref rename require reset return reverse rewinddir rindex rmdir s scalar seek seekdir select semctl semget semop send setgrent sethostent setnetent setpgrp setpriority setprotoent setpwent setservent setsockopt shift shmctl shmget shmread shmwrite shutdown sin sleep socket socketpair sort splice split sprintf sqrt srand stat study sub substr symlink syscall sysopen sysread sysseek system syswrite tell telldir tie tied time times tr truncate uc ucfirst umask undef unless unlink unpack unshift untie until use utime values vec wait waitpid wantarray warn while write x xor y
/bin/cygwin1.dll: _getgrent
/bin/cygwin1.dll: _getgrent32
/bin/cygwin1.dll: getgrent

[-- Attachment #3: cygwin64.txt --]
[-- Type: text/plain, Size: 4194 bytes --]

/bin/bash.exe: getgrent
/bin/cppcheck.exe: fgetgrent
/bin/cppcheck.exe: getgrent
/bin/cygdrop.exe: getgrent
/bin/emacs-nox.exe: getgrent
/bin/emacs-w32.exe: getgrent
/bin/emacs-X11.exe: getgrent
/bin/filan.exe: getgrent
/bin/mc.exe: getgrent
/bin/nedit.exe: 	library functions:"<((?# arithmetic functions)abs|atan2|cos|exp|int|log|rand|sin|sqrt|srand|time|(?# conversion functions)chr|gmtime|hex|localtime|oct|ord|vec|(?# structure conversion)pack|unpack|(?# string functions)chomp|chop|crypt|eval(?=\s*[^{])|index|lc|lcfirst|length|quotemeta|rindex|substr|uc|ucfirst|(?# array and hash functions)delete|each|exists|grep|join|keys|map|pop|push|reverse|scalar|shift|sort|splice|split|unshift|values|(?# search and replace functions)pos|study|(?# file operations)chmod|chown|link|lstat|mkdir|readlink|rename|rmdir|stat|symlink|truncate|unlink|utime|(?# input/output)binmode|close|eof|fcntl|fileno|flock|getc|ioctl|open|pipe|print|printf|read|readline|readpipe|seek|select|sprintf|sysopen|sysread|sysseek|syswrite|tell|(?# formats)formline|write|(?# tying variables)tie|tied|untie|(?# directory reading routines)closedir|opendir|readdir|rewinddir|seekdir|telldir|(?# system interaction)alarm|chdir|chroot|die|exec|exit|fork|getlogin|getpgrp|getppid|getpriority|glob|kill|setpgrp|setpriority|sleep|syscall|system|times|umask|wait|waitpid|warn|(?# networking)accept|bind|connect|getpeername|getsockname|getsockopt|listen|recv|send|setsockopt|shutdown|socket|socketpair|(?# system V ipc)msgctl|msgget|msgrcv|msgsnd|semctl|semget|semop|shmctl|shmget|shmread|shmwrite|(?# miscellaneous)defined|do|dump|eval(?=\s*\{)|local|my|ref|reset|undef|(?# informations from system databases)endpwent|getpwent|getpwnam|getpwuid|setpwent|endgrent|getgrent|getgrgid|getgrnam|setgrent|endnetent|getnetbyaddr|getnetbyname|getnetent|setnetent|endhostend|gethostbyaddr|gethostbyname|gethostent|sethostent|endservent|getservbyname|getservbyport|getservent|setservent|endprotoent|getprotobyname|getprotobynumber|getprotoent|setprotoent)>":::Subroutine::
/bin/parl.exe: getgrent
/bin/parl.exe: getgrent
/bin/procan.exe: getgrent
/bin/sh.exe: getgrent
/bin/socat.exe: getgrent
/bin/tcsh.exe: getgrent
/bin/xgettext.exe: getgrent
/bin/cygguile-17.dll: getgrent
/bin/cygobt-2.dll: getgrent
/bin/cygoctave-1.dll: _ZN12octave_group8getgrentERSs
/bin/cygoctave-1.dll: _ZN12octave_group8getgrentEv
/bin/cygoctave-1.dll: getgrent
/bin/cygperl5_14.dll: getgrent
/bin/cygperl5_14.dll: getgrent
/bin/cygqscintilla2-9.dll: NULL __FILE__ __LINE__ __PACKAGE__ __DATA__ __END__ AUTOLOAD BEGIN CORE DESTROY END EQ GE GT INIT LE LT NE CHECK abs accept alarm and atan2 bind binmode bless caller chdir chmod chomp chop chown chr chroot close closedir cmp connect continue cos crypt dbmclose dbmopen defined delete die do dump each else elsif endgrent endhostent endnetent endprotoent endpwent endservent eof eq eval exec exists exit exp fcntl fileno flock for foreach fork format formline ge getc getgrent getgrgid getgrnam gethostbyaddr gethostbyname gethostent getlogin getnetbyaddr getnetbyname getnetent getpeername getpgrp getppid getpriority getprotobyname getprotobynumber getprotoent getpwent getpwnam getpwuid getservbyname getservbyport getservent getsockname getsockopt glob gmtime goto grep gt hex if index int ioctl join keys kill last lc lcfirst le length link listen local localtime lock log lstat lt m map mkdir msgctl msgget msgrcv msgsnd my ne next no not oct open opendir or ord our pack package pipe pop pos print printf prototype push q qq qr quotemeta qu qw qx rand read readdir readline readlink readpipe recv redo ref rename require reset return reverse rewinddir rindex rmdir s scalar seek seekdir select semctl semget semop send setgrent sethostent setnetent setpgrp setpriority setprotoent setpwent setservent setsockopt shift shmctl shmget shmread shmwrite shutdown sin sleep socket socketpair sort splice split sprintf sqrt srand stat study sub substr symlink syscall sysopen sysread sysseek system syswrite tell telldir tie tied time times tr truncate uc ucfirst umask undef unless unlink unpack unshift untie until use utime values vec wait waitpid wantarray warn while write x xor y
/bin/cygwin1.dll: getgrent

[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/7/2014 06:53, David Stacey wrote:
> On 07/02/2014 09:49, Corinna Vinschen wrote:
>> On Feb  6 14:43, Warren Young wrote:
>>> I know a guy who currently has all of
>>> Cygwin downloaded and ready to re-install, to test this.:)
>> Try this:  strings -f/bin/*.exe/bin/*.dll | grep getgrent
>
> Let me save you the trouble of reinstalling it

Thank you!

 > Results attached.

The cppcheck, cygqscintilla*.dll and nedit results are bogus. (i.e. They 
contain these strings for other reasons.)  Here's a better check that 
doesn't give false positives:

     $ cat <<END > checkfile
     #!/bin/sh
     if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
     END
     $ find /bin -name \*.exe -exec ./checkfile {} \;

cygperl*.dll includes the functions only because it wraps them.  I 
assume Perl's configuration script is smart enough to cope if they're 
missing.

I assume the shells (bash, mc, and tcsh) also have alternate code paths 
for systems where these functions don't exist.

ytree's use of these functions is obvious.  I don't know that I'd bet on 
it having an alternate code path, though.

I have no guess why socat needs these functions, but given that it was 
recently updated, I will guess the socat devs aren't interested in 
moving to more modern APIs.  Programs like this, where user management 
isn't any kind of real focus, are where you expect to find old, easy 
APIs used in preference to more modern, more complicated APIs.

cygwin1.dll is obvious. :)

Emacs...why am I not surprised that it uses these old functions?

Vim does not, therefore Vim is the superior text editor.  (I kid.)


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

On Feb  7 10:51, Warren Young wrote:
> On 2/7/2014 06:53, David Stacey wrote:
> >On 07/02/2014 09:49, Corinna Vinschen wrote:
> >>On Feb  6 14:43, Warren Young wrote:
> >>>I know a guy who currently has all of
> >>>Cygwin downloaded and ready to re-install, to test this.:)
> >>Try this:  strings -f/bin/*.exe/bin/*.dll | grep getgrent
> >
> >Let me save you the trouble of reinstalling it
> 
> Thank you!
> 
> > Results attached.
> 
> The cppcheck, cygqscintilla*.dll and nedit results are bogus. (i.e.
> They contain these strings for other reasons.)  Here's a better
> check that doesn't give false positives:
> 
>     $ cat <<END > checkfile
>     #!/bin/sh
>     if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
>     END
>     $ find /bin -name \*.exe -exec ./checkfile {} \;
> 
> cygperl*.dll includes the functions only because it wraps them.  I
> assume Perl's configuration script is smart enough to cope if
> they're missing.

Just to clarify:  The entry points won't go away, but they might be
*sort of* non-functional.  They will, for a start, only enumerate
the accounts which are cached in the process, which may be only
two (current user and her primary group), or none at all.

> Emacs...why am I not surprised that it uses these old functions?
> 
> Vim does not, therefore Vim is the superior text editor.  (I kid.)

You dont ;)


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On Feb 7, 2014, at 10:51 AM, Warren Young <warren@etr-usa.com> wrote:

> Here's a better check that doesn't give false positives:
> 
>    $ cat <<END > checkfile
>    #!/bin/sh
>    if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
>    END
>    $ find /bin -name \*.exe -exec ./checkfile {} \;

The strings(1) call got left out of that test.  Here’s a working variant, which tests for getpwent() calls instead:

    $ cat <<END > checkfile
    if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
    END
    $ find /bin -name \*.exe -exec sh checkfile {} \;

Alas, Vim *does* call this API.  Results:

/bin/ali.exe
/bin/biber.exe
/bin/cppcheck.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/emacs-X11.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/mc.exe
/bin/nano.exe
/bin/otp.exe
/bin/pure-pwconvert.exe
/bin/tcsh.exe
/bin/vim.exe
/bin/xdvi-xaw.exe
/bin/xemacs-21.4.22.exe
/bin/xgettext.exe
/bin/ytree.exe

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]

On Feb  9 08:45, Warren Young wrote:
> On Feb 7, 2014, at 10:51 AM, Warren Young <warren@etr-usa.com> wrote:
> 
> > Here's a better check that doesn't give false positives:
> > 
> >    $ cat <<END > checkfile
> >    #!/bin/sh
> >    if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
> >    END
> >    $ find /bin -name \*.exe -exec ./checkfile {} \;
> 
> The strings(1) call got left out of that test.  Here’s a working variant, which tests for getpwent() calls instead:
> 
>     $ cat <<END > checkfile
>     if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
>     END
>     $ find /bin -name \*.exe -exec sh checkfile {} \;
> 
> Alas, Vim *does* call this API.  Results:

It does so to evaluate a ~user in a path and I guess that's what other
apps do as well.  This is kind of weird since it would obviously not be
necessary to enumerate accounts for this functionality:

  if (~ in path)
    if (~ is solo)
      getpwuid(getuid ())
    else
      getpwname (username after the ~);


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

On Feb  9 17:10, Corinna Vinschen wrote:
> On Feb  9 08:45, Warren Young wrote:
> > On Feb 7, 2014, at 10:51 AM, Warren Young <warren@etr-usa.com> wrote:
> > 
> > > Here's a better check that doesn't give false positives:
> > > 
> > >    $ cat <<END > checkfile
> > >    #!/bin/sh
> > >    if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
> > >    END
> > >    $ find /bin -name \*.exe -exec ./checkfile {} \;
> > 
> > The strings(1) call got left out of that test.  Here’s a working variant, which tests for getpwent() calls instead:
> > 
> >     $ cat <<END > checkfile
> >     if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
> >     END
> >     $ find /bin -name \*.exe -exec sh checkfile {} \;
> > 
> > Alas, Vim *does* call this API.  Results:
> 
> It does so to evaluate a ~user in a path and I guess that's what other
> apps do as well.  This is kind of weird since it would obviously not be
> necessary to enumerate accounts for this functionality:
> 
>   if (~ in path)
>     if (~ is solo)
>       getpwuid(getuid ())
>     else
>       getpwname (username after the ~);

Huh, even though tcsh calls getpwent as well, this works fine:

  $ cd ~foo

with foo being some not yet cached user name.

However, with my current code this:

  $ cd ~<TAB>

only returns the cached user entries.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Ken Brown]

On 2/9/2014 11:16 AM, Corinna Vinschen wrote:
> On Feb  9 17:10, Corinna Vinschen wrote:
>> On Feb  9 08:45, Warren Young wrote:
>>> On Feb 7, 2014, at 10:51 AM, Warren Young <warren@etr-usa.com> wrote:
>>>
>>>> Here's a better check that doesn't give false positives:
>>>>
>>>>     $ cat <<END > checkfile
>>>>     #!/bin/sh
>>>>     if egrep -q '_getgrent(32|64)' "$1" ; then echo $1 ; fi
>>>>     END
>>>>     $ find /bin -name \*.exe -exec ./checkfile {} \;
>>>
>>> The strings(1) call got left out of that test.  Here’s a working variant, which tests for getpwent() calls instead:
>>>
>>>      $ cat <<END > checkfile
>>>      if strings "$1" | egrep -q '^getpwent$' ; then echo $1 ; fi
>>>      END
>>>      $ find /bin -name \*.exe -exec sh checkfile {} \;
>>>
>>> Alas, Vim *does* call this API.  Results:
>>
>> It does so to evaluate a ~user in a path and I guess that's what other
>> apps do as well.  This is kind of weird since it would obviously not be
>> necessary to enumerate accounts for this functionality:
>>
>>    if (~ in path)
>>      if (~ is solo)
>>        getpwuid(getuid ())
>>      else
>>        getpwname (username after the ~);
>
> Huh, even though tcsh calls getpwent as well, this works fine:
>
>    $ cd ~foo
>
> with foo being some not yet cached user name.
>
> However, with my current code this:
>
>    $ cd ~<TAB>
>
> only returns the cached user entries.

emacs uses getpwent (on systems that provide it) for a similar reason: 
to provide tab completion on commands that expect a user name.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[David Stacey]

On 09/02/2014 15:45, Warren Young wrote:
> Results:
>
> /bin/cppcheck.exe

This intrigued me. cppcheck is a static analyser, so what's it doing 
with getpwent()? I had a nosy in the source code, and it appears that 
cppcheck has a rule checking for POSIX calls that are not re-entrant. If 
a call to getpwent() is found in your source code, cppcheck will 
recommend that you replace this with getpwent_r().

As far as I can tell, cppcheck doesn't actually call getpwent() at all; 
this is a false positive turned up by strings(1).

Dave.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On Feb 9, 2014, at 9:37 AM, David Stacey <drstacey@tiscali.co.uk> wrote:

> On 09/02/2014 15:45, Warren Young wrote:
>> Results:
>> 
>> /bin/cppcheck.exe
> 
> As far as I can tell, cppcheck doesn't actually call getpwent() at all; this is a false positive turned up by strings(1).

Yeah, there *has* to be a better way than strings(1) to extract an EXE's list of DLL imports.  The linker does it somehow...
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Peter Rosin]

On 2014-02-10 10:02, Warren Young wrote:
> On Feb 9, 2014, at 9:37 AM, David Stacey <drstacey@tiscali.co.uk> wrote:
> 
>> On 09/02/2014 15:45, Warren Young wrote:
>>> Results:
>>>
>>> /bin/cppcheck.exe
>>
>> As far as I can tell, cppcheck doesn't actually call getpwent() at all; this is a false positive turned up by strings(1).
> 
> Yeah, there *has* to be a better way than strings(1) to extract an EXE's list of DLL imports.  The linker does it somehow...

objdump -x /bin/foo.exe

Cheers,
Peter


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/10/2014 04:16, Peter Rosin wrote:
> On 2014-02-10 10:02, Warren Young wrote:
>>
>> there *has* to be a better way than strings(1) to extract an EXE's list of DLL imports.
>
> objdump -x /bin/foo.exe

Thank you!

-x turns on 6 other flags, the only one of which that really matters 
here is -p.  The output is complex enough that I decided to write a 
better parser than a grep call.  Here's my new checkfile script:


#!/usr/bin/perl -w

my ($exe, $symbol) = @ARGV;
my $in_cygdll = 0;

die "usage: $0 exename symbol\n" unless length($symbol);
open my $dump, '-|', "objdump -p '$exe'" or die "Can't dump $exe: $!\n";

while (<$dump>) {
     if (m/DLL Name: cygwin1.dll/) {
         $in_cygdll = 1;
     }
     elsif (m/DLL Name: /) {
         last;   # Last cygwin1.dll symbol found; on to another DLL
     }
     elsif ($in_cygdll) {
         my @parts = split;
         if (@parts == 3 and $parts[2] eq $symbol) {
             print "$exe\n";
             last;
         }
     }
}


Run it like before, except that it takes the import name to search for 
as a second parameter now:

     $ find /bin -name \*.exe -exec ./checkfile {} getpwent \;

I don't have my "almost everything" Cygwin install here to run it 
against, so unless someone beats me to it, I won't be posting results 
for many hours at least.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[David Stacey]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

On 10/02/2014 19:49, Warren Young wrote:
> I don't have my "almost everything" Cygwin install here to run it 
> against, so unless someone beats me to it, I won't be posting results 
> for many hours at least.

Delighted to oblige. I ran your perl script on all executables and DLLs 
in /bin. Results attached.

Dave.


[-- Attachment #2: cygwin32.txt --]
[-- Type: text/plain, Size: 294 bytes --]

/bin/ali.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/lefty.exe
/bin/nano.exe
/bin/pure-pwconvert.exe
/bin/vim.exe
/bin/ytree.exe
/bin/cygedit-0.dll
/bin/cygreadline7.dll
/bin/libpython2.7.dll
/bin/libpython3.2m.dll
/bin/libzsh-5.0.2.dll

[-- Attachment #3: cygwin64.txt --]
[-- Type: text/plain, Size: 249 bytes --]

/bin/ali.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/lefty.exe
/bin/mc.exe
/bin/nano.exe
/bin/vim.exe
/bin/cygreadline7.dll
/bin/libpython2.7.dll
/bin/libpython3.2m.dll
/bin/libzsh-5.0.2.dll


[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, David Stacey!

>> I don't have my "almost everything" Cygwin install here to run it
>> against, so unless someone beats me to it, I won't be posting results 
>> for many hours at least.

> Delighted to oblige. I ran your perl script on all executables and DLLs 
> in /bin. Results attached.

Curious that mc is missing from 32-bit list.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 11.02.2014, <06:24>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[David Stacey]

[-- Attachment #1: Type: text/plain, Size: 2002 bytes --]

On 11/02/2014 02:25, Andrey Repin wrote:
> Greetings, David Stacey!

Greetings, Andrey Repin! (I've wanted to type that for such a long time...)

>
>>>   I don't have my "almost everything" Cygwin install here to run it
>>>   against, so unless someone beats me to it, I won't be posting results
>>>   for many hours at least.
>>   Delighted to oblige. I ran your perl script on all executables and DLLs
>>   in /bin. Results attached.
> Curious that mc is missing from 32-bit list.

You are quite correct. I've looked at the source code for Midnight 
Commander, and getpwent() is called in three different places. The 
problem was an assumption made in the 'checkfile' perl script: it was 
assumed that cygwin1.dll is the first DLL listed by objdump. For mc.exe, 
this isn't the case - cyggcc_s-1.dll is listed first, with cygwin1.dll 
second.

I made a quick amendment to said perl script to cater for this, and the 
new results are attached. The differences between the two lists are 
mainly due to packages not being available for 64-bit Cygwin (e.g. 
xemacs, ytree). Guile is worthy of a mention: cygguile-12.dll is in the 
32-bit list, but there is no equivalent in the 64-bit list. This is 
probably because the 64-bit version of guile is at a later version 
compared to its 32-bit cousin.

Hopefully third time lucky!

Dave.

PS: For reference, the revised perl script is as follows:

#!/usr/bin/perl -w

my ($exe, $symbol) = @ARGV;
my $in_cygdll = 0;

die "usage: $0 exename symbol\n" unless length($symbol);
open my $dump, '-|', "objdump -p '$exe'" or die "Can't dump $exe: $!\n";

while (<$dump>) {
     if (m/DLL Name: cygwin1.dll/) {
         $in_cygdll = 1;
     }
     elsif (m/DLL Name: /) {
         if ($in_cygdll) {
             last;   # Last cygwin1.dll symbol found; on to another DLL
         }
     }
     elsif ($in_cygdll) {
         my @parts = split;
         if (@parts == 3 and $parts[2] eq $symbol) {
             print "$exe\n";
             last;
         }
     }
}



[-- Attachment #2: cygwin32.txt --]
[-- Type: text/plain, Size: 591 bytes --]

/bin/ali.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/emacs-X11.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/lefty.exe
/bin/mc.exe
/bin/nano.exe
/bin/otp.exe
/bin/pure-pwconvert.exe
/bin/tcsh.exe
/bin/vim.exe
/bin/xdvi-xaw.exe
/bin/xemacs-21.4.22.exe
/bin/ytree.exe
/bin/cygedit-0.dll
/bin/cyggtk-1-2-0.dll
/bin/cyggtk-x11-2.0-0.dll
/bin/cygguile-12.dll
/bin/cygguile-17.dll
/bin/cyggutils-1.dll
/bin/cygoctave-1.dll
/bin/cygperl5_14.dll
/bin/cygreadline5.dll
/bin/cygreadline6.dll
/bin/cygreadline7.dll
/bin/libpython2.7.dll
/bin/libpython3.2m.dll
/bin/libzsh-5.0.2.dll

[-- Attachment #3: cygwin64.txt --]
[-- Type: text/plain, Size: 464 bytes --]

/bin/ali.exe
/bin/emacs-nox.exe
/bin/emacs-w32.exe
/bin/emacs-X11.exe
/bin/fish.exe
/bin/gdb.exe
/bin/gvim.exe
/bin/joe.exe
/bin/lefty.exe
/bin/mc.exe
/bin/nano.exe
/bin/otp.exe
/bin/tcsh.exe
/bin/vim.exe
/bin/xdvi-xaw.exe
/bin/cygedit-0.dll
/bin/cyggtk-1-2-0.dll
/bin/cyggtk-x11-2.0-0.dll
/bin/cygguile-17.dll
/bin/cyggutils-1.dll
/bin/cygoctave-1.dll
/bin/cygperl5_14.dll
/bin/cygreadline7.dll
/bin/libpython2.7.dll
/bin/libpython3.2m.dll
/bin/libzsh-5.0.2.dll


[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/11/2014 16:25, David Stacey wrote:
> getpwent() is called in three different places.

To those of you who have investigated these code paths: do any of them 
look like they couldn't be replaced by getpwnam() or other calls that 
would let cygwin1.dll do single-record AD/SAM lookups, rather than 
whole-table/tree scans?

That is, do any of these programs really need to visit every record in 
/etc/passwd?

> problem was an assumption made in the 'checkfile' perl script: it was
> assumed that cygwin1.dll is the first DLL listed by objdump.

Details, details. :)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

On 02/11/2014 05:06 PM, Warren Young wrote:
> On 2/11/2014 16:25, David Stacey wrote:
>> getpwent() is called in three different places.
> 
> To those of you who have investigated these code paths: do any of them
> look like they couldn't be replaced by getpwnam() or other calls that
> would let cygwin1.dll do single-record AD/SAM lookups, rather than
> whole-table/tree scans?
> 
> That is, do any of these programs really need to visit every record in
> /etc/passwd?

libreadline wants to know how to tab-complete ~foo; to do that, it has
to find all usernames beginning with foo.  How would you do that without
visiting every single record?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On Feb 11 19:06, Eric Blake wrote:
> On 02/11/2014 05:06 PM, Warren Young wrote:
> > On 2/11/2014 16:25, David Stacey wrote:
> >> getpwent() is called in three different places.
> > 
> > To those of you who have investigated these code paths: do any of them
> > look like they couldn't be replaced by getpwnam() or other calls that
> > would let cygwin1.dll do single-record AD/SAM lookups, rather than
> > whole-table/tree scans?
> > 
> > That is, do any of these programs really need to visit every record in
> > /etc/passwd?
> 
> libreadline wants to know how to tab-complete ~foo; to do that, it has
> to find all usernames beginning with foo.  How would you do that without
> visiting every single record?

This seems to be the major usage of getpwent these days.  The question
is, how bad is it if only a handful entries, or even only a single one
(of oneself) show up?

Either way, implementing a full getpwent requires to return the local
users, the users of the primary domain, and the users of all trusted
domains.  I know of domains with 200K users and there are probably
bigger ones.  How long should a search take when a user presses <TAB>
after the ~?  And then, shall the process running the getpwent actually
cache all of them?  This seems really excessive.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Corinna Vinschen!

> Either way, implementing a full getpwent requires to return the local
> users, the users of the primary domain, and the users of all trusted
> domains.  I know of domains with 200K users and there are probably
> bigger ones.  How long should a search take when a user presses <TAB>
> after the ~?  And then, shall the process running the getpwent actually
> cache all of them?  This seems really excessive.

IMO, such practice should be actually discouraged. I mean, listing ALL users
of ALL... well, you got it. For the reasons you mentioned above - the possible
results of such operation are largely unpredictable.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 12.02.2014, <15:49>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Richard]

On Wed, 12 Feb 2014, Corinna Vinschen wrote:
> On Feb 11 19:06, Eric Blake wrote:
>> On 02/11/2014 05:06 PM, Warren Young wrote:
>>> On 2/11/2014 16:25, David Stacey wrote:
>>>> getpwent() is called in three different places.
>>>
>>> To those of you who have investigated these code paths: do any of them
>>> look like they couldn't be replaced by getpwnam() or other calls that
>>> would let cygwin1.dll do single-record AD/SAM lookups, rather than
>>> whole-table/tree scans?
>>>
>>> That is, do any of these programs really need to visit every record in
>>> /etc/passwd?
>>
>> libreadline wants to know how to tab-complete ~foo; to do that, it has
>> to find all usernames beginning with foo.  How would you do that without
>> visiting every single record?
>
> This seems to be the major usage of getpwent these days.  The question
> is, how bad is it if only a handful entries, or even only a single one
> (of oneself) show up?
>
> Either way, implementing a full getpwent requires to return the local
> users, the users of the primary domain, and the users of all trusted
> domains.  I know of domains with 200K users and there are probably
> bigger ones.  How long should a search take when a user presses <TAB>
> after the ~?  And then, shall the process running the getpwent actually
> cache all of them?  This seems really excessive.
>

Not only is it excessive, it's a massive security hole. ...I don't know 
why the present crop of geniuses don't know one of the most fundamental 
rules of security: don't give up usernames. (Yet, they do it all the 
time.)

I vote get rid of the damned thing - not that my vote counts or that this 
is the place for it anyway!

Richard

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Ken Brown]

On 2/12/2014 4:08 AM, Corinna Vinschen wrote:
> On Feb 11 19:06, Eric Blake wrote:
>> On 02/11/2014 05:06 PM, Warren Young wrote:
>>> On 2/11/2014 16:25, David Stacey wrote:
>>>> getpwent() is called in three different places.
>>>
>>> To those of you who have investigated these code paths: do any of them
>>> look like they couldn't be replaced by getpwnam() or other calls that
>>> would let cygwin1.dll do single-record AD/SAM lookups, rather than
>>> whole-table/tree scans?
>>>
>>> That is, do any of these programs really need to visit every record in
>>> /etc/passwd?
>>
>> libreadline wants to know how to tab-complete ~foo; to do that, it has
>> to find all usernames beginning with foo.  How would you do that without
>> visiting every single record?
>
> This seems to be the major usage of getpwent these days.  The question
> is, how bad is it if only a handful entries, or even only a single one
> (of oneself) show up?
>
> Either way, implementing a full getpwent requires to return the local
> users, the users of the primary domain, and the users of all trusted
> domains.  I know of domains with 200K users and there are probably
> bigger ones.  How long should a search take when a user presses <TAB>
> after the ~?  And then, shall the process running the getpwent actually
> cache all of them?  This seems really excessive.

What about the following compromise:  If /etc/passwd exists, then 
getpwent behaves as it does currently.  Otherwise, it returns a handful 
of entries, or possibly just the current user.  This gives users a 
choice.  If tab-completion in this situation is important to them, they 
can keep their /etc/passwd file.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Richard]

On Wed, 12 Feb 2014, Ken Brown wrote:
> On 2/12/2014 4:08 AM, Corinna Vinschen wrote:
>> On Feb 11 19:06, Eric Blake wrote:
>>> On 02/11/2014 05:06 PM, Warren Young wrote:
>>>> On 2/11/2014 16:25, David Stacey wrote:
>>>>> getpwent() is called in three different places.
>>>> 
>>>> To those of you who have investigated these code paths: do any of them
>>>> look like they couldn't be replaced by getpwnam() or other calls that
>>>> would let cygwin1.dll do single-record AD/SAM lookups, rather than
>>>> whole-table/tree scans?
>>>> 
>>>> That is, do any of these programs really need to visit every record in
>>>> /etc/passwd?
>>> 
>>> libreadline wants to know how to tab-complete ~foo; to do that, it has
>>> to find all usernames beginning with foo.  How would you do that without
>>> visiting every single record?
>> 
>> This seems to be the major usage of getpwent these days.  The question
>> is, how bad is it if only a handful entries, or even only a single one
>> (of oneself) show up?
>> 
>> Either way, implementing a full getpwent requires to return the local
>> users, the users of the primary domain, and the users of all trusted
>> domains.  I know of domains with 200K users and there are probably
>> bigger ones.  How long should a search take when a user presses <TAB>
>> after the ~?  And then, shall the process running the getpwent actually
>> cache all of them?  This seems really excessive.
>
> What about the following compromise:  If /etc/passwd exists, then getpwent 
> behaves as it does currently.  Otherwise, it returns a handful of entries, or 
> possibly just the current user.  This gives users a choice.  If 
> tab-completion in this situation is important to them, they can keep their 
> /etc/passwd file.

Works for me! And I'd vote "just the current user" in lieu of "a handful" 
because even a handful has a huge overhead associated with it.

Richard

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Ken Brown!

> What about the following compromise:  If /etc/passwd exists, then
> getpwent behaves as it does currently.  Otherwise, it returns a handful 
> of entries, or possibly just the current user.  This gives users a 
> choice.  If tab-completion in this situation is important to them, they 
> can keep their /etc/passwd file.

Makes sense. You get your current user name substituted in all cases, if you
want more - you fill in the list of the names you want to see.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 12.02.2014, <21:03>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]

On Feb 12 11:16, Ken Brown wrote:
> On 2/12/2014 4:08 AM, Corinna Vinschen wrote:
> >On Feb 11 19:06, Eric Blake wrote:
> >>On 02/11/2014 05:06 PM, Warren Young wrote:
> >>>On 2/11/2014 16:25, David Stacey wrote:
> >>>>getpwent() is called in three different places.
> >>>
> >>>To those of you who have investigated these code paths: do any of them
> >>>look like they couldn't be replaced by getpwnam() or other calls that
> >>>would let cygwin1.dll do single-record AD/SAM lookups, rather than
> >>>whole-table/tree scans?
> >>>
> >>>That is, do any of these programs really need to visit every record in
> >>>/etc/passwd?
> >>
> >>libreadline wants to know how to tab-complete ~foo; to do that, it has
> >>to find all usernames beginning with foo.  How would you do that without
> >>visiting every single record?
> >
> >This seems to be the major usage of getpwent these days.  The question
> >is, how bad is it if only a handful entries, or even only a single one
> >(of oneself) show up?
> >
> >Either way, implementing a full getpwent requires to return the local
> >users, the users of the primary domain, and the users of all trusted
> >domains.  I know of domains with 200K users and there are probably
> >bigger ones.  How long should a search take when a user presses <TAB>
> >after the ~?  And then, shall the process running the getpwent actually
> >cache all of them?  This seems really excessive.
> 
> What about the following compromise:  If /etc/passwd exists, then
> getpwent behaves as it does currently.

This part is relatively easy to implement.

> Otherwise, it returns a
> handful of entries, or possibly just the current user.

The handful entries would be the ones the process has cached at that
point in time.  The tricky part is that getpwent would have to keep
track which entries from the file are in the cache so that those are not
accidentally enumerated twice.

> This gives
> users a choice.  If tab-completion in this situation is important to
> them, they can keep their /etc/passwd file.

There's only one tiny problem.  Whatever I think about the full
enumerate being right or wrong, I have this vague feeling that I'd like
to have this implemented fully at one point.  My cat disapproves, but we
can't agree on everything, I guess.  Another configuration option in
/etc/nsswitch.conf might comfort her.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Christopher Faylor]

On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote:
>On Feb 12 11:16, Ken Brown wrote:
>> On 2/12/2014 4:08 AM, Corinna Vinschen wrote:
>> >On Feb 11 19:06, Eric Blake wrote:
>> >>On 02/11/2014 05:06 PM, Warren Young wrote:
>> >>>On 2/11/2014 16:25, David Stacey wrote:
>> >>>>getpwent() is called in three different places.
>> >>>
>> >>>To those of you who have investigated these code paths: do any of them
>> >>>look like they couldn't be replaced by getpwnam() or other calls that
>> >>>would let cygwin1.dll do single-record AD/SAM lookups, rather than
>> >>>whole-table/tree scans?
>> >>>
>> >>>That is, do any of these programs really need to visit every record in
>> >>>/etc/passwd?
>> >>
>> >>libreadline wants to know how to tab-complete ~foo; to do that, it has
>> >>to find all usernames beginning with foo.  How would you do that without
>> >>visiting every single record?
>> >
>> >This seems to be the major usage of getpwent these days.  The question
>> >is, how bad is it if only a handful entries, or even only a single one
>> >(of oneself) show up?
>> >
>> >Either way, implementing a full getpwent requires to return the local
>> >users, the users of the primary domain, and the users of all trusted
>> >domains.  I know of domains with 200K users and there are probably
>> >bigger ones.  How long should a search take when a user presses <TAB>
>> >after the ~?  And then, shall the process running the getpwent actually
>> >cache all of them?  This seems really excessive.
>> 
>> What about the following compromise:  If /etc/passwd exists, then
>> getpwent behaves as it does currently.
>
>This part is relatively easy to implement.
>
>> Otherwise, it returns a
>> handful of entries, or possibly just the current user.
>
>The handful entries would be the ones the process has cached at that
>point in time.  The tricky part is that getpwent would have to keep
>track which entries from the file are in the cache so that those are not
>accidentally enumerated twice.
>
>> This gives
>> users a choice.  If tab-completion in this situation is important to
>> them, they can keep their /etc/passwd file.
>
>There's only one tiny problem.  Whatever I think about the full
>enumerate being right or wrong, I have this vague feeling that I'd like
>to have this implemented fully at one point.  My cat disapproves, but we
>can't agree on everything, I guess.  Another configuration option in
>/etc/nsswitch.conf might comfort her.

I don't know if this has been mentioned but would a cache help here,
i.e., nscd?  I think that's how Linux deals with this type of situation.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Feb 12 16:37, Christopher Faylor wrote:
> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote:
> >There's only one tiny problem.  Whatever I think about the full
> >enumerate being right or wrong, I have this vague feeling that I'd like
> >to have this implemented fully at one point.  My cat disapproves, but we
> >can't agree on everything, I guess.  Another configuration option in
> >/etc/nsswitch.conf might comfort her.
> 
> I don't know if this has been mentioned but would a cache help here,
> i.e., nscd?  I think that's how Linux deals with this type of situation.

Caching is wonderful for the usual requests for single entries from the
DB, and for this we have already two caches, the LSA cache and Cygwin's
own cache.  But caching doesn't help at all when enumerating.

There's also the problem to rely on an external program.  If it turns
out that the current implementation is too slow, I'm prepared to add
caching to cygserver to have a system-wide caching server, but Cygwin
shouldn't *require* that cygserver runs.  And either way, it still
wouldn't help when enumerating all accounts.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Corinna Vinschen!

>> >There's only one tiny problem.  Whatever I think about the full
>> >enumerate being right or wrong, I have this vague feeling that I'd like
>> >to have this implemented fully at one point.  My cat disapproves, but we
>> >can't agree on everything, I guess.  Another configuration option in
>> >/etc/nsswitch.conf might comfort her.
>> 
>> I don't know if this has been mentioned but would a cache help here,
>> i.e., nscd?  I think that's how Linux deals with this type of situation.

> Caching is wonderful for the usual requests for single entries from the
> DB, and for this we have already two caches, the LSA cache and Cygwin's
> own cache.  But caching doesn't help at all when enumerating.

> There's also the problem to rely on an external program.  If it turns
> out that the current implementation is too slow, I'm prepared to add
> caching to cygserver to have a system-wide caching server, but Cygwin
> shouldn't *require* that cygserver runs.  And either way, it still
> wouldn't help when enumerating all accounts.

Another rock in the direction of caching. Caching security-relevant data 'en
masse' is usually a not-so-good idea, and should be considered very carefully.
Even for Linux, common recommendation is to disable nscd to improve security
and reliability of the accounting data.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 13.02.2014, <15:36>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Christopher Faylor]

On Thu, Feb 13, 2014 at 11:00:25AM +0100, Corinna Vinschen wrote:
>On Feb 12 16:37, Christopher Faylor wrote:
>> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote:
>> >There's only one tiny problem.  Whatever I think about the full
>> >enumerate being right or wrong, I have this vague feeling that I'd like
>> >to have this implemented fully at one point.  My cat disapproves, but we
>> >can't agree on everything, I guess.  Another configuration option in
>> >/etc/nsswitch.conf might comfort her.
>> 
>> I don't know if this has been mentioned but would a cache help here,
>> i.e., nscd?  I think that's how Linux deals with this type of situation.
>
>Caching is wonderful for the usual requests for single entries from the
>DB, and for this we have already two caches, the LSA cache and Cygwin's
>own cache.  But caching doesn't help at all when enumerating.
>
>There's also the problem to rely on an external program.

But that's no different than Linux.  I've never looked at the code but
apparently libc has hooks for talking to nscd.  We could do the same
with cygserver.

>If it turns out that the current implementation is too slow, I'm
>prepared to add caching to cygserver to have a system-wide caching
>server, but Cygwin shouldn't *require* that cygserver runs.  And either
>way, it still wouldn't help when enumerating all accounts.

nscd does more than just keep information around in memory.  As I said,
it's how Linux deals with this situation.  I know because I didn't install
nscd when setting up a minimal Fedora 20 server at work and was met with
awful lags and timeouts in services which tried to read from our nis.
So Fedora doesn't require nscd but it sure does help.

But, even after having set it up, I still have to remember not to do
ls ~cg<tab> because it just takes forever.  So, if it is possible to
enumerate users then I think you just do it and let people learn the
cost.

I also would be pretty reluctant about deprecating a standard linux
function since there is nothing here that I can see which makes this
a distinctly linux problem unless you actually can't enumerate users.

cgf


cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2755 bytes --]

On Feb 13 09:35, Christopher Faylor wrote:
> On Thu, Feb 13, 2014 at 11:00:25AM +0100, Corinna Vinschen wrote:
> >On Feb 12 16:37, Christopher Faylor wrote:
> >> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote:
> >> >There's only one tiny problem.  Whatever I think about the full
> >> >enumerate being right or wrong, I have this vague feeling that I'd like
> >> >to have this implemented fully at one point.  My cat disapproves, but we
> >> >can't agree on everything, I guess.  Another configuration option in
> >> >/etc/nsswitch.conf might comfort her.
> >> 
> >> I don't know if this has been mentioned but would a cache help here,
> >> i.e., nscd?  I think that's how Linux deals with this type of situation.
> >
> >Caching is wonderful for the usual requests for single entries from the
> >DB, and for this we have already two caches, the LSA cache and Cygwin's
> >own cache.  But caching doesn't help at all when enumerating.
> >
> >There's also the problem to rely on an external program.
> 
> But that's no different than Linux.  I've never looked at the code but
> apparently libc has hooks for talking to nscd.  We could do the same
> with cygserver.
> 
> >If it turns out that the current implementation is too slow, I'm
> >prepared to add caching to cygserver to have a system-wide caching
> >server, but Cygwin shouldn't *require* that cygserver runs.  And either
> >way, it still wouldn't help when enumerating all accounts.
> 
> nscd does more than just keep information around in memory.  As I said,
> it's how Linux deals with this situation.  I know because I didn't install
> nscd when setting up a minimal Fedora 20 server at work and was met with
> awful lags and timeouts in services which tried to read from our nis.
> So Fedora doesn't require nscd but it sure does help.
> 
> But, even after having set it up, I still have to remember not to do
> ls ~cg<tab> because it just takes forever.  So, if it is possible to
> enumerate users then I think you just do it and let people learn the
> cost.

Yes, I think so too.  I have some preliminary code (actually, just
empty function shells right now) which are supposed to implement
full enumerating.

However, system admins might not exactly approve.  I discussed this
with our Linux folks, and I learned that NSS backends like SSSD or
winbind default to NOT allowing enumerating, but giving the admin a
choice to enable it.

So I think for our case a configuration option in /etc/nsswitch.conf
to limit the scope of the enumeration might be feasible.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Christopher Faylor]

On Thu, Feb 13, 2014 at 03:44:19PM +0100, Corinna Vinschen wrote:
>On Feb 13 09:35, Christopher Faylor wrote:
>> On Thu, Feb 13, 2014 at 11:00:25AM +0100, Corinna Vinschen wrote:
>> >On Feb 12 16:37, Christopher Faylor wrote:
>> >> On Wed, Feb 12, 2014 at 08:59:31PM +0100, Corinna Vinschen wrote:
>> >> >There's only one tiny problem.  Whatever I think about the full
>> >> >enumerate being right or wrong, I have this vague feeling that I'd like
>> >> >to have this implemented fully at one point.  My cat disapproves, but we
>> >> >can't agree on everything, I guess.  Another configuration option in
>> >> >/etc/nsswitch.conf might comfort her.
>> >> 
>> >> I don't know if this has been mentioned but would a cache help here,
>> >> i.e., nscd?  I think that's how Linux deals with this type of situation.
>> >
>> >Caching is wonderful for the usual requests for single entries from the
>> >DB, and for this we have already two caches, the LSA cache and Cygwin's
>> >own cache.  But caching doesn't help at all when enumerating.
>> >
>> >There's also the problem to rely on an external program.
>> 
>> But that's no different than Linux.  I've never looked at the code but
>> apparently libc has hooks for talking to nscd.  We could do the same
>> with cygserver.
>> 
>> >If it turns out that the current implementation is too slow, I'm
>> >prepared to add caching to cygserver to have a system-wide caching
>> >server, but Cygwin shouldn't *require* that cygserver runs.  And either
>> >way, it still wouldn't help when enumerating all accounts.
>> 
>> nscd does more than just keep information around in memory.  As I said,
>> it's how Linux deals with this situation.  I know because I didn't install
>> nscd when setting up a minimal Fedora 20 server at work and was met with
>> awful lags and timeouts in services which tried to read from our nis.
>> So Fedora doesn't require nscd but it sure does help.
>> 
>> But, even after having set it up, I still have to remember not to do
>> ls ~cg<tab> because it just takes forever.  So, if it is possible to
>> enumerate users then I think you just do it and let people learn the
>> cost.
>
>Yes, I think so too.  I have some preliminary code (actually, just
>empty function shells right now) which are supposed to implement
>full enumerating.
>
>However, system admins might not exactly approve.  I discussed this
>with our Linux folks, and I learned that NSS backends like SSSD or
>winbind default to NOT allowing enumerating, but giving the admin a
>choice to enable it.
>
>So I think for our case a configuration option in /etc/nsswitch.conf
>to limit the scope of the enumeration might be feasible.

Or, nscd.conf which has stuff like:

    enable-cache            passwd          yes
    positive-time-to-live   passwd          600
    negative-time-to-live   passwd          20
    suggested-size          passwd          211
    check-files             passwd          yes
    persistent              passwd          yes
    shared                  passwd          yes
    max-db-size             passwd          33554432
    auto-propagate          passwd          yes

I understand why a sysadmin might not want you to be able to enumerate
user names but that really isn't, IMO, a reason not to implement the
functionality (not that you are proposing this).  You obviously can't
assume that people won't exercise the capability if it is available.

Security through obscurity...?  Nah.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2131 bytes --]

On Feb 13 10:43, Christopher Faylor wrote:
> On Thu, Feb 13, 2014 at 03:44:19PM +0100, Corinna Vinschen wrote:
> >Yes, I think so too.  I have some preliminary code (actually, just
> >empty function shells right now) which are supposed to implement
> >full enumerating.
> >
> >However, system admins might not exactly approve.  I discussed this
> >with our Linux folks, and I learned that NSS backends like SSSD or
> >winbind default to NOT allowing enumerating, but giving the admin a
> >choice to enable it.
> >
> >So I think for our case a configuration option in /etc/nsswitch.conf
> >to limit the scope of the enumeration might be feasible.
> 
> Or, nscd.conf which has stuff like:
> 
>     enable-cache            passwd          yes
>     positive-time-to-live   passwd          600
>     negative-time-to-live   passwd          20
>     suggested-size          passwd          211
>     check-files             passwd          yes
>     persistent              passwd          yes
>     shared                  passwd          yes
>     max-db-size             passwd          33554432
>     auto-propagate          passwd          yes

I know that nsswitch.conf is not quite the right place for the
configuration variables, but I was reluctant to introduce YA file to
read at startup.  If nobody cares, we can also go with a limited
nscd.conf approach for the configuration variables.

> I understand why a sysadmin might not want you to be able to enumerate
> user names but that really isn't, IMO, a reason not to implement the
> functionality (not that you are proposing this).  You obviously can't
> assume that people won't exercise the capability if it is available.
> 
> Security through obscurity...?  Nah.

Nah.  But restricting the capability for pure networking reasons is on
order, IMHO.  Assuming that Cygwin has been setup by an admin and the
/etc files are not writable by the ordinary user, of course.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent?

[Achim Gratz]

Corinna Vinschen writes:
> Caching is wonderful for the usual requests for single entries from the
> DB, and for this we have already two caches, the LSA cache and Cygwin's
> own cache.  But caching doesn't help at all when enumerating.

Would it be possible to only look (for user name completion purposes) at
the current user plus whatever is in %SystemDrive%\Users plus whatever
is found in /etc/passwd?  That way no beans are spilled about domain
users that couldn't be gleaned from the local file system and in almost
all cases that's the list one would want to complete from anyway.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent?

[Andrey Repin]

Greetings, Achim Gratz!

> Corinna Vinschen writes:
>> Caching is wonderful for the usual requests for single entries from the
>> DB, and for this we have already two caches, the LSA cache and Cygwin's
>> own cache.  But caching doesn't help at all when enumerating.

> Would it be possible to only look (for user name completion purposes) at
> the current user plus whatever is in %SystemDrive%\Users

Assuming, the mentioned directory at all exists... And contain relevant
data... And the data is full... and not redundant... Too many assumptions to
work from.

> plus whatever is found in /etc/passwd?
> That way no beans are spilled about domain users that couldn't be gleaned
> from the local file system and in almost all cases that's the list one would
> want to complete from anyway. 

You have to know, how easy it is to confuse a user, who expect a file system
being solid source of security-related information.
Starting from profiles of nonexistent accounts, and down to relocated profiles
and complete absence of the directory in question at all.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 13.02.2014, <22:34>

Sorry for my terrible english...

Re: get rid of getpwent?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

On Feb 13 18:50, Achim Gratz wrote:
> Corinna Vinschen writes:
> > Caching is wonderful for the usual requests for single entries from the
> > DB, and for this we have already two caches, the LSA cache and Cygwin's
> > own cache.  But caching doesn't help at all when enumerating.
> 
> Would it be possible to only look (for user name completion purposes) at
> the current user plus whatever is in %SystemDrive%\Users plus whatever
> is found in /etc/passwd?  That way no beans are spilled about domain
> users that couldn't be gleaned from the local file system and in almost
> all cases that's the list one would want to complete from anyway.

Oh, hmm.  Well, it might be possible, but somehow I'm not excited by the
idea.  While it looks like getpwent is mostly used for this purpose, you
don't really know it.  I think I'll try to implement it fully and then
let the admin decide what to allow.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent?

[Achim Gratz]

Corinna Vinschen writes:
> Oh, hmm.  Well, it might be possible, but somehow I'm not excited by the
> idea.  While it looks like getpwent is mostly used for this purpose, you
> don't really know it.  I think I'll try to implement it fully and then
> let the admin decide what to allow.

Configurable would be OK of course.  I'm not sure when I'll get around
to testing the new snapshot, but I can certainly try how long it takes
when I do "~<TAB>" in the shell.  IIRC, mkpasswd took several minutes to
pull down the domain accounts (I've then deleted all but the handful
that was used for network shares).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

On Feb 13 21:41, Achim Gratz wrote:
> Corinna Vinschen writes:
> > Oh, hmm.  Well, it might be possible, but somehow I'm not excited by the
> > idea.  While it looks like getpwent is mostly used for this purpose, you
> > don't really know it.  I think I'll try to implement it fully and then
> > let the admin decide what to allow.
> 
> Configurable would be OK of course.  I'm not sure when I'll get around
> to testing the new snapshot, but I can certainly try how long it takes
> when I do "~<TAB>" in the shell.  IIRC, mkpasswd took several minutes to
> pull down the domain accounts (I've then deleted all but the handful
> that was used for network shares).

Keep in mind, the search in getpwent/getgrent is not yet implemented.
You only get what the application cached so far.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Warren Young!

>> problem was an assumption made in the 'checkfile' perl script: it was
>> assumed that cygwin1.dll is the first DLL listed by objdump.

> Details, details. :)

That's where is the devil, we know, right?


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 12.02.2014, <08:23>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/7/2014 02:49, Corinna Vinschen wrote:
> On Feb  6 14:43, Warren Young wrote:
>> On 2/6/2014 07:13, Corinna Vinschen wrote:
>
> it would, of course, be possible to implement Cygwin
> command line tools along the lines of useradd/usermod/groupdel.  For AD,
> they would just have to use LDAP,

If by "use LDAP" you mean the ldap_* functions in the OpenLDAP library, 
I can't recommend it.  (See my other post on LDAP books.)

Such programs need not be portable.  It doesn't look like it will be 
helpful to start with the ones from Linux's shadow-utils, since those 
modify /etc/foo directly.  They don't even attempt to abstract away 
/etc/foo, NIS and LDAP, so they're no use to us except as design 
exemplars.  Cygwin equivalents should use the same command line format 
and flags, they should use /etc/skel, etc.

I don't see why such programs shouldn't be written straight to the 
Windows API, even though this is naughty on Cygwin.  The Win32 security 
API fills the same role as libldap does on a Linux box configured for LDAP.

You're right that such programs are probably going to be necessary, if 
Cygwin moves to SAM/AD as primary.  Windows Home edition user management 
probably won't be powerful enough to do what Cygwin needs, if SAM is 
Cygwin's Single Point of Truth on such systems.

>> I don't see a reason for this to change, given that so many other
>> POSIX systems share aspects of this behavior.
>
> Sorry, I lost you there.  What exactly of the current behaviour do
> you want to keep?

I want getpwent() and friends to remain available, but to switch to 
AD/SAM as primary, like OS X does all the time, or like Linux does when 
LDAP is enabled in NSS.[*]

I want the mkpasswd and mkgroup utilities to remain available to 
generate these files on systems where there are programs that insist on 
scanning them directly.  Such files will be understood to be potentially 
stale caches of records that really live in SAM/AD.  Plus, these files 
are useful as debug dumps, and for casual grepping.

Corinna, an earlier post of yours suggested that /etc/foo was being kept 
as primary for speed reasons, but are you comparing to SAM or to AD? 
And have you tested it lately?

I don't think it's fair or even useful to compare the speed of /etc/foo 
to AD.  Sites using AD have implicitly said they're willing to pay a 
speed hit to get certain benefits.  The fact that AD is slower is like 
pointing out that container ships are slower than a Maserati.

I'd expect a SAM lookup to be *faster* than any /etc/foo lookup, even 
with very small /etc/foo files.

This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:

      #include <pwd.h>
      #include <stdio.h>
      #include <stdlib.h>

      int main(int argc, const char* argv[])
      {
          int i;
          const char* user = argv[1];

          if (!user) {
              printf("usage: %s username\n", argv[0]);
              exit(1);
          }

          for (i = 0; i < 1000000; ++i) {
              struct passwd* pw = getpwnam(user);
              if (!pw) {
                  printf("User %s doesn't exist!\n", user);
                  exit(2);
              }
              else if (i == 0) {
                  printf("User %s is UID %d\n", user, pw->pw_uid);
              }
          }
      }

So, each getpwnam() call takes 7.1 microseconds on average.

It used to take 5.5 seconds, until I moved my user record down to the 
end of the /etc/passwd file, just to make it a worst-case test.  3 lines 
of movement added ~1.6 us to the call time.  Then I moved it to the top 
of the file, and the program ran in 1.2 seconds.  6x delta between best 
and worst case, in such a small file!

If you send me the Win32 code for a SAM equivalent, I'll run it here, 
for comparison.  Please keep the unnecessary overhead, like the 
printf()s and logic inside the main work loop, so we can null out the 
difference in the comparison.

> Also, the usual problem:  What exactly
> do you write to passwd/group?  If you write all users of a big corp,
> the files get really big.

If Cygwin moves to SAM/AD as primary, then this doesn't matter.

The disk space hit is small in modern terms.  Didn't someone calculate 
135 bytes on average per /etc/passwd line recently?  So, a massive 
100,000 user corporation's /etc/passwd would be 13.5 MB.  Big, but not 
TeX big.  Not even Vim big, these days.  Sigh.

In such a system, Joe User probably isn't allowed to pull all 100,000 
records anyway, for security reasons.  Won't mkpasswd skip all those?

The speed hit doesn't really matter, since it will only be paid by 
programs that insist on scanning these files directly, if AD/SAM are 
primary.  That's going to be things like misguided Perl scripts that 
parse the file directly, instead of using Perl's built-in getpwent() and 
such.

>> (I assume getpwnam() consults SAM/AD in Cygwin, now or in Cygwin.next.)
>
> It checks for the file first, then it asks SAM/AD.

I realize getting rid of /etc/foo checks in these functions is a very 
large change, from a system operations point of view.  Large enough to 
call the result Cygwin 1.9, IMO, simply to give people a forking point, 
to buy migration time.




-----

[*] The comparison to NSS breaks down quickly.  A "files" fall-back 
doesn't make sense, since you can't make up the proper NT security token 
to populate /etc/passwd with a Cygwin-only user that doesn't exist in 
SAM/AD.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On 2/7/2014 13:09, Warren Young wrote:
>
> I want getpwent() and friends to remain available, but to switch to
> AD/SAM as primary, like OS X does all the time,

I just realized that this means getpwent() turns into an AD database 
linear scan at AD sites.

Hmmm...

I think I'm still in favor of the idea.  It seems like a "Doctor, it 
hurts when I do a complete linear AD scan!" issue.  ("So don't do that, 
then!")

This could make Cygwin a positive force for getting these old programs 
updated to use modern APIs.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Feb  7 13:25, Warren Young wrote:
> On 2/7/2014 13:09, Warren Young wrote:
> >
> >I want getpwent() and friends to remain available, but to switch to
> >AD/SAM as primary, like OS X does all the time,
> 
> I just realized that this means getpwent() turns into an AD database
> linear scan at AD sites.
> 
> Hmmm...

So you see the problem.  Also:
http://cygwin.com/ml/cygwin/2014-02/msg00146.html


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1.1: Type: text/plain, Size: 2558 bytes --]

On Feb  7 13:09, Warren Young wrote:
> On 2/7/2014 02:49, Corinna Vinschen wrote:
> >On Feb  6 14:43, Warren Young wrote:
> >>On 2/6/2014 07:13, Corinna Vinschen wrote:
> >
> >it would, of course, be possible to implement Cygwin
> >command line tools along the lines of useradd/usermod/groupdel.  For AD,
> >they would just have to use LDAP,
> 
> If by "use LDAP" you mean the ldap_* functions in the OpenLDAP
> library, I can't recommend it.  (See my other post on LDAP books.)

You can also use the calls from wldap32.dll which is available anyway.

> Such programs need not be portable.

Never said so.

> I don't see why such programs shouldn't be written straight to the
> Windows API, even though this is naughty on Cygwin.  The Win32
> security API fills the same role as libldap does on a Linux box
> configured for LDAP.

The underlying protocol is LDAP, so why not use it, given that lots of
changes to AD cannot be done using the "High Level" Net API anyway.

> You're right that such programs are probably going to be necessary,
> if Cygwin moves to SAM/AD as primary.  Windows Home edition user
> management probably won't be powerful enough to do what Cygwin
> needs, if SAM is Cygwin's Single Point of Truth on such systems.

Again, it isn't.  We will keep the passwd and group files for users
who are more comfortable with them.

We will also have an nsswitch.conf file for configuration.  I attached
my local sandbox version below.

> I want the mkpasswd and mkgroup utilities to remain available

They will, with slight changes.  The default values generated for
uid/gid numbers should preferredly reflect the automatism when reading
from SAM/AD.

> Corinna, an earlier post of yours suggested that /etc/foo was being
> kept as primary for speed reasons, but are you comparing to SAM or
> to AD? And have you tested it lately?

Did I really write something about speed?  I think SAM/AD will be mostly
quicker but they will be especially less hassle and allow centralized
maintainance, which is a real boon for admins.

Also, the new Cygwin will only read and cache the requested entries
from the passwd/group files, not the entire file.

Also, if it turns out that AD is too slow for some reason or in some
environment, we should consider to use cygserver as a centralized local
cache.  But this is something for later.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #1.2: nsswitch.conf --]
[-- Type: text/plain, Size: 1353 bytes --]

# /etc/nsswitch.conf
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.
#
# passwd:
# group:
#
#    "files"	only use /etc/passwd or /etc/group file.
#    "db"	only use SAM/AD retrieval.
#    "files db"	both, files preferred.  This is the default.
#
#    "db files"	does not make any sense
#
passwd: files db
group:  files db
#
# Configuration of "db" style passwd/group handling:
#
# db_prefix:
#
#    "auto" 	If "auto", prepend domain to account name if the account
#		is not a member of the machine's primary domain.  Prepend
#		just the separator char if the account is a well-known
#		or builtin group.
#
#    "primary" 	"primary" is like "auto", but prepend domain to account name
#		as well, if the account is a member of the machine's primary
#		domain.
#
#    "always" 	If set to "always", always prepend domain, even for
#		well-known and builtin accounts.
#
db_prefix: auto
#
# db_cache:
#
#    "yes" 	If yes, cache once retrieved DB values in local process,
#		hand cache down to child processes.
#
#    "no"	If no, fetch passwd or group entries anew, every time an
#		entry is requested.  Default is "no".
#
db_cache: no
#
#  db_separator:
#
#		Set separator character between domain and account name to
#		the ASCII char X.  Default is '+'.
#
db_separator: +

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

RE: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> I think SAM/AD will be mostly quicker

I do not want to be a party pooper here, but have you checked how
the AD approach will work from the unmanaged Windows service accounts?

We've been experiencing rather nasty effects of the M$ design that
when a host changes its password (it is required to, every so many
days), it is no longer considered an "authorized" agent (rather,
anonymous).  Accessing AD anonymously (esp. from system-managed
service account) is limited;  like when you request a list,
you get only first 100 (who at M$ had invented this?!) entries.
Which means that if your code is scanning, it won't find
more than 100 users (and they are alphabetized, so the "excess"
users will simply disappear from view).  That creates false-positive
nonexistent users / groups.  The only remedy is to restart the host...

P.S.  I'm not an AD person, and some of the info from the above
comes from our sysadmins (how they see things unfolding).

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

On Feb  7 21:49, Lavrentiev, Anton (NIH/NLM/NCBI) [C] wrote:
> > I think SAM/AD will be mostly quicker
> 
> I do not want to be a party pooper here, but have you checked how
> the AD approach will work from the unmanaged Windows service accounts?

No, and I fail to see how this is related to the on-the-fly generation
of passwd and group entries.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

RE: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> and I fail to see how this is related to the on-the-fly generation of passwd and group entries

Well, if a cygwin app was run under such an account, it might be affected, that's all...

Anton Lavrentiev
Contractor NIH/NLM/NCBI

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> and I fail to see how this is related to the on-the-fly generation of passwd and group entries

Well, if a cygwin app was run under such an account, it might be affected, that's all...

Anton Lavrentiev
Contractor NIH/NLM/NCBI

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Larry Hall (Cygwin)]

On 2/7/2014 3:09 PM, Warren Young wrote:
> This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
>
>       #include <pwd.h>
>       #include <stdio.h>
>       #include <stdlib.h>
>
>       int main(int argc, const char* argv[])
>       {
>           int i;
>           const char* user = argv[1];
>
>           if (!user) {
>               printf("usage: %s username\n", argv[0]);
>               exit(1);
>           }
>
>           for (i = 0; i < 1000000; ++i) {
>               struct passwd* pw = getpwnam(user);
>               if (!pw) {
>                   printf("User %s doesn't exist!\n", user);
>                   exit(2);
>               }
>               else if (i == 0) {
>                   printf("User %s is UID %d\n", user, pw->pw_uid);
>               }
>           }
>       }
>
> So, each getpwnam() call takes 7.1 microseconds on average.

I think you forgot to put an "exit(0);" after the last printf().  Without
it, you're checking for the same user a million times, which is certainly
going to take a little time. ;-)

-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[David Stacey]

On 07/02/14 21:44, Larry Hall (Cygwin) wrote:
> On 2/7/2014 3:09 PM, Warren Young wrote:
>> This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
>>
>>       #include <pwd.h>
>>       #include <stdio.h>
>>       #include <stdlib.h>
>>
>>       int main(int argc, const char* argv[])
>>       {
>>           int i;
>>           const char* user = argv[1];
>>
>>           if (!user) {
>>               printf("usage: %s username\n", argv[0]);
>>               exit(1);
>>           }
>>
>>           for (i = 0; i < 1000000; ++i) {
>>               struct passwd* pw = getpwnam(user);
>>               if (!pw) {
>>                   printf("User %s doesn't exist!\n", user);
>>                   exit(2);
>>               }
>>               else if (i == 0) {
>>                   printf("User %s is UID %d\n", user, pw->pw_uid);
>>               }
>>           }
>>       }
>>
>> So, each getpwnam() call takes 7.1 microseconds on average.
>
> I think you forgot to put an "exit(0);" after the last printf(). Without
> it, you're checking for the same user a million times, which is certainly
> going to take a little time. ;-)
>

I thought the point of the programme /was/ to call getpwnam() a million 
times. Time this as accurately as you can. Then, with a quick division, 
you get the time for one call.

Dave.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Larry Hall (Cygwin)]

On 2/7/2014 5:45 PM, David Stacey wrote:
> On 07/02/14 21:44, Larry Hall (Cygwin) wrote:
>> On 2/7/2014 3:09 PM, Warren Young wrote:
>>> This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
>>>
>>>       #include <pwd.h>
>>>       #include <stdio.h>
>>>       #include <stdlib.h>
>>>
>>>       int main(int argc, const char* argv[])
>>>       {
>>>           int i;
>>>           const char* user = argv[1];
>>>
>>>           if (!user) {
>>>               printf("usage: %s username\n", argv[0]);
>>>               exit(1);
>>>           }
>>>
>>>           for (i = 0; i < 1000000; ++i) {
>>>               struct passwd* pw = getpwnam(user);
>>>               if (!pw) {
>>>                   printf("User %s doesn't exist!\n", user);
>>>                   exit(2);
>>>               }
>>>               else if (i == 0) {
>>>                   printf("User %s is UID %d\n", user, pw->pw_uid);
>>>               }
>>>           }
>>>       }
>>>
>>> So, each getpwnam() call takes 7.1 microseconds on average.
>>
>> I think you forgot to put an "exit(0);" after the last printf(). Without
>> it, you're checking for the same user a million times, which is certainly
>> going to take a little time. ;-)
>>
>
> I thought the point of the programme /was/ to call getpwnam() a million
> times. Time this as accurately as you can. Then, with a quick division, you
> get the time for one call.

Hm, I missed that he summarized with _microseconds_, even though I quoted
that too in my response. :-(

My average was much closer to 2 microseconds per call but that could be
machine differences.

In any case, sorry for the noise.

-- 
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Andrey Repin]

Greetings, Larry Hall (Cygwin)!

>>>> This takes 7.1 seconds on my system, with a 12-line /etc/passwd file:
>>>>
>>>>       #include <pwd.h>
>>>>       #include <stdio.h>
>>>>       #include <stdlib.h>
>>>>
>>>>       int main(int argc, const char* argv[])
>>>>       {
>>>>           int i;
>>>>           const char* user = argv[1];
>>>>
>>>>           if (!user) {
>>>>               printf("usage: %s username\n", argv[0]);
>>>>               exit(1);
>>>>           }
>>>>
>>>>           for (i = 0; i < 1000000; ++i) {
>>>>               struct passwd* pw = getpwnam(user);
>>>>               if (!pw) {
>>>>                   printf("User %s doesn't exist!\n", user);
>>>>                   exit(2);
>>>>               }
>>>>               else if (i == 0) {
>>>>                   printf("User %s is UID %d\n", user, pw->pw_uid);
>>>>               }
>>>>           }
>>>>       }
>>>>
>>>> So, each getpwnam() call takes 7.1 microseconds on average.
>>>
>>> I think you forgot to put an "exit(0);" after the last printf(). Without
>>> it, you're checking for the same user a million times, which is certainly
>>> going to take a little time. ;-)
>>>
>>
>> I thought the point of the programme /was/ to call getpwnam() a million
>> times. Time this as accurately as you can. Then, with a quick division, you
>> get the time for one call.

> Hm, I missed that he summarized with _microseconds_, even though I quoted
> that too in my response. :-(

> My average was much closer to 2 microseconds per call but that could be
> machine differences.

> In any case, sorry for the noise.

In either case, repeatedly requesting the same record in a short amount of
time will only test the system level cache.


--
WBR,
Andrey Repin (anrdaemon@yandex.ru) 08.02.2014, <04:39>

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On Feb 7, 2014, at 5:40 PM, Andrey Repin <anrdaemon@yandex.ru> wrote:

>>> I thought the point of the programme /was/ to call getpwnam() a million
>>> times.

Precisely.

> In either case, repeatedly requesting the same record in a short amount of
> time will only test the system level cache.

If that were true, moving the requested record around in /etc/passwd wouldn’t cause a 6x spread in test results. That’s O(N) type behavior, whereas a cache approaches O(1).

If SAM uses a tree-structured DB, as I suspect, it will be some logarithmic function, like O(log2(N)).
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: get rid of getpwent? (Was: cygwin-1.7.28 getpwent header declaration changes ?)

[Warren Young]

On Feb 8, 2014, at 8:19 AM, Warren Young <warren@etr-usa.com> wrote:

> On Feb 7, 2014, at 5:40 PM, Andrey Repin <anrdaemon@yandex.ru> wrote:
> 
>> In either case, repeatedly requesting the same record in a short amount of
>> time will only test the system level cache.
> 
> If that were true, moving the requested record around in /etc/passwd wouldn’t cause a 6x spread in test results. That’s O(N) type behavior, whereas a cache approaches O(1).

A clarification:

I agree that /etc/passwd is being cached for the entire test.  My 12-line file is a bit over 1K, so it fits easily into a 4K RAM page.  Also, I repeated my tests several times, so even the initial disk read is cached.

The fact that we see a 6x speed difference depending on where in the file — or RAM page, if you prefer — the target record is tells us that lookups are dominated by parsing time.

Parsing overhead shouldn’t exist for SAM or AD. They’re “pre-parsed.”

The equivalent overhead in SAM/AD terms is tree and index traversal, which should happen a lot faster than O(N).
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Marco Atzeri]

On 06/02/2014 10:01, Corinna Vinschen wrote:
> On Feb  6 08:29, Marco Atzeri wrote:
>> on cygwin-1.7.28 64 bit but not on cygwin-1.7.27,
>> (only system difference) compiling octave I see:
>>
>> ----------------------------------------------------------------
>> /pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
>> :114:25: error: '::getpwent' has not been declared
>>     return octave_passwd (::getpwent (), msg);
>>                           ^
>> [...]
>> Is something slightly changed in the  relative system headers ?
>
> I don't know anything about octave, but what has changed is that newlib
> is finally introducing BSD-like visibility macros.  The pwd.h header now
> contains this:
>
>    #if __XSI_VISIBLE >= 500
>    struct passwd   *getpwent (void);
>    void             setpwent (void);
>    void             endpwent (void);
>    #endif
>
> The default settings per the sys/cdefs.h header are:
>
>    #define __POSIX_VISIBLE         200809
>    #define __XSI_VISIBLE           700
>    #define __BSD_VISIBLE           1
>    #define __ISO_C_VISIBLE         2011
>
> unless your environment defines one of
>
>    _XOPEN_SOURCE
>    _POSIX_C_SOURCE
>    _ANSI_SOURCE
>    _C99_SOURCE
>    _C11_SOURCE
>    _GNU_SOURCE
>
> Have a look at the end of /usr/include/sys/cdefs.h, it explains things.
>
>
> Corinna
>

probably I am missing something obvious, but should not

/usr/include/sys/cdefs.h

be included in some way in
/usr/include/pwd.h

to make working the

     #if __XSI_VISIBLE >= 500
     struct passwd   *getpwent (void);
     void             setpwent (void);
     void             endpwent (void);
     #endif

?

Regards
Marco



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

On Feb  9 20:50, Marco Atzeri wrote:
> 
> 
> On 06/02/2014 10:01, Corinna Vinschen wrote:
> >On Feb  6 08:29, Marco Atzeri wrote:
> >>on cygwin-1.7.28 64 bit but not on cygwin-1.7.27,
> >>(only system difference) compiling octave I see:
> >>
> >>----------------------------------------------------------------
> >>/pub/devel/octave/octave-3.8.0-2/src/octave-3.8.0/liboctave/system/oct-passwd.cc
> >>:114:25: error: '::getpwent' has not been declared
> >>    return octave_passwd (::getpwent (), msg);
> >>                          ^
> >>[...]
> >>Is something slightly changed in the  relative system headers ?
> >
> >I don't know anything about octave, but what has changed is that newlib
> >is finally introducing BSD-like visibility macros.  The pwd.h header now
> >contains this:
> >
> >   #if __XSI_VISIBLE >= 500
> >   struct passwd   *getpwent (void);
> >   void             setpwent (void);
> >   void             endpwent (void);
> >   #endif
> >
> >The default settings per the sys/cdefs.h header are:
> >
> >   #define __POSIX_VISIBLE         200809
> >   #define __XSI_VISIBLE           700
> >   #define __BSD_VISIBLE           1
> >   #define __ISO_C_VISIBLE         2011
> >
> >unless your environment defines one of
> >
> >   _XOPEN_SOURCE
> >   _POSIX_C_SOURCE
> >   _ANSI_SOURCE
> >   _C99_SOURCE
> >   _C11_SOURCE
> >   _GNU_SOURCE
> >
> >Have a look at the end of /usr/include/sys/cdefs.h, it explains things.
> >
> >
> >Corinna
> >
> 
> probably I am missing something obvious, but should not
> 
> /usr/include/sys/cdefs.h
> 
> be included in some way in
> /usr/include/pwd.h
> 
> to make working the
> 
>     #if __XSI_VISIBLE >= 500
>     struct passwd   *getpwent (void);
>     void             setpwent (void);
>     void             endpwent (void);
>     #endif
> 
> ?

You are oh so right.  I'm just uploading a new cygwin-1.7.18-2
package which fixes this and the sys/file.h bug.


HTH,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Marco Atzeri]

On 09/02/2014 21:09, Corinna Vinschen wrote:
> On Feb  9 20:50, Marco Atzeri wrote:
>>
>>

>>
>> probably I am missing something obvious, but should not
>>
>> /usr/include/sys/cdefs.h
>>
>> be included in some way in
>> /usr/include/pwd.h
>>
>> to make working the
>>
>>      #if __XSI_VISIBLE >= 500
>>      struct passwd   *getpwent (void);
>>      void             setpwent (void);
>>      void             endpwent (void);
>>      #endif
>>
>> ?
>
> You are oh so right.  I'm just uploading a new cygwin-1.7.18-2
> package which fixes this and the sys/file.h bug.
>
>
> HTH,
> Corinna
>

works as charm

Tschüs
Marco

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Jan Nijtmans]

2014-02-06 15:41 GMT+01:00 Corinna Vinschen:
> Jan, can you please check if SQlite builds *without* defining
> __BSD_VISIBLE, but adding the following lines to /usr/include/sys/file.h;

That addition makes SQLite build fine again. You can try it easily
by installing the "sqlite3-debug" package and then:

    gcc -c /usr/src/debug/sqlite3-3.8.3-2/sqlite3.c

If this command produces an "sqlite3.o" file in the
current directory without complaining, everything is OK.
(The current sqlite3-3.8.3-2 package doesn't set
__BSD_VISIBLE, and with your proposed change
it won't be necessary to add it)

Thank you very much!

Regards,
       Jan Nijtmans

> Index: sys/file.h
> ===================================================================
> RCS file: /cvs/src/src/winsup/cygwin/include/sys/file.h,v
> retrieving revision 1.8
> diff -u -p -r1.8 file.h
> --- sys/file.h  17 Jan 2014 11:01:46 -0000      1.8
> +++ sys/file.h  6 Feb 2014 14:41:23 -0000
> @@ -31,4 +31,16 @@
>  #define L_INCR         SEEK_CUR
>  #define L_XTND         SEEK_END
>
> +/* Including <sys/file.h> always defines flock & macros. */
> +#if __BSD_VISIBLE - 0  == 0
> +
> +#define LOCK_SH                0x01            /* shared file lock */
> +#define LOCK_EX                0x02            /* exclusive file lock */
> +#define LOCK_NB                0x04            /* don't block when locking */
> +#define LOCK_UN                0x08            /* unlock file */
> +
> +extern int flock _PARAMS ((int, int));
> +
> +#endif
> +
>  #endif

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-1.7.28 getpwent header declaration changes ?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Feb  7 10:27, Jan Nijtmans wrote:
> 2014-02-06 15:41 GMT+01:00 Corinna Vinschen:
> > Jan, can you please check if SQlite builds *without* defining
> > __BSD_VISIBLE, but adding the following lines to /usr/include/sys/file.h;
> 
> That addition makes SQLite build fine again. You can try it easily
> by installing the "sqlite3-debug" package and then:
> 
>     gcc -c /usr/src/debug/sqlite3-3.8.3-2/sqlite3.c
> 
> If this command produces an "sqlite3.o" file in the
> current directory without complaining, everything is OK.
> (The current sqlite3-3.8.3-2 package doesn't set
> __BSD_VISIBLE, and with your proposed change
> it won't be necessary to add it)
> 
> Thank you very much!

Thanks for testing.  I applied the patch to CVS.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]