nis compat depreciation and test failures

nis compat depreciation and test failures

[Steve Ellcey]

I have a question about libnss_compat.  There was a change to nis/Makefile
to stop building this library by default and only build it when configuring
with --enable-obsolete-nsl.  However some glibc tests are still testing this
library whether it is built or not.  In nss/spwd-lookup.c, nss/pwd-lookup.c,
and nss/grp-lookup.c there is a definition of DEFAULT_CONFIG that includes
'compat' as a lookup method and that will cause some tests to try and load
libnss_compat even when it has not been built.

On most systems this will not cause a failure because when the dynamic loader
cannot find libnss_compat in the glibc build directories it will look in
the system directories and if it finds libnss_compat there it will use it
and no failures show up.  When I test with ILP32 aarch64 there is no
libnss_compat system library, the load of libnss_compat fails and I get these
extra failures:

FAIL: grp/testgrp
FAIL: nss/bug17079
FAIL: posix/globtest
FAIL: pwd/tst-getpw

I don't know if there is anyway that we can stop the testing from using
the standard library locations but I think the tests should at least
be changed to not use the compat lookup method and/or use it only when
configured in.

Steve Ellcey
sellcey@cavium.com

Re: nis compat depreciation and test failures

[Joseph Myers]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

On Mon, 14 Aug 2017, Steve Ellcey wrote:

> and no failures show up.  When I test with ILP32 aarch64 there is no
> libnss_compat system library, the load of libnss_compat fails and I get these
> extra failures:
> 
> FAIL: grp/testgrp
> FAIL: nss/bug17079
> FAIL: posix/globtest
> FAIL: pwd/tst-getpw

That explains the failures of those tests I saw for some MIPS ABIs and 
noted on the wiki as having unknown cause.  Could you update the 
architecture-independent section of 
<https://sourceware.org/glibc/wiki/Release/2.26> with a discussion of 
these failures and the circumstances under which they occur (probably 
linking to your message)?  (Of course, we should fix them for the future, 
and hopefully there will be a fix suitable to backport to the release 
branch.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: nis compat depreciation and test failures

[DJ Delorie]

Wait... what's the real failure here?  What *should* happen when a
method is listed for which a service doesn't exist?  Are we handing it
incorrectly, which causes the test failure, or is the test just
dependent on those services?  Or dependent on the host's data?

I mean, we can take that out of the test if we don't need it, but we
should then add a test for "make sure the right thing happens if a
service is missing".

Also, if these tests depend on a "properly configured" *host*, we should
consider moving them to the nss directory and using the new test
services framework instead, so we can provide specific test data for
them.  Or find some other way to ensure we have the right test data.
(/me is looking into test-in-installed-chroot for example)

Re: nis compat depreciation and test failures

[Florian Weimer]

On 08/14/2017 06:49 PM, Steve Ellcey wrote:
> I have a question about libnss_compat.  There was a change to nis/Makefile
> to stop building this library by default and only build it when configuring
> with --enable-obsolete-nsl.  However some glibc tests are still testing this
> library whether it is built or not.  In nss/spwd-lookup.c, nss/pwd-lookup.c,
> and nss/grp-lookup.c there is a definition of DEFAULT_CONFIG that includes
> 'compat' as a lookup method and that will cause some tests to try and load
> libnss_compat even when it has not been built.

Thanks for noticing this.  We should remove “compat” from the default
configuration and replace it with “files”, at least for the next
release.  For the 2.26 releases, we may have to make this conditional on
whether --enable-obsolete-nsl has been specified.

The default for the hosts database, "dns [!UNAVAIL=return] files" looks
wrong as well.  We should always prefer “files” (hopefully with entries
for localhost) over “dns”.

Thanks,
Florian

Re: nis compat depreciation and test failures

[Steve Ellcey]

On Mon, 2017-08-14 at 17:14 +0000, Joseph Myers wrote:
> On Mon, 14 Aug 2017, Steve Ellcey wrote:
> 
> > 
> > and no failures show up.  When I test with ILP32 aarch64 there is no
> > libnss_compat system library, the load of libnss_compat fails and I get these
> > extra failures:
> > 
> > FAIL: grp/testgrp
> > FAIL: nss/bug17079
> > FAIL: posix/globtest
> > FAIL: pwd/tst-getpw
> That explains the failures of those tests I saw for some MIPS ABIs
> and 
> noted on the wiki as having unknown cause.  Could you update the 
> architecture-independent section of 
> <https://sourceware.org/glibc/wiki/Release/2.26> with a discussion of 
> these failures and the circumstances under which they occur (probably 
> linking to your message)?  (Of course, we should fix them for the future, 
> and hopefully there will be a fix suitable to backport to the release 
> branch.)

I haven't been able to update the Wiki.  I can't seem to log in.  I am
in EditorGroup but I don't remember my password and I cannot get the
Wiki page to email me a token for resetting it.  I am not really 
sure what email address that I used to setup my wiki access (sje <AT>
sourceware.org or sje <AT> gcc.gnu.org or sellcey <AT> cavium.com), but
none of them seem to be working.

Steve Ellcey
sellcey@cavium.com

P.S.  The Wiki (and Bugzilla) seem really slow.  I am not sure if the  
      machine is just really busy or if something else is going on.

Re: nis compat depreciation and test failures

[Joseph Myers]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Tue, 15 Aug 2017, Steve Ellcey wrote:

> P.S.  The Wiki (and Bugzilla) seem really slow.  I am not sure if the  
>       machine is just really busy or if something else is going on.

Please report wiki problems to overseers@sourceware.org.  We know there 
are GCC wiki problems arising from last night's filesystem corruption 
(pages missing), problems seen with glibc or other wikis need to be 
reported.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: nis compat depreciation and test failures

[Carlos O'Donell]

On 08/15/2017 06:11 PM, Joseph Myers wrote:
> On Tue, 15 Aug 2017, Steve Ellcey wrote:
> 
>> P.S.  The Wiki (and Bugzilla) seem really slow.  I am not sure if the  
>>       machine is just really busy or if something else is going on.
> 
> Please report wiki problems to overseers@sourceware.org.  We know there 
> are GCC wiki problems arising from last night's filesystem corruption 
> (pages missing), problems seen with glibc or other wikis need to be 
> reported.
 
I have fixed this for Steve.

-- 
Cheers,
Carlos.

Re: nis compat depreciation and test failures

[Carlos O'Donell]

On 08/14/2017 02:00 PM, DJ Delorie wrote:
> 
> Wait... what's the real failure here?  What *should* happen when a
> method is listed for which a service doesn't exist?  Are we handing it
> incorrectly, which causes the test failure, or is the test just
> dependent on those services?  Or dependent on the host's data?

The tests are dependent on the host configuration, and that is the bug,
we have many such tests which need to be changed into tests that are
isolated from the host configuration using a namespace/chroot.

The library is compiled with --disable-obsolete-nsl, which disables
building all of the NIS and compat (used for NIS) NSS service modules.
This is done because such modules are being replaced by TIRPC-based
versions which support IPv6.

The problem is that with such a configuration of glibc, you need to
get your compat NSS module from somewhere else, and most people don't
arrange for that in their normal test setup.

> I mean, we can take that out of the test if we don't need it, but we
> should then add a test for "make sure the right thing happens if a
> service is missing".

We aren't taking it out of the test, but rather removing 'compat' as
an NSS module from the *default* service modules to try when doing
a shadow password lookup.

The secondary bug here is: Why do we have compat in the default config
for shadow passwords when compat (NIS) hasn't really been the default
for Linux for a long time.

> Also, if these tests depend on a "properly configured" *host*, we should
> consider moving them to the nss directory and using the new test
> services framework instead, so we can provide specific test data for
> them.  Or find some other way to ensure we have the right test data.
> (/me is looking into test-in-installed-chroot for example)
 
I think that's also a good idea.

In summary:

(a) Make tests independent of host configuration.

(b) Add more tests to nss/ subdirectory to test for things like the
    behaviour when specifying a missing NSS module.

(c) Remove 'compat' from DEFAULT_CONFIG for all glibc-built services.

-- 
Cheers,
Carlos.

Re: nis compat depreciation and test failures

[Steve Ellcey]

On Mon, 2017-08-14 at 17:14 +0000, Joseph Myers wrote:
> On Mon, 14 Aug 2017, Steve Ellcey wrote:
> 
> > 
> > and no failures show up.  When I test with ILP32 aarch64 there is no
> > libnss_compat system library, the load of libnss_compat fails and I get these
> > extra failures:
> > 
> > FAIL: grp/testgrp
> > FAIL: nss/bug17079
> > FAIL: posix/globtest
> > FAIL: pwd/tst-getpw
> That explains the failures of those tests I saw for some MIPS ABIs and 
> noted on the wiki as having unknown cause.  Could you update the 
> architecture-independent section of 
> <https://sourceware.org/glibc/wiki/Release/2.26> with a discussion of 
> these failures and the circumstances under which they occur (probably 
> linking to your message)?  (Of course, we should fix them for the future, 
> and hopefully there will be a fix suitable to backport to the release 
> branch.)

FYI: I have updated the wiki page.

Steve Ellcey
sellcey@cavium.com

Re: nis compat depreciation and test failures

[Steve Ellcey]

On Wed, 2017-08-16 at 16:32 -0400, Carlos O'Donell wrote:
> 
> I think that's also a good idea.
> 
> In summary:
> 
> (a) Make tests independent of host configuration.
> 
> (b) Add more tests to nss/ subdirectory to test for things like the
>     behaviour when specifying a missing NSS module.
> 
> (c) Remove 'compat' from DEFAULT_CONFIG for all glibc-built services.


I have no idea on how to do (a) or what kind of tests we would need for
(b) but (c) seems easy enough.  If I change DEFAULT_CONFIG in
nss/grp-lookup.c, nss/pwd-lookup.c, and nss/spwd-lookup.c from
"compat [NOTFOUND=return] files" to just "files", the tests work fine
on my x86 and aarch64 boxes.  Does that seem like the right
DEFAULT_CONFIG?  If so, I can submit a patch.

Steve Ellcey
sellcey@cavium.com

Re: nis compat depreciation and test failures

[DJ Delorie]

Steve Ellcey <sellcey@cavium.com> writes:
> I have no idea on how to do (a) or what kind of tests we would need for

The new framework I recently committed to nss/ allows individual tests
to request one or more "test" services and provide the specific data
they need for those services.  So the test includes both the getpw*()
calls and the pwent tables it expects.

So instead of specifying "files" and using the hosts /etc/passwd, you'd
specify "test1" and provide an in-test equivalent of /etc/passwd.

See nss/tst-nss-test2.c for an example.

> (b) but (c) seems easy enough.  If I change DEFAULT_CONFIG in
> nss/grp-lookup.c, nss/pwd-lookup.c, and nss/spwd-lookup.c from
> "compat [NOTFOUND=return] files" to just "files", the tests work fine
> on my x86 and aarch64 boxes.  Does that seem like the right
> DEFAULT_CONFIG?  If so, I can submit a patch.

I suspect just "files" is a reasonable default *for glibc* since that's
pretty much the only service we build by default now.  Distros of course
should provide defaults appropriate for the services they're including
by default (ldap, nis, krb5, whatever) but that's outside of our realm.

Re: nis compat depreciation and test failures

[Carlos O'Donell]

On 08/17/2017 04:06 PM, DJ Delorie wrote:
> Steve Ellcey <sellcey@cavium.com> writes:
>> I have no idea on how to do (a) or what kind of tests we would need for
> 
> The new framework I recently committed to nss/ allows individual tests
> to request one or more "test" services and provide the specific data
> they need for those services.  So the test includes both the getpw*()
> calls and the pwent tables it expects.
> 
> So instead of specifying "files" and using the hosts /etc/passwd, you'd
> specify "test1" and provide an in-test equivalent of /etc/passwd.
> 
> See nss/tst-nss-test2.c for an example.
> 
>> (b) but (c) seems easy enough.  If I change DEFAULT_CONFIG in
>> nss/grp-lookup.c, nss/pwd-lookup.c, and nss/spwd-lookup.c from
>> "compat [NOTFOUND=return] files" to just "files", the tests work fine
>> on my x86 and aarch64 boxes.  Does that seem like the right
>> DEFAULT_CONFIG?  If so, I can submit a patch.
> 
> I suspect just "files" is a reasonable default *for glibc* since that's
> pretty much the only service we build by default now.  Distros of course
> should provide defaults appropriate for the services they're including
> by default (ldap, nis, krb5, whatever) but that's outside of our realm.

I agree, I think "files" is just about the only thing we should list in
a default fallback.

-- 
Cheers,
Carlos.