On the removal of nscd from Fedora, and the future of nscd.

On the removal of nscd from Fedora, and the future of nscd.

[Carlos O'Donell]

In Fedora 36 we officially removed nscd from the distribution with this
system-wide change request:
https://fedoraproject.org/wiki/Changes/RemoveNSCD

I say "we" but it was the entire Fedora glibc team, which includes various
developers also on this list like Arjun, Florin, and DJ. We took this action
because over time glibc should focus on the important parts of being a core
runtime. We felt sssd and systemd had grown to the point where they were
well supported and and supporting nscd in Fedora was no longer required.

Ludovic reached out to me regarding the removal because Guix uses nscd
for the interesting use case of supporting multiple system libcs:
https://guix.gnu.org/manual/devel/en/html_node/Application-Setup.html#Name-Service-Switch-1

The idea is very clever in that if you always make the lookup in the system
nscd then the process with the alternative libc is decoupled from the system
libc and all the system plugins required to answer the lookup request.

Ludovic and I compared notes on the future of nscd, and we discussed a what-if
scenario. What if we kept nscd, but removed all the caches, and left it as a
daemon that could answer requests from system processes but the only goal was
to decouple the libc required to service the request and the libc in the process?

This is already starting to sound like Florian's getent idea where by a static
process uses a known binary to carry out the NSS lookup and resolution [1].

What if instead of getent we just thinned down nscd to something we could audit
and trust? If we really wanted to we could even compile the same code into getent
as a fallback e.g. try nscd first (long lived) then getent (short lived).

You might say "Well then nscd is mis-named!" I'd argue it's a "cache" for the shared
objects and state required to do the lookup, but not the data itself, and if you
didn't run nscd then you'd go through NSS normally, or "getent" if you were a
statically compiled binary.

Thoughts?

-- 
Cheers,
Carlos.

[1] https://sourceware.org/legacy-ml/libc-alpha/2017-12/msg00831.html

Re: On the removal of nscd from Fedora, and the future of nscd.

[DJ Delorie]

"Carlos O'Donell" <carlos@redhat.com> writes:
> Ludovic reached out to me regarding the removal because Guix uses nscd
> for the interesting use case of supporting multiple system libcs:
> https://guix.gnu.org/manual/devel/en/html_node/Application-Setup.html#Name-Service-Switch-1

I think this is a weak argument.  If Guix can dlopen the host system's
objects, you risk incompatibilities anyway.  And we don't have a rule
that says that nsswitch modules have to be built against the exact
version of glibc you're running against, although if the module is too
new there may be problems.  But Guix would have this problem anyway if
both sets of objects are available.  If Guix wants to run its own glibc,
it needs to be isolated better - meaning, its own nscd, its own path to
nss modules, etc.  Almost like a container or flatpack.  It may not even
be able to parse /etc/nsswitch.conf if the host system is newer.

> You might say "Well then nscd is mis-named!" I'd argue it's a "cache" for the shared
> objects and state required to do the lookup, but not the data itself, and if you
> didn't run nscd then you'd go through NSS normally, or "getent" if you were a
> statically compiled binary.

If we want a separate service and API for name lookups, outside of
glibc, let's discuss.  But recently we've been moving such services
*into* glibc, to better support statically linked applications.  I don't
see an easy path forward here.

Re: On the removal of nscd from Fedora, and the future of nscd.

[Joseph Myers]

On Mon, 28 Feb 2022, DJ Delorie via Libc-alpha wrote:

> If we want a separate service and API for name lookups, outside of
> glibc, let's discuss.  But recently we've been moving such services
> *into* glibc, to better support statically linked applications.  I don't
> see an easy path forward here.

We've moved certain NSS modules such as nss_files and nss_dns into libc.  
I think there is a reasonable case that all NSS modules that aren't in 
libc (especially those not shipped with glibc) should be loaded only in a 
separate process, not in the main process doing the name lookup, to avoid 
issues with name lookup loading more or less arbitrary libraries into the 
process doing the lookup.

That doesn't say what the interface should look like to any separate 
process that handles doing lookup with not-in-libc NSS modules.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: On the removal of nscd from Fedora, and the future of nscd.

[DJ Delorie]

Joseph Myers <joseph@codesourcery.com> writes:
> I think there is a reasonable case that all NSS modules that aren't in 
> libc (especially those not shipped with glibc) should be loaded only in a 
> separate process, not in the main process doing the name lookup, to avoid 
> issues with name lookup loading more or less arbitrary libraries into the 
> process doing the lookup.

Hmm... I hadn't thought of it that way, but the security implications
are compelling... and we don't hardcode the search path either, so the
user can override it via environment (unless setuid), I think.

 {
    char *shlib_name;
    if (__asprintf (&shlib_name, "libnss_%s.so%s",
                    module->name, __nss_shlib_revision) < 0)
      /* This is definitely a temporary failure.  Do not update
         module->state.  This will trigger another attempt at the next
         call.  */
      return false;

    handle = __libc_dlopen (shlib_name);
    free (shlib_name);
  }

Re: On the removal of nscd from Fedora, and the future of nscd.

[Ludovic Courtès]

Hello,

Thank you Carlos for the introduction!

DJ Delorie <dj@redhat.com> skribis:

> "Carlos O'Donell" <carlos@redhat.com> writes:
>> Ludovic reached out to me regarding the removal because Guix uses nscd
>> for the interesting use case of supporting multiple system libcs:
>> https://guix.gnu.org/manual/devel/en/html_node/Application-Setup.html#Name-Service-Switch-1
>
> I think this is a weak argument.  If Guix can dlopen the host system's
> objects, you risk incompatibilities anyway.  And we don't have a rule
> that says that nsswitch modules have to be built against the exact
> version of glibc you're running against, although if the module is too
> new there may be problems.  But Guix would have this problem anyway if
> both sets of objects are available.  If Guix wants to run its own glibc,
> it needs to be isolated better - meaning, its own nscd, its own path to
> nss modules, etc.  Almost like a container or flatpack.  It may not even
> be able to parse /etc/nsswitch.conf if the host system is newer.

To be clear, Guix applications have been used on top of other distros
just fine.  This nscd requirement is one of the few must-haves to
ensure, as Joseph writes, that processes (in particular those linked
against Guix’s libc) do not end up dlopening arbitrary, possibly
incompatible libraries.

(The problem applies to other deployment tools that are not limited to a
single system libc such as Nix.)

I like the idea of keeping a trimmed-down nscd as Carlos wrote.  Calling
out to ‘getent’ would also work, but it potentially means having one
‘getent’ process per application, if I understand Florian’s proposal
correctly.

Thanks,
Ludo’.

Re: On the removal of nscd from Fedora, and the future of nscd.

[DJ Delorie]

Ludovic Courts <ludo@gnu.org> writes:
> This nscd requirement is one of the few must-haves to ensure, as
> Joseph writes, that processes (in particular those linked against
> Guix’s libc) do not end up dlopening arbitrary, possibly incompatible
> libraries.

My point is, if there's a risk that a Guix binary *could* load a host
dso, then Guix is insufficiently isolated from the host system.  nscd is
just one example of how this could happen.  If you accept this
non-isolation, you need to accept that dsos need to be cross-usable.

That Guix is accepting some sharing (like /etc/nsswitch.conf) but not
all sharing (like libnss_*.so) opens Guix up to these types of problems,
and thus, the onus is mostly on Guix to deal with them.

Hence my feeling that "keep ncsd around because we're not isolated
enough" is a weak (but still valid) reason to keep nscd.  nscd is just
the most obvious problem point, but not the only problem point.  The
right solution is to complete the isolation, although I admit that may
be an impossible goal at this point.

Note that I don't feel strongly about the above; just pointing it out.

Re: On the removal of nscd from Fedora, and the future of nscd.

[Ludovic Courtès]

DJ Delorie <dj@redhat.com> skribis:

> Ludovic Courts <ludo@gnu.org> writes:
>> This nscd requirement is one of the few must-haves to ensure, as
>> Joseph writes, that processes (in particular those linked against
>> Guix’s libc) do not end up dlopening arbitrary, possibly incompatible
>> libraries.
>
> My point is, if there's a risk that a Guix binary *could* load a host
> dso, then Guix is insufficiently isolated from the host system.  nscd is
> just one example of how this could happen.  If you accept this
> non-isolation, you need to accept that dsos need to be cross-usable.

Actually, by default, a Guix program on (say) Fedora-without-nscd won’t
dlopen Fedora’s libnss_sssd.so.  Instead, it’ll search in vain for
libnss_sssd.so in its search path (which does not include /usr/lib), and
have its name lookups fail with EAI_SYSTEM, ENOENT, or some other
unclear error.

IMO the situation of NSS is singular.  The use case of Guix, Nix, and
others is also very real, used today just fine even without full
isolation (separate namespaces & co.).

I hope this clarifies the context.

Ludo’.

Re: On the removal of nscd from Fedora, and the future of nscd.

[DJ Delorie]

Ludovic Courts <ludo@gnu.org> writes:
> Actually, by default, a Guix program on (say) Fedora-without-nscd won’t
> dlopen Fedora’s libnss_sssd.so.  Instead, it’ll search in vain for
> libnss_sssd.so in its search path (which does not include /usr/lib), and
> have its name lookups fail with EAI_SYSTEM, ENOENT, or some other
> unclear error.

Ah, that's different - mostly.  It still means you're using the host's
/etc/nsswitch.conf, but that just exposes a different problem - how to
handle site-wide services in isolated "programs" (flatpacks, containers,
chroots, statically linked apps, Guix).

NSCD certainly could act as that gateway, but I'd hate to rely on it
without an RFC defining the protocol, and such an RFC would enable it to
solve the problem in a wider context too.

Doing so, however, means that nscd (or some other equivalent) MUST be
present and running on all systems...

Re: On the removal of nscd from Fedora, and the future of nscd.

[Ludovic Courtès]

Hi,

DJ Delorie <dj@redhat.com> skribis:

> Ludovic Courts <ludo@gnu.org> writes:
>> Actually, by default, a Guix program on (say) Fedora-without-nscd won’t
>> dlopen Fedora’s libnss_sssd.so.  Instead, it’ll search in vain for
>> libnss_sssd.so in its search path (which does not include /usr/lib), and
>> have its name lookups fail with EAI_SYSTEM, ENOENT, or some other
>> unclear error.
>
> Ah, that's different - mostly.  It still means you're using the host's
> /etc/nsswitch.conf, but that just exposes a different problem - how to
> handle site-wide services in isolated "programs" (flatpacks, containers,
> chroots, statically linked apps, Guix).
>
> NSCD certainly could act as that gateway, but I'd hate to rely on it
> without an RFC defining the protocol, and such an RFC would enable it to
> solve the problem in a wider context too.

Yes.  In practice, I haven’t seen nscd protocol changes in 10 years of
Guix.

> Doing so, however, means that nscd (or some other equivalent) MUST be
> present and running on all systems...

Yes, exactly.

Ludo’.

Re: On the removal of nscd from Fedora, and the future of nscd.

[John Ericson]

Hi, I am coming from NixOS/Nixpkgs and this [PR comment] in particular.
I am *far* from an expert on this NSS, but I have been bitten by it on
a few occasions in conjunction with various sorts of sand-boxing, and I
certainly share the desire for more simplicity and security, hand in hand.

Specifically, it seems like this thread is reaching consensus anyways,
but wanted to back up a few things Ludo was saying.

On 3/3/22 08:40, Ludovic Courtès wrote:
> DJ Delorie <dj@redhat.com> skribis:
>> Ah, that's different - mostly.  It still means you're using the host's
>> /etc/nsswitch.conf, but that just exposes a different problem - how to
>> handle site-wide services in isolated "programs" (flatpacks, containers,
>> chroots, statically linked apps, Guix).

Yes. we have a chance to establish some new idioms here that can be used
in all those contexts (and Nix!). Good and exciting!

>> 
>> NSCD certainly could act as that gateway, but I'd hate to rely on it
>> without an RFC defining the protocol, and such an RFC would enable it to
>> solve the problem in a wider context too.
>> 

Stepping back, we can ask: "What makes a de facto standard?"

I would say a) stability and b) multiple implementations.

> Yes.  In practice, I haven?t seen nscd protocol changes in 10 years of
> Guix.

That's one, and a quick search turned up:

https://github.com/pikhq/musl-nscd

whose read-me says:

> musl-nscd is an implementation of the NSCD protocol. It makes use of NSS
> modules, just like with glibc. This permits alternative backends for the user
> and group databases for musl libc. The protocol it uses is a subset of that
> used by glibc. 

That's the other. Both criteria for "de facto standard" met!

>> Doing so, however, means that nscd (or some other equivalent) MUST be
>> present and running on all systems...
>> 
> Yes, exactly.

If I may push back on this slightly, a libc could should ship with the sssd and
systemd-resolved clients too. For sake of argument, one could imagine they are
even statically linked, so there is no dlopen to be suspicious of. The binary could
try those protocols first before falling back on the nscd protocol.

The goal here, I think, is to support the existing NSCD protocol as the "lingua
franca" we have *today*. It's not to hinder the uptake of anything else, or even a
value judgement on whether the NSCD protocol is "good" in any deep sense,
just acknowledging its what we've got.

----

Having found that musl-nscd, I was curious more generally what the Musl
community thought about this. I found [musl thread] which was referred to back
when the Fedora change was first proposed, and while the thread wasn't super
conclusive, I think it is broadly in agreement with what is discussed here:

- dlopen-ing arbitrary libraries is very bad
- static linking should work just as well
- Keep rump NSCD with caching and other inessential complexity removed

This [reply] a bit buried down from Rich seems to share the exact ethos about
just keeping NSCD for continuity's sake too. That's a good place to end.

Cheers,

John

[PR comment]: https://github.com/NixOS/nixpkgs/pull/155655#issuecomment-1055721088
[musl thread]: https://www.openwall.com/lists/musl/2020/10/19/1
[reply]: https://www.openwall.com/lists/musl/2020/10/23/15

Re: On the removal of nscd from Fedora, and the future of nscd.

[Florian Klink]

>>> NSCD certainly could act as that gateway, but I'd hate to rely on it
>>> without an RFC defining the protocol, and such an RFC would enable it to
>>> solve the problem in a wider context too.
>>>
>
>Stepping back, we can ask: "What makes a de facto standard?"
>
>I would say a) stability and b) multiple implementations.
>
>> Yes.  In practice, I haven?t seen nscd protocol changes in 10 years of
>> Guix.
>
>That's one, and a quick search turned up:
>
>https://github.com/pikhq/musl-nscd
>
>whose read-me says:
>
>> musl-nscd is an implementation of the NSCD protocol. It makes use of NSS
>> modules, just like with glibc. This permits alternative backends for the user
>> and group databases for musl libc. The protocol it uses is a subset of that
>> used by glibc.
>
>That's the other. Both criteria for "de facto standard" met!
>
>>> Doing so, however, means that nscd (or some other equivalent) MUST be
>>> present and running on all systems...
>>>
>> Yes, exactly.
>
>If I may push back on this slightly, a libc could should ship with the sssd and
>systemd-resolved clients too. For sake of argument, one could imagine they are
>even statically linked, so there is no dlopen to be suspicious of. The binary could
>try those protocols first before falling back on the nscd protocol.
>
>The goal here, I think, is to support the existing NSCD protocol as the "lingua
>franca" we have *today*. It's not to hinder the uptake of anything else, or even a
>value judgement on whether the NSCD protocol is "good" in any deep sense,
>just acknowledging its what we've got.

I recently took a closer look at the protocol, while adding host lookup
support to [nsncd][nsncd-pr], so there's plenty of implementations of
this protocol.

[nsncd-blog] describes the reasons why Nix/GUIX needs this *as a pure
request dispatcher* in a bit more detail, too.

There's currently some scary request types to pass around FDs to
internal nscd cache structures (GETFD*). But client code already falls
back gracefully to querying records manually, if you don't actually hand
over a FD.

If there's plans to drop nscd from glibc codebase, what about keeping
the client code, but simplified, without all client-side cache peeking?

That way, Nix, GUIX and other peoples usecases would still keep working,
while the amount of code in glibc on that can be reduced.

Regards,
flokli

[nsncd-pr]: https://github.com/twosigma/nsncd/pull/49
[nsncd-blog]: https://flokli.de/posts/2022-11-18-nsncd/