[Bug nscd/14906] New: inotify failed when /etc/hosts file change

[Bug nscd/14906] New: inotify failed when /etc/hosts file change

[binli at opensuse dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14906

             Bug #: 14906
           Summary: inotify failed when /etc/hosts file change
           Product: glibc
           Version: 2.11
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nscd
        AssignedTo: unassigned@sourceware.org
        ReportedBy: binli@opensuse.org
                CC: drepper.fsp@gmail.com
    Classification: Unclassified


Created attachment 6770
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6770
The test program

This test program shows the problem.  I'm creating a watch on /etc/hosts
(looking for modification or deletion), and it's catching only changes of 
the type "echo '#bla' >> /etc/hosts".  It doesn't catch changes via copy
to temporary file, do changes, rename temp file back to /etc/hosts.  That's
what sed -i is using to atomically do the in place changes.

I thought that adding IN_MOVE should help when setting up the watch,
but it doesn't (commented out in the test program).  Even if it would have
helped I would claim the kernel to be broken.  If I include IN_MODIFY
in a watch for /etc/hosts, I'm of course also interested if somebody renames
some random file to the watched filename, as it very certainly is a
modification.

Basically as soon as "sed -i" is done on the file the kernel lost track
of the watch and no changes to the "new" /etc/hosts are tracked anymore.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[binli at opensuse dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14906

--- Comment #1 from Bin Li <binli at opensuse dot org> 2012-12-03 10:38:53 UTC ---
The original bug is from SUSE Linux.
https://bugzilla.novell.com/show_bug.cgi?id=779531

Michael Matz's comments.
--------------------------------
I can make the program work as expected also in the light of overwriting
renames, when I'm handling a IN_IGNORED event by recreating the watch on
/etc/hosts, like this in the inner loop:

      if (buf.wd == watch && (buf.mask & IN_IGNORED) != 0)
        {
          /* Recreate watch */
          watch = inotify_add_watch (fd, "/etc/hosts", IN_DELETE_SELF |
IN_MODIFY /* | IN_MOVE */);
          printf ("Added watch on /etc/hosts, as watch %d\n", watch);
          if (watch < 0)
            bark ("inotify_add_watch");
        }

That's for sure surprising limitation of the interface.  Is this really how
it's supposed to be used?
--------------------------------

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[binli at opensuse dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14906

--- Comment #2 from Bin Li <binli at opensuse dot org> 2012-12-03 10:40:11 UTC ---
Miklos Szeredi 2012-10-15 15:27:29 UTC

(In reply to comment #20)
> I can make the program work as expected also in the light of overwriting
> renames, when I'm handling a IN_IGNORED event by recreating the watch on
> /etc/hosts, like this in the inner loop:
> 
>       if (buf.wd == watch && (buf.mask & IN_IGNORED) != 0)
>         {
>           /* Recreate watch */
>           watch = inotify_add_watch (fd, "/etc/hosts", IN_DELETE_SELF |
> IN_MODIFY /* | IN_MOVE */);
>           printf ("Added watch on /etc/hosts, as watch %d\n", watch);
>           if (watch < 0)
>             bark ("inotify_add_watch");
>         }
> 
> That's for sure surprising limitation of the interface.  Is this really how
> it's supposed to be used?

The watch descriptor returned by inotify_add_watch() refers to the inode, not
the path name.  This is analogous to open(2) returning a file descriptor that
refers to the inode, not the pathname.

So yes, adding IN_DELETE_SELF and IN_MOVE_SELF to the watches is a good way to
monitor whether changes to the file _name_ were made.

Note: inotify is an unreliable interface by design and the application should
provide a backup mechanism for theoretical cases when events are lost (e.g. by
checking the modification time of the file periodically).

Also see "Limitations and caveats" section in the inofity(7) manpage.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[binli at opensuse dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14906

--- Comment #3 from Bin Li <binli at opensuse dot org> 2012-12-03 10:42:29 UTC ---
Created attachment 6771
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6771
the patch for suggested fix

Miklos Szeredi 2012-10-18 11:04:49 UTC

I looked at the inotify code in nscd code and it's full of bugs.  Apparently
the cache pruning by inotify wasn't tested to any useful extent.  And latest
upstream version carries all those bugs too.  We can fix all these but that's a
whole little project.  Or there are some simpler options:

a) disable inotify completely and do it the old way (check file modification
times regularly)

b) enable inotify but also check modification times in case the inotify code
fails to work.

c) option b) plus something like comment 20

The attached patch does option b).  I've not tested it.

The glibc package maintainers will have to make the decision on how to fix it,
this patch is just a suggestion.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[binli at opensuse dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14906

--- Comment #4 from Bin Li <binli at opensuse dot org> 2012-12-03 10:44:32 UTC ---
Hi, Maintainer

 I just coy some analyze from Novell's bugzilla, and could you help review the
patch in comment#3? Thanks a lot!

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14906

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at redhat dot com

--- Comment #5 from Carlos O'Donell <carlos at redhat dot com> ---
Created attachment 8137
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8137&action=edit
glibc-14906.diff

The following patch reimplements the inotify support in glibc's nscd.

It expands the support to include watching the parent directory for the
creation, deltion, and move-in of the configuration file under watch.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14906

--- Comment #6 from Carlos O'Donell <carlos at redhat dot com> ---
Final version posted on libc-alpha for review:

https://sourceware.org/ml/libc-alpha/2015-02/msg00504.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=14906

--- Comment #7 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, master has been updated
       via  cf9313e7d1dd42addd6cf8c9277f0f18a62cdeff (commit)
      from  7d67a196b6548562070ac11fc673c0d3d263d846 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=cf9313e7d1dd42addd6cf8c9277f0f18a62cdeff

commit cf9313e7d1dd42addd6cf8c9277f0f18a62cdeff
Author: Carlos O'Donell <carlos@systemhalted.org>
Date:   Fri Mar 13 09:49:24 2015 -0400

    Enhance nscd's inotify support (Bug 14906).

    In bug 14906 the user complains that the inotify support in nscd
    is not sufficient when it comes to detecting changes in the
    configurationfiles that should be watched for the various databases.

    The current nscd implementation uses inotify to watch for changes in
    the configuration files, but adds watches only for IN_DELETE_SELF and
    IN_MODIFY. These watches are insufficient to cover even the most basic
    uses by a system administrator. For example using emacs or vim to edit
    a configuration file should trigger a reload but it might not if
    the editors use move to atomically update the file. This atomic update
    changes the inode and thus removes the notification on the file (as
    inotify is based on inodes). Thus the inotify support in nscd for
    configuration files is insufficient to account for the average use
    cases of system administrators and users.

    The inotify support is significantly enhanced and described here:
    https://www.sourceware.org/ml/libc-alpha/2015-02/msg00504.html

    Tested on x86_64 with and without inotify support.

-----------------------------------------------------------------------

Summary of changes:
 ChangeLog                  |   24 +++
 NEWS                       |   12 +-
 nscd/cache.c               |   28 +++-
 nscd/connections.c         |  358 ++++++++++++++++++++++++++++++++------------
 nscd/nscd.h                |   60 +++++++-
 nss/nss_db/db-init.c       |   28 +---
 nss/nss_files/files-init.c |   46 +++---
 7 files changed, 398 insertions(+), 158 deletions(-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug nscd/14906] inotify failed when /etc/hosts file change

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14906

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #8 from Carlos O'Donell <carlos at redhat dot com> ---
Fixed. Please open new bugs if you have problems with the new
inotify/stat-based support, it should cover all scenarios with file changes.

-- 
You are receiving this mail because:
You are on the CC list for the bug.