[Bug libc/12926] New: getaddrinfo()/make_request() may spin forever

[Bug libc/12926] New: getaddrinfo()/make_request() may spin forever

[ppluzhnikov at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12926

           Summary: getaddrinfo()/make_request() may spin forever
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: ppluzhnikov@google.com


I am not sure whether this is a kernel bug or a glibc bug.

Even if it is a kernel bug, it might be wise to fix it in glibc, since
buggy kernels are observed "in the wild".

Over last two weeks, we observed several Java programs that have a thread
spinning here:

#0  0xf7717430 in __kernel_vsyscall ()
#1  0xf757e448 in recvmsg () at ../sysdeps/unix/sysv/linux/i386/socket.S:97
#2  0xf76797a6 in make_request (fd=1590, pid=<value optimized out>,
seen_ipv4=0x7dceea7b, seen_ipv6=0x7dceea7a, in6ai=0x7dceea70,
in6ailen=0x7dceea6c) at ../sysdeps/unix/sysv/linux/check_pf.c:123
#3  0xf7679bd4 in __check_pf (seen_ipv4=0x7dceea7b, seen_ipv6=0x7dceea7a,
in6ai=0x7dceea70, in6ailen=0x7dceea6c) at
../sysdeps/unix/sysv/linux/check_pf.c:275
#4  0xf761ff4c in getaddrinfo (name=0x988f330 "<a-valid-host-name>",
service=0x0, hints=0x7dceeaf8, pai=0x7dceeb18) at
../sysdeps/posix/getaddrinfo.c:2109
...

strace shows un-ending stream of recvmsg() calls, which all return 0.

Looking in check_pf.c, it seems clear that if recvmsg() keeps returning 0,
then the for() loop on line 131 (current git source) will not execute,
and the do-while loop on line 113 will spin forever.

We do not know what conditions provoke the kernel (2.6.34-smp and 2.6.26-smp
have been observed) to return zero here, and the problem is not repeatable
on the same machine.


The following patch will make make_request() fail under these conditions.
Not sure whether that's the right thing to do.

diff --git a/sysdeps/unix/sysv/linux/check_pf.c
b/sysdeps/unix/sysv/linux/check_pf.c
index c053adc..47cf034 100644
--- a/sysdeps/unix/sysv/linux/check_pf.c
+++ b/sysdeps/unix/sysv/linux/check_pf.c
@@ -121,7 +121,7 @@ make_request (int fd, pid_t pid, bool *seen_ipv4, bool
*seen_ipv6,
        };

       ssize_t read_len = TEMP_FAILURE_RETRY (__recvmsg (fd, &msg, 0));
-      if (read_len < 0)
+      if (read_len <= 0)
        goto out_fail;

       if (msg.msg_flags & MSG_TRUNC)

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[nick.jones@network-box.com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12926

nick.jones@network-box.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nick.jones@network-box.com

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[ppluzhnikov at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12926

Paul Pluzhnikov <ppluzhnikov at google dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID

--- Comment #1 from Paul Pluzhnikov <ppluzhnikov at google dot com> 2011-07-16 05:42:43 UTC ---
Further investigation showed that this is most likely an application (Java NIO
actually) bug.

It appears that NIO, under some conditions, may execute the following sequence
of calls:

1. accept(...) = N
2. close(N)
3. dup2(X, N) = N

which opens a race: if between 2 and 3 another thread opens N (e.g. a NETLINK
socket), that thread will lose.

In our case, the victim happened to be DNS resolver thread.

Since there is nothing glibc can really do to protect itself from such
application code, resolving as invalid.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=12926

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #2 from Rich Felker <bugdal at aerifal dot cx> 2011-07-17 03:32:00 UTC ---
Applications with this bug can be fixed not to close the old file descriptor
before using dup2 to replace it. If there is no old file descriptor,
fcntl/F_DUPFD can be used as a non-destructive dup2 that will choose the next
available fd >= the requested fd if the requested fd is taken.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #4 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, master has been updated
       via  fda389c8f0311dd5786be91a7b54b9f935fcafa1 (commit)
      from  fcb32af153a745414b0d949e707c9485ab77d6ba (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=fda389c8f0311dd5786be91a7b54b9f935fcafa1

commit fda389c8f0311dd5786be91a7b54b9f935fcafa1
Author: Siddhesh Poyarekar <siddhesh@redhat.com>
Date:   Tue Oct 14 21:05:33 2014 +0530

    Fix infinite loop in check_pf (BZ #12926)

    The recvmsg could return 0 under some conditions and cause the
    make_request function to be stuck in an infinite loop.

    Thank you Jim King <jim.king@simplivity.com> for posting Paul's patch
    on the list.

-----------------------------------------------------------------------

Summary of changes:
 ChangeLog                          |    6 ++++++
 NEWS                               |    2 +-
 sysdeps/unix/sysv/linux/check_pf.c |    2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[siddhesh at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
                 CC|                            |siddhesh at redhat dot com
         Resolution|---                         |FIXED

--- Comment #5 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
Fixed in master.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[fengtiantian at huawei dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

fengtiantian <fengtiantian at huawei dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
                 CC|                            |fengtiantian at huawei dot com
            Version|2.15                        |2.13
         Resolution|FIXED                       |---

--- Comment #7 from fengtiantian <fengtiantian at huawei dot com> ---
I met the same error when I use python programs, and a thread spinning here:
#0  0x00007ff4ae826dd0 in __recvmsg_nocancel () from /lib64/libc.so.6
#1  0x00007ff4ae848891 in make_request () from /lib64/libc.so.6
#2  0x00007ff4ae848daa in __check_pf () from /lib64/libc.so.6
#3  0x00007ff4ae811a23 in getaddrinfo () from /lib64/libc.so.6
#4  0x00007ff4ad919fd1 in setipaddr () from
/usr/lib64/python2.6/lib-dynload/_socket.so
#5  0x00007ff4ad91abdf in getsockaddrarg () from
/usr/lib64/python2.6/lib-dynload/_socket.so
#6  0x00007ff4ad91b086 in sock_sendto () from
/usr/lib64/python2.6/lib-dynload/_socket.so

the OS is suse 11 SP3 ,kernel 3.0.93.and the glibc version is
glibc-2.11.3-17.82.11, which have emerge the patch make-request-loop.patch

  1 2014-10-14  Paul Pluzhnikov  <ppluzhnikov@google.com>
  2
  3     [BZ #12926]
  4     * sysdeps/unix/sysv/linux/check_pf.c (make_request): Avoid
  5     infinite loop when __recvmsg returns 0.
  6
  7 Index: glibc-2.11.3/sysdeps/unix/sysv/linux/check_pf.c
  8 ===================================================================
  9 --- glibc-2.11.3.orig/sysdeps/unix/sysv/linux/check_pf.c
 10 +++ glibc-2.11.3/sysdeps/unix/sysv/linux/check_pf.c
 11 @@ -219,7 +219,7 @@ make_request (int fd, pid_t pid, bool *s
 12     };
 13
 14        ssize_t read_len = TEMP_FAILURE_RETRY (__recvmsg (fd, &msg, 0));
 15 -      if (read_len < 0)
 16 +      if (read_len <= 0)
 17     goto out_fail;
 18
 19        if (msg.msg_flags & MSG_TRUN 

So I think the patch do not fix my bug.

I do not understand ,in make_request function, while it use the (__recvmsg (fd,
&msg, 0),the blocking mode , and do not set the timeout. If the kernel not send
the NLMSG_DONE message, the __recvmsg will hung a long time.

I am not sur this is a kernel bug or glibc bug,if glibc think the kernel not
send the NLMSG_DONE message is a bug ?  I see in kernel source, some exception
branch is will not send the NLMSG_DONE .

can glibc  use the MSG_DONTWAIT flag to receive the message? If not receive the
message, return false?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/12926] getaddrinfo()/make_request() may spin forever

[jim.king at simplivity dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #8 from James E. King, III <jim.king at simplivity dot com> ---
So we originally believed that we had seen this as well which is why I found
and re-posted this patch.  This patch is still correct, if you examine the man
page for recv a zero response is possible and was not properly handled. 
Recommend you examine https://sourceware.org/bugzilla/show_bug.cgi?id=15946 and
see if that is the root cause of your issue.  In that defect a file descriptor
is not handled properly by glibc and that turned out to be the root cause of
the issue.  Since we pulled in that specific patch we have not seed the
check_pf hang again and we were able to reproduce it rather easily before.  If
the other issue resolves this, please post here letting us know so we can
re-resolve this.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[jsm28 at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|libc                        |network

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|security-                   |security?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[jim.king at simplivity dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #12 from James E. King, III <jim.king at simplivity dot com> ---
The issue I originally experienced was causes by defect 15946, however I still
believe that since the interface documentation for "recvmsg" clearly states
that 0 is a valid return for a disconnection then the glibc code must honor
that even if the kernel is not expected to use it.  The kernel might change
because the interface documentation says that it can, and a future kernel
change might then break things in glibc, where glibc would enter an infinite
loop.  So even if we never expect 0 to come back from recvmsg on the NETLINK
socket, we still need to code for it because it is a valid response.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #13 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Carlos O'Donell from comment #11)

> How can we assume all supported kernels from 2.6.32 and up are not
> vulnerable? AFAIK glibc has to be defensive in this case.

We could probably make an exception here.  In general, I really dislike the
idea of working around kernel security bugs, but this might be an exception. 
(Although there are likely corner cases the libc check won't cover.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #14 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Florian Weimer from comment #13)
> (In reply to Carlos O'Donell from comment #11)
> 
> > How can we assume all supported kernels from 2.6.32 and up are not
> > vulnerable? AFAIK glibc has to be defensive in this case.
> 
> We could probably make an exception here.  In general, I really dislike the
> idea of working around kernel security bugs, but this might be an exception.
> (Although there are likely corner cases the libc check won't cover.)

We can conditionlize the code on upstream kernel version with the fix, similar
to the way we conditionalize with interface availability and eventually remove
the code when the minimum kernel version is new enough.

I don't like it either, but it is a fact of developing on a library with a wide
range of supported kernel versions.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[joseph at codesourcery dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #15 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
Note that when 2.6.32 ceases to be maintained 
(<https://www.kernel.org/category/releases.html> still says "Mid-2015") I 
intend to propose moving to 3.2 as minimum kernel version, though it seems 
that still predates the kernel fix in question.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/12926] getaddrinfo()/make_request() may spin forever

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12926

--- Comment #16 from Florian Weimer <fweimer at redhat dot com> ---
Revised patch posted:
https://sourceware.org/ml/libc-alpha/2015-10/msg00865.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.