[Bug network/14308] New: getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[Bug network/14308] New: getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[law at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14308

             Bug #: 14308
           Summary: getaddrinfo DNS referral response returns host not
                    found when A and AAAA questions are sent and one
                    response is a referral
           Product: glibc
           Version: 2.15
            Status: NEW
          Severity: normal
          Priority: P2
         Component: network
        AssignedTo: unassigned@sourceware.org
        ReportedBy: law@redhat.com
    Classification: Unclassified


Created attachment 6494
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6494
Wireshark Packet Capture

[ This is a cut-n-paste from a bug report in Red Hat's bugzilla database.
Refiling it here on Gino's behalf as this is really the right place. ]


I'm seeing a very edge-case issue with Linux hosts complaining about "host not
found" name resolution errors when using the getaddrinfo() API call as provided
by glibc. When getaddrinfo() is called on hosts with the IPv6 module loaded, it
sends a parallel query for A and AAAA records. If one of the responses is a
referral response and the other is not (e.g. A is not-referral but AAAA is
referral), the function will terminate and return value -2, "No address
associated with hostname" error.

Version-Release number of selected component (if applicable):

Any Linux host running glibc 2.9 or later
Affected distributions:
- Fedora 10-16, rawhide
- EL 6.0 and later

How reproducible:
100% under specific conditions:
- Linux running glibc 2.9 or later with IPv6 enabled
- Citrix Netscaler as a DNS server

Steps to Reproduce:
1. Install Fedora 16 (e.g. Live ISO)
2. Setup and install any DNS server (e.g. ISC Bind) and configure it to be an
authoritative name server for a zone (e.g. example.com)
3. Setup and install a Citrix Netscaler load-balancer (e.g. VPX appliance) and
configure a DNS proxy VIP
4. Enable DNS caching on the Citrix Netscaler load-balancer (set dns parameter
-cache YES)
5. Configure the Linux host to use the DNS VIP as its resolver
6. Disable nscd on the Linux host to bypass name caching (alternatively, use
nscd -i hosts in between calls)
7. Run a program that uses getaddrinfo() (e.g. curl, wget, telnet, etc) to
resolve a hostname

Actual results:
curl -v hostname.example.com
* getaddrinfo(3) failed for hostname.example.com:80
* Couldn't resolve host 'hostname.example.com
* Closing connection #0
curl: (6) Couldn't resolve host 'hostname.example.com'

Expected results:
Connection should be fine

Additional Info:
A buggy DNS server is exposing an edge-case bug in glibc

Related Bugs:
Citrix SR 60783234

[reply] [-]
Private
Comment 1 Gino LV. Ledesma 2012-05-22 21:14:19 EDT

Created attachment 586223 [details]
Wireshark Packet Capture

[reply] [-]
Private
Comment 2 Gino LV. Ledesma 2012-05-22 21:14:34 EDT

I took a packet capture of the DNS request-response scenario (dns.pcap) and
found the following

Case 1: Response is served fresh (not from cache)
Frame 01    Client:    A?    active-mrepo.me.com
Frame 02    Client:    AAAA?    active-mrepo.me.com
Frame 03    VIP:    A    1/1/1 active-mrepo.me.com. 17.172.194.16
Frame 04    VIP:    AAAA    0/1/0 hostmaster... 

Case 2: Response is served from cache
Frame 05    Client:    A?    active-mrepo.me.com
Frame 06    Client: AAAA?    active-mrepo.me.com
Frame 07    VIP:    A    1/0/0 active-mrepo.me.com. 17.172.194.16
Frame 08    VIP:    AAAA    0/0/0 hostmaster... 

# Try 2 (this is done automatically by glibc)
Frame 09    Client:    A?    active-mrepo.me.com
Frame 10    Client:    AAAA?    active-mrepo.me.com
Frame 11    VIP:    A    1/0/0 active-mrepo.me.com. 17.172.194.16
Frame 12    VIP:    AAAA    0/0/0 hostmaster... 

As best as I can tell, there are two things happening here:
1) This DNS server is serving a DNS response that falls under NODATA type 3
(RFC 2308 section 2.2)
2) glibc's (mis-?) interpretation of a referral response and short-circuiting
its logic

glibc interprets DNS responses as "referral" if the following conditions are
met (see glibc 2.15, resolv/res_send.c lines 1301-1303):
a) rcode == NOERROR
b) ancount == 0
c) aa == 0
d) ra == 0
e) arcount == 0

I see that this change was introduced in glibc 2.9 and is still present in
2.15.

In the above situation when two responses come in (A response = authoritative,
AAAA = referral), send_dg() immediately returns 0, causing __libc_res_nsend to
try the next nameserver and repeat the query, ignoring any valid responses that
may have come in.

--

Here is a debug call-trace of glibc with RES_DEBUG enabled:

looking up: active-mrepo
;; res_setoptions(" timeout:600 debug
", "conf")..
;;    debug
dots=0, statp->ndots=1, trailing_dot=0, name=active-mrepo
;; res_nquerydomain(active-mrepo, me.com, 1, 62321)
;; res_query(active-mrepo.me.com, 1, 62321)
;; res_nmkquery(QUERY, active-mrepo.me.com, IN, A)
;; res_nmkquery(QUERY, active-mrepo.me.com, IN, AAAA)
;; res_send()
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49477
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;;    active-mrepo.me.com, type = A, class = IN
;; Querying server (# 1) address = 17.230.128.24
referred query:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8813
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;;    active-mrepo.me.com, type = AAAA, class = IN
;; got answer:
;; ns_initparse: Message too long
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8813
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;;    active-mrepo.me.com, type = AAAA, class = IN

The call trace won't show the A response because the AAAA referral response
returns 0 immediately.

Adjusting the next_ns: goto label to be moved above the if(buf2 != null) under
the SERVAIL/NOTIMP/REFUSED code checks seems like a work-around but is most
likely incorrect (due to the subsequent goto wait call).

--

On a side note, I can only make this happen with the Citrix Netscaler acting as
a DNS cache. Other caching resolvers (bind, dnsmasq, dnscache, pdns-recursor,
etc) do not expose this behavior in glibc because they will either have the
"aa" or "ra" bits set to 1. The Netscaler seems to be unique in serving the
following combination of flags for type=AAAA:

rcode == NOERROR
ancount == 0
ra == 0
arcount == 0
nscount=0

I've filed a bug with Citrix for this problem: Citrix SR 60783234

--


Possibly related bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=459756
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/326718

--


Work-arounds:
1. Disable Netscaler caching (set dns param -cache NO)
2. Switch service type from DNS/DNS_TCP to UDP/TCP
3. Make Netscaler authoritative for all zones that its DNS proxying for (add ns
soaRecord ...)
4. Disable IPv6 on client-side

Some notes / tidbits:
1. This behavior does NOT affect older glibc hosts (e.g. EL 5.x), presumably
because gethostbyname3_r does two separate calls for A/AAAA
2. The order of the response (whether A or AAAA comes first) doesn't seem to
matter


--




I just installed Fedora 17 i386 on a VM to test this. Problem still persists
there (glibc-2.15-37.fc17.i686).

I've also confirmed that the problem exists with upstream, stock glibc 2.15.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[phattanon at nettree dot co.th]

http://sourceware.org/bugzilla/show_bug.cgi?id=14308

Phattanon Duangdara <phattanon at nettree dot co.th> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |phattanon at nettree dot
                   |                            |co.th

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[gauryogesh.nsit at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14308

Yogesh Gaur <gauryogesh.nsit at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gauryogesh.nsit at gmail
                   |                            |dot com

--- Comment #1 from Yogesh Gaur <gauryogesh.nsit at gmail dot com> 2013-02-09 08:50:17 UTC ---
Hello All,

This issue is coming because of both message type of T_A and T_AAAA are being
send in parallel for processing through api _nss_dns_gethostbyname4_r().

Actually following conditions happens before coming this issue:
1> Receive request of type T_A, creating session table entry.
                        T_A -- Host address
This is non-referral response and would create its own session table entry.
2> Receive request of type T_AAAA on port x, reusing session table entry from
#1
                        T_AAAA -- Ip6 Address
This is referral response and would reuse previous created session table entry.
3> Respond to T_AAAA request on port x and remove session table entry.
4> Load-balancer receives response from DNS server for T_A request, but since
session table entry is gone, it simply forwards the traffic. 
Client receives a reply from a different IP not from the IP it expects and
simply ignores it.
5> This then lead to message failure and returns with error-code as EAI_NODATA
i.e. No address associated with hostname.

This issue is only coming with some bad DNS server as they are not able to
process the request correctly. There is no proper fix for this is yet been
released by glibc and this bug is being reported from glibc-2.9.

I found this link which contains patch information using which work-around for
this issue can be applied.

http://sourceware.org/bugzilla/show_bug.cgi?id=7060

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[atsushi at onoe dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14308

Atsushi Onoe <atsushi at onoe dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |atsushi at onoe dot org

--- Comment #2 from Atsushi Onoe <atsushi at onoe dot org> ---
Created attachment 7049
  --> http://sourceware.org/bugzilla/attachment.cgi?id=7049&action=edit
Proposed patch to handle DNS responding referral for AAAA

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[atsushi at onoe dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=14308

--- Comment #3 from Atsushi Onoe <atsushi at onoe dot org> ---
Comment on attachment 7049
  --> http://sourceware.org/bugzilla/attachment.cgi?id=7049
Proposed patch to handle DNS responding referral for AAAA

If there is at least one correct response within parallel query, it should be
used, just like the case of SERVFAIL/NOTIMP/REFUSED.

Actually, the code to skip referral response already exists in glibc since 2.4.

Unfortunately, the parallel query code via _nss_dns_gethostbyname4_r() is
inserted without adopting the issue.

I think the label next_ns in send_dg() should be moved before checking another
response of parallel query.  Attached patch works properly to get 'A' record
from the DNS respond referral response for 'AAAA' query.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[siddhesh at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14308

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |siddhesh at redhat dot com
           Assignee|unassigned at sourceware dot org   |siddhesh at redhat dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=14308

--- Comment #4 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, master has been updated
       via  16b293a7a6f65d8ff348a603d19e8fd4372fa3a9 (commit)
      from  136785eacd13f0c05fefd92b029269efec35dc0a (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=16b293a7a6f65d8ff348a603d19e8fd4372fa3a9

commit 16b293a7a6f65d8ff348a603d19e8fd4372fa3a9
Author: Siddhesh Poyarekar <siddhesh@redhat.com>
Date:   Wed Apr 30 11:48:43 2014 +0530

    Do not fail if one of the two responses to AF_UNSPEC fails (BZ #14308)

    [Fixes BZ #14308, #12994, #13651]

    AF_UNSPEC results in sending two queries in parallel, one for the A
    record and the other for the AAAA record.  If one of these is a
    referral, then the query fails, which is wrong.  It should return at
    least the one successful response.

    The fix has two parts.  The first part makes the referral fall back to
    the SERVFAIL path, which results in using the successful response.
    There is a bug in that path however, due to which the second part is
    necessary.  The bug here is that if the first response is a failure
    and the second succeeds, __libc_res_nsearch does not detect that and
    assumes a failure.  The case where the first response is a success and
    the second fails, works correctly.

    This condition is produced by buggy routers, so here's a crude
    interposable library that can simulate such a condition.  The library
    overrides the recvfrom syscall and modifies the header of the packet
    received to reproduce this scenario.  It has two key variables:
    mod_packet and first_error.

    The mod_packet variable when set to 0, results in odd packets being
    modified to be a referral.  When set to 1, even packets are modified
    to be a referral.

    The first_error causes the first response to be a failure so that a
    domain-appended search is performed to test the second part of the
    __libc_nsearch fix.

    The driver for this fix is a simple getaddrinfo program that does an
    AF_UNSPEC query.  I have omitted this since it should be easy to
    implement.

    I have tested this on x86_64.

    The interceptor library source:

    /* Override recvfrom and modify the header of the first DNS response to
make it
       a referral and reproduce bz #845218.  We have to resort to this ugly
hack
       because we cannot make bind return the buggy response of a referral for
the
       AAAA record and an authoritative response for the A record.  */
     #define _GNU_SOURCE
     #include <sys/types.h>
     #include <sys/socket.h>
     #include <netinet/in.h>
     #include <arpa/inet.h>
     #include <stdio.h>
     #include <stdbool.h>
     #include <endian.h>
     #include <dlfcn.h>
     #include <stdlib.h>

    /* Lifted from resolv/arpa/nameser_compat.h.  */
    typedef struct {
        unsigned        id :16;         /*%< query identification number */
     #if BYTE_ORDER == BIG_ENDIAN
        /* fields in third byte */
        unsigned        qr: 1;          /*%< response flag */
        unsigned        opcode: 4;      /*%< purpose of message */
        unsigned        aa: 1;          /*%< authoritive answer */
        unsigned        tc: 1;          /*%< truncated message */
        unsigned        rd: 1;          /*%< recursion desired */
        /* fields
         * in
         * fourth
         * byte
         * */
        unsigned        ra: 1;          /*%< recursion available */
        unsigned        unused :1;      /*%< unused bits (MBZ as of 4.9.3a3) */
        unsigned        ad: 1;          /*%< authentic data from named */
        unsigned        cd: 1;          /*%< checking disabled by resolver */
        unsigned        rcode :4;       /*%< response code */
     #endif
     #if BYTE_ORDER == LITTLE_ENDIAN || BYTE_ORDER == PDP_ENDIAN
        /* fields
         * in
         * third
         * byte
         * */
        unsigned        rd :1;          /*%< recursion desired */
        unsigned        tc :1;          /*%< truncated message */
        unsigned        aa :1;          /*%< authoritive answer */
        unsigned        opcode :4;      /*%< purpose of message */
        unsigned        qr :1;          /*%< response flag */
        /* fields
         * in
         * fourth
         * byte
         * */
        unsigned        rcode :4;       /*%< response code */
        unsigned        cd: 1;          /*%< checking disabled by resolver */
        unsigned        ad: 1;          /*%< authentic data from named */
        unsigned        unused :1;      /*%< unused bits (MBZ as of 4.9.3a3) */
        unsigned        ra :1;          /*%< recursion available */
     #endif
        /* remaining
         * bytes
         * */
        unsigned        qdcount :16;    /*%< number of question entries */
        unsigned        ancount :16;    /*%< number of answer entries */
        unsigned        nscount :16;    /*%< number of authority entries */
        unsigned        arcount :16;    /*%< number of resource entries */
    } HEADER;

    static int done = 0;

    /* Packets to modify.  0 for the odd packets and 1 for even packets.  */
    static const int mod_packet = 0;

    /* Set to true if the first request should result in an error, resulting in
a
       search query.  */
    static bool first_error = true;

    static ssize_t (*real_recvfrom) (int sockfd, void *buf, size_t len, int
flags,
                  struct sockaddr *src_addr, socklen_t *addrlen);

    void
    __attribute__ ((constructor))
    init (void)
    {
      real_recvfrom = dlsym (RTLD_NEXT, "recvfrom");

      if (real_recvfrom == NULL)
        {
          printf ("Failed to get reference to recvfrom: %s\n", dlerror ());
          printf ("Cannot simulate test\n");
          abort ();
        }
    }

    /* Modify the second packet that we receive to set the header in a manner
as to
       reproduce BZ #845218.  */
    static void
    mod_buf (HEADER *h, int port)
    {
      if (done % 2 == mod_packet || (first_error && done == 1))
        {
          printf ("(Modifying header)");

          if (first_error && done == 1)
        h->rcode = 3;
          else
        h->rcode = 0;    /* NOERROR == 0.  */
          h->ancount = 0;
          h->aa = 0;
          h->ra = 0;
          h->arcount = 0;
        }
      done++;
    }

    ssize_t
    recvfrom (int sockfd, void *buf, size_t len, int flags,
          struct sockaddr *src_addr, socklen_t *addrlen)
    {
      ssize_t ret = real_recvfrom (sockfd, buf, len, flags, src_addr, addrlen);
      int port = htons (((struct sockaddr_in *) src_addr)->sin_port);
      struct in_addr addr = ((struct sockaddr_in *) src_addr)->sin_addr;
      const char *host = inet_ntoa (addr);
      printf ("\n*** From %s:%d: ", host, port);

      mod_buf (buf, port);

      printf ("returned %zd\n", ret);
      return ret;
    }

-----------------------------------------------------------------------

Summary of changes:
 ChangeLog          |   11 +++++++++++
 NEWS               |   14 +++++++-------
 resolv/res_query.c |    7 +++++--
 resolv/res_send.c  |    2 +-
 4 files changed, 24 insertions(+), 10 deletions(-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[siddhesh at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14308

--- Comment #5 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
Fixed in master.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[siddhesh at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14308

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #6 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
... and changing the status of the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/14308] getaddrinfo DNS referral response returns host not found when A and AAAA questions are sent and one response is a referral

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=14308

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.