[Bug libc/10652] getaddrinfo causes segfault if multithreaded and linked statically

[Bug libc/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bobby at sharedrealm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

Robert G. Jakabosky <bobby at sharedrealm dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bobby at sharedrealm dot
                   |                            |com

--- Comment #12 from Robert G. Jakabosky <bobby at sharedrealm dot com> 2011-03-26 07:09:49 UTC ---
The same crash happens if the host program is not compiled with "-pthread" and
dynamically loads a module which is linked to libpthread.so and calls
getaddrinfo() from multiple threads.

I will attache two example C files that show case this problem.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bobby at sharedrealm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #13 from Robert G. Jakabosky <bobby at sharedrealm dot com> 2011-03-26 07:15:24 UTC ---
Created attachment 5325
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5325
Example module which calls getaddrinfo() from many threads.

Compile this example module with:
gcc -o crash_getaddrinfo.so -Wall -fPIC -shared -pthread crash_getaddrinfo.c

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bobby at sharedrealm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #14 from Robert G. Jakabosky <bobby at sharedrealm dot com> 2011-03-26 07:24:33 UTC ---
Created attachment 5326
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5326
Simple host program to dynamically load a module with dlopen().

Compile without -pthread:
gcc -ldl -Wall -o crash_main_no_pthread crash_main.c

Compile with -pthread:
gcc -ldl -Wall -o crash_main_pthread crash_main.c -pthread

By default the program will try to load a module named:
/tmp/crash_getaddrinfo.so

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bobby at sharedrealm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #15 from Robert G. Jakabosky <bobby at sharedrealm dot com> 2011-03-26 07:36:11 UTC ---
I first ran into this problem when using a Lua C module (ZeroMQ bindings for
Lua) that uses IO threads in the background.  The only work-around is to either
compile the Lua VM with -pthread (This shouldn't be required, since not all Lua
scripts need pthread support) or to use "LD_PRELOAD=/lib/libpthread.so
host_program".

I would prefer an option where the host program (Lua VM) didn't have to either
be compiled with -pthread or wrapped in a script to preload libpthread.so.

Also the example program will even crash on a single-cpu(single-core) computer
running Debian 6.0, glibc 2.11.2.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bobby at sharedrealm dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #16 from Robert G. Jakabosky <bobby at sharedrealm dot com> 2011-03-26 07:58:37 UTC ---
Created attachment 5327
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5327
Valgrind output shows some invalid reads into freed memory before the program
crashes on a NULL pointer.

This problem seems to be caused by a race condition between the threads calling
getaddrinfo().  With a small number of threads it doesn't always happen. 
Atleast the backtrace has always been the same.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/10652] getaddrinfo causes segfault if multithreaded and linked statically

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|libc                        |stdio

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/10652] getaddrinfo causes segfault if multithreaded and linked statically

[aj at suse dot de]

http://sourceware.org/bugzilla/show_bug.cgi?id=10652

Andreas Jaeger <aj at suse dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aj at suse dot de

--- Comment #17 from Andreas Jaeger <aj at suse dot de> 2013-01-03 12:13:25 UTC ---
(In reply to comment #12)
> The same crash happens if the host program is not compiled with "-pthread" and
> dynamically loads a module which is linked to libpthread.so and calls
> getaddrinfo() from multiple threads.
> 
> I will attache two example C files that show case this problem.

A comment on this case is at:
http://sourceware.org/ml/libc-alpha/2012-10/msg00361.html

my advise for now is to link your application against libpthread until somebody
really digs into this and figures out what is supposed to work and how.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[neleai at seznam dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Ondrej Bilka <neleai at seznam dot cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |neleai at seznam dot cz
          Component|stdio                       |network

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at redhat dot com
           Assignee|drepper.fsp at gmail dot com       |unassigned at sourceware dot org

--- Comment #18 from Carlos O'Donell <carlos at redhat dot com> ---
This bug is still reproducible in 2.18.90.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |NEW

--- Comment #19 from Carlos O'Donell <carlos at redhat dot com> ---
In a test case where the application doesn't link against libpthread, but a
dlopen'd library does, parallel calls to getaddrinfo cause corruption in the IO
layers and eventually a crash.

Even though libpthread.so.1 has been loaded the weak-ref-and-check idiom in the
NSS code isn't working. The GOT entry stays zero and therefore the nss code
skips doing any locking and we get serious corruption via
get_contents->__GI_fgets_unlocked (doing unlocked file IO with multiple threads
causes data races and corruption). 

The skipped locks are in _nss_files_gethostbyname4_r (libnss_files.so). When
the application is compiled with -lpthread the GOT entry has a non-zero value
of 0x00007ffff77bc460 which is "0x7ffff77bc460 <__GI___pthread_mutex_lock>:   
sub    $0x8,%rsp" and therefore correct. That entry is the GOT entry #40 with
relocation: 000000000020bfd8  0000001a00000006 R_X86_64_GLOB_DAT     
0000000000000000 __pthread_mutex_lock + 0.

If libpthread is loaded *after* libnss_files.so is loaded I don't see that
there is anything you can do to make the NSS code use locks since the GOT
relocation has already been processed. However in this case libpthread is
loaded *before* libnss_files.so, but it appears as if the resolution scope
prevents the symbols from libpthread being made available to libnss_files.so?

e.g.
     20987:     object=/home/carlos/build/glibc/nss/libnss_files.so.2 [0]
     20987:      scope 0: ./crash_main_no_pthread
/home/carlos/build/glibc/dlfcn/libdl.so.2 /home/carlos/build/glibc/libc.so.6
/home/carlos/build/glibc/elf/ld.so
     20987:      scope 1: /home/carlos/build/glibc/nss/libnss_files.so.2
/home/carlos/build/glibc/libc.so.6 /home/carlos/build/glibc/elf/ld.so

Notice libnss_files.so.2 is in it's own scope without libpthread. As opposed to
crash_getaddrinfo.so's scope with libpthread in it

e.g.
     20987:     object=/home/carlos/support/2013-11-22/crash_getaddrinfo.so [0]
     20987:      scope 0: ./crash_main_no_pthread
/home/carlos/build/glibc/dlfcn/libdl.so.2 /home/carlos/build/glibc/libc.so.6
/home/carlos/build/glibc/elf/ld.so
     20987:      scope 1: /home/carlos/support/2013-11-22/crash_getaddrinfo.so
/home/carlos/build/glibc/nptl/libpthread.so.0
/home/carlos/build/glibc/libc.so.6 /home/carlos/build/glibc/elf/ld.so

I don't know what's the right answer here. There are really only two resolution
scopes, global and local, the scopes listed above are internal details of
glibc's dyanmic loader. Why libpthread's symbols wouldn't be used for the
relocation in libnss_files.so is what baffles me, one would have to track down
the exact relocation and determine why the libpthread symbol isn't used.

I'm not working on this so I'm flipping this to NEW, but I thought I'd post
what I saw during my analysis of a similar internal Red Hat bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bugdal at aerifal dot cx]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #20 from Rich Felker <bugdal at aerifal dot cx> ---
Why is getaddrinfo trying to "optimize" out the locking for single-threaded
programs anyway? Certainly the time spent in getaddrinfo is dominated by actual
lookups, not by locking overhead.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #21 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Rich Felker from comment #20)
> Why is getaddrinfo trying to "optimize" out the locking for single-threaded
> programs anyway? Certainly the time spent in getaddrinfo is dominated by
> actual lookups, not by locking overhead.

I can only assume it does this to avoid requiring libpthread. The actual
lookups might also be very fast if they are resolved by /etc/hosts or some
other local file-based NSS backend. Requiring the thread library would have a
non-zero impact on performance for single-threaded applications. What other
reason could there be for using the weak-ref-and-check idiom (which I know you
don't like)?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[bugdal at aerifal dot cx]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #22 from Rich Felker <bugdal at aerifal dot cx> ---
As long as libpthread is a separate DSO, avoiding loading it makes sense, yes.
However it seems that all the internal locking in glibc components (including
nss modules) could be done with lock functions available unconditionally in
libc rather than needing the pthread lock functions. I'm not familiar enough
with the glibc internals to know whether such functions are already available,
but it would certainly make for a cleaner solution to this and many other
problems if they are. Note that the locking requirements for internal use are
much simpler than pthread requirements; there are no difficult issues like
different mutex types, self-synchronized destruction, etc.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

--- Comment #23 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Rich Felker from comment #22)
> As long as libpthread is a separate DSO, avoiding loading it makes sense,
> yes. However it seems that all the internal locking in glibc components
> (including nss modules) could be done with lock functions available
> unconditionally in libc rather than needing the pthread lock functions. I'm
> not familiar enough with the glibc internals to know whether such functions
> are already available, but it would certainly make for a cleaner solution to
> this and many other problems if they are. Note that the locking requirements
> for internal use are much simpler than pthread requirements; there are no
> difficult issues like different mutex types, self-synchronized destruction,
> etc.

No, you make a good point, and internally glibc already uses just plain futexes
for __libc_lock_lock, but for non-libc modules like libnss_files.so.2 (loaded
as part of the NSS plugin mechanism) the __libc_lock_lock defines redirect to
__pthread_mutex_lock. I see no reason at the moment why they couldn't just use
futexes for serializing threaded access. There was certainly no futex support
when these NSS modules were written so it might be a legacy issue. Switching
them over to futex locking would solve this problem and the uncontended lock
case is an atomic operation that should always succeeds.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[jackie.rosen at hushmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Jackie Rosen <jackie.rosen at hushmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jackie.rosen at hushmail dot com

--- Comment #24 from Jackie Rosen <jackie.rosen at hushmail dot com> ---
*** Bug 260998 has been marked as a duplicate of this bug. ***
Seen from the domain http://volichat.com
Page where seen: http://volichat.com/adult-chat-rooms
Marked for reference. Resolved as fixed @bugzilla.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[schwab at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Andreas Schwab <schwab at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|jackie.rosen at hushmail dot com   |

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug network/10652] getaddrinfo causes segfault if multithreaded and linked statically

[izbyshev at ispras dot ru]

https://sourceware.org/bugzilla/show_bug.cgi?id=10652

Alexey Izbyshev <izbyshev at ispras dot ru> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |izbyshev at ispras dot ru

--- Comment #29 from Alexey Izbyshev <izbyshev at ispras dot ru> ---
Still occurs with 2.31 in the same place as in the previous comment. Note that
heavy multithreading is not required, the following crashes too:

#include <netdb.h>
#include <pthread.h>

void *run(void *arg) {
  gethostbyname("localhost");
  return NULL;
}

int main() {
    gethostbyname("localhost");
    pthread_t t;
    pthread_create(&t, NULL, run, NULL);
    pthread_join(t, NULL);
    return 0;
}

The crash goes away if any of gethostbyname() calls is removed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.