[Bug libstdc++/42679] New: RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[Bug libstdc++/42679] New: RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[mjtruog at fastmail dot ca]

I do not know why this occurs but it only seems to occur for libstdc++
std::ostream code.  I have had SEGFAULTs with std::cerr but not std::clog when
simply outputting errors but tried to ignore it... until I found a different
problem.  I was getting SIGABRTs from a free() assert within libc when using
std::ostringstream for simple stream operators building from static 'char const
* const' like:

std::ostringstream s;
s << "bad";

I don't have a simple test to replicate this unfortunately, but I do have an
example crash dump.  I was using the linker -rpath to always tie the executable
and the shared library to the gcc dependencies.  I have no problems with my
setup when I remove the dlopen RTLD_DEEPBIND option when loading my shared
library that includes the std::ostream usage (like that which I described for
std::ostringstream).  This particular problem is a "heisenbug" because it only
appears when I turn optimization on for the shared library (-O1 was enough to
make it crash).  The example crash dump is below:

#0  0x00002b92e27af015 in *__GI_raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
        in ../nptl/sysdeps/unix/sysv/linux/raise.c
(gdb) back
#0  0x00002b92e27af015 in *__GI_raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002b92e27b0b83 in *__GI_abort () at abort.c:88
#2  0x00002b92e27f00c8 in __libc_message (do_abort=2, 
    fmt=0x2b92e28b8dd0 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:170
#3  0x00002b92e27f5a58 in malloc_printerr (action=2, 
    str=0x2b92e28b63a9 "free(): invalid pointer", ptr=<value optimized out>)
    at malloc.c:5949
#4  0x00002b92e27f80a6 in *__GI___libc_free (mem=0x2b92e28aef40)
    at malloc.c:3625
#5  0x00002b92e228b0d8 in std::basic_stringbuf<char, std::char_traits<char>,
std::allocator<char> >::overflow (this=0x40a87948, __c=<value optimized out>)
    at
/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:231
#6  0x00002b92e228eee5 in std::basic_streambuf<char, std::char_traits<char>
>::xsputn (this=0x40a87948, 
    __s=0xeea868
"01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567
", __n=98)
    at
/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/streambuf.tcc:97
#7  0x00002b92e22871d5 in std::__ostream_insert<char, std::char_traits<char> >
    (__out=@0x40a87940, __s=<value optimized out>, __n=98)
    at
/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/streambuf:427
#8  0x00002aaaaaab1998 in random_task (stop=@0xeaf7a2, 
    variable0=<value optimized out>, 
    variable1=<value optimized out>, 
    variable2=<value optimized out>, 
    variable3=<value optimized out>, 
    variable4=<value optimized out>, variable5=0, 
    variable6=0, variable7=0, variable8=0, 
    variable9=0, variable10=0, variable11=0, 
    variable12=0, stream=@0x40a87cf0)
    at
/.../install/g++/releases/gcc-4.4.2_install/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.2/../../../../include/c++/4.4.2/bits/basic_string.h:2506
#9  0x00002aaaaaabd543 in do_work (stop=<value optimized out>, id=0, 
    taskData=@0xe9d8a0, taskDataSize=88, queriesOut=@0x40a87f70)
    at lib/cloud_job_all_patterns/src/cloud_job_random_task.cpp:106
#10 0x000000000041320d in
WorkerController::WorkerExecution::ThreadPool::ThreadFunctionObject::operator()
(this=0x40a88030, stopped=@0xeaf7a2, 
    allocator=<value optimized out>)
    at lib/cloud_worker/src/worker_execution.cpp:501
warning: (Internal error: pc 0x418d35 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x418d35 in read in psymtab, but not in symtab.)

#11 0x0000000000418d36 in
boost::detail::thread_data<WorkerController::WorkerExecution::ThreadPool::ThreadObject>::run
(this=<value optimized out>)
    at lib/cloud_worker/src/worker_execution.cpp:236
warning: (Internal error: pc 0x418d35 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x418c80 in read in psymtab, but not in symtab.)

#12 0x00002b92e15ae350 in thread_proxy ()
   from
/.../src/lib/boost/releases/boost_1_40_0_install/lib/libboost_thread.so.1.40.0
#13 0x00002b92e2d0c3ea in start_thread (arg=<value optimized out>)
    at pthread_create.c:297
#14 0x00002b92e2862cbd in clone () from /usr/lib/debug/libc.so.6
#15 0x0000000000000000 in ?? ()


libc gives its own little backtrace:
*** glibc detected *** _release/lib/cloud-0.0.8/priv/cloud_worker_port: free():
invalid pointer: 0x00002b92e24e50a0 ***
======= Backtrace: =========
/usr/lib/debug/libc.so.6[0x2b92e27f5a58]
/usr/lib/debug/libc.so.6(cfree+0x76)[0x2b92e27f80a6]
/.../src/lib/g++/releases/gcc-4.4.2_install/lib64/libstdc++.so.6(_ZNSt15basic_stringbufIcSt11char_traitsIcESaIcEE8overflowEi+0x168)[0x2b92e228b0d8]
/.../src/lib/g++/releases/gcc-4.4.2_install/lib64/libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x35)[0x2b92e228eee5]
_release/lib/cloud-0.0.8/priv/work_types/libcloud_job_random_task.so(_Z21random_taskRKbP7pg_connRKSsS4_jjjjjjjjjjRSo+0x229)[0x2aaaaaab1998]
_release/lib/cloud-0.0.8/priv/work_types/libcloud_job_random_task.so(do_work+0x74a)[0x2aaaaaabd543]
release/lib/cloud-0.0.8/priv/cloud_worker_port[0x41320d]

You could probably replicate my scenario with the code I am using, but the
setup would be a pain (code is at
http://sourceforge.net/projects/cloudi/files/0.0.8a/cloudi-0.0.8.tar.gz/download)

Why won't std::ostream classes play well with when loaded into an executable
with RTLD_DEEPBIND?  Isn't this some sort of design problem within the
std::ostream code?


-- 
           Summary: RTLD_DEEPBIND dlopen option for shared library that uses
                    libstdc++ std::ostream crashes
           Product: gcc
           Version: 4.4.2
            Status: UNCONFIRMED
          Severity: major
          Priority: P3
         Component: libstdc++
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: mjtruog at fastmail dot ca
 GCC build triplet: x86_64-unknown-linux-gnu
  GCC host triplet: x86_64-unknown-linux-gnu
GCC target triplet: x86_64-unknown-linux-gnu


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #1 from paolo dot carlini at oracle dot com  2010-01-10 10:42 -------
At the very minimum we need a small reproducer.


-- 

paolo dot carlini at oracle dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[mjtruog at fastmail dot ca]

------- Comment #2 from mjtruog at fastmail dot ca  2010-01-10 18:16 -------
I have been trying to make a simple test case for the ostringstream problem but
have not yet found one.  However, I have been able to make a test case for the
crash when using std::cerr.  The problems are probably related in some way, but
the std::ostringstream one I found may be harder to recreate.  I have the files
to replicate the std::cerr crash scenario below:

main.cpp:

#include <dlfcn.h>
#include <iostream>

// compile with: g++ -g -O0 main.cpp -ldl

int main()
{
    char const * const library_name = "./library.so";
    void * handle = dlopen(library_name, RTLD_NOW | RTLD_LOCAL |
RTLD_DEEPBIND);
    char const * const dlopen_error = dlerror();
    if (! handle)
    {
        std::cerr << "dlopen error: " << dlopen_error << std::endl;
        return 1;
    }
    typedef void (*library_function_type)();
    void * function = dlsym(handle, "library_function");
    char const * const dlsym_error = dlerror();
    if (dlsym_error)
    {
        std::cerr << "dlsym error: " << dlsym_error << std::endl;
        dlclose(handle);
        return 1;
    }
    reinterpret_cast<library_function_type>(function)();
    dlclose(handle);
    return 0;
}

library1.cpp:

#include <sstream>
#include <iostream>

// example of SEGFAULT when writing to std::cerr

// compile with:
// g++ -g -O0 -rdynamic -c -fPIC -o library.o library1.cpp
// g++ -shared -Wl,-export-dynamic -o library.so library.o

extern "C"
{

void library_function()
{
    std::clog << "std::cerr will segfault" << std::endl;
    std::cerr << "crash me";
}

}


-- 

mjtruog at fastmail dot ca changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |UNCONFIRMED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #3 from paolo dot carlini at oracle dot com  2010-01-10 19:18 -------
Well, these are two interesting data points: 1- The crash definitely is not
new, happens also with 4.2.4; 2- The same 4.4.x library, but ICC as C++
compiler, doesn't crash, maybe it's a random behavior, sure, but I would be
cautious with "design problems". Maybe Jakub can spot something...


-- 

paolo dot carlini at oracle dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at redhat dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

-- 

paolo dot carlini at oracle dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|major                       |normal


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[mjtruog at fastmail dot ca]

------- Comment #4 from mjtruog at fastmail dot ca  2010-01-22 07:10 -------
I can help try to determine the problem if you want since it seems everyone is
either busy, doesn't necessarily care right now, or both.  However, if someone
could provide any hints as to what RTLD_DEEPBIND might be doing to libstdc++
internals that is atypical behavior, it would be helpful.  I just suggested
that it might be a design problem since it seems like the problem might have a
broad scope.  I don't wish to suggest that libstdc++ is directly responsible
for no reason.  I was expecting that there was some strange compiler specific
linking going on somewhere but have not set out to try to find anything strange
like that since I assumed other people would have a better idea of what might
be causing the problem.  The problem with RTLD_DEEPBIND might be a new one
since it appeared in glibc 2.3.4.  While the option is not critical, it seems
helpful for keeping dynamic libraries in isolation if it is an N-to-1
relationship where many dynamic libraries are used for a single executable, at
runtime based on runtime events.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #5 from paolo dot carlini at oracle dot com  2010-01-28 03:10 -------
Jakub, any idea? Thanks in advance.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[jakub at gcc dot gnu dot org]

------- Comment #6 from jakub at gcc dot gnu dot org  2010-01-28 08:03 -------
If libstdc++ relies on ODR, then RTLD_DEEPBIND can break its assumptions, as
then the same symbol which is defined in multiple shared libraries can resolve
to different addresses (RTLD_DEEPBIND means first the dlopened library and its
dependencies, and only after it the global search scope, will be searched).

STB_GNU_UNIQUE object ought to cure it, though it is only in 4.5 (and 4.4-RH)
gcc and sufficiently new binutils and glibc are needed for it too.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #7 from paolo dot carlini at oracle dot com  2010-01-28 09:50 -------
Many thanks Jakub. I have to give this issue much more thought. For the time
being I'm going to add Jason in CC in case he wants to suggest something in
particular, I see he was involved in the STB_GNU_UNIQUE idea quite directly.


-- 

paolo dot carlini at oracle dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jason at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[jason at gcc dot gnu dot org]

------- Comment #8 from jason at gcc dot gnu dot org  2010-01-28 20:01 -------
The issue Jakub describes is indeed the motivation for STB_GNU_UNIQUE.  But I
don't know why RTLD_DEEPBIND would cause trouble when the library is only
loaded once.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[jakub at gcc dot gnu dot org]

------- Comment #9 from jakub at gcc dot gnu dot org  2010-01-28 22:07 -------
The theoretical example is libstdc++ defining some object where things
misbehave if it is not the only one (say foo) and some other library has that
object as well.  When that other library isn't an direct or indirect dependency
of the executable, but libstdc++ is, in all libraries loaded before start of
the program foo resolves to the libstdc++ copy.  When you dlopen RTLD_DEEPBIND
this other library, foo in it and all its dependencies will resolve to the
other library, as it comes earlier in the search scope, before global scope.
I guess LD_DEBUG=all could shed some light into what exactly is going on, it
could be the empty string, or something similar.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #10 from paolo dot carlini at oracle dot com  2010-01-28 22:35 -------
Jakub, Jason, thanks for the additional info, hopefully we can make progress on
this issue. By the way, I'm not sure if you noticed that we have an actual 
reproducer in Comment #2, is it consistent with your theoretical analysis?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[jakub at gcc dot gnu dot org]

------- Comment #11 from jakub at gcc dot gnu dot org  2010-01-28 22:49 -------
LD_DEBUG=all ./main 2>&1 | grep _ZSt4cerr
     12758:     symbol=_ZSt4cerr;  lookup in file=./main [0]
     12758:     binding file /usr/lib64/libstdc++.so.6 [0] to ./main [0]:
normal symbol `_ZSt4cerr' [GLIBCXX_3.4]
     12758:     symbol=_ZSt4cerr;  lookup in file=/lib64/libdl.so.2 [0]
     12758:     symbol=_ZSt4cerr;  lookup in file=/usr/lib64/libstdc++.so.6 [0]
     12758:     binding file ./main [0] to /usr/lib64/libstdc++.so.6 [0]:
normal symbol `_ZSt4cerr' [GLIBCXX_3.4]
     12758:     symbol=_ZSt4cerr;  lookup in file=./library.so [0]
     12758:     symbol=_ZSt4cerr;  lookup in file=/usr/lib64/libstdc++.so.6 [0]
     12758:     binding file ./library.so [0] to /usr/lib64/libstdc++.so.6 [0]:
normal symbol `_ZSt4cerr' [GLIBCXX_3.4]

The first lookup is for std::cerr relocations in libstdc++, the second one is
just to find out what should be the std::cerr COPY relocation in main be
initialized for.  Thus, the executable uses std::cerr inside of main's .bss.
But during RTLD_DEEPBIND first library.so and its dependencies are searched, so
the copy in libstdc++.so.6 (which hasn't been initialized at runtime) is used.

Note that std::cerr isn't STB_GNU_UNIQUE, so this crashes even on Fedora 12.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #12 from paolo dot carlini at oracle dot com  2010-01-28 22:59 -------
Good. Now, if I can have a little bit more of your time, I have two main
curiosities: can you guess why the same 4.4.x *.so + headers don't lead to a
crash with ICC? Then, we should of course come to a possible fix: the idea
would be checking at configure time that the underlying glibc is recent enough
(is 10.1 ok?) and then marking STB_GNU_UNIQUE some selected exported symbols?
The static allocated ones? I'm guessing a bit...


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #13 from paolo dot carlini at oracle dot com  2010-01-29 00:46 -------
This is the LD_DEBUG output for Icc 11.1, thus no crash.

     17439:     symbol=_ZSt4cerr;  lookup in file=./a.out [0]
     17439:     binding file /usr/lib64/libstdc++.so.6 [0] to ./a.out [0]:
normal symbol `_ZSt4cerr' [GLIBCXX_3.4]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/libdl.so.2 [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/libm.so.6 [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/usr/lib64/libstdc++.so.6 [0]
     17439:     binding file ./a.out [0] to /usr/lib64/libstdc++.so.6 [0]:
normal symbol `_ZSt4cerr' [GLIBCXX_3.4]
     17439:     symbol=_ZSt4cerr;  lookup in file=./library.so [0]
     17439:     symbol=_ZSt4cerr;  lookup in
file=/opt/intel/Compiler/11.1/064/lib/intel64/libimf.so [0]
     17439:     symbol=_ZSt4cerr;  lookup in
file=/opt/intel/Compiler/11.1/064/lib/intel64/libsvml.so [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/libm.so.6 [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/libgcc_s.so.1 [0]
     17439:     symbol=_ZSt4cerr;  lookup in
file=/opt/intel/Compiler/11.1/064/lib/intel64/libintlc.so.5 [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/libc.so.6 [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/libdl.so.2 [0]
     17439:     symbol=_ZSt4cerr;  lookup in file=/lib64/ld-linux-x86-64.so.2
[0]
     17439:     symbol=_ZSt4cerr;  lookup in file=./a.out [0]
     17439:     binding file ./library.so [0] to ./a.out [0]: normal symbol
`_ZSt4cerr'


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[jakub at gcc dot gnu dot org]

------- Comment #14 from jakub at gcc dot gnu dot org  2010-01-29 07:52 -------
Clearly this works with icc, because library.so isn't linked against
libstdc++.so.  I guess if you link the library with gcc instead of g++, it will
work too.
Any reason why you need to use RTLD_DEEPBIND?  AFAIK it has been mainly to
support libraries linked against a different version of libstdc++ from the rest
of the app (say the program and all its libraries linked against GCC 3.2/3.3
libstdc++, while some library it dlopens linked against GCC 3.4+ libstdc++.
I'd say there is nothing that should be done on the libstdc++ side, this can
happen with any kinds of symbols in any library.
What can help is compile the program with -fpic/-fPIC, then it won't have copy
relocation and thus this problem won't exist.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[paolo dot carlini at oracle dot com]

------- Comment #15 from paolo dot carlini at oracle dot com  2010-01-29 10:08 -------
Thanks Jakub, let's see what submitter has to say.

Just want to add here that I'm building submitter' app **exactly** in the same
way with g++ / icc, on the very same machine indeed, and in the former case the
final executable crashes, in the latter it doesn't. But there are still many
details I do not understand about this new feature, sorry if I'm saying
something wrong. I suspect however that other lay users having icc installed
could also report to us a puzzling behavior.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[mjtruog at fastmail dot ca]

------- Comment #16 from mjtruog at fastmail dot ca  2010-01-29 17:04 -------
I am using "-fpic/-fPIC" and in fact am using:
// g++ -g -O0 main.cpp -ldl
// g++ -g -O0 -rdynamic -c -fPIC -o library.o library1.cpp
// g++ -shared -Wl,-export-dynamic -o library.so library.o

I do want to use RTLD_DEEPBIND to keep the libraries in isolation as you
mention, since I could have libraries that depend on different versions of GCC.
 I understand that it doesn't work right now with g++ linking, so I might try
to switch to gcc linking.  I just was concerned because it appeared to be
connected to libstdc++, but I guess it is a g++ linking peculiarity.  I don't
think ignoring the RTLD_DEEPBIND feature in glibc would help.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679

[Bug libstdc++/42679] RTLD_DEEPBIND dlopen option for shared library that uses libstdc++ std::ostream crashes

[mjtruog at fastmail dot ca]

------- Comment #17 from mjtruog at fastmail dot ca  2010-03-07 17:53 -------
I have found this doesn't fix the problem.  It may fix the problem in the
example, but not in all cases.  I have a few new crash dumps:

Core was generated by `_release/lib/cloud-0.0.9/priv/cloud_worker_port'.
Program terminated with signal 6, Aborted.
[New process 10276]
[New process 10273]
[New process 10277]
#0  0x00002b766a9a7015 in raise () from /lib/libc.so.6
(gdb) back
#0  0x00002b766a9a7015 in raise () from /lib/libc.so.6
#1  0x00002b766a9a8b83 in abort () from /lib/libc.so.6
#2  0x00002b766a9e80c8 in __libc_message () from /lib/libc.so.6
#3  0x00002b766a9eda58 in malloc_printerr () from /lib/libc.so.6
#4  0x00002b766a9f00a6 in free () from /lib/libc.so.6
#5  0x00002b766ad8758e in std::string::_M_mutate (this=0x407f0ea0, __pos=0, 
    __len1=0, __len2=32)
    at
/home/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:231
#6  0x00002b766ad875dc in std::string::_M_replace_safe (this=0x2821, 
    __pos1=10276, __n1=6, __s=0x407f0741 "243f6a8885a308d313198a2e03707344", 
    __n2=47787595689792)
    at
/home/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:662
#7  0x00002aaaaaab2445 in bbp_pi (abortTask=<value optimized out>, 
    digitIndex=<value optimized out>, digitStep=<value optimized out>, 
    piSequence=<value optimized out>)
    at lib/cloud_job_tests/src/piqpr8_gmp.cpp:241
#8  0x00002aaaaaab05db in do_work (abortTask=@0x12387a2, 
    workInstance=<value optimized out>, id=<value optimized out>, 
    taskData=@0x1226ca0, taskDataSize=10, queriesOut=@0x407f0f70)
    at lib/cloud_job_tests/src/cloud_job_tests.cpp:145
#9  0x000000000041390d in
WorkerController::WorkerExecution::ThreadPool::ThreadFunctionObject::operator()
(this=0x407f1040, stopped=@0x12387a2, 
    allocator=<value optimized out>)
    at lib/cloud_worker/src/worker_execution.cpp:501
warning: (Internal error: pc 0x419054 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x419054 in read in psymtab, but not in symtab.)

#10 0x0000000000419055 in
boost::detail::thread_data<WorkerController::WorkerExecution::ThreadPool::ThreadObject>::run
(this=<value optimized out>)
    at lib/cloud_worker/src/worker_execution.cpp:236
warning: (Internal error: pc 0x419054 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x418fa0 in read in psymtab, but not in symtab.)

#11 0x00002b7669b05870 in thread_proxy ()
   from
/home/.../src/lib/boost/releases/boost_1_42_0_install/lib/libboost_thread.so.1.42.0
#12 0x00002b766b27a3ea in start_thread () from /lib/libpthread.so.0
#13 0x00002b766aa5acbd in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()
(gdb) 

I also continue to reproduce the same stderr problem (in debug mode):
Core was generated by `_release/lib/cloud-0.0.9/priv/cloud_worker_port'.
Program terminated with signal 11, Segmentation fault.
[New process 11089]
[New process 11090]
#0  sentry (this=0x7fff4a32a8d0, __os=@0x2abb61e70c20)
    at
/home/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:51
51            if (__os.tie() && __os.good())
(gdb) back
#0  sentry (this=0x7fff4a32a8d0, __os=@0x2abb61e70c20)
    at
/home/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:51
#1  0x00002abb61c15048 in std::__ostream_insert<char, std::char_traits<char> >
    (__out=@0x2abb61e70c20, __s=<value optimized out>, __n=47)
    at
/home/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:80
#2  0x00002abb61c1544f in operator<< <std::char_traits<char> > (
    __out=@0x2abb61e70c20, 
    __s=0x2aaaaaab7360 "cloud_job_tests global/static data initialize()")
    at
/home/.../src/lib/g++/releases/gcc-4.4.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ostream:510
#3  0x00002aaaaaab30d6 in initialize ()
    at lib/cloud_job_tests/src/cloud_job_tests.cpp:100
#4  0x00002aaaaaab7326 in __do_global_ctors_aux ()
   from _release/lib/cloud-0.0.9/priv/work_types/libcloud_job_tests.so
#5  0x00002aaaaaab2423 in _init ()
   from _release/lib/cloud-0.0.9/priv/work_types/libcloud_job_tests.so
#6  0x00002abb00000000 in ?? ()
#7  0x00002abb60781a20 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#8  0x00002abb6078649a in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#9  0x00002abb60781676 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x00002abb60785b2b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#11 0x00002abb611e8f5b in dlopen_doit () from /lib/libdl.so.2
#12 0x00002abb60781676 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#13 0x00002abb611e92cc in _dlerror_run () from /lib/libdl.so.2
#14 0x00002abb611e8ec1 in dlopen@@GLIBC_2.2.5 () from /lib/libdl.so.2
#15 0x0000000000425f17 in library (this=0x172a970, name=@0x7fff4a32ae60)
    at lib/cloud_worker/src/library.cpp:68
#16 0x0000000000417cec in WorkerController::WorkerExecution::input (
    this=0x672218, pTask=@0x7fff4a32b518)
    at lib/cloud_worker/src/worker_execution.cpp:574
#17 0x000000000040f379 in WorkerController::WorkerQuery::store (this=0x672258, 
    ptr=@0x7fff4a32b518) at lib/cloud_worker/src/worker_controller.cpp:432
#18 0x00000000004100aa in WorkerController::store (this=0x6721c0, 
    ptr=@0x7fff4a32b518) at lib/cloud_worker/src/worker_controller.hpp:195
#19 0x000000000040e278 in WorkerController::receive_work (this=0x6721c0, fd=8)
    at lib/cloud_worker/src/worker_controller.cpp:244
#20 0x000000000040ba4f in handle_node_connection (index=5, 
    controller=@0x6721c0) at lib/cloud_worker/src/node_connections.cpp:499
#21 0x000000000040bc25 in NodeConnections::worker_loop (
    buffer=0x7fff4a32b610 "", controller=@0x6721c0)
    at lib/cloud_worker/src/node_connections.cpp:560
#22 0x000000000042587e in main (argv=0x7fff4a3336f8)
    at lib/cloud_worker/src/cloud_worker.cpp:114
(gdb) 

This is with using gcc for linking instead of g++.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42679