Problem with static linking

Problem with static linking

[Zachary Turner]

Hello, I've been trying to write a program that links to static
libraries, and I've been having a lot of difficulties.  Was wondering
if someone can help me identify what's going wrong.

The codebase is large, but is new to linux.  It was originally
developed on windows and then ported to linux.  It makes heavy use of
C++, STL, and boost and we'd like to (if possible) link *everything*
statically.  This means libc, libgcc, libstdc++, boost, libpthread,
etc.

First things first however.  For the past few weeks we've had
everything compiling and linking just fine so we didn't suspect
anything.  We just knew that we had sporadic errors that didn't really
make sense but figured they were just obscure coding errors.  We
decided to run the program under valgrind and it generated about
60,000 errors.  Currently we are *assuming* that this is what's
causing all the other strange errors in our program, so we'd like to
fix them.  We spent a day or two playing around with various linker
settings / compiler and finally realized that removing the -static
command line options brings valgrind errors down to about 5, all of
which appear to be actual problems with our own code.

To get a baseline for comparison, we tried to make the simplest C++
program imaginable and compile it with -static.  Even that doesn't
work.  Technically it runs without segfaulting, but it still generates
19 valgrind errors, so we feel like the problem is still present even
in this simple case.  The source of said program is as follows:

//test.cpp
int main(int argc, char** argv) { return 0; }

That's it.   When I change the filename to test.c and compile it using
gcc instead of g++, I get the exact same problem if I use -static, and
the problem again goes away if I don't use -static.The output from g++
-v, as well as the output from valgrind when I run this program is as
follows (I've snipped valgrind errors down to only a few for the sake
of brevity.  The rest of the errors are somewhat similar):

% g++ -static test.cpp -v -o test
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --with-system-zlib --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --enable-nls
--with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc
--enable-mpfr --enable-targets=all --with-tune=generic
--enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu
--target=i486-linux-gnu
Thread model: posix
gcc version 4.2.4 (Debian 4.2.4-6)
 /usr/lib/gcc/i486-linux-gnu/4.2.4/cc1plus -quiet -v -D_GNU_SOURCE
test.cpp -quiet -dumpbase test.cpp -mtune=generic -auxbase test
-version -o /tmp/ccjuzsji.s
ignoring nonexistent directory "/usr/local/include/i486-linux-gnu"
ignoring nonexistent directory
"/usr/lib/gcc/i486-linux-gnu/4.2.4/../../../../i486-linux-gnu/include"
ignoring nonexistent directory "/usr/include/i486-linux-gnu"
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/4.2
 /usr/include/c++/4.2/i486-linux-gnu
 /usr/include/c++/4.2/backward
 /usr/local/include
 /usr/lib/gcc/i486-linux-gnu/4.2.4/include
 /usr/include
End of search list.
GNU C++ version 4.2.4 (Debian 4.2.4-6) (i486-linux-gnu)
 compiled by GNU C version 4.2.4 (Debian 4.2.4-6).
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: f71704aa7b35be12ec19d4deb7ce4f04
 as -V -Qy -o /tmp/ccmPieVt.o /tmp/ccjuzsji.s
GNU assembler version 2.18.0 (i486-linux-gnu) using BFD version (GNU
Binutils for Debian) 2.18.0.20080103
 /usr/lib/gcc/i486-linux-gnu/4.2.4/collect2 -m elf_i386
--hash-style=both -static -o test
/usr/lib/gcc/i486-linux-gnu/4.2.4/../../../../lib/crt1.o
/usr/lib/gcc/i486-linux-gnu/4.2.4/../../../../lib/crti.o
/usr/lib/gcc/i486-linux-gnu/4.2.4/crtbeginT.o
-L/usr/lib/gcc/i486-linux-gnu/4.2.4
-L/usr/lib/gcc/i486-linux-gnu/4.2.4
-L/usr/lib/gcc/i486-linux-gnu/4.2.4/../../../../lib -L/lib/../lib
-L/usr/lib/../lib -L/usr/lib/gcc/i486-linux-gnu/4.2.4/../../..
/tmp/ccmPieVt.o -lstdc++ -lm --start-group -lgcc -lgcc_eh -lc
--end-group /usr/lib/gcc/i486-linux-gnu/4.2.4/crtend.o
/usr/lib/gcc/i486-linux-gnu/4.2.4/../../../../lib/crtn.o



% valgrind ./test
==16624== Memcheck, a memory error detector.
==16624== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==16624== Using LibVEX rev 1854, a library for dynamic binary translation.
==16624== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==16624== Using valgrind-3.3.1-Debian, a dynamic binary
instrumentation framework.
==16624== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==16624== For more details, rerun with: -v
==16624==
==16624== Conditional jump or move depends on uninitialised value(s)
==16624==    at 0x805BF16: __register_atfork (in /home/zturner/test)
==16624==    by 0x805663C: ptmalloc_init (in /home/zturner/test)
==16624==    by 0x8059AB5: malloc_hook_ini (in /home/zturner/test)
==16624==    by 0x8059957: malloc (in /home/zturner/test)
==16624==    by 0x807C96B: _dl_init_paths (in /home/zturner/test)
==16624==    by 0x805C75B: _dl_non_dynamic_init (in /home/zturner/test)
==16624==    by 0x805CFB5: __libc_init_first (in /home/zturner/test)
==16624==    by 0x8052AC0: (below main) (in /home/zturner/test)
==16624==
==16624== Conditional jump or move depends on uninitialised value(s)
==16624==    at 0x805BF93: __register_atfork (in /home/zturner/test)
==16624==    by 0x805663C: ptmalloc_init (in /home/zturner/test)
==16624==    by 0x8059AB5: malloc_hook_ini (in /home/zturner/test)
==16624==    by 0x8059957: malloc (in /home/zturner/test)
==16624==    by 0x807C96B: _dl_init_paths (in /home/zturner/test)
==16624==    by 0x805C75B: _dl_non_dynamic_init (in /home/zturner/test)
==16624==    by 0x805CFB5: __libc_init_first (in /home/zturner/test)
==16624==    by 0x8052AC0: (below main) (in /home/zturner/test)
==16624==
==16624== Conditional jump or move depends on uninitialised value(s)
==16624==    at 0x805998B: malloc (in /home/zturner/test)
==16624==    by 0x8059957: malloc (in /home/zturner/test)
==16624==    by 0x807C96B: _dl_init_paths (in /home/zturner/test)
==16624==    by 0x805C75B: _dl_non_dynamic_init (in /home/zturner/test)
==16624==    by 0x805CFB5: __libc_init_first (in /home/zturner/test)
==16624==    by 0x8052AC0: (below main) (in /home/zturner/test)
==16624==
==16624== Conditional jump or move depends on uninitialised value(s)
==16624==    at 0x80599C0: malloc (in /home/zturner/test)
==16624==    by 0x8059957: malloc (in /home/zturner/test)
==16624==    by 0x807C96B: _dl_init_paths (in /home/zturner/test)
==16624==    by 0x805C75B: _dl_non_dynamic_init (in /home/zturner/test)
==16624==    by 0x805CFB5: __libc_init_first (in /home/zturner/test)
==16624==    by 0x8052AC0: (below main) (in /home/zturner/test)



Note for the record that I've tried -static-libgcc in conjuction with
-static, but no matter what I do I cannot compile even the simplest
program with static linkage without receiving tons of errors like
this.  Note that in this simple program I'm not even using libstdc++
at all or including any header files of any kind.  What's worse, this
same problem happens across multiple versions of GCC (at the very
least it happens on 4.2.4 and 4.4.0) and multiple linux distributions
/ kernels.

So I guess I have three questions.  1) Is this actually a problem or
are these errors spurious?  2) Why do they disappear when I delete the
-static option?  and 3) Is there a way to fix them?  I've even gone so
far as to manually run collect2 specifying my own hand edited command
line, but nothing I've tried there has worked either.

I feel really insecure just writing these off as spurious warnings
just for the simple fact that our program behaves in unexplainable
ways.

Re: Problem with static linking

[Ian Lance Taylor]

Zachary Turner <divisortheory@gmail.com> writes:

> The codebase is large, but is new to linux.  It was originally
> developed on windows and then ported to linux.  It makes heavy use of
> C++, STL, and boost and we'd like to (if possible) link *everything*
> statically.  This means libc, libgcc, libstdc++, boost, libpthread,
> etc.

This message is not appropriate for the gcc@gcc.gnu.org mailing list.
It would be appropriate for the gcc-help@gcc.gnu.org mailing list.
Please take any followups to gcc-help.  Thanks.

This is actually not a gcc issue.  This is a library issue.  On
GNU/Linux, the library is glibc.  The glibc maintainers have decided
that they do not want to support static linking (I personally disagree
with this position).  They only support dynamically linking against
libc.  Static linking sort of grudgingly works, but some things will
fail.  For example, if your statically linked program does DNS lookups,
it will generally fail on any system which is not running the precise
version of glibc as the system on which it was built.

My guess is that the problems you are encountering are problems
statically linking with glibc.  You can certainly bring these up with
the glibc maintainers--see http://sourceware.org/glibc .  However, they
will ignore you with prejudice.

You may want to consider using -Bstatic and -Bdynamic to statically link
everything except glibc.

Ian

Re: Problem with static linking

[Andrew Haley]

Zachary Turner wrote:
> Hello, I've been trying to write a program that links to static
> libraries, and I've been having a lot of difficulties.  Was wondering
> if someone can help me identify what's going wrong.
> 
> The codebase is large, but is new to linux.  It was originally
> developed on windows and then ported to linux.  It makes heavy use of
> C++, STL, and boost and we'd like to (if possible) link *everything*
> statically.  This means libc, libgcc, libstdc++, boost, libpthread,
> etc.

Ian Taylor has explained that the messages about libc are spurious, so
you can either ignore those or install some kind of a Valgrind exception.

However, I really implore you: by all means link statically to everything
else, but leave libc dynamically linked.  I'm not aware of any reason not
to link libc dynamically, and not doing so leads to a ton of problems.

Andrew.

Re: Problem with static linking

[Jakub Jelinek]

On Wed, Jul 15, 2009 at 09:37:32PM -0500, Zachary Turner wrote:
> So I guess I have three questions.  1) Is this actually a problem or
> are these errors spurious?  2) Why do they disappear when I delete the

They are likely spurious.  You get tons of valgrind warnings with dynamically
linked ld.so/libc.so as well, they are just suppressed through a default valgrind
suppression file.  Obviously it is not possible to do the same with
statically linked programs.  valgrind doesn't grok tricks used e.g. in SSE2 strlen
and a bunch of other memory/string ops.

> -static option?  and 3) Is there a way to fix them?  I've even gone so
> far as to manually run collect2 specifying my own hand edited command
> line, but nothing I've tried there has worked either.

Don't link statically, there are many reasons not to and only very few
reasons for it (primarily exception is some system recovery tools that are
supposed to work even when shared libraries are hosed).
See http://people.redhat.com/drepper/no_static_linking.html

	Jakub

Re: Problem with static linking

[Alfred M. Szmidt]

   However, I really implore you: by all means link statically to
   everything else, but leave libc dynamically linked.  I'm not aware
   of any reason not to link libc dynamically, and not doing so leads
   to a ton of problems.

Problems also arise if one uses functions that use NSS (eg. getXbyY
functions like gethostbyname).  In which case, the GNU C library will
try to do a dlopen on several libraries to find the right one.

Re: Problem with static linking

[Ian Lance Taylor]

Jakub Jelinek <jakub@redhat.com> writes:

>> -static option?  and 3) Is there a way to fix them?  I've even gone so
>> far as to manually run collect2 specifying my own hand edited command
>> line, but nothing I've tried there has worked either.
>
> Don't link statically, there are many reasons not to and only very few
> reasons for it (primarily exception is some system recovery tools that are
> supposed to work even when shared libraries are hosed).
> See http://people.redhat.com/drepper/no_static_linking.html

The main reason to link statically is the inverse of Ulrich's first
reason to link dynamically: if you link dynamically, your program is
vulnerable to changes in the shared libraries.  If your program is
carefully tuned and tested, then changes to shared libraries can
introduce unexpected performance changes, or even, in the worst case,
unexpected bugs.  Another way to say "by fixing a dynamic library you
can fix a bug in every program in one place" is "by distributing a new
dynamic library, you can break every program at once."

I think the argument against static linking does not consider the world
where one builds a carefully tuned executable and then runs it for many
years.  In that environment, each change to a dynamic library requires
careful retesting of all the affected programs.  But it does not follow
that we never want to change the library, because not all programs are
performance sensitive, and new programs require new features.

Ian

Re: Problem with static linking

[Frank Ch. Eigler]

Andrew Haley <aph@redhat.com> writes:

>> [...] It makes heavy use of
>> C++, STL, and boost and we'd like to (if possible) link *everything*
>> statically.  This means libc, libgcc, libstdc++, boost, libpthread,
>> etc.
> [...]
> However, I really implore you: by all means link statically to everything
> else, but leave libc dynamically linked.  I'm not aware of any reason not
> to link libc dynamically, and not doing so leads to a ton of problems.

If they actually encounter that ton of problems, then they will
change their minds about libc, regardless of preemptive imploring.


- FChE

Re: Problem with static linking

[Zachary Turner]

On Thu, Jul 16, 2009 at 12:57 PM, Ian Lance Taylor <iant@google.com> wrote:
>
> Jakub Jelinek <jakub@redhat.com> writes:
>
> >> -static option?  and 3) Is there a way to fix them?  I've even gone so
> >> far as to manually run collect2 specifying my own hand edited command
> >> line, but nothing I've tried there has worked either.
> >
> > Don't link statically, there are many reasons not to and only very few
> > reasons for it (primarily exception is some system recovery tools that are
> > supposed to work even when shared libraries are hosed).
> > See http://people.redhat.com/drepper/no_static_linking.html
>
> The main reason to link statically is the inverse of Ulrich's first
> reason to link dynamically: if you link dynamically, your program is
> vulnerable to changes in the shared libraries.  If your program is
> carefully tuned and tested, then changes to shared libraries can
> introduce unexpected performance changes, or even, in the worst case,
> unexpected bugs.  Another way to say "by fixing a dynamic library you
> can fix a bug in every program in one place" is "by distributing a new
> dynamic library, you can break every program at once."
>
> I think the argument against static linking does not consider the world
> where one builds a carefully tuned executable and then runs it for many
> years.  In that environment, each change to a dynamic library requires
> careful retesting of all the affected programs.  But it does not follow
> that we never want to change the library, because not all programs are
> performance sensitive, and new programs require new features.
>
> Ian

There's also much less to deal with from a Q/A and tech support
perspective if you use static linking with a closed source
application, since you can produce 1 binary which works across
multiple distributions and kernels without the user compiling it,
which is what I'm dealing with.  Not necessarily enough to mandate
linking everything statically, I actually have been advocating for a
while to link everything dynamically.  But it is how it is for now,
and I suspect it will take more nagging and more time for there to be
a real switch

Re: Problem with static linking

[Andrew Haley]

On 07/16/2009 09:08 PM, Zachary Turner wrote:
> 
> There's also much less to deal with from a Q/A and tech support
> perspective if you use static linking with a closed source
> application, since you can produce 1 binary which works across
> multiple distributions and kernels without the user compiling it,
> which is what I'm dealing with.

That's more of an assumption than an advantage.  It may well appply to
some libraries, but libc is really part of the host OS.  Anyone
linking statically with libc on GNU/Linux is probably making a
mistake.

Andrew.