[BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Hal Black]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]

Using GNU ld version 2.13.90.0.18 20030206 (comes with Redhat 9)
Also experienced with previous 3.x and 2.9x versions.

When linking with a bunch of .o files, the desired behavior (static 
initializers are called) happens.  But when linking .a files composed of 
those .o files, static initializers are not called.  On the other hand, 
when the .cpp files are combined to make a single .o file, which is then 
  used to form a .a file, that works.

This was previously reported to gcc bugs, but they sent me here (I've 
been lurking for quite a while to see if it was okay to post bugs here):
http://gcc.gnu.org/cgi-bin/gnatsweb.pl (bug 7769)

There is some good discussion there.

Full source code and makefile follows in the attachment.  See below for 
a quick glance of what I'm talking about.

one.cc
------
#include <stdio.h>

class A {
public:
	A() { printf("Hello world!\n"); }
};
A a;
------

two.cc
------
int main() {}
------

combined.cc
-----------
#include "one.cc"
#include "two.cc"
-----------

Makefile
--------
CPP=g++
test:main1 main2 main3 main4
	./main1
	./main2
	./main3
	./main4
%.o:%.cc
	$(CPP) -c $<
lib%.a:%.o
	ar rv $@ $<
both.a:one.o two.o
	ar rv $@ $^
main1:one.o two.o
	$(CPP) one.o two.o -o main1
main2:libone.a libtwo.a
	$(CPP) libone.a libtwo.a -o main2
main3:both.a
	$(CPP) both.a -o main3
main4:libcombined.a
	$(CPP) libcombined.a -o main4
clean:
	rm -f *.a *.o main*
--------


thanks for your time.
   Hal

[-- Attachment #2: static_init.tar.bz2 --]
[-- Type: application/octet-stream, Size: 554 bytes --]

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Alexandre Oliva]

On Apr 11, 2003, Hal Black <hablack@vt.edu> wrote:

> But when linking .a files composed
> of those .o files, static initializers are not called.

And, if you look closer, you'll notice the object files that contain
the initializers aren't linked in at all.  Which is perfectly fine,
given that no symbols from it are referenced.  Maybe you want
--whole-archive?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Alan Modra]

On Thu, Apr 10, 2003 at 11:43:53PM -0400, Hal Black wrote:
> When linking with a bunch of .o files, the desired behavior (static 
> initializers are called) happens.  But when linking .a files composed of 
> those .o files, static initializers are not called.

See http://sources.redhat.com/ml/binutils/2003-02/msg00155.html

-- 
Alan Modra
IBM OzLabs - Linux Technology Centre

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Hal Black]

Alexandre Oliva wrote:
> On Apr 11, 2003, Hal Black <hablack@vt.edu> wrote:
> 
> 
>>But when linking .a files composed
>>of those .o files, static initializers are not called.
> 
> 
> And, if you look closer, you'll notice the object files that contain
> the initializers aren't linked in at all.  Which is perfectly fine,
> given that no symbols from it are referenced.  Maybe you want
> --whole-archive?
> 

Thanks Alexandre, for replying and the --whole-archive tip - it works 
great (has to be -Wl,-whole-archive when passed to g++, though, as noted 
on the man page).

I still believe this is a bug, however.  Even though there aren't any 
references to static intializers, there is an implicit reference to 
them.  When using .o files, there are no references either, but the 
static intializer is included in that case.  Why should it be different 
for a .a file with the same .o files in it?  Should .a files not be used 
for C++?

Perhaps the solution is to pass --whole-archive by default to ld when 
linking with g++.

But, at least there is a workaround.

Thanks also to Alan for the reference, I don't know how I missed that 
thread.

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Alexandre Oliva]

On Apr 12, 2003, Hal Black <hablack@vt.edu> wrote:

> Why should it be different for a .a file with the same .o files in
> it?  Should .a files not be used for C++?

Because .o files are copied to the output, whereas .a files are used
as libraries, i.e., only members that offer symbols that are being
looked for are copied to the output.  Can you imagine the bloat if any
time you linked with a static library, all of the code in it was
copied to your executables?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Hal Black]

Alexandre Oliva wrote:
> On Apr 12, 2003, Hal Black <hablack@vt.edu> wrote:
> 
> 
>>Why should it be different for a .a file with the same .o files in
>>it?  Should .a files not be used for C++?
> 
> 
> Because .o files are copied to the output, whereas .a files are used
> as libraries, i.e., only members that offer symbols that are being
> looked for are copied to the output.  Can you imagine the bloat if any
> time you linked with a static library, all of the code in it was
> copied to your executables?

What I'm saying is not to use all the symbols, but to include all static 
initializers as "symbols that are being looked for".  Even if they 
aren't explicitly referenced by annother piece of code, they are 
implicitly referenced by virtue of being static initializers.


------------------
   From the C++ draft spec 2 December 1996:

  Section 3.7.1:
  2 If an object of static storage duration has initialization or a
  destructor with side effects, it shall not be eliminated even if it
  appears to be unused, except that a class object or its copy may be
  eliminated as specified in 12.8.
------------------

I would say that not looking for static initializers constitutes 
eliminating them...

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Ian Lance Taylor]

Hal Black <black@ieee.org> writes:

> What I'm saying is not to use all the symbols, but to include all
> static initializers as "symbols that are being looked for".  Even if
> they aren't explicitly referenced by annother piece of code, they are
> implicitly referenced by virtue of being static initializers.

That would mean that every static initializer in a .a file would be
brought in, which would doubtless bring in other object files as well.
It means that including any class with a static initializer in a .a
file would mean that that class would be included in all programs
linked with that .a file.  This would go against the very idea of a .a
file, which is that only required objects are included.

> I would say that not looking for static initializers constitutes
> eliminating them...

No, because they were never included in the first place.

It's reasonably easy to avoid this problem: put the static initializer
in the same source file as a class virtual function.  Since the
class virtual function must be included, the initializer will be
brought in as well.

If you don't have any virtual functions, then you do need to take some
other action to force that object file to be included, such as using
global variables in some way.

Or maybe .a files are simply the wrong tool for the job.  Maybe you
should use ld -r to form several .o files into one large .o file.
Linking against that will ensure that everything is brought in.

Ian

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Hal Black]

Thanks for replying, Ian.

Ian Lance Taylor wrote:

> Hal Black <black@ieee.org> writes:
> 
> 
>>What I'm saying is not to use all the symbols, but to include all
>>static initializers as "symbols that are being looked for".  Even if
>>they aren't explicitly referenced by annother piece of code, they are
>>implicitly referenced by virtue of being static initializers.
> 
> 
> That would mean that every static initializer in a .a file would be
> brought in

Yes, as desired.

 > which would doubtless bring in other object files as well.

If the static initializer calls other functions, then those would be 
required as well, just as normal ld behavior.

> It means that including any class with a static initializer in a .a
> file would mean that that class would be included in all programs
> linked with that .a file.

Yes.  That's the desired behavior.  If a developer were to develop a 
library with a static initializer with side-effects, it would be 
something that should be called whenever that library was used.

Other classes with no static initializers would only be used as needed, 
as they are now.

If you don't think this is the proper usage, what is your interpretation 
of the meaning of having a static intializer (or items with static 
storage duration in general) in a library?

 >  This would go against the very idea of a .a
> file, which is that only required objects are included.

My claim is that if it has a static initializer, it is required.  If it 
is called from any included object, it is required (as it works now). 
Therefore this doesn't break the idea the only required objects are 
included.

-----
Also, just to clarify since it got lost in the quoting, I'm talking 
about this scenario:

class A {
public:
     A() { printf("Hello world!\n"); }
};
A a;

I claim if this code is put in a library, printf should be called. 
Because "a" has static storage duration and has side effects, its 
initializer cannot be eliminated per 3.7.1.2 of the draft standard.

So, when I say static initializer in the context of this discussion, the 
above code fragment is what I mean.

>>I would say that not looking for static initializers constitutes
>>eliminating them...
> 
> No, because they were never included in the first place.

The claim is that never including them in the first place is elimination.

> It's reasonably easy to avoid this problem: put the static initializer
> in the same source file as a class virtual function.  Since the
> class virtual function must be included, the initializer will be
> brought in as well.

The claim is that static initializers as specified above are in this 
same category of things that must be included.

> If you don't have any virtual functions, then you do need to take some
> other action to force that object file to be included, such as using
> global variables in some way.
> 
> Or maybe .a files are simply the wrong tool for the job.  Maybe you
> should use ld -r to form several .o files into one large .o file.
> Linking against that will ensure that everything is brought in.

Thanks for large .o idea.  This, and other workarounds such as the one 
provided by Alexandre, would work great for my immediate purposes.

However, I'm not having a problem finding a workaround, I'm addressing 
the point of how static initializers should behave in a library.  The 
workarounds proposed, though they work great as a fix for what I was 
having problems with, don't provide library functionality (only include 
what's needed).  Also, they require extra steps to use common 
implementations of automatic registration patterns in a library.

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Alexandre Oliva]

On Apr 12, 2003, Hal Black <black@ieee.org> wrote:

>> That would mean that every static initializer in a .a file would be
>> brought in

> Yes, as desired.

Maybe in your specific application.  But think of the Standard C++
library.  It may contain hundreds, if not thousands, of global
initializers, that are of no use for most programs that don't happen
to use the particular feature that depend on some of these
initializers.  Bringing them in just because you would like it to be
so is not exactly a reasonable proposition.

> If you don't think this is the proper usage, what is your
> interpretation of the meaning of having a static intializer (or items
> with static storage duration in general) in a library?

No different from having it in an object file: if the object file is
linked in, the static initializer is run.  The difference is that
object files listed in the command line are always linked in, whereas
those in a static library get linked in only if they would resolve
some symbol the linker is looking for.

> My claim is that if it has a static initializer, it is required.

If we implemented this mis-feature, you'll come back tomorrow and
complain about the bloat from all these modules being linked in that
are not needed for your program to run, and the answer will be that
they do contain static initializers so, per your request, they have to
be brought in.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Hal Black]

Alexandre Oliva wrote:

> On Apr 12, 2003, Hal Black <black@ieee.org> wrote:
> 
> 
>>>That would mean that every static initializer in a .a file would be
>>>brought in
> 
> 
>>Yes, as desired.
> 
> 
> Maybe in your specific application.  But think of the Standard C++
> library.  It may contain hundreds, if not thousands, of global
> initializers, that are of no use for most programs that don't happen
> to use the particular feature that depend on some of these
> initializers.

I'm not an expert on g++ symbols, but I'm assuming the static intializer 
of the sort I'm talking about is is named 
__static_initialization_and_detruction.* because that's what nm reports 
for my test program.

When I run nm on libstdc++*.a and grep for static_initialization, I get 
not thousands, not hundreds, but zero matches.  So, if I'm right about 
the naming, the program would get no bigger if the changes I suggested 
were implemented.

OTOH, I could be totally wrong.  8')  I really know nothing about the 
details of how g++ does naming of C++ constructs.

[caveat: I don't have access to the latest stuff right now, this is with 
mandrake 9.0 - maybe they have added thousands of static initializers 
since then]

I counted 18 static initializers total in /usr/lib.  I don't know 
whether these are meant to be called every time the library is linked 
against, or just when a certain feature is used.

 > Bringing them in just because you would like it to be
> so is not exactly a reasonable proposition.

It's not a question of me liking it to be so.  I have already stated 
that I am fine with the work around.  I am raising the point how the C++ 
spec is interpreted in terms of libraries for discussion.  The way to 
dispute this point is to argue against the point, not wave it off 
because you infer endearment of an individual to it, which is irrelevant 
on whether it is the right way to do it or not.

I just hadn't heard any valid reasons against having it be as I 
describe, and it seems like the correct interpretation of the spec.

>>If you don't think this is the proper usage, what is your
>>interpretation of the meaning of having a static intializer (or items
>>with static storage duration in general) in a library?
> 
> 
> No different from having it in an object file: if the object file is
> linked in, the static initializer is run.  The difference is that
> object files listed in the command line are always linked in, whereas
> those in a static library get linked in only if they would resolve
> some symbol the linker is looking for.

I understand the behavior of the tools.  What I asked was what is the 
meaning of having objects with static storage in a library.

Based on an earlier portion of your reply, I am taking your 
interpretation to mean that a static initializer in a library is only 
called when something references an instance of its class or members, 
but not otherwise.  I don't believe this is a valid view from reading 
the C++ spec.

>>My claim is that if it has a static initializer, it is required.
> 
> 
> If we implemented this mis-feature, you'll come back tomorrow and
> complain about the bloat from all these modules being linked in that
> are not needed for your program to run, and the answer will be that
> they do contain static initializers so, per your request, they have to
> be brought in.

If they were not needed for the program to run, the behavior (not 
counting performance/memory footprint for the moment) would be the same 
whether they were included or not.  Desired side-effects of the static 
initializers don't happen when they're not included, therefore they are 
required for the program to run.

I see your point that if the custom is to put in instances of static 
storage whose initializers aren't meant to be called all the time, some 
uneccesary bloat could result.  If they are supposed to be called, 
however, they aren't unneccesary.

I don't know whether the 18 static initializers in my /usr/lib are meant 
to be used the way I describe or the way you describe, so I can't say 
whether these are unneccesary or neccesary.

Regardless, if my interpretation of the C++ spec is correct, it is a 
flaw in the library to create static storage items with initializers 
with side effects that aren't meant to be run for all users of the 
library.  In that case, the linker would be following the spec and would 
deserve no complaints.  It's not the auto-maker's fault if the car is 
working fine and you get caught speeding.

But, if you don't want to implement it to avoid complaints, it's your 
project, not mine, so I can't argue with that.

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Ian Lance Taylor]

Hal Black <black@ieee.org> writes:

> > It means that including any class with a static initializer in a .a
> > file would mean that that class would be included in all programs
> > linked with that .a file.
> 
> Yes.  That's the desired behavior.  If a developer were to develop a
> library with a static initializer with side-effects, it would be
> something that should be called whenever that library was used.

No Unix linker has ever worked that way.

Basically, you misunderstand how ld and ar work.  If you feel that you
have been misled by existing documentation, please let us know so that
we can fix it.

Also, while this is not the reason for not making the change, it turns
out that it is not an efficient change to make.  It would require the
linker to examine every object in every archive, which obviates the
entire point of having archives in the first place.  Again, this
inefficiency, while severe, is not the reason for not making the
change; the reason for not making the change is that ld and ar
implement behaviour which has not changed for decades, and should not
be changed now.

Ian

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Zack Weinberg]

Ian Lance Taylor <ian@airs.com> writes:

> Hal Black <black@ieee.org> writes:
>> Yes.  That's the desired behavior.  If a developer were to develop a
>> library with a static initializer with side-effects, it would be
>> something that should be called whenever that library was used.
>
> No Unix linker has ever worked that way.
>
> Basically, you misunderstand how ld and ar work.  If you feel that you
> have been misled by existing documentation, please let us know so that
> we can fix it.

I'd like to point out that while this is true for static (.a)
libraries, it is not true for shared (.so) libraries -- the linker
generates DT_NEEDED entries for every shared library on the command
line, whether or not they satisfy undefined symbols.  Personally I
consider this a bug.

zw

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Ian Lance Taylor]

Zack Weinberg <zack@codesourcery.com> writes:

> Ian Lance Taylor <ian@airs.com> writes:
> 
> > Hal Black <black@ieee.org> writes:
> >> Yes.  That's the desired behavior.  If a developer were to develop a
> >> library with a static initializer with side-effects, it would be
> >> something that should be called whenever that library was used.
> >
> > No Unix linker has ever worked that way.
> >
> > Basically, you misunderstand how ld and ar work.  If you feel that you
> > have been misled by existing documentation, please let us know so that
> > we can fix it.
> 
> I'd like to point out that while this is true for static (.a)
> libraries, it is not true for shared (.so) libraries -- the linker
> generates DT_NEEDED entries for every shared library on the command
> line, whether or not they satisfy undefined symbols.  Personally I
> consider this a bug.

I think that one is debatable.  It's certainly true that shared
libraries and archive libraries don't work the same way; as you say,
one obvious difference is in fact the very one we are talking
about--all static initializers in a shared library are run, whether
they are required or not.  There are a number of other
differences--only in simple case can you just replace a .a file with a
.so file.

I think the general attitude of linker implementors has been that if
you list a shared library on the command line, you must want it in
there.  That is, a .so file is in that sense more like a .o file than
a .a file.

But I agree that it could be argued the other way.  I don't know of
any ELF linker which does not generate a DT_NEEDED for a .so listed on
the command line, but perhaps there is one.

Ian

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Ulrich Drepper]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Zack Weinberg wrote:

> I'd like to point out that while this is true for static (.a)
> libraries, it is not true for shared (.so) libraries -- the linker
> generates DT_NEEDED entries for every shared library on the command
> line, whether or not they satisfy undefined symbols.  Personally I
> consider this a bug.

Agreed, I argued like that for years.  If somebody wants the DSO
included s/he can use -u to create and artificial dependency.

- -- 
- --------------.                        ,-.            444 Castro Street
Ulrich Drepper \    ,-----------------'   \ Mountain View, CA 94041 USA
Red Hat         `--' drepper at redhat.com `---------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+mfH72ijCOnn/RHQRAtHmAKCnq3OcfkKWRnvmmZI9cyTLCzkeGgCfej6g
ZErCJSskzy+8TJD3Ld+CWos=
=9jUv
-----END PGP SIGNATURE-----

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Hal Black]

Ian Lance Taylor wrote:

> Hal Black <black@ieee.org> writes:
> 
> 
>>>It means that including any class with a static initializer in a .a
>>>file would mean that that class would be included in all programs
>>>linked with that .a file.
>>
>>Yes.  That's the desired behavior.  If a developer were to develop a
>>library with a static initializer with side-effects, it would be
>>something that should be called whenever that library was used.
> 
> 
> No Unix linker has ever worked that way.

Well, for most of the lifetime of Unix, there were no static 
initializers, since there was no C++, so this is not surprising.

> Also, while this is not the reason for not making the change, it turns
> out that it is not an efficient change to make.  It would require the
> linker to examine every object in every archive, which obviates the
> entire point of having archives in the first place.  Again, this
> inefficiency, while severe, is not the reason for not making the
> change; the reason for not making the change is that ld and ar
> implement behaviour which has not changed for decades, and should not
> be changed now.

For the C code that ld has been linking for decades, I agree with you 100%.

But, C++ has additional requirements.  Either the tool needs to change 
to meet these requirements, or a different tool needs to be used to meet 
them.

If .a is meant only for what it has been used for decades without 
changing to accomodate the full C++ specification, you're right, I 
misunderstood the usage for it.  That is definately not what I want. 
Based on the other replies, I guess that .so can be used to acheive the 
desired behavior even though it includes too much.

Anyway, thank you for responding and for your time, Ian.

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Ian Lance Taylor]

Hal Black <black@ieee.org> writes:

> > No Unix linker has ever worked that way.
> 
> Well, for most of the lifetime of Unix, there were no static
> initializers, since there was no C++, so this is not surprising.

C++ has had static initializers at least since 1985, so it's not all
that new.

> If .a is meant only for what it has been used for decades without
> changing to accomodate the full C++ specification, you're right, I
> misunderstood the usage for it.  That is definately not what I
> want. Based on the other replies, I guess that .so can be used to
> acheive the desired behavior even though it includes too much.

I'm not sure in what sense a .so file includes too much.  A .so file
is a very rough replacement for a .a file, so the fact that it
includes everything which is in the .a file is a feature.  It's true
that a .so file will always run all the static initializers which are
found in the .a file, but that's the very feature you want to
implement.  There are various differences in the implementation of
common symbols, but C++ disallows common symbols so that shouldn't
matter too much.  Warning symbols no longer work correctly, but that
is a GNU extension anyhow.  A .so is admittedly slightly slower, but
that rarely matters these days.

So what are your concerns with using a .so?

Ian

Re: [BUG] ld behavior varies for C++ static initializer depending on .a or .o input

[Alexandre Oliva]

On Apr 12, 2003, Hal Black <black@ieee.org> wrote:

> I'm not an expert on g++ symbols, but I'm assuming the static
> intializer of the sort I'm talking about is is named
> __static_initialization_and_detruction.* because that's what nm
> reports for my test program.

Watch out for .ctor, .init or .init_array sections as well.

> I just hadn't heard any valid reasons against having it be as I
> describe, and it seems like the correct interpretation of the spec.

I disagree.  The compiler, which is the entity that deals with
translation units, isn't dropping any initializers, just like mandated
by the spec.  Now, by the Standard, a program is one or more
translation units linked together [basic.link], and that's it.
Nowhere does it get into details of what a linker is, nevermind how
static archives should behave, since this is all far beyond the scope
of the C++ Standard.

Your point that archives should be scanned for initializers would be
perfectly reasonable, except for the bloat one wouldn't be able to
avoid, but, like Ian says, this is not the way linkers have dealt with
archives in the past and, barring a strong reason to change, they
should change.  Standard compliance is not such a reason, because the
nowhere does the standard determine how to select the translation
units that go into a program.  

> Based on an earlier portion of your reply, I am taking your
> interpretation to mean that a static initializer in a library is only
> called when something references an instance of its class or members,
> but not otherwise.  I don't believe this is a valid view from reading
> the C++ spec.

My interpretation is that a static initializer in a translation unit
is only executed if the translation unit makes it to the program.  The
point under debate is whether the translation unit must make it to the
program in spite of not being referenced by any other translation unit
listed in the program.  If you have such requirements, using an
archive is the wrong approach: you want to list the object files
instead.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer