Re: using templates in shared library

Re: using templates in shared library

[Mike Stump]

> From: Kasper Peeters <K.Peeters@damtp.cam.ac.uk>
> Date: Wed, 9 Dec 1998 11:37:03 +0000 (GMT)

> > You can always use -fno-implicit-templates or -frepo

> I know, but I like the automatic instantiation.

> If I have a few zillion other templates in main.cc it's a pain
> having to instantiate all them explicitly by hand.

This doesn't sound like -frepo, it is not a manual scheme, it is meant
to be an automatic scheme.

Re: using templates in shared library

[Joe Buck]

[ deleting duplicate template instantiations at link time ]

> > Ideally, we should also be able to do the same with shared libraries.
> 
> This was my first impression, as well. But then, the implicit
> instantiation in the shared library might go away, which would mean
> that we have to recompile all applications relying on it.

Just the same, some libraries may be template-heavy enough so that
if the library doesn't have the needed template instantiations, far
less code will be shared.

If we want the library to be highly reusable, and are willing to pay a
space penalty to achieve this, we can get rid of all but one of the
instantiations in the library, and use that instantiation to satisfy
internal references (for calls within the library), but not export it to
external code.

But some people may want maximum space efficiency even if it means
more recompiles for library changes.

Re: using templates in shared library

[Martin von Loewis]

> Ideally, we should also be able to do the same with shared libraries.

This was my first impression, as well. But then, the implicit
instantiation in the shared library might go away, which would mean
that we have to recompile all applications relying on it. This is not
what we want: The user didn't *request* that the instantiation is in
the shared library, she shouldn't get into trouble if it suddenly
isn't anymore (because of, say, removing an unused variable in a
function).

Regards,
Martin

Re: using templates in shared library

[Kasper Peeters]

> a) he who uses shared libraries always has to include code that is
>    useless to some application; linking statically gives better ways
>    to omit unneeded things

He who's always linking statically has been sleeping for the last
decade or so. I can't make modular systems with re-usable code if
everything is in one binary.

> b) why do you care about the duplicate instantiations? Your program
>    still should work, it is just using somewhat more disk space.

In my case, it's at least 3 times as big as necessary ('library 1' of
my example is rather big, while 'library 2' happens to be fairly
small). If it was something like 10% I could live with it, now it just
looks really stupid.

Those code duplications will have to be downloaded by people as
well. Most people still feel the difference when your binary goes up
in size by a factor of 3.

Kasper

Re: using templates in shared library

[Martin von Loewis]

> Ok, I'll redo the design, this is obviously not a feature that anyone
> has thought of before ;-)

There are actually two different conclusions

a) he who uses shared libraries always has to include code that is
   useless to some application; linking statically gives better ways
   to omit unneeded things

b) why do you care about the duplicate instantiations? Your program
   still should work, it is just using somewhat more disk space.

Regards,
Martin

Re: using templates in shared library

[Joe Buck]

> All symbols _are_ already instantiated in the library. The question is
> how to get rid of them in the library or program linking to it.

The GNU linker already removes redundant templates from an executable.
It can do this because templates are put into special "linkonce" sections.
This feature works on ELF platforms such as Linux or Solaris.  This
means that you are paying in disk space to store extra templates in
.o files, but you are not paying for disk space or memory for executable
files.

Ideally, we should also be able to do the same with shared libraries.

Re: using templates in shared library

[Kasper Peeters]

> If you want the linker to consider this instantiation intentional, and
> as a public interface of the library, you need to instantiate
> explicitely.

All symbols _are_ already instantiated in the library. The question is
how to get rid of them in the library or program linking to it. I
would maybe be happy with a method to tell the compiler by hand which
templates it should _not_ instantiate, but there's no such 1-x thing
it seems.

Instantiating manually is, for all practical purposes with loads of
STL in use, a nightmare. 

Ok, I'll redo the design, this is obviously not a feature that anyone
has thought of before ;-) I wonder how other compilers handle this though.

Kasper

Re: using templates in shared library

[Martin von Loewis]

> Is there any reason why _in principle_ the linker couldn't do what I
> want?

I suppose it could. In order to do drop a linkonce section, it would
need to determine that all weak symbols are already provided
elsewhere. This is not how linkonce currently works: the linker drops
a linkonce section if there is a duplicate of the entire section (i.e
a section with the same name). When dropping a section, it doesn't care
what symbols it looses.

Regards,
Martin

Re: using templates in shared library

[Martin von Loewis]

> and link it to the library. Obviously, since the library already
> exists, it is possible to see that 'vector<int>' is present in the
> library. I therefore want the 'main.o' or 'a.out' NOT to include the
> 'vector<int>' instance. It will only make it bigger if the symbols
> were included, since the library is going to override the weak symbols
> anyway.

This is not how shared-library versioning works. A newer version of
the library still needs to export the same set of global symbols to be
binary compatible.

However, if an function implementation slightly changes, people do
expect that they don't need to recompile all applications. With your
proposal implemented, they might need to: the slight change could
cause an implicit template instantiation to go away, and programs
relying on the implicit instatiation to be in the library would break.

If you want the linker to consider this instantiation intentional, and
as a public interface of the library, you need to instantiate
explicitely.

Regards,
Martin

Re: using templates in shared library

[Jason Merrill]

>>>>> Kasper Peeters <K.Peeters@damtp.cam.ac.uk> writes:

 > On my machine, -frepo repeatedly compiles my sources until all
 > templates are done. When it does that n times, it is roughly n times
 > slower than the default, which instantiates all templates
 > immediately.

This is only true for the first link.  On subsequent links, the cached
information from before speeds the process up greatly.

Jason

Re: using templates in shared library

[Kasper Peeters]

> ?  Give us some hard numbers as to the cycles wasted.  I don't think
> it'll amount to much in a real project.

On my machine, -frepo repeatedly compiles my sources until all
templates are done. When it does that n times, it is roughly n times
slower than the default, which instantiates all templates
immediately. (I know, it's not that bad, but this estimate is pretty
close in a real-world, slow nfs, bad caching situation).

> I see no reason why it can't, we look forward to a submission for this
> enhancement from you.  :-)

Sorry, there are only 25 hours in a day and mine are filled twice
already. If someone could hold my hand and point me at the proper
place in the source tree I might be tempted to give it a try, but I
don't have time for an extensive study of the egcs internals first.

I'm just a little surprised to hear that this feature is not already
somewhere high on the wish list or even implemented a long time
ago. Does everyone really do all instantiations by hand when things
get more complicated than creating a single large binary? Or do you
all accept duplicate code on disk? I've never seen any project that
uses the -frepo solution in the way Mike proposed it either.

Kasper

Re: using templates in shared library

[Mike Stump]

> From: Kasper Peeters <K.Peeters@damtp.cam.ac.uk>
> Date: Wed, 9 Dec 1998 20:43:54 +0000 (GMT)

> But more importantly: you can't be serious about pushing this as the
> proper method to do it. -frepo wastes CPU cycles

?  Give us some hard numbers as to the cycles wasted.  I don't think
it'll amount to much in a real project.

> and during build I have to confront my poor users with screens full
> of error messages?  Yes, I can redirect to /dev/null too, but this
> is definitely not what I want.

?  If they are redirected, why would your poor users be confronted
with them.  I agree it does feel a little messy though.

> Is there any reason why _in principle_ the linker couldn't do what I
> want? It seems so completely trivial to me (at least, _much_ easier
> than most other template things egcs can do these days).

I see no reason why it can't, we look forward to a submission for this
enhancement from you.  :-) If you want to wait for someone else to do
it, it may take a while.

Re: using templates in shared library

[Kasper Peeters]

> Now, you can try that or let us know if it doesn't work and why.

It fails on including something, but that's probably my fault. The
idea should work I agree.

But more importantly: you can't be serious about pushing this as the
proper method to do it. -frepo wastes CPU cycles, and during build I
have to confront my poor users with screens full of error messages?
Yes, I can redirect to /dev/null too, but this is definitely not what
I want.

Is there any reason why _in principle_ the linker couldn't do what I
want? It seems so completely trivial to me (at least, _much_ easier
than most other template things egcs can do these days).

Kasper

Re: using templates in shared library

[Mike Stump]

> From: Kasper Peeters <K.Peeters@damtp.cam.ac.uk>
> Date: Wed, 9 Dec 1998 19:27:39 +0000 (GMT)

I agree with your frustration...  We need better docs for this stuff.

> Unless I misunderstood your procedure, this also doesn't work. Let
> me state my problem again: I am building _two_ shared libraries and
> _no_ executables. Library 1 uses vector<int> and so does library
> 2. When I build library 1, the default egcs options will result in
> vector<int> being instantiated in library 1. Good. Then I build
> library 2 and during the link phase, I tell it to link dynamically
> to library 1.

> It should be trivial for the linker to figure out that library 1
> already has vector<int>. I want it to drop the instantiation of
> vector<int> from library 2. 

> The -frepo only seems to work if I have an executable too. I could
> make a trivial demo that uses everything in library 1 and 2, but that
> makes the compiler recompile the demo, not the library. 

Assume we have the structure:

       lib1: lib1a.cc lib1b.cc
       lib2: lib2a.cc lib2b.cc lib1.so
       app: app.cc lib1.so lib2.so

You:
	gcc -frepo -c lib1a.cc
	gcc -frepo -c lib1b.cc
	-gcc -frepo -o delete_me lib1a.o lib1b.o
	gcc -shared -o lib1.so lib1a.o lib1b.o

	gcc -frepo -c lib2a.cc
	gcc -frepo -c lib2b.cc
	-gcc -frepo -o delete_me lib2a.o lib2b.o lib1.so
	gcc -shared -o lib2.so lib2a.o lib2b.o

Now, you can try that or let us know if it doesn't work and why.

For the app, one can use default compile or if one wants, one can use
-frepo.

	gcc -o app app.cc lib2.so lib1.so
or
	gcc -frepo -c app.cc
	gcc -frepo -o app app.o lib2.so lib1.so

I think this should work.

Re: using templates in shared library

[Kasper Peeters]

> -frepo.  You do a full link with the two on the command line, and let
> it error out.  But as a side effect, it will instantiate everything
> just once.

Unless I misunderstood your procedure, this also doesn't work. Let me
state my problem again: I am building _two_ shared libraries and _no_
executables. Library 1 uses vector<int> and so does library 2. When I
build library 1, the default egcs options will result in vector<int>
being instantiated in library 1. Good. Then I build library 2 and
during the link phase, I tell it to link dynamically to library 1. 

It should be trivial for the linker to figure out that library 1
already has vector<int>. I want it to drop the instantiation of
vector<int> from library 2. 

The -frepo only seems to work if I have an executable too. I could
make a trivial demo that uses everything in library 1 and 2, but that
makes the compiler recompile the demo, not the library. 

Kasper

RE: using templates in shared library

[Kasper Peeters]

> -fno-implicit-templates might do what you want.

That flag doesn't just drop duplicates, it drops a lot more, forcing
me to instantiate things by hand. 

Kasper

Re: using templates in shared library

[Mike Stump]

> From: Kasper Peeters <K.Peeters@damtp.cam.ac.uk>
> Date: Wed, 9 Dec 1998 11:17:48 +0000 (GMT)

> But regardless of the philosophy, let me rephrase my question: how
> can I force the compiler NOT to instantiate symbols in an object
> file which are already present in a given shared library on my
> system?

-frepo.  You do a full link with the two on the command line, and let
it error out.  But as a side effect, it will instantiate everything
just once.

Re: using templates in shared library

[Kasper Peeters]

> > Well, the first library is just required to contain the symbols
> > associated to my templates. If a new version doesn't have them, it's
> > not a new version but a broken version.
>
> The linker coulnd't know that :-)

It can. Maybe I'm not explaining things very well. Say I have a
library with all code/symbols for

   vector<int>

Now I make a program main.cc, 

   main()
      {
      vector<int> foo;
      }

and link it to the library. Obviously, since the library already
exists, it is possible to see that 'vector<int>' is present in the
library. I therefore want the 'main.o' or 'a.out' NOT to include the
'vector<int>' instance. It will only make it bigger if the symbols
were included, since the library is going to override the weak symbols
anyway.

> You can always use -fno-implicit-templates or -frepo, together with
> explicit template instantiation, to control where templates should
> go.

I know, but I like the automatic instantiation. The linker can look
into the shared library and see which symbols are there. I _only_ want
to discard those in my new object file, all the others should go in!

If I have a few zillion other templates in main.cc it's a pain having
to instantiate all them explicitly by hand. 

Kasper

Re: using templates in shared library

[Alexandre Oliva]

On Dec  9, 1998, Kasper Peeters <K.Peeters@damtp.cam.ac.uk> wrote:

>> that's because the component object files contain those symbols
>> already.  I don't think the linker should remove those symbols from
>> a shared library; can you think of the implications of replacing the
>> other library your library depended upon?

> Well, the first library is just required to contain the symbols
> associated to my templates. If a new version doesn't have them, it's
> not a new version but a broken version.

The linker coulnd't know that :-)

> But regardless of the philosophy, let me rephrase my question: how
> can I force the compiler NOT to instantiate symbols in an object
> file which are already present in a given shared library on my
> system?

You can always use -fno-implicit-templates or -frepo, together with
explicit template instantiation, to control where templates should
go.

-- 
Alexandre Oliva  http://www.dcc.unicamp.br/~oliva  aoliva@{acm.org}
oliva@{dcc.unicamp.br,gnu.org,egcs.cygnus.com,samba.org}
Universidade Estadual de Campinas, SP, Brasil

Re: using templates in shared library

[Kasper Peeters]

> that's because the component object files contain those symbols
> already.  I don't think the linker should remove those symbols from
> a shared library; can you think of the implications of replacing the
> other library your library depended upon?

Well, the first library is just required to contain the symbols
associated to my templates. If a new version doesn't have them, it's
not a new version but a broken version.

But regardless of the philosophy, let me rephrase my question: how can
I force the compiler NOT to instantiate symbols in an object file
which are already present in a given shared library on my system?

(Please CC to me, I'm not on this list)

Kasper

Re: using templates in shared library

[Alexandre Oliva]

On Dec  8, 1998, Kasper Peeters <K.Peeters@damtp.cam.ac.uk> wrote:

> I have a shared library which contains instances of my template
> classes; 'nm --dynamic --defined' shows them as W, weak symbols. When
> I use these template classes in a second library and then link with
> the first one, the second library _still_ contains instances of these
> templates too! (ie. nm shows exactly the same 'W' symbols).

that's because the component object files contain those symbols
already.  I don't think the linker should remove those symbols from
a shared library; can you think of the implications of replacing the
other library your library depended upon?

-- 
Alexandre Oliva  http://www.dcc.unicamp.br/~oliva  aoliva@{acm.org}
oliva@{dcc.unicamp.br,gnu.org,egcs.cygnus.com,samba.org}
Universidade Estadual de Campinas, SP, Brasil

using templates in shared library

[Kasper Peeters]

Sorry for asking this question here, but I wasn't able to find
appropriate documentation anywhere else (is there an up to date egcs
manual on the web somewhere?)

(this is all using egcs-1.1 on linux/ix86 with binutils 2.9.1)

I have a shared library which contains instances of my template
classes; 'nm --dynamic --defined' shows them as W, weak symbols. When
I use these template classes in a second library and then link with
the first one, the second library _still_ contains instances of these
templates too! (ie. nm shows exactly the same 'W' symbols).

Is there a way to tell the linker not to bother and use the instances
in the first library instead? Or am I misinterpreting my nm output?
(it's of course only a diskspace problem, but still).

Please CC to me; I'm not on this list.

Kasper