Incremental linking

Incremental linking

[Dave Taylor]

I've been noticing the "-r" command in ld for some time now, and as our
project is almost at 1000 files and takes a solid 30 seconds to link with
debugging symbols, I've been lusting to figure out how incremental linking
works.

I grok that -r generates an output file which in turn can be used to link
with, and I've gotten that to work. 

However, the desired use for this relinkable file is to be able to say
something like:

gcc relinkable.o module1.o module2.o ... -o my_proggy

Where module1.o and module2.o may already be in the relinkable.o file, but
because they've been specified on the command line, the symbols in those
module*.o files override the symbols in the relinkable.o file.  Is there a
way to get ld to behave this way?  Or was that half of the incremental
linking feature never implemented?  Because when I try that, it complains
that there are duplicate symbols and is clearly not trying to override the
old symbols. 

I'm interested in any alternatives you know of to shorten the link time.

A big problem seems to be that the output file is a whopping 12Mb, and the
*.o files take up a healthy 50Mb themselves.  This stuff is partially
bloated because of the size of the debugging info added by -g.  We've
noticed that MSVC puts the debugging symbols in a seperate file.  Perhaps
they did this to speed up linking? 

We're using Linux, and we've already tried using RAM drives and getting
lots of RAM.  Unfortunately, we don't entirely grok how to allocate a
certain amt of RAM to the ramdrive.  The technique shown to us was to
mke2fs on /dev/ram1, specifying how many blocks, and then mounting the
/dev/ram1 file.  Doing this made my system unstable, and interestingly, I
couldn't umount it.

We've also tried creating shared libraries which I guess basically defers
the link until each module is needed at execution time.  This speeds
things up rather a lot, but we noticed that gdb doesn't debug shared
libraries very well for us.  If you step into a function that's in a
shared library, instead of popping in, it gets confused about what line
number you're on, or it'll start stepping through what looks like
hash-table glue code.  Not being able to debug hurts.

We're using egcs 1.0.2, and we make extensive use of C++.  I've heard
rumors that libg++ isn't terribly well-tested for interoperability with
other libs.. ?

Thanks for any comments or suggestions.  My e-mail is ddt@crack.com.

	=-ddt->

---------
Dave Taylor, Owner
Crack dot Com, Inc.  http://crack.com

Re: Incremental linking

[Jeffrey A Law]

  In message < Pine.LNX.3.96.980409133155.9469B-100000@crack.crack.com >you write:
  > I grok that -r generates an output file which in turn can be used to link
  > with, and I've gotten that to work. 
Right.  Basically it just takes all the given .o files and makes one
big .o file from them.  Generally that's all that happens -- ie no
symbol resolution is done by most linkers for -r.


  > However, the desired use for this relinkable file is to be able to say
  > something like:
  > 
  > gcc relinkable.o module1.o module2.o ... -o my_proggy
  > 
  > Where module1.o and module2.o may already be in the relinkable.o file, but
  > because they've been specified on the command line, the symbols in those
  > module*.o files override the symbols in the relinkable.o file.  Is there a
  > way to get ld to behave this way?
ld doesn't work that way.

What you want is shared libraries :-)

  > A big problem seems to be that the output file is a whopping 12Mb, and the
  > *.o files take up a healthy 50Mb themselves.  This stuff is partially
  > bloated because of the size of the debugging info added by -g.  We've
  > noticed that MSVC puts the debugging symbols in a seperate file.  Perhaps
  > they did this to speed up linking? 
Quite possible.  Dealing with debug symbols is a significant amount
of time for the linker.  These days it's mostly the IO costs I 
suspect (contrast to a.out style stabs which are computationally
expensive as well as IO expensive).  You aren't using a.out are you?!?

  > We've also tried creating shared libraries which I guess basically defers
  > the link until each module is needed at execution time.  This speeds
  > things up rather a lot, but we noticed that gdb doesn't debug shared
  > libraries very well for us.  If you step into a function that's in a
  > shared library, instead of popping in, it gets confused about what line
  > number you're on, or it'll start stepping through what looks like
  > hash-table glue code.  Not being able to debug hurts.
Report them as bugs to the gdb folks :-)   Odds are gdb just needs
some code to analyze the x86 shared library sequences.  It's too bad
there's no standard way for the linker to describe the stub code to
the debugger....  

jeff

Re: Incremental linking

[Ian Lance Taylor]

   Date: Thu, 09 Apr 1998 22:55:28 -0600
   From: Jeffrey A Law <law@cygnus.com>

     > A big problem seems to be that the output file is a whopping 12Mb, and the
     > *.o files take up a healthy 50Mb themselves.  This stuff is partially
     > bloated because of the size of the debugging info added by -g.  We've
     > noticed that MSVC puts the debugging symbols in a seperate file.  Perhaps
     > they did this to speed up linking? 
   Quite possible.  Dealing with debug symbols is a significant amount
   of time for the linker.  These days it's mostly the IO costs I 
   suspect (contrast to a.out style stabs which are computationally
   expensive as well as IO expensive).  You aren't using a.out are you?!?

The GNU linker since version 2.7 compresses ELF stabs debugging
information, which means that it does interpret the stabs strings, and
there are computational costs.  I do not know how significant they
are.

You can see this compression in effect when the 50MB of .o files
become a 12MB output file.  I expect that most of that change is due
to debugging compression.  You can confirm this, if you care, by
running the linker with the --traditional-format option, which turns
off the debugging compression.


To shorten the link time, I agree with Jeff: use shared libraries if
they are a possibility.

Also, the linker is a heavy memory user.  Check the memory size of the
linker process while it is running; if it is being forced out to swap,
then adding RAM may be a cheap way to speed up linking.

I would not expect using -r to shorten your link time significantly.
It will cut down on the number of open and read calls, and there will
be some reduction in the number of global symbols, but the linker
still has to process basically the same amount of data.  You can also
cut down the number of open and read calls to a lesser degree by using
archives, perhaps with the --whole-archive option if appropriate for
your application.


In general, I believe that changing gdb to read debugging information
from the .o files would permit a significant decrease in link time, at
the cost of an increase in debugging time.  It would be good if
somebody looked into adding that feature to gdb.

Ian

Re: Incremental linking

[Andi Kleen]

Ian Lance Taylor <ian@cygnus.com> writes:
> 
> In general, I believe that changing gdb to read debugging information
> from the .o files would permit a significant decrease in link time, at
> the cost of an increase in debugging time.  It would be good if
> somebody looked into adding that feature to gdb.

It would allow another interesting feature. I could release stripped binaries
and still run gdb with symbolic information on resulting core dumps sent in by 
just keeping the object files around.

-Andi

Re: Incremental linking

[Samuel Tardieu]

>>>>> "Andi" == Andi Kleen <ak@muc.de> writes:

Andi> It would allow another interesting feature. I could release
Andi> stripped binaries and still run gdb with symbolic information on
Andi> resulting core dumps sent in by just keeping the object files
Andi> around.

Can't you do this already by keeping the unstripped binary around too?

  Sam
-- 
Samuel Tardieu -- sam@inf.enst.fr

Re: Incremental linking

[Richard Henderson]

On Thu, Apr 09, 1998 at 01:46:28PM -0500, Dave Taylor wrote:
> However, the desired use for this relinkable file is to be able to say
> something like:
> 
> gcc relinkable.o module1.o module2.o ... -o my_proggy
> 
> Where module1.o and module2.o may already be in the relinkable.o file, but
> because they've been specified on the command line, the symbols in those
> module*.o files override the symbols in the relinkable.o file.  Is there a
> way to get ld to behave this way?

No.  The misunderstanding here is that ld is not doing "incremental linking,"
but rather "relocatable linking".  Ie you get a new object file that is in
no way different from any other object file.

> I'm interested in any alternatives you know of to shorten the link time.

Depending on prevailing conditions, `ld --no-keep-memory' may help.

> We're using Linux, and we've already tried using RAM drives and getting
> lots of RAM.  Unfortunately, we don't entirely grok how to allocate a
> certain amt of RAM to the ramdrive.

With Linux after say 1.2, a ramdrive probably won't help, and may even
hurt.  The page cache in 2.0 is really fairly effective.

> We've also tried creating shared libraries which I guess basically defers
> the link until each module is needed at execution time.

Yes.  

The trade-off here is that if you use PIC code, you loose a register,
and if non-PIC code, startup time can be much longer due to the number
of dynamic relocations.  Furthermore, the fact that you're now going
through a PLT for calls between DSOs, some of the instruction prefetch
mechanisms of the ppro core are thwarted. 

All-in-all probably not a great idea to use them carelessly.


r~

Re: Incremental linking

[H.J. Lu]

> However, the desired use for this relinkable file is to be able to say
> something like:
> 
> gcc relinkable.o module1.o module2.o ... -o my_proggy
> 
> Where module1.o and module2.o may already be in the relinkable.o file, but
> because they've been specified on the command line, the symbols in those
> module*.o files override the symbols in the relinkable.o file.  Is there a
> way to get ld to behave this way?  Or was that half of the incremental

I am not sure. I think it is a linker bug.

> 
> We've also tried creating shared libraries which I guess basically defers
> the link until each module is needed at execution time.  This speeds
> things up rather a lot, but we noticed that gdb doesn't debug shared
> libraries very well for us.  If you step into a function that's in a

That is another linker bug.

> shared library, instead of popping in, it gets confused about what line
> number you're on, or it'll start stepping through what looks like
> hash-table glue code.  Not being able to debug hurts.

Please get binutils 2.8.1.0.30 from

ftp://ftp.yggdrasil.com/private/hjl

It should solve your shared library debugging problem. Let me know
if it doesn't work for you.


H.J.

Re: Incremental linking

[Ian Lance Taylor]

   From: hjl@lucon.org (H.J. Lu)
   Date: Fri, 10 Apr 1998 07:49:37 -0700 (PDT)

   > However, the desired use for this relinkable file is to be able to say
   > something like:
   > 
   > gcc relinkable.o module1.o module2.o ... -o my_proggy
   > 
   > Where module1.o and module2.o may already be in the relinkable.o file, but
   > because they've been specified on the command line, the symbols in those
   > module*.o files override the symbols in the relinkable.o file.  Is there a
   > way to get ld to behave this way?  Or was that half of the incremental

   I am not sure. I think it is a linker bug.

As others have also said, this is not a linker bug.  This is expected
behaviour.

Ian

Incremental Linking

[Mike Ludwig]

Does GCC support incremental linking and if so what options do I have to 
use?

I checked the mailing list archives and someone asked a similiar
question and the answer was to use shared libraries.  Shared libraries
are not the answer that I am looking for.  What I want to do is link a 
number of objects to gether to create an executable and then relink  in one 
of the (modified) objects again.
Something like:
gcc -o myexe a.o b.o c.o
Make some changes to b.c and recompile:
gcc -c b.c
I then want to relink b.o directly into myexe. Something like
gcc -o myexe myexe b.o

Can I do this somehow?  When I try I get a bunch of errors about duplicate 
symbols.

Thanks,
Mike.
______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com

Re: Incremental Linking

[Martin v. Loewis]

> Does GCC support incremental linking and if so what options do I
> have to use?

Sorry, no, it doesn't. For one thing, gcc is just a compiler; the
linker is a separate package. Depending on the architecture, you may
be able to use an incremental linker already available. For example
(without trying), it should be possible to configure gcc to work with
ild on Solaris.

Regards,
Martin