FW: Huge binaries

FW: Huge binaries

[Hite, Christopher]

Hi, I'm upgrading from 4.4.3  => 4.5.2 and the size of my already large
binaries got even bigger.

60MB - stripped old or new
180MB - old release binary with debug info (-g)
1200MB - new compiler with same flags

strings are only a small part of the debug info:
45MB - strings <binary> 

I'm using a bunch of meta-programming.

Did something big happen between these versions?
Are there any options between the stripped binary and the 1.2GB version?
Can I debug a core from a stripped binary using a non-stripped binary?

Chris

Re: FW: Huge binaries

[Ian Lance Taylor]

"Hite, Christopher" <Christopher.Hite@partner.commerzbank.com> writes:

> Hi, I'm upgrading from 4.4.3  => 4.5.2 and the size of my already large
> binaries got even bigger.
>
> 60MB - stripped old or new
> 180MB - old release binary with debug info (-g)
> 1200MB - new compiler with same flags
>
> strings are only a small part of the debug info:
> 45MB - strings <binary> 
>
> I'm using a bunch of meta-programming.
>
> Did something big happen between these versions?
> Are there any options between the stripped binary and the 1.2GB version?
> Can I debug a core from a stripped binary using a non-stripped binary?

There is certainly more debug info generated by gcc 4.5.2.  Are you
saying that your stripped binary went from 60MB to 1200MB without
counting any debug info?  That seems difficult to understand.  If that
is what happened you're going to need to track down the specific part of
your program which increased that much.  Using nm --size-sort may help.

Ian

Re: FW: Huge binaries

[Andrew Haley]

On 02/21/2011 03:59 PM, Hite, Christopher wrote:
> Hi, I'm upgrading from 4.4.3  =>  4.5.2 and the size of my already large
> binaries got even bigger.
>
> 60MB - stripped old or new
> 180MB - old release binary with debug info (-g)
> 1200MB - new compiler with same flags
>
> strings are only a small part of the debug info:
> 45MB - strings<binary>
>
> I'm using a bunch of meta-programming.
>
> Did something big happen between these versions?
> Are there any options between the stripped binary and the 1.2GB version?

Debuginfo got better.

> Can I debug a core from a stripped binary using a non-stripped binary?

Yes.  gdb knows how to read separate debuginfo, and objcopy(1) knows how to
extract it.

  $ gcc -g hello.c
  $ objcopy --only-keep-debug hello hello.debug
  $ strip -g hello
  $ objcopy --add-gnu-debuglink=hello.debug hello
  $ gdb hello
GNU gdb (GDB) Fedora (7.2-41.fc14)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/aph/hello...Reading symbols from /home/aph/hello.debug...done.
done.
(gdb) b main
Breakpoint 1 at 0x400508: file hello.c, line 5.
(gdb) r
Starting program: /home/aph/hello

Breakpoint 1, main () at hello.c:5
5	    fprintf(stderr, "Hello, world!\n");

RE: FW: Huge binaries

[Hite, Christopher]

> There is certainly more debug info generated by gcc 4.5.2.  Are you
saying that your stripped binary went from 60MB to 1200MB without
counting any debug info?  That seems difficult to understand.  If that
is what happened you're 
> going to need to track down the specific part of your program which
increased that much.  Using nm --size-sort may help.

No the stripped binary is with the old and new compiler is 60MB.  That
seems to be the size of my code.  So it's not an issue code bloat issue.
The binary with debug info goes from 180MB to  1200MB when I upgrade the
compiler.

I can see the .o files having to do with my MPL generated en/decoder are
huge.

Does debug information affect performance in any way?

Chris

Re: FW: Huge binaries

[Andrew Haley]

On 02/21/2011 04:37 PM, Hite, Christopher wrote:
>> There is certainly more debug info generated by gcc 4.5.2.  Are you
> saying that your stripped binary went from 60MB to 1200MB without
> counting any debug info?  That seems difficult to understand.  If that
> is what happened you're 
>> going to need to track down the specific part of your program which
> increased that much.  Using nm --size-sort may help.
> 
> No the stripped binary is with the old and new compiler is 60MB.  That
> seems to be the size of my code.  So it's not an issue code bloat issue.
> The binary with debug info goes from 180MB to  1200MB when I upgrade the
> compiler.
> 
> I can see the .o files having to do with my MPL generated en/decoder are
> huge.
> 
> Does debug information affect performance in any way?

Assuming you're using GNU/Linux, no.  The pages that contain the info
don't even get allocated any memory until they're needed.

Andrew.

Re: FW: Huge binaries

[Ian Lance Taylor]

"Hite, Christopher" <Christopher.Hite@partner.commerzbank.com> writes:

> No the stripped binary is with the old and new compiler is 60MB.  That
> seems to be the size of my code.  So it's not an issue code bloat issue.
> The binary with debug info goes from 180MB to  1200MB when I upgrade the
> compiler.

Ah, then it is improved debug info.  At least, I hope it's better.
There are various options you can use to control the size of the debug
info, such as -femit-struct-debug-XXX.

> Does debug information affect performance in any way?

No.

Ian

RE: FW: Huge binaries

[Hite, Christopher]

I've traced down an area where the debug info seems to be growing more
quadratically to what one would expect (see N in code).  It uses some
boost header only libraries, which I'm not sure I can reasonably
recreate.

//------------------------------------------------
// code 
#include <boost/mpl/range_c.hpp>
#include <boost/mpl/transform.hpp>
#include <boost/mpl/copy.hpp>
#include <boost/mpl/list.hpp>
#include <boost/mpl/front_inserter.hpp>
#include <boost/mpl/for_each.hpp>

#include <boost/fusion/include/map.hpp>
#include <boost/fusion/include/at_key.hpp>
#include <boost/fusion/include/as_map.hpp>
#include <boost/fusion/include/mpl.hpp>

const unsigned N=10;
// 50 is realistic for my program
// debug info grows > quadratically ; N=40 creates ~100MB of debug info

typedef boost::mpl::range_c<int,0,N> IntRange;

typedef boost::mpl::copy<
	IntRange,
	boost::mpl::front_inserter< boost::mpl::list0<> >
>::type                               IntList;


struct Int2FusionPair{
	template<typename Int>
	struct apply{
		typedef boost::fusion::pair<Int, double> type;
	};
};

typedef typename boost::mpl::transform<IntList,Int2FusionPair>
	::type FusionPairList;

typedef typename boost::fusion::result_of::as_map<FusionPairList>
	::type FusionMap;



struct Initer{
	template< typename U > void operator()(U x)
	{
		boost::fusion::at_key< U >(fm) = U::value * 0.01;
	}

	Initer(FusionMap& fm_): fm(fm_) {}

	FusionMap& fm;
};

void init(FusionMap& fm){
	boost::mpl::for_each<IntRange>(Initer(fm));
}


int main(){
	FusionMap fm;
	init(fm);
	return 0;
}

//------------------------------------------------
To compile:

g++  -ftemplate-depth-256 -O0 -fno-inline -Wall -g -pthread
-DBOOST_MPL_CFG_NO_PREPROCESSED_HEADERS -DBOOST_MPL_LIMIT_LIST_SIZE=50
-DBOOST_MPL_LIMIT_MAP_SIZE=50 -DFUSION_MAX_MAP_SIZE=50
-DFUSION_MAX_VECTOR_SIZE=50  -I"/fs/tools/L2/boost_1_43_0"
big_db_file.cpp

g++ (GCC) 4.5.2 .   You mentioned things might get better with 4.6.  Was
that only because multiple copies of debug info are removed or will it
also help a single module like this?

Chris


-----Original Message-----
From: Ian Lance Taylor [mailto:iant@google.com] 
Sent: Wednesday, February 23, 2011 1:41 PM
To: Hite, Christopher
Cc: gcc-help@gcc.gnu.org; bill@rkirkpat.net
Subject: Re: FW: Huge binaries

"Hite, Christopher" <Christopher.Hite@partner.commerzbank.com> writes:

> If I lock the process into memory is does it lock the debug info into 
> memory too?

No.

> Perhaps I can run a stripped binary in production and then analize 
> it's cores with the unstripped version.  Will that work?

Yes.

But be aware that the debug info costs you nothing at runtime.  The only
cost is the disk space that it occupies and the bandwidth required to
transmit the extra data when copying the binary between machines.

Ian

Re: FW: Huge binaries

[Ian Lance Taylor]

"Hite, Christopher" <Christopher.Hite@partner.commerzbank.com> writes:

> If I lock the process into memory is does it lock the debug info into
> memory too?

No.

> Perhaps I can run a stripped binary in production and then analize it's
> cores with the unstripped version.  Will that work?

Yes.

But be aware that the debug info costs you nothing at runtime.  The only
cost is the disk space that it occupies and the bandwidth required to
transmit the extra data when copying the binary between machines.

Ian

Re: FW: Huge binaries

[Jonathan Wakely]

On 23 February 2011 10:29, Hite, Christopher wrote:
>
> Perhaps I can run a stripped binary in production and then analize it's
> cores with the unstripped version.  Will that work?

See Andrew's instructions on creating a separate file containing the
debuginfo, which gdb will use when debugging the stripped binary:
http://gcc.gnu.org/ml/gcc-help/2011-02/msg00331.html

RE: FW: Huge binaries

[Hite, Christopher]

> > So it seems to be something in g2, but not in g1.  Is there anything

> > else I can turn off?

> I don't think so.

I'm guessing it's all the classes being define, but never instantiated
like int_<1>, int_<2>, boost::mpl::list<...> etc.  They aren't used
"indirrectly" like (T*(p))->field.

I do have a lot of:
template<...>
struct X{
	static void foo();
};


> For your case (C++ with heavy template usage) gcc 4.6 will do a much
better job of eliminating duplicate debug info when using -gdwarf-4.
> But I don't think there is much else you can do until 4.6 is released.

Cool!  How far away is that?  Looks close.

If I lock the process into memory is does it lock the debug info into
memory too?
mlockall(MCL_CURRENT|MCL_FUTURE)

Perhaps I can run a stripped binary in production and then analize it's
cores with the unstripped version.  Will that work?


Thanks for your help.

Chris

Re: FW: Huge binaries

[Ian Lance Taylor]

"Hite, Christopher" <Christopher.Hite@partner.commerzbank.com> writes:

> So it seems to be something in g2, but not in g1.  Is there anything
> else I can turn off?

I don't think so.

For your case (C++ with heavy template usage) gcc 4.6 will do a much
better job of eliminating duplicate debug info when using -gdwarf-4.
But I don't think there is much else you can do until 4.6 is released.

Ian

RE: FW: Huge binaries

[Hite, Christopher]

> Sorry, Chris, I couldn't debug a stripped binary with gdb because the
computer said there wasn't any debugger information in that file.  
I'm hoping if I have a core from the stripped binary I can debug it with
the non-stripped version.

> Did the compiler tell your computer to optimize the programs that
caused the huge binaries?
Debug binaries are also larger:

1300MB - debug binary with 4.5.2
  59MB - debug build stripped
 850MB - debug build with 4.4.3
 180MB - release with 4.4.3
1200MB - release with 4.5.2
  45MB - strings of release with 4.5.2
  60MB - stripped release with 4.5.2

> By the way, years ago, when I used GNAT 95, Professor Michael Feldman
said that GNAT's linker would include in the binary each program from
every library the Ada 95 program needed.  If you told the machine to
optimize the binary, it only deleted the extra routines from the binary.

I'm not using ADA.  It's C++ with heave use of meta programming.  The
actual code the program needs to run isn't an issue.  It's the debug
information.


I did some playing with the different gcc flags for compiling a single
.o

218M -g2
 94M -g1 
 20M -g0
179M -femit-struct-debug-baseonly
218M -femit-struct-debug-reduced
218M -feliminate-dwarf2-dups
179M -femit-struct-debug-detailed=none
218M -fvar-tracking-assignments-toggle
218M -feliminate-unused-debug-types
173M -gstrict-dwarf
218M -ggdb
218M -feliminate-unused-debug-symbols
294M -gdwarf-version=4  (without -g)
216M -gdwarf-3 (without -g)

So it seems to be something in g2, but not in g1.  Is there anything
else I can turn off?


Chris


-----Original Message-----
From: Bill McEnaney [mailto:bill@rkirkpat.net] 
Sent: Tuesday, February 22, 2011 12:38 AM
To: Hite, Christopher; gcc-help@gcc.gnu.org
Subject: Re: FW: Huge binaries

Sorry, Chris, I couldn't debug a stripped binary with gdb because the
computer said there wasn't any debugger information in that file.  

Did the compiler tell your computer to optimize the programs that caused
the huge binaries?

By the way, years ago, when I used GNAT 95, Professor Michael Feldman
said that GNAT's linker would include in the binary each program from
every library the Ada 95 program needed.  If you told the machine to
optimize the binary, it only deleted the extra routines from the binary.

Cheers,
Bill
Chris Hite writes:

> Hi, I'm upgrading from 4.4.3  => 4.5.2 and the size of my already 
> large binaries got even bigger.
> 
> 60MB - stripped old or new
> 180MB - old release binary with debug info (-g) 1200MB - new compiler 
> with same flags
> 
> strings are only a small part of the debug info:
> 45MB - strings <binary>
> 
> I'm using a bunch of meta-programming.
> 
> Did something big happen between these versions?
> Are there any options between the stripped binary and the 1.2GB
version?
> Can I debug a core from a stripped binary using a non-stripped binary?
> 
> Chris
> 
> 

________________________________________________________________
Please visit a saintly hero:
http://www.jakemoore.org

Re: FW: Huge binaries

[Bill McEnaney]

Sorry, Chris, I couldn't debug a stripped binary with gdb because the computer said there 
wasn't any debugger information in that file.  

Did the compiler tell your computer to optimize the programs that caused the huge binaries?

By the way, years ago, when I used GNAT 95, Professor Michael Feldman said that GNAT's 
linker would include in the binary each program from every library the Ada 95 program 
needed.  If you told the machine to optimize the binary, it only deleted the extra routines 
from the binary.

Cheers,
Bill
Chris Hite writes:

> Hi, I'm upgrading from 4.4.3  => 4.5.2 and the size of my already large
> binaries got even bigger.
> 
> 60MB - stripped old or new
> 180MB - old release binary with debug info (-g)
> 1200MB - new compiler with same flags
> 
> strings are only a small part of the debug info:
> 45MB - strings <binary> 
> 
> I'm using a bunch of meta-programming.
> 
> Did something big happen between these versions?
> Are there any options between the stripped binary and the 1.2GB version?
> Can I debug a core from a stripped binary using a non-stripped binary?
> 
> Chris
> 
> 

________________________________________________________________
Please visit a saintly hero:
http://www.jakemoore.org