Re: Debugging a large program

Re: Debugging a large program

[Jason Molenda]

On Tue, Oct 05, 2004 at 10:02:14AM -0400, Andrew Cagney wrote:

> involves very few symbols and addresses yet GDB is slurping all the 
> following:
> 
> - entire minsymtab - O(<nr minsyms>) at least
> 
> - simplified symtab a.k.a. partial-symtab - O(<nr syms>) at least
> 
> and then when the first symbol request occures:
> 
> - full symtab - O(<nr syms>) at least
> 
> - address information - O(<?>) (or is this done above?)
> 
> GDB needs to find ways to achieve (assuming co-operation from GCC and 
> BFD) an initial:
> 
> - process symtab sections (for address ranges) - O(<nr sections>)
> 
> and then when a symbol is requested:
> 
> - lookup symbol - O(log (<nr symbols>)) * O(<nr solibs>) for first time 
> (better?); O(1) for re-searches
> 
> Doing this means abandoning the psymtab and instead having symbol code 
> directly read each symbol or address as it is needed, and with the 
> minimum auxulary information (i.e., direct from disk).



For what it's worth, we've been working on a less ambitious scheme
at Apple for the past month, with similar benefits.

We've had code in our gdb where the user can set "load rules" which
specify how much gdb will know about the solibs that are loaded,
e.g.  nothing, address ranges only, minsyms, normal debug handling.
Jim has added code so we can treat everything at the minsym level
by default, and when we're looking up one of those symbols or we're
in an address contained by that solib, we read in the psymtabs for
that solib and let gdb do its usual thing.  e.g. when you hit a 
breakpoint or interrupt execution and do a backtrace, the psymtabs
for all the solibs containing functions in the stack are read in;
the relevant psymtabs are upgraded to symtabs as they normally would
as needed.

The amount of changes was not especially large given the "object
file load level" code already being present.

The only drawback to our approach is that static functions aren't
known to gdb when they're in a minsym-level solib -- so a user
putting a breakpoint on a static function will fail unless something
else in that solib has caused it to be psymtab'ed already.  We're
only using this delayed symbol reading optimization when the Xcode
UI is used, though, so we can mitigate that problem through some
clever UI.

Andrew's approach is much cooler, but it's definitely not something
that could be done with stabs (as it exists today), and we're stuck
on stabs on MacOS X for a while still.

Anyway, just a point of information.  It was a HUGE win for us on
extremely large applications where they have many large solibs and
the developer is only debugging code in a couple of them.

J

Re: Debugging a large program

[Daniel Berlin]

>
> Andrew's approach is much cooler,

Which is probably why i suggested exactly that years ago :)

The cooperation you need is a bit more than just debug_pubnames, pubtypes, 
and debug_aranges though (so that info functions and whatnot still worked 
like it does now), but nobody has ever asked any gcc guys to implement it 
for them (heck, i'd be happy to if you guys put together a definitive list 
of things you need, and proposed any new sections to the dwarf3 mailing 
list).

If you don't care about info functions output, tab completion of names, or 
are willing to live with them only showing "public" symbols, i believe you 
could implement this stuff now.

bfd's window stuff is currently undefined unless want_mmap is set to true 
(by the configure option --with-mmap, the default is false, it seems).
You need that to be able to just get pieces of the section.

It looks like it has a non-mmap fallback implemented in bfdwin.c, but 
"bfd_generic_get_section_contents_in_window" aborts if you don't have 
USE_MMAP set.

So uh, just wake me someone gets all the bfd stuff sorted out, and 
you guys figure out what you need more than what you've got to do 
this in the way you want it done, and i'll happily help propose these 
things on the dwarf3 reflector/implement them for gcc.

--Dan

Re: Debugging a large program

[Andrew Cagney]

> 
>>
>> Andrew's approach is much cooler,

> Which is probably why i suggested exactly that years ago :)

I remember the mmap stuff but not much else.
At one stage I ran the basics past Jim but he didn't mention your stuff 
(just described it has hard to impossible).  Do you have a URL.

>> GDB needs to find ways to achieve (assuming co-operation from GCC and 
>> BFD) an initial:
>> 
>> - process symtab sections (for address ranges) - O(<nr sections>)
>> 
>> and then when a symbol is requested:
>> 
>> - lookup symbol - O(log (<nr symbols>)) * O(<nr solibs>) for first time 
>> (better?); O(1) for re-searches

Right now, I think this is the most important aspect.  It establishes an 
absolute bound on the performance.  No matter what is done, it needs to 
be measured against that.

> The cooperation you need is a bit more than just debug_pubnames, pubtypes, and debug_aranges though (so that info functions and whatnot still worked like it does now), but nobody has ever asked any gcc guys to implement it for them (heck, i'd be happy to if you guys put together a definitive list of things you need, and proposed any new sections to the dwarf3 mailing list).
> 
> If you don't care about info functions output, tab completion of names, or are willing to live with them only showing "public" symbols, i believe you could implement this stuff now.
> 
> bfd's window stuff is currently undefined unless want_mmap is set to true (by the configure option --with-mmap, the default is false, it seems).
> You need that to be able to just get pieces of the section.
> 
> It looks like it has a non-mmap fallback implemented in bfdwin.c, but "bfd_generic_get_section_contents_in_window" aborts if you don't have USE_MMAP set.
> 
> So uh, just wake me someone gets all the bfd stuff sorted out, and you guys figure out what you need more than what you've got to do this in the way you want it done, and i'll happily help propose these things on the dwarf3 reflector/implement them for gcc.

Andrew

Re: Debugging a large program

[Andrew Cagney]

>> Doing this means abandoning the psymtab and instead having symbol code 
>> directly read each symbol or address as it is needed, and with the 
>> minimum auxulary information (i.e., direct from disk).

> For what it's worth, we've been working on a less ambitious scheme
> at Apple for the past month, with similar benefits.

(dejavu, sigh)

> Andrew's approach is much cooler, but it's definitely not something
> that could be done with stabs (as it exists today), and we're stuck
> on stabs on MacOS X for a while still.
> 
> Anyway, just a point of information.  It was a HUGE win for us on
> extremely large applications where they have many large solibs and
> the developer is only debugging code in a couple of them.

The existing symfile code can be pushed down a level, hiding it behind a 
new symtab proxy.

Andrew

Re: Debugging a large program

[Nick Savoiu]

----- Original Message ----- 
From: "Daniel Jacobowitz" <drow@false.org>
To: "Nick Savoiu" <savoiu@ics.uci.edu>
Cc: <gdb@sources.redhat.com>
Sent: Monday, October 04, 2004 4:25 PM
Subject: Re: Debugging a large program


> On Mon, Oct 04, 2004 at 04:21:28PM -0700, Nick Savoiu wrote:
> > ----- Original Message ----- 
> > From: "Daniel Jacobowitz" <drow@false.org>
> > To: "Nick Savoiu" <savoiu@ics.uci.edu>; <gdb@sources.redhat.com>
> > Sent: Monday, October 04, 2004 3:14 PM
> > Subject: Re: Debugging a large program
> >
> >
> > > On Mon, Oct 04, 2004 at 05:15:14PM -0400, Daniel Jacobowitz wrote:
> > > > > BTW, 405MB was for gdb running to main() not just what I said
above :)
> > There
> > > > > probably are a few global variables but I don't think they should
take
> > up
> > > > > too much space.
> > > >
> > > > That requires loading symbol information for all the shared
libraries,
> > > > which is probably what's taking all the space.  400MB+ is a bit
> > > > unusual, but I don't know how big your libraries are.
> > >
> > > Your libraries contain 288MB of DWARF2 debug information.  We're
> > > winding up with less than twice that in memory usage for reading in
> > > partial symbols.  It probably could be reduced somewhat - say, 30%.
> > > But some of this data we've just got to hold in memory.
> >
> > I see. If one of the .so files is not actually used, can I somehow
prevent
> > GDB from loading even its partial symbols? How much memory do partial
> > symbols use?
>
> Partial symbols are most or all of that memory.  Try "set
> auto-solib-add 0", followed by "sharedlibrary REGEXP-YOU-WANT-LOADED".

This seems to work reasonably well (a bit tedious since one has to know what
libs to load and when). I wish that GDB could infer that automagically.

Nick

Re: Debugging a large program

[Daniel Jacobowitz]

On Tue, Oct 05, 2004 at 10:02:14AM -0400, Andrew Cagney wrote:
> This, unfortunatly, isn't true.  The "classic" user interaction:
> 	gdb ...
> 	run
> 	<segmentation fault>
> 	backtrace
> 	print variable
> involves very few symbols and addresses yet GDB is slurping all the 
> following:
> 
> - entire minsymtab - O(<nr minsyms>) at least
> 
> - simplified symtab a.k.a. partial-symtab - O(<nr syms>) at least
> 
> and then when the first symbol request occures:
> 
> - full symtab - O(<nr syms>) at least

No, that's not right.  O(<nr syms in the one relevant symtab>).

> - address information - O(<?>) (or is this done above?)

Yes, it is.

-- 
Daniel Jacobowitz

Re: Debugging a large program

[Andrew Cagney]

> On Mon, Oct 04, 2004 at 01:44:55PM -0700, Nick Savoiu wrote:
> 
>>> Hi,
>>> 
>>> I'm using GDB to debug a rather large program and I'm running into memory
>>> usage problems that slow down debugging considerably. Just invoking GDB on
>>> the executable (without issuing 'run') results in GDB using up 450MB of
>>> memory.
>>> 
>>> I think that this is caused by GDB reading in all the symbol info. However,
>>> the code that I'm debugging uses but a small fraction of the code that's
>>> present in the program. Can I somehow tell GDB to only load the symbols it
>>> needs?

Tell me about it :-)

> GDB already reads in only what it needs, and more lazily - however,
> there's some information about every symbol that's needed.  450MB is
> pretty remarkable; how big is the application?  readelf -S output would
> be the best way to answer the question.

This, unfortunatly, isn't true.  The "classic" user interaction:
	gdb ...
	run
	<segmentation fault>
	backtrace
	print variable
involves very few symbols and addresses yet GDB is slurping all the 
following:

- entire minsymtab - O(<nr minsyms>) at least

- simplified symtab a.k.a. partial-symtab - O(<nr syms>) at least

and then when the first symbol request occures:

- full symtab - O(<nr syms>) at least

- address information - O(<?>) (or is this done above?)

GDB needs to find ways to achieve (assuming co-operation from GCC and 
BFD) an initial:

- process symtab sections (for address ranges) - O(<nr sections>)

and then when a symbol is requested:

- lookup symbol - O(log (<nr symbols>)) * O(<nr solibs>) for first time 
(better?); O(1) for re-searches

Doing this means abandoning the psymtab and instead having symbol code 
directly read each symbol or address as it is needed, and with the 
minimum auxulary information (i.e., direct from disk).

The first time I made this observation, the response was "impossible", 
the second time "hard".  I guess I'm making progress :-)

Andrew

Re: Debugging a large program

[Daniel Jacobowitz]

On Mon, Oct 04, 2004 at 04:21:28PM -0700, Nick Savoiu wrote:
> ----- Original Message ----- 
> From: "Daniel Jacobowitz" <drow@false.org>
> To: "Nick Savoiu" <savoiu@ics.uci.edu>; <gdb@sources.redhat.com>
> Sent: Monday, October 04, 2004 3:14 PM
> Subject: Re: Debugging a large program
> 
> 
> > On Mon, Oct 04, 2004 at 05:15:14PM -0400, Daniel Jacobowitz wrote:
> > > > BTW, 405MB was for gdb running to main() not just what I said above :)
> There
> > > > probably are a few global variables but I don't think they should take
> up
> > > > too much space.
> > >
> > > That requires loading symbol information for all the shared libraries,
> > > which is probably what's taking all the space.  400MB+ is a bit
> > > unusual, but I don't know how big your libraries are.
> >
> > Your libraries contain 288MB of DWARF2 debug information.  We're
> > winding up with less than twice that in memory usage for reading in
> > partial symbols.  It probably could be reduced somewhat - say, 30%.
> > But some of this data we've just got to hold in memory.
> 
> I see. If one of the .so files is not actually used, can I somehow prevent
> GDB from loading even its partial symbols? How much memory do partial
> symbols use?

Partial symbols are most or all of that memory.  Try "set
auto-solib-add 0", followed by "sharedlibrary REGEXP-YOU-WANT-LOADED".

-- 
Daniel Jacobowitz

Re: Debugging a large program

[Nick Savoiu]

----- Original Message ----- 
From: "Daniel Jacobowitz" <drow@false.org>
To: "Nick Savoiu" <savoiu@ics.uci.edu>; <gdb@sources.redhat.com>
Sent: Monday, October 04, 2004 3:14 PM
Subject: Re: Debugging a large program


> On Mon, Oct 04, 2004 at 05:15:14PM -0400, Daniel Jacobowitz wrote:
> > > BTW, 405MB was for gdb running to main() not just what I said above :)
There
> > > probably are a few global variables but I don't think they should take
up
> > > too much space.
> >
> > That requires loading symbol information for all the shared libraries,
> > which is probably what's taking all the space.  400MB+ is a bit
> > unusual, but I don't know how big your libraries are.
>
> Your libraries contain 288MB of DWARF2 debug information.  We're
> winding up with less than twice that in memory usage for reading in
> partial symbols.  It probably could be reduced somewhat - say, 30%.
> But some of this data we've just got to hold in memory.

I see. If one of the .so files is not actually used, can I somehow prevent
GDB from loading even its partial symbols? How much memory do partial
symbols use?

Thanks,
Nick

Re: Debugging a large program

[Daniel Jacobowitz]

On Mon, Oct 04, 2004 at 05:15:14PM -0400, Daniel Jacobowitz wrote:
> > BTW, 405MB was for gdb running to main() not just what I said above :) There
> > probably are a few global variables but I don't think they should take up
> > too much space.
> 
> That requires loading symbol information for all the shared libraries,
> which is probably what's taking all the space.  400MB+ is a bit
> unusual, but I don't know how big your libraries are.

Your libraries contain 288MB of DWARF2 debug information.  We're
winding up with less than twice that in memory usage for reading in
partial symbols.  It probably could be reduced somewhat - say, 30%.
But some of this data we've just got to hold in memory.

-- 
Daniel Jacobowitz

Re: Debugging a large program

[Daniel Jacobowitz]

On Mon, Oct 04, 2004 at 02:12:46PM -0700, Nick Savoiu wrote:
> >On Mon, Oct 04, 2004 at 01:44:55PM -0700, Nick Savoiu wrote:
> >> Hi,
> >>
> >> I'm using GDB to debug a rather large program and I'm running into memory
> >> usage problems that slow down debugging considerably. Just invoking GDB
> on
> >> the executable (without issuing 'run') results in GDB using up 450MB of
> >> memory.
> >>
> >> I think that this is caused by GDB reading in all the symbol info.
> However,
> >> the code that I'm debugging uses but a small fraction of the code that's
> >> present in the program. Can I somehow tell GDB to only load the symbols
> it
> >> needs?
> >
> >GDB already reads in only what it needs, and more lazily - however,
> >there's some information about every symbol that's needed.  450MB is
> >pretty remarkable; how big is the application?  readelf -S output would
> >be the best way to answer the question.
> 
> Here's what readelf -S says:
> 
> There are 30 section headers, starting at offset 0x57ec:
> 
>   [12] .text             PROGBITS        0804ad40 002d40 0000f0 00  AX  0
> 0 16
>   [23] .stab             PROGBITS        00000000 0034e4 0007a4 0c     24
> 0  4
>   [24] .stabstr          STRTAB          00000000 003c88 001983 00      0
> 0  1

This is an extremely small application.  I guess this is the KDE-style
everything-is-in-a-shared-library setup?

> BTW, 405MB was for gdb running to main() not just what I said above :) There
> probably are a few global variables but I don't think they should take up
> too much space.

That requires loading symbol information for all the shared libraries,
which is probably what's taking all the space.  400MB+ is a bit
unusual, but I don't know how big your libraries are.

-- 
Daniel Jacobowitz

Re: Debugging a large program

[Nick Savoiu]

>On Mon, Oct 04, 2004 at 01:44:55PM -0700, Nick Savoiu wrote:
>> Hi,
>>
>> I'm using GDB to debug a rather large program and I'm running into memory
>> usage problems that slow down debugging considerably. Just invoking GDB
on
>> the executable (without issuing 'run') results in GDB using up 450MB of
>> memory.
>>
>> I think that this is caused by GDB reading in all the symbol info.
However,
>> the code that I'm debugging uses but a small fraction of the code that's
>> present in the program. Can I somehow tell GDB to only load the symbols
it
>> needs?
>
>GDB already reads in only what it needs, and more lazily - however,
>there's some information about every symbol that's needed.  450MB is
>pretty remarkable; how big is the application?  readelf -S output would
>be the best way to answer the question.

Here's what readelf -S says:

There are 30 section headers, starting at offset 0x57ec:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk
Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0
0  0
  [ 1] .interp           PROGBITS        080480f4 0000f4 000013 00   A  0
0  1
  [ 2] .note.ABI-tag     NOTE            08048108 000108 000020 00   A  0
0  4
  [ 3] .hash             HASH            08048128 000128 000054 04   A  4
0  4
  [ 4] .dynsym           DYNSYM          0804817c 00017c 000100 10   A  5
1  4
  [ 5] .dynstr           STRTAB          0804827c 00027c 002a23 00   A  0
0  1
  [ 6] .gnu.version      VERSYM          0804aca0 002ca0 000020 02   A  4
0  2
  [ 7] .gnu.version_r    VERNEED         0804acc0 002cc0 000020 00   A  5
1  4
  [ 8] .rel.dyn          REL             0804ace0 002ce0 000008 08   A  4
0  4
  [ 9] .rel.plt          REL             0804ace8 002ce8 000010 08   A  4
b  4
  [10] .init             PROGBITS        0804acf8 002cf8 000018 00  AX  0
0  4
  [11] .plt              PROGBITS        0804ad10 002d10 000030 04  AX  0
0  4
  [12] .text             PROGBITS        0804ad40 002d40 0000f0 00  AX  0
0 16
  [13] .fini             PROGBITS        0804ae30 002e30 00001e 00  AX  0
0  4
  [14] .rodata           PROGBITS        0804ae50 002e50 000008 00   A  0
0  4
  [15] .data             PROGBITS        0804b000 003000 00000c 00  WA  0
0  4
  [16] .eh_frame         PROGBITS        0804b00c 00300c 000004 00  WA  0
0  4
  [17] .dynamic          DYNAMIC         0804b010 003010 0004a8 08  WA  5
0  4
  [18] .ctors            PROGBITS        0804b4b8 0034b8 000008 00  WA  0
0  4
  [19] .dtors            PROGBITS        0804b4c0 0034c0 000008 00  WA  0
0  4
  [20] .jcr              PROGBITS        0804b4c8 0034c8 000004 00  WA  0
0  4
  [21] .got              PROGBITS        0804b4cc 0034cc 000018 04  WA  0
0  4
  [22] .bss              NOBITS          0804b4e4 0034e4 000004 00  WA  0
0  4
  [23] .stab             PROGBITS        00000000 0034e4 0007a4 0c     24
0  4
  [24] .stabstr          STRTAB          00000000 003c88 001983 00      0
0  1
  [25] .comment          PROGBITS        00000000 00560b 0000c2 00      0
0  1
  [26] .note             NOTE            00000000 0056cd 00003c 00      0
0  1
  [27] .shstrtab         STRTAB          00000000 005709 0000e3 00      0
0  1
  [28] .symtab           SYMTAB          00000000 005c9c 000470 10     29
34  4
  [29] .strtab           STRTAB          00000000 00610c 0001cf 00      0
0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

BTW, 405MB was for gdb running to main() not just what I said above :) There
probably are a few global variables but I don't think they should take up
too much space.

Nick

Re: Debugging a large program

[Daniel Jacobowitz]

On Mon, Oct 04, 2004 at 01:44:55PM -0700, Nick Savoiu wrote:
> Hi,
> 
> I'm using GDB to debug a rather large program and I'm running into memory
> usage problems that slow down debugging considerably. Just invoking GDB on
> the executable (without issuing 'run') results in GDB using up 450MB of
> memory.
> 
> I think that this is caused by GDB reading in all the symbol info. However,
> the code that I'm debugging uses but a small fraction of the code that's
> present in the program. Can I somehow tell GDB to only load the symbols it
> needs?

GDB already reads in only what it needs, and more lazily - however,
there's some information about every symbol that's needed.  450MB is
pretty remarkable; how big is the application?  readelf -S output would
be the best way to answer the question.

-- 
Daniel Jacobowitz

Debugging a large program

[Nick Savoiu]

Hi,

I'm using GDB to debug a rather large program and I'm running into memory
usage problems that slow down debugging considerably. Just invoking GDB on
the executable (without issuing 'run') results in GDB using up 450MB of
memory.

I think that this is caused by GDB reading in all the symbol info. However,
the code that I'm debugging uses but a small fraction of the code that's
present in the program. Can I somehow tell GDB to only load the symbols it
needs?

Thanks,
Nick