Re: memory increased rapidly when adding a break

[DeJiang Zhu <doujiang24@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 15441 bytes --]

On Mon, Nov 14, 2022 at 8:26 AM Simon Marchi <simark@simark.ca> wrote:

>
>
> On 11/13/22 05:01, DeJiang Zhu via Gdb wrote:
> > Hi,
> >
> > I compiled envoy(a big c++ project) by using gcc-12.2.0, debug it by
> using
> > gdb 12.1.
> >
> > But, memory increased rapidly(over 40+GB, until OOM), when adding a
> break.
> >
> > I got this backtrace, after attach the gdb process, when memory
> increasing.
> >
> > ```
> > (gdb) bt
> > #0  0x00007fa6893dc935 in _int_malloc () from /lib64/libc.so.6
> > #1  0x00007fa6893df6fc in malloc () from /lib64/libc.so.6
> > #2  0x0000000000468278 in xmalloc (size=4064) at alloc.c:60
> > #3  0x00000000008ecd95 in call_chunkfun (size=<optimized out>,
> > h=0x17a246ed0) at ./obstack.c:94
> > #4  _obstack_begin_worker (h=0x17a246ed0, size=<optimized out>,
> > alignment=<optimized out>) at ./obstack.c:141
> > #5  0x000000000052d0d3 in demangle_parse_info::demangle_parse_info
> > (this=0x17a246ec0) at cp-name-parser.y:1973
> > #6  cp_demangled_name_to_comp (demangled_name=demangled_name@entry
> =0x8d12c8c0
> > "std::stack<unsigned int, std::deque<unsigned int,
> std::allocator<unsigned
> > int> > >::size_type", errmsg=errmsg@entry=0x0) at cp-name-parser.y:2040
> > #7  0x000000000052ff5e in cp_canonicalize_string
> > (string=string@entry=0x8d12c8c0
> > "std::stack<unsigned int, std::deque<unsigned int,
> std::allocator<unsigned
> > int> > >::size_type") at cp-support.c:635
> > #8  0x0000000000570b98 in dwarf2_canonicalize_name (name=0x8d12c8c0
> > "std::stack<unsigned int, std::deque<unsigned int,
> std::allocator<unsigned
> > int> > >::size_type", cu=<optimized out>, objfile=0x2c3af10) at
> > dwarf2/read.c:22908
> > #9  0x0000000000590265 in dwarf2_compute_name (name=0x7fa55773c524
> > "size_type", die=0x172590eb0, cu=0xe2aeefd0, physname=0) at
> > dwarf2/read.c:10095
> > #10 0x000000000058bf39 in dwarf2_full_name (cu=0xe2aeefd0,
> die=0x172590eb0,
> > name=0x0) at dwarf2/read.c:10123
> > #11 read_typedef (cu=0xe2aeefd0, die=0x172590eb0) at dwarf2/read.c:17687
> > #12 read_type_die_1 (cu=0xe2aeefd0, die=0x172590eb0) at
> dwarf2/read.c:22531
> > #13 read_type_die (die=0x172590eb0, cu=0xe2aeefd0) at dwarf2/read.c:22473
> > #14 0x000000000059acda in dwarf2_add_type_defn (cu=0xe2aeefd0,
> > die=0x172590eb0, fip=0x7ffd8a1be3e0) at dwarf2/read.c:14740
> > #15 handle_struct_member_die (child_die=0x172590eb0, type=0x17a6becd0,
> > fi=0x7ffd8a1be3e0, template_args=<optimized out>, cu=0xe2aeefd0) at
> > dwarf2/read.c:15867
> > #16 0x0000000000597044 in process_structure_scope (cu=0xe2aeefd0,
> > die=0x172590920) at dwarf2/read.c:15908
> > #17 process_die (die=0x172590920, cu=0xe2aeefd0) at dwarf2/read.c:9698
> > #18 0x000000000059646d in read_namespace (cu=0xe2aeefd0, die=0x16802e140)
> > at dwarf2/read.c:17068
> > #19 process_die (die=0x16802e140, cu=0xe2aeefd0) at dwarf2/read.c:9737
> > #20 0x0000000000598df9 in read_file_scope (die=0x1594e8360,
> cu=0xe2aeefd0)
> > at dwarf2/read.c:10648
> > #21 0x0000000000595f32 in process_die (die=0x1594e8360, cu=0xe2aeefd0) at
> > dwarf2/read.c:9669
> > #22 0x000000000059c0c8 in process_full_comp_unit
> > (pretend_language=<optimized out>, cu=0xe2aeefd0) at dwarf2/read.c:9439
> > #23 process_queue (per_objfile=0x9d546c0) at dwarf2/read.c:8652
> > #24 dw2_do_instantiate_symtab (per_cu=<optimized out>,
> > per_objfile=0x9d546c0, skip_partial=<optimized out>) at
> dwarf2/read.c:2311
> > #25 0x000000000059c5f0 in dw2_instantiate_symtab (per_cu=0x9c886f0,
> > per_objfile=0x9d546c0, skip_partial=<optimized out>) at
> > dwarf2/read.c:2335#26 0x000000000059c78a in
> > dw2_expand_symtabs_matching_one(dwarf2_per_cu_data *, dwarf2_per_objfile
> *,
> > gdb::function_view<bool(char const*, bool)>,
> > gdb::function_view<bool(compunit_symtab*)>) (per_cu=<optimized out>,
> > per_objfile=<optimized out>, file_matcher=..., expansion_notify=...) at
> > dwarf2/read.c:4204
> > #27 0x000000000059c94b in
> > dwarf2_gdb_index::expand_symtabs_matching(objfile*,
> gdb::function_view<bool
> > (char const*, bool)>, lookup_name_info const*, gdb::function_view<bool
> > (char const*)>, gdb::function_view<bool (compunit_symtab*)>,
> > enum_flags<block_search_flag_values>, domain_enum_tag, search_domain)
> > (this=<optimized out>, objfile=<optimized out>, file_matcher=...,
> > lookup_name=<optimized out>, symbol_matcher=
> > ..., expansion_notify=..., search_flags=..., domain=UNDEF_DOMAIN,
> > kind=<optimized out>) at dwarf2/read.c:4421
> > #28 0x0000000000730feb in objfile::map_symtabs_matching_filename(char
> > const*, char const*, gdb::function_view<bool (symtab*)>) (this=0x2c3af10,
> > name=<optimized out>, name@entry=0x586f26f0 "utility.h",
> > real_path=<optimized out>, real_path@entry=0x0, callback=...) at
> > symfile-debug.c:207
> > #29 0x0000000000741abd in iterate_over_symtabs(char const*,
> > gdb::function_view<bool (symtab*)>) (name=name@entry=0x586f26f0
> > "utility.h", callback=...) at symtab.c:624
> > #30 0x00000000006311d7 in collect_symtabs_from_filename (file=0x586f26f0
> > "utility.h", search_pspace=<optimized out>) at linespec.c:3716
> > #31 0x0000000000631212 in symtabs_from_filename (filename=0x586f26f0
> > "utility.h", search_pspace=<optimized out>) at linespec.c:3736
> > #32 0x0000000000635e9f in parse_linespec (parser=0x7ffd8a1bf1b0,
> > arg=<optimized out>, match_type=<optimized out>) at linespec.c:2557
> > #33 0x0000000000636cac in event_location_to_sals (parser=0x7ffd8a1bf1b0,
> > location=0x51ed4da0) at linespec.c:3082
> > #34 0x0000000000636f73 in decode_line_full (location=location@entry
> =0x51ed4da0,
> > flags=flags@entry=1, search_pspace=search_pspace@entry=0x0,
> > default_symtab=<optimized out>, default_line=<optimized out>,
> > canonical=0x7ffd8a1bf4e0, select_mode=0x0, filter=<optimized out>) at
> > linespec.c:3161
> > #35 0x00000000004b1683 in parse_breakpoint_sals (location=0x51ed4da0,
> > canonical=0x7ffd8a1bf4e0) at breakpoint.c:8730
> > #36 0x00000000004b5d03 in create_breakpoint (gdbarch=0xeca5dc0,
> > location=location@entry=0x51ed4da0, cond_string=cond_string@entry=0x0,
> > thread=<optimized out>, thread@entry=-1, extra_string=0x0,
> > extra_string@entry=0x7ffd8a1bf650 "",
> > force_condition=force_condition@entry=false,
> > parse_extra=0, tempflag=0, type_wanted=bp_breakpoint, ignore_count=0,
> > pending_break_support=AUTO_BOOLEAN_TRUE, ops=0xc23c00
> > <bkpt_breakpoint_ops>, from_tty=0, enabled=1, internal=0, flags=0) at
> > breakpoint.c:9009
> > #37 0x0000000000674ba8 in mi_cmd_break_insert_1 (dprintf=0,
> argv=<optimized
> > out>, argc=<optimized out>, command=<optimized out>) at
> > mi/mi-cmd-break.c:361
> > ```
> >
> > Also, I found it's loop in `dwarf2_gdb_index::expand_symtabs_matching`.
> > I added a break on `dw2_expand_symtabs_matching_one`, it hit this break
> > repeatly.
> >
> > ```
> >   if (lookup_name == nullptr)
> >     {
> >       for (dwarf2_per_cu_data *per_cu
> >         : all_comp_units_range (per_objfile->per_bfd))
> >       {
> >          QUIT;
> >          if (!dw2_expand_symtabs_matching_one (per_cu, per_objfile,
> > file_matcher, expansion_notify))
> >            return false;
> >       }
> >       return true;
> >     }
> > ```
> >
> > Seems, `per_bfd->all_comp_units.size()` is `28776`.
> > I'm not sure if this is a reasonable value.
>
> I think that's possible, if it's a big project.  For instance, my gdb
> binary has about 660 compile units, and gdb is not really big.
>
> >
> > ```
> > (gdb) p per_objfile->per_bfd->all_comp_units
> > $423 = {<std::_Vector_base<std::unique_ptr<dwarf2_per_cu_data,
> > dwarf2_per_cu_data_deleter>,
> > std::allocator<std::unique_ptr<dwarf2_per_cu_data,
> > dwarf2_per_cu_data_deleter> > >> = {_M_impl =
> > {<std::allocator<std::unique_ptr<dwarf2_per_cu_data,
> > dwarf2_per_cu_data_deleter> >> =
> > {<__gnu_cxx::new_allocator<std::unique_ptr<dwarf2_per_cu_data,
> > dwarf2_per_cu_data_deleter> >> = {<No data fields>}, <No data fields>},
> > <std::_Vector_base<std::unique_ptr<dwarf2_per_cu_data,
> > dwarf2_per_cu_data_deleter>,
> > std::allocator<std::unique_ptr<dwarf2_per_cu_data,
> > dwarf2_per_cu_data_deleter> > >::_Vector_impl_data> = {_M_start =
> > 0x3b6f980, _M_finish = 0x3b769e8, _M_end_of_storage = 0x3b769e8}, <No
> data
> > fields>}}, <No data fields>}
> > (gdb) p 0x3b769e8-0x3b6f980
> > $424 = 28776
> > ```
> >
> > I can see the memory increasing rapidly in the for loop.
> > I'm new to the gdb internal implementation.
> > I'm not sure where could be the problem, gcc or gdb, or just a wrong use.
> >
> > Could you help to point the direction? I have the files to reproduce it
> > stablely.
>
> GDB works in two steps to read compile units.  From you stack trace, it
> looks like you are using an index (the .gdb_index kind).  When GDB first
> loads you binary, it reads in an index present in the binary (or in the
> index cache) that lists all the entity names present in each compile
> unit of the program.  When you set a breakpoint using a name, GDB
> "expands" all the compile units with something in it that matches what
> you asked for.  "Expand" means that GDB reads the full debug information
> from the DWARF for that compile unit, creating some internal data
> structures to represent it.
>
> It sounds like the breakpoint spec string you passed matches a lot of
> compile units, and a lot of them get expanded.  That creates a lot of
> in-memory objects, eventually reaching some limit.
>
> Out of curiosity, what is the string you used to create your breakpoint?
> From you stack trace, it sounds like it's "utility.h:LINE".
>

Thanks for your detailed explanation.
Yes, it's `utility.h:560`, I added this break from vscode.

> creating some internal data structures to represent it

I wonder where would allocate so much memory, (at least 40GB),
while the binary is 985MB with the whole debug info.
Maybe many match results need too many memories?

```
$ ls -lh envoy-deadloop
-r-xr-xr-x 1 zhudejiang.pt users 985M Nov 13 15:34 envoy-deadloop

$ readelf -SW envoy-deadloop
There are 48 section headers, starting at offset 0x3d856690:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES
Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00
     0   0  0
  [ 1] .interp           PROGBITS        00000000000002e0 0002e0 00001c 00
  A  0   0  1
  [ 2] .note.ABI-tag     NOTE            00000000000002fc 0002fc 000020 00
  A  0   0  4
  [ 3] .dynsym           DYNSYM          0000000000000320 000320 003810 18
  A  7   1  8
  [ 4] .gnu.version      VERSYM          0000000000003b30 003b30 0004ac 02
  A  3   0  2
  [ 5] .gnu.version_r    VERNEED         0000000000003fdc 003fdc 0001d0 00
  A  7   6  4
  [ 6] .gnu.hash         GNU_HASH        00000000000041b0 0041b0 0005ac 00
  A  3   0  8
  [ 7] .dynstr           STRTAB          000000000000475c 00475c 001cce 00
  A  0   0  1
  [ 8] .rela.dyn         RELA            0000000000006430 006430 4c20d8 18
  A  3   0  8
  [ 9] .rela.plt         RELA            00000000004c8508 4c8508 002550 18
  A  3  29  8
  [10] .rodata           PROGBITS        00000000004cb000 4cb000 c1b474 00
AMS  0   0 4096
  [11] .gcc_except_table PROGBITS        00000000010e6474 10e6474 5f231c 00
  A  0   0  4
  [12] protodesc_cold    PROGBITS        00000000016d87a0 16d87a0 079d68 00
  A  0   0 32
  [13] flags_help_cold   PROGBITS        0000000001752520 1752520 001202 00
  A  0   0 32
  [14] .eh_frame_hdr     PROGBITS        0000000001753724 1753724 4a1e4c 00
  A  0   0  4
  [15] .eh_frame         PROGBITS        0000000001bf5570 1bf5570 13a6b0c
00   A  0   0  8
  [16] .text             PROGBITS        0000000002f9d080 2f9c080 3bbc38f
00  AX  0   0 64
  [17] .init             PROGBITS        0000000006b59410 6b58410 00001a 00
 AX  0   0  4
  [18] .fini             PROGBITS        0000000006b5942c 6b5842c 000009 00
 AX  0   0  4
  [19] malloc_hook       PROGBITS        0000000006b59436 6b58436 000786 00
 AX  0   0  2
  [20] google_malloc     PROGBITS        0000000006b59bc0 6b58bc0 005bd0 00
 AX  0   0 64
  [21] .plt              PROGBITS        0000000006b5f790 6b5e790 0018f0 00
 AX  0   0 16
  [22] .tdata            PROGBITS        0000000006b62080 6b60080 000088 00
WAT  0   0 64
  [23] .tbss             NOBITS          0000000006b62140 6b60108 002500 00
WAT  0   0 64
  [24] .fini_array       FINI_ARRAY      0000000006b62108 6b60108 000008 08
 WA  0   0  8
  [25] .init_array       INIT_ARRAY      0000000006b62110 6b60110 0064b0 08
 WA  0   0  8
  [26] .data.rel.ro      PROGBITS        0000000006b685c0 6b665c0 1d7448 00
 WA  0   0 32
  [27] .dynamic          DYNAMIC         0000000006d3fa08 6d3da08 000200 10
 WA  7   0  8
  [28] .got              PROGBITS        0000000006d3fc08 6d3dc08 000760 00
 WA  0   0  8
  [29] .got.plt          PROGBITS        0000000006d40368 6d3e368 000c88 00
 WA  0   0  8
  [30] .data             PROGBITS        0000000006d42000 6d3f000 060c18 00
 WA  0   0 64
  [31] .tm_clone_table   PROGBITS        0000000006da2c18 6d9fc18 000000 00
 WA  0   0  8
  [32] .bss              NOBITS          0000000006da2c40 6d9fc18 2119e0 00
 WA  0   0 64
  [33] .comment          PROGBITS        0000000000000000 6d9fc18 000060 01
 MS  0   0  1
  [34] .debug_info       PROGBITS        0000000000000000 6d9fc78 8a32aa 00
     0   0  1
  [35] .debug_abbrev     PROGBITS        0000000000000000 7642f22 093663 00
     0   0  1
  [36] .debug_aranges    PROGBITS        0000000000000000 76d6590 62d2cd0
00      0   0 16
  [37] .debug_line       PROGBITS        0000000000000000 d9a9260 128ef333
00      0   0  1
  [38] .debug_str        PROGBITS        0000000000000000 20298593 1a392b
01  MS  0   0  1
  [39] .debug_addr       PROGBITS        0000000000000000 2043bebe 44b00a0
00      0   0  1
  [40] .debug_rnglists   PROGBITS        0000000000000000 248ebf5e 196f4db
00      0   0  1
  [41] .debug_loclists   PROGBITS        0000000000000000 2625b439 22e7fb
00      0   0  1
  [42] .debug_frame      PROGBITS        0000000000000000 26489c38 000060
00      0   0  8
  [43] .debug_line_str   PROGBITS        0000000000000000 26489c98 000569
01  MS  0   0  1
  [44] .gdb_index        PROGBITS        0000000000000000 2648a201 eef593d
00      0   0  1
  [45] .symtab           SYMTAB          0000000000000000 3537fb40 194d828
18     47 418626  8
  [46] .shstrtab         STRTAB          0000000000000000 36ccd368 0001fe
00      0   0  1
  [47] .strtab           STRTAB          0000000000000000 36ccd566 6b89128
00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)
```


> Expanding that many CUs could be legitimate, if there's really something
> matching in all these CUs, or it could be a bug where GDB expands
> unrelated CUs.  There is an open bug related to a problem like this:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=29105


This looks like a CPU performance issue.
May not be my issue, in my opinion.


>
> Although I'm not sure this is what you see.
>
> Is the project you build something open source that other people could
> build and try?
>

Yes, it's an open source project.
It builds with Bazel and depends on Go, which may be a bit complicated.
This is the doc for build & run.
https://github.com/mosn/envoy-go-extension#envoy


>
> Simon
>