From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x1130.google.com (mail-yw1-x1130.google.com [IPv6:2607:f8b0:4864:20::1130]) by sourceware.org (Postfix) with ESMTPS id A708E38518BE for ; Mon, 14 Nov 2022 03:58:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A708E38518BE Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-yw1-x1130.google.com with SMTP id 00721157ae682-376596ae449so95250407b3.5 for ; Sun, 13 Nov 2022 19:58:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=nn6LmlAJFZNVtbXKZlXczbfjkf3s80+JCjxnzBN2UZM=; b=ik4wjxgjtOSp5fcwzP0zCeDYMSJRhFuwMAQsCkJj6ppVvFydxmy95cGlGXLXEYEJNS xVqWsEhNRv0tUFsZne2OfPv0+vmUy9NYV2y0xrQ73eSs0cyocvOrSrHJp7OP98DQm6Hj VI5vesxVRGuKfV0FMtJmTsE8tM2o3P18KHdpCeZUzW735KyXeMPx+W0VsKDtGGal0xWE zHMCnvQ8Cbjs3/3+d6CSqBT/HNAoubmivZq7nNRNUccyQ2pfuuJO8gfDdCZNRWwMj1up /LJLukC9nq1YTHshVRSZ95MPPWPdqTAOAm28UHk6ZAqmgf07S8Ef6LFGjOwEQFB6M6zU 12JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nn6LmlAJFZNVtbXKZlXczbfjkf3s80+JCjxnzBN2UZM=; b=UFdyD3VhCdJvR+jRnbNnXaQafhDLtgZSh/3CzQHVKUgC0bJDPEuZsMyKV1M2U4MN8Z YAybPqYYbjOChB0V/Hj+saMR4ntVnGVN443+YN0zn6vGaOvUTE/ASkPdZSQPgc22T3gu y+ISInogZU4if2z2PRbC5kcM+yiT/CiqVGBGlQVIIx44WWhUS3BeLma0UMf/mAQIO5Wl ki4YxEtYqsfV8j7MtEloo2HBsjIZRiZHZYD8DAiIPWS/R+M3EprUDOgA5OEsFl3RAle8 ab1MIDi4e3NB7ef62ckZ7/esqa8VfA1baOOqc3la5COrEq5yQ7gbD0ZmJDmBprPcUWF3 j7VA== X-Gm-Message-State: ANoB5pnmAyVnMi7CHkViFiiAq4uc/nxNwvBG0cDJbLDsY6WGBDKU52io 0xwSoW2YJ/tAqY0rB3/qNGlppjDVCmLgcnr879ELXifU X-Google-Smtp-Source: AA0mqf6AtKnR6vFuCR95S8ITcIsjk5akosLKsT0Uf637KocDZiX94Hn998mY+zdHFJU1N5gJ4pBazYralYcC6BAehjk= X-Received: by 2002:a81:1487:0:b0:370:59d2:31f4 with SMTP id 129-20020a811487000000b0037059d231f4mr11410494ywu.16.1668398322906; Sun, 13 Nov 2022 19:58:42 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: DeJiang Zhu Date: Mon, 14 Nov 2022 11:58:31 +0800 Message-ID: Subject: Re: memory increased rapidly when adding a break To: Simon Marchi Cc: gdb@sourceware.org Content-Type: multipart/alternative; boundary="000000000000617da305ed664183" X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --000000000000617da305ed664183 Content-Type: text/plain; charset="UTF-8" On Mon, Nov 14, 2022 at 8:26 AM Simon Marchi wrote: > > > On 11/13/22 05:01, DeJiang Zhu via Gdb wrote: > > Hi, > > > > I compiled envoy(a big c++ project) by using gcc-12.2.0, debug it by > using > > gdb 12.1. > > > > But, memory increased rapidly(over 40+GB, until OOM), when adding a > break. > > > > I got this backtrace, after attach the gdb process, when memory > increasing. > > > > ``` > > (gdb) bt > > #0 0x00007fa6893dc935 in _int_malloc () from /lib64/libc.so.6 > > #1 0x00007fa6893df6fc in malloc () from /lib64/libc.so.6 > > #2 0x0000000000468278 in xmalloc (size=4064) at alloc.c:60 > > #3 0x00000000008ecd95 in call_chunkfun (size=, > > h=0x17a246ed0) at ./obstack.c:94 > > #4 _obstack_begin_worker (h=0x17a246ed0, size=, > > alignment=) at ./obstack.c:141 > > #5 0x000000000052d0d3 in demangle_parse_info::demangle_parse_info > > (this=0x17a246ec0) at cp-name-parser.y:1973 > > #6 cp_demangled_name_to_comp (demangled_name=demangled_name@entry > =0x8d12c8c0 > > "std::stack std::allocator > int> > >::size_type", errmsg=errmsg@entry=0x0) at cp-name-parser.y:2040 > > #7 0x000000000052ff5e in cp_canonicalize_string > > (string=string@entry=0x8d12c8c0 > > "std::stack std::allocator > int> > >::size_type") at cp-support.c:635 > > #8 0x0000000000570b98 in dwarf2_canonicalize_name (name=0x8d12c8c0 > > "std::stack std::allocator > int> > >::size_type", cu=, objfile=0x2c3af10) at > > dwarf2/read.c:22908 > > #9 0x0000000000590265 in dwarf2_compute_name (name=0x7fa55773c524 > > "size_type", die=0x172590eb0, cu=0xe2aeefd0, physname=0) at > > dwarf2/read.c:10095 > > #10 0x000000000058bf39 in dwarf2_full_name (cu=0xe2aeefd0, > die=0x172590eb0, > > name=0x0) at dwarf2/read.c:10123 > > #11 read_typedef (cu=0xe2aeefd0, die=0x172590eb0) at dwarf2/read.c:17687 > > #12 read_type_die_1 (cu=0xe2aeefd0, die=0x172590eb0) at > dwarf2/read.c:22531 > > #13 read_type_die (die=0x172590eb0, cu=0xe2aeefd0) at dwarf2/read.c:22473 > > #14 0x000000000059acda in dwarf2_add_type_defn (cu=0xe2aeefd0, > > die=0x172590eb0, fip=0x7ffd8a1be3e0) at dwarf2/read.c:14740 > > #15 handle_struct_member_die (child_die=0x172590eb0, type=0x17a6becd0, > > fi=0x7ffd8a1be3e0, template_args=, cu=0xe2aeefd0) at > > dwarf2/read.c:15867 > > #16 0x0000000000597044 in process_structure_scope (cu=0xe2aeefd0, > > die=0x172590920) at dwarf2/read.c:15908 > > #17 process_die (die=0x172590920, cu=0xe2aeefd0) at dwarf2/read.c:9698 > > #18 0x000000000059646d in read_namespace (cu=0xe2aeefd0, die=0x16802e140) > > at dwarf2/read.c:17068 > > #19 process_die (die=0x16802e140, cu=0xe2aeefd0) at dwarf2/read.c:9737 > > #20 0x0000000000598df9 in read_file_scope (die=0x1594e8360, > cu=0xe2aeefd0) > > at dwarf2/read.c:10648 > > #21 0x0000000000595f32 in process_die (die=0x1594e8360, cu=0xe2aeefd0) at > > dwarf2/read.c:9669 > > #22 0x000000000059c0c8 in process_full_comp_unit > > (pretend_language=, cu=0xe2aeefd0) at dwarf2/read.c:9439 > > #23 process_queue (per_objfile=0x9d546c0) at dwarf2/read.c:8652 > > #24 dw2_do_instantiate_symtab (per_cu=, > > per_objfile=0x9d546c0, skip_partial=) at > dwarf2/read.c:2311 > > #25 0x000000000059c5f0 in dw2_instantiate_symtab (per_cu=0x9c886f0, > > per_objfile=0x9d546c0, skip_partial=) at > > dwarf2/read.c:2335#26 0x000000000059c78a in > > dw2_expand_symtabs_matching_one(dwarf2_per_cu_data *, dwarf2_per_objfile > *, > > gdb::function_view, > > gdb::function_view) (per_cu=, > > per_objfile=, file_matcher=..., expansion_notify=...) at > > dwarf2/read.c:4204 > > #27 0x000000000059c94b in > > dwarf2_gdb_index::expand_symtabs_matching(objfile*, > gdb::function_view > (char const*, bool)>, lookup_name_info const*, gdb::function_view > (char const*)>, gdb::function_view, > > enum_flags, domain_enum_tag, search_domain) > > (this=, objfile=, file_matcher=..., > > lookup_name=, symbol_matcher= > > ..., expansion_notify=..., search_flags=..., domain=UNDEF_DOMAIN, > > kind=) at dwarf2/read.c:4421 > > #28 0x0000000000730feb in objfile::map_symtabs_matching_filename(char > > const*, char const*, gdb::function_view) (this=0x2c3af10, > > name=, name@entry=0x586f26f0 "utility.h", > > real_path=, real_path@entry=0x0, callback=...) at > > symfile-debug.c:207 > > #29 0x0000000000741abd in iterate_over_symtabs(char const*, > > gdb::function_view) (name=name@entry=0x586f26f0 > > "utility.h", callback=...) at symtab.c:624 > > #30 0x00000000006311d7 in collect_symtabs_from_filename (file=0x586f26f0 > > "utility.h", search_pspace=) at linespec.c:3716 > > #31 0x0000000000631212 in symtabs_from_filename (filename=0x586f26f0 > > "utility.h", search_pspace=) at linespec.c:3736 > > #32 0x0000000000635e9f in parse_linespec (parser=0x7ffd8a1bf1b0, > > arg=, match_type=) at linespec.c:2557 > > #33 0x0000000000636cac in event_location_to_sals (parser=0x7ffd8a1bf1b0, > > location=0x51ed4da0) at linespec.c:3082 > > #34 0x0000000000636f73 in decode_line_full (location=location@entry > =0x51ed4da0, > > flags=flags@entry=1, search_pspace=search_pspace@entry=0x0, > > default_symtab=, default_line=, > > canonical=0x7ffd8a1bf4e0, select_mode=0x0, filter=) at > > linespec.c:3161 > > #35 0x00000000004b1683 in parse_breakpoint_sals (location=0x51ed4da0, > > canonical=0x7ffd8a1bf4e0) at breakpoint.c:8730 > > #36 0x00000000004b5d03 in create_breakpoint (gdbarch=0xeca5dc0, > > location=location@entry=0x51ed4da0, cond_string=cond_string@entry=0x0, > > thread=, thread@entry=-1, extra_string=0x0, > > extra_string@entry=0x7ffd8a1bf650 "", > > force_condition=force_condition@entry=false, > > parse_extra=0, tempflag=0, type_wanted=bp_breakpoint, ignore_count=0, > > pending_break_support=AUTO_BOOLEAN_TRUE, ops=0xc23c00 > > , from_tty=0, enabled=1, internal=0, flags=0) at > > breakpoint.c:9009 > > #37 0x0000000000674ba8 in mi_cmd_break_insert_1 (dprintf=0, > argv= > out>, argc=, command=) at > > mi/mi-cmd-break.c:361 > > ``` > > > > Also, I found it's loop in `dwarf2_gdb_index::expand_symtabs_matching`. > > I added a break on `dw2_expand_symtabs_matching_one`, it hit this break > > repeatly. > > > > ``` > > if (lookup_name == nullptr) > > { > > for (dwarf2_per_cu_data *per_cu > > : all_comp_units_range (per_objfile->per_bfd)) > > { > > QUIT; > > if (!dw2_expand_symtabs_matching_one (per_cu, per_objfile, > > file_matcher, expansion_notify)) > > return false; > > } > > return true; > > } > > ``` > > > > Seems, `per_bfd->all_comp_units.size()` is `28776`. > > I'm not sure if this is a reasonable value. > > I think that's possible, if it's a big project. For instance, my gdb > binary has about 660 compile units, and gdb is not really big. > > > > > ``` > > (gdb) p per_objfile->per_bfd->all_comp_units > > $423 = { > dwarf2_per_cu_data_deleter>, > > std::allocator > dwarf2_per_cu_data_deleter> > >> = {_M_impl = > > { > dwarf2_per_cu_data_deleter> >> = > > {<__gnu_cxx::new_allocator > dwarf2_per_cu_data_deleter> >> = {}, }, > > > dwarf2_per_cu_data_deleter>, > > std::allocator > dwarf2_per_cu_data_deleter> > >::_Vector_impl_data> = {_M_start = > > 0x3b6f980, _M_finish = 0x3b769e8, _M_end_of_storage = 0x3b769e8}, data > > fields>}}, } > > (gdb) p 0x3b769e8-0x3b6f980 > > $424 = 28776 > > ``` > > > > I can see the memory increasing rapidly in the for loop. > > I'm new to the gdb internal implementation. > > I'm not sure where could be the problem, gcc or gdb, or just a wrong use. > > > > Could you help to point the direction? I have the files to reproduce it > > stablely. > > GDB works in two steps to read compile units. From you stack trace, it > looks like you are using an index (the .gdb_index kind). When GDB first > loads you binary, it reads in an index present in the binary (or in the > index cache) that lists all the entity names present in each compile > unit of the program. When you set a breakpoint using a name, GDB > "expands" all the compile units with something in it that matches what > you asked for. "Expand" means that GDB reads the full debug information > from the DWARF for that compile unit, creating some internal data > structures to represent it. > > It sounds like the breakpoint spec string you passed matches a lot of > compile units, and a lot of them get expanded. That creates a lot of > in-memory objects, eventually reaching some limit. > > Out of curiosity, what is the string you used to create your breakpoint? > From you stack trace, it sounds like it's "utility.h:LINE". > Thanks for your detailed explanation. Yes, it's `utility.h:560`, I added this break from vscode. > creating some internal data structures to represent it I wonder where would allocate so much memory, (at least 40GB), while the binary is 985MB with the whole debug info. Maybe many match results need too many memories? ``` $ ls -lh envoy-deadloop -r-xr-xr-x 1 zhudejiang.pt users 985M Nov 13 15:34 envoy-deadloop $ readelf -SW envoy-deadloop There are 48 section headers, starting at offset 0x3d856690: Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .interp PROGBITS 00000000000002e0 0002e0 00001c 00 A 0 0 1 [ 2] .note.ABI-tag NOTE 00000000000002fc 0002fc 000020 00 A 0 0 4 [ 3] .dynsym DYNSYM 0000000000000320 000320 003810 18 A 7 1 8 [ 4] .gnu.version VERSYM 0000000000003b30 003b30 0004ac 02 A 3 0 2 [ 5] .gnu.version_r VERNEED 0000000000003fdc 003fdc 0001d0 00 A 7 6 4 [ 6] .gnu.hash GNU_HASH 00000000000041b0 0041b0 0005ac 00 A 3 0 8 [ 7] .dynstr STRTAB 000000000000475c 00475c 001cce 00 A 0 0 1 [ 8] .rela.dyn RELA 0000000000006430 006430 4c20d8 18 A 3 0 8 [ 9] .rela.plt RELA 00000000004c8508 4c8508 002550 18 A 3 29 8 [10] .rodata PROGBITS 00000000004cb000 4cb000 c1b474 00 AMS 0 0 4096 [11] .gcc_except_table PROGBITS 00000000010e6474 10e6474 5f231c 00 A 0 0 4 [12] protodesc_cold PROGBITS 00000000016d87a0 16d87a0 079d68 00 A 0 0 32 [13] flags_help_cold PROGBITS 0000000001752520 1752520 001202 00 A 0 0 32 [14] .eh_frame_hdr PROGBITS 0000000001753724 1753724 4a1e4c 00 A 0 0 4 [15] .eh_frame PROGBITS 0000000001bf5570 1bf5570 13a6b0c 00 A 0 0 8 [16] .text PROGBITS 0000000002f9d080 2f9c080 3bbc38f 00 AX 0 0 64 [17] .init PROGBITS 0000000006b59410 6b58410 00001a 00 AX 0 0 4 [18] .fini PROGBITS 0000000006b5942c 6b5842c 000009 00 AX 0 0 4 [19] malloc_hook PROGBITS 0000000006b59436 6b58436 000786 00 AX 0 0 2 [20] google_malloc PROGBITS 0000000006b59bc0 6b58bc0 005bd0 00 AX 0 0 64 [21] .plt PROGBITS 0000000006b5f790 6b5e790 0018f0 00 AX 0 0 16 [22] .tdata PROGBITS 0000000006b62080 6b60080 000088 00 WAT 0 0 64 [23] .tbss NOBITS 0000000006b62140 6b60108 002500 00 WAT 0 0 64 [24] .fini_array FINI_ARRAY 0000000006b62108 6b60108 000008 08 WA 0 0 8 [25] .init_array INIT_ARRAY 0000000006b62110 6b60110 0064b0 08 WA 0 0 8 [26] .data.rel.ro PROGBITS 0000000006b685c0 6b665c0 1d7448 00 WA 0 0 32 [27] .dynamic DYNAMIC 0000000006d3fa08 6d3da08 000200 10 WA 7 0 8 [28] .got PROGBITS 0000000006d3fc08 6d3dc08 000760 00 WA 0 0 8 [29] .got.plt PROGBITS 0000000006d40368 6d3e368 000c88 00 WA 0 0 8 [30] .data PROGBITS 0000000006d42000 6d3f000 060c18 00 WA 0 0 64 [31] .tm_clone_table PROGBITS 0000000006da2c18 6d9fc18 000000 00 WA 0 0 8 [32] .bss NOBITS 0000000006da2c40 6d9fc18 2119e0 00 WA 0 0 64 [33] .comment PROGBITS 0000000000000000 6d9fc18 000060 01 MS 0 0 1 [34] .debug_info PROGBITS 0000000000000000 6d9fc78 8a32aa 00 0 0 1 [35] .debug_abbrev PROGBITS 0000000000000000 7642f22 093663 00 0 0 1 [36] .debug_aranges PROGBITS 0000000000000000 76d6590 62d2cd0 00 0 0 16 [37] .debug_line PROGBITS 0000000000000000 d9a9260 128ef333 00 0 0 1 [38] .debug_str PROGBITS 0000000000000000 20298593 1a392b 01 MS 0 0 1 [39] .debug_addr PROGBITS 0000000000000000 2043bebe 44b00a0 00 0 0 1 [40] .debug_rnglists PROGBITS 0000000000000000 248ebf5e 196f4db 00 0 0 1 [41] .debug_loclists PROGBITS 0000000000000000 2625b439 22e7fb 00 0 0 1 [42] .debug_frame PROGBITS 0000000000000000 26489c38 000060 00 0 0 8 [43] .debug_line_str PROGBITS 0000000000000000 26489c98 000569 01 MS 0 0 1 [44] .gdb_index PROGBITS 0000000000000000 2648a201 eef593d 00 0 0 1 [45] .symtab SYMTAB 0000000000000000 3537fb40 194d828 18 47 418626 8 [46] .shstrtab STRTAB 0000000000000000 36ccd368 0001fe 00 0 0 1 [47] .strtab STRTAB 0000000000000000 36ccd566 6b89128 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), l (large), p (processor specific) ``` > Expanding that many CUs could be legitimate, if there's really something > matching in all these CUs, or it could be a bug where GDB expands > unrelated CUs. There is an open bug related to a problem like this: > > https://sourceware.org/bugzilla/show_bug.cgi?id=29105 This looks like a CPU performance issue. May not be my issue, in my opinion. > > Although I'm not sure this is what you see. > > Is the project you build something open source that other people could > build and try? > Yes, it's an open source project. It builds with Bazel and depends on Go, which may be a bit complicated. This is the doc for build & run. https://github.com/mosn/envoy-go-extension#envoy > > Simon > --000000000000617da305ed664183--