[Bug gdb/29188] New: gdb too slow and hogs memory for debug binary generated with gcc 11.2

[Bug gdb/29188] New: gdb too slow and hogs memory for debug binary generated with gcc 11.2

[rajpal.gusain at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

            Bug ID: 29188
           Summary: gdb too slow and hogs memory for debug binary
                    generated with gcc 11.2
           Product: gdb
           Version: 11.2
            Status: UNCONFIRMED
          Severity: critical
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: rajpal.gusain at gmail dot com
  Target Milestone: ---

Created attachment 14115
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14115&action=edit
pstack output captured when gdb was stuck

We recently migrated our projects to gcc 11.2 and it has dwarf 5 as default
debugging format (unsure if it's consequential).

And I notice that gdb is not liking it, debugging 11.2 compiled binary with gdb
11.2 is pathetically slow and sometimes even backtrace takes 3 mins and memory
consumption of gdb process also goes up significantly. 
I then tried old gdb 8.3 and I still have same problem


I dumped pstack on gdb process (with gdb 8.3) when it was churning wheels doing
backtrace and breakpoint completion and that's the only information I can share
right now.

Debugging in general is painstakingly slow and almost impractical. 


I did try to compile with dwarf4 format but it doesn't seem to help with the
gdb performance. 

Are there any suggestions/workarounds to deal with this issue?

Let me know if any other information is needed.

Binary was compiled with -g3 option.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

Simon Marchi <simark at simark dot ca> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |simark at simark dot ca

--- Comment #1 from Simon Marchi <simark at simark dot ca> ---
By the looks of you backtraces (GDB stuck in processing macro information), I
am guessing that it's the same as:

https://sourceware.org/bugzilla/show_bug.cgi?id=27754

If you run "readelf --debug-dump=macro" on your binary, do you get a ton of

  DW_MACRO_import - offset : 0x0

?  If so it's a problem in the compiler or linker.  It has apparently been
fixed (you can follow the paper trail on that bug), but I don't really
understand the details on that side.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[rajpal.gusain at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #2 from gusain <rajpal.gusain at gmail dot com> ---
Thanks for your suggestion but no there isn't even a single occurrence of it.

% readelf --debug-dump=macro <mybinary> 
%

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #3 from Simon Marchi <simark at simark dot ca> ---
(In reply to gusain from comment #2)
> Thanks for your suggestion but no there isn't even a single occurrence of it.
> 
> % readelf --debug-dump=macro <mybinary> 
> %

This is surprising given that you said you compiled with -g3.  And your
backtraces show GDB reading some macro information.  It could also be in a
shared library, though.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[rajpal.gusain at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #4 from gusain <rajpal.gusain at gmail dot com> ---
I did check shared libraries and I couldn't  find it there either. Is there any
other way to confirm it's the same issue?

If there is more information needed to narrow it further, can you let me know ?

Also, I checked other ticket and I have two comments
1. LTO is not in picture here
2. I have tried lld and gold as well but none seems to fix the issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #5 from Simon Marchi <simark at simark dot ca> ---
Well, your backtraces do show GDB reading macro information, so there's gotta
be some somewhere.

Actually, you can turn on:

(gdb) set debug dwarf-read 1

prior to running your experiment.  You should get some messages like this, that
will help you know which CU in which file GDB is slow at reading:

[dwarf-read] process_queue: Expanding one or more symtabs of objfile
/home/simark/build/binutils-gdb-one-target/gdb/gdb ...
[dwarf-read] process_queue: Expanding symtab of CU at offset 0x1054

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at sourceware dot org

--- Comment #6 from Tom Tromey <tromey at sourceware dot org> ---
I tend to think this is a dup as well.

The way gcc emits the macro info also triggered a problem with mold.
Maybe gdb should just ignore imports of 0x0 as a workaround.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #7 from Simon Marchi <simark at simark dot ca> ---
(In reply to Tom Tromey from comment #6)
> I tend to think this is a dup as well.
> 
> The way gcc emits the macro info also triggered a problem with mold.
> Maybe gdb should just ignore imports of 0x0 as a workaround.

I'm just worried that there could be legitimate cases of "import 0".

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[rajpal.gusain at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #8 from gusain <rajpal.gusain at gmail dot com> ---
(In reply to Simon Marchi from comment #5)
> Well, your backtraces do show GDB reading macro information, so there's
> gotta be some somewhere.
> 
> Actually, you can turn on:
> 
> (gdb) set debug dwarf-read 1
> 
> prior to running your experiment.  You should get some messages like this,
> that will help you know which CU in which file GDB is slow at reading:
> 
> [dwarf-read] process_queue: Expanding one or more symtabs of objfile
> /home/simark/build/binutils-gdb-one-target/gdb/gdb ...
> [dwarf-read] process_queue: Expanding symtab of CU at offset 0x1054


Thanks, and yes now it prints and appears to be stuck when I try breakpoint
completion by hitting tab after name hint

[dwarf-read] process_queue: Expanding symtab of CU at offset 0x5d2cd20
[dwarf-read] process_queue: Done expanding CU at offset 0x5d2cd20
[dwarf-read] process_queue: Done expanding symtabs of <binary>.
[dwarf-read] process_queue: Expanding one or more symtabs of objfile <binary>
...
[dwarf-read] process_queue: Expanding symtab of CU at offset 0x3644296
[dwarf-read] process_queue: Done expanding CU at offset 0x3644296
[dwarf-read] process_queue: Done expanding symtabs of <binary>.
[dwarf-read] process_queue: Expanding one or more symtabs of objfile <binary>
...
[dwarf-read] process_queue: Expanding symtab of CU at offset 0x2cc08bb
[dwarf-read] process_queue: Done expanding CU at offset 0x2cc08bb
[dwarf-read] process_queue: Done expanding symtabs of <binary>.
[dwarf-read] process_queue: Expanding one or more symtabs of objfile <binary>
...
[dwarf-read] process_queue: Expanding symtab of CU at offset 0x18f13e
[dwarf-read] process_queue: Done expanding CU at offset 0x18f13e
[dwarf-read] process_queue: Done expanding symtabs of <binary>.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[rajpal.gusain at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #9 from gusain <rajpal.gusain at gmail dot com> ---
(In reply to gusain from comment #8)
> (In reply to Simon Marchi from comment #5)
> > Well, your backtraces do show GDB reading macro information, so there's
> > gotta be some somewhere.
> > 
> > Actually, you can turn on:
> > 
> > (gdb) set debug dwarf-read 1
> > 
> > prior to running your experiment.  You should get some messages like this,
> > that will help you know which CU in which file GDB is slow at reading:
> > 
> > [dwarf-read] process_queue: Expanding one or more symtabs of objfile
> > /home/simark/build/binutils-gdb-one-target/gdb/gdb ...
> > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x1054
> 
> 
> Thanks, and yes now it prints and appears to be stuck when I try breakpoint
> completion by hitting tab after name hint
> 
> [dwarf-read] process_queue: Expanding symtab of CU at offset 0x5d2cd20
> [dwarf-read] process_queue: Done expanding CU at offset 0x5d2cd20
> [dwarf-read] process_queue: Done expanding symtabs of <binary>.
> [dwarf-read] process_queue: Expanding one or more symtabs of objfile
> <binary> ...
> [dwarf-read] process_queue: Expanding symtab of CU at offset 0x3644296
> [dwarf-read] process_queue: Done expanding CU at offset 0x3644296
> [dwarf-read] process_queue: Done expanding symtabs of <binary>.
> [dwarf-read] process_queue: Expanding one or more symtabs of objfile
> <binary> ...
> [dwarf-read] process_queue: Expanding symtab of CU at offset 0x2cc08bb
> [dwarf-read] process_queue: Done expanding CU at offset 0x2cc08bb
> [dwarf-read] process_queue: Done expanding symtabs of <binary>.
> [dwarf-read] process_queue: Expanding one or more symtabs of objfile
> <binary> ...
> [dwarf-read] process_queue: Expanding symtab of CU at offset 0x18f13e
> [dwarf-read] process_queue: Done expanding CU at offset 0x18f13e
> [dwarf-read] process_queue: Done expanding symtabs of <binary>.

Is there any other information I can get to confirm it's same as other issue
you mentioned ?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #10 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to gusain from comment #9)

> Is there any other information I can get to confirm it's same as other issue
> you mentioned ?

Try something like:

readelf --debug-dump=macro | grep 'import.* 0x0$'

See bug #27754

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[rajpal.gusain at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

--- Comment #11 from gusain <rajpal.gusain at gmail dot com> ---
Thanks for your help. It does seem like it's similar to 27754 but let me share
few observations

% readelf --debug-dump=macro ld-g | grep 'import.*0x0$' | wc -l 
277652

% readelf --debug-dump=macro lld-g | grep 'import.*0x0$' | wc -l 
277655

% readelf --debug-dump=macro gold-g | grep 'import.*0x0$' | wc -l 
0

You had a comment on other ticket that it doesn't seem to affect lld but my
binary linked with ldd doesn't suggest it. 
With gold, I have 0 occurrences but binary is no good because when I bring it
up in gdb, source information seems to be incorrect. When I put a breakpoint in
my code, gdb tells me that it's in some libstdc++ file.


Do you still think it's not gdb bug ? If not, whose bug is it ?

Are you aware of any workaround (gold link doesn't work for me and I have other
issues with it as I mentioned before) ?

Do you think it's gold linker bug and I should file a ticket for it ? Is there
information I can get which can shed more light on this gold-linked binary
issue  ? 



(In reply to Tom Tromey from comment #10)
> (In reply to gusain from comment #9)
> 
> > Is there any other information I can get to confirm it's same as other issue
> > you mentioned ?
> 
> Try something like:
> 
> readelf --debug-dump=macro | grep 'import.* 0x0$'
> 
> See bug #27754

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29188] gdb too slow and hogs memory for debug binary generated with gcc 11.2

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29188

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |DUPLICATE

--- Comment #12 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to gusain from comment #11)

> With gold, I have 0 occurrences but binary is no good because when I bring
> it up in gdb, source information seems to be incorrect. When I put a
> breakpoint in my code, gdb tells me that it's in some libstdc++ file.

This sounds like a different problem, which we'd have to investigate
separately.

> Do you still think it's not gdb bug ? If not, whose bug is it ?

I think the mold author put in a workaround for this issue.
While pedantically it may be a gcc bug, in practice the fixes are in the
linker.

> Are you aware of any workaround (gold link doesn't work for me and I have
> other issues with it as I mentioned before) ?

GNU ld should work, or an updated mold.  I'm unsure about lld.

> Do you think it's gold linker bug and I should file a ticket for it ? Is
> there information I can get which can shed more light on this gold-linked
> binary issue  ? 

I'm afraid I don't know the answer here.

Since I think this particular bug is another instance of the macro-import
problem, I'm closing it as a dup.

*** This bug has been marked as a duplicate of bug 27754 ***

-- 
You are receiving this mail because:
You are on the CC list for the bug.