Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[R. Diez]

Hi all:

I recently upgraded my cross-compilation toolchain for ARM Cortex-M4F to these component versions:

  Binutils 2.36.1
  GCC 10.3
  GDB 10.1

For more information, see this comment of mine:

https://sourceware.org/bugzilla/show_bug.cgi?id=23710#c18

I noticed that GDB is now using much more RAM than before, and not just for LTO builds, which is what that bug report was about.

I just tried with this open-source firmware of mine:

https://github.com/rdiez/JtagDue

For a debug build (non LTO), firmware.elf weighs around 1.5 MB. Most of it is debug information, because the firmware.bin file only weighs 76 kB.

The debug information is not compressed.

When GDB needs to load the symbols, because you are touching some C++ source code, there is a noticeable pause, and GDB uses 385 MiB of RAM. Is that not too much for this smallish project?

I tried a release build, compiled with LTO, and GDB became unresponsive in the same situation. I killed GDB manually when it got to consume over 2 GiB of RAM, because I have had such a misbehaving GDB freeze my Ubuntu 20.04.2, not even the mouse was moving. I do not have swap, so I would have expected the Out of Memory Killer to kick in, but it did not. I had to physically switch my computer off.

Before I spend more time investigating this issue: is this a known problem?

The bug report mentioned above seems to suggest that there is something foul in this area, but there does not seem to be much interest in the past months. The trouble is, I am seeing now pauses and high memory usage even in non-LTO builds. GDB is still usable, but it is becoming annoying.

I have been using this toolchain for many years, and I am getting this kind of problem with GDB only relatively recently.

Best regards,
  rdiez

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[David Blaikie]

> The debug information is not compressed.

How are you determining that ^ ?

(I ask, because historically compressed debug info was opt-in and used
a section name mangling (.zdebug_*) to notate compressed debug
sections - but recent versions of GCC have started compressing by
default and using an ELF section flag (SHF_COMPRESSED) without section
name mangling (so dumping section names would not be enough to
determine whether compressed debug info was used)

But, yeah, that does seem like a lot of work, for all of 76kB of
non-debug executable...

On Sat, Apr 17, 2021 at 3:01 PM R. Diez via Gdb <gdb@sourceware.org> wrote:
>
> Hi all:
>
> I recently upgraded my cross-compilation toolchain for ARM Cortex-M4F to these component versions:
>
>   Binutils 2.36.1
>   GCC 10.3
>   GDB 10.1
>
> For more information, see this comment of mine:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=23710#c18
>
> I noticed that GDB is now using much more RAM than before, and not just for LTO builds, which is what that bug report was about.
>
> I just tried with this open-source firmware of mine:
>
> https://github.com/rdiez/JtagDue
>
> For a debug build (non LTO), firmware.elf weighs around 1.5 MB. Most of it is debug information, because the firmware.bin file only weighs 76 kB.
>
> The debug information is not compressed.
>
> When GDB needs to load the symbols, because you are touching some C++ source code, there is a noticeable pause, and GDB uses 385 MiB of RAM. Is that not too much for this smallish project?
>
> I tried a release build, compiled with LTO, and GDB became unresponsive in the same situation. I killed GDB manually when it got to consume over 2 GiB of RAM, because I have had such a misbehaving GDB freeze my Ubuntu 20.04.2, not even the mouse was moving. I do not have swap, so I would have expected the Out of Memory Killer to kick in, but it did not. I had to physically switch my computer off.
>
> Before I spend more time investigating this issue: is this a known problem?
>
> The bug report mentioned above seems to suggest that there is something foul in this area, but there does not seem to be much interest in the past months. The trouble is, I am seeing now pauses and high memory usage even in non-LTO builds. GDB is still usable, but it is becoming annoying.
>
> I have been using this toolchain for many years, and I am getting this kind of problem with GDB only relatively recently.
>
> Best regards,
>   rdiez

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[R. Diez]

>> The debug information is not compressed.
> 
> How are you determining that ^ ?
> 
> (I ask, because historically compressed debug info was opt-in and used
> a section name mangling (.zdebug_*) to notate compressed debug
> sections - but recent versions of GCC have started compressing by
> default and using an ELF section flag (SHF_COMPRESSED) without section
> name mangling (so dumping section names would not be enough to
> determine whether compressed debug info was used)


I do not know about SHF_COMPRESSED.

I checked with:

$ ./arm-none-eabi-ld --help

[...]
--compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
     Compress DWARF debug sections using zlib
     Default: none
[...]


I did not quite trust the "Default: none" statement, so I tested in the past by passing the following flag:

-Wl,--compress-debug-sections=zlib

And the resulting elf file was less than half the size.

That is how I assume that the debug information is not being compressed.

I am not compressing because for such a little project it is not worth it.

Regards,
   rdiez

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[David Blaikie]

Yeah, that checks out then. (you can also confirm that SHF_COMPRESSED
is not used by readelf -S or llvm-readelf -S and checking for the "C"
value in the Flags column)

& yeah, that does seem like quite a bit of RAM usage for a relatively
small amount of debug info. Though I'm not a regular/frequent gdb
developer, so I don't have any particular insight there - if no one
else chimes in, might be worth running valgrind --tool=massif to get a
memory profile, might point to what part of gdb is using all the RAM.

On Sun, Apr 18, 2021 at 1:16 AM R. Diez <rdiezmail-binutils@yahoo.de> wrote:
>
>
> >> The debug information is not compressed.
> >
> > How are you determining that ^ ?
> >
> > (I ask, because historically compressed debug info was opt-in and used
> > a section name mangling (.zdebug_*) to notate compressed debug
> > sections - but recent versions of GCC have started compressing by
> > default and using an ELF section flag (SHF_COMPRESSED) without section
> > name mangling (so dumping section names would not be enough to
> > determine whether compressed debug info was used)
>
>
> I do not know about SHF_COMPRESSED.
>
> I checked with:
>
> $ ./arm-none-eabi-ld --help
>
> [...]
> --compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
>      Compress DWARF debug sections using zlib
>      Default: none
> [...]
>
>
> I did not quite trust the "Default: none" statement, so I tested in the past by passing the following flag:
>
> -Wl,--compress-debug-sections=zlib
>
> And the resulting elf file was less than half the size.
>
> That is how I assume that the debug information is not being compressed.
>
> I am not compressing because for such a little project it is not worth it.
>
> Regards,
>    rdiez

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[Simon Marchi]

On 2021-04-17 5:30 p.m., R. Diez via Gdb wrote:
> Hi all:
> 
> I recently upgraded my cross-compilation toolchain for ARM Cortex-M4F to these component versions:
> 
>   Binutils 2.36.1
>   GCC 10.3
>   GDB 10.1
> 
> For more information, see this comment of mine:
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=23710#c18
> 
> I noticed that GDB is now using much more RAM than before, and not just for LTO builds, which is what that bug report was about.
> 
> I just tried with this open-source firmware of mine:
> 
> https://github.com/rdiez/JtagDue
> 
> For a debug build (non LTO), firmware.elf weighs around 1.5 MB. Most of it is debug information, because the firmware.bin file only weighs 76 kB.

I don't have anything useful to add.  But I was wondering if you could
share a set of commands to configure / build your project exactly like
you configure / build it yourself.  And then a set of GDB commands to do
something where you experience such a slowdown.  We could start by
trying to reproduce it, that would be a good first step to investigate
it.  To have an isolated environment that we can reproduce (make sure
GDB doesn't load some .gdbinit file), start your GDB with -nx.

> The debug information is not compressed.
> 
> When GDB needs to load the symbols, because you are touching some C++ source code, there is a noticeable pause, and GDB uses 385 MiB of RAM. Is that not too much for this smallish project?

I never really paid attention, so I can't tell.  I have noticed some
strange slowdowns recently when I debug GDB with GDB (probably a bit
bigger than your program).  For example, when I type "backtrace", there
is sometimes a noticeable pause between the display of some frames.

> I tried a release build, compiled with LTO, and GDB became unresponsive in the same situation. I killed GDB manually when it got to consume over 2 GiB of RAM, because I have had such a misbehaving GDB freeze my Ubuntu 20.04.2, not even the mouse was moving. I do not have swap, so I would have expected the Out of Memory Killer to kick in, but it did not. I had to physically switch my computer off.

Yep, that's really bad :(.

> Before I spend more time investigating this issue: is this a known problem?
> 
> The bug report mentioned above seems to suggest that there is something foul in this area, but there does not seem to be much interest in the past months. The trouble is, I am seeing now pauses and high memory usage even in non-LTO builds. GDB is still usable, but it is becoming annoying.
> 
> I have been using this toolchain for many years, and I am getting this kind of problem with GDB only relatively recently.

Do you have a version in mind that you remember worked better?  We could
compare the profiling numbers between the two versions.

Simon

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[David Blaikie]

4My high water mark reported my massif for the non-lto build was 197MB -
most of that seems to come from decoding the macro information
(dwarf_decode_macros). You could try dropping down from -g3 to -g2 to see
if that helps. (I realize the linux kernel uses a bunch of macros and
benefits from debug info for macros, but it'd at least help isolate the
problem - might help clarify whether the lto case is related or. not)

On Sun, Apr 18, 2021 at 10:02 AM R. Diez <rdiezmail-binutils@yahoo.de>
wrote:

>
>  > [...]
> > & yeah, that does seem like quite a bit of RAM usage for a relatively
> > small amount of debug info. Though I'm not a regular/frequent gdb
> > developer, so I don't have any particular insight there - if no one
> > else chimes in, might be worth running valgrind --tool=massif to get a
> > memory profile, might point to what part of gdb is using all the RAM.
>
> I am worried that I may not be building the toolchain and/or GDB correctly.
>
> Could you do the following test for me, just to confirm that you are
> indeed seeing such a high memory consumption?
>
> Could you build GDB 10.1 for ARM? This is how I am configuring it for
> cross-debugging (somewhat simplified):
>
> configure \
>    CFLAGS="-g0 -O3 -flto=9 -march=native" \
>    CXXFLAGS="-g0 -O3 -flto=9 -march=native" \
>    --target=arm-none-eabi
>
> Maybe you want to build it without -O3 and LTO, just using the defaults. I
> have been using those flags for years with previous versions without any
> issues.
>
> Then load one of the .elf files in the attachment like this. There is no
> need to have any ARM CPU available:
>
> ./arm-none-eabi-gdb firmware-debug-non-lto.elf
>
> At this point, GDB should be using less than 15 MiB of RAM.
>
> Now issue this GDB command:
>
> print StartOfUserCode
>
> You should see an output like this:
>
> $1 = {void (void)} 0x866d8 <StartOfUserCode()>
>
> Did that take more than 1 second? How much RAM is your GDB using now?
>
> If you repeat that with firmware-release-lto.elf , how much memory is your
> GDB using? Be prepared to kill it before it breaks your system though. I
> normally kill it after a few seconds when it reaches 2 GiB of RAM.
>
> Thanks in advance,
>    rdiez
>

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[David Blaikie]

Oh, managed to get the lto case to stop & valgrind to log it.

Yeah... peaked out over 5GB of memory usage, and:

74.12% (4,372,282,848B) 0x3A4D42: macro_alloc

On Sun, Apr 18, 2021 at 9:57 PM David Blaikie <dblaikie@gmail.com> wrote:

> 4My high water mark reported my massif for the non-lto build was 197MB -
> most of that seems to come from decoding the macro information
> (dwarf_decode_macros). You could try dropping down from -g3 to -g2 to see
> if that helps. (I realize the linux kernel uses a bunch of macros and
> benefits from debug info for macros, but it'd at least help isolate the
> problem - might help clarify whether the lto case is related or. not)
>
> On Sun, Apr 18, 2021 at 10:02 AM R. Diez <rdiezmail-binutils@yahoo.de>
> wrote:
>
>>
>>  > [...]
>> > & yeah, that does seem like quite a bit of RAM usage for a relatively
>> > small amount of debug info. Though I'm not a regular/frequent gdb
>> > developer, so I don't have any particular insight there - if no one
>> > else chimes in, might be worth running valgrind --tool=massif to get a
>> > memory profile, might point to what part of gdb is using all the RAM.
>>
>> I am worried that I may not be building the toolchain and/or GDB
>> correctly.
>>
>> Could you do the following test for me, just to confirm that you are
>> indeed seeing such a high memory consumption?
>>
>> Could you build GDB 10.1 for ARM? This is how I am configuring it for
>> cross-debugging (somewhat simplified):
>>
>> configure \
>>    CFLAGS="-g0 -O3 -flto=9 -march=native" \
>>    CXXFLAGS="-g0 -O3 -flto=9 -march=native" \
>>    --target=arm-none-eabi
>>
>> Maybe you want to build it without -O3 and LTO, just using the defaults.
>> I have been using those flags for years with previous versions without any
>> issues.
>>
>> Then load one of the .elf files in the attachment like this. There is no
>> need to have any ARM CPU available:
>>
>> ./arm-none-eabi-gdb firmware-debug-non-lto.elf
>>
>> At this point, GDB should be using less than 15 MiB of RAM.
>>
>> Now issue this GDB command:
>>
>> print StartOfUserCode
>>
>> You should see an output like this:
>>
>> $1 = {void (void)} 0x866d8 <StartOfUserCode()>
>>
>> Did that take more than 1 second? How much RAM is your GDB using now?
>>
>> If you repeat that with firmware-release-lto.elf , how much memory is
>> your GDB using? Be prepared to kill it before it breaks your system though.
>> I
>> normally kill it after a few seconds when it reaches 2 GiB of RAM.
>>
>> Thanks in advance,
>>    rdiez
>>
>

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[R. Diez]

> [...]
> You could try dropping down from -g3 to -g2 to see if that helps.

Many thanks for your help.

I did a quick test and dropping to -g2 solves the problem indeed.

I'll do some more testing and create a new bug about this.

Best regards,
   rdiez

Re: Greatly increased GDB memory and CPU usage with newest embedded ARM toolchain

[R. Diez]

> [...]
> Yeah... peaked out over 5GB of memory usage, and:
 > [...]


For future reference, I have created a bug for this issue:

   Excessive CPU load and memory usage with -g3 debug info

   https://sourceware.org/bugzilla/show_bug.cgi?id=27754

Best regards,
   rdiez