[Bug demangler/106641] New: Endless Looping & Abnormal Memory Occupatio

[Bug demangler/106641] New: Endless Looping & Abnormal Memory Occupatio

[chkunq at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106641

            Bug ID: 106641
           Summary: Endless Looping & Abnormal Memory Occupatio
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: demangler
          Assignee: unassigned at gcc dot gnu.org
          Reporter: chkunq at gmail dot com
  Target Milestone: ---

Created attachment 53465
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=53465&action=edit
The input object file of binutils/nm that trigger an endless looping in
rust-demangler

When I executed binutils/nm-new with the command `nm-new -C [file in
attachment]`, I found that nm-new could not be executed within 24h and was
taking up an unusually large amount of memory.
Using gdb and inserting printf statements, I traced the problem to
libiberty/rust-demangle.c.

When parsing the symbol _RYOFGFF1FFFFFF_array_start, demangle_binder()
internally enters a dead loop (line 657).

When I implanted the printf statement and recompiled, I found that the number
of loops was unusually large (line 653), with a value of 9096425505278371,
which is clearly not normal for looping so many times.
And during the loop, it takes up a lot of memory (>30G), which may cause a
memory overflow.

Here is the call stack (from gdb)

#4  0x0000555555644036 in demangle_binder (rdm=0x7fffffffd9b0)
    at ./rust-demangle.c:662
#5  0x0000555555644f2c in demangle_type (rdm=0x7fffffffd9b0)
    at ./rust-demangle.c:956
#6  0x0000555555644dc4 in demangle_type (rdm=0x7fffffffd9b0)
    at ./rust-demangle.c:920
#7  0x00005555556445a6 in demangle_path (
    rdm=rdm@entry=0x7fffffffd9b0, in_value=in_value@entry=1)
    at ./rust-demangle.c:759
#8  0x00005555556456fd in demangle_path (in_value=1, 
    rdm=0x7fffffffd9b0) at ./rust-demangle.c:1482
#9  rust_demangle_callback (mangled=<optimized out>, 
    options=options@entry=259, 
    callback=callback@entry=0x555555645500 <str_buf_demangle_callback>,
opaque=opaque@entry=0x7fffffffda20) at ./rust-demangle.c:1482
#10 0x000055555564590d in rust_demangle (mangled=<optimized out>, 
    options=options@entry=259) at ./rust-demangle.c:1593
#11 0x000055555563226f in cplus_demangle (
    mangled=mangled@entry=0x5555556a7cc5 "_RYOFGFF1FFFFFF_array_start",
options=259, options@entry=3) at ./cplus-dem.c:166
#12 0x000055555558cfa3 in bfd_demangle (abfd=<optimized out>, 
    name=0x5555556a7cc5 "_RYOFGFF1FFFFFF_array_start", options=3)
    at bfd.c:2428
#13 0x00005555555827cb in print_symname (form=0x555555656c2c " %s", 
    info=0x7fffffffdb80, 
    name=0x5555556a7cc5 "_RYOFGFF1FFFFFF_array_start",