[Bug c++/105115] New: `demangle_const` causes infinite recursion and stack overflow

[Bug c++/105115] New: `demangle_const` causes infinite recursion and stack overflow

[ronghua at iu dot edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105115

            Bug ID: 105115
           Summary: `demangle_const` causes infinite recursion and stack
                    overflow
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: ronghua at iu dot edu
  Target Milestone: ---

Created attachment 52721
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52721&action=edit
output.txt is output of ASAN, and inf-recursion is the poc

At commit `b8e92c571baed4e794bd62b7bf417fa8bbaf5c95` of `binutils-gdb`, the
given poc can produces crash on `cxxfilt`:

build2/binutils/cxxfilt < inf-recursion

[Bug c++/105115] `demangle_const` causes infinite recursion and stack overflow

[ronghua at iu dot edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105115

--- Comment #1 from ronghua at iu dot edu ---
Hello, It's already more than one month since my post, and the bug still exists
at latest commit (e.i. e84412c6a3c3eed1a4eca5fe18f329bfa7efea40), so I would
like to know why it is still `UNCONFIRMED`.

Thanks.

[Bug c++/105115] `demangle_const` causes infinite recursion and stack overflow

[ronghua at iu dot edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105115

--- Comment #2 from ronghua at iu dot edu ---
The simplified PoC:

$ echo _RYAbB2_ | ./binutils/cxxfilt
Segmentation fault


Root Cause Analysis:

1. In `rust_demangle_callback`, "_R" at beginning causes `rdm.version ==
0`[1][2], so `demangle_path` in the else-branch is called[3] with `rdm.sym ==
"YAbB2_"`, with `rdm.next` field initialized to 0.

2. Next char 'Y' causes `demangle_type` to be called[4] in a switch-case
branch, with `rdm.next == 1`.

3. Next char 'A' causes `demangle_type` to be called first[5] to consume char
'b', then `demangle_const` is called[6] with `rdm.next == 3`.

4. Next char 'B' is consumed[7], and "2_" is parsed as `integer_62`[8] to be
value 3, and this vlaue is assigned to `rdm.next`, then `demangle_const` is
called recursively[9]. However, currently the situation is `rdm.next == 3`
again, which is exactly same as last call to `demangle_const`, and this causes
infinite recursion.

[1]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L1356
[2]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L1362
[3]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L1453
[4]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L759
[5]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L934
[6]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L938
[7]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L1151
[8]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L1153
[9]
https://github.com/bminor/binutils-gdb/blob/626d0e40e55c35a4f143b70def498734e8ed3c2a/libiberty/rust-demangle.c#L1158

Possible Fix:

The infinite recursion check added in this
[commit](https://github.com/bminor/binutils-gdb/commit/f10f8617a302f45dae721eae0cd659911f03d864)
should also be applied to `demangle_const`