[gcc r12-3139] analyzer: Impose recursion limit on indirect calls.

[Ankur saini <arsenic@gcc.gnu.org>]

https://gcc.gnu.org/g:43a5d46feabd93ba78983919234f05f5fc9a0982

commit r12-3139-g43a5d46feabd93ba78983919234f05f5fc9a0982
Author: Ankur Saini <arsenic@sourceware.org>
Date:   Wed Aug 25 12:33:06 2021 +0530

    analyzer: Impose recursion limit on indirect calls.
    
    2021-08-25  Ankur Saini  <arsenic@sourceware.org>
    
    gcc/analyzer/ChangeLog:
            PR analyzer/101980
            * engine.cc (exploded_graph::maybe_create_dynamic_call): Don't create
            calls if max recursion limit is reached.

Diff:
---
 gcc/analyzer/engine.cc | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/gcc/analyzer/engine.cc b/gcc/analyzer/engine.cc
index 4ee92794941..9c604d1eb8c 100644
--- a/gcc/analyzer/engine.cc
+++ b/gcc/analyzer/engine.cc
@@ -3059,6 +3059,20 @@ exploded_graph::maybe_create_dynamic_call (const gcall *call,
 
       new_point.push_to_call_stack (sn_exit,
                                     next_point.get_supernode());
+
+      /* Impose a maximum recursion depth and don't analyze paths
+         that exceed it further.
+         This is something of a blunt workaround, but it only
+         applies to recursion (and mutual recursion), not to
+         general call stacks.  */
+      if (new_point.get_call_string ().calc_recursion_depth ()
+          > param_analyzer_max_recursion_depth)
+      {
+        if (logger)
+          logger->log ("rejecting call edge: recursion limit exceeded");
+        return false;
+      }
+
       next_state.push_call (*this, node, call, uncertainty);
 
       if (next_state.m_valid)