[binutils-gdb] gdb/tui: disable tui mode when an assert triggers

[binutils-gdb] gdb/tui: disable tui mode when an assert triggers

[Andrew Burgess]

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=a4f8f2902704e47b2e70a95995248a54d6df8beb

commit a4f8f2902704e47b2e70a95995248a54d6df8beb
Author: Andrew Burgess <aburgess@redhat.com>
Date:   Wed Jan 4 09:48:31 2023 +0000

    gdb/tui: disable tui mode when an assert triggers
    
    When an assert triggers in tui mode the output is not great, the
    internal backtrace that is generated is printed directly to the file
    descriptor for gdb_stderr, and, as a result, does not currently format
    itself correctly - the output uses only '\n' at the end of each line,
    and so, when the terminal is in raw mode, the cursor does not return
    to the start of each line after the '\n'.
    
    This is mostly fixable, we could update bt-utils.c to use '\r\n'
    instead of just '\n', and this would fix most of the problems.  The
    one we can't easily fix is if/when GDB is built to use execinfo
    instead of libbacktrace, in this case we use backtrace_symbols_fd to
    print the symbols, and this function only uses '\n' as the line
    terminator.  Fixing this would require switching to backtrace_symbols,
    but that API uses malloc, which is something we're trying to
    avoid (this code is called when GDB hits an error, so ideally we don't
    want to rely on malloc).
    
    However, the execinfo code is only used when libbacktrace is not
    available (or the user specifically disables libbacktrace) so maybe we
    can ignore that problem...
    
    ... but there is another problem.  When the backtrace is printed in
    raw mode, it is possible that the backtrace fills the screen.  With
    the terminal in raw mode we don't have the ability to scroll back,
    which means we loose some of the backtrace, which isn't ideal.
    
    In this commit I propose that we should disable tui mode whenever we
    handle a fatal signal, or when we hit the internal error code
    path (e.g. when an assert triggers).  With this done then we don't
    need to update the bt-utils.c code, and the execinfo version of the
    code (using backtrace_symbols_fd) works just fine.  We also get the
    ability to scroll back to view the error message and all of the
    backtrace, assuming the users terminal supports scrolling back.
    
    The only downside I see with this change is if the tui_disable call
    itself causes an error for some reason, or, if we handle a single at a
    time when it is not safe to call tui_disable, in these cases the extra
    tui_disable call might cause GDB to loose the original error.
    
    However, I think (just from personal experience) that the above two
    issues are pretty rare and the benefits from this change far out
    weighs the possible drawbacks.

Diff:
---
 gdb/event-top.c | 8 ++++++++
 gdb/utils.c     | 7 ++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/gdb/event-top.c b/gdb/event-top.c
index 4a46e1b9346..14984707df1 100644
--- a/gdb/event-top.c
+++ b/gdb/event-top.c
@@ -48,6 +48,10 @@
 #include "readline/readline.h"
 #include "readline/history.h"
 
+#ifdef TUI
+#include "tui/tui.h"
+#endif
+
 /* readline defines this.  */
 #undef savestring
 
@@ -940,6 +944,10 @@ unblock_signal (int sig)
 static void ATTRIBUTE_NORETURN
 handle_fatal_signal (int sig)
 {
+#ifdef TUI
+  tui_disable ();
+#endif
+
 #ifdef GDB_PRINT_INTERNAL_BACKTRACE
   const auto sig_write = [] (const char *msg) -> void
   {
diff --git a/gdb/utils.c b/gdb/utils.c
index 734c5bf7f70..95adbe58e4a 100644
--- a/gdb/utils.c
+++ b/gdb/utils.c
@@ -29,7 +29,8 @@
 #endif /* HAVE_SYS_RESOURCE_H */
 
 #ifdef TUI
-#include "tui/tui.h"		/* For tui_get_command_dimension.   */
+/* For tui_get_command_dimension and tui_disable.   */
+#include "tui/tui.h"
 #endif
 
 #ifdef __GO32__
@@ -354,6 +355,10 @@ internal_vproblem (struct internal_problem *problem,
       }
   }
 
+#ifdef TUI
+  tui_disable ();
+#endif
+
   /* Create a string containing the full error/warning message.  Need
      to call query with this full string, as otherwize the reason
      (error/warning) and question become separated.  Format using a