Re: [PATCH 3/4] gdb: use python to colorize disassembler output

[Andrew Burgess <andrew.burgess@embecosm.com>]

Tom,

Thanks for the feedback.

* Tom Tromey <tom@tromey.com> [2021-10-27 14:38:04 -0600]:

> >>>>> "Andrew" == Andrew Burgess <andrew.burgess@embecosm.com> writes:
> 
> Andrew> In this commit I make use of the Python Pygments package to provide
> Andrew> the styling.  I did investigate making use of libsource-highlight,
> Andrew> however, I found the highlighting results to be inferior to those of
> Andrew> Pygments; only some mnemonics were highlighted, and highlighting of
> Andrew> register names such as r9d and r8d (on x86-64) was incorrect.
> 
> FWIW I think source highlight also only handles x86 assembly.
> 
> I didn't look to see if Pygments is any better about this... is it?

It's nothing special, but it doesn't seem to be restricted to looking
for specific mnemonics, at least it always seems to be highlighting
the mnemonic in x86-64 and risc-v code that I've thrown at it.  It
then highlights registers, immediates, and comments differently.

> 
> Andrew> One possibly odd choice I made with the new hook is to pass a
> Andrew> gdb.Architecture through, even though this is currently unused.
> ...
> Andrew> However, even though the Python function used to perform styling of
> Andrew> disassembly output is not part of any undocumented API, I don't want
> Andrew> to close the door on a user overriding this function to provide
> Andrew> architecture specific styling.
> 
> Sounds very reasonable to me.
> 
> Andrew> I don't know how much of a problem this is, for me, having the
> Andrew> disassembler fully styled is a big enough win.  But, if people see
> Andrew> this as a huge problem we can investigate mechanisms to restore the
> Andrew> print_address styling (for the case where Pygments is not available).
> 
> It does seem like a step backward when Pygments isn't available.
> 
> Maybe one idea would be to disable the hook on the first failure -- just
> fall back to the gdb styling, with an output stream that accepts
> styling, and re-disassemble the single failing instruction.  Would this
> work?

I've done this.  I'd be grateful for your thoughts on my proposed
solution, it relies on in-place new to recreate the string_file with
different properties, not sure what your thoughts are on this.

I'm sure there must be alternative solutions that don't require this
trick, if this seems too distasteful...

> 
> Andrew> +  if (!gdbpy_is_string (result.get ()))
> Andrew> +    return {};
> Andrew> +
> Andrew> +  gdbpy_ref<> unic = python_string_to_unicode (result.get ());
> Andrew> +  if (unic == nullptr)
> 
> I think the call to gdbpy_is_string is maybe not needed, as
> python_string_to_unicode already does type checking.
> 
> Andrew> +    {
> Andrew> +      gdbpy_print_stack ();
> Andrew> +      return {};
> Andrew> +    }
> Andrew> +  gdbpy_ref<> host_str (PyUnicode_AsEncodedString (unic.get (),
> Andrew> +						   host_charset (),
> Andrew> +						   nullptr));
> 
> .. though actually perhaps the whole sequence could be replace with
> python_string_to_host_string.  Though then you may prefer to use a
> unique_xmalloc_ptr rather than a std::string in the rest of the API.

I replaces the sequence with python_string_to_host_string as you
suggested.  I've left this function returning a std::string though.
The result is going to be placed into a string_file, which just wraps
a std::string anyway, so there doesn't seem like a huge benefit to
having this return anything other than std::string, and this way the
source code "colorize", and the disassembler "colorize_disasm" APIs
are pretty similar.

New patch is below.

Thanks,
Andrew

---

commit 468dc239e58de38634136691ebb509fc5df200be
Author: Andrew Burgess <andrew.burgess@embecosm.com>
Date:   Mon Oct 25 17:26:57 2021 +0100

    gdb: use python to colorize disassembler output
    
    This commit adds styling support to the disassembler output, as such
    two new commands are added to GDB:
    
      set style disassembler on|off
      show style disassembler
    
    In this commit I make use of the Python Pygments package to provide
    the styling.  I did investigate making use of libsource-highlight,
    however, I found the highlighting results to be inferior to those of
    Pygments; only some mnemonics were highlighted, and highlighting of
    register names such as r9d and r8d (on x86-64) was incorrect.
    
    To enable disassembler highlighting via Pygments, I've added a new
    extension language hook, which is then implemented for Python.  This
    hook is very similar to the existing hook for source code
    colorization.
    
    One possibly odd choice I made with the new hook is to pass a
    gdb.Architecture through, even though this is currently unused.  The
    reason this argument is not used is that, currently, styling is
    performed identically for all architectures.
    
    However, even though the Python function used to perform styling of
    disassembly output is not part of any undocumented API, I don't want
    to close the door on a user overriding this function to provide
    architecture specific styling.  To do this, the user would inevitably
    require access to the gdb.Architecture, and so I decided to add this
    field now.
    
    The styling is applied within gdb_disassembler::print_insn, to achieve
    this, gdb_disassembler now writes its output into a temporary buffer,
    styling is then applied to the contents of this buffer.  Finally the
    gdb_disassembler buffer is copied out to its final destination stream.
    
    There's a new test to check that the disassembler output includes some
    escape sequences, though I don't check for specific colours; the
    precise colors will depend on which instructions are in the
    disassembler output.
    
    The only negative change with this commit relates to how addresses are
    printed, in the case when the Python Pygments package is not
    available.  Addresses are printed via calls to GDB's print_address
    function.  Traditionally, this would provide styling for the address
    and symbol name if the output ui_file* supported styling.  Now that we
    want to apply styling after the disassembler has finished, all
    disassembler output is written into a temporary string_file, which is
    configured not to support styling.  As a result, the print_address
    call no longer performs styling.
    
    If Pygments is available, this isn't a huge problem, the output will
    be fully styled one the disassembler has finished.  However, if
    Pygments is not available, or fails for some reason, then, it is now
    too late to go back and have print_address apply styling.  We have
    lost all print_address styling in this case.
    
    I don't know how much of a problem this is, for me, having the
    disassembler fully styled is a big enough win.  But, if people see
    this as a huge problem we can investigate mechanisms to restore the
    print_address styling (for the case where Pygments is not available).

diff --git a/gdb/NEWS b/gdb/NEWS
index d001a03145d..37c58c527a9 100644
--- a/gdb/NEWS
+++ b/gdb/NEWS
@@ -32,6 +32,12 @@ maint show internal-warning backtrace
   internal-error, or an internal-warning.  This is on by default for
   internal-error and off by default for internal-warning.
 
+set style disassembly on|off
+show style disassembly
+  If GDB is compiled with Python support, and the Python Pygments
+  package is available, then, when this setting is on, disassembler
+  output will have styling applied.
+
 * Python API
 
   ** New function gdb.add_history(), which takes a gdb.Value object
diff --git a/gdb/cli/cli-style.c b/gdb/cli/cli-style.c
index 228fa698c13..d4e49e3df5b 100644
--- a/gdb/cli/cli-style.c
+++ b/gdb/cli/cli-style.c
@@ -38,6 +38,11 @@ bool cli_styling = true;
 
 bool source_styling = true;
 
+/* True if disassembler styling is enabled.  Note that this is only
+   consulted when cli_styling is true.  */
+
+bool disassembler_styling = true;
+
 /* Name of colors; must correspond to ui_file_style::basic_color.  */
 static const char * const cli_colors[] = {
   "none",
@@ -302,6 +307,18 @@ show_style_sources (struct ui_file *file, int from_tty,
     fprintf_filtered (file, _("Source code styling is disabled.\n"));
 }
 
+/* Implement 'show style disassembler'.  */
+
+static void
+show_style_disassembler (struct ui_file *file, int from_tty,
+			 struct cmd_list_element *c, const char *value)
+{
+  if (disassembler_styling)
+    fprintf_filtered (file, _("Disassembler output styling is enabled.\n"));
+  else
+    fprintf_filtered (file, _("Disassembler output styling is disabled.\n"));
+}
+
 void _initialize_cli_style ();
 void
 _initialize_cli_style ()
@@ -337,6 +354,15 @@ available if the appropriate extension is available at runtime."
 			   ), set_style_enabled, show_style_sources,
 			   &style_set_list, &style_show_list);
 
+  add_setshow_boolean_cmd ("disassembler", no_class, &disassembler_styling, _("\
+Set whether disassembler output styling is enabled."), _("\
+Show whether disassembler output styling is enabled."), _("\
+If enabled, disassembler output is styled.  Disassembler highlighting\n\
+requires the Python Pygments library, if this library is not available\n\
+then disassembler highlighting will not be possible."
+			   ), set_style_enabled, show_style_disassembler,
+			   &style_set_list, &style_show_list);
+
   file_name_style.add_setshow_commands (no_class, _("\
 Filename display styling.\n\
 Configure filename colors and display intensity."),
diff --git a/gdb/cli/cli-style.h b/gdb/cli/cli-style.h
index 78bc2cd6f1e..5361a644a95 100644
--- a/gdb/cli/cli-style.h
+++ b/gdb/cli/cli-style.h
@@ -128,6 +128,9 @@ extern cli_style_option version_style;
 /* True if source styling is enabled.  */
 extern bool source_styling;
 
+/* True if disassembler styling is enabled.  */
+extern bool disassembler_styling;
+
 /* True if styling is enabled.  */
 extern bool cli_styling;
 
diff --git a/gdb/disasm.c b/gdb/disasm.c
index c045dfc94a6..8f7ec00486e 100644
--- a/gdb/disasm.c
+++ b/gdb/disasm.c
@@ -753,9 +753,12 @@ get_all_disassembler_options (struct gdbarch *gdbarch)
 gdb_disassembler::gdb_disassembler (struct gdbarch *gdbarch,
 				    struct ui_file *file,
 				    di_read_memory_ftype read_memory_func)
-  : m_gdbarch (gdbarch)
+  : m_gdbarch (gdbarch),
+    m_buffer (!use_ext_lang_colorization_p && disassembler_styling
+	      && file->can_emit_style_escape ()),
+    m_dest (file)
 {
-  init_disassemble_info (&m_di, file, dis_asm_fprintf);
+  init_disassemble_info (&m_di, &m_buffer, dis_asm_fprintf);
   m_di.flavour = bfd_target_unknown_flavour;
   m_di.memory_error_func = dis_asm_memory_error;
   m_di.print_address_func = dis_asm_print_address;
@@ -784,14 +787,66 @@ gdb_disassembler::~gdb_disassembler ()
   disassemble_free_target (&m_di);
 }
 
+/* See disasm.h.  */
+
+bool gdb_disassembler::use_ext_lang_colorization_p = true;
+
+/* See disasm.h.  */
+
 int
 gdb_disassembler::print_insn (CORE_ADDR memaddr,
 			      int *branch_delay_insns)
 {
   m_err_memaddr.reset ();
+  m_buffer.clear ();
 
   int length = gdbarch_print_insn (arch (), memaddr, &m_di);
 
+  /* If we have successfully disassembled an instruction, styling is
+     on, we think that the extension language might be able to perform
+     styling for us, and the destination can support styling, then lets
+     call into the extension languages in order to style this output.  */
+  if (length > 0 && disassembler_styling
+      && use_ext_lang_colorization_p
+      && m_dest->can_emit_style_escape ())
+    {
+      gdb::optional<std::string> ext_contents;
+      ext_contents = ext_lang_colorize_disasm (m_buffer.string (), arch ());
+      if (ext_contents.has_value ())
+	m_buffer.string () = std::move (*ext_contents);
+      else
+	{
+	  /* The extension language failed to add styling to the
+	     disassembly output.  Set the static flag so that next time we
+	     disassemble we don't even bother attempting to use the
+	     extension language for styling.  */
+	  use_ext_lang_colorization_p = false;
+
+	  /* The instruction we just disassembled, and the extension
+	     languages failed to highlight, might have otherwise had some
+	     minimal highlighting applied by GDB.  In regain that
+	     highlighting we need to recreate m_buffer, but this time with
+	     styling support.
+
+	     To do this we perform an in-place new, but this time turn on
+	     the styling support, then we can re-disassembly the
+	     instruction, and gain any minimal styling GDB might add.  */
+	  gdb_static_assert ((std::is_same<decltype (m_buffer),
+			      string_file>::value));
+	  gdb_assert (!m_buffer.term_out ());
+	  m_buffer.~string_file ();
+	  new (&m_buffer) string_file (true);
+	  length = gdbarch_print_insn (arch (), memaddr, &m_di);
+	  gdb_assert (length > 0);
+	}
+    }
+
+  /* Push any disassemble output to the real destination stream.  We do
+     this even if the disassembler reported failure (-1) as the
+     disassembler may have printed something to its output stream.  */
+  m_di.fprintf_func (m_dest, "%s", m_buffer.c_str ());
+
+  /* If the disassembler failed then report an appropriate error.  */
   if (length < 0)
     {
       if (m_err_memaddr.has_value ())
diff --git a/gdb/disasm.h b/gdb/disasm.h
index f6de33e3db8..cb7f3768995 100644
--- a/gdb/disasm.h
+++ b/gdb/disasm.h
@@ -82,6 +82,31 @@ class gdb_disassembler
      non-memory error.  */
   gdb::optional<CORE_ADDR> m_err_memaddr;
 
+  /* Disassembler output is built up into this buffer.  Whether this
+     string_file is created with styling support or not depends on the
+     value of use_ext_lang_colorization_p, as well as whether disassembler
+     styling in general is turned on, and also, whether *m_dest supports
+     styling or not.  */
+  string_file m_buffer;
+
+  /* The stream to which disassembler output will be written.  */
+  ui_file *m_dest;
+
+  /* When true, m_buffer will be created without styling support,
+     otherwise, m_buffer will be created with styling support.
+
+     This field will initially be true, but will be set to false if
+     ext_lang_colorize_disasm fails to add styling at any time.
+
+     If the extension language is going to add the styling then m_buffer
+     should be created without styling support, the extension language will
+     then add styling at the end of the disassembly process.
+
+     If the extension language is not going to add the styling, then we
+     create m_buffer with styling support, and GDB will add minimal styling
+     (currently just to addresses and symbols) as it goes.  */
+  static bool use_ext_lang_colorization_p;
+
   static int dis_asm_fprintf (void *stream, const char *format, ...)
     ATTRIBUTE_PRINTF(2,3);
 
diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
index 631a7c03b31..6644bd517ff 100644
--- a/gdb/doc/gdb.texinfo
+++ b/gdb/doc/gdb.texinfo
@@ -26071,6 +26071,21 @@
 
 @item show style sources
 Show the current state of source code styling.
+
+@item set style disassembly @samp{on|off}
+Enable or disable disassembly styling.  This affects whether
+disassembly output, such as the output of the @code{disassemble}
+command, is styled.  Disassembly styling only works if styling in
+general is enabled (with @code{set style enabled on}), and if a source
+highlighting library is available to @value{GDBN}.
+
+To highlight disassembler output, @value{GDBN} must be compiled with
+Python support, and the Python Pygments package must be available.  If
+these requirements are not met then @value{GDBN} will not highlight
+disassembler output, even when this option is @samp{on}.
+
+@item show style disassembly
+Show the current state of disassembly styling.
 @end table
 
 Subcommands of @code{set style} control specific forms of styling.
diff --git a/gdb/extension-priv.h b/gdb/extension-priv.h
index 77f23e0f911..b2150624dde 100644
--- a/gdb/extension-priv.h
+++ b/gdb/extension-priv.h
@@ -257,6 +257,12 @@ struct extension_language_ops
      or an empty option.  */
   gdb::optional<std::string> (*colorize) (const std::string &name,
 					  const std::string &contents);
+
+  /* Colorize a single line of disassembler output, CONTENT.  This should
+     either return colorized (using ANSI terminal escapes) version of the
+     contents, or an empty optional.  */
+  gdb::optional<std::string> (*colorize_disasm) (const std::string &content,
+						 gdbarch *gdbarch);
 };
 
 /* State necessary to restore a signal handler to its previous value.  */
diff --git a/gdb/extension.c b/gdb/extension.c
index 89ab29f3d1c..6862147ac2f 100644
--- a/gdb/extension.c
+++ b/gdb/extension.c
@@ -904,6 +904,26 @@ ext_lang_colorize (const std::string &filename, const std::string &contents)
   return result;
 }
 
+/* See extension.h.  */
+
+gdb::optional<std::string>
+ext_lang_colorize_disasm (const std::string &content, gdbarch *gdbarch)
+{
+  gdb::optional<std::string> result;
+
+  for (const struct extension_language_defn *extlang : extension_languages)
+    {
+      if (extlang->ops == nullptr
+	  || extlang->ops->colorize_disasm == nullptr)
+	continue;
+      result = extlang->ops->colorize_disasm (content, gdbarch);
+      if (result.has_value ())
+	return result;
+    }
+
+  return result;
+}
+
 /* Called via an observer before gdb prints its prompt.
    Iterate over the extension languages giving them a chance to
    change the prompt.  The first one to change the prompt wins,
diff --git a/gdb/extension.h b/gdb/extension.h
index 2f2ca3e7743..e8d2fbc7fc3 100644
--- a/gdb/extension.h
+++ b/gdb/extension.h
@@ -319,6 +319,14 @@ extern void get_matching_xmethod_workers
 extern gdb::optional<std::string> ext_lang_colorize
   (const std::string &filename, const std::string &contents);
 
+/* Try to colorize a single line of disassembler output, CONTENT for
+   GDBARCH.  This will return either a colorized (using ANSI terminal
+   escapes) version of CONTENT, or an empty value if colorizing could not
+   be done.  */
+
+extern gdb::optional<std::string> ext_lang_colorize_disasm
+  (const std::string &content, gdbarch *gdbarch);
+
 #if GDB_SELF_TEST
 namespace selftests {
 extern void (*hook_set_active_ext_lang) ();
diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
index 7b6d8701548..6ab797c8c5e 100644
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -243,8 +243,19 @@ try:
         except:
             return None
 
+    def colorize_disasm(content, gdbarch):
+        # Don't want any errors.
+        try:
+            lexer = lexers.get_lexer_by_name("asm")
+            formatter = formatters.TerminalFormatter()
+            return highlight(content, lexer, formatter).rstrip()
+        except:
+            return None
 
 except:
 
     def colorize(filename, contents):
         return None
+
+    def colorize_disasm(content, gdbarch):
+        return None
diff --git a/gdb/python/python.c b/gdb/python/python.c
index c7b5e7faa8e..d407ca66405 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -121,6 +121,8 @@ static enum ext_lang_rc gdbpy_before_prompt_hook
   (const struct extension_language_defn *, const char *current_gdb_prompt);
 static gdb::optional<std::string> gdbpy_colorize
   (const std::string &filename, const std::string &contents);
+static gdb::optional<std::string> gdbpy_colorize_disasm
+  (const std::string &content, gdbarch *gdbarch);
 
 /* The interface between gdb proper and loading of python scripts.  */
 
@@ -162,6 +164,8 @@ static const struct extension_language_ops python_extension_ops =
   gdbpy_get_matching_xmethod_workers,
 
   gdbpy_colorize,
+
+  gdbpy_colorize_disasm,
 };
 
 /* The main struct describing GDB's interface to the Python
@@ -1181,6 +1185,69 @@ gdbpy_colorize (const std::string &filename, const std::string &contents)
   return std::string (PyBytes_AsString (host_str.get ()));
 }
 
+/* This is the extension_language_ops.colorize_disasm "method".  */
+
+static gdb::optional<std::string>
+gdbpy_colorize_disasm (const std::string &content, gdbarch *gdbarch)
+{
+  if (!gdb_python_initialized)
+    return {};
+
+  gdbpy_enter enter_py (get_current_arch (), current_language);
+
+  if (gdb_python_module == nullptr
+      || !PyObject_HasAttrString (gdb_python_module, "colorize_disasm"))
+    return {};
+
+  gdbpy_ref<> hook (PyObject_GetAttrString (gdb_python_module,
+					    "colorize_disasm"));
+  if (hook == nullptr)
+    {
+      gdbpy_print_stack ();
+      return {};
+    }
+
+  if (!PyCallable_Check (hook.get ()))
+    return {};
+
+  gdbpy_ref<> content_arg (PyString_FromString (content.c_str ()));
+  if (content_arg == nullptr)
+    {
+      gdbpy_print_stack ();
+      return {};
+    }
+
+  gdbpy_ref<> gdbarch_arg (gdbarch_to_arch_object (gdbarch));
+  if (gdbarch_arg == nullptr)
+    {
+      gdbpy_print_stack ();
+      return {};
+    }
+
+  gdbpy_ref<> result (PyObject_CallFunctionObjArgs (hook.get (),
+						    content_arg.get (),
+						    gdbarch_arg.get (),
+						    nullptr));
+  if (result == nullptr)
+    {
+      gdbpy_print_stack ();
+      return {};
+    }
+
+  if (result == Py_None)
+    return {};
+
+  gdb::unique_xmalloc_ptr<char> str
+    = python_string_to_host_string (result.get ());
+  if (str == nullptr)
+    {
+      gdbpy_print_stack ();
+      return {};
+    }
+
+ return std::string (str.get ());
+}
+
 \f
 
 /* Printing.  */
diff --git a/gdb/testsuite/gdb.base/style.exp b/gdb/testsuite/gdb.base/style.exp
index 91d3059612d..4cf367ec82d 100644
--- a/gdb/testsuite/gdb.base/style.exp
+++ b/gdb/testsuite/gdb.base/style.exp
@@ -182,12 +182,25 @@ proc run_style_tests { } {
 
 	gdb_test_no_output "set width 0"
 
-	set main [limited_style main function]
-	set func [limited_style some_called_function function]
-	# Somewhere should see the call to the function.
-	gdb_test "disassemble main" \
-	    [concat "Dump of assembler code for function $main:.*" \
-		 "[limited_style $hex address].*$func.*"]
+	# Disassembly highlighting is done by Python, so, if the
+	# required modules are not available we'll not get the full
+	# highlighting.
+	if { $::python_disassembly_highlighting } {
+	    # Check that the header line of the disassembly output is
+	    # styled correctly, the address at the start of the first
+	    # disassembly line is styled correctly, and that there is at
+	    # least one escape sequence in the disassembly output.
+	    set main [limited_style main function]
+	    gdb_test "disassemble main" \
+		[concat "Dump of assembler code for function $main:\\r\\n" \
+		     "\\s+[limited_style $hex address]\\s+<\\+$decimal>:\[^\\r\\n\]+\033\\\[${decimal}\[^\\r\\n\]+.*" ""]
+	} else {
+	    set main [limited_style main function]
+	    # Somewhere should see the call to the function.
+	    gdb_test "disassemble main" \
+		[concat "Dump of assembler code for function $main:.*" \
+		     "[limited_style $hex address].*<some_called_function>.*"]
+	}
 
 	set ifield [limited_style int_field variable]
 	set sfield [limited_style string_field variable]
@@ -312,6 +325,25 @@ proc test_startup_version_string { } {
     gdb_test "" "${vers}.*" "version is styled at startup"
 }
 
+# Check to see if the Python highlighting of disassembler output is
+# expected or not, this highlighting requires Python support in GDB,
+# and the Python pygments module to be available.
+clean_restart ${binfile}
+if {![skip_python_tests]} {
+    gdb_test_multiple "python import pygments" "" {
+	-re "ModuleNotFoundError: No module named 'pygments'.*$gdb_prompt $" {
+	    set python_disassembly_highlighting false
+	}
+	-re "ImportError: No module named pygments.*$gdb_prompt $" {
+	    set python_disassembly_highlighting false
+	}
+	-re "^python import pygments\r\n$gdb_prompt $" {
+	    set python_disassembly_highlighting true
+	}
+    }
+} else {
+    set python_disassembly_highlighting false
+}
 
 # Run tests with all styles in their default state.
 with_test_prefix "all styles enabled" {