[PATCH,RFC 1/3] Add new function lang_GNU_GIMPLE

[PATCH,RFC 1/3] Add new function lang_GNU_GIMPLE

[Indu Bhagat]

gcc/ChangeLog:

	* langhooks.c (lang_GNU_GIMPLE): New Function.
	* langhooks.h (lang_GNU_GIMPLE): New Prototype.

---
 gcc/ChangeLog   | 5 +++++
 gcc/langhooks.c | 9 +++++++++
 gcc/langhooks.h | 1 +
 3 files changed, 15 insertions(+)

diff --git a/gcc/langhooks.c b/gcc/langhooks.c
index 2df97f2..f3a64c1 100644
--- a/gcc/langhooks.c
+++ b/gcc/langhooks.c
@@ -825,3 +825,12 @@ lang_GNU_OBJC (void)
 {
   return strncmp (lang_hooks.name, "GNU Objective-C", 15) == 0;
 }
+
+/* Returns true if the current lang_hooks represents the GNU GIMPLE
+   frontend.  */
+
+bool
+lang_GNU_GIMPLE (void)
+{
+  return strncmp (lang_hooks.name, "GNU GIMPLE", 10) == 0;
+}
diff --git a/gcc/langhooks.h b/gcc/langhooks.h
index a45579b..0ac794e 100644
--- a/gcc/langhooks.h
+++ b/gcc/langhooks.h
@@ -570,5 +570,6 @@ extern bool lang_GNU_C (void);
 extern bool lang_GNU_CXX (void);
 extern bool lang_GNU_Fortran (void);
 extern bool lang_GNU_OBJC (void);
+extern bool lang_GNU_GIMPLE (void);
 
 #endif /* GCC_LANG_HOOKS_H */
-- 
1.8.3.1

[PATCH,RFC 3/3] Create CTF debug hooks

[Indu Bhagat]

Initialize CTF debug hooks when -gtLEVEL is specified.  Import the CTF header
from binutils.

gcc/ChangeLog:

	* Makefile.in: Add ctfout.* files to GTFILES.
	* ctfout.c: New file.
	* ctfout.h: Likewise.
	* debug.h (ctf_debug_init): New function.
	* gengtype.c (open_base_files): Add ctfout.h to ifiles.
	* toplev.c (process_options): Warn and ignore -gtLEVEL if frontend is
	not C.
	(toplev::finalize): Finalize CTF containers.

gcc/testsuite/ChangeLog:

	* gcc.dg/debug/ctf/ctf-1.c: New test.
	* gcc.dg/debug/ctf/ctf-preamble-1.c: Likewise.
	* gcc.dg/debug/ctf/ctf.exp: Add CTF testsuite.
	* gcc.dg/debug/dwarf2-ctf-1.c: New test.

include/ChangeLog:

	* ctf.h: Import from binutils.

---
 gcc/ChangeLog                                   |  11 +
 gcc/Makefile.in                                 |   3 +
 gcc/ctfout.c                                    | 171 +++++++++
 gcc/ctfout.h                                    |  41 +++
 gcc/debug.h                                     |   4 +
 gcc/gengtype.c                                  |   4 +-
 gcc/testsuite/ChangeLog                         |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c          |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp          |  41 +++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c       |   7 +
 gcc/toplev.c                                    |  24 ++
 include/ChangeLog                               |   4 +
 include/ctf.h                                   | 471 ++++++++++++++++++++++++
 14 files changed, 803 insertions(+), 2 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 6677f77..44d9078 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1254,6 +1254,7 @@ OBJS = \
 	cfgloopanal.o \
 	cfgloopmanip.o \
 	cfgrtl.o \
+	ctfout.o \
 	symtab.o \
 	cgraph.o \
 	cgraphbuild.o \
@@ -2532,6 +2533,8 @@ GTFILES = $(CPPLIB_H) $(srcdir)/input.h $(srcdir)/coretypes.h \
   $(srcdir)/dwarf2asm.c \
   $(srcdir)/dwarf2cfi.c \
   $(srcdir)/dwarf2out.c \
+  $(srcdir)/ctfout.h \
+  $(srcdir)/ctfout.c \
   $(srcdir)/tree-vect-generic.c \
   $(srcdir)/dojump.c $(srcdir)/emit-rtl.h \
   $(srcdir)/emit-rtl.c $(srcdir)/except.h $(srcdir)/explow.c $(srcdir)/expr.c \
diff --git a/gcc/ctfout.c b/gcc/ctfout.c
new file mode 100644
index 0000000..9e9c48f
--- /dev/null
+++ b/gcc/ctfout.c
@@ -0,0 +1,171 @@
+/* Output ctf format from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+<http://www.gnu.org/licenses/>.  */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "target.h"
+#include "rtl.h"
+#include "tree.h"
+#include "memmodel.h"
+#include "tm_p.h"
+#include "varasm.h"
+#include "output.h"
+#include "dwarf2asm.h"
+#include "debug.h"
+#include "ctfout.h"
+
+/* CTF debug hooks.  This is initialized by ctf_debug_init.  */
+static struct gcc_debug_hooks ctf_debug_hooks;
+
+/* The real debug hooks.  */
+static const struct gcc_debug_hooks *real_debug_hooks;
+
+/* A CTF container object - one per translation unit.  */
+static GTY(()) ctf_container_ref ctf_container;
+
+static int ctf_label_num;
+
+/* Pointers to various ctf sections.  */
+static GTY(()) section *ctf_info_section;
+
+/* Section names used to hold CTF debugging information.  */
+#ifndef CTF_INFO_SECTION_NAME
+#define CTF_INFO_SECTION_NAME  ".ctf"
+#endif
+
+/* Section flags for .ctf section.  */
+
+/* CTF debug info section.  */
+#define CTF_INFO_SECTION_FLAGS (SECTION_DEBUG)
+
+/* Maximum size (in bytes) of an artificially generated ctf label.  */
+#define MAX_CTF_LABEL_BYTES 40
+
+static char ctf_info_section_label[MAX_CTF_LABEL_BYTES];
+
+#ifndef CTF_INFO_SECTION_LABEL
+#define CTF_INFO_SECTION_LABEL			"Lctf"
+#endif
+
+/* Forward declarations for functions defined in this file.  */
+static inline ctf_container_ref
+new_ctf_container (unsigned char ctp_flags);
+
+static void output_ctf_preamble (void);
+static void output_ctf_header (void);
+
+/* Allocate a new ctf container with the desired flags.  */
+static inline ctf_container_ref
+new_ctf_container (unsigned char ctp_flags)
+{
+  ctf_container = ggc_cleared_alloc<ctf_container_t> ();
+
+  ctf_container->ctf_magic = CTF_MAGIC;
+  ctf_container->ctf_version = CTF_VERSION;
+  ctf_container->ctf_flags = ctp_flags;
+
+  return ctf_container;
+}
+
+static inline void
+delete_ctf_container (ctf_container_ref ctfc)
+{
+  ggc_free (ctfc);
+  ctfc = NULL;
+}
+/* Initialize a ctf container per translation unit.  */
+static void
+init_ctf_containers (void)
+{
+  ctf_container = new_ctf_container (0);
+}
+
+/* Output the ctf preamble.  */
+static void
+output_ctf_preamble (void)
+{
+  dw2_asm_output_data (2, ctf_container->ctf_magic,
+		       "CTF preamble magic number");
+  dw2_asm_output_data (1, ctf_container->ctf_version, "CTF preamble version");
+  dw2_asm_output_data (1, ctf_container->ctf_flags, "CTF preamble flags");
+}
+
+/* Output the ctf header.  */
+static void
+output_ctf_header (void)
+{
+  switch_to_section (ctf_info_section);
+  ASM_OUTPUT_LABEL (asm_out_file, ctf_info_section_label);
+
+  output_ctf_preamble ();
+}
+
+/* Initialize the various sections and labels for ctf output.  */
+void
+init_ctf_sections (void)
+{
+  /* CTF debug info is generated for C frontend only, at the moment.  */
+
+  ctf_info_section = get_section (CTF_INFO_SECTION_NAME,
+				  CTF_INFO_SECTION_FLAGS,
+				  NULL);
+  ASM_GENERATE_INTERNAL_LABEL (ctf_info_section_label,
+			       CTF_INFO_SECTION_LABEL, ctf_label_num++);
+}
+
+static void
+ctf_define (unsigned int lineno, const char *buffer)
+{
+  real_debug_hooks->define (lineno, buffer);
+}
+
+static void
+ctf_finish (const char *filename)
+{
+  init_ctf_sections ();
+  init_ctf_containers ();
+
+  output_ctf_header ();
+
+  real_debug_hooks->finish (filename);
+}
+
+const struct gcc_debug_hooks *
+ctf_debug_init (const struct gcc_debug_hooks *hooks)
+{
+  ctf_debug_hooks = *hooks;
+  real_debug_hooks = hooks;
+
+  ctf_debug_hooks.finish = ctf_finish;
+  ctf_debug_hooks.define = ctf_define;
+
+  return &ctf_debug_hooks;
+}
+
+/* Reset all state within ctfout.c sot that we can rerun the compiler
+   within the same process.  For use by toplev::finalize.  */
+void ctfout_c_finalize (void)
+{
+  ctf_info_section = NULL;
+
+  delete_ctf_container (ctf_container);
+}
+
+#include "gt-ctfout.h"
diff --git a/gcc/ctfout.h b/gcc/ctfout.h
new file mode 100644
index 0000000..c129d6d
--- /dev/null
+++ b/gcc/ctfout.h
@@ -0,0 +1,41 @@
+/* ctfout.h - Various declarations for functions found in ctfout.c
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+<http://www.gnu.org/licenses/>.  */
+
+#ifndef GCC_CTFOUT_H
+#define GCC_CTFOUT_H 1
+
+#include "ctf.h"
+
+/* A ctf container for the context passed around when generating ctf debug
+ * info.  There is one container per translation unit.  */
+typedef struct GTY(()) ctf_container
+{
+  /* CTF Preamble.  */
+  unsigned short ctf_magic;
+  unsigned char ctf_version;
+  unsigned char ctf_flags;
+  /* Global type definitions.  */
+  //  htab_t ctf_types;
+} ctf_container_t;
+
+typedef ctf_container_t * ctf_container_ref;
+
+void ctfout_c_finalize (void);
+
+#endif /* GCC_CTFOUT_H */
diff --git a/gcc/debug.h b/gcc/debug.h
index 412e7ca..846784e 100644
--- a/gcc/debug.h
+++ b/gcc/debug.h
@@ -256,6 +256,10 @@ extern bool dwarf2out_default_as_locview_support (void);
 extern const struct gcc_debug_hooks *
 dump_go_spec_init (const char *, const struct gcc_debug_hooks *);
 
+/* For CTF debug info generation setup.  */
+extern const struct gcc_debug_hooks *
+ctf_debug_init (const struct gcc_debug_hooks *);
+
 /* Instance discriminator mapping table.  See final.c.  */
 typedef hash_map<const_tree, int> decl_to_instance_map_t;
 extern decl_to_instance_map_t *decl_to_instance_map;
diff --git a/gcc/gengtype.c b/gcc/gengtype.c
index 5331733..39de737 100644
--- a/gcc/gengtype.c
+++ b/gcc/gengtype.c
@@ -1724,8 +1724,8 @@ open_base_files (void)
       "tree-dfa.h", "tree-ssa.h", "reload.h", "cpplib.h", "tree-chrec.h",
       "except.h", "output.h",  "cfgloop.h", "target.h", "lto-streamer.h",
       "target-globals.h", "ipa-ref.h", "cgraph.h", "symbol-summary.h",
-      "ipa-prop.h", "ipa-fnsummary.h", "dwarf2out.h", "omp-general.h",
-      "omp-offload.h", NULL
+      "ipa-prop.h", "ipa-fnsummary.h", "dwarf2out.h", "ctfout.h",
+      "omp-general.h", "omp-offload.h", NULL
     };
     const char *const *ifp;
     outf_p gtype_desc_c;
diff --git a/gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c b/gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
new file mode 100644
index 0000000..1be0dfb
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
@@ -0,0 +1,6 @@
+/* { dg-do compile } */
+/* { dg-options "-gt" } */
+
+void func(void)
+{
+}
diff --git a/gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c b/gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
new file mode 100644
index 0000000..fa377ad
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
@@ -0,0 +1,11 @@
+/* Verify the CTF preamble in the ctf section.  */
+
+/* { dg-do compile )  */
+/* { dg-options "-O0 -gt -dA" } */
+/* { dg-final { scan-assembler "0xdff2.*CTF preamble magic number" } } */
+/* { dg-final { scan-assembler "0x4.*CTF preamble version" } } */
+/* { dg-final { scan-assembler "0.*CTF preamble flags" } } */
+
+void func (void)
+{
+}
diff --git a/gcc/testsuite/gcc.dg/debug/ctf/ctf.exp b/gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
new file mode 100644
index 0000000..46055f8
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
@@ -0,0 +1,41 @@
+#   Copyright (C) 2002-2019 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING3.  If not see
+# <http://www.gnu.org/licenses/>.
+
+# GCC testsuite that uses the `dg.exp' driver.
+
+# Load support procs.
+load_lib gcc-dg.exp
+
+# Disable on ptx (in sync with DWARF testsuite)
+if { [istarget nvptx-*-*] } {
+    return
+}
+
+# If a testcase doesn't have special options, use these.
+global DEFAULT_CFLAGS
+if ![info exists DEFAULT_CFLAGS] then {
+    set DEFAULT_CFLAGS " -ansi -pedantic-errors"
+}
+
+# Initialize `dg'.
+dg-init
+
+# Main loop.
+dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/*.\[cS\] ]] \
+	"" $DEFAULT_CFLAGS
+
+# All done.
+dg-finish
diff --git a/gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c b/gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
new file mode 100644
index 0000000..4ffa8ee
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
@@ -0,0 +1,7 @@
+/* Verify that CTF debug info can co-exist with other debug formats.  */
+/* { dg-do compile } */
+/* { dg-options "-gt -dA" } */
+/* { dg-final { scan-assembler "0xdff2.*CTF preamble magic number" } } */
+
+void func (void)
+{ }
diff --git a/gcc/toplev.c b/gcc/toplev.c
index d300ac2..3e0028a 100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -72,6 +72,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "pass_manager.h"
 #include "auto-profile.h"
 #include "dwarf2out.h"
+#include "ctfout.h"
 #include "ipa-reference.h"
 #include "symbol-summary.h"
 #include "tree-vrp.h"
@@ -1312,6 +1313,8 @@ parse_alignment_opts (void)
 static void
 process_options (void)
 {
+  const char *language_string = lang_hooks.name;
+
   /* Just in case lang_hooks.post_options ends up calling a debug_hook.
      This can happen with incorrect pre-processed input. */
   debug_hooks = &do_nothing_debug_hooks;
@@ -1484,6 +1487,17 @@ process_options (void)
 	debug_info_level = DINFO_LEVEL_NONE;
     }
 
+  /* CTF is supported for only C at this time.
+     Compiling with -flto results in frontend language of GNU GIMPLE.  */
+  if (!lang_GNU_C () && !lang_GNU_GIMPLE ()
+      && ctf_debug_info_level > CTFINFO_LEVEL_NONE)
+    {
+      warning_at (UNKNOWN_LOCATION, 0,
+		  "CTF debug info requested, but not supported for %s frontend",
+		  language_string);
+      ctf_debug_info_level = CTFINFO_LEVEL_NONE;
+    }
+
   if (flag_dump_final_insns && !flag_syntax_only && !no_backend)
     {
       FILE *final_output = fopen (flag_dump_final_insns, "w");
@@ -1556,6 +1570,15 @@ process_options (void)
       flag_var_tracking_uninit = 0;
     }
 
+  /* The debug hooks are used to generate ctf debug info.  The API of debug
+   * hooks, hence, cannot be used for both -fdump-go-spec and -gt.  */
+  if (flag_dump_go_spec != NULL && ctf_debug_info_level > CTFINFO_LEVEL_NONE)
+    sorry ("CTF debug info generation is not supported with "
+	   "-fdump-go-spec");
+
+  if (ctf_debug_info_level > CTFINFO_LEVEL_NONE)
+    debug_hooks = ctf_debug_init (debug_hooks);
+
   /* The debug hooks are used to implement -fdump-go-spec because it
      gives a simple and stable API for all the information we need to
      dump.  */
@@ -2387,6 +2410,7 @@ toplev::finalize (void)
   cgraph_c_finalize ();
   cgraphunit_c_finalize ();
   dwarf2out_c_finalize ();
+  ctfout_c_finalize ();
   gcse_c_finalize ();
   ipa_cp_c_finalize ();
   ira_costs_c_finalize ();
diff --git a/include/ctf.h b/include/ctf.h
new file mode 100644
index 0000000..cab92d1
--- /dev/null
+++ b/include/ctf.h
@@ -0,0 +1,471 @@
+/* CTF format description.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+   This file is part of libctf.
+
+   libctf is free software; you can redistribute it and/or modify it under
+   the terms of the GNU General Public License as published by the Free
+   Software Foundation; either version 3, or (at your option) any later
+   version.
+
+   This program is distributed in the hope that it will be useful, but
+   WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+   See the GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; see the file COPYING.  If not see
+   <http://www.gnu.org/licenses/>.  */
+
+#ifndef	_CTF_H
+#define	_CTF_H
+
+#include <sys/types.h>
+#include <limits.h>
+#include <stdint.h>
+
+
+#ifdef	__cplusplus
+extern "C"
+{
+#endif
+
+/* CTF - Compact ANSI-C Type Format
+
+   This file format can be used to compactly represent the information needed
+   by a debugger to interpret the ANSI-C types used by a given program.
+   Traditionally, this kind of information is generated by the compiler when
+   invoked with the -g flag and is stored in "stabs" strings or in the more
+   modern DWARF format.  CTF provides a representation of only the information
+   that is relevant to debugging a complex, optimized C program such as the
+   operating system kernel in a form that is significantly more compact than
+   the equivalent stabs or DWARF representation.  The format is data-model
+   independent, so consumers do not need different code depending on whether
+   they are 32-bit or 64-bit programs; libctf automatically compensates for
+   endianness variations.  CTF assumes that a standard ELF symbol table is
+   available for use in the debugger, and uses the structure and data of the
+   symbol table to avoid storing redundant information.  The CTF data may be
+   compressed on disk or in memory, indicated by a bit in the header.  CTF may
+   be interpreted in a raw disk file, or it may be stored in an ELF section,
+   typically named .ctf.  Data structures are aligned so that a raw CTF file or
+   CTF ELF section may be manipulated using mmap(2).
+
+   The CTF file or section itself has the following structure:
+
+   +--------+--------+---------+----------+----------+-------+--------+
+   |  file  |  type  |  data   | function | variable | data  | string |
+   | header | labels | objects |   info   |   info   | types | table  |
+   +--------+--------+---------+----------+----------+-------+--------+
+
+   The file header stores a magic number and version information, encoding
+   flags, and the byte offset of each of the sections relative to the end of the
+   header itself.  If the CTF data has been uniquified against another set of
+   CTF data, a reference to that data also appears in the the header.  This
+   reference is the name of the label corresponding to the types uniquified
+   against.
+
+   Following the header is a list of labels, used to group the types included in
+   the data types section.  Each label is accompanied by a type ID i.  A given
+   label refers to the group of types whose IDs are in the range [0, i].
+
+   Data object and function records are stored in the same order as they appear
+   in the corresponding symbol table, except that symbols marked SHN_UNDEF are
+   not stored and symbols that have no type data are padded out with zeroes.
+   For each data object, the type ID (a small integer) is recorded.  For each
+   function, the type ID of the return type and argument types is recorded.
+
+   Variable records (as distinct from data objects) provide a modicum of support
+   for non-ELF systems, mapping a variable name to a CTF type ID.  The variable
+   names are sorted into ASCIIbetical order, permitting binary searching.
+
+   The data types section is a list of variable size records that represent each
+   type, in order by their ID.  The types themselves form a directed graph,
+   where each node may contain one or more outgoing edges to other type nodes,
+   denoted by their ID.
+
+   Strings are recorded as a string table ID (0 or 1) and a byte offset into the
+   string table.  String table 0 is the internal CTF string table.  String table
+   1 is the external string table, which is the string table associated with the
+   ELF symbol table for this object.  CTF does not record any strings that are
+   already in the symbol table, and the CTF string table does not contain any
+   duplicated strings.
+
+   If the CTF data has been merged with another parent CTF object, some outgoing
+   edges may refer to type nodes that exist in another CTF object.  The debugger
+   and libctf library are responsible for connecting the appropriate objects
+   together so that the full set of types can be explored and manipulated.
+
+   This connection is done purely using the ctf_import() function.  There is no
+   notation anywhere in the child CTF file indicating which parent it is
+   connected to: it is the debugger's responsibility to track this.  */
+
+#define CTF_MAX_TYPE	0xfffffffe	/* Max type identifier value.  */
+#define CTF_MAX_PTYPE	0x7fffffff	/* Max parent type identifier value.  */
+#define CTF_MAX_NAME 0x7fffffff		/* Max offset into a string table.  */
+#define CTF_MAX_VLEN	0xffffff /* Max struct, union, enum members or args.  */
+
+/* See ctf_type_t */
+#define CTF_MAX_SIZE	0xfffffffe	/* Max size of a v2 type in bytes. */
+#define CTF_LSIZE_SENT	0xffffffff	/* Sentinel for v2 ctt_size.  */
+
+
+  /* Start of actual data structure definitions.
+
+     Every field in these structures must have corresponding code in the
+     endianness-swapping machinery in libctf/ctf-open.c.  */
+
+typedef struct ctf_preamble
+{
+  unsigned short ctp_magic;	/* Magic number (CTF_MAGIC).  */
+  unsigned char ctp_version;	/* Data format version number (CTF_VERSION).  */
+  unsigned char ctp_flags;	/* Flags (see below).  */
+} ctf_preamble_t;
+
+typedef struct ctf_header
+{
+  ctf_preamble_t cth_preamble;
+  uint32_t cth_parlabel;	/* Ref to name of parent lbl uniq'd against.  */
+  uint32_t cth_parname;		/* Ref to basename of parent.  */
+  uint32_t cth_lbloff;		/* Offset of label section.  */
+  uint32_t cth_objtoff;		/* Offset of object section.  */
+  uint32_t cth_funcoff;		/* Offset of function section.  */
+  uint32_t cth_varoff;		/* Offset of variable section.  */
+  uint32_t cth_typeoff;		/* Offset of type section.  */
+  uint32_t cth_stroff;		/* Offset of string section.  */
+  uint32_t cth_strlen;		/* Length of string section in bytes.  */
+} ctf_header_t;
+
+#define cth_magic   cth_preamble.ctp_magic
+#define cth_version cth_preamble.ctp_version
+#define cth_flags   cth_preamble.ctp_flags
+
+#define CTF_MAGIC	0xdff2	/* Magic number identifying header.  */
+
+/* Data format version number.  */
+
+
+#define CTF_VERSION_3 4
+#define CTF_VERSION CTF_VERSION_3 /* Current version.  */
+
+#define CTF_F_COMPRESS	0x1	/* Data buffer is compressed by libctf.  */
+
+typedef struct ctf_lblent
+{
+  uint32_t ctl_label;		/* Ref to name of label.  */
+  uint32_t ctl_type;		/* Last type associated with this label.  */
+} ctf_lblent_t;
+
+typedef struct ctf_varent
+{
+  uint32_t ctv_name;		/* Reference to name in string table.  */
+  uint32_t ctv_type;		/* Index of type of this variable.  */
+} ctf_varent_t;
+
+/* In format v2, type sizes, measured in bytes, come in two flavours.  Nearly
+   all of them fit into a (UINT_MAX - 1), and thus can be stored in the ctt_size
+   member of a ctf_stype_t.  The maximum value for these sizes is CTF_MAX_SIZE.
+   Types larger than this must be stored in the ctf_lsize member of a
+   ctf_type_t.  Use of this member is indicated by the presence of
+   CTF_LSIZE_SENT in ctt_size.  */
+
+typedef struct ctf_stype
+{
+  uint32_t ctt_name;		/* Reference to name in string table.  */
+  uint32_t ctt_info;		/* Encoded kind, variant length (see below).  */
+#ifndef __GNUC__
+  union
+  {
+    uint32_t _size;		/* Size of entire type in bytes.  */
+    uint32_t _type;		/* Reference to another type.  */
+  } _u;
+#else
+  __extension__
+  union
+  {
+    uint32_t ctt_size;		/* Size of entire type in bytes.  */
+    uint32_t ctt_type;		/* Reference to another type.  */
+  };
+#endif
+} ctf_stype_t;
+
+typedef struct ctf_type
+{
+  uint32_t ctt_name;		/* Reference to name in string table.  */
+  uint32_t ctt_info;		/* Encoded kind, variant length (see below).  */
+#ifndef __GNUC__
+union
+  {
+    uint32_t _size;		/* Always CTF_LSIZE_SENT.  */
+    uint32_t _type;		/* Do not use.  */
+  } _u;
+#else
+  __extension__
+  union
+  {
+    uint32_t ctt_size;		/* Always CTF_LSIZE_SENT.  */
+    uint32_t ctt_type;		/* Do not use.  */
+  };
+#endif
+  uint32_t ctt_lsizehi;		/* High 32 bits of type size in bytes.  */
+  uint32_t ctt_lsizelo;		/* Low 32 bits of type size in bytes.  */
+} ctf_type_t;
+
+#ifndef __GNUC__
+#define ctt_size _u._size	/* For fundamental types that have a size.  */
+#define ctt_type _u._type	/* For types that reference another type.  */
+#endif
+
+/* The following macros and inline functions compose and decompose values for
+   ctt_info and ctt_name, as well as other structures that contain name
+   references.  Use outside libdtrace-ctf itself is explicitly for access to CTF
+   files directly: types returned from the library will always appear to be
+   CTF_V2.
+
+   v1: (transparently upgraded to v2 at open time: may be compiled out of the
+   library)
+               ------------------------
+   ctt_info:   | kind | isroot | vlen |
+               ------------------------
+               15   11    10    9     0
+
+   v2:
+               ------------------------
+   ctt_info:   | kind | isroot | vlen |
+               ------------------------
+               31    26    25  24     0
+
+   CTF_V1 and V2 _INFO_VLEN have the same interface:
+
+   kind = CTF_*_INFO_KIND(c.ctt_info);     <-- CTF_K_* value (see below)
+   vlen = CTF_*_INFO_VLEN(fp, c.ctt_info); <-- length of variable data list
+
+   stid = CTF_NAME_STID(c.ctt_name);     <-- string table id number (0 or 1)
+   offset = CTF_NAME_OFFSET(c.ctt_name); <-- string table byte offset
+
+   c.ctt_info = CTF_TYPE_INFO(kind, vlen);
+   c.ctt_name = CTF_TYPE_NAME(stid, offset);  */
+
+
+#define CTF_V2_INFO_KIND(info)		(((info) & 0xfc000000) >> 26)
+#define CTF_V2_INFO_ISROOT(info)	(((info) & 0x2000000) >> 25)
+#define CTF_V2_INFO_VLEN(info)		(((info) & CTF_MAX_VLEN))
+
+#define CTF_NAME_STID(name)		((name) >> 31)
+#define CTF_NAME_OFFSET(name)		((name) & CTF_MAX_NAME)
+
+/* V2 only. */
+#define CTF_TYPE_INFO(kind, isroot, vlen) \
+	(((kind) << 26) | (((isroot) ? 1 : 0) << 25) | ((vlen) & CTF_MAX_VLEN))
+
+#define CTF_TYPE_NAME(stid, offset) \
+	(((stid) << 31) | ((offset) & CTF_MAX_NAME))
+
+/* The next set of macros are for public consumption only.  Not used internally,
+   since the relevant type boundary is dependent upon the version of the file at
+   *opening* time, not the version after transparent upgrade.  Use
+   ctf_type_isparent() / ctf_type_ischild() for that.  */
+
+#define CTF_V2_TYPE_ISPARENT(fp, id)	((id) <= CTF_MAX_PTYPE)
+#define CTF_V2_TYPE_ISCHILD(fp, id)	((id) > CTF_MAX_PTYPE)
+#define CTF_V2_TYPE_TO_INDEX(id)	((id) & CTF_MAX_PTYPE)
+#define CTF_V2_INDEX_TO_TYPE(id, child) ((child) ? ((id) | (CTF_MAX_PTYPE+1)) : (id))
+
+
+/* Valid for both V1 and V2. */
+#define CTF_TYPE_LSIZE(cttp) \
+	(((uint64_t)(cttp)->ctt_lsizehi) << 32 | (cttp)->ctt_lsizelo)
+#define CTF_SIZE_TO_LSIZE_HI(size)	((uint32_t)((uint64_t)(size) >> 32))
+#define CTF_SIZE_TO_LSIZE_LO(size)	((uint32_t)(size))
+
+#define CTF_STRTAB_0	0	/* String table id 0 (in-CTF).  */
+#define CTF_STRTAB_1	1	/* String table id 1 (ELF strtab).  */
+
+/* Values for CTF_TYPE_KIND().  If the kind has an associated data list,
+   CTF_INFO_VLEN() will extract the number of elements in the list, and
+   the type of each element is shown in the comments below. */
+
+#define CTF_K_UNKNOWN	0	/* Unknown type (used for padding).  */
+#define CTF_K_INTEGER	1	/* Variant data is CTF_INT_DATA (see below).  */
+#define CTF_K_FLOAT	2	/* Variant data is CTF_FP_DATA (see below).  */
+#define CTF_K_POINTER	3	/* ctt_type is referenced type.  */
+#define CTF_K_ARRAY	4	/* Variant data is single ctf_array_t.  */
+#define CTF_K_FUNCTION	5	/* ctt_type is return type, variant data is
+				   list of argument types (unsigned short's for v1,
+				   uint32_t's for v2).  */
+#define CTF_K_STRUCT	6	/* Variant data is list of ctf_member_t's.  */
+#define CTF_K_UNION	7	/* Variant data is list of ctf_member_t's.  */
+#define CTF_K_ENUM	8	/* Variant data is list of ctf_enum_t's.  */
+#define CTF_K_FORWARD	9	/* No additional data; ctt_name is tag.  */
+#define CTF_K_TYPEDEF	10	/* ctt_type is referenced type.  */
+#define CTF_K_VOLATILE	11	/* ctt_type is base type.  */
+#define CTF_K_CONST	12	/* ctt_type is base type.  */
+#define CTF_K_RESTRICT	13	/* ctt_type is base type.  */
+#define CTF_K_SLICE	14	/* Variant data is a ctf_slice_t.  */
+
+#define CTF_K_MAX	63	/* Maximum possible (V2) CTF_K_* value.  */
+
+/* Values for ctt_type when kind is CTF_K_INTEGER.  The flags, offset in bits,
+   and size in bits are encoded as a single word using the following macros.
+   (However, you can also encode the offset and bitness in a slice.)  */
+
+#define CTF_INT_ENCODING(data) (((data) & 0xff000000) >> 24)
+#define CTF_INT_OFFSET(data)   (((data) & 0x00ff0000) >> 16)
+#define CTF_INT_BITS(data)     (((data) & 0x0000ffff))
+
+#define CTF_INT_DATA(encoding, offset, bits) \
+       (((encoding) << 24) | ((offset) << 16) | (bits))
+
+#define CTF_INT_SIGNED	0x01	/* Integer is signed (otherwise unsigned).  */
+#define CTF_INT_CHAR	0x02	/* Character display format.  */
+#define CTF_INT_BOOL	0x04	/* Boolean display format.  */
+#define CTF_INT_VARARGS	0x08	/* Varargs display format.  */
+
+/* Use CTF_CHAR to produce a char that agrees with the system's native
+   char signedness.  */
+#if CHAR_MIN == 0
+# define CTF_CHAR (CTF_INT_CHAR)
+#else
+# define CTF_CHAR (CTF_INT_CHAR | CTF_INT_SIGNED)
+#endif
+
+/* Values for ctt_type when kind is CTF_K_FLOAT.  The encoding, offset in bits,
+   and size in bits are encoded as a single word using the following macros.
+   (However, you can also encode the offset and bitness in a slice.)  */
+
+#define CTF_FP_ENCODING(data)  (((data) & 0xff000000) >> 24)
+#define CTF_FP_OFFSET(data)    (((data) & 0x00ff0000) >> 16)
+#define CTF_FP_BITS(data)      (((data) & 0x0000ffff))
+
+#define CTF_FP_DATA(encoding, offset, bits) \
+       (((encoding) << 24) | ((offset) << 16) | (bits))
+
+/* Variant data when kind is CTF_K_FLOAT is an encoding in the top eight bits.  */
+#define CTF_FP_ENCODING(data)	(((data) & 0xff000000) >> 24)
+
+#define CTF_FP_SINGLE	1	/* IEEE 32-bit float encoding.  */
+#define CTF_FP_DOUBLE	2	/* IEEE 64-bit float encoding.  */
+#define CTF_FP_CPLX	3	/* Complex encoding.  */
+#define CTF_FP_DCPLX	4	/* Double complex encoding.  */
+#define CTF_FP_LDCPLX	5	/* Long double complex encoding.  */
+#define CTF_FP_LDOUBLE	6	/* Long double encoding.  */
+#define CTF_FP_INTRVL	7	/* Interval (2x32-bit) encoding.  */
+#define CTF_FP_DINTRVL	8	/* Double interval (2x64-bit) encoding.  */
+#define CTF_FP_LDINTRVL	9	/* Long double interval (2x128-bit) encoding.  */
+#define CTF_FP_IMAGRY	10	/* Imaginary (32-bit) encoding.  */
+#define CTF_FP_DIMAGRY	11	/* Long imaginary (64-bit) encoding.  */
+#define CTF_FP_LDIMAGRY	12	/* Long double imaginary (128-bit) encoding.  */
+
+#define CTF_FP_MAX	12	/* Maximum possible CTF_FP_* value */
+
+/* A slice increases the offset and reduces the bitness of the referenced
+   ctt_type, which must be a type which has an encoding (fp, int, or enum).  We
+   also store the referenced type in here, because it is easier to keep the
+   ctt_size correct for the slice than to shuffle the size into here and keep
+   the ctt_type where it is for other types.  */
+
+typedef struct ctf_slice
+{
+  uint32_t cts_type;
+  unsigned char cts_offset;
+  unsigned char cts_bits;
+} ctf_slice_t;
+
+typedef struct ctf_array_v1
+{
+  unsigned short cta_contents;	/* Reference to type of array contents.  */
+  unsigned short cta_index;	/* Reference to type of array index.  */
+  uint32_t cta_nelems;		/* Number of elements.  */
+} ctf_array_v1_t;
+
+typedef struct ctf_array
+{
+  uint32_t cta_contents;	/* Reference to type of array contents.  */
+  uint32_t cta_index;		/* Reference to type of array index.  */
+  uint32_t cta_nelems;		/* Number of elements.  */
+} ctf_array_t;
+
+/* Most structure members have bit offsets that can be expressed using a short.
+   Some don't.  ctf_member_t is used for structs which cannot contain any of
+   these large offsets, whereas ctf_lmember_t is used in the latter case.  If
+   any member of a given struct has an offset that cannot be expressed using a
+   uint32_t, all members will be stored as type ctf_lmember_t.  This is expected
+   to be very rare (but nonetheless possible).  */
+
+#define CTF_LSTRUCT_THRESH	536870912
+
+
+typedef struct ctf_member_v2
+{
+  uint32_t ctm_name;		/* Reference to name in string table.  */
+  uint32_t ctm_offset;		/* Offset of this member in bits.  */
+  uint32_t ctm_type;		/* Reference to type of member.  */
+} ctf_member_t;
+
+typedef struct ctf_lmember_v2
+{
+  uint32_t ctlm_name;		/* Reference to name in string table.  */
+  uint32_t ctlm_offsethi;	/* High 32 bits of member offset in bits.  */
+  uint32_t ctlm_type;		/* Reference to type of member.  */
+  uint32_t ctlm_offsetlo;	/* Low 32 bits of member offset in bits.  */
+} ctf_lmember_t;
+
+#define	CTF_LMEM_OFFSET(ctlmp) \
+	(((uint64_t)(ctlmp)->ctlm_offsethi) << 32 | (ctlmp)->ctlm_offsetlo)
+#define	CTF_OFFSET_TO_LMEMHI(offset)	((uint32_t)((uint64_t)(offset) >> 32))
+#define	CTF_OFFSET_TO_LMEMLO(offset)	((uint32_t)(offset))
+
+typedef struct ctf_enum
+{
+  uint32_t cte_name;		/* Reference to name in string table.  */
+  int cte_value;		/* Value associated with this name.  */
+} ctf_enum_t;
+
+/* The ctf_archive is a collection of ctf_file_t's stored together. The format
+   is suitable for mmap()ing: this control structure merely describes the
+   mmap()ed archive (and overlaps the first few bytes of it), hence the
+   greater care taken with integral types.  All CTF files in an archive
+   must have the same data model.  (This is not validated.)
+
+   All integers in this structure are stored in little-endian byte order.
+
+   The code relies on the fact that everything in this header is a uint64_t
+   and thus the header needs no padding (in particular, that no padding is
+   needed between ctfa_ctfs and the unnamed ctfa_archive_modent array
+   that follows it).  */
+
+#define CTFA_MAGIC 0x8b47f2a4d7623eeb	/* Random.  */
+struct ctf_archive
+{
+  /* Magic number.  (In loaded files, overwritten with the file size
+     so ctf_arc_close() knows how much to munmap()).  */
+  uint64_t ctfa_magic;
+
+  /* CTF data model.  */
+  uint64_t ctfa_model;
+
+  /* Number of CTF files in the archive.  */
+  uint64_t ctfa_nfiles;
+
+  /* Offset of the name table.  */
+  uint64_t ctfa_names;
+
+  /* Offset of the CTF table.  Each element starts with a size (a uint64_t
+     in network byte order) then a ctf_file_t of that size.  */
+  uint64_t ctfa_ctfs;
+};
+
+/* An array of ctfa_nnamed of this structure lies at
+   ctf_archive[ctf_archive->ctfa_modents] and gives the ctfa_ctfs or
+   ctfa_names-relative offsets of each name or ctf_file_t.  */
+
+typedef struct ctf_archive_modent
+{
+  uint64_t name_offset;
+  uint64_t ctf_offset;
+} ctf_archive_modent_t;
+
+#ifdef	__cplusplus
+}
+#endif
+
+#endif				/* _CTF_H */
-- 
1.8.3.1

[PATCH,RFC 2/3] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

-gtLEVEL is used to request CTF debug information and also to specify how much
CTF debug information.

gcc/ChangeLog:

	* common.opt: Add CTF debug info options.
	* doc/invoke.texi: Document the CTF debug info options.
	* flag-types.h (enum ctf_debug_info_levels): New enum.
	* opts.c (set_ctf_debug_level): New Function.
	(common_handle_option):  Handle the new CTF debug info options.

---
 gcc/ChangeLog       |  8 ++++++++
 gcc/common.opt      |  9 +++++++++
 gcc/doc/invoke.texi | 16 ++++++++++++++++
 gcc/flag-types.h    | 13 +++++++++++++
 gcc/opts.c          | 26 ++++++++++++++++++++++++++
 5 files changed, 72 insertions(+)

diff --git a/gcc/common.opt b/gcc/common.opt
index d342c4f..2b5cfb8 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -125,6 +125,11 @@ enum debug_info_levels debug_info_level = DINFO_LEVEL_NONE
 Variable
 bool use_gnu_debug_info_extensions
 
+; Level of CTF debugging information we are producing.  See flag-types.h
+; for the definitions of the different possible levels.
+Variable
+enum ctf_debug_info_levels ctf_debug_info_level = CTFINFO_LEVEL_NONE
+
 ; Original value of maximum field alignment in bytes, specified via
 ; -fpack-struct=<value>.
 Variable
@@ -2995,6 +3000,10 @@ gcolumn-info
 Common Driver Var(debug_column_info,1) Init(1)
 Record DW_AT_decl_column and DW_AT_call_column in DWARF.
 
+gt
+Common Driver RejectNegative JoinedOrMissing
+Generate CTF debug information at default level.
+
 gdwarf
 Common Driver JoinedOrMissing Negative(gdwarf-)
 Generate debug information in default version of DWARF format.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 5e3e887..2ed39c0 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -373,6 +373,7 @@ Objective-C and Objective-C++ Dialects}.
 @item Debugging Options
 @xref{Debugging Options,,Options for Debugging Your Program}.
 @gccoptlist{-g  -g@var{level}  -gdwarf  -gdwarf-@var{version} @gol
+-gt  -gt@var{level} @gol
 -ggdb  -grecord-gcc-switches  -gno-record-gcc-switches @gol
 -gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf @gol
 -gas-loc-support  -gno-as-loc-support @gol
@@ -7766,6 +7767,21 @@ other DWARF-related options such as
 @option{-fno-dwarf2-cfi-asm}) retain a reference to DWARF Version 2
 in their names, but apply to all currently-supported versions of DWARF.
 
+@item -gt
+@itemx -gt@var{level}
+@opindex gt
+Request CTF debug information and use level to specify how much CTF debug
+information should be produced.  If -gt is specified without a value for level,
+the default level of CTF debug information is 2.
+
+Level 0 produces no CTF debug information at all.  Thus, -gt0 negates -gt.
+
+Level 1 produces CTF information for tracebacks only.  This includes callsite
+information, but does not include type information.
+
+Level 2 produces type information for entities (functions, data objects etc.)
+at file-scope or global-scope only.
+
 @item -gstabs
 @opindex gstabs
 Produce debugging information in stabs format (if that is supported),
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index a210328..61a1432 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -105,6 +105,19 @@ enum dwarf_gnat_encodings
 				       Emit GNAT encodings for the rest.  */
 };
 
+/* CTF debug info levels.
+   CTF debug info levels are untied with DWARF debug info levels because CTF
+   may co-exist with DWARF.  */
+enum ctf_debug_info_levels
+{
+  CTFINFO_LEVEL_NONE = 0,     /* Write no CTF debug info.  */
+  CTFINFO_LEVEL_TERSE = 1,    /* Write CTF information to support tracebacks
+				 only.  Not Implemented.  */
+  CTFINFO_LEVEL_NORMAL = 2    /* Write CTF type information for all entities
+				 (functions, data objects, variables etc.)
+				 at file-scope or global-scope only.  */
+};
+
 /* Enumerate Objective-c instance variable visibility settings. */
 
 enum ivar_visibility
diff --git a/gcc/opts.c b/gcc/opts.c
index 879ab17..dfd646b 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -195,6 +195,8 @@ static void set_debug_level (enum debug_info_type type, int extended,
 			     const char *arg, struct gcc_options *opts,
 			     struct gcc_options *opts_set,
 			     location_t loc);
+static void set_ctf_debug_level (const char *arg,
+				 struct gcc_options *opts, location_t loc);
 static void set_fast_math_flags (struct gcc_options *opts, int set);
 static void decode_d_option (const char *arg, struct gcc_options *opts,
 			     location_t loc, diagnostic_context *dc);
@@ -2689,6 +2691,10 @@ common_handle_option (struct gcc_options *opts,
       opts->x_flag_stack_usage_info = value != 0;
       break;
 
+    case OPT_gt:
+      set_ctf_debug_level (arg, opts, loc);
+      break;
+
     case OPT_g:
       set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENSIONS, arg, opts, opts_set,
                        loc);
@@ -3011,6 +3017,26 @@ set_debug_level (enum debug_info_type type, int extended, const char *arg,
     }
 }
 
+/* Handle a debug output -gt switch for options OPTS.  */
+static void
+set_ctf_debug_level (const char *arg,
+		     struct gcc_options *opts, location_t loc)
+{
+  /* CTF debug flag without a level defaults to level 2.  */
+  if (*arg == '\0')
+    opts->x_ctf_debug_info_level = CTFINFO_LEVEL_NORMAL;
+  else
+    {
+      int argval = integral_argument (arg);
+      if (argval == -1)
+	error_at (loc, "unrecognized ctf debug output level %qs", arg);
+      else if (argval > CTFINFO_LEVEL_NORMAL)
+	error_at (loc, "ctf debug output level %qs is too high", arg);
+      else
+	opts->x_ctf_debug_info_level = (enum ctf_debug_info_levels) argval;
+    }
+}
+
 /* Arrange to dump core on error for diagnostic context DC.  (The
    regular error message is still printed first, except in the case of
    abort ().)  */
-- 
1.8.3.1

[PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Background :
CTF is the Compact Ansi-C Type Format. It is a format designed to express some
characteristics (specifically Type information) of the data types in a C
program. CTF format is compact and fast; It was originally designed for
use-cases like dynamic tracing, online in-application debugging among others.

A patch to the binutils mailing list to add libctf is currently under
review (https://sourceware.org/ml/binutils/2019-05/msg00154.html and
	https://sourceware.org/ml/binutils/2019-05/msg00212.html).  libctf
provides means to create, update, read and manipulate CTF information.

This GCC patch set is preliminary work and the purpose is to gather comments and
feedback about CTF support in GCC.

(For technical introduction into the CTF format, the CTF header or
https://sourceware.org/ml/binutils/2019-04/msg00277.html will be useful.)

Project Details :
The project aims to add the support for CTF in the GNU toolchain. Adding CTF
support in the GNU toolchain will help the community in developing and 
converging the tools and use-cases where a light-weight debug format is needed.

De-duplication is a key aspect of the CTF format which ensures its compactness.
A parallel effort is ongoing to support de-duplication of CTF types at the
link-time.

In phase 1, we are making the compiler, linker and the debugger (GDB) capable
of handling the CTF format.

CTF format, in its present form, does not have callsite information.  We are
working on this as well. Once the CTF format extensions are agreed upon, the 
-gt1 option (see below) will begin to take form, in phase 2 of the project.

GCC RFC patch set :
Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
GIMPLE frontend.
Patch 2 and Patch 3 set up the framework for CTF support in GCC :
-- Patch 2 adds the new command line option for generating CTF. CTF generation
   is enabled in the compiler by specifying an explicit -gt or
   -gtLEVEL[LEVEL=1,2] :
    
    -gtLEVEL

    This is used to request CTF debug information and to specify how much CTF
    debug information, LEVEL[=0,1,2] can be specified. If -gt is specified
    (with no LEVEL), the default value of LEVEL is 2.

    -gt0 (Level 0) produces no CTF debug information at all. Thus, -gt0
    negates -gt.

    -gt1 (Level 1) produces CTF information for tracebacks only. This includes
    CTF callsite information, but does not include type information for other
    entities.

    -gt2 (Level 2) produces type information for entities (functions, variables
    etc.) at file-scope or global-scope only. This level of information can be
    used by dynamic tracers like DTrace.

--  Patch 3 adds the CTF debug hooks and initializes them if the required 
    user-level options are specified. 
    CTF debug hooks are wrappers around the DWARF debug hooks.

One of the main high-level design requirements that is relevant in the context
of the current GCC patch set is that - CTF and DWARF must be able to co-exist.
A user may want CTF debug information in isolation or with other debug formats.
A .ctf section is small and unlike other debug sections, ideally should not
need to be stripped out of the binary/executable.

High-level proposed plan (phase 1) :
In the next few patches, the functionality to generate contents of the CTF
section (.ctf) for a single compilation unit will be added.
Once CTF generation for a single compilation unit stabilizes, LTO and CTF
generation will be looked at.

Feedback and suggestions welcome.

Thanks

Indu Bhagat (3):
  Add new function lang_GNU_GIMPLE
  Add CTF command line options : -gtLEVEL
  Create CTF debug hooks

 gcc/ChangeLog                                   |  24 ++
 gcc/Makefile.in                                 |   3 +
 gcc/common.opt                                  |   9 +
 gcc/ctfout.c                                    | 171 +++++++++
 gcc/ctfout.h                                    |  41 +++
 gcc/debug.h                                     |   4 +
 gcc/doc/invoke.texi                             |  16 +
 gcc/flag-types.h                                |  13 +
 gcc/gengtype.c                                  |   4 +-
 gcc/langhooks.c                                 |   9 +
 gcc/langhooks.h                                 |   1 +
 gcc/opts.c                                      |  26 ++
 gcc/testsuite/ChangeLog                         |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c          |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp          |  41 +++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c       |   7 +
 gcc/toplev.c                                    |  24 ++
 include/ChangeLog                               |   4 +
 include/ctf.h                                   | 471 ++++++++++++++++++++++++
 20 files changed, 890 insertions(+), 2 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

-- 
1.8.3.1

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Richard Biener]

On Mon, May 20, 2019 at 7:56 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
> Background :
> CTF is the Compact Ansi-C Type Format. It is a format designed to express some
> characteristics (specifically Type information) of the data types in a C
> program. CTF format is compact and fast; It was originally designed for
> use-cases like dynamic tracing, online in-application debugging among others.
>
> A patch to the binutils mailing list to add libctf is currently under
> review (https://sourceware.org/ml/binutils/2019-05/msg00154.html and
>         https://sourceware.org/ml/binutils/2019-05/msg00212.html).  libctf
> provides means to create, update, read and manipulate CTF information.
>
> This GCC patch set is preliminary work and the purpose is to gather comments and
> feedback about CTF support in GCC.
>
> (For technical introduction into the CTF format, the CTF header or
> https://sourceware.org/ml/binutils/2019-04/msg00277.html will be useful.)
>
> Project Details :
> The project aims to add the support for CTF in the GNU toolchain. Adding CTF
> support in the GNU toolchain will help the community in developing and
> converging the tools and use-cases where a light-weight debug format is needed.
>
> De-duplication is a key aspect of the CTF format which ensures its compactness.
> A parallel effort is ongoing to support de-duplication of CTF types at the
> link-time.
>
> In phase 1, we are making the compiler, linker and the debugger (GDB) capable
> of handling the CTF format.
>
> CTF format, in its present form, does not have callsite information.  We are
> working on this as well. Once the CTF format extensions are agreed upon, the
> -gt1 option (see below) will begin to take form, in phase 2 of the project.
>
> GCC RFC patch set :
> Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
> GIMPLE frontend.

I don't think you should need this - the GIMPLE "frontend" is intended for
unit testing only, I wouldn't like it to be exposed more.

> Patch 2 and Patch 3 set up the framework for CTF support in GCC :
> -- Patch 2 adds the new command line option for generating CTF. CTF generation
>    is enabled in the compiler by specifying an explicit -gt or
>    -gtLEVEL[LEVEL=1,2] :
>
>     -gtLEVEL
>
>     This is used to request CTF debug information and to specify how much CTF
>     debug information, LEVEL[=0,1,2] can be specified. If -gt is specified
>     (with no LEVEL), the default value of LEVEL is 2.
>
>     -gt0 (Level 0) produces no CTF debug information at all. Thus, -gt0
>     negates -gt.
>
>     -gt1 (Level 1) produces CTF information for tracebacks only. This includes
>     CTF callsite information, but does not include type information for other
>     entities.
>
>     -gt2 (Level 2) produces type information for entities (functions, variables
>     etc.) at file-scope or global-scope only. This level of information can be
>     used by dynamic tracers like DTrace.
>
> --  Patch 3 adds the CTF debug hooks and initializes them if the required
>     user-level options are specified.
>     CTF debug hooks are wrappers around the DWARF debug hooks.
>
> One of the main high-level design requirements that is relevant in the context
> of the current GCC patch set is that - CTF and DWARF must be able to co-exist.
> A user may want CTF debug information in isolation or with other debug formats.
> A .ctf section is small and unlike other debug sections, ideally should not
> need to be stripped out of the binary/executable.
>
> High-level proposed plan (phase 1) :
> In the next few patches, the functionality to generate contents of the CTF
> section (.ctf) for a single compilation unit will be added.
> Once CTF generation for a single compilation unit stabilizes, LTO and CTF
> generation will be looked at.
>
> Feedback and suggestions welcome.

You probably got asked this question multiple times already, but,
can CTF information be generated from DWARF instead?

The meaning of the CTF acronym suggests that there's nothing
like locations, registers, etc. but just a representation of the
types?

Generally we are trying to walk away from supporting multiple
debug info formats because that gets in the way of being
more precise from the frontend side.  Since DWARF is the
defacto standard, extensible and with a rich feature set the
line of thinking is that other formats (like STABS) can be
generated by "post-processing" DWARF.  Such
post-processing could happen on the object files or
on the GCC internal DWARF data structures by
providing alternate output routines.  That is, the mid-term
design goal is to make DWARF generation the "API"
for GCC frontends to use when creating high-level
debug information rather than trying to abstract from
the debuginfo format via the current debug-hooks or
the other way around via language-hooks.

Richard.

> Thanks
>
> Indu Bhagat (3):
>   Add new function lang_GNU_GIMPLE
>   Add CTF command line options : -gtLEVEL
>   Create CTF debug hooks
>
>  gcc/ChangeLog                                   |  24 ++
>  gcc/Makefile.in                                 |   3 +
>  gcc/common.opt                                  |   9 +
>  gcc/ctfout.c                                    | 171 +++++++++
>  gcc/ctfout.h                                    |  41 +++
>  gcc/debug.h                                     |   4 +
>  gcc/doc/invoke.texi                             |  16 +
>  gcc/flag-types.h                                |  13 +
>  gcc/gengtype.c                                  |   4 +-
>  gcc/langhooks.c                                 |   9 +
>  gcc/langhooks.h                                 |   1 +
>  gcc/opts.c                                      |  26 ++
>  gcc/testsuite/ChangeLog                         |   7 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c          |   6 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf.exp          |  41 +++
>  gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c       |   7 +
>  gcc/toplev.c                                    |  24 ++
>  include/ChangeLog                               |   4 +
>  include/ctf.h                                   | 471 ++++++++++++++++++++++++
>  20 files changed, 890 insertions(+), 2 deletions(-)
>  create mode 100644 gcc/ctfout.c
>  create mode 100644 gcc/ctfout.h
>  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
>  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
>  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
>  create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
>  create mode 100644 include/ctf.h
>
> --
> 1.8.3.1
>

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Thanks for your feedback. Comments inline.


On 05/21/2019 03:28 AM, Richard Biener wrote:
>> GCC RFC patch set :
>> Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
>> GIMPLE frontend.
> I don't think you should need this - the GIMPLE "frontend" is intended for
> unit testing only, I wouldn't like it to be exposed more.

When using -gt with -flto, I would still like the CTF hooks to be initialized
so that CTF can be generated when -flto is used. So the check in toplev.c is
done to allow only C and GNU GIMPLE.  I am fine with doing a string compare
with the language.hooks string if you suggest to go that way.

>
>> One of the main high-level design requirements that is relevant in the context
>> of the current GCC patch set is that - CTF and DWARF must be able to co-exist.
>> A user may want CTF debug information in isolation or with other debug formats.
>> A .ctf section is small and unlike other debug sections, ideally should not
>> need to be stripped out of the binary/executable.
>>
>> High-level proposed plan (phase 1) :
>> In the next few patches, the functionality to generate contents of the CTF
>> section (.ctf) for a single compilation unit will be added.
>> Once CTF generation for a single compilation unit stabilizes, LTO and CTF
>> generation will be looked at.
>>
>> Feedback and suggestions welcome.
> You probably got asked this question multiple times already, but,
> can CTF information be generated from DWARF instead?

Yes and No :) And that is indeed one of the motivation of the project - to
allow CTF generation where it's most suited aka the toolchain.

There do exist utilties for generation of CTF from DWARF. For example, one of
them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
to transform DWARF generated by the compiler into CTF.

A dependency of an external conversion utility for "post-processing" DWARF
offline poses several problems:

1. Deployment problems: the converter should be distributed and integrated in
    the build system of the program.  This, on occasions, can be intrusive.  For
    example, in terms of dependencies: the dwarf2ctf converter depends on
    libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
    compress the CTF information) and libctf (which both reads and writes the
    CTF data).

2. Performance problems: the conversion from DWARF to CTF can take a long time,
    especially in big programs such as the Linux kernel.

3. Maintainability problems: the converter should be maintained in order to
    reflect potential changes in the DWARF generated by the compiler.

4. Adoption problem: it is difficult for applications to adopt the usage of
    CTF, even if it happens to provide what they need, since it would require to
    write a conversion utility or integrate DTrace's.


>
> The meaning of the CTF acronym suggests that there's nothing
> like locations, registers, etc. but just a representation of the
> types?

Yes. CTF is simply put Type information; no locations, registers etc.

>
> Generally we are trying to walk away from supporting multiple
> debug info formats because that gets in the way of being
> more precise from the frontend side.  Since DWARF is the

With regard to whether the support for CTF imposes infeasible or distinct
requirements on the frontend - it does not appear to be the case (I have
been working on CTF generation in GCC for a SINGLE compilation unit; More see
below). I agree that CTF debug information generation should ideally not impose
additional requirements on the frontend.

> defacto standard, extensible and with a rich feature set the
> line of thinking is that other formats (like STABS) can be
> generated by "post-processing" DWARF.  Such
> post-processing could happen on the object files or
> on the GCC internal DWARF data structures by
> providing alternate output routines.  That is, the mid-term
> design goal is to make DWARF generation the "API"
> for GCC frontends to use when creating high-level
> debug information rather than trying to abstract from
> the debuginfo format via the current debug-hooks or
> the other way around via language-hooks.

I am not sure if I understood the last part very well, so I will state how CTF
generation is intended to work. Does the following fit the design goal you
state ?

( Caveat : I have been working on the functionality to generate CTF for a SINGLE
   compilation unit. LTO bits remain. )

So far, there are no additional requirements on the frontend side. CTF hooks
are wrappers around DWARF debug hooks (much like go-dump hooks, and vms dbg
hooks).  We did notice that GCC does not have the infrastructure to register or
enlist multiple debug hooks; and now from your comments it is clear that this
is by design. Thanks for clarifying that.

Having said that, I use CTF hooks to go from TREE --> update CTF internal
structures or output CTF routines depending on the hook (e.g., type_decl or
finish respectively), rather than changing the dwarf* files with CTF APIs. The
CTF debug hooks relay control to the DWARF debug hooks at an appropriate point.
TREE input references to the CTF debug hooks are readonly in the context of CTF
generation.

The CTF debug information is kept in a CTF container distinct from the frontend
structures.  HashMaps are used to avoid generation of duplicate CTF and to
book-keep the generated CTF.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Richard Biener]

On Wed, May 22, 2019 at 12:34 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
> Thanks for your feedback. Comments inline.
>
>
> On 05/21/2019 03:28 AM, Richard Biener wrote:
>
> GCC RFC patch set :
> Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
> GIMPLE frontend.
>
> I don't think you should need this - the GIMPLE "frontend" is intended for
> unit testing only, I wouldn't like it to be exposed more.
>
> When using -gt with -flto, I would still like the CTF hooks to be initialized
> so that CTF can be generated when -flto is used. So the check in toplev.c is
> done to allow only C and GNU GIMPLE.  I am fine with doing a string compare
> with the language.hooks string if you suggest to go that way.

I see.  Note that starting with GCC 8 type and declaration debug information
(the only you are interested in it seems) for LTO is generated at compile-time
while LTO ("GNU GIMPLE") only amends it with location information.  So
I believe you do not want to emit CTF from LTO but from the compilation-stage
driven by the appropriate language frontend which is where debug information
for types and decls is emitted.

>
> One of the main high-level design requirements that is relevant in the context
> of the current GCC patch set is that - CTF and DWARF must be able to co-exist.
> A user may want CTF debug information in isolation or with other debug formats.
> A .ctf section is small and unlike other debug sections, ideally should not
> need to be stripped out of the binary/executable.
>
> High-level proposed plan (phase 1) :
> In the next few patches, the functionality to generate contents of the CTF
> section (.ctf) for a single compilation unit will be added.
> Once CTF generation for a single compilation unit stabilizes, LTO and CTF
> generation will be looked at.
>
> Feedback and suggestions welcome.
>
> You probably got asked this question multiple times already, but,
> can CTF information be generated from DWARF instead?
>
> Yes and No :) And that is indeed one of the motivation of the project - to
> allow CTF generation where it's most suited aka the toolchain.
>
> There do exist utilties for generation of CTF from DWARF. For example, one of
> them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
> to transform DWARF generated by the compiler into CTF.
>
> A dependency of an external conversion utility for "post-processing" DWARF
> offline poses several problems:
>
> 1. Deployment problems: the converter should be distributed and integrated in
>    the build system of the program.  This, on occasions, can be intrusive.  For
>    example, in terms of dependencies: the dwarf2ctf converter depends on
>    libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
>    compress the CTF information) and libctf (which both reads and writes the
>    CTF data).
>
> 2. Performance problems: the conversion from DWARF to CTF can take a long time,
>    especially in big programs such as the Linux kernel.
>
> 3. Maintainability problems: the converter should be maintained in order to
>    reflect potential changes in the DWARF generated by the compiler.
>
> 4. Adoption problem: it is difficult for applications to adopt the usage of
>    CTF, even if it happens to provide what they need, since it would require to
>    write a conversion utility or integrate DTrace's.
>
>
>
> The meaning of the CTF acronym suggests that there's nothing
> like locations, registers, etc. but just a representation of the
> types?
>
> Yes. CTF is simply put Type information; no locations, registers etc.
>
>
> Generally we are trying to walk away from supporting multiple
> debug info formats because that gets in the way of being
> more precise from the frontend side.  Since DWARF is the
>
>
> With regard to whether the support for CTF imposes infeasible or distinct
> requirements on the frontend - it does not appear to be the case (I have
> been working on CTF generation in GCC for a SINGLE compilation unit; More see
> below). I agree that CTF debug information generation should ideally not impose
> additional requirements on the frontend.
>
> defacto standard, extensible and with a rich feature set the
> line of thinking is that other formats (like STABS) can be
> generated by "post-processing" DWARF.  Such
> post-processing could happen on the object files or
> on the GCC internal DWARF data structures by
> providing alternate output routines.  That is, the mid-term
> design goal is to make DWARF generation the "API"
> for GCC frontends to use when creating high-level
> debug information rather than trying to abstract from
> the debuginfo format via the current debug-hooks or
> the other way around via language-hooks.
>
> I am not sure if I understood the last part very well, so I will state how CTF
> generation is intended to work. Does the following fit the design goal you
> state ?
>
> ( Caveat : I have been working on the functionality to generate CTF for a SINGLE
>   compilation unit. LTO bits remain. )
>
> So far, there are no additional requirements on the frontend side. CTF hooks
> are wrappers around DWARF debug hooks (much like go-dump hooks, and vms dbg
> hooks).  We did notice that GCC does not have the infrastructure to register or
> enlist multiple debug hooks; and now from your comments it is clear that this
> is by design. Thanks for clarifying that.
>
> Having said that, I use CTF hooks to go from TREE --> update CTF internal
> structures or output CTF routines depending on the hook (e.g., type_decl or
> finish respectively), rather than changing the dwarf* files with CTF APIs. The
> CTF debug hooks relay control to the DWARF debug hooks at an appropriate point.
> TREE input references to the CTF debug hooks are readonly in the context of CTF
> generation.
>
> The CTF debug information is kept in a CTF container distinct from the frontend
> structures.  HashMaps are used to avoid generation of duplicate CTF and to
> book-keep the generated CTF.

OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
data structures?  That is, in my view CTF should be emitted by
dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
is generated from).  It would be nice to avoid extra bookkeeping data structures
for CTF since those of DWARF should include all necessary information already.

Btw, do I read the CTF document posted to the binutils list (but not
cross-referenced
here :/) correctly in that you only want CTF debug for objects defined
in the file and
type information for the types refered to from that?  At
dwarf2out_early_finish time
it isn't fully known which symbols will end up being emitted (and with
LTO you only
would know at link time).  It seems to me that linker support to garbage collect
unused entries would be the way to go forward (probably easy for the
declarations
but not so for the types)?

Richard.

>

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

On 05/22/2019 02:04 AM, Richard Biener wrote:
>> The CTF debug information is kept in a CTF container distinct from the frontend
>> structures.  HashMaps are used to avoid generation of duplicate CTF and to
>> book-keep the generated CTF.
> OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
> data structures?  That is, in my view CTF should be emitted by
> dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
> is generated from).  It would be nice to avoid extra bookkeeping data structures
> for CTF since those of DWARF should include all necessary information already.

CTF format has some characteristics which make it necessary to "pre-process"
the generated CTF data before asm'ing out into a section. E.g. few cases of why
"pre-processing" CTF is required before asm'ing out :
  1. CTF types do need to be emitted in "some" order :
     CTF types can have references to other CTF types. This consequently implies
     that the referenced type must appear BEFORE the referring type.
  2. CTF preamble holds offsets to the various subsections - function info,
     variables, data types and CTF string table. To calculate the offsets, the
     compiler needs to know the size in bytes of these sub-sections.  CTF
     representation for some types like structures, functions, enums have
     variable length of bytes trailing them (depending on the defintion of the
     type).
  3. CTF variable entries need to be placed in the order of the names.

Because of some of these "features" of the CTF format, the compiler does need
to do a transition from runtime CTF generated data --> CTF binary data format
for a clean and readable code.

So, I think the needs are different enough to vouch for an implementation
segregated from dwarf* codebase.


> Btw, do I read the CTF document posted to the binutils list (but not
> cross-referenced
> here :/) correctly in that you only want CTF debug for objects defined
> in the file and
> type information for the types refered to from that?  At

Yes. CTF is emitted for types at file-scope and global-scope only.  Types, vars
at function-scope should be skipped.

> dwarf2out_early_finish time
> it isn't fully known which symbols will end up being emitted (and with
> LTO you only
> would know at link time).

In nutshell, I am processing all decl at early_global_decl () time except
TYPE_DECL (Similar to DWARF, based on the thinking that if they are required
they will be reached at via other DECL).
In addition, I process all decl at type_decl () time except function-scope,
no-name decl, builtins.

Currently, it does look like CTF for possibly to-be-omitted symbols will be
generated... I assume even DWARF needs to handle this case. Can you point me to
how DWARF does this ?

> It seems to me that linker support to garbage collect
> unused entries would be the way to go forward (probably easy for the
> declarations
> but not so for the types)?

Hmm, garbage collecting unused types in linker - Let me get back to you on
this. It does not look easy. Decl should be doable though.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Richard Biener]

On Thu, May 23, 2019 at 10:31 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
>
>
> On 05/22/2019 02:04 AM, Richard Biener wrote:
>
> The CTF debug information is kept in a CTF container distinct from the frontend
> structures.  HashMaps are used to avoid generation of duplicate CTF and to
> book-keep the generated CTF.
>
> OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
> data structures?  That is, in my view CTF should be emitted by
> dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
> is generated from).  It would be nice to avoid extra bookkeeping data structures
> for CTF since those of DWARF should include all necessary information already.
>
> CTF format has some characteristics which make it necessary to "pre-process"
> the generated CTF data before asm'ing out into a section. E.g. few cases of why
> "pre-processing" CTF is required before asm'ing out :
>  1. CTF types do need to be emitted in "some" order :
>     CTF types can have references to other CTF types. This consequently implies
>     that the referenced type must appear BEFORE the referring type.
>  2. CTF preamble holds offsets to the various subsections - function info,
>     variables, data types and CTF string table. To calculate the offsets, the
>     compiler needs to know the size in bytes of these sub-sections.  CTF
>     representation for some types like structures, functions, enums have
>     variable length of bytes trailing them (depending on the defintion of the
>     type).
>  3. CTF variable entries need to be placed in the order of the names.
>
> Because of some of these "features" of the CTF format, the compiler does need
> to do a transition from runtime CTF generated data --> CTF binary data format
> for a clean and readable code.
>
> So, I think the needs are different enough to vouch for an implementation
> segregated from dwarf* codebase.
>
>
> Btw, do I read the CTF document posted to the binutils list (but not
> cross-referenced
> here :/) correctly in that you only want CTF debug for objects defined
> in the file and
> type information for the types refered to from that?  At
>
> Yes. CTF is emitted for types at file-scope and global-scope only.  Types, vars
> at function-scope should be skipped.
>
> dwarf2out_early_finish time
> it isn't fully known which symbols will end up being emitted (and with
> LTO you only
> would know at link time).
>
> In nutshell, I am processing all decl at early_global_decl () time except
> TYPE_DECL (Similar to DWARF, based on the thinking that if they are required
> they will be reached at via other DECL).
> In addition, I process all decl at type_decl () time except function-scope,
> no-name decl, builtins.
>
> Currently, it does look like CTF for possibly to-be-omitted symbols will be
> generated... I assume even DWARF needs to handle this case. Can you point me to
> how DWARF does this ?

It emits the debug information.  DWARF outputs a representation of the source,
not only emitted objects.  We prune some "unused" bits if the user prefers us
to do that but we do not omit information on types or decls that are used in
the source but later eventually optimized away.

> It seems to me that linker support to garbage collect
> unused entries would be the way to go forward (probably easy for the
> declarations
> but not so for the types)?
>
> Hmm, garbage collecting unused types in linker - Let me get back to you on
> this. It does not look easy. Decl should be doable though.

For example DWARF has something like type units that can be refered
to via hashes.  GCC can output those into separate sections and I can
envision outputting separate debug (CTF) sections for each declaration.
The linker could then merge sections for declarations that survived
and pick up all referenced type sections.  Restrictions on ordering
for CTF may make this a bit difficult though, essentially forcing a
separate intermediate "unlinked" format and the linker regenerating
the final one.  OTOH CTF probably simply concatenates data from
different CUs?

Richard.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Michael Matz]

Hello,

On Thu, 23 May 2019, Indu Bhagat wrote:

> > OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
> > data structures?  That is, in my view CTF should be emitted by
> > dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
> > is generated from).  It would be nice to avoid extra bookkeeping data
> > structures
> > for CTF since those of DWARF should include all necessary information
> > already.
> 
> CTF format has some characteristics which make it necessary to "pre-process"
> the generated CTF data before asm'ing out into a section. E.g. few cases of
> why "pre-processing" CTF is required before asm'ing out :
>  1. CTF types do need to be emitted in "some" order :
>     CTF types can have references to other CTF types. This consequently
>     implies
>     that the referenced type must appear BEFORE the referring type.
>  2. CTF preamble holds offsets to the various subsections - function info,
>     variables, data types and CTF string table. To calculate the offsets, the
>     compiler needs to know the size in bytes of these sub-sections.  CTF
>     representation for some types like structures, functions, enums have
>     variable length of bytes trailing them (depending on the defintion of the
>     type).
>  3. CTF variable entries need to be placed in the order of the names.
> 
> Because of some of these "features" of the CTF format, the compiler does 
> need to do a transition from runtime CTF generated data --> CTF binary 
> data format for a clean and readable code.

Sure, but this whole process could still be triggered from within 
dwarf2out_early_finish, by walking the DWARF tree (instead of getting fed 
types and decls via hooks) and generating the appropriate CTF data 
structures.  (It's just a possibility, it might end up uglier that using 
GCC trees)

Imagine a world where debug hooks wouldn't exist (which is where we would 
like to end up in a far away future), how would you then add CTF debug 
info to the compiler (assuming it already emits DWARF)?  You would hook 
yourself either into the DWARF routines that currently are fed the 
entities or you would hook yourself into somewhere late in the pipeline 
where the DWARF debug info is complete and you would generate CTF from 
that.

> So, I think the needs are different enough to vouch for an implementation
> segregated from dwarf* codebase.

Of course.  We are merely discussing of where the triggering of processing 
starts: debug hooks, or something like:

dwarf2out_early_finish() {
  ...
  if (ctf)
    ctf_emit();
}

(and then in addition if the current DWARF info would be the source of CTF 
info, or if it'd be whatever the compiler gives you as trees)

The thing is, with debug hooks you'd have to invent a scheme of stacking 
hooks on top of each other (because we want to generate DWARF and CTF from 
the same compilation).  That seems like a wasted effort when our wish is 
for the hooks to go away alltogether.


Ciao,
Michael.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Jakub Jelinek]

On Tue, May 21, 2019 at 03:44:47PM -0700, Indu Bhagat wrote:
> Yes and No :) And that is indeed one of the motivation of the project - to
> allow CTF generation where it's most suited aka the toolchain.
> 
> There do exist utilties for generation of CTF from DWARF. For example, one of
> them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
> to transform DWARF generated by the compiler into CTF.

So, if there is a conversion utility, why don't we just change gcc so that
if some new option is passed on the gcc driver link line, then that
post-processing utility will be invoked too?

> A dependency of an external conversion utility for "post-processing" DWARF
> offline poses several problems:
> 
> 1. Deployment problems: the converter should be distributed and integrated in
>    the build system of the program.  This, on occasions, can be intrusive.  For
>    example, in terms of dependencies: the dwarf2ctf converter depends on
>    libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
>    compress the CTF information) and libctf (which both reads and writes the
>    CTF data).

I don't see this as a problem.

> 2. Performance problems: the conversion from DWARF to CTF can take a long time,
>    especially in big programs such as the Linux kernel.

So optimize it?  Linux kernel certainly doesn't have extra large debug
information, compared to other projects.

> 3. Maintainability problems: the converter should be maintained in order to
>    reflect potential changes in the DWARF generated by the compiler.

If you integrate the support into GCC, then it will need to be maintained
there as well, I bet it will be more work than on the conversion utility.

	Jakub

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Jakub Jelinek]

On Fri, May 24, 2019 at 03:24:34PM +0200, Jakub Jelinek wrote:
> On Tue, May 21, 2019 at 03:44:47PM -0700, Indu Bhagat wrote:
> > Yes and No :) And that is indeed one of the motivation of the project - to
> > allow CTF generation where it's most suited aka the toolchain.
> > 
> > There do exist utilties for generation of CTF from DWARF. For example, one of
> > them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
> > to transform DWARF generated by the compiler into CTF.
> 
> So, if there is a conversion utility, why don't we just change gcc so that
> if some new option is passed on the gcc driver link line, then that
> post-processing utility will be invoked too?

It could be even written as a linker plugin.

	Jakub

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Hi Michael,

On 05/24/2019 06:04 AM, Michael Matz wrote:
> Hello,
>
> On Thu, 23 May 2019, Indu Bhagat wrote:
>
>>> OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
>>> data structures?  That is, in my view CTF should be emitted by
>>> dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
>>> is generated from).  It would be nice to avoid extra bookkeeping data
>>> structures
>>> for CTF since those of DWARF should include all necessary information
>>> already.
>> CTF format has some characteristics which make it necessary to "pre-process"
>> the generated CTF data before asm'ing out into a section. E.g. few cases of
>> why "pre-processing" CTF is required before asm'ing out :
>>   1. CTF types do need to be emitted in "some" order :
>>      CTF types can have references to other CTF types. This consequently
>>      implies
>>      that the referenced type must appear BEFORE the referring type.
>>   2. CTF preamble holds offsets to the various subsections - function info,
>>      variables, data types and CTF string table. To calculate the offsets, the
>>      compiler needs to know the size in bytes of these sub-sections.  CTF
>>      representation for some types like structures, functions, enums have
>>      variable length of bytes trailing them (depending on the defintion of the
>>      type).
>>   3. CTF variable entries need to be placed in the order of the names.
>>
>> Because of some of these "features" of the CTF format, the compiler does
>> need to do a transition from runtime CTF generated data --> CTF binary
>> data format for a clean and readable code.
> Sure, but this whole process could still be triggered from within
> dwarf2out_early_finish, by walking the DWARF tree (instead of getting fed
> types and decls via hooks) and generating the appropriate CTF data
> structures.  (It's just a possibility, it might end up uglier that using
> GCC trees)

I think not only is the code messier, but it's also wasted effort if user only
wants to generate CTF.

> Imagine a world where debug hooks wouldn't exist (which is where we would
> like to end up in a far away future), how would you then add CTF debug
> info to the compiler (assuming it already emits DWARF)?  You would hook
> yourself either into the DWARF routines that currently are fed the
> entities or you would hook yourself into somewhere late in the pipeline
> where the DWARF debug info is complete and you would generate CTF from
> that.
>
>> So, I think the needs are different enough to vouch for an implementation
>> segregated from dwarf* codebase.
> Of course.  We are merely discussing of where the triggering of processing
> starts: debug hooks, or something like:
>
> dwarf2out_early_finish() {
>    ...
>    if (ctf)
>      ctf_emit();
> }
>
> (and then in addition if the current DWARF info would be the source of CTF
> info, or if it'd be whatever the compiler gives you as trees)
>
> The thing is, with debug hooks you'd have to invent a scheme of stacking
> hooks on top of each other (because we want to generate DWARF and CTF from
> the same compilation).  That seems like a wasted effort when our wish is
> for the hooks to go away alltogether.
>
When the debug hooks go away, the functionality can be folded in. Much like
above, the ctf proposed implementation will do :

ctf_early_global_decl (tree decl)
{
   ctf_decl (decl);

   real_debug_hooks->early_global_decl (decl);
}

These ctf_* debug hooks wrappers are as lean as shown above.

I do understand now that if debug hooks are destined to go away, all the
implementation which wraps debug hooks (go dump hooks, vms debug hooks,
and now the proposed ctf debug hooks) will need some merging. But to generate
CTF, I think working on type or decl instead of DWARF dies to is a better
implementation because if user wants only CTF, no DWARF trees need to be
created.

This way we keep DWARF and CTF generation independent of each other (as the
user may want either one of these or both).

> Ciao,
> Michael.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Richard Biener]

On Mon, May 27, 2019 at 8:12 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
> Hi Michael,
>
> On 05/24/2019 06:04 AM, Michael Matz wrote:
> > Hello,
> >
> > On Thu, 23 May 2019, Indu Bhagat wrote:
> >
> >>> OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
> >>> data structures?  That is, in my view CTF should be emitted by
> >>> dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
> >>> is generated from).  It would be nice to avoid extra bookkeeping data
> >>> structures
> >>> for CTF since those of DWARF should include all necessary information
> >>> already.
> >> CTF format has some characteristics which make it necessary to "pre-process"
> >> the generated CTF data before asm'ing out into a section. E.g. few cases of
> >> why "pre-processing" CTF is required before asm'ing out :
> >>   1. CTF types do need to be emitted in "some" order :
> >>      CTF types can have references to other CTF types. This consequently
> >>      implies
> >>      that the referenced type must appear BEFORE the referring type.
> >>   2. CTF preamble holds offsets to the various subsections - function info,
> >>      variables, data types and CTF string table. To calculate the offsets, the
> >>      compiler needs to know the size in bytes of these sub-sections.  CTF
> >>      representation for some types like structures, functions, enums have
> >>      variable length of bytes trailing them (depending on the defintion of the
> >>      type).
> >>   3. CTF variable entries need to be placed in the order of the names.
> >>
> >> Because of some of these "features" of the CTF format, the compiler does
> >> need to do a transition from runtime CTF generated data --> CTF binary
> >> data format for a clean and readable code.
> > Sure, but this whole process could still be triggered from within
> > dwarf2out_early_finish, by walking the DWARF tree (instead of getting fed
> > types and decls via hooks) and generating the appropriate CTF data
> > structures.  (It's just a possibility, it might end up uglier that using
> > GCC trees)
>
> I think not only is the code messier, but it's also wasted effort if user only
> wants to generate CTF.
>
> > Imagine a world where debug hooks wouldn't exist (which is where we would
> > like to end up in a far away future), how would you then add CTF debug
> > info to the compiler (assuming it already emits DWARF)?  You would hook
> > yourself either into the DWARF routines that currently are fed the
> > entities or you would hook yourself into somewhere late in the pipeline
> > where the DWARF debug info is complete and you would generate CTF from
> > that.
> >
> >> So, I think the needs are different enough to vouch for an implementation
> >> segregated from dwarf* codebase.
> > Of course.  We are merely discussing of where the triggering of processing
> > starts: debug hooks, or something like:
> >
> > dwarf2out_early_finish() {
> >    ...
> >    if (ctf)
> >      ctf_emit();
> > }
> >
> > (and then in addition if the current DWARF info would be the source of CTF
> > info, or if it'd be whatever the compiler gives you as trees)
> >
> > The thing is, with debug hooks you'd have to invent a scheme of stacking
> > hooks on top of each other (because we want to generate DWARF and CTF from
> > the same compilation).  That seems like a wasted effort when our wish is
> > for the hooks to go away alltogether.
> >
> When the debug hooks go away, the functionality can be folded in. Much like
> above, the ctf proposed implementation will do :
>
> ctf_early_global_decl (tree decl)
> {
>    ctf_decl (decl);
>
>    real_debug_hooks->early_global_decl (decl);
> }
>
> These ctf_* debug hooks wrappers are as lean as shown above.
>
> I do understand now that if debug hooks are destined to go away, all the
> implementation which wraps debug hooks (go dump hooks, vms debug hooks,
> and now the proposed ctf debug hooks) will need some merging. But to generate
> CTF, I think working on type or decl instead of DWARF dies to is a better
> implementation because if user wants only CTF, no DWARF trees need to be
> created.
>
> This way we keep DWARF and CTF generation independent of each other (as the
> user may want either one of these or both).

The user currently can't have both DWARF and STABS either.  That things like
godump uses debug hooks is just (convenient?) abuse.

In the end frontends will not call sth like dwarf2out_decl but maybe
gen_subroutine_die () or gen_template_die ().  So how do you expect
the "wrapping" to work there?

I understand you want CTF for "actually emitted" decls so I propose you
instead hook into the symtab code which would end up calling the
early_global_decl debug hook.  But please don't add new debug hook
users.

Richard.

> > Ciao,
> > Michael.
>

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

On 05/24/2019 02:26 AM, Richard Biener wrote:
>> Currently, it does look like CTF for possibly to-be-omitted symbols will be
>> generated... I assume even DWARF needs to handle this case. Can you point me to
>> how DWARF does this ?
> It emits the debug information.  DWARF outputs a representation of the source,
> not only emitted objects.  We prune some "unused" bits if the user prefers us
> to do that but we do not omit information on types or decls that are used in
> the source but later eventually optimized away.
>
>> It seems to me that linker support to garbage collect
>> unused entries would be the way to go forward (probably easy for the
>> declarations
>> but not so for the types)?
>>
>> Hmm, garbage collecting unused types in linker - Let me get back to you on
>> this. It does not look easy. Decl should be doable though.
> For example DWARF has something like type units that can be refered
> to via hashes.  GCC can output those into separate sections and I can
> envision outputting separate debug (CTF) sections for each declaration.
> The linker could then merge sections for declarations that survived
> and pick up all referenced type sections.  Restrictions on ordering
> for CTF may make this a bit difficult though, essentially forcing a
> separate intermediate "unlinked" format and the linker regenerating
> the final one.  OTOH CTF probably simply concatenates data from
> different CUs?

Yes, I cannot see this happening with CTF easily without some format changes.

At link-time, there needs to be de-duplication of CTF types across CUs. This
linker component needs work at this time, although we do have a working
prototype.

Regarding the type units in DWARF, are the shared/common types duplicated
across the type units ? If not duplicated, how are the referenced types
maintained/denoted across type units ?

Thanks!
Indu

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

On 05/29/2019 12:15 AM, Richard Biener wrote:
>>> Of course.  We are merely discussing of where the triggering of processing
>>> starts: debug hooks, or something like:
>>>
>>> dwarf2out_early_finish() {
>>>     ...
>>>     if (ctf)
>>>       ctf_emit();
>>> }
>>>
>>> (and then in addition if the current DWARF info would be the source of CTF
>>> info, or if it'd be whatever the compiler gives you as trees)
>>>
>>> The thing is, with debug hooks you'd have to invent a scheme of stacking
>>> hooks on top of each other (because we want to generate DWARF and CTF from
>>> the same compilation).  That seems like a wasted effort when our wish is
>>> for the hooks to go away alltogether.
>>>
>> When the debug hooks go away, the functionality can be folded in. Much like
>> above, the ctf proposed implementation will do :
>>
>> ctf_early_global_decl (tree decl)
>> {
>>     ctf_decl (decl);
>>
>>     real_debug_hooks->early_global_decl (decl);
>> }
>>
>> These ctf_* debug hooks wrappers are as lean as shown above.
>>
>> I do understand now that if debug hooks are destined to go away, all the
>> implementation which wraps debug hooks (go dump hooks, vms debug hooks,
>> and now the proposed ctf debug hooks) will need some merging. But to generate
>> CTF, I think working on type or decl instead of DWARF dies to is a better
>> implementation because if user wants only CTF, no DWARF trees need to be
>> created.
>>
>> This way we keep DWARF and CTF generation independent of each other (as the
>> user may want either one of these or both).
> The user currently can't have both DWARF and STABS either.  That things like
> godump uses debug hooks is just (convenient?) abuse.
>
> In the end frontends will not call sth like dwarf2out_decl but maybe
> gen_subroutine_die () or gen_template_die ().  So how do you expect
> the "wrapping" to work there?
>
> I understand you want CTF for "actually emitted" decls so I propose you
> instead hook into the symtab code which would end up calling the
> early_global_decl debug hook.  But please don't add new debug hook
> users.

OK.

Will I need to tap both the callsites of the early_global_decl () debug hook ? :
   1. symbol_table::finalize_compilation_unit () in cgraphunit.c
   2. rest_of_decl_compilation () in passes.c
Or is the last one for something specific to godump debug hooks and C++ ?

I guess the above will take care of the CTF generation bit. For emission,
something similar should be done because DWARF hooks will not be initialized if
DWARF debuginfo is not requested by the user. So I cannot have the CTF emission
code in the dwarf2out*finish () debug hooks as suggested earlier.

Curious to know how the current debug hook users like dbx debug hooks will be
taken care of in the future design ? Is it just the wrapping/stacking of debug
hooks that's problematic and not the clean instances like dbx debug hooks ?

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Richard Biener]

On Sat, Jun 1, 2019 at 2:14 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
>
>
> On 05/29/2019 12:15 AM, Richard Biener wrote:
>
> Of course.  We are merely discussing of where the triggering of processing
> starts: debug hooks, or something like:
>
> dwarf2out_early_finish() {
>    ...
>    if (ctf)
>      ctf_emit();
> }
>
> (and then in addition if the current DWARF info would be the source of CTF
> info, or if it'd be whatever the compiler gives you as trees)
>
> The thing is, with debug hooks you'd have to invent a scheme of stacking
> hooks on top of each other (because we want to generate DWARF and CTF from
> the same compilation).  That seems like a wasted effort when our wish is
> for the hooks to go away alltogether.
>
> When the debug hooks go away, the functionality can be folded in. Much like
> above, the ctf proposed implementation will do :
>
> ctf_early_global_decl (tree decl)
> {
>    ctf_decl (decl);
>
>    real_debug_hooks->early_global_decl (decl);
> }
>
> These ctf_* debug hooks wrappers are as lean as shown above.
>
> I do understand now that if debug hooks are destined to go away, all the
> implementation which wraps debug hooks (go dump hooks, vms debug hooks,
> and now the proposed ctf debug hooks) will need some merging. But to generate
> CTF, I think working on type or decl instead of DWARF dies to is a better
> implementation because if user wants only CTF, no DWARF trees need to be
> created.
>
> This way we keep DWARF and CTF generation independent of each other (as the
> user may want either one of these or both).
>
> The user currently can't have both DWARF and STABS either.  That things like
> godump uses debug hooks is just (convenient?) abuse.
>
> In the end frontends will not call sth like dwarf2out_decl but maybe
> gen_subroutine_die () or gen_template_die ().  So how do you expect
> the "wrapping" to work there?
>
> I understand you want CTF for "actually emitted" decls so I propose you
> instead hook into the symtab code which would end up calling the
> early_global_decl debug hook.  But please don't add new debug hook
> users.
>
> OK.
>
> Will I need to tap both the callsites of the early_global_decl () debug hook ? :
>   1. symbol_table::finalize_compilation_unit () in cgraphunit.c
>   2. rest_of_decl_compilation () in passes.c
> Or is the last one for something specific to godump debug hooks and C++ ?

You need to handle both, one is for variables and one for functions.

> I guess the above will take care of the CTF generation bit. For emission,
> something similar should be done because DWARF hooks will not be initialized if
> DWARF debuginfo is not requested by the user. So I cannot have the CTF emission
> code in the dwarf2out*finish () debug hooks as suggested earlier.

True.

> Curious to know how the current debug hook users like dbx debug hooks will be
> taken care of in the future design ? Is it just the wrapping/stacking of debug
> hooks that's problematic and not the clean instances like dbx debug hooks ?

dbx format support will go away, it's unmaintained and not up to the
task handling
modern languages.

Richard.

>

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Richard Biener]

On Fri, May 31, 2019 at 9:17 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
>
>
> On 05/24/2019 02:26 AM, Richard Biener wrote:
>
> Currently, it does look like CTF for possibly to-be-omitted symbols will be
> generated... I assume even DWARF needs to handle this case. Can you point me to
> how DWARF does this ?
>
> It emits the debug information.  DWARF outputs a representation of the source,
> not only emitted objects.  We prune some "unused" bits if the user prefers us
> to do that but we do not omit information on types or decls that are used in
> the source but later eventually optimized away.
>
> It seems to me that linker support to garbage collect
> unused entries would be the way to go forward (probably easy for the
> declarations
> but not so for the types)?
>
> Hmm, garbage collecting unused types in linker - Let me get back to you on
> this. It does not look easy. Decl should be doable though.
>
> For example DWARF has something like type units that can be refered
> to via hashes.  GCC can output those into separate sections and I can
> envision outputting separate debug (CTF) sections for each declaration.
> The linker could then merge sections for declarations that survived
> and pick up all referenced type sections.  Restrictions on ordering
> for CTF may make this a bit difficult though, essentially forcing a
> separate intermediate "unlinked" format and the linker regenerating
> the final one.  OTOH CTF probably simply concatenates data from
> different CUs?
>
> Yes, I cannot see this happening with CTF easily without some format changes.
>
> At link-time, there needs to be de-duplication of CTF types across CUs. This
> linker component needs work at this time, although we do have a working
> prototype.
>
> Regarding the type units in DWARF, are the shared/common types duplicated
> across the type units ? If not duplicated, how are the referenced types
> maintained/denoted across type units ?

I think type units can refer to each other just fine but the linker will not
split them up further, just throw away duplicates.

Richard.

> Thanks!
> Indu
>

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Hello,


On 05/24/2019 06:24 AM, Jakub Jelinek wrote:
> On Tue, May 21, 2019 at 03:44:47PM -0700, Indu Bhagat wrote:
>> Yes and No :) And that is indeed one of the motivation of the project - to
>> allow CTF generation where it's most suited aka the toolchain.
>>
>> There do exist utilties for generation of CTF from DWARF. For example, one of
>> them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
>> to transform DWARF generated by the compiler into CTF.
> So, if there is a conversion utility, why don't we just change gcc so that
> if some new option is passed on the gcc driver link line, then that
> post-processing utility will be invoked too?

Performing DWARF to CTF conversion at link-time is not recommended because for
any large project, it is time-consuming to generate, read in, and convert DWARF
to CTF in a post-processing step. These costs are prohibitive enough and affect
CTF adoption.

Data for some projects below.

>> A dependency of an external conversion utility for "post-processing" DWARF
>> offline poses several problems:
>>
>> 1. Deployment problems: the converter should be distributed and integrated in
>>     the build system of the program.  This, on occasions, can be intrusive.  For
>>     example, in terms of dependencies: the dwarf2ctf converter depends on
>>     libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
>>     compress the CTF information) and libctf (which both reads and writes the
>>     CTF data).
> I don't see this as a problem.
>
>> 2. Performance problems: the conversion from DWARF to CTF can take a long time,
>>     especially in big programs such as the Linux kernel.
> So optimize it?  Linux kernel certainly doesn't have extra large debug
> information, compared to other projects.

DWARF generation only for it to be transformed into something compact like CTF
and eventually to be stripped away is not an efficient workflow; Further, only
a subset of this is amenable to "optimizations", i.e. compile-time generation
of DWARF is unavoidable in this flow.

For example, for the Linux kernel, the debuginfo in the object files is ~9GiB
(The object files without debuginfo is ~1.5GiB). That's a 500% increase in the
space requirements. Generating the DWARF at compile time adds 20% to the compile
time. Next, reading in this DWARF using libdwarf from elftuils takes ~1 min 10
seconds. Next, a conversion utility like dwarf2ctf will then need to perform
de-duplication, on this rather voluminous DWARF, as you see, for the purpose on
hand.

Not just the kernel, for another relatively large internal application we
measured a 19% and 23% increase in compile time for generating DWARF for
a randomly chosen subsets of C and C++ components respectively. For the entire
application, it already takes an order of 3-4 hours to perform a parallel build.
The space requirements for building the entire application with -g also increase
to about 5.5x.

These numbers I state above are on sufficiently beefy platforms. On the
"wimpier" platforms or single-threaded builds for that matter, the increase in
time costs to generate DWARF and then use an external utility (at link-time
or offline) just aggravates the pain for developers, who may need to do even
multiple builds a day.

So, in summary, this increase in build times and space costs is noticeable
(even prohibitive in some cases) and does impact a subset of kernel and
application developers who need to use CTF. Above all, effectively it limits
CTF adoption.

>
>> 3. Maintainability problems: the converter should be maintained in order to
>>     reflect potential changes in the DWARF generated by the compiler.
> If you integrate the support into GCC, then it will need to be maintained
> there as well, I bet it will be more work than on the conversion utility.

Yes, work is needed on all components of the toolchain to integrate CTF
generation. We are working on the compiler, linker and debugger (GDB) currently.

We are committing to maintain the CTF generation in GCC, just like we are doing
for Binutils, and will be doing for GDB.

>
> 	Jakub