Re: [PATCH,RFC,V3 0/5] Support for CTF in GCC

[Nix <nix@esperi.org.uk>]

On 5 Jul 2019, Jakub Jelinek outgrape:

> On Fri, Jul 05, 2019 at 07:28:12PM +0100, Nix wrote:
>> > What makes it superior to DWARF stripped down to the above feature set?
>> 
>> Increased compactness. DWARF fundamentally trades off compactness in
>> favour of its regular structure, which makes it easier to parse (but not
>> easier to interpret) but very hard to make it much smaller than it is
>> now. Where DWARF uses word-sized and larger entities for everything, CTF
>> packs everything much more tightly -- and this is quite visible in the
>
> That is just not true, most of the data in DWARF are uleb128/sleb128
> encoded, or often even present just in the abbreviation table
> (DW_FORM_flag_present, DW_FORM_implicit_const), word-sized is typically only
> stuff that needs relocations (at assembly time and more importantly at link
> time).

Hm. I may have misread the spec.

The fact remains that DWARF is (in typical usage) both large and slow to
use: it is not entirely untrue to say that you can spot a DWARF consumer
because it takes ten seconds to start up. This may be something that can
be avoided with sufficiently clever implementations, but I've never seen
any such implementation and we don't appear to be approaching one
terribly fast :( meanwhile, in CTF we already have a working system that
can reduce multigigabyte DWARF input down to 6MiB of compressed CTF
loading in fractions of a second, though it is true that not all of that
input was global-scope type info, so a large portion of that
multigigabyte input would simply have been dropped and should not be
considered relevant. I'm not sure how to determine how much of the input
is type DIEs at global scope... (The 6MiB figure is slightly misleading,
too, since only 1439845 bytes of that is type data: the rest is mostly
compressed string table.)

Possibly sufficiently clever deduplication can do a similar scrunching
job for DWARF, but I note that what DWARF deduplication GCC did in
earlier releases has subsequently been removed because it never really
worked very well. (Having written code that deduplicates DWARF, I can
see why: it's a complex job when you just have C to think about. Doing
it for C++ as well must have made people's brains dribble out of their
ears).

Type signatures in DWARF 4 were supposed to provide this sort of thing,
too, but yet again the promise does not seem to have been borne out:
DWARF debuginfo remains immense and there is no discussion of leaving
unstripped binaries on production systems for the sake of continuous
tracing tools or introspection, because the debuginfo in those binaries
would still be many times the size of the binaries they relate to, and
obviously leaving it unstripped in that case is ridiculous. Meanwhile,
FreeBSD has a leg-up in continuous debugging because they generate (an
older form of) CTF for everything and deduplicate it, and it's small
enough that they can leave it linked into the binaries rather than
stripping it out, and tracers can and do use it. I'm trying to give us
all that advantage, while not leaving us tied to a format with as many
limitations as FreeBSD's CTF.


As a side note, I tried switching to ULEB128 for the representations of
unsigned integers in CTF a while back, but never even pushed it anywhere
because while it shrank the output a little, the compressed sizes
worsened noticeably, by about 10%, and we don't want to hurt the
compressed sizes any more than we do the uncompressed ones. I found this
quite annoying. So I'm not convinced that ULEB actually buys you
much of anything once compressors get into the mix.

Something similar happened when I tried to do clever things with string
tables last month, sharing common string suffixes, slicing strtabs up on
underscores and changes of case and replacing strings where beneficial
with offset tables pointing into the sliced-up pieces: the uncompressed
size shrank by about 50% and the compressed size grew by 20%... I found
this *very* annoying. :)

> For DWARF you also have various techniques at reducing size and optimizing
> redundancies away, look e.g. at the dwz utility.

... interesting! I'll be looking through this and seeing if any of it is
applicable to CTF as well, that's for sure.