Re: [PATCH 02/19] include: new header ctf-api.h

[Nick Alcock <nick.alcock@oracle.com>]

On 2 May 2019, Nick Clifton verbalised:

> Hi Nick,
>
>> This non-installed header is the means by which libctf consumers
>> communicate with libctf.
>
> By "non-installed" do you mean that it would not be placed into /usr/include ?

Yes. Mostly because I assumed that the bar was higher for installed
headers than for non-installed ones, and there are no users of libctf
yet.

> If so, then how do consumers know how to communicate with libctf, given that
> they might be compiled without access to the ctf sources ?

At the moment, libctf is an automatically-synched copy of the upstream
libdtrace-ctf project. Our aim is to eventually make binutils the
upstream, whereupon we can drop libdtrace-ctf entirely, and turn this
into an installed header :)

(If we do make it an installed header, we want to set up things like
symbol versioning first. libdtrace-ctf already has all of that.)

>> +/* Clients can open one or more CTF containers and obtain a pointer to an
>> +   opaque ctf_file_t.  Types are identified by an opaque ctf_id_t token.
>> +   They can also open or create read-only archives of CTF containers in a
>> +   ctf_archive_t.
>
> Are CTF archives just arbitrary collections of CTF containers or is there more
> to them than that ?

They are usually arbitrary collections of *related* CTF containers (with
associated names): e.g. a parent container and a bunch of children that
reuse types from the parent.

The format is given in ctf-impl.h: it's just an mmappable archive which
isn't tar or cpio and so doesn't have to deal with the infinite horrors
of tar or cpio versioning. :) One feature it does have that tar and cpio
don't is that, like zip files, the compression happens at the level of
individual archive members, via the CTF_F_COMPRESS flag bit in the
header.

This means that you can elect to not compress CTF files that are already
"small enough" (say, smaller than a page), whereupon they can get read
via mmap() directly out of the CTF archive, with no copying or
decompression overhead. CTF files with parents are often very small:
looking at Linux kernel 5.0, of 2938 CTF files in the archive I
generated in my most recent test build, all but 188 are under 4096
bytes: but they all have as a parent the 1.2MiB shared CTF file that
contains types shared across kernel modules (in a non-kernel context,
the parent container will usually contain types shared across TUs).

(Nearly 2000 of them are under 512 bytes long. By this stage just the
gzip header is bloating the thing up substantially, and you gain nothing
from compression of things so small.)