From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 94819 invoked by alias); 9 Oct 2019 07:41:25 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Received: (qmail 94811 invoked by uid 89); 9 Oct 2019 07:41:25 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.5 required=5.0 tests=AWL,BAYES_00,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=ham version=3.3.1 spammy=H*i:sk:088f530, H*f:sk:088f530, online X-HELO: mail-lf1-f65.google.com Received: from mail-lf1-f65.google.com (HELO mail-lf1-f65.google.com) (209.85.167.65) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 09 Oct 2019 07:41:23 +0000 Received: by mail-lf1-f65.google.com with SMTP id 72so870898lfh.6 for ; Wed, 09 Oct 2019 00:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3A5F9PXUzcwHbU1FW5+V1NITMPz78CFZ8x+AzXNR4ts=; b=KEH+yBvyn4EnBJ6zFnRfbbaHRgS2cVAvHfPmQCs7FBCM4B48E8rkuOdmorTER6GZ+0 woGcS+rpF8vIHNu/1Z0QYFv+64ki2m0ivuxKoGeA3jzCeq7ljlzvZ3IQTO3X1QEdN1J8 U2XianOqjUKrGSq4+QXVUZwzfujkGUFk7icu1v9jBfvP6l3csiKBzt1lrhT79/MXG/wa ede0WNF0dy2+a56P439YFCYClo2EoYQ+PbUxMcRvx2BwDI3GJi3VGGFS/pCZaVuP3ec/ cxoKl+pj17xawSJpYBgwcaVKm2OYLA1ETAiEdhrxmGbadr07bmoKVanTLE+CtWs/pxaO LLow== MIME-Version: 1.0 References: <6ade3f1a-08a5-f8df-c53a-c98d60b0f12a@oracle.com> <088f530c-959b-ce76-5b87-8889e82f62df@oracle.com> In-Reply-To: <088f530c-959b-ce76-5b87-8889e82f62df@oracle.com> From: Richard Biener Date: Wed, 09 Oct 2019 07:43:00 -0000 Message-ID: Subject: Re: Type representation in CTF and DWARF To: Indu Bhagat Cc: Pedro Alves , gcc-patches , mark@klomp.org Content-Type: multipart/mixed; boundary="00000000000033c0d105947569a6" X-IsSubscribed: yes X-SW-Source: 2019-10/txt/msg00586.txt.bz2 --00000000000033c0d105947569a6 Content-Type: text/plain; charset="UTF-8" Content-length: 5759 On Wed, Oct 9, 2019 at 7:26 AM Indu Bhagat wrote: > > > > On 10/08/2019 08:37 AM, Pedro Alves wrote: > > On 10/4/19 8:23 PM, Indu Bhagat wrote: > >> Hello, > >> > >> At GNU Tools Cauldron this year, some folks were curious to know more on how > >> the "type representation" in CTF compares vis-a-vis DWARF. > > I was one of those, and I brought this up to Jose, after your > > presentation. Glad to see the follow up! Thanks much for this. > > > > In your Cauldron presentation we saw CTF compared to full blown DWARF > > as justification for CTF, > > Hmm. And I thought I made the effort reqd to clarify my position that comparing > full-blown DWARF sizes to type-only CTF section sizes is not appropriate, let > alone to not use as a justification for CTF. My intention to show those numbers was > only to give some perspective to users curious to know the sizes of CTF debug > info (as generated by dwarf2ctf) because these sections will ideally be not > stripped out of shipped binaries. > > The justification for CTF is and will remain - a compact, faster debug format > for type information and support some online debugging use-cases (like > backtraces) in future. > > > but I was more interested in a comparison between > > CTF and a DWARF subset containing exactly only what you have available in > > CTF. Because if DWARF with everything-you-don't-need stripped out > > is in the same ballpark, then I am puzzled on why add/maintain a new > > Debug format, with all the duplication of effort that entails going > > forward. > > I shared some numbers on this in the previous emails in this thread. I thought > comparing DWARF's de-duplication-amenable offering (using > -fdebug-types-section) will be useful in this context. > > For binaries compiled with -fdebug-types-section -gdwarf-4, here is some data. > The CTF sections are generated with dwarf2ctf because CTF link-time de-dup is > being worked on currently. The end result of link-time CTF de-dup is expected > to be at par with these .ctf section sizes. > > The .ctf section sizes below include the CTF string table (.debug_str is > excluded from the calculations however): > > (coreutils-0.22) > .debug_info(D1) | .debug_abbrev(D2) | .debug_str | .debug_types(D3) | .ctf (uncompressed) | ratio (.ctf/(D1+D2+D3)) > ls 109806 | 18876 | 22042 | 12413 | 26240 | 0.18 > pwd 27902 | 7914 | 10851 | 5753 | 13929 | 0.33 > groups 26920 | 8173 | 10674 | 5070 | 13378 | 0.33 > > (emacs-26.3) > .debug_info(D1) | .debug_abbrev(D2) | .debug_str | .debug_types(D3) | .ctf (uncompressed) | ratio (.ctf/(D1+D2+D3)) > emacs 3755083 | 202926 | 431926 | 143462 | 273910 | 0.06 > > > It is not easy to get an estimate of 'DWARF with everything-you-don't-need > stripped out'. At this time, I don't know of an easy way to make this comparison > more meaningful. Any suggestions ? There's a mechanism to get type (and decl - I suppose CTF also contains debug info for function declarations not only its type?) info as part of early debug generation. The attached "hack" simply mangles dwarf2out to output this early info as the only debug info (only verified on a small .c file). We still have things like file, line and column numbers for entities (not sure if CTF has those). It should be possible to "hide" the hack behind a -gdwarf-like-ctf or similar. I guess -g0.5 isn't desirable and we've taken both -g0 and -g1 already... (and -g1 doesn't include types but just decls). Richard. > > Also, it's my understanding that the current CTF format doesn't yet > > support C++, Vector registers, etc., maybe other things, so if DWARF > > was sufficient for your needs, then in the long run it sounds like > > a better option to me, as then you wouldn't have to extend CTF _and_ > > DWARF whenever some feature is needed. > > Yes, CTF does not support C++ at this time. To cover all of C (including > GNU C extensions), we need to add representation for things like Vector type, > non IEEE float etc. (somewhat infrequently occurring constructs) > > The issue is not that DWARF cannot represent the required type information. > DWARF is voluminous and secondly, the current workflow to get to CTF from > source programs without direct toolchain support is tiresome and lengthy. > > For current and future users of CTF, having the support for the format in the > toolchain is the best way to promote adoption and enhance community experience. > > > Maybe it would make sense to work on integrating CTF into the DWARF > > standard itself, not sure? > > > > I was also curious on your plans for adding unwinding support to CTF, > > while the kernel (the main CTF user, IIUC), already has plans to > > use its own unwinding format (ORC)? > > Kernel's unwinding format (ORC) helps generate backtrace with function > identifiers. For some (ORCL) internal customers, the requirement is to go beyond > that and support input arg values. The requirement there is to generate > backtraces in a fast way, without relying on DWARF. > > > So with all those questions, I came out of the presentation > > thinking that I could not really justify CTF if I were asked to. > > Thanks for discussing this openly. I believe there are other GCC > maintainers who are undecided as well :) > > I hope I have answered some of your concerns. > > > (Side note: the Cauldron page is missing slides for your > > presentation, so I couldn't go and recheck some things > > mentioned above.) > > > > Thanks, > > Pedro Alves > > > I mailed the organizers my slides. They should be online soon. > > Thanks > --00000000000033c0d105947569a6 Content-Type: application/octet-stream; name=p Content-Disposition: attachment; filename=p Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_k1iyrvjb0 Content-length: 2522 SW5kZXg6IGdjYy9kd2FyZjJvdXQuYwo9PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 Ci0tLSBnY2MvZHdhcmYyb3V0LmMJKHJldmlzaW9uIDI3Njc1MikKKysrIGdj Yy9kd2FyZjJvdXQuYwkod29ya2luZyBjb3B5KQpAQCAtMzE0ODcsNiArMzE0 ODcsNyBAQCBkd2FyZjJvdXRfZmluaXNoIChjb25zdCBjaGFyICpmaWxlbmFt ZSkKICAgICAgIHByaW50X2RpZSAoY29tcF91bml0X2RpZSAoKSwgZHVtcF9m aWxlKTsKICAgICB9CiAKKyAgcmV0dXJuOwogICAvKiBJbml0aWFsaXplIHNl Y3Rpb25zIGFuZCBsYWJlbHMgdXNlZCBmb3IgYWN0dWFsIGFzc2VtYmxlciBv dXRwdXQuICAqLwogICB1bnNpZ25lZCBnZW5lcmF0aW9uID0gaW5pdF9zZWN0 aW9uc19hbmRfbGFiZWxzIChmYWxzZSk7CiAKQEAgLTMyMjY0LDYgKzMyMjY1 LDcgQEAgZHdhcmYyb3V0X2Vhcmx5X2ZpbmlzaCAoY29uc3QgY2hhciAqZmls ZQogICAgICAgcHJpbnRfZGllIChjb21wX3VuaXRfZGllICgpLCBkdW1wX2Zp bGUpOwogICAgIH0KIAorI2lmIDAKICAgLyogRG8gbm90IGdlbmVyYXRlIERX QVJGIGFzc2VtYmxlciBub3cgd2hlbiBub3QgcHJvZHVjaW5nIExUTyBieXRl Y29kZS4gICovCiAgIGlmICgoIWZsYWdfZ2VuZXJhdGVfbHRvICYmICFmbGFn X2dlbmVyYXRlX29mZmxvYWQpCiAgICAgICAvKiBGSVhNRTogRGlzYWJsZSBk ZWJ1ZyBpbmZvIGdlbmVyYXRpb24gZm9yIChQRS0pQ09GRiB0YXJnZXRzIHNp bmNlIHRoZQpAQCAtMzIyNzEsMTIgKzMyMjczLDEzIEBAIGR3YXJmMm91dF9l YXJseV9maW5pc2ggKGNvbnN0IGNoYXIgKmZpbGUKIAkgbGliaWJlcnR5IGlz IG5vdCBpbXBsZW1lbnRlZCBmb3IgdGhlbSB5ZXQuICAqLwogICAgICAgfHwg VEFSR0VUX1BFQ09GRiB8fCBUQVJHRVRfQ09GRikKICAgICByZXR1cm47Cisj ZW5kaWYKIAogICAvKiBOb3cgYXMgd2UgYXJlIGdvaW5nIHRvIG91dHB1dCBm b3IgTFRPIGluaXRpYWxpemUgc2VjdGlvbnMgYW5kIGxhYmVscwogICAgICB0 byB0aGUgTFRPIHZhcmlhbnRzLiAgV2UgZG9uJ3QgbmVlZCBhIHJhbmRvbS1z ZWVkIHBvc3RmaXggYXMgb3RoZXIKICAgICAgTFRPIHNlY3Rpb25zIGFzIGxp bmtpbmcgdGhlIExUTyBkZWJ1ZyBzZWN0aW9ucyBpbnRvIG9uZSBpbiBhIHBh cnRpYWwKICAgICAgbGluayBpcyBmaW5lLiAgKi8KLSAgaW5pdF9zZWN0aW9u c19hbmRfbGFiZWxzICh0cnVlKTsKKyAgaW5pdF9zZWN0aW9uc19hbmRfbGFi ZWxzIChmYWxzZSAvKnRydWUqLyk7CiAKICAgLyogVGhlIG91dHB1dCBiZWxv dyBpcyBtb2RlbGVkIGFmdGVyIGR3YXJmMm91dF9maW5pc2ggd2l0aCBhbGwK ICAgICAgbG9jYXRpb24gcmVsYXRlZCBvdXRwdXQgcmVtb3ZlZCBhbmQgc29t ZSBMVE8gc3BlY2lmaWMgY2hhbmdlcy4KQEAgLTMyMzc5LDcgKzMyMzgyLDgg QEAgZHdhcmYyb3V0X2Vhcmx5X2ZpbmlzaCAoY29uc3QgY2hhciAqZmlsZQog ICAvKiBFbWl0IGEgc2tlbGV0b24gZGVidWdfbGluZSBzZWN0aW9uLiAgKi8K ICAgc3dpdGNoX3RvX3NlY3Rpb24gKGRlYnVnX2xpbmVfc2VjdGlvbik7CiAg IEFTTV9PVVRQVVRfTEFCRUwgKGFzbV9vdXRfZmlsZSwgZGVidWdfbGluZV9z ZWN0aW9uX2xhYmVsKTsKLSAgb3V0cHV0X2xpbmVfaW5mbyAodHJ1ZSk7Cisg IGlmICghIG91dHB1dF9hc21fbGluZV9kZWJ1Z19pbmZvICgpKQorICAgIG91 dHB1dF9saW5lX2luZm8gKHRydWUpOwogCiAgIC8qIElmIHdlIGVtaXR0ZWQg YW55IGluZGlyZWN0IHN0cmluZ3MsIG91dHB1dCB0aGUgc3RyaW5nIHRhYmxl IHRvby4gICovCiAgIGlmIChkZWJ1Z19zdHJfaGFzaCB8fCBza2VsZXRvbl9k ZWJ1Z19zdHJfaGFzaCkK --00000000000033c0d105947569a6--