[Bug default/21023] New: The abidw tool does not appear to read dwarf from .dwp files associated with executables

[Bug default/21023] New: The abidw tool does not appear to read dwarf from .dwp files associated with executables

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

            Bug ID: 21023
           Summary: The abidw tool does not appear to read dwarf from .dwp
                    files associated with executables
           Product: libabigail
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: default
          Assignee: dodji at redhat dot com
          Reporter: andrew.c.morrow at gmail dot com
                CC: libabigail at sourceware dot org
  Target Milestone: ---

$ abidw --version
1.0.rc7
I'm not actually sure if this is a bug in libabigail, or one of the supporting
libraries like libelf or libdw, but I'm starting here, since the end state I
want is that abidw works binaries built using DebugFission.

I would expect that abidw would search for DWARF info in the associated .dwp
file, however, it does not appear to do so.

On my Ubuntu 16.04 machine:

$ g++ --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

$ dwp --version
GNU dwp (GNU Binutils for Ubuntu) 2.26.1

$ cat hello_world.cpp
#include <cstdlib>
#include <iostream>

int main(int argc, char* argv[]) {
    std::cout << "Hello, World!\n" << argc;
    return EXIT_SUCCESS;
}

$ g++ -g ./hello_world.cpp -o hello_world
$ abidw ./hello_world | cksum
1150852500 53174

$ g++ -gsplit-dwarf -g ./hello_world.cpp -o hello_world
$ dwp -e ./hello_world
$ abidw ./hello_world | cksum
2156308181 1319

As you can see from the sizes, running abidw against the DebugFission version
results in severely truncated output. Running under strace appears to confirm
that the dwp file was never stat'ed or read:

$ strace -f -o trace abidw ./hello_world | cksum
2156308181 1319
$ grep -c dwp trace
0

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

--- Comment #2 from Mark Wielaard <mark at klomp dot org> ---
This probably is something that elfutils should transparently support for
libabigail. But currently elfutils libdw doesn't support GNU DebugFission
dwp/split-dwarf. Parts of it are in the process of being standardized now for
DWARF5. But the standard hasn't been finalized and code for elfutils is not yet
there (and after DWARF5 public beta was released some details were changed).

For now I would recommend not using -gsplit-dwarf and dwp. If you are looking
for smaller or "split" DWARF then yse using eu-strip -o and dwz to create debug
multi-files. That is supported.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #4 from Mark Wielaard <mark at klomp dot org> ---
(In reply to andrew.c.morrow from comment #3)
> Thank you for clarifying. It sounds from your description like this will
> eventually work automatically, which is great.

It will however take some work, it is not really trivial to support. Also while
GNU DebugFission is being standardized in the upcoming DWARFv5 there have been
various changes. So it isn't clear yet which tools will support which version.

> I have experimented with dwz as well, with the aim of reducing the size of
> our debug symbol packages, but so far without success:
> https://bugzilla.redhat.com/show_bug.cgi?id=1399866. If you have any
> additional thoughts on that ticket it would be greatly appreciated. I still
> don't have an explanation for why dwz cannot process the output of ld.gold.

I commented on that bug report. It looks like the linker is creating a bogus
.bss section, but there is not really enough information in the bug report to
know for sure.

> I'm also unclear on the intended relationship between dwp and dwz. When I
> tried running dwz on dwp files, my recollection is that it didn't work, at
> all. I'd presume that the process would be to collect the .dwo files
> associated with each binary into a .dwp file for that binary with the dwp
> tool, and then run dwz across all the dwp files to extract common symbols
> into a shared file, along with any other optimization/compression. I would
> then hope that abidw run on the binary would be able to present the full
> ABI, extracting information from both the associated .dwp file, and any
> common underling file produced by dwz.
> 
> Or, perhaps the functionality of dwz should be folded directly into dwp? The
> documentation on these tools and the intended direction for development is
> somewhat unclear to me.

The have been developed completely independently and I don't think anybody has
really thought about how to combine them. They also serve somewhat different
purposes. And not many tools support both DWARF debuginfo data/file variants.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #3 from andrew.c.morrow at gmail dot com ---
Hi Mark - 

Thank you for clarifying. It sounds from your description like this will
eventually work automatically, which is great.

My actual goal for using DebugFission was more for the build time improvements,
rather than debug symbol packaging. My use of libabigail/abidw is somewhat
unusual, and also focused on build time improvements:

https://sourceware.org/bugzilla/show_bug.cgi?id=18838#c4
http://scons-users.scons.narkive.com/E9RiycaY/replacement-for-sharedlibrarysignatureoverride

I had hoped to be able to use both DebugFission and abidw driven re-link
pruning simultaneously, but it appears that this will not work, for the moment.

I have experimented with dwz as well, with the aim of reducing the size of our
debug symbol packages, but so far without success:
https://bugzilla.redhat.com/show_bug.cgi?id=1399866. If you have any additional
thoughts on that ticket it would be greatly appreciated. I still don't have an
explanation for why dwz cannot process the output of ld.gold.

I'm also unclear on the intended relationship between dwp and dwz. When I tried
running dwz on dwp files, my recollection is that it didn't work, at all. I'd
presume that the process would be to collect the .dwo files associated with
each binary into a .dwp file for that binary with the dwp tool, and then run
dwz across all the dwp files to extract common symbols into a shared file,
along with any other optimization/compression. I would then hope that abidw run
on the binary would be able to present the full ABI, extracting information
from both the associated .dwp file, and any common underling file produced by
dwz.

Or, perhaps the functionality of dwz should be folded directly into dwp? The
documentation on these tools and the intended direction for development is
somewhat unclear to me.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #1 from andrew.c.morrow at gmail dot com ---
A gentle ping on this issue?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #5 from andrew.c.morrow at gmail dot com ---
This recently ended up on my radar again. Is there a ticket tracker for
elfutils where I could file an issue, or an already filed issue I should watch?
My understanding is that DWARF5 is now an issued standard, and it would be
great to see tools like abidw which are built above elfutils be able to reach
out to .dwo or .dwp files when -gsplit-dwarf (or its DWARF5 equivalent) is in
play.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #6 from Mark Wielaard <mark at klomp dot org> ---
elfutils 0.171 was recently released with support for DWARF5 and split-dwarf
(including GNU DebugFission as extension to older DWARF).

https://sourceware.org/ml/elfutils-devel/2018-q2/msg00141.html

For now .dwo files are supported, but not yet .dwp files.
(Also the binutils dwp tools doesn't work with DWARF5 split dwarf.)
It will take a bit more programming to also add .dwp support, but hopefully the
structure of the elfutils code is now so that this isn't major surgery.

Although most things should work as is, libabigail will need to use a few new
interfaces to properly handle split dwarf, iterate through CUs using the new
dwarf_get_units, and change from using offsets and dwarf_dieoffset/dwarf_offdie
to using the new dwarf_die_addr_die interface (because DIEs can now come from
multiple files, so using offsets alone isn't enough to identify a DIE).

While we update libabigail for this could you try out the new elfutils-0.171
tools to see if they work correctly in your setup?

And could you indicate how important .dwp support is? Would a libabigail that
just works with .dwo files be enough for now?

The interfaces should be generic enough that when .dwp support is added to
elfutils libdw, then they should work as is and libabigail shouldn't really
care whether the split dwarf units come from the .dwo files or the .dwp package
file.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #7 from andrew.c.morrow at gmail dot com ---
(In reply to Mark Wielaard from comment #6)
> elfutils 0.171 was recently released with support for DWARF5 and split-dwarf
> (including GNU DebugFission as extension to older DWARF).
> 
> https://sourceware.org/ml/elfutils-devel/2018-q2/msg00141.html

That is great news, thanks.

> 
> For now .dwo files are supported, but not yet .dwp files.
> (Also the binutils dwp tools doesn't work with DWARF5 split dwarf.)
> It will take a bit more programming to also add .dwp support, but hopefully
> the structure of the elfutils code is now so that this isn't major surgery.
>

Support for .dwo files is currently more important for me than .dwp.


> Although most things should work as is, libabigail will need to use a few
> new interfaces to properly handle split dwarf, iterate through CUs using the
> new dwarf_get_units, and change from using offsets and
> dwarf_dieoffset/dwarf_offdie to using the new dwarf_die_addr_die interface
> (because DIEs can now come from multiple files, so using offsets alone isn't
> enough to identify a DIE).
> 
> While we update libabigail for this could you try out the new elfutils-0.171
> tools to see if they work correctly in your setup?

A quick and dirty test suggests that it is working:

$ g++ -g -gsplit-dwarf -c -o hello_world.o ./hello_world.cpp -std=c++14
$ g++ -o hello_world hello_world.o
$ strace -o trace ~/opt/bin/eu-readelf --debug-dump=info+ --all ./hello_world
$ grep dwo trace
write(1, "           GNU_dwo_name         "..., 57) = 57
write(1, "           GNU_dwo_id           "..., 59) = 59
openat(AT_FDCWD, "/home/andrew/hello_world.dwo", O_RDONLY) = 4
readlink("/proc/27201/fd/4", "/home/andrew/hello_world.dwo", 4095) = 28
lstat("/home/andrew/hello_world.dwo", {st_mode=S_IFREG|0664, st_size=21696,
...}) = 0
write(1, "           GNU_dwo_id           "..., 59) = 59 

More complete testing will need to wait on the libabigail support.

> 
> And could you indicate how important .dwp support is? Would a libabigail
> that just works with .dwo files be enough for now?

I can definitely make use of a libabigail that only supports .dwo, perhaps
indefinitely. Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[tschwinge at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

Thomas Schwinge <tschwinge at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-12-06
             Status|UNCONFIRMED                 |NEW
                 CC|                            |tschwinge at sourceware dot org

--- Comment #8 from Thomas Schwinge <tschwinge at sourceware dot org> ---
So I wanted to use libabigail on GCC '-gsplit-dwarf' object files
('*.o'/'*.dwo'), and found that it doesn't work.  Then found this existing PR;
confirming that status is still as per Andrew's 2018-06-16 comment:
'eu-readelf'/elfutils (Ubuntu 0.176-1.1build1 in my case) appears to work, but
'abidw'/libabigail (current master branch) doesn't even open the '*.dwo' file,
per 'strace'.

If I wanted to try help implement that (but no promises how far I'll get...),
are Mark's 2018-06-05 comments here re "libabigail will need to use a few new
interfaces" still applicable, to get started?

Additionally, in PR25042 I saw Mark's 2019-09-30 comment that "logic [there] is
broken, [...] when dealing with split-dwarf DWO files", so I suppose that needs
to be reworked, too?  Per Dodji's 2019-10-01 comment, "this optimization will
be unnecessary, hopefully".

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #9 from Mark Wielaard <mark at klomp dot org> ---
Hi Thomas,

(In reply to Thomas Schwinge from comment #8)
> So I wanted to use libabigail on GCC '-gsplit-dwarf' object files
> ('*.o'/'*.dwo'), and found that it doesn't work.  Then found this existing
> PR; confirming that status is still as per Andrew's 2018-06-16 comment:
> 'eu-readelf'/elfutils (Ubuntu 0.176-1.1build1 in my case) appears to work,
> but 'abidw'/libabigail (current master branch) doesn't even open the '*.dwo'
> file, per 'strace'.
> 
> If I wanted to try help implement that (but no promises how far I'll
> get...), are Mark's 2018-06-05 comments here re "libabigail will need to use
> a few new interfaces" still applicable, to get started?

Yes, that is still applicable.

Where libabigail now uses dwarf_next_unit, it should use dwarf_get_units.
dwarf_get_units provides the type of the unit, if it is a type unit or split
unit, it will not only provide the CU DIE, but also the "Sub DIE", that is the
Type DIE (for type units) or Split DIE (for split units). It can then follow
the Split DIE to get the actual DIE tree for that unit (the CU DIE is just the
Skeleton DIE in that case).

These DIEs might come from different files (Dwarf objects) than the main Dwarf.
So where libabigail now uses Dwarf_Off as a (unique) identifier of a DIE (and
uses dwarf_offdie or dwarf_offdie_types to reconstruct a Dwarf_Die from a
Dwarf_Off), it should now use the Dwarf_Die void *addr field as unique
identifier (and use dwarf_die_addr_die to reconstruct the Dwarf_Die).

See libdw.h for more documentation. Specifically:

/* Gets the next Dwarf_CU (unit), version, unit type and if available
   the CU DIE and sub (type) DIE of the unit.  Returns 0 on success,
   -1 on error or 1 if there are no more units.  To start iterating
   provide NULL for CU.  If version < 5 the unit type is set from the
   CU DIE if available (DW_UT_compile for DW_TAG_compile_unit,
   DW_UT_type for DW_TAG_type_unit or DW_UT_partial for
   DW_TAG_partial_unit), otherwise it is set to zero.  If unavailable
   (the version or unit type is unknown) the CU DIE is cleared.
   Likewise if the sub DIE isn't isn't available (the unit type is not
   DW_UT_type or DW_UT_split_type) the sub DIE tag is cleared.  */
extern int dwarf_get_units (Dwarf *dwarf, Dwarf_CU *cu, Dwarf_CU **next_cu,
                            Dwarf_Half *version, uint8_t *unit_type,
                            Dwarf_Die *cudie, Dwarf_Die *subdie)
     __nonnull_attribute__ (3);

/* Given a Dwarf_Die addr returns a (reconstructed) Dwarf_Die, or NULL
   if the given addr didn't come from a valid Dwarf_Die.  In particular
   it will make sure that the correct Dwarf_CU pointer is set for the
   Dwarf_Die, the Dwarf_Abbrev pointer will not be set up yet (it will
   only be once the Dwarf_Die is used to read attributes, children or
   siblings).  This functions can be used to keep a reference to a
   Dwarf_Die which you want to refer to later.  The addr, and the result
   of this function, is only valid while the associated Dwarf is valid.  */
extern Dwarf_Die *dwarf_die_addr_die (Dwarf *dbg, void *addr,
                                      Dwarf_Die *result)
     __nonnull_attribute__ (3);

One note to keep in mind. Each .dwo is also represented as a Dwarf handle, and
a Dwarf_Die from a split dwarf file will have that as Dwarf handle. But the
above functions work on the main Dwarf handle (which owns the sub Dwarf handles
and will clean them up when you drop the main Dwarf handle).

Cheers,

Mark

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/21023] The abidw tool does not appear to read dwarf from .dwp files associated with executables

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=21023

--- Comment #10 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Thomas Schwinge from comment #8)
> Additionally, in PR25042 I saw Mark's 2019-09-30 comment that "logic [there]
> is broken, [...] when dealing with split-dwarf DWO files", so I suppose that
> needs to be reworked, too?  Per Dodji's 2019-10-01 comment, "this
> optimization will be unnecessary, hopefully".

libdw is designed so you don't need to know the specific DW_FORM that an
attribute is encoded in. For strings it expects your simply call
dwarf_formstring which provides you with a char * (and libdw takes care of
figuring out where precisely this string comes from for you).

libabigail however takes advantage of the fact that normally all strings come
from one section .debug_str which is a unique string pool. That allows some
optimizations when comparing strings.

But especially with split Dwarf (but also with small strings embedded in
.debug_info) there is no longer one .debug_str pool that contains all unique
strings (but one per .dwo file).

So you might have to go through the code to see where/how strings are compared
to make sure it still does the right thing given multiple string pools.

Maybe if string comparisons are really that important to optimize you'll have
to create a new string pool so you get string pointers/offsets that get you the
most optimal string compare results.

-- 
You are receiving this mail because:
You are on the CC list for the bug.