Porting pahole from dwarf_next_unit() to dwarf_get_units()

Porting pahole from dwarf_next_unit() to dwarf_get_units()

[Dimitri John Ledkov]

Currently pahole warns and does nothing upon hitting
DW_TAG_skeleton_unit as implemented at
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf

In elfutils, a while back a new API got added that aids with discovery
and processing of such tags -
https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade

It seems to me if pahole is ported from using dwarf_next_unit() to
instead use dwarf_get_units() native support can be added for
split-dwarf (dwo) files.

I am trying to write such a port, but it is proving to be very
difficult. I am entirely unfamiliar with neither pahole nor libdw nor
the dwarf file format. Thus it is very confusing when both pahole and
dwarf library use very similar type names and structs. For example
libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct.

What are the differences between dwarf_nextcu(), dwarf_next_unit(),
dwarf_get_units() and when should one use each one of them? (or nest
them?)

Is a port of https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
to use dwarf_get_units() a right approach and would be welcomed?

Is anyone else interested in providing any help, or guidance?

-- 
Dimitri

Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()

[Arnaldo Carvalho de Melo]

Em Tue, Dec 05, 2023 at 01:03:01PM +0000, Dimitri John Ledkov escreveu:
> Currently pahole warns and does nothing upon hitting
> DW_TAG_skeleton_unit as implemented at
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> 
> In elfutils, a while back a new API got added that aids with discovery
> and processing of such tags -
> https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade
> 
> It seems to me if pahole is ported from using dwarf_next_unit() to
> instead use dwarf_get_units() native support can be added for
> split-dwarf (dwo) files.

That would be awesome!
 
> I am trying to write such a port, but it is proving to be very
> difficult.

I did some work on supporting split-dwarf months ago, but got
sidetracked with other work, BTF related and then the code bitrotted, I
have to go back looking at it to swap back the details into my brain:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf

The patches after:

45c044860c2abce7 dwarf_loader: Sync with LINUX_ELFNOTE_LTO_INFO macro from kernel

Are the ones to suport alt dwarf.

> I am entirely unfamiliar with neither pahole nor libdw nor
> the dwarf file format. Thus it is very confusing when both pahole and
> dwarf library use very similar type names and structs. For example
> libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct.
 
> What are the differences between dwarf_nextcu(), dwarf_next_unit(),
> dwarf_get_units() and when should one use each one of them? (or nest
> them?)

> Is a port of https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> to use dwarf_get_units() a right approach and would be welcomed?

Yes, we need to support DWARF5 fully.
 
> Is anyone else interested in providing any help, or guidance?

I'm interested, and I think if Mark could help it would be great as
well.

- ARnaldo

Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()

[Dimitri John Ledkov]

I

On Tue, 5 Dec 2023, 15:47 Arnaldo Carvalho de Melo, <acme@kernel.org> wrote:
>
> Em Tue, Dec 05, 2023 at 01:03:01PM +0000, Dimitri John Ledkov escreveu:
> > Currently pahole warns and does nothing upon hitting
> > DW_TAG_skeleton_unit as implemented at
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> >
> > In elfutils, a while back a new API got added that aids with discovery
> > and processing of such tags -
> > https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade
> >
> > It seems to me if pahole is ported from using dwarf_next_unit() to
> > instead use dwarf_get_units() native support can be added for
> > split-dwarf (dwo) files.
>
> That would be awesome!
>
> > I am trying to write such a port, but it is proving to be very
> > difficult.
>
> I did some work on supporting split-dwarf months ago, but got
> sidetracked with other work, BTF related and then the code bitrotted, I
> have to go back looking at it to swap back the details into my brain:
>
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf
>
> The patches after:
>
> 45c044860c2abce7 dwarf_loader: Sync with LINUX_ELFNOTE_LTO_INFO macro from kernel
>
> Are the ones to suport alt dwarf.

I will read into those thanks.

>
> > I am entirely unfamiliar with neither pahole nor libdw nor
> > the dwarf file format. Thus it is very confusing when both pahole and
> > dwarf library use very similar type names and structs. For example
> > libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct.
>
> > What are the differences between dwarf_nextcu(), dwarf_next_unit(),
> > dwarf_get_units() and when should one use each one of them? (or nest
> > them?)
>
> > Is a port of https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> > to use dwarf_get_units() a right approach and would be welcomed?
>
> Yes, we need to support DWARF5 fully.

ack

>
> > Is anyone else interested in providing any help, or guidance?
>
> I'm interested, and I think if Mark could help it would be great as
> well.

I have something that sort of works, but then like aboarts with
invalid free's on exit - which the purist in me cares, but not sure if
it is of practical value or not.
And eu-readelf code also mentions that it deliberary leaks memory,
because life is hard.
I will try to address or warn about memory leaks to see if stuff
works, and post and RFC.

Regards,

Dimitri.

Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()

[Arnaldo Carvalho de Melo]

Em Tue, Dec 05, 2023 at 04:11:06PM +0000, Dimitri John Ledkov escreveu:
> I
> 
> On Tue, 5 Dec 2023, 15:47 Arnaldo Carvalho de Melo, <acme@kernel.org> wrote:
> >
> > Em Tue, Dec 05, 2023 at 01:03:01PM +0000, Dimitri John Ledkov escreveu:
> > > Currently pahole warns and does nothing upon hitting
> > > DW_TAG_skeleton_unit as implemented at
> > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> > >
> > > In elfutils, a while back a new API got added that aids with discovery
> > > and processing of such tags -
> > > https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade
> > >
> > > It seems to me if pahole is ported from using dwarf_next_unit() to
> > > instead use dwarf_get_units() native support can be added for
> > > split-dwarf (dwo) files.
> >
> > That would be awesome!
> >
> > > I am trying to write such a port, but it is proving to be very
> > > difficult.
> >
> > I did some work on supporting split-dwarf months ago, but got
> > sidetracked with other work, BTF related and then the code bitrotted, I
> > have to go back looking at it to swap back the details into my brain:
> >
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf
> >
> > The patches after:
> >
> > 45c044860c2abce7 dwarf_loader: Sync with LINUX_ELFNOTE_LTO_INFO macro from kernel
> >
> > Are the ones to suport alt dwarf.
> 
> I will read into those thanks.

Hopefully what you've been work ends up less convoluted by using
new elfutils functions, but it may be useful to help understand how the
internal pahole code deals with Dwarf offsets to reduce them to 32 bits
for conversion to CTF/BTF (where it is not really that important as
libbpf does this work).
 
> > > I am entirely unfamiliar with neither pahole nor libdw nor
> > > the dwarf file format. Thus it is very confusing when both pahole and
> > > dwarf library use very similar type names and structs. For example
> > > libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct.
> >
> > > What are the differences between dwarf_nextcu(), dwarf_next_unit(),
> > > dwarf_get_units() and when should one use each one of them? (or nest
> > > them?)
> >
> > > Is a port of https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> > > to use dwarf_get_units() a right approach and would be welcomed?
> >
> > Yes, we need to support DWARF5 fully.

> ack
 
> > > Is anyone else interested in providing any help, or guidance?
> >
> > I'm interested, and I think if Mark could help it would be great as
> > well.
> 
> I have something that sort of works, but then like aboarts with
> invalid free's on exit - which the purist in me cares, but not sure if
> it is of practical value or not.
> And eu-readelf code also mentions that it deliberary leaks memory,
> because life is hard.

Off course ;-) Frees on exit are interesting to try to evaluate if
things that are not just frees on exit are leaking, when we're sure that
is the case, don't free on exit, as its just overhead.

> I will try to address or warn about memory leaks to see if stuff
> works, and post and RFC.

Great, thanks a lot for working on this!

- Arnaldo

Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()

[Mark Wielaard]

Hi Dimitri,

Sorry, this arrived before my vacation and then the new year happened.

On Tue, Dec 05, 2023 at 01:03:01PM +0000, Dimitri John Ledkov wrote:
> Currently pahole warns and does nothing upon hitting
> DW_TAG_skeleton_unit as implemented at
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf
> 
> In elfutils, a while back a new API got added that aids with discovery
> and processing of such tags -
> https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade
> 
> It seems to me if pahole is ported from using dwarf_next_unit() to
> instead use dwarf_get_units() native support can be added for
> split-dwarf (dwo) files.
> 
> I am trying to write such a port, but it is proving to be very
> difficult. I am entirely unfamiliar with neither pahole nor libdw nor
> the dwarf file format. Thus it is very confusing when both pahole and
> dwarf library use very similar type names and structs. For example
> libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct.
> 
> What are the differences between dwarf_nextcu(), dwarf_next_unit(),
> dwarf_get_units() and when should one use each one of them? (or nest
> them?)

The dwarf_nextcu was the original way to iterate over the CUs from
.debug_info. Then dwarf_next_unit was added when type units could come
from a .debug_types section. Both functions use and return offsets to
iterate through the section and then get the CU DIE using dwarf_offdie
(or dwarf_offdie_types). This requires the user to know beforehand
where to DIE data is stored (in the .debug_info or .debug_types
section).  For type units one also needs to use the type offset to
create the actual type DIE. In DWARF5 DIEs can come from even more
data locations. And there are also skeleton units which require the
user to find the associated split compile unit DIE (which would come
from a different file).
    
The new dwarf_get_units function simplifies iterating over the units
in a DWARF file. It doesn't require the user to know where the DIE
data is stored, it will automagically iterate over all know data
sources (sections) returning the Dwarf_CU and the associated Dwarf_Die
if requested. If the user requests to know the associated "subdie" it
will also be resolved.
    
A subdie is either a type DIE for a type unit or a split unit DIE for
a skeleton unit. The same (and some more) info about DWARF_CUs can
also be gotten through the dwarf_cu_info function.

You should either use dwarf_nextcu or dwarf_next_unit with
dwarf_offdie to get the (top-level) DIE. Or use dwarf_get_units and
possibly dwarf_cu_info. In general you shouldn't mix them.

Hope this helps and let me know if you need more info.

Cheers,

Mark