Latest version of dwarflint?

Latest version of dwarflint?

[Ananthakrishna Sowda (asowda)]

I found the dwarflint and locstats tools in the ‘dwarf ‘ branch of elfutils.
   https://sourceware.org/elfutils/DwarfLint
Any plans of merging this into main branch?

I get following error from locstats:

error: .debug_info: DIE 0x3f10fc: dwarf_getlocation_addr: invalid DWARF

This error is reported on DW_OP_GNU_entry_value in DWARF location list. Any newer version of these tools which can handle DWARF5 ops such as DW_OP_GNU_entry-value?
Thanks
Ananth

Re: Latest version of dwarflint?

[Mark Wielaard]

On Wed, 2017-08-02 at 19:47 +0000, Ananthakrishna Sowda (asowda) wrote:
> I found the dwarflint and locstats tools in the ‘dwarf ‘ branch of elfutils.
>    https://sourceware.org/elfutils/DwarfLint
> Any plans of merging this into main branch?

Currently not, sorry.

> I get following error from locstats:
> 
> error: .debug_info: DIE 0x3f10fc: dwarf_getlocation_addr: invalid DWARF
> 
> This error is reported on DW_OP_GNU_entry_value in DWARF location list. Any newer version of these tools which can handle DWARF5 ops such as DW_OP_GNU_entry-value?

There is a standalone version of dwlocstat here:
https://github.com/pmachata/dwlocstat

It should handle DW_OP_GNU_entry_value but is also a bit old so might
not handle the latest DWARF extensions.

Cheers,

Mark

Re: Latest version of dwarflint?

[Ananthakrishna Sowda (asowda)]

Any interest in seeing some patches which upgrade dwarflint to latest elfutils-0.170?    Djordje Todorovic  has more details about the patches.
Thanks,
Ananth

On 8/3/17, 2:13 AM, "Mark Wielaard" <mark@klomp.org> wrote:

    On Wed, 2017-08-02 at 19:47 +0000, Ananthakrishna Sowda (asowda) wrote:
    > I found the dwarflint and locstats tools in the ‘dwarf ‘ branch of elfutils.
    >    https://sourceware.org/elfutils/DwarfLint
    > Any plans of merging this into main branch?
    
    Currently not, sorry.
    
    > I get following error from locstats:
    > 
    > error: .debug_info: DIE 0x3f10fc: dwarf_getlocation_addr: invalid DWARF
    > 
    > This error is reported on DW_OP_GNU_entry_value in DWARF location list. Any newer version of these tools which can handle DWARF5 ops such as DW_OP_GNU_entry-value?
    
    There is a standalone version of dwlocstat here:
    https://github.com/pmachata/dwlocstat
    
    It should handle DW_OP_GNU_entry_value but is also a bit old so might
    not handle the latest DWARF extensions.
    
    Cheers,
    
    Mark
    

Re: Latest version of dwarflint?

[Djordje Todorovic]

Hi All,

I have managed to integrate dwarflint on the top of elfutils-0.170 release, so dwarflint and locstats can use all conveniences and improvements of the newest libdw library (such as 
handling ops like DW_OP_GNU_entry_value).

It is used internally by our compiler team. I have also managed to add some new enhancements for locstats, such as reporting summary either for formal parameter or local variable, 
reporting some kind of verbose info about particular variable or formal parameter of particular function.
But, beside these enhancements that were interested for our compiler team, dwarflint and locstats have a lot of good utilities that can be really useful to compiler developers. For 
example, dwarflint tool can detect mistakes in debug sections such as debug info, debug line etc., and help compiler developers to get rid of those mistakes. Locstats can 
apparently report useful summary and show progress of making better coverage for local variables or formal parameter in debug sections, so it can save a lot of time to compiler 
developers.

At least, maybe considering a making new branch called „dwarflint“ on elfuitls git repository, which follows the newest source code, would be good for the beginning, the same as it 
was on elfutils-0.152? Because, getting more feedback from users, these tools would be better and more useful, so it can be, one day, merged on to master branch.

If you are interested, I can share my patches.

Best regards,
Djordje Todorovic

On 21.09.2017. 17:59, Ananthakrishna Sowda (asowda) wrote:
> Any interest in seeing some patches which upgrade dwarflint to latest elfutils-0.170?    Djordje Todorovic  has more details about the patches.
> Thanks,
> Ananth
> 
> On 8/3/17, 2:13 AM, "Mark Wielaard" <mark@klomp.org> wrote:
> 
>      On Wed, 2017-08-02 at 19:47 +0000, Ananthakrishna Sowda (asowda) wrote:
>      > I found the dwarflint and locstats tools in the ‘dwarf ‘ branch of elfutils.
>      >    https://sourceware.org/elfutils/DwarfLint
>      > Any plans of merging this into main branch?
>      
>      Currently not, sorry.
>      
>      > I get following error from locstats:
>      >
>      > error: .debug_info: DIE 0x3f10fc: dwarf_getlocation_addr: invalid DWARF
>      >
>      > This error is reported on DW_OP_GNU_entry_value in DWARF location list. Any newer version of these tools which can handle DWARF5 ops such as DW_OP_GNU_entry-value?
>      
>      There is a standalone version of dwlocstat here:
>      https://github.com/pmachata/dwlocstat
>      
>      It should handle DW_OP_GNU_entry_value but is also a bit old so might
>      not handle the latest DWARF extensions.
>      
>      Cheers,
>      
>      Mark
>      
> 

Re: Latest version of dwarflint?

[Mark Wielaard]

Hi Djordje,

On Thu, 2017-09-21 at 18:07 +0200, Djordje Todorovic wrote:
> I have managed to integrate dwarflint on the top of elfutils-0.170
> release, so dwarflint and locstats can use all conveniences and
> improvements of the newest libdw library (such as 
> handling ops like DW_OP_GNU_entry_value).

Very nice.

> It is used internally by our compiler team. I have also managed to
> add some new enhancements for locstats, such as reporting summary
> either for formal parameter or local variable, 
> reporting some kind of verbose info about particular variable or
> formal parameter of particular function.
> But, beside these enhancements that were interested for our compiler
> team, dwarflint and locstats have a lot of good utilities that can be
> really useful to compiler developers. For 
> example, dwarflint tool can detect mistakes in debug sections such as
> debug info, debug line etc., and help compiler developers to get rid
> of those mistakes. Locstats can 
> apparently report useful summary and show progress of making better
> coverage for local variables or formal parameter in debug sections,
> so it can save a lot of time to compiler 
> developers.

Yes, I know some compiler hackers have found it useful. And I was a
little sad nobody had time to keep dwarflint up to date. It is really
great to hear you have.

> At least, maybe considering a making new branch called „dwarflint“ on
> elfuitls git repository, which follows the newest source code, would
> be good for the beginning, the same as it 
> was on elfutils-0.152? Because, getting more feedback from users,
> these tools would be better and more useful, so it can be, one day,
> merged on to master branch.
> 
> If you are interested, I can share my patches.

Yes please do share. I cannot promise to have a lot of time reviewing
the code. But I am certainly interested. Creating a public "dwarflint"
branch sounds like a good idea.

Could you take a look at the CONTRIBUTING file? We don't have to be
very formal for code not directly going onto master. But having at
least a Signed-off-by line on the patches makes sure we can eventually
integrate them officially.

Thanks,

Mark

Re: Latest version of dwarflint?

[Djordje Todorovic]

Hi Mark,

Thank you for your comments!

>Could you take a look at the CONTRIBUTING file?
Sure.

How should I send this patches? Should I wait for you to make a new "dwarflint" branch which follows master branch or?

Thanks,
Djordje

-------- Original Message --------
Subject: Re: Latest version of dwarflint?
Date: Thursday, September 21, 2017 21:00 CEST
From: Mark Wielaard <mark@klomp.org>
To: Djordje Todorovic <djordje.todorovic@rt-rk.com>, "Ananthakrishna Sowda	(asowda)" <asowda@cisco.com>
CC: "elfutils-devel@sourceware.org" <elfutils-devel@sourceware.org>
References: <81113170-F590-4C51-BEA0-C8EF3A4422DA@contoso.com>	 <1501751620.3741.132.camel@klomp.org>	 <624331B2-472C-44AD-9034-05AD3CBBAE8A@cisco.com>	 <cc97238e-81a3-683b-a2ea-1e4d82382f40@rt-rk.com>



> Hi Djordje,
>
> On Thu, 2017-09-21 at 18:07 +0200, Djordje Todorovic wrote:
> > I have managed to integrate dwarflint on the top of elfutils-0.170
> > release, so dwarflint and locstats can use all conveniences and
> > improvements of the newest libdw library (such as 
> > handling ops like DW_OP_GNU_entry_value).
>
> Very nice.
>
> > It is used internally by our compiler team. I have also managed to
> > add some new enhancements for locstats, such as reporting summary
> > either for formal parameter or local variable, 
> > reporting some kind of verbose info about particular variable or
> > formal parameter of particular function.
> > But, beside these enhancements that were interested for our compiler
> > team, dwarflint and locstats have a lot of good utilities that can be
> > really useful to compiler developers. For 
> > example, dwarflint tool can detect mistakes in debug sections such as
> > debug info, debug line etc., and help compiler developers to get rid
> > of those mistakes. Locstats can 
> > apparently report useful summary and show progress of making better
> > coverage for local variables or formal parameter in debug sections,
> > so it can save a lot of time to compiler 
> > developers.
>
> Yes, I know some compiler hackers have found it useful. And I was a
> little sad nobody had time to keep dwarflint up to date. It is really
> great to hear you have.
>
> > At least, maybe considering a making new branch called „dwarflint“ on
> > elfuitls git repository, which follows the newest source code, would
> > be good for the beginning, the same as it 
> > was on elfutils-0.152? Because, getting more feedback from users,
> > these tools would be better and more useful, so it can be, one day,
> > merged on to master branch.
> >
> > If you are interested, I can share my patches.
>
> Yes please do share. I cannot promise to have a lot of time reviewing
> the code. But I am certainly interested. Creating a public "dwarflint"
> branch sounds like a good idea.
>
> Could you take a look at the CONTRIBUTING file? We don't have to be
> very formal for code not directly going onto master. But having at
> least a Signed-off-by line on the patches makes sure we can eventually
> integrate them officially.
>
> Thanks,
>
> Mark

Re: Latest version of dwarflint?

[Mark Wielaard]

On Fri, 2017-09-22 at 15:37 +0200, Djordje Todorovic wrote:
> How should I send this patches? Should I wait for you to make a new
> "dwarflint" branch which follows master branch or?

However is most convenient to you.
Do you have patches against the origin/dwarf branch that contained the
original dwarlint? Or against current master (or the 0.170 release)
branch? If you have them in a git branch and it is easy to extract them
then using git send-email to this list would probably be easiest. Then
we can create a new branch as base to apply them on top.

Thanks,

Mark

Re: Latest version of dwarflint?

[Djordje Todorovic]

 > Do you have patches against the origin/dwarf branch that contained the
 >original dwarlint? Or against current master (or the 0.170 release)
 >branch?
I have patches that can be applied on top of 0.170 release. I have moved dwarflint and libdw/c++ directories from origin/dwarf branch and then add support in configure and 
makefiles to build it successfully. After that I made some fixes and improvements of the utilities.

 > If you have them in a git branch and it is easy to extract them
 >then using git send-email to this list would probably be easiest. Then
 >we can create a new branch as base to apply them on top.

Yes, I have git branch, but these patches are pretty cumulative and I’m breaking up them into more logical pieces, so it would be easier for maintaining. I will be back in a few days.

Thanks,
Djordje

Re: Latest version of dwarflint?

[Mark Wielaard]

Hi Djordje,

On Thu, Sep 28, 2017 at 03:36:14PM +0200, Djordje Todorovic wrote:
> Please find attached compressed folder with dwarflint patches. I’m sending it in this way because I think changes are not that a small to be sent in a mail body or with git mail.

Thanks. It looks like the mailinglist didn't like the big zip file.
So for now I have put it here:
https://gnu.wildebeest.org/~mark/dwarflint_patches.zip

> All of the patches can be applied on to elfutils-0.170 release. I have run
> dwarflint test suites and all of the tests are PASS at this point.

I haven't gotten them all applied yet.
I am adjusting the Makefile.in changes to the Makefile.am files,
so we don't have generated files in git. And since I am on RHEL7
with an older g++ I had to add some autoconf magic to get C++11 support:
https://www.gnu.org/software/autoconf-archive/ax_cxx_compile_stdcxx.html

As soon as I have it cleaned up to build I'll add the patches to a
new dwarflint branch.

> Besides bug fixes and code updates, please note a new option added to
> locstats tool (e.g. --sub-category=formal_parameters), which reports summary
> either for formal parameters or local variables.
> 
> Also, I’m working on adding a new option which can preset debug info about
> particular variable of particular function in more “human readable” way. At
> this point, that option can handle only C functions and I’m customizing it
> also to handle methods of C++ classes as well. There can be output more
> additional info, e.g. compilation unit etc. I think it would be very useful
> to users and when I’m done with it, I’ll share it.
> 
> Running the tool on GDB executable with option that I have mentioned looks as following:
> 
> eu-locstats  --show=amd64_epilogue_frame_cache[this_frame] gdb
> Function: amd64_epilogue_frame_cache
> Formal parameter: this_frame
> Low PC: 0x40b4e0
> High PC relative: 0x156
> Range of the function: [0x40b4e0, 0x40b636)
> Location list of the formal parameter:
> 	PC range: [40b4e0,40b504) expression: {DW_OP_reg5}
> 	PC range: [40b504,40b519) expression: {DW_OP_reg6}
> 	PC range: [40b519,40b538) expression: {DW_OP_GNU_entry_value(1) {DW_OP_reg5}, DW_OP_stack_value}
> 
> So, for example, if compiler developers adding a handling of a new DWARF operation, in quick way they can see if there is appropriate information generated.
> 
> Thanks,
> Djordje

Re: Latest version of dwarflint?

[Mark Wielaard]

On Fri, Sep 29, 2017 at 01:58:43PM +0200, Mark Wielaard wrote:
> On Thu, Sep 28, 2017 at 03:36:14PM +0200, Djordje Todorovic wrote:
> > Please find attached compressed folder with dwarflint patches. I’m
> > sending it in this way because I think changes are not that a small
> > to be sent in a mail body or with git mail.
> 
> Thanks. It looks like the mailinglist didn't like the big zip file.
> So for now I have put it here:
> https://gnu.wildebeest.org/~mark/dwarflint_patches.zip

BTW. The *bz2 test binaries under dwarflint/tests were not in the zip
file. If any of them didn't come from the dwarf branch could you sent
them separately?

Thanks,

Mark

Re: Latest version of dwarflint?

[Djordje Todorovic]

 > BTW. The *bz2 test binaries under dwarflint/tests were not in the zip
 > file. If any of them didn't come from the dwarf branch could you sent
 > them separately?
Are you applied the patch 0002-Add-dwarflint-from-origin-dwarf-branch.patch with 'git apply'?
Because, the 'patch' command would not include *bz2 files.

Re: Latest version of dwarflint?

[Mark Wielaard]

On Fri, Sep 29, 2017 at 02:44:31PM +0200, Djordje Todorovic wrote:
> > BTW. The *bz2 test binaries under dwarflint/tests were not in the zip
> > file. If any of them didn't come from the dwarf branch could you sent
> > them separately?
> Are you applied the patch 0002-Add-dwarflint-from-origin-dwarf-branch.patch with 'git apply'?
> Because, the 'patch' command would not include *bz2 files.

Ah, right, doh. Got them.
The real problem was that I build with srcdir != builddir.
Will fix the check target to support that.

Thanks,

Mark

Re: Latest version of dwarflint?

[Mark Wielaard]

Hi,

On Fri, 2017-09-29 at 13:58 +0200, Mark Wielaard wrote:
> I haven't gotten them all applied yet.
> I am adjusting the Makefile.in changes to the Makefile.am files,
> so we don't have generated files in git. And since I am on RHEL7
> with an older g++ I had to add some autoconf magic to get C++11
> support:
> https://www.gnu.org/software/autoconf-archive/ax_cxx_compile_stdcxx.h
> tml
> 
> As soon as I have it cleaned up to build I'll add the patches to a
> new dwarflint branch.

I finally finished this. The code should now build fine on older
setups, with srcdir != destdir and make distcheck mostly works
(dwarflint doesn't support --version which I haven't fixed yet).
In a couple of places I did add slightly newer versions of the files
from the dwarf branch (if only to get the correct license heaers).

There is now dwarflint branch in git against latest git master.
https://sourceware.org/git/?p=elfutils.git;a=shortlog;h=refs/heads/dwarflint

The shortlog looks like:

Djordje Todorovic (12):
      libdw: Add ELFUTILS_PRIVATE_TMP version
      Add libdw/c++/ from origin/dwarf branch
      Add dwarflint/ from origin/dwarf branch
      Add support for building C++
      Add files needed by c++ module from dwarf branch.
      dwarflint: Update code to work with current version of elfutils
      libdw: Update code to work with current version of elfutils
      dwarflint: Report summary for sub-category of variables
      dwarflint: Avoid corrupted access
      libdw: Proper handling of DW_AT_high_pc
      dwarflint: Remember the referring CU when read src lines
      Add support for dwarflint testing

Mark Wielaard (2):
      C++ build fixes.
      Add support for DW_OP_GNU_parameter_ref to dwarf-opcodes.h

Where I tweaked your commits a little I updated the commit message and
added my own Signed-off-by.

Thanks a lot for this. There were already happy users on irc
(irc.freenode.net #elfutils). And I hope it will help with my own
DWARF5 work (of course the code will need to be updated for that).

There is still a lot of work to do to get this in a releasable state
(where we can merge it to master). I didn't really review all the code.
And there is a lot. I hope we can trim down the new C++ files a bit.
Because I don't think we need it all. Also there are currently various
hacks to get things to build and run than would need to be cleaned up.
But it is in a useful state now for people who don't mind to build from
source.

Thanks,

Mark