DWZ 0.14 released

DWZ 0.14 released

[Tom de Vries]

Hi,

DWZ 0.14 has been released.

You can download dwz from the sourceware FTP server here:

        https://sourceware.org/ftp/dwz/releases/
        ftp://sourceware.org/pub/dwz/releases/

The vital stats:

  Size    md5sum                            Name
  184KiB  cf60e4a65d9cc38c7cdb366e9a29ca8e  dwz-0.14.tar.gz
  144KiB  1f1225898bd40d63041d54454fcda5b6  dwz-0.14.tar.xz

There is a web page for DWZ at:

        https://sourceware.org/dwz/

DWZ 0.14 includes the following changes and enhancements:

* DWARF 5 support. The tool now handles most of DWARF version 5
  (at least everything emitted by GCC when using -gdwarf-5).

  Not yet supported are DW_UT_type units (DWARF 4 .debug_types
  are supported), .debug_names (.gdb_index is supported) and some
  forms and sections that are only emitted by GCC when
  generating Split DWARF (DW_FORM_strx and .debug_str_offsets,
  DW_FORM_addrx and .debug_addr, DW_FORM_rnglistx and
  DW_FORM_loclistsx). https://sourceware.org/PR24726

* .debug_sup support. DWARF Supplementary Object Files
  (DWARF 5, section 7.3.6) can now be generated when using
  the --dwarf-5 option. To keep compatibility with existing DWARF
  consumers this isn't the default yet.

  Without the --dwarf-5 option instead of a .debug_sup section dwz
  will generate a .gnu_debugaltlink section and will use
  DW_FORM_GNU_strp_alt and DW_FORM_GNU_reg_alt, instead of
  DW_FORM_strp_sup and DW_FORM_ref_sup

* An experimental optimization has been added that exploits the
  One-Definition-Rule of C++.  It's enabled using the --odr option, and
  off by default.  This optimization causes struct/union/class DIEs with
  the same name to be considered equal.  The optimization can be set to
  a lower aggressiveness level using --odr-mode=basic, to possibly be
  able to workaround problems without having to switch off the
  optimization altogether.

* The clean-up of temporary files in hardlink mode has been fixed.

* The DIE limits --low-mem-die-limit <n> / -l <n> and
  --max-die-limit <n> / -L <n> can now be disabled using respectively
  -l none and -L none.  Note that -l none disables the limit, whereas
  -l 0 sets the limit to zero.

* The usage message has been:
  - updated to show that -r and -M are exclusive.
  - updated to show at -v and -? cannot be combined with other options.
  - extended to list all options in detail.
  - restyled to wrap at 80 chars.

* An option --no-import-optimize was added that switches off an
  optimization that attempts to reduce the number of
  DW_TAG_imported_unit DIEs.  This can be used f.i. in case the
  optimization takes too long.

* A heuristic has been added that claims more memory earlier (without
  increasing the peak memory usage) to improve compression time.

* A heuristic has been added that estimates whether one of the two DIE
  limits will be hit.  If so, it will do an exact DIE count to verify
  this.  If the exact DIE count finds that the low-mem DIE limit is
  indeed hit, processing is done in low-mem mode from the start, rather
  than processing in regular mode first.  If the exact DIE count finds
  that the max DIE limit is indeed hit, processing is skipped
  altogether.

* Various other performance improvements.

* A case where previously we would either hit the assertion
  "dwz: dwz.c:9461: write_die: Assertion `refd != NULL' failed" (in
  regular mode) or a segmentation fault (in low-mem mode), now is
  handled by "dwz: Couldn't find DIE at DW_FORM_ref_addr offset 0x<n>".

* A case where a reference from a partial unit to a compile unit was
  generated has been fixed.  This could happen if a DIE was referenced
  using a CU-relative DWARF operator.

* A case has been fixed for low-mem mode where instead of issuing
  "dwz: Couldn't find DIE referenced by  DW_OP_GNU_implicit_pointer" dwz
  would run into a segfault instead.

* A multi-file case where we run into ".debug_line reference above end
  of section" has been fixed.

* The following assertion failures were fixed:
  - dwz: dwz.c:9310: write_die: Assertion `
      value && refdcu->cu_kind != CU_ALT
    ' failed.
  - dwz: dwz.c:9920: recompute_abbrevs: Assertion `
      off == cu_size
    ' failed.

* The assert condition of this assertion has been fixed:
  - write_types: Assertion `ref && ref->die_dup == NULL'.

Re: DWZ 0.14 released

[Allan Sandfeld Jensen]

Btw, question for gcc/binutils

Any reason the work done by tools like dwz couldn't be done in the compiler or 
linker? Seems a bit odd to have a post-linker that optimizes the generated 
code, when optimizations should already be enabled.

Best regards
Allan

On Montag, 8. März 2021 13:43:11 CET Tom de Vries wrote:
> Hi,
> 
> DWZ 0.14 has been released.
> 
> You can download dwz from the sourceware FTP server here:
> 
>         https://sourceware.org/ftp/dwz/releases/
>         ftp://sourceware.org/pub/dwz/releases/
> 
> The vital stats:
> 
>   Size    md5sum                            Name
>   184KiB  cf60e4a65d9cc38c7cdb366e9a29ca8e  dwz-0.14.tar.gz
>   144KiB  1f1225898bd40d63041d54454fcda5b6  dwz-0.14.tar.xz
> 
> There is a web page for DWZ at:
> 
>         https://sourceware.org/dwz/
> 
> DWZ 0.14 includes the following changes and enhancements:
> 
> * DWARF 5 support. The tool now handles most of DWARF version 5
>   (at least everything emitted by GCC when using -gdwarf-5).
> 
>   Not yet supported are DW_UT_type units (DWARF 4 .debug_types
>   are supported), .debug_names (.gdb_index is supported) and some
>   forms and sections that are only emitted by GCC when
>   generating Split DWARF (DW_FORM_strx and .debug_str_offsets,
>   DW_FORM_addrx and .debug_addr, DW_FORM_rnglistx and
>   DW_FORM_loclistsx). https://sourceware.org/PR24726
> 
> * .debug_sup support. DWARF Supplementary Object Files
>   (DWARF 5, section 7.3.6) can now be generated when using
>   the --dwarf-5 option. To keep compatibility with existing DWARF
>   consumers this isn't the default yet.
> 
>   Without the --dwarf-5 option instead of a .debug_sup section dwz
>   will generate a .gnu_debugaltlink section and will use
>   DW_FORM_GNU_strp_alt and DW_FORM_GNU_reg_alt, instead of
>   DW_FORM_strp_sup and DW_FORM_ref_sup
> 
> * An experimental optimization has been added that exploits the
>   One-Definition-Rule of C++.  It's enabled using the --odr option, and
>   off by default.  This optimization causes struct/union/class DIEs with
>   the same name to be considered equal.  The optimization can be set to
>   a lower aggressiveness level using --odr-mode=basic, to possibly be
>   able to workaround problems without having to switch off the
>   optimization altogether.
> 
> * The clean-up of temporary files in hardlink mode has been fixed.
> 
> * The DIE limits --low-mem-die-limit <n> / -l <n> and
>   --max-die-limit <n> / -L <n> can now be disabled using respectively
>   -l none and -L none.  Note that -l none disables the limit, whereas
>   -l 0 sets the limit to zero.
> 
> * The usage message has been:
>   - updated to show that -r and -M are exclusive.
>   - updated to show at -v and -? cannot be combined with other options.
>   - extended to list all options in detail.
>   - restyled to wrap at 80 chars.
> 
> * An option --no-import-optimize was added that switches off an
>   optimization that attempts to reduce the number of
>   DW_TAG_imported_unit DIEs.  This can be used f.i. in case the
>   optimization takes too long.
> 
> * A heuristic has been added that claims more memory earlier (without
>   increasing the peak memory usage) to improve compression time.
> 
> * A heuristic has been added that estimates whether one of the two DIE
>   limits will be hit.  If so, it will do an exact DIE count to verify
>   this.  If the exact DIE count finds that the low-mem DIE limit is
>   indeed hit, processing is done in low-mem mode from the start, rather
>   than processing in regular mode first.  If the exact DIE count finds
>   that the max DIE limit is indeed hit, processing is skipped
>   altogether.
> 
> * Various other performance improvements.
> 
> * A case where previously we would either hit the assertion
>   "dwz: dwz.c:9461: write_die: Assertion `refd != NULL' failed" (in
>   regular mode) or a segmentation fault (in low-mem mode), now is
>   handled by "dwz: Couldn't find DIE at DW_FORM_ref_addr offset 0x<n>".
> 
> * A case where a reference from a partial unit to a compile unit was
>   generated has been fixed.  This could happen if a DIE was referenced
>   using a CU-relative DWARF operator.
> 
> * A case has been fixed for low-mem mode where instead of issuing
>   "dwz: Couldn't find DIE referenced by  DW_OP_GNU_implicit_pointer" dwz
>   would run into a segfault instead.
> 
> * A multi-file case where we run into ".debug_line reference above end
>   of section" has been fixed.
> 
> * The following assertion failures were fixed:
>   - dwz: dwz.c:9310: write_die: Assertion `
>       value && refdcu->cu_kind != CU_ALT
>     ' failed.
>   - dwz: dwz.c:9920: recompute_abbrevs: Assertion `
>       off == cu_size
>     ' failed.
> 
> * The assert condition of this assertion has been fixed:
>   - write_types: Assertion `ref && ref->die_dup == NULL'.

Re: DWZ 0.14 released

[Mark Wielaard]

Hi Allan,

On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote:
> Btw, question for gcc/binutils
> 
> Any reason the work done by tools like dwz couldn't be done in the compiler or 
> linker? Seems a bit odd to have a post-linker that optimizes the generated 
> code, when optimizations should already be enabled.

dwz does two kinds of optimization. First it attempts to optimize the
DWARF debugging information for a given object (executable or shared
library). Secondly it tries to put shared pieces of a list of given
objects into a supplemental file that gets referenced from all the
given object files.

Technically the first optimization could be done by the linker. But
the second optimization is really a post-linker step.

Cheers,

Mark

Re: DWZ 0.14 released

[Hannes Domani]

 Am Dienstag, 9. März 2021, 10:10:47 MEZ hat Mark Wielaard <mark@klomp.org> Folgendes geschrieben:

> Hi Allan,
>
> On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote:
> > Btw, question for gcc/binutils
> >
> > Any reason the work done by tools like dwz couldn't be done in the compiler or
> > linker? Seems a bit odd to have a post-linker that optimizes the generated
> > code, when optimizations should already be enabled.
>
>
> dwz does two kinds of optimization. First it attempts to optimize the
> DWARF debugging information for a given object (executable or shared
> library). Secondly it tries to put shared pieces of a list of given
> objects into a supplemental file that gets referenced from all the
> given object files.
>
> Technically the first optimization could be done by the linker. But
> the second optimization is really a post-linker step.

Related question: If it were part of binutils, maybe it could be adapted to
optimize DWARF debugging information of PE files as well.


Hannes

Re: DWZ 0.14 released

[Jakub Jelinek]

On Tue, Mar 09, 2021 at 11:38:07AM +0000, Hannes Domani via Dwz wrote:
>  Am Dienstag, 9. März 2021, 10:10:47 MEZ hat Mark Wielaard <mark@klomp.org> Folgendes geschrieben:
> 
> > Hi Allan,
> >
> > On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote:
> > > Btw, question for gcc/binutils
> > >
> > > Any reason the work done by tools like dwz couldn't be done in the compiler or
> > > linker? Seems a bit odd to have a post-linker that optimizes the generated
> > > code, when optimizations should already be enabled.
> >
> >
> > dwz does two kinds of optimization. First it attempts to optimize the
> > DWARF debugging information for a given object (executable or shared
> > library). Secondly it tries to put shared pieces of a list of given
> > objects into a supplemental file that gets referenced from all the
> > given object files.
> >
> > Technically the first optimization could be done by the linker. But
> > the second optimization is really a post-linker step.
> 
> Related question: If it were part of binutils, maybe it could be adapted to
> optimize DWARF debugging information of PE files as well.

dwz intentionally uses libelf, it often deals with very large amounts of
debug info that only barely fit into the address space limitations on
certain arches or physical memory for good performance, and any kind of
abstraction penalty (e.g. bfd) would make it slower and more memory hungry.
So no, PE support is not possible.

	Jakub

Re: DWZ 0.14 released

[Nick Alcock]

On 9 Mar 2021, Jakub Jelinek via Binutils spake thusly:

> On Tue, Mar 09, 2021 at 11:38:07AM +0000, Hannes Domani via Dwz wrote:
>>  Am Dienstag, 9. März 2021, 10:10:47 MEZ hat Mark Wielaard <mark@klomp.org> Folgendes geschrieben:
>> 
>> > Hi Allan,
>> >
>> > On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote:
>> > > Btw, question for gcc/binutils
>> > >
>> > > Any reason the work done by tools like dwz couldn't be done in the compiler or
>> > > linker? Seems a bit odd to have a post-linker that optimizes the generated
>> > > code, when optimizations should already be enabled.
>> >
>> >
>> > dwz does two kinds of optimization. First it attempts to optimize the
>> > DWARF debugging information for a given object (executable or shared
>> > library). Secondly it tries to put shared pieces of a list of given
>> > objects into a supplemental file that gets referenced from all the
>> > given object files.
>> >
>> > Technically the first optimization could be done by the linker. But
>> > the second optimization is really a post-linker step.
>> 
>> Related question: If it were part of binutils, maybe it could be adapted to
>> optimize DWARF debugging information of PE files as well.
>
> dwz intentionally uses libelf, it often deals with very large amounts of
> debug info that only barely fit into the address space limitations on
> certain arches or physical memory for good performance, and any kind of
> abstraction penalty (e.g. bfd) would make it slower and more memory hungry.

Well, it's not *impossible*. You could in theory do what CTF has done:
move the dedup machinery into a library which is then called both from
the linker (providing its input via BFD and explicitly not supporting
multifile unification of DWARF) and from a separate dedup tool still
called 'dwz' (providing its input via libelf, and flipping a switch
allowing dedup across files and producing an additional output which the
tool then writes to the unified file).

Howver, link speed would likely be affected if dedup is on by default
(it's still a concern of mine with libctf even though none of my
testcases take more than a couple of seconds to dedup: I know how to
shave a *lot* of time off that, I just haven't done it yet). It's quite
possible that you'll save as much time by not having to write as much
DWARF out as you lose deduplicating -- but the write time is usually
hidden from the user anyway since writeback is usually buffered on most
operating systems. So this is probably not as helpful as it might appear
:(

Also, it might not be acceptable to have dwz depend on a shared library
provided by binutils, nor to have binutils depend (even optionally?) on
a shared library provided by dwz...