[Bug gdb/29584] New: gdb: support zstd compressed .gnu_debugdata

[Bug gdb/29584] New: gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

            Bug ID: 29584
           Summary: gdb: support zstd compressed .gnu_debugdata
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: i at maskray dot me
  Target Milestone: ---

.gnu_debugdata is compressed with xz, which is very slow to compress and
decompress.

zstd compressing with a large level (e.g. 18 used by Arch Linux packaging, 19
used by openSUSE packaging) can be slower than xz, but the output is much
faster to decompress (e.g. time(xz -18) / time(zstd -18) > 5 for some
workload).

This depends on PR29563 .

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

Fangrui Song <i at maskray dot me> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |29563


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=29563
[Bug 29563] gdb: support zstd for SHF_COMPRESSED debug sections
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584
Bug 29584 depends on bug 29563, which changed state.

Bug 29563 Summary: gdb: support zstd for SHF_COMPRESSED debug sections
https://sourceware.org/bugzilla/show_bug.cgi?id=29563

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

--- Comment #1 from Fangrui Song <i at maskray dot me> ---
If this is a good idea, can we get an agreement about the format even if the
actual implementation may take a while?

xz compressed .gnu_debugdata can be used this way:

objcopy --only-keep-debug a a.debug
xz a.debug
objcopy -S --add-section=.gnu_debugdata=a.debug.xz a a.stripped

The data has a header magic of 0xFD, '7', 'z', 'X', 'Z', 0x00.

I suggest that if the first 4 bytes are little-endian value 0xFD2FB528 (zstd
header), than interpret .gnu_debugdata as zstd compressed, i.e.:

objcopy --only-keep-debug a a.debug
zstd a.debug
objcopy -S --add-section=.gnu_debugdata=a.debug.zstd a a.stripped

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at sourceware dot org

--- Comment #2 from Tom Tromey <tromey at sourceware dot org> ---
I haven't looked at the implementation but I wonder if this
is perhaps more of a BFD issue than a gdb one?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

--- Comment #3 from Fangrui Song <i at maskray dot me> ---
(In reply to Tom Tromey from comment #2)
> I haven't looked at the implementation but I wonder if this
> is perhaps more of a BFD issue than a gdb one?

binutils/BFD do not need to know .gnu_debugdata. Their directories don't even
contain the string ".gdb_debugdata" :)

The only file needing a change is in gdb: gdb/minidebug.c (see commit
b181bbd0d6ae15cfa0b4a8d0d426f550dd6b398f).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

--- Comment #4 from Mark Wielaard <mark at klomp dot org> ---
If we are changing the format, Can't we use standard ELF section compression?
That supports zstd these days. Then the .gnu_debugdata can be just a normal ELF
section, that is a mini-ELF image, which happens to be zstd compressed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

--- Comment #5 from Fangrui Song <i at maskray dot me> ---
(In reply to Mark Wielaard from comment #4)
> If we are changing the format, Can't we use standard ELF section compression?
> That supports zstd these days. Then the .gnu_debugdata can be just a normal
> ELF section, that is a mini-ELF image, which happens to be zstd compressed.

We can do that but it likely doesn't provide an overall advantage.

The .gnu_debugdata content is an ELF file, not a regular section.
We don't benefit from information like ch_type/ch_size/ch_addrline
in the Chdr header.

To embed an ELF file (with .symtab, .strtab, and optionally .debug_*), we need
one `objcopy --add-section` command, then one `eu-elfcompress -n` command (with
no binutils equivalent https://sourceware.org/bugzilla/show_bug.cgi?id=27452).
Even eu-elfcompress -n doesn't provide the various zstd options that a
distribution
may toggle. Also, this process involves two binary manipulation commands on
the ELF file.

If we use `zstd a.debug; objcopy -S --add-section=.gnu_debugdata=a.debug.zstd a
a.stripped`,
there is just one binary manipulation command, and we can toggle zstd options
easily.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

--- Comment #6 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Fangrui Song from comment #3)

> The only file needing a change is in gdb: gdb/minidebug.c (see commit
> b181bbd0d6ae15cfa0b4a8d0d426f550dd6b398f).

No, there's also libbacktrace in gcc.
It was probably a mistake to even let this not be gzip.
We were chastised for this when the libbacktrace patch went in.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

--- Comment #7 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Fangrui Song from comment #5)
> (In reply to Mark Wielaard from comment #4)
> > If we are changing the format, Can't we use standard ELF section compression?
> > That supports zstd these days. Then the .gnu_debugdata can be just a normal
> > ELF section, that is a mini-ELF image, which happens to be zstd compressed.
> 
> We can do that but it likely doesn't provide an overall advantage.
>
> The .gnu_debugdata content is an ELF file, not a regular section.
> We don't benefit from information like ch_type/ch_size/ch_addrline
> in the Chdr header.

I think it does provide an advantage because programs already need to handle
standard compressed Elf sections, so they don't need any special code for
decompressing this (assuming we'll drop the old compression scheme eventually).

> To embed an ELF file (with .symtab, .strtab, and optionally .debug_*), we
> need
> one `objcopy --add-section` command, then one `eu-elfcompress -n` command
> (with
> no binutils equivalent
> https://sourceware.org/bugzilla/show_bug.cgi?id=27452).
> Even eu-elfcompress -n doesn't provide the various zstd options that a
> distribution may toggle.

There is already support in eu-elfcompress, it will show up in the next release
(0.189, hopefully later this month):

  -n, --name=SECTION         SECTION name to (de)compress, SECTION is an
                             extended wildcard pattern (defaults to
                             '.?(z)debug*')
[...]
  -t, --type=TYPE            What type of compression to apply. TYPE can be
                             'none' (decompress), 'zlib' (ELF ZLIB compression,
                             the default, 'zlib-gabi' is an alias), 'zlib-gnu'
                             (.zdebug GNU style compression, 'gnu' is an alias)
                             or 'zstd' (ELF ZSTD compression)

> Also, this process involves two binary manipulation commands on
> the ELF file.
> 
> If we use `zstd a.debug; objcopy -S
> --add-section=.gnu_debugdata=a.debug.zstd a a.stripped`,
> there is just one binary manipulation command, and we can toggle zstd
> options easily.

It is slightly more that that. See debugedit find-debuginfo add_mini how the
section is created:
https://sourceware.org/git/?p=debugedit.git;a=blob;f=scripts/find-debuginfo.in;hb=HEAD#l332

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

--- Comment #8 from Fangrui Song <i at maskray dot me> ---
(In reply to Mark Wielaard from comment #7)
> > To embed an ELF file (with .symtab, .strtab, and optionally .debug_*), we
> > need
> > one `objcopy --add-section` command, then one `eu-elfcompress -n` command
> > (with
> > no binutils equivalent
> > https://sourceware.org/bugzilla/show_bug.cgi?id=27452).
> > Even eu-elfcompress -n doesn't provide the various zstd options that a
> > distribution may toggle.
> 
> There is already support in eu-elfcompress, it will show up in the next
> release (0.189, hopefully later this month):
> 
>   -n, --name=SECTION         SECTION name to (de)compress, SECTION is an
>                              extended wildcard pattern (defaults to
>                              '.?(z)debug*')
> [...]
>   -t, --type=TYPE            What type of compression to apply. TYPE can be
>                              'none' (decompress), 'zlib' (ELF ZLIB
> compression,
>                              the default, 'zlib-gabi' is an alias),
> 'zlib-gnu'
>                              (.zdebug GNU style compression, 'gnu' is an
> alias)
>                              or 'zstd' (ELF ZSTD compression)

I have seen
https://sourceware.org/git/?p=elfutils.git;a=commit;h=a5b07cdf9c491fb7a4a16598c482c68b718f59b9
(support ZSTD compression algorithm). I am thinking about Operation modifiers
like the compression level, window size, dictionary, etc. A Linux distribution
may have many packages which are not used for debugging, spending additional
time on the producer side is probably a win. Some use cases may prefer a fast
producer.

> > Also, this process involves two binary manipulation commands on
> > the ELF file.
> > 
> > If we use `zstd a.debug; objcopy -S
> > --add-section=.gnu_debugdata=a.debug.zstd a a.stripped`,
> > there is just one binary manipulation command, and we can toggle zstd
> > options easily.
> 
> It is slightly more that that. See debugedit find-debuginfo add_mini how the
> section is created:
> https://sourceware.org/git/?p=debugedit.git;a=blob;f=scripts/find-debuginfo.
> in;hb=HEAD#l332

Notes. You are right:) I noticed this when writing
https://maskray.me/blog/2022-10-30-distribution-of-debug-information#minidebuginfo
but I don't have experience with the package, so did not recall it off the top
my head...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/29584] gdb: support zstd compressed .gnu_debugdata

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29584

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.