[Bug dynamic-link/26151] New: Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[Bug dynamic-link/26151] New: Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

            Bug ID: 26151
           Summary: Add ET_* markers for debug files, and enhance ldd
                    (ld.so) to use them.
           Product: glibc
           Version: 2.32
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: dynamic-link
          Assignee: unassigned at sourceware dot org
          Reporter: carlos at redhat dot com
  Target Milestone: ---

Users use ldd to process directories full of files.

In some cases ldd is used to process what appear to be ELF files.

These ELF files are actually debuginfo objects that are only intended to be
used with debuggers.

The implementation of debuginfo objects is missing clear identification
information because it does not allow for the easy disambiguation of these ELF
files as debuginfo. When ldd is run on these objects files the corrupt and
invalid ELF data leads to crashes in ldd as it considers the objects to be
valid ELF files (they contain valid ELF headers).

There are a few solutions to this problem:

(a) Duplicate in ldd what eu-elfclassify does. In an attempt to create a robust
identification tooling for object files Mark Wielaard and Florian Weimer
created eu-elfclassify. This solution is conceptually wrong for ldd which
should need to look at section headers, but would need to in order to duplicate
what eu-elfclassify does.

(b) Add ET_* markup for debuginfo files to clearly mark them as debuginfo files
and avoid the problem by teaching the downstream tools, ldd first, that such
files are not be loaded or handled like normal ELF files. From an ELF
perspective they are invalid and incomplete and must be parsed with different
kinds of rules than normal ELF files.

My preferred solution is (b), but doing that may take a long time to deploy.

The short-term alternative is (a) followed eventually by (b) to remove the
section header processing at a later date.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #1 from Carlos O'Donell <carlos at redhat dot com> ---
Other issues to consider at the same time:

(1) Kernel objects.
- In Solaris these have special markup to avoid reading them as-if they were
plain ET_REL objects.

(2) dwz multi files are also ET_REL.

One of the proposals for DWARFv5 is Split DWARF Objects (DWO) which is already
implemented in some parts of the GNU toolchain.

https://gcc.gnu.org/wiki/DebugFission
http://dwarfstd.org/ShowIssue.php?issue=130313.4

This would leave some parts of the DWARF info in the .dwo (ET_REL) file, so the
linker doesn't have to process it. Later these .dwo would be packaged together
in a .dwp file for use with debuggers.

---

In the case of (1) it's a distinct issue that needs a new markup. Solaris uses
DF_1_KMOD to mark kernel modules. This means that with
https://docs.oracle.com/cd/E88353_01/html/E37839/ld-1.html you can use -z type
to say: exec, kmod, reloc, shared, pie, and static. That means you don't have
to guess.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://sourceware.org/bugz
                   |                            |illa/show_bug.cgi?id=22101

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://sourceware.org/bugz
                   |                            |illa/show_bug.cgi?id=22136

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #2 from Carlos O'Donell <carlos at redhat dot com> ---
The binutils work is tracked here:
https://sourceware.org/bugzilla/show_bug.cgi?id=22136

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hjl.tools at gmail dot com

--- Comment #3 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Carlos O'Donell from comment #1)
> Other issues to consider at the same time:
> 
> (1) Kernel objects.
> - In Solaris these have special markup to avoid reading them as-if they were
> plain ET_REL objects.
> 
> (2) dwz multi files are also ET_REL.
> 
> One of the proposals for DWARFv5 is Split DWARF Objects (DWO) which is
> already implemented in some parts of the GNU toolchain.
> 
> https://gcc.gnu.org/wiki/DebugFission
> http://dwarfstd.org/ShowIssue.php?issue=130313.4
> 
> This would leave some parts of the DWARF info in the .dwo (ET_REL) file, so
> the linker doesn't have to process it. Later these .dwo would be packaged
> together in a .dwp file for use with debuggers.
> 
> ---
> 
> In the case of (1) it's a distinct issue that needs a new markup. Solaris
> uses DF_1_KMOD to mark kernel modules. This means that with
> https://docs.oracle.com/cd/E88353_01/html/E37839/ld-1.html you can use -z
> type to say: exec, kmod, reloc, shared, pie, and static. That means you
> don't have to guess.

I prefer DF_1_XXX over ET_XXX.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #4 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to H.J. Lu from comment #3)
> (In reply to Carlos O'Donell from comment #1)
> > In the case of (1) it's a distinct issue that needs a new markup. Solaris
> > uses DF_1_KMOD to mark kernel modules. This means that with
> > https://docs.oracle.com/cd/E88353_01/html/E37839/ld-1.html you can use -z
> > type to say: exec, kmod, reloc, shared, pie, and static. That means you
> > don't have to guess.
> 
> I prefer DF_1_XXX over ET_XXX.

It doesn't make sense to me.

The use of DF_1_* requires PT_DYANMIC/.dynamic and DT_FLAGS_1.

The PT_DYNAMIC/.dynamic is not present in debuginfo objects, it is stripped and
NOBITS and should come from the original executable. This makes it less than
useful to use as a way to self-identify the files.

Likewise kernel modules have no PT_DYNAMIC/.dyanmic because they are ET_REL. So
there is additional work to pursue a solution based on DF_1_* markup.

The DF_1_* markup is also semantically about marking up dynamic objects, and
debug info is not a dynamic object, it would make more sense to add new
fundamental ELF types since that is what they are.

Do you have any specific objection to a new ET_* markup?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #5 from H.J. Lu <hjl.tools at gmail dot com> ---
Who are the consumers of this kind of special debug files?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #6 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to H.J. Lu from comment #5)
> Who are the consumers of this kind of special debug files?

The general problem is as follows.

Downstream distributions need to build, split, and package debug information.

During the process of building the debug information, for example in Fedora,
you use a special scripts which include find-debuginfo.sh.

The point is that because source files change frequently and you don't want to
have to keep altering a static list of debug information files so the scripts
automatically generate these files.

In order to automate these processes there is a mix of file list scanning and
analysis of built files being done. In some cases the lists of files to be
scanned are inaccurate so some users use ldd to distinguish between certain
"types" of files.

We continue to get examples of cases where ldd can crash. My goal is to try
make ldd more robust against crashing:

(a) Tell users to stop using ldd and recommend eu-elfclassify. This action is
already taken and eu-elfclassify is getting used in places where we need robust
classification.

(b) Make ldd more robust. This action is not yet taken and is the reason I
filed this bug. If the dynamic loader could easily exclude processing certain
files then existing user scripts using ldd to examine files would continue to
work and be more robust.

(c) Eventually replace ldd with eu-ldd that can do more complex analysis and
assure users that they will never run code, and support container inspection.

In the short term we've done (a), I think (b) has value, and I think (c) is the
right long-term solution.

To answer your question the user of the markup is going to be the dynamic
loader ldd implementation (LD_DEBUG_TRACED_OBJECTS), eu-elfclassify could use
the same markup to avoid guessing, future eu-ldd would use it also.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #7 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to H.J. Lu from comment #5)
> Who are the consumers of this kind of special debug files?

Split debug files are used by debuggers.

The problem is that tooling to build packages has to manipulate lists of files
and distinguish between different types of files in a robust manner to build
installed file lists.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #8 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Carlos O'Donell from comment #7)
> (In reply to H.J. Lu from comment #5)
> > Who are the consumers of this kind of special debug files?
> 
> Split debug files are used by debuggers.
> 
> The problem is that tooling to build packages has to manipulate lists of
> files and distinguish between different types of files in a robust manner to
> build installed file lists.

Since debuggers can exam special debug files, will a single ET_GNU_DEBUG
be sufficient?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #9 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to H.J. Lu from comment #8)
> (In reply to Carlos O'Donell from comment #7)
> > (In reply to H.J. Lu from comment #5)
> > > Who are the consumers of this kind of special debug files?
> > 
> > Split debug files are used by debuggers.
> > 
> > The problem is that tooling to build packages has to manipulate lists of
> > files and distinguish between different types of files in a robust manner to
> > build installed file lists.
> 
> Since debuggers can exam special debug files, will a single ET_GNU_DEBUG
> be sufficient?

Yes, I think that would suffice.

When the ELF file requires completely different semantics for processing then
my opinion is that the ET_* value should change.

Likewise I think we need ET_GNU_KMOD, but given that we haven't seen any
problem there, and *.ko's are limited to specific packages, the issue isn't as
general as *all* packages having to handle split debug information.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #10 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Carlos O'Donell from comment #9)
> (In reply to H.J. Lu from comment #8)
> > (In reply to Carlos O'Donell from comment #7)
> > > (In reply to H.J. Lu from comment #5)
> > > > Who are the consumers of this kind of special debug files?
> > > 
> > > Split debug files are used by debuggers.
> > > 
> > > The problem is that tooling to build packages has to manipulate lists of
> > > files and distinguish between different types of files in a robust manner to
> > > build installed file lists.
> > 
> > Since debuggers can exam special debug files, will a single ET_GNU_DEBUG
> > be sufficient?
> 
> Yes, I think that would suffice.
> 
> When the ELF file requires completely different semantics for processing
> then my opinion is that the ET_* value should change.

Let's make a proposal for ET_DEBUG to gABI.  If it is rejected, we send
ET_GNU_DEBUG to gnu-gABI.

> Likewise I think we need ET_GNU_KMOD, but given that we haven't seen any
> problem there, and *.ko's are limited to specific packages, the issue isn't
> as general as *all* packages having to handle split debug information.

How different are .ko files from .o files?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #11 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to H.J. Lu from comment #10)
> (In reply to Carlos O'Donell from comment #9)
> > (In reply to H.J. Lu from comment #8)
> > > (In reply to Carlos O'Donell from comment #7)
> > > > (In reply to H.J. Lu from comment #5)
> > > > > Who are the consumers of this kind of special debug files?
> > > > 
> > > > Split debug files are used by debuggers.
> > > > 
> > > > The problem is that tooling to build packages has to manipulate lists of
> > > > files and distinguish between different types of files in a robust manner to
> > > > build installed file lists.
> > > 
> > > Since debuggers can exam special debug files, will a single ET_GNU_DEBUG
> > > be sufficient?
> > 
> > Yes, I think that would suffice.
> > 
> > When the ELF file requires completely different semantics for processing
> > then my opinion is that the ET_* value should change.
> 
> Let's make a proposal for ET_DEBUG to gABI.  If it is rejected, we send
> ET_GNU_DEBUG to gnu-gABI.

Sounds good to me.

> > Likewise I think we need ET_GNU_KMOD, but given that we haven't seen any
> > problem there, and *.ko's are limited to specific packages, the issue isn't
> > as general as *all* packages having to handle split debug information.
> 
> How different are .ko files from .o files?

No different really, some special sections, but that's all. They are already
marked ET_REL, so the problem of "correct identification" is academic.

I don't consider *.ko to be a problem since ldd and other tools won't confuse
them with a complete ELF shared library or program that can be correctly
interpreted.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

--- Comment #12 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Carlos O'Donell from comment #11)
> > 
> > Let's make a proposal for ET_DEBUG to gABI.  If it is rejected, we send
> > ET_GNU_DEBUG to gnu-gABI.
> 
> Sounds good to me.
> 

Done.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/26151] Add ET_* markers for debug files, and enhance ldd (ld.so) to use them.

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26151

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.