[Bug libdw/30980] New: offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[Bug libdw/30980] New: offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[cebtenzzre at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

            Bug ID: 30980
           Summary: offline.c:53: dwfl_offline_section_address: Assertion
                    `mod->e_type == ET_REL' failed.
           Product: elfutils
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libdw
          Assignee: unassigned at sourceware dot org
          Reporter: cebtenzzre at gmail dot com
                CC: elfutils-devel at sourceware dot org
  Target Milestone: ---

Created attachment 15181
  --> https://sourceware.org/bugzilla/attachment.cgi?id=15181&action=edit
The file that elfutils crashes on while trying to read debug info.

I ran into this crash while systemd-coredump was trying to process a coredump
from evolution.


evolution backtrace:

#0  0x0000000000000000 in  ()
#1  0x00007f8d92ec6ba1 in compute_next_step (assistant=0x55a0bfe3ea60)
    at ../gtk/gtk/gtkassistant.c:1035
#2  gtk_assistant_next_page (assistant=0x55a0bfe3ea60) at
../gtk/gtk/gtkassistant.c:1610
#3  0x00007f8d7a34ad5e in  () at /usr/lib/evolution/libevolution-mail.so
#4  0x00007f8d8e356252 in e_simple_async_result_complete ()
    at /usr/lib/evolution/libevolution-util.so
#5  0x00007f8d8e3562b9 in  () at /usr/lib/evolution/libevolution-util.so
#6  0x00007f8d93834f19 in g_main_dispatch (context=0x55a0be1fefc0)
    at ../glib/glib/gmain.c:3476
#7  0x00007f8d938932b7 in g_main_context_dispatch_unlocked
(context=0x55a0be1fefc0)
    at ../glib/glib/gmain.c:4284
#8  g_main_context_iterate_unlocked.isra.0
    (context=0x55a0be1fefc0, block=block@entry=1, dispatch=dispatch@entry=1,
self=<optimized out>) at ../glib/glib/gmain.c:4349
#9  0x00007f8d93835b47 in g_main_loop_run (loop=0x55a0bed6cde0)
    at ../glib/glib/gmain.c:4551
#10 0x00007f8d930337ed in gtk_main () at ../gtk/gtk/gtkmain.c:1329
#11 0x000055a0bcea857f in main ()


elfutils backtrace:

#5  0x00007fb1fae54d26 in __assert_fail
(assertion=assertion@entry=0x7fb1fa4e1c30 "mod->e_type == ET_REL",
file=file@entry=0x7fb1fa4e1c26 "offline.c", line=line@entry=53, 
    function=function@entry=0x7fb1fa4fded0 <__PRETTY_FUNCTION__.0.lto_priv.43>
"dwfl_offline_section_address") at assert.c:101
#6  0x00007fb1fa4c3c30 in dwfl_offline_section_address (mod=<optimized out>,
userdata=<optimized out>, modname=<optimized out>, base=<optimized out>,
secname=<optimized out>, 
    shndx=<optimized out>, shdr=0x7fff255196b0, addr=0x7fff255196c0) at
../libdwfl/offline.c:53
#7  0x00007fb1fa4c859c in __libdwfl_relocate_value
(mod=mod@entry=0x5624c5cd80d0, elf=elf@entry=0x5624c605dc30,
shstrndx=shstrndx@entry=0x7fff25519760, shndx=4, 
    value=value@entry=0x7fff25519768) at ../libdwfl/relocate.c:72
#8  0x00007fb1fa4c8772 in find_elf_build_id (mod=mod@entry=0x5624c5cd80d0,
e_type=1, elf=elf@entry=0x5624c605dc30,
build_id_bits=build_id_bits@entry=0x7fff25519890, 
    build_id_elfaddr=build_id_elfaddr@entry=0x7fff25519888,
build_id_len=build_id_len@entry=0x7fff25519884) at
../libdwelf/dwelf_elf_gnu_build_id.c:113
#9  0x00007fb1fa4c88e3 in __libdwfl_find_elf_build_id
(mod=mod@entry=0x5624c5cd80d0, elf=0x5624c605dc30,
build_id_bits=build_id_bits@entry=0x7fff25519890, 
    build_id_elfaddr=build_id_elfaddr@entry=0x7fff25519888,
build_id_len=build_id_len@entry=0x7fff25519884) at
../libdwelf/dwelf_elf_gnu_build_id.c:142
#10 0x00007fb1fa4c8992 in __libdwfl_find_build_id
(mod=mod@entry=0x5624c5cd80d0, set=set@entry=false, elf=<optimized out>) at
../libdwfl/dwfl_module_build_id.c:70
#11 0x00007fb1fa4c939e in validate (debuglink_crc=<optimized out>,
check=<optimized out>, fd=<optimized out>, mod=0x5624c5cd80d0) at
../libdwfl/find-debuginfo.c:141
#12 find_debuginfo_in_path (mod=mod@entry=0x5624c5cd80d0,
file_name=file_name@entry=0x5624c5cd82b0
"/usr/lib/libjavascriptcoregtk-4.1.so.0", 
    debuglink_file=debuglink_file@entry=0x7faf25bbedc8 "crti.o.debug",
debuglink_crc=debuglink_crc@entry=465747295,
debuginfo_file_name=debuginfo_file_name@entry=0x5624c5cd8128)
    at ../libdwfl/find-debuginfo.c:326
#13 0x00007fb1fa4ccfc0 in dwfl_standard_find_debuginfo (mod=0x5624c5cd80d0,
userdata=<optimized out>, modname=<optimized out>, base=<optimized out>, 
    file_name=0x5624c5cd82b0 "/usr/lib/libjavascriptcoregtk-4.1.so.0",
debuglink_file=0x7faf25bbedc8 "crti.o.debug", debuglink_crc=465747295,
debuginfo_file_name=0x5624c5cd8128)
    at ../libdwfl/find-debuginfo.c:386
#14 0x00007fb1fa4c5b83 in find_debuginfo (mod=mod@entry=0x5624c5cd80d0) at
../libdwfl/dwfl_module_getdwarf.c:538
#15 0x00007fb1fa4cfa60 in find_dw (mod=0x5624c5cd80d0) at
../libdwfl/dwfl_module_getdwarf.c:1412
#16 dwfl_module_getdwarf (mod=mod@entry=0x5624c5cd80d0, bias=0x7fff25519b88) at
../libdwfl/dwfl_module_getdwarf.c:1446
#17 0x00007fb1fa4d8bd8 in dwfl_module_addrdie (mod=0x5624c5cd80d0,
addr=140245923026755, bias=<optimized out>) at
../libdwfl/dwfl_module_addrdie.c:38
#18 0x00007fb1fb0ed9e9 in frame_callback (frame=<optimized out>,
userdata=0x7fff25519e50) at ../systemd-stable/src/shared/elf-util.c:203
#19 0x00007fb1fa4de175 in dwfl_thread_getframes (thread=0x7fff25519ce0,
callback=0x7fb1fb0ed920 <frame_callback>, arg=0x7fff25519e50) at
../libdwfl/dwfl_frame.c:428
#20 0x00007fb1fb0edd08 in thread_callback (thread=0x7fff25519ce0,
userdata=0x7fff25519e50) at ../systemd-stable/src/shared/elf-util.c:262
#21 0x00007fb1fa4d4275 in dwfl_getthreads (dwfl=0x5624c5a791b0,
callback=0x7fb1fb0edc90 <thread_callback>, arg=0x7fff25519e50) at
../libdwfl/dwfl_frame.c:284
#22 0x00007fb1fb0f3e44 in parse_core (ret_package_metadata=<optimized out>,
ret=<optimized out>, executable=<optimized out>, fd=<optimized out>)
    at ../systemd-stable/src/shared/elf-util.c:619
#23 parse_elf (ret_package_metadata=0x7fff25519db8, ret=<optimized out>,
executable=<optimized out>, fd=<optimized out>) at
../systemd-stable/src/shared/elf-util.c:665
#24 parse_elf_object (fd=fd@entry=6, executable=0x5624c5a6b41d
"/usr/bin/evolution", fork_disable_dump=<optimized out>,
ret=ret@entry=0x7fff25519fa8, 
    ret_package_metadata=ret_package_metadata@entry=0x7fff25519fb0) at
../systemd-stable/src/shared/elf-util.c:810
#25 0x00005624c4125ff4 in submit_coredump
(context=context@entry=0x7fff2551a5e0, iovw=iovw@entry=0x7fff2551a560,
input_fd=input_fd@entry=5) at ../systemd-stable/src/coredump/coredump.c:897
#26 0x00005624c41285f2 in process_socket (fd=3) at
../systemd-stable/src/coredump/coredump.c:1134
#27 0x00005624c411fc31 in run (argv=0x7fff2551a838, argc=1) at
../systemd-stable/src/coredump/coredump.c:1522
#28 main (argc=1, argv=0x7fff2551a838) at
../systemd-stable/src/coredump/coredump.c:1528


The crash happens while elfutils tries to call dwfl_offline_section_address on
/usr/lib/libjavascriptcoregtk-4.1.so.0. That function is expecting a
relocatable file (ET_REL), but that is a shared object (ET_DYN).

A copy of that file is attached.

Downstream bug report: https://github.com/systemd/systemd/issues/29585

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

--- Comment #1 from Mark Wielaard <mark at klomp dot org> ---
Comment on attachment 15181
  --> https://sourceware.org/bugzilla/attachment.cgi?id=15181
The file that elfutils crashes on while trying to read debug info.

One slightly odd thing, although probably not the real cause of this bug, is
that the build-id stored in the libjavascriptcoregtk-4.1.so.0.4.10 is only 8
bytes wide:

Note section [ 1] '.note.gnu.build-id' of 24 bytes at offset 0x318:
  Owner          Data size  Type
  GNU                    8  GNU_BUILD_ID
    Build ID: 289bbb6d9dfaf6e3

Normally they are ~20 bytes wide. Which guarantees they are globally unique.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

--- Comment #2 from Mark Wielaard <mark at klomp dot org> ---
This however is really odd and might explain why we get onto an ET_REL path:

$ eu-readelf -x .gnu_debuglink
~/Downloads/libjavascriptcoregtk-4.1.so.0.4.10.zst

Hex dump of section [28] '.gnu_debuglink', 84 bytes at offset 0x1fbedc8:
  0x00000000 63727469 2e6f2e64 65627567 00000000 crti.o.debug....
  0x00000010 2aa4dad4 63727462 6567696e 532e6f2e *...crtbeginS.o.
  0x00000020 64656275 67000000 39c05e5d 63727465 debug...9.^]crte
  0x00000030 6e64532e 6f2e6465 62756700 6b271cc4 ndS.o.debug.k'..
  0x00000040 6372746e 2e6f2e64 65627567 00000000 crtn.o.debug....
  0x00000050 5fbdc21b                            _...

So when looking for the separate .debug file, when we don't have a build-id
match, we will search for crti.o.debug...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

--- Comment #3 from Mark Wielaard <mark at klomp dot org> ---
Would you happen to know where systemd-stable/src/coredump/coredump.c and
systemd-stable/src/shared/elf-util.c come from? I am trying to figure out how
the dwfl has been setup that parse_core uses, but I cannot find those files or
the parse_core function in current systemd.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

--- Comment #4 from Mark Wielaard <mark at klomp dot org> ---
O, apparently systemd isn't updated on freedesktop.org anymore. So my git repo
is stale. The copy on github (sigh) does have those files and the parse_core
function.

The dwfl setup is done with:

        const Dwfl_Callbacks callbacks = {
                .find_elf = sym_dwfl_build_id_find_elf,
                .section_address = sym_dwfl_offline_section_address,
                .find_debuginfo = sym_dwfl_standard_find_debuginfo,
        };

Where sym_dwfl_offline_section_address is through some magic the libdw.so
dwfl_offline_section_address.

At least that explains why dwfl_offline_section_address is called.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org
           See Also|                            |https://github.com/systemd/
                   |                            |systemd/issues/29585

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[amerey at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

Aaron Merey <amerey at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |amerey at redhat dot com
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #5 from Aaron Merey <amerey at redhat dot com> ---
A workaround for this abort has been added in the following commit:

commit a34c5faad861efdd26d1c52b4f8d9d4077e03131
Author: Aaron Merey <amerey@redhat.com>
Date:   Wed Nov 1 16:15:16 2023 -0400

    dwfl_offline_section_address: replace asserts with early return

    dwfl_offline_section_address asserts that the current module is ET_REL.

    A possibly corrupt .gnu_debuglink can cause an abort by calling
    dwfl_offline_section_address on an ET_DYN module.

    Prevent this abort and similar ones by replacing
    dwfl_offline_section_address initial asserts with an early return.

    https://sourceware.org/bugzilla/show_bug.cgi?id=30980

    Signed-off-by: Aaron Merey <amerey@redhat.com>

If you are still having issues with this please reopen this bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

--- Comment #6 from Mark Wielaard <mark at klomp dot org> ---
Thanks, that seems to be the correct thing to do here.

Note that there are a couple more asserts on this code path. In particular in
__libdwfl_find_elf_build_id there is what apparently is the root of the
confusion here:

  // MOD->E_TYPE is zero here.
  assert (ehdr->e_type != ET_REL || mod != NULL);

Which passes because mod != NULL. But probably should have been a red flag.
This is my fault, I refactored this code and kept these asserts without fully
understanding why there were (not) needed.

We should go over all asserts in the library code and rewrite/remove them. It
is never a good thing to trigger an abort in library code.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libdw/30980] offline.c:53: dwfl_offline_section_address: Assertion `mod->e_type == ET_REL' failed.

[amerey at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30980

--- Comment #7 from Aaron Merey <amerey at redhat dot com> ---
(In reply to Mark Wielaard from comment #6)
> We should go over all asserts in the library code and rewrite/remove them.
> It is never a good thing to trigger an abort in library code.

I created a PR for this: https://sourceware.org/bugzilla/show_bug.cgi?id=31027

-- 
You are receiving this mail because:
You are on the CC list for the bug.