Re: Analyzing process and core dump files in changed root environments

Re: Analyzing process and core dump files in changed root environments

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2693 bytes --]

Hi Jakub,

On Tue, 2016-11-29 at 13:37 +0100, h wrote:
> I have a question regarding analyzing processes running in a change root 
> environment or in an own mount namespace with re-mounted /.
> I want to debug a process running in a non-system root using elfutils or gdb
> from system root, and vice versa.
> More precisely, I want to create a container shipping ABRT + GDB + elfutils 
> and the container should allow users to analyze core dump files on Fedora 
> Atomic. I need to run eu-stack on a core dump file generated by a program 
> from system root and I need to run it from a container (non-system root).
> 
> For example, if I run the following command on Fedora Atomic 25, it crashes 
> in the system namespace and ABRT saves its core file to /var/spool/abrt/
> ccpp* in ABRT's namespace. If I enter the ABRT container and run eu-stack on
> the core dump file, I get an error because the container misses some 
> libraries:
> [...]
> You may ask what is the binary argument. It's a copy of the crashed program.
> It can be replaced with the `/host/bin/ostree` path (I bind mount the hots' 
> / to the container's /host).
> 
> GDB allows me to configure 'sysroot', so it knows where to look for 
> libraries.  Thus if I run GDB on the same core dump file, I get the 
> following ouptut:
> $ gdb --batch --ex 'set sysroot /host' --ex 'file binary' --ex 'core 
> coredump' --ex 'bt'
> [...]
> Do elfutils support such an option too?

We already talked briefly on irc. And the short answer is:  No, at the
moment not. Thanks for filing
https://bugzilla.redhat.com/show_bug.cgi?id=1399637

The slightly longer versions is: No, not "globally" and only for some
files/types. You can set how to find debuginfo files by providing a path
or alternative callback when setting up a Dwfl. See Dwfl_Callbacks
debuginfo_path and find_debuginfo. Usually you would use
dwfl_standard_find_debuginfo for this. You can also provide a find_elf
callback that finds the Elf image for a module. Usually you would use
either dwfl_build_id_find_elf, dwfl_linux_proc_find_elf or
dwfl_linux_kernel_find_elf for that.

We probably should provide alternatives to those that take some sysroot.
But there are some parts inside the link_map scanning code that cannot
easily use those callbacks at the moment and use direct open calls that
also would need sysroot "hooks" (there are already some XXX hook for
sysroot comments).

But maybe the above is too flexible. Which (types of) files would you
expect a "sysroot" setting to affect? Just direct file paths, lookups
through build-ids, separate debuginfo files, kernel module lookups,
etc.?

Thanks,

Mark

Analyzing process and core dump files in changed root environments

[h]

[-- Attachment #1: Type: text/plain, Size: 2770 bytes --]

Hello,

I have a question regarding analyzing processes running in a change root 
environment or in an own mount namespace with re-mounted /.
I want to debug a process running in a non-system root using elfutils or gdb
from system root, and vice versa.
More precisely, I want to create a container shipping ABRT + GDB + elfutils 
and the container should allow users to analyze core dump files on Fedora 
Atomic. I need to run eu-stack on a core dump file generated by a program 
from system root and I need to run it from a container (non-system root).

For example, if I run the following command on Fedora Atomic 25, it crashes 
in the system namespace and ABRT saves its core file to /var/spool/abrt/
ccpp* in ABRT's namespace. If I enter the ABRT container and run eu-stack on
the core dump file, I get an error because the container misses some 
libraries:

$ ostree admin instutil grub2-generate

$ eu-stack --core coredump -e binary                                        
           
PID 1693 - core
TID 1693:
#0  0x00007fbd9c7c892f raise
#1  0x00007fbd9c7ca52a abort
#2  0x00007fbd9d2b0cc5 g_assertion_message
#3  0x00007fbd9d2b0d5a g_assertion_message_expr
#4  0x00007fbd9e713e7e
eu-stack: dwfl_thread_getframes tid 1693 at 0x7fbd9e713e7d in /usr/lib64/
libostree-1.so.1.0.0: Callback returned failure

You may ask what is the binary argument. It's a copy of the crashed program.
It can be replaced with the `/host/bin/ostree` path (I bind mount the hots' 
/ to the container's /host).

GDB allows me to configure 'sysroot', so it knows where to look for 
libraries.  Thus if I run GDB on the same core dump file, I get the 
following ouptut:
$ gdb --batch --ex 'set sysroot /host' --ex 'file binary' --ex 'core 
coredump' --ex 'bt'
Core was generated by `ostree admin instutil grub2-generate'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fbd9c7c892f in raise () from /host/lib64/libc.so.6
#0  0x00007fbd9c7c892f in raise () from /host/lib64/libc.so.6
#1  0x00007fbd9c7ca52a in abort () from /host/lib64/libc.so.6
#2  0x00007fbd9d2b0cc5 in g_assertion_message () from /host/lib64/libglib-
2.0.so.0
#3  0x00007fbd9d2b0d5a in g_assertion_message_expr () from /host/lib64/
libglib-2.0.so.0
#4  0x00007fbd9e713e7e in _ostree_bootloader_grub2_generate_config () from /
host/lib64/libostree-1.so.1
#5  0x000055b04f4300c1 in ot_admin_instutil_builtin_grub2_generate ()
#6  0x000055b04f42da53 in ot_admin_builtin_instutil ()
#7  0x000055b04f41fd93 in ostree_builtin_admin ()
#8  0x000055b04f4294d7 in ostree_run ()
#9  0x000055b04f41fae0 in main ()

Do elfutils support such an option too?



Kind regards,
Jakub Filak


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3124 bytes --]