Re: Enable gdb to open Linux kernel dumps

[Jeff Mahoney <jeffm@suse.com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/1/16 10:01 AM, Jeff Mahoney wrote:
> On 2/1/16 9:32 AM, Ales Novak wrote:
>> On 2016-2-1 12:51, Kieran Bingham wrote:
> 
>>>
>>> On 01/02/16 11:27, Kieran Bingham wrote:
>>>> Hi Ales,
>>>>
>>>> I'm just checking out your tree now to try locally.
>>>>
>>>> It sounds like there is a high level of cross over in our work,
>>>> but I believe our work can complement each other's if we work
>>>> together.
> 
>> Yes. Our primary intention is to open kdumps (i.e. dead images of
>> the fallen kernels), but what can be shared between live and dead
>> kernel debugging, should be shared...
> 
>>>> On 31/01/16 21:44, Ales Novak wrote:
>>>>> Following patches are adding basic ability to access Linux
>>>>> kernel dumps using the libkdumpfile library. They're creating
>>>>> new target "kdump", so all one has to do is to provide
>>>>> appropriate debuginfo and then run "target kdump
>>>>> /path/to/vmcore".
>>>>>
>>>>> The tasks of the dumped kernel are mapped to threads in gdb.
>>>>>
>>>>> Except for that, there's a code adding understanding of Linux
>>>>> SLAB memory allocator, which means we can tell for the given
>>>>> address to which SLAB does it belong, or list objects for
>>>>> give SLAB name - and more.
>>>>>
>>>>> Patches are against "gdb-7.10-release" (but will apply
>>>>> elsewhere).
>>>>>
>>>>> Note: registers of task are fetched accordingly - either from
>>>>> the dump metadata (the active tasks) or from their stacks. It
>>>>> should be noted that as this mechanism varies amongst the
>>>>> kernel versions and configurations, my naive implementation
>>>>> currently covers only the dumps I encounter, handling of
>>>>> different kernel versions is to be added.
>>>> In the work that I am doing, I had expected this to be done in
>>>> python for exactly this reason. The kernel version specifics,
>>>> (and architecture specifics) can then live alongside their
>>>> respective trees.
>>>>> In the near future, our plan is to remove the clumsy C-code
>>>>> handling this and reimplement it in Python - only the binding
>>>>> to certain gdb structures (e.g. thread, regcache) has to be
>>>>> added. A colleague of mine is already working on that.
>>>> This sounds exactly like the work I am doing right now. Could
>>>> you pass on my details to your colleague so we can discuss?
>>>
>>> Aha, or is your colleague Andreas Arnez? I'm just about to reply
>>> to his mail over on gbd@ next.
> 
>> No, it's Jeff Mahoney. His current efforts, which include Python
>> binding to threads' regcaches and more, are at:
> 
>> https://jeffm.io/git/cgit.cgi/gnu/binutils-gdb/log/
> 
>> And yes, you're right I've incorrectly removed autorship from some
>> of his older patches (which in fact are not necessary for the
>> current gdb-kdump to work, they are extending the Python binding).
> 
>> And as you've already found, his older patches are at:
> 
>> https://github.com/jeffmahoney/py-crash
> 
> Hi guys -
> 
> Ales gave me the heads up that you were discussing these.  The github
> repo is old and I haven't touched it in a year or so.  The link to my
> git server is the active one, but I should be clear that this is
> currently a WIP from my perspective.  I've been doing my work in the
> rel-7.10.1-kdump branch, which is based on the gdb-7.10.1-release tag,
> plus some SUSE patches to handle build-ids and external debuginfo files.
> 
> This branch is subject to rebasing as I make progess, but there should
> be a stable base underneath it that I can condense and put into a
> separate branch for public consumption.

Hi guys -

I spent a decent amount of time on this in the past week
or so and have something usable to present.  At least in
terms of baseline functionality.  My branches have been
churning quite a bit, much to the annoyance of my colleagues,
I'm sure. :)

Here's the end result:

https://jeffm.io/git/cgit.cgi/gnu/binutils-gdb/log/?h=snapshots/python-working-target-20160209

There is no more kdump.c or py-kdump.c.  All the functionality added to gdb itself should be sufficiently generic, though I expect there may be some discussion points.  The target itself
is implemented as an interface between the gdb python API and the libkdump python API, entirely in Python.  As of this evening, it does depend on this commit:

https://github.com/jeffmahoney/libkdumpfile/commit/9488340227f3d69c893599101d8bdae1106da44b

... on top of the current libkdumpfile master branch.

The interface is such now that my test.py script consists of:
import kdump.target
kdump.target.Target('/var/crash/2015-12-08-15:18/vmcore')

... and I can do:
$ gdb /var/crash/2015-12-08-15\:18/vmlinux-3.16.7-29-desktop
[...]
(gdb) source ../test.py
kdump (<open file '/var/crash/2015-12-08-15:18/vmcore', mode 'r' at 0x7f64c1f32ae0>)
(gdb) thread 409
[Switching to thread 409 (pid 8110)]
#0  0xffffffff8161f172 in context_switch (next=<optimized out>,
    prev=<optimized out>, rq=<optimized out>) at ../kernel/sched/core.c:2334
2334	../kernel/sched/core.c: No such file or directory.
(gdb) bt
#0  0xffffffff8161f172 in context_switch (next=<optimized out>,
    prev=<optimized out>, rq=<optimized out>) at ../kernel/sched/core.c:2334
#1  __schedule () at ../kernel/sched/core.c:2795
#2  0xffffffff8161f62a in schedule () at ../kernel/sched/core.c:2831
#3  0xffffffff8105e9ea in do_wait (wo=0xffff880136a7ff08)
    at ../kernel/exit.c:1506
#4  0xffffffff8105fa87 in SYSC_wait4 (ru=<optimized out>,
    options=<optimized out>, stat_addr=<optimized out>, upid=<optimized out>)
    at ../kernel/exit.c:1615
#5  SyS_wait4 (upid=<optimized out>, stat_addr=140728330276444,
    options=<optimized out>, ru=<optimized out>) at ../kernel/exit.c:1584
#6  <signal handler called>
#7  0x00007fb93274ca5c in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb) python import crash.commands.log
(gdb) pydmesg
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.16.7-29-desktop (geeko@buildhost) (gcc version 4.8.3 20140627 [gcc-4_8-branch revision 212064] (SUSE Linux) ) #1 SMP PREEMPT Fri Oct 23 00:46:04 UTC 2015 (6be6a97)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.7-29-desktop root=UUID=886331b0-64cb-4f49-9db6-aa03562a8df0 eth0=dhcp console=tty0 console=ttyS0,115200 resume=/dev/disk/by-id/ata-GB0500EAFYL_WCASYE758932-part5 splash=silent quiet showopts crashkernel=1024M-:512M
[...]

It's a start.  Now that I have what I think should be a mostly stable base, I'm going to turn my efforts into wrangling my existing python crash projects into something that can work with this a bit better.  Obviously dmesg already works.  ps shouldn't be far behind.

- -Jeff

- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQIcBAEBAgAGBQJWuq1WAAoJEB57S2MheeWyfOsQAKZVllQ378T5untDOMuTLm4h
8tfiuq4+toDcNBwXjWEds8AwnAphSqV4Q/U61Z18adcNDTl2ajRvQHRGhH65DJHA
Nu532HSPl/4LzkwrLUs9KdKYX0ROK05jHWbqTvG8Bf98S/eWrTtRnhbGrxv7O9wK
dBPVoRwEsRlXfWakNUB4x8BiT7dtX3Sdx/buqz6yCVLEsCXs5M4keYWLzn8bzbmS
I/2M4XTfiZQOfImcCqWL7N7uds8EBZCIOmFbEFZ9hVXrbWKsakqvAJRofIyuNq9N
6gjTRjVgxt3Y/fTf96ol0tPJC/J7GIBv5qCfYX3Y58/jEu9Zm9oC2GN+r8mMMvuJ
lEklDn+7hV9wErh61stUOtr2qqIaZE/phH74dVj4S3+8HVZdP/BAvl03sJj71+ju
XDJVTqC+6+TYBEPpiGXJjdQ8LKvZ0aqY1KC+DskIjnLdCLutayFnn8kzVHQ7PAZZ
L/mPHZLGLaHmvllk35txLTYeVywkK4/JWn42EFmM9xggVKUt7rpEsKIXSe51fp5J
7kzZII+D3c/n2+0gQEqaenjeuZqFQMz30Ke7qPcNWXAHPN1hOd96TTQ++C4ePSH4
scgQqx9MKxfB1l8dM3Bi+nVY7dZ/z2gyU93ZPgcdw+n/UufcfpoXmuWESSskPwsx
V5hTGusS1rqTqTyy1p5W
=dp6N
-----END PGP SIGNATURE-----