Re: Query: Kdump: Core Image ELF Format

Re: Query: Kdump: Core Image ELF Format

[Vivek Goyal]

On Tue, 2005-03-08 at 11:00 -0700, Eric W. Biederman wrote: 
> vivek goyal <vgoyal@in.ibm.com> writes:
> 
> > Hi,
> > 
> > Kdump (A kexec based crash dumping mechanism) is going to export the
> > kernel core image in ELF format. ELF was chosen as a format, keeping in
> > mind that gdb can be used for limited debugging and "Crash" can be used
> > for advanced debugging.
> 
> When I suggested ELF for this purpose it was not so much that it was
> directly usable.  But rather it was an existing file format that could
> do the job, was well understood, and had enough extensibility
> through the PT_NOTES segment to handle the weird cases.
> 
> > Core image ELF headers are prepared before crash and stored at a safe
> > place in memory. These headers are retrieved over a kexec boot and final
> > elf core image is prepared for analysis. 
> > 
> > Given the fact physical memory can be dis-contiguous, One program header
> > of type PT_LOAD is created for every contiguous memory chunk present in
> > the system. Other information like register states etc. is captured in
> > notes section.
> > 
> > Now the issue is, on i386, whether to prepare core headers in ELF32 or
> > ELF64 format. gdb can not analyze ELF64 core image for i386 system. I
> > don't know about "crash". Can "crash" support ELF64 core image file for
> > i386 system?
> > 
> > Given the limitation of analysis tools, if core headers are prepared in
> > ELF32 format then how to handle PAE systems? 
> > 
> > Any thoughts or suggestions on this?
> 
> Generate it ELF64.  We also have the problem that the kernels virtual
> addresses are not used in the core dump either.   Which a post-processing
> tool will also have to address as well. 

That sounds good. But we loose the advantage of doing limited debugging
with gdb. Crash (or other analysis tools) will still take considerable
amount of time before before they are fully ready and tested.

How about giving user the flexibility to choose. What I mean is
introducing a command line option in kexec-tools to choose between ELF32
and ELF64 headers. For the users who are not using PAE systems, they can
very well go with ELF32 headers and do the debugging using gdb.

This also requires, setting the kernel virtual addresses while preparing
the headers. KVA for linearly mapped region is known in advance and can
be filled at header creation time and gdb can directly operate upon this
region.

> 
> What I aim on at was a simple picture of memory decorated with the
> register state.  We should be able to derive everything beyond that.
> And the fact that it is all in user space should make it straight
> forward to change if needed.
> 
> Eric
>  

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Eric W. Biederman]

Vivek Goyal <vgoyal@in.ibm.com> writes:

> On Tue, 2005-03-08 at 11:00 -0700, Eric W. Biederman wrote: 
> That sounds good. But we loose the advantage of doing limited debugging
> with gdb. Crash (or other analysis tools) will still take considerable
> amount of time before before they are fully ready and tested.
> 
> How about giving user the flexibility to choose. What I mean is
> introducing a command line option in kexec-tools to choose between ELF32
> and ELF64 headers. For the users who are not using PAE systems, they can
> very well go with ELF32 headers and do the debugging using gdb.
> 
> This also requires, setting the kernel virtual addresses while preparing
> the headers. KVA for linearly mapped region is known in advance and can
> be filled at header creation time and gdb can directly operate upon this
> region.

I have no problems decorating the ELF header you are generating
in user space with virtual addresses assuming we can reliably
get that information.  And before a kernel crashes looks like a reasonable
time to ask that question.  I don't currently see where you could
derive that information.

Beyond that I prefer a little command line tool that will do the
ELF64 to ELF32 conversion and possibly add in the kva mapping to
make the core dump usable with gdb.  Doing it in a separate tool
means it is the developer who is doing the analysis who cares
not the user who is capturing the system core dump.

But I do agree that it a use case worth solving.

Eric

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Dipankar Sarma]

On Wed, Mar 09, 2005 at 07:17:49AM -0700, Eric W. Biederman wrote:
> Vivek Goyal <vgoyal@in.ibm.com> writes:
> 
> > On Tue, 2005-03-08 at 11:00 -0700, Eric W. Biederman wrote: 
> > 
> > This also requires, setting the kernel virtual addresses while preparing
> > the headers. KVA for linearly mapped region is known in advance and can
> > be filled at header creation time and gdb can directly operate upon this
> > region.
> 
> I have no problems decorating the ELF header you are generating
> in user space with virtual addresses assuming we can reliably
> get that information.  And before a kernel crashes looks like a reasonable
> time to ask that question.  I don't currently see where you could
> derive that information.
> 
> Beyond that I prefer a little command line tool that will do the
> ELF64 to ELF32 conversion and possibly add in the kva mapping to
> make the core dump usable with gdb.  Doing it in a separate tool
> means it is the developer who is doing the analysis who cares
> not the user who is capturing the system core dump.

Well, as a kernel developer, I am both :) For me, having to install
half-a-dozen different command line tools to get and analyze a crash dump
is a PITA, not to mention potential version mismatches. As someone
who would like very much to use crash dump for debugging, I would
much rather be able to force a dump and then use gdb for
a quick debug. I agree that a customer would see a different
situation. It would be nice if we can cater to both the kinds.

Thanks
Dipankar

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Eric W. Biederman]

Dipankar Sarma <dipankar@in.ibm.com> writes:

> On Wed, Mar 09, 2005 at 07:17:49AM -0700, Eric W. Biederman wrote:

> > Beyond that I prefer a little command line tool that will do the
> > ELF64 to ELF32 conversion and possibly add in the kva mapping to
> > make the core dump usable with gdb.  Doing it in a separate tool
> > means it is the developer who is doing the analysis who cares
> > not the user who is capturing the system core dump.
> 
> Well, as a kernel developer, I am both :) For me, having to install
> half-a-dozen different command line tools to get and analyze a crash dump
> is a PITA, not to mention potential version mismatches. As someone
> who would like very much to use crash dump for debugging, I would
> much rather be able to force a dump and then use gdb for
> a quick debug. I agree that a customer would see a different
> situation. It would be nice if we can cater to both the kinds.

Crash dumps seem to be a when all else fails kind of solution.  Or
something to make a detailed record of what happened so information
can be logged.

I think are differences are largely a matter of emphasis.  I worry
about the end user and the whole cycle.  For that we need a fixed
simple crash dump format whit no knobs bells or whistles, that can
be given to developers and eventually supported natively by all of
the tools.

I doubt tweaking gdb so it can handle a 64bit ELF core dump even
on 32bit architectures would be very hard.  Once that is in place
the 64->32bit conversion doesn't matter.  The virtual addresses
matter a little more although I am more inclined to teach gdb
about the physical and virtual address differences of whole machine
crash dumps.

I do agree that the ability to tweak things in the short term
to work like a process that does not have the virtual/physical address
distinction is useful.  

The issue of tool versioning problems is bogus.  That is solved
by simply picking a good interface between tools and sticking
with it.  Occasionally there will be paradigm shifts (like threading)
that will cause change, but generally everything will stay the same.
In addition if tools are distributed together it does not matter,
if there are several of them because there is only one update.

Eric

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Vivek Goyal]

On Wed, 2005-03-09 at 07:17 -0700, Eric W. Biederman wrote:
> Vivek Goyal <vgoyal@in.ibm.com> writes:
> 
> > On Tue, 2005-03-08 at 11:00 -0700, Eric W. Biederman wrote: 
> > That sounds good. But we loose the advantage of doing limited debugging
> > with gdb. Crash (or other analysis tools) will still take considerable
> > amount of time before before they are fully ready and tested.
> > 
> > How about giving user the flexibility to choose. What I mean is
> > introducing a command line option in kexec-tools to choose between ELF32
> > and ELF64 headers. For the users who are not using PAE systems, they can
> > very well go with ELF32 headers and do the debugging using gdb.
> > 
> > This also requires, setting the kernel virtual addresses while preparing
> > the headers. KVA for linearly mapped region is known in advance and can
> > be filled at header creation time and gdb can directly operate upon this
> > region.
> 
> I have no problems decorating the ELF header you are generating
> in user space with virtual addresses assuming we can reliably
> get that information.  And before a kernel crashes looks like a reasonable
> time to ask that question.  I don't currently see where you could
> derive that information.

I want to fill the virtual addresses of linearly mapped region. That is
physical addresses from 0 to MAXMEM (896 MB) are mapped by kernel at
virtual addresses PAGE_OFFSET to (PAGE_OFFSET + MAXMEM). Values of
PAGE_OFFSET and MAXMEM are already known and hard-coded.

I think I used the terminology kernel virtual address and that is adding
to the confusion. Kernel virtual addresses are not necessarily linearly
mapped. What I meant was kernel logical addresses whose associated
physical addresses differ only by a constant offset.

> 
> Beyond that I prefer a little command line tool that will do the
> ELF64 to ELF32 conversion and possibly add in the kva mapping to
> make the core dump usable with gdb.  Doing it in a separate tool
> means it is the developer who is doing the analysis who cares
> not the user who is capturing the system core dump.
> 
> But I do agree that it a use case worth solving.
> 
> Eric
> 

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Eric W. Biederman]

Vivek Goyal <vgoyal@in.ibm.com> writes:

> I want to fill the virtual addresses of linearly mapped region. That is
> physical addresses from 0 to MAXMEM (896 MB) are mapped by kernel at
> virtual addresses PAGE_OFFSET to (PAGE_OFFSET + MAXMEM). Values of
> PAGE_OFFSET and MAXMEM are already known and hard-coded.

PAGE_OFFSET has a common value of 0xc0000000, on x86.  However
that value is by no means fixed.  The 4G/4G split changes it
as do some other patches floating around at the time.
On x86-64 I don't know how stable those kinds of offsets are.
 
> I think I used the terminology kernel virtual address and that is adding
> to the confusion. Kernel virtual addresses are not necessarily linearly
> mapped. What I meant was kernel logical addresses whose associated
> physical addresses differ only by a constant offset.

I know what you meant.  I simply meant that things don't look that
constant to me.  Especially in Linux where there are enough people
to try most of the reasonable possibilities.

I don't even think it is a bad idea.  But I do think we have a different
idea of what is constant.

Eric

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Vivek Goyal]

On Wed, 2005-03-09 at 23:56 -0700, Eric W. Biederman wrote:
> Vivek Goyal <vgoyal@in.ibm.com> writes:
> 
> > I want to fill the virtual addresses of linearly mapped region. That is
> > physical addresses from 0 to MAXMEM (896 MB) are mapped by kernel at
> > virtual addresses PAGE_OFFSET to (PAGE_OFFSET + MAXMEM). Values of
> > PAGE_OFFSET and MAXMEM are already known and hard-coded.
> 
> PAGE_OFFSET has a common value of 0xc0000000, on x86.  However
> that value is by no means fixed.  The 4G/4G split changes it
> as do some other patches floating around at the time.
> On x86-64 I don't know how stable those kinds of offsets are.

Agreed. Then how about, exporting this information to user space.
Probably through sysfs. May be the range of linearly mapped region can
be exported. (PAGE_OFFSET to (PAGE_OFFSET + x)).

>  
> > I think I used the terminology kernel virtual address and that is adding
> > to the confusion. Kernel virtual addresses are not necessarily linearly
> > mapped. What I meant was kernel logical addresses whose associated
> > physical addresses differ only by a constant offset.
> 
> I know what you meant.  I simply meant that things don't look that
> constant to me.  Especially in Linux where there are enough people
> to try most of the reasonable possibilities.
> 
> I don't even think it is a bad idea.  But I do think we have a different
> idea of what is constant.
> 
> Eric
> 

Re: [Fastboot] Re: Query: Kdump: Core Image ELF Format

[Itsuro Oda]

Hi,

Using ELF format to construct dump information (registers, physical adress
range, and physical memory etc.) is OK. It's not bad idea.

But it is not necessary to indend to use a particular analysis tool.
Do simple.

Thanks.
-- 
Itsuro ODA <oda@valinux.co.jp>