Problem with getExe and testInsertedBreakpoint

Problem with getExe and testInsertedBreakpoint

[Kris Van Hees]

Recently, a problem has surfaced related to getExe() in relation to core
files and its use in the testInsertedBreakpoint() test.  Specifically,
the core file seems to store the first 79 characters of the full
executable pathname only.  This results in truncation in cases where the
frysk build tree is located fairly far down a directory hierarchy (which
is a implicit test in itself), and the testInsertedBreakpoint test tries
to read the executable using a truncated (and thus invalid) executable
path name.

As far as the test is concerned, I think it might be easiest to simply
use the (known) path name to the executable name directly because this
particular test is *not* verifying whether the getExe() processing
works.  Avoiding this problem altogether in this test will at least
avoid the current problem.

Whether this problem can be resolved in general remains to be seen.  If
the executable name is simply not available, there is nothing we can do.
However, there might be a tiny hope that we can get to the executable
path name anyway.  Running 'strings' on the core files I checked
displays the untruncated version twice.  I'm currently looking whether
there is a clean, dependable way to get to that information.

	Cheers,
	Kris

Re: Problem with getExe and testInsertedBreakpoint

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2324 bytes --]

Hi Kris,

On Thu, 2007-07-26 at 06:23 -0700, Kris Van Hees wrote:
> Recently, a problem has surfaced related to getExe() in relation to core
> files and its use in the testInsertedBreakpoint() test.  Specifically,
> the core file seems to store the first 79 characters of the full
> executable pathname only.  This results in truncation in cases where the
> frysk build tree is located fairly far down a directory hierarchy (which
> is a implicit test in itself), and the testInsertedBreakpoint test tries
> to read the executable using a truncated (and thus invalid) executable
> path name.
> 
> As far as the test is concerned, I think it might be easiest to simply
> use the (known) path name to the executable name directly because this
> particular test is *not* verifying whether the getExe() processing
> works.  Avoiding this problem altogether in this test will at least
> avoid the current problem.

Nice catch. But the test case seems to actually be testing the wrong
thing. It should test the symbols found in the actual core file, but it
tries to open Proc.getExe() which points to the (truncated) path of the
executable that created the core file.

> Whether this problem can be resolved in general remains to be seen.  If
> the executable name is simply not available, there is nothing we can do.
> However, there might be a tiny hope that we can get to the executable
> path name anyway.  Running 'strings' on the core files I checked
> displays the untruncated version twice.  I'm currently looking whether
> there is a clean, dependable way to get to that information.

As pointed out earlier today in the fexe command thread it would be nice
to split getExe() into 2 semantically different things.

1) String Proc.getExeName();
Which provides the possible name of the executable. Using /proc/pid/exe
symlink following or the name stored in the core file. Both might not be
completely accurate because they were truncated or moved. This should be
used to display to the user what command was run.
2) File Proc.getExeFile();
Which provides the actual file path to open to get at the elf image
(/proc/pid/exe opened "raw", not following the symlink, or the actual
File that the core file was loaded from).

Does that make sense/would that work?

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Problem with getExe and testInsertedBreakpoint

[Phil Muldoon]

Mark Wielaard wrote:
> Hi Kris,
>
> On Thu, 2007-07-26 at 06:23 -0700, Kris Van Hees wrote:
>   
>> Recently, a problem has surfaced related to getExe() in relation to core
>> files and its use in the testInsertedBreakpoint() test.  Specifically,
>> the core file seems to store the first 79 characters of the full
>> executable pathname only.  This results in truncation in cases where the
>> frysk build tree is located fairly far down a directory hierarchy (which
>> is a implicit test in itself), and the testInsertedBreakpoint test tries
>> to read the executable using a truncated (and thus invalid) executable
>> path name.
>>
>> As far as the test is concerned, I think it might be easiest to simply
>> use the (known) path name to the executable name directly because this
>> particular test is *not* verifying whether the getExe() processing
>> works.  Avoiding this problem altogether in this test will at least
>> avoid the current problem.
>>     
>
> Nice catch. But the test case seems to actually be testing the wrong
> thing. It should test the symbols found in the actual core file, but it
> tries to open Proc.getExe() which points to the (truncated) path of the
> executable that created the core file.
>
>   
>> Whether this problem can be resolved in general remains to be seen.  If
>> the executable name is simply not available, there is nothing we can do.
>> However, there might be a tiny hope that we can get to the executable
>> path name anyway.  Running 'strings' on the core files I checked
>> displays the untruncated version twice.  I'm currently looking whether
>> there is a clean, dependable way to get to that information.
>>     
>
> As pointed out earlier today in the fexe command thread it would be nice
> to split getExe() into 2 semantically different things.
>
> 1) String Proc.getExeName();
> Which provides the possible name of the executable. Using /proc/pid/exe
> symlink following or the name stored in the core file. Both might not be
> completely accurate because they were truncated or moved. This should be
> used to display to the user what command was run.
> 2) File Proc.getExeFile();
> Which provides the actual file path to open to get at the elf image
> (/proc/pid/exe opened "raw", not following the symlink, or the actual
> File that the core file was loaded from).
>   

When a core file is modeled in Frysk, it could be a long time from when 
the core was created. An executable file might not be even available at 
that time. For this reason there are two layers of meta data built:

1) Basic meta-data. No executable available. No maps can be produced 
(but address ranges can). No solib tables can be built. No elided 
segment memory access. At this point we can model the information that 
is in the core file, and allow access to non-elided sections of the core 
file. This is basic inspection. Can't do much other than look at memory, 
and registers.

2) Enhanced meta-data. Executable is available. In this case we can find 
where the link-map was in the core file when it was dumped by looking at 
the executable's dynamic segment. We can build a table of what solibs 
were loaded and mapped into the process when it was dumped, and we can 
build a "rich" set of maps  almost identical to what you would get in 
/proc/$$/maps. We can now allow elided memory access because we know 
what solib is at which address, so the host can open that solib and read 
that memory. This allows sophisticated operations as stack back traces, 
and so on.

At no time is an executable ever loaded into memory during this process, 
or at any time. It's all simulated via CorefileByteBuffer, and 
dead/LinuxHost. The process might "look" alive from the interface, but 
it is well and truly, and probably forever will be, dead ;) All these 
operations are done by looking at the Elf image on disk. So relying on 
anything in /proc/$$ won't work in that instance.

Hope that sheds a bit of light.

Regards

Phil

Re: Problem with getExe and testInsertedBreakpoint

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

Hi Phil,

On Thu, 2007-07-26 at 10:34 -0500, Phil Muldoon wrote:
> When a core file is modeled in Frysk, it could be a long time from when 
> the core was created. An executable file might not be even available at 
> that time. For this reason there are two layers of meta data built:
> 
> 1) Basic meta-data. No executable available. No maps can be produced 
> (but address ranges can). No solib tables can be built. No elided 
> segment memory access. At this point we can model the information that 
> is in the core file, and allow access to non-elided sections of the core 
> file. This is basic inspection. Can't do much other than look at memory, 
> and registers.

What does "elided" mean in this context?

> 2) Enhanced meta-data. Executable is available. In this case we can find 
> where the link-map was in the core file when it was dumped by looking at 
> the executable's dynamic segment. We can build a table of what solibs 
> were loaded and mapped into the process when it was dumped, and we can 
> build a "rich" set of maps  almost identical to what you would get in 
> /proc/$$/maps. We can now allow elided memory access because we know 
> what solib is at which address, so the host can open that solib and read 
> that memory. This allows sophisticated operations as stack back traces, 
> and so on.

OK, so does this mean not all code segments might be available in the
core elf file itself, but that you need to cross-reference them always
against the actual executable (but you aren't guaranteed access to the
orgininal executable since even the core file itself doesn't have a
reliable link to it)?

My question really is what about code segments that are modified (as we
do in the testcase that Kris pointed at)? Or maybe more realistically,
what about code segments generated dynamically by the process (like in
the case of a just in time, hotspot, compiler)?

Finally, how does the rest of the core get access to these code
segments? For example in this testcase or in the DebugInfo, access to
the Elf object for the Proc is obtained by doing new Elf(proc.getExe()).
But since for core Procs that would give the (possible not full, or no
longer existing) path to the original executable that seems the wrong
approach. And from your explanation I now understand that you shouldn't
do it this way (need to audit the code a bit more to find all cases
where this happens), but go through the getMaps() interface and create a
Dwfl from that.

> Hope that sheds a bit of light.

Yes, thanks!

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Problem with getExe and testInsertedBreakpoint

[Phil Muldoon]

Mark Wielaard wrote:
> Hi Phil,
>
> On Thu, 2007-07-26 at 10:34 -0500, Phil Muldoon wrote:
>   
>
> What does "elided" mean in this context?
>   

With apologies to Roland who coined and explained these terms to me. Any 
errors I make here are mine, but:

elided means the segment is recorded in the program headers, but was not 
included in the core file dump (ie filesz = 0, memsz > 0). Segments are 
elided when they can be reconstructed after a core file dump (ie 
segments that have not been written to).

included means both the program header and the segment itself is written 
(ie filesz >0, memsz >0 but filesz is never more than memsz)

omitted means neither the program header entry or the segment is written 
for that segment (ie in future it will not exist in core file 
reconstruction).

>   
>> 2) Enhanced meta-data. Executable is available. In this case we can find 
>> where the link-map was in the core file when it was dumped by looking at 
>> the executable's dynamic segment. We can build a table of what solibs 
>> were loaded and mapped into the process when it was dumped, and we can 
>> build a "rich" set of maps  almost identical to what you would get in 
>> /proc/$$/maps. We can now allow elided memory access because we know 
>> what solib is at which address, so the host can open that solib and read 
>> that memory. This allows sophisticated operations as stack back traces, 
>> and so on.
>>     
>
> OK, so does this mean not all code segments might be available in the
> core elf file itself, but that you need to cross-reference them always
> against the actual executable (but you aren't guaranteed access to the
> orgininal executable since even the core file itself doesn't have a
> reliable link to it)?
>   

Yes it means that though we might have the code executable segments in 
the core file, we lose the ability of basic navigation as _all the other 
executable segments_ might not be there, and certainly not in an easily 
readable/locatable way. That is why we need the executable for anything 
other than basic segment and note access,  so we can find positional 
information (like the link_map). The only place the executable name is 
stored in the core file that we can get to reliably is in the 
elf_prpsinfo notes.

> My question really is what about code segments that are modified (as we
> do in the testcase that Kris pointed at)? Or maybe more realistically,
> what about code segments generated dynamically by the process (like in
> the case of a just in time, hotspot, compiler)?
>   

They may or may not be there in the hot-spot case. If the segment has 
been written to, though, it will be included as it cannot be recreated 
post core dump. Essentially I suspect a core file of a Java process 
will  be a core file of the JVM with one of the threads being the (user 
Java) program itself.

> Finally, how does the rest of the core get access to these code
> segments? For example in this testcase or in the DebugInfo, access to
> the Elf object for the Proc is obtained by doing new Elf(proc.getExe()).
>   


The core file code does not care one whit about how the user/implementor 
uses the information it provides. It cannot, and should not. It would be 
asking for safety checks beyond the api barrier.

It simulates the process image _as close to it can_ to when that process 
was dumped. It is important to note that that this is still a lot of 
guessing and calculating and estimating - even if an executable is 
provided. If no executable was provided or found then that data is just 
not present in the first place. If you try to access elided data on a 
basic meta-data core file, it will throw a RuntimeException noting it 
cannot access that memory address. Proc.getExe() in a Corefile will 
return the executable string that was found in the notes.


> But since for core Procs that would give the (possible not full, or no
> longer existing) path to the original executable that seems the wrong
> approach. 


It will provide the information it has. Because core files are 
traditionally associated with dead or dying processes, the level of 
paranoia when dealing with a core file should be appropriate. There can 
be any number of corrupt, badly written, busted, and generally 
misleading information in there.


> And from your explanation I now understand that you shouldn't
> do it this way (need to audit the code a bit more to find all cases
> where this happens), but go through the getMaps() interface and create a
> Dwfl from that.
>   


Right, and I've talked a bit about this and learned that we should have 
done core files right from that start. Right now (or rather we had) code 
that makes assumptions that a process is alive and has entries in proc. 
As I find them, I correct them, or ask the author to correct them. The 
golden rule here is all information should always come from the Proc or 
the Task. In the case of maps, it should be getMaps(). In the case of 
memory, it should be via the Task's memory interface and so on.

Hope that helps!

Regards

Phil

Re: Problem with getExe and testInsertedBreakpoint

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 3865 bytes --]

Hi Phil,

On Thu, 2007-07-26 at 14:32 -0500, Phil Muldoon wrote:
> With apologies to Roland who coined and explained these terms to me. Any 
> errors I make here are mine, but:
> 
> elided means the segment is recorded in the program headers, but was not 
> included in the core file dump (ie filesz = 0, memsz > 0). Segments are 
> elided when they can be reconstructed after a core file dump (ie 
> segments that have not been written to).
> 
> included means both the program header and the segment itself is written 
> (ie filesz >0, memsz >0 but filesz is never more than memsz)
> 
> omitted means neither the program header entry or the segment is written 
> for that segment (ie in future it will not exist in core file 
> reconstruction).

Aha, that also explains the meaning of the new kernel /proc/sys sysctrl
parameters fs.binfmt_elf.dumpwhole_segments to get all text sections
included and fs.binfmt_elf.dump_elf_headers to get partial text sections
that I saw Roland added. Does fcore have some similar parameters?

> Yes it means that though we might have the code executable segments in 
> the core file, we lose the ability of basic navigation as _all the other 
> executable segments_ might not be there, and certainly not in an easily 
> readable/locatable way. That is why we need the executable for anything 
> other than basic segment and note access,  so we can find positional 
> information (like the link_map). The only place the executable name is 
> stored in the core file that we can get to reliably is in the 
> elf_prpsinfo notes.

Right, and elf_prpsinfo only allows for truncated executable names (16
chars). Bleah. So something like BuildID is really badly needed to
present the user with something that is more often usable I guess. I saw
that the BuildID support proposal was approved as one of the Fedora 8
Features: http://fedoraproject.org/wiki/Releases/FeatureBuildId
http://fedoraproject.org/wiki/Releases/8/FeatureList
Will it be possible for frysk to also support that in the Fedora 8
timeframe?

> > My question really is what about code segments that are modified (as we
> > do in the testcase that Kris pointed at)? Or maybe more realistically,
> > what about code segments generated dynamically by the process (like in
> > the case of a just in time, hotspot, compiler)?
> >   
> 
> They may or may not be there in the hot-spot case. If the segment has 
> been written to, though, it will be included as it cannot be recreated 
> post core dump.

OK, and I assume the same would be true for any code segments that were
altered through ptrace poke?

> > And from your explanation I now understand that you shouldn't
> > do it this way (need to audit the code a bit more to find all cases
> > where this happens), but go through the getMaps() interface and create a
> > Dwfl from that.
> >   
> Right, and I've talked a bit about this and learned that we should have 
> done core files right from that start. Right now (or rather we had) code 
> that makes assumptions that a process is alive and has entries in proc. 
> As I find them, I correct them, or ask the author to correct them. The 
> golden rule here is all information should always come from the Proc or 
> the Task. In the case of maps, it should be getMaps(). In the case of 
> memory, it should be via the Task's memory interface and so on.

Thanks. I filed bug #4852 to clean up the code where I abused it and
will go through the other stuff in the core listed there that uses the
wrong approach (I know I just copied what I did from some other existing
example, so it is good to get this fully cleaned up).

> Hope that helps!

It helped greatly. Thanks for sharing. Some of the things you said
suddenly made me see the bigger picture and connect the dots.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Problem with getExe and testInsertedBreakpoint

[Phil Muldoon]

Mark Wielaard wrote:
> Hi Phil,
>
> On Thu, 2007-07-26 at 14:32 -0500, Phil Muldoon wrote:
>   
>
> Aha, that also explains the meaning of the new kernel /proc/sys sysctrl
> parameters fs.binfmt_elf.dumpwhole_segments to get all text sections
> included and fs.binfmt_elf.dump_elf_headers to get partial text sections
> that I saw Roland added. Does fcore have some similar parameters?
>
>   

Yup, fcore -a dump all possible segments. However that is not the 
default. That has been there since fcore was written because I perhaps 
made the brave assumption that in userland core dumps we can hang around 
and dump more segments as the process is "fine". In kernel core dumps I 
think the aim it so do this all as quickly as possible and clobber the 
process.


> Right, and elf_prpsinfo only allows for truncated executable names (16
> chars). Bleah. So something like BuildID is really badly needed to
> present the user with something that is more often usable I guess. I saw
> that the BuildID support proposal was approved as one of the Fedora 8
> Features: http://fedoraproject.org/wiki/Releases/FeatureBuildId
> http://fedoraproject.org/wiki/Releases/8/FeatureList
> Will it be possible for frysk to also support that in the Fedora 8
> timeframe?
>
>   


Given that there is a lot to do, and not enough of us to do it, I've 
taken the rather (perhaps cheeky) approach of letting Jan and Roland 
thrash out the details, then pick it up and incorporate it when done. I 
already owe a Roland an answer to the partially elided segment question 
and Frysk.


> OK, and I assume the same would be true for any code segments that were
> altered through ptrace poke?
>
>   

I'm not sure of kernel mechanics. The core dump class just looks at the 
the bits in /proc/$$/maps to see if it has been written to.

> Thanks. I filed bug #4852 to clean up the code where I abused it and
> will go through the other stuff in the core listed there that uses the
> wrong approach (I know I just copied what I did from some other existing
> example, so it is good to get this fully cleaned up).
>
>   
>> Hope that helps!
>>     
>
> It helped greatly. Thanks for sharing. Some of the things you said
> suddenly made me see the bigger picture and connect the dots.
>
>   

Awesome, glad to be of help

Regards

Phil

Re: Problem with getExe and testInsertedBreakpoint

[Roland McGrath]

> Will it be possible for frysk to also support that in the Fedora 8
> timeframe?

I intend to have some libdwfl additions for build-id before F8.
Those will make it very easy to do something in Frysk.

> OK, and I assume the same would be true for any code segments that were
> altered through ptrace poke?

Correct.  The kernel's test is on its internal VM data structures, so it
gets any private file mapping that has had some COW happen.


Thanks,
Roland

Re: Problem with getExe and testInsertedBreakpoint

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

Hi Roland,

On Fri, 2007-07-27 at 12:03 -0700, Roland McGrath wrote:
> > Will it be possible for frysk to also support that in the Fedora 8
> > timeframe?
> 
> I intend to have some libdwfl additions for build-id before F8.
> Those will make it very easy to do something in Frysk.

That would make it easy to fixup the core file consumer part in frysk
indeed. But we would still need to make sure that when we produce core
files we also include the BuildID.

> > OK, and I assume the same would be true for any code segments that were
> > altered through ptrace poke?
> 
> Correct.  The kernel's test is on its internal VM data structures, so it
> gets any private file mapping that has had some COW happen.

Nice to know.
Somewhat related, is there some way to revert this?
In the case of inserting low-level breakpoints all we really do is
setting one instruction and then later resetting the original
instruction. Is there any way to make the kernel know that the page can
be remerged again because all writes to it have been reversed?

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Problem with getExe and testInsertedBreakpoint

[Roland McGrath]

> But we would still need to make sure that when we produce core files we
> also include the BuildID.

What the changed kernel does is easy to replicate.  When you otherwise
would have elided a segment and it's a private mapping (...p in 2nd column
of /proc/pid/maps) at offset 0 (3rd column), check if its contents begin
with "\177ELF".  If so, don't elide the first page of that segment.

> Is there any way to make the kernel know that the page can
> be remerged again because all writes to it have been reversed?

No.  This is one of the benefits that would come from the "fancy VM tricks"
future item (don't bother asking for details, there aren't any).


Thanks,
Roland

Re: Problem with getExe and testInsertedBreakpoint

[Phil Muldoon]

Kris Van Hees wrote:
> Recently, a problem has surfaced related to getExe() in relation to core
> files and its use in the testInsertedBreakpoint() test.  Specifically,
> the core file seems to store the first 79 characters of the full
> executable pathname only.  
For context here. The executable name is usually stored in two places, 
both in the same note:

eu-readelf -n core.2843

Note segment of 584 bytes at offset 0x294:
  Owner          Data size  Type
  CORE                 124  PRPSINFO
    state: T (84),  zombie: 49,  nice: 48
    flags: 00400000,  uid: 500,  gid: 500
    pid: 2843,  ppid: 2807,  pgrp: 2843,  sid: 2807
    fname: funit-child
     args: 
/home/pmuldoon/frysk_bin/frysk-core/frysk/pkglibdir/funit-child 10000 
--busy-lo


The first is fname, and the second ... assuming no tinkering with the 
args, is arg[0] in the args list.

In the struct elf_prpsinfo, these two are defined as

  char           pr_fname[16];  /* Filename of 
executable                    */
  char           pr_psargs[80]; /* Initial part of arg 
list                  */

So the length is limited in the notes section in the core file. There is 
nothing fcore, or gcore can do to overcome this.

The fcore command uses automatic executable location, which tries to 
find the executable from arg[0], or /usr/bin, or /bin. This is because 
there are two constructors in dead/LinuxHost. One that just takes just a 
core file and tries to find the core file itself (with hopefully 
build-id patches in the future), and one that takes a core file and its 
corresponding executable. From an api point of view, the latter will 
always be more accurate. Perhaps fcore and other core file utilities 
need to utilize the second constructor? Personally I would prefer this 
approach.


> This results in truncation in cases where the
> frysk build tree is located fairly far down a directory hierarchy (which
> is a implicit test in itself), and the testInsertedBreakpoint test tries
> to read the executable using a truncated (and thus invalid) executable
> path name.
>   

The truncation is always going to occur in that case :(
> As far as the test is concerned, I think it might be easiest to simply
> use the (known) path name to the executable name directly because this
> particular test is *not* verifying whether the getExe() processing
> works.  Avoiding this problem altogether in this test will at least
> avoid the current problem.
>   

The second constructor in dead/LinuxHost will satisfy that. Anyway, good 
catch.

Regards

Phil

Re: Problem with getExe and testInsertedBreakpoint

[Phil Muldoon]

Phil Muldoon wrote:
> Perhaps fcore and other core file utilities need to utilize the second 
> constructor?

Sorry made a mistake here. fcore should be fstack, fhpd, and source window.