[RFC] a prototype checkpoint-restart using core files

[RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

Folks, this isn't for commit, just for discussion.

Attached is an experimental patch that adds a command
"restore-core-file" or "rcore", which is the inverse of
"generate-core-file" (gcore).  Instead of copying the
memory and register state of a process into a file,
it takes an existing corefile, and copies its memory
and register state into the child process.

The idea was to experiment with the concept of doing
checkpoint and restore, by using a corefile as the
checkpoint file.  Obviously it has limitations --
it doesn't save any kernel state, I/O state etc.
Just user state.

But it turns out that if you avoid those limitations,
it works!  As a conservative rule of thumb, you can
go back to an earlier state so long as you don't cross
a system call.  And in practice there are lots of
system calls that can be regarded as "stateless",
or that change only user state -- so you can cross
those.

I've only tried it on Linux, but it seems to me that
it should be pretty portable, at least to hosts for
which 'gcore' works.

This may be useful for anyone who wants to experiment
with the idea of checkpoint and restart, get a feel
for how it would work in practice, and look for
"gotchas" where the sudden state change might interfere
with something else in gdb.  For a "real" checkpoint
and restart facility, we would probably want to handle
some amount of kernel state and I/O state, but
gcore/rcore might even be somewhat useful as-is
(at least to a knowledgeable user who understands
their limitations).




[-- Attachment #2: rcore.diff --]
[-- Type: text/plain, Size: 4948 bytes --]

Index: gcore.c
===================================================================
RCS file: /cvs/src/src/gdb/gcore.c,v
retrieving revision 1.17
diff -p -r1.17 gcore.c
*** gcore.c	15 Feb 2005 15:49:10 -0000	1.17
--- gcore.c	3 Nov 2005 01:19:53 -0000
*************** gcore_memory_sections (bfd *obfd)
*** 488,493 ****
--- 488,621 ----
    return 1;
  }
  
+ /* OK now, I want to add a new command to read a corefile, 
+    and restore its state into the inferior process.  Obviously
+    dangerous, probably want to make certain that they are 
+    actually the same process!  But we can put that off till
+    later.  Let's see what's required.  This should actually
+    be pretty easy.  */
+ 
+ static void
+ load_core_sections (bfd *abfd, asection *asect, void *arg)
+ {
+   unsigned long from_tty = (unsigned long) arg;
+   char *memhunk;
+ 
+   if ((bfd_section_size (abfd, asect) > 0) &&
+       (bfd_get_section_flags (abfd, asect) & SEC_LOAD))
+     {
+       if (info_verbose && from_tty)
+ 	{
+ 	  printf_filtered (_("Load core section %s"), 
+ 			   bfd_section_name (abfd, asect));
+ 	  printf_filtered (_(", address 0x%08lx"), 
+ 			   (unsigned long) bfd_section_vma (abfd, asect));
+ 	  printf_filtered (_(", size = %d"), 
+ 			   (int) bfd_section_size (abfd, asect));
+ 	  printf_filtered (_(".\n"));
+ 	}
+       /* Fixme cleanup? */
+       memhunk = xmalloc (bfd_section_size (abfd, asect));
+       bfd_get_section_contents (abfd, asect, memhunk, 0, 
+ 				bfd_section_size (abfd, asect));
+       target_write_memory (bfd_section_vma (abfd, asect), 
+ 			   memhunk, 
+ 			   bfd_section_size (abfd, asect));
+       xfree (memhunk);
+     }
+ }
+ 
+ #include <fcntl.h>
+ #ifndef O_BINARY
+ #define O_BINARY 0
+ #endif
+ 
+ #include "regcache.h"
+ #include "regset.h"
+ 
+ static void
+ rcore_command (char *args, int from_tty)
+ {
+   /* corelow.c core_open */
+   /* scratch_chan = open (filename)
+      temp_bfd = bfd_fdopenr (filename, gnutarget, scratch_chan)
+      bfd_check_format (temp_bfd, bfd_core)
+      build_section_table (core_bfd, to_sections, to_sections_end)
+      bfd_map_over_sections (core_bfd, myfunc)
+      myfunc will check for loadable, contents, and size, 
+      and then write the section contents into memory at vma.
+   */
+   char *corefilename, corefilename_buffer[40], *scratch_path;
+   int  scratch_chan;
+   bfd  *core_bfd;
+ 
+   /* Can't restore a corefile without a target process.  */
+   if (!target_has_execution)
+     noprocess ();
+ 
+   if (args && *args)
+     corefilename = args;
+   else
+     {
+       /* Default corefile name is "core.PID".  */
+       sprintf (corefilename_buffer, "core.%d", PIDGET (inferior_ptid));
+       corefilename = corefilename_buffer;
+     }
+ 
+   if (info_verbose)
+     fprintf_filtered (gdb_stdout,
+ 		      _("Opening corefile '%s' for input.\n"), corefilename);
+ 
+   scratch_chan = openp (getenv ("PATH"), OPF_TRY_CWD_FIRST, corefilename, 
+ 			O_BINARY | O_RDONLY | O_LARGEFILE, 0, &scratch_path);
+   if (scratch_chan < 0)
+     perror_with_name (corefilename);
+ 
+   core_bfd = bfd_fdopenr (scratch_path, gnutarget, scratch_chan);
+   if (!core_bfd)
+     perror_with_name (scratch_path);
+ 
+   if (!bfd_check_format (core_bfd, bfd_core))
+     {
+       make_cleanup_bfd_close (core_bfd);
+       error (_("\"%s\" is not a core file: %s"), 
+ 	     corefilename, bfd_errmsg (bfd_get_error ()));
+     }
+ 
+   bfd_map_over_sections (core_bfd, load_core_sections, (void *) from_tty);
+   /* Now need to get/set registers.  */
+   {
+     struct bfd_section *regsect = bfd_get_section_by_name (core_bfd, ".reg");
+     char *contents;
+     int size;
+ 
+     if (!regsect)
+       error (_("Couldn't find .reg section."));
+ 
+     size = bfd_section_size (core_bfd, regsect);
+     contents = xmalloc (size);
+     bfd_get_section_contents (core_bfd, regsect, contents, 0, size);
+ 
+     if (gdbarch_regset_from_core_section_p (current_gdbarch))
+       {
+ 	const struct regset *regset;
+ 
+ 	regset = gdbarch_regset_from_core_section (current_gdbarch, 
+ 						   ".reg", size);
+ 	if (!regset)
+ 	  error (_("Failed to allocate regset."));
+ 
+ 	registers_changed ();
+ 	regset->supply_regset (regset, current_regcache, 
+ 			       -1, contents, size);
+ 	reinit_frame_cache ();
+ 	target_store_registers (-1);
+       }
+   }
+ 
+   bfd_close (core_bfd);
+ }
+ 
  void
  _initialize_gcore (void)
  {
*************** Argument is optional filename.  Default 
*** 497,500 ****
--- 625,633 ----
  
    add_com_alias ("gcore", "generate-core-file", class_files, 1);
    exec_set_find_memory_regions (objfile_find_memory_regions);
+ 
+   add_com ("restore-core-file", class_files, rcore_command, _("\
+ Restore the machine state from a core file into the debugged process.\n\
+ Argument is optional filename.  Default filename is 'core.<process_id>'."));
+   add_com_alias ("rcore", "restore-core-file", class_files, 1);
  }

Re: [RFC] a prototype checkpoint-restart using core files

[Daniel Jacobowitz]

On Wed, Nov 02, 2005 at 05:35:15PM -0800, Michael Snyder wrote:
> Folks, this isn't for commit, just for discussion.
> 
> Attached is an experimental patch that adds a command
> "restore-core-file" or "rcore", which is the inverse of
> "generate-core-file" (gcore).  Instead of copying the
> memory and register state of a process into a file,
> it takes an existing corefile, and copies its memory
> and register state into the child process.
> 
> The idea was to experiment with the concept of doing
> checkpoint and restore, by using a corefile as the
> checkpoint file.  Obviously it has limitations --
> it doesn't save any kernel state, I/O state etc.
> Just user state.

I've got to say that this is amazingly cool.

Afraid that's all the comments I have time for at the moment :-)


-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [RFC] a prototype checkpoint-restart using core files

[Eli Zaretskii]

> Date: Sun, 6 Nov 2005 19:19:37 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: gdb@sources.redhat.com, gdb-patches@sources.redhat.com
> 
> I've got to say that this is amazingly cool.
> 
> Afraid that's all the comments I have time for at the moment :-)

I agree, and I'd add that getting this into GDB soon would be good.

Re: [RFC] a prototype checkpoint-restart using core files

[Mark Kettenis]

> Date: Mon, 07 Nov 2005 06:40:22 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Sun, 6 Nov 2005 19:19:37 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: gdb@sources.redhat.com, gdb-patches@sources.redhat.com
> > 
> > I've got to say that this is amazingly cool.
> > 
> > Afraid that's all the comments I have time for at the moment :-)
> 
> I agree, and I'd add that getting this into GDB soon would be good.

Heh, I'd expected Eli to ask for documentation ;-)

Anyway, in this cause I think that's important since I expect a lot of
users won't understand its limitations.

If I read the code correctly, there is one rather serious limitation
though: restoring mmapped area's will fail if the same area isn't
mapped in the target process.  Especially on systems that randomize
the location of mmapped memory this will make the usefullness of this
feature pretty limited :(.

Mark

Re: [RFC] a prototype checkpoint-restart using core files

[Daniel Jacobowitz]

On Mon, Nov 07, 2005 at 07:57:25PM +0100, Mark Kettenis wrote:
> Heh, I'd expected Eli to ask for documentation ;-)
> 
> Anyway, in this cause I think that's important since I expect a lot of
> users won't understand its limitations.
> 
> If I read the code correctly, there is one rather serious limitation
> though: restoring mmapped area's will fail if the same area isn't
> mapped in the target process.  Especially on systems that randomize
> the location of mmapped memory this will make the usefullness of this
> feature pretty limited :(.

Why should it?  The expected use is to restore these dumps into the
same running session - just after stepping a bit.  So unless you step
across a very large free(), it should be fine.

I admit the general-purpose rcore command has a lot of limitations, but
not as part of a checkpoint-restart system.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

Mark Kettenis wrote:
>>Date: Mon, 07 Nov 2005 06:40:22 +0200
>>From: Eli Zaretskii <eliz@gnu.org>
>>
>>>Date: Sun, 6 Nov 2005 19:19:37 -0500
>>>From: Daniel Jacobowitz <drow@false.org>
>>>Cc: gdb@sources.redhat.com, gdb-patches@sources.redhat.com
>>>
>>>I've got to say that this is amazingly cool.
>>>
>>>Afraid that's all the comments I have time for at the moment :-)
>>
>>I agree, and I'd add that getting this into GDB soon would be good.
> 
> 
> Heh, I'd expected Eli to ask for documentation ;-)
> 
> Anyway, in this cause I think that's important since I expect a lot of
> users won't understand its limitations.
> 
> If I read the code correctly, there is one rather serious limitation
> though: restoring mmapped area's will fail if the same area isn't
> mapped in the target process.  Especially on systems that randomize
> the location of mmapped memory this will make the usefullness of this
> feature pretty limited :(.

Yes, I'm still puzzling over that one.  However, it would
mostly only happen (sic) if you are going forward in time,
not backward.  In particular, if you create a new process,
then try to feed it a corefile from an old process that
was further along in its execution.

If you save a gcore file, run forward, and then reload
the previously saved gcore file back into the *same* process
(ie. go backward in time), the mmaped areas will almost
always be present -- unles someone explicitly unmaps them.

As for your broader point, I agree, I don't think this is
ready for exposure to average users.  I would like to see
it get some play from "sophisticated" users such as you
guys, who may understand the limitations and help to
characterize them.

Re: [RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

Daniel Jacobowitz wrote:
> On Mon, Nov 07, 2005 at 07:57:25PM +0100, Mark Kettenis wrote:
> 
>>Heh, I'd expected Eli to ask for documentation ;-)
>>
>>Anyway, in this cause I think that's important since I expect a lot of
>>users won't understand its limitations.
>>
>>If I read the code correctly, there is one rather serious limitation
>>though: restoring mmapped area's will fail if the same area isn't
>>mapped in the target process.  Especially on systems that randomize
>>the location of mmapped memory this will make the usefullness of this
>>feature pretty limited :(.
> 
> 
> Why should it?  The expected use is to restore these dumps into the
> same running session - just after stepping a bit.  So unless you step
> across a very large free(), it should be fine.
> 
> I admit the general-purpose rcore command has a lot of limitations, but
> not as part of a checkpoint-restart system.

Yes, the next interesting task (before publication, I should think)
is to characterize and refine those limitations.  Experimentally,
I have found that I can jump backward across many system calls,
but not all.  I'd like to work on some idea of which ones are
always ok, which ones are sometimes ok, and which ones will never
be ok.

For instance, I wrote a simple test program that copies stdin
to stdout, one byte at a time, in a loop.  I ran it with input
and output redirected (so that it copies a file).  I checkpointed
it near the beginning, let it run thru 20,000 loops, then went
back to the checkpoint -- and then ran it to completion (so that
the first 20,000 bytes were copied twice).  The resulting input
and output files compared identical.

But when i repeated the experiment, but jumped backward across
50,000 loops, the output file was corrupted.

This is just a crude shot-in-the-dark experiment, obviously;
but it tends to show that we can get away with somewhat more
than we might at first suppose.

[this was ix86-linux native]

Re: [RFC] a prototype checkpoint-restart using core files

[Mark Kettenis]

> Date: Mon, 7 Nov 2005 14:07:24 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> > If I read the code correctly, there is one rather serious limitation
> > though: restoring mmapped area's will fail if the same area isn't
> > mapped in the target process.  Especially on systems that randomize
> > the location of mmapped memory this will make the usefullness of this
> > feature pretty limited :(.
> 
> Why should it?  The expected use is to restore these dumps into the
> same running session - just after stepping a bit.  So unless you step
> across a very large free(), it should be fine.

Ah, somehow I forgar about the "same running session" part.  Guess
that's one of the things that needs to be clearly documented then ;-).

Thanks Daniel,

Mark

Re: [RFC] a prototype checkpoint-restart using core files

[Daniel Jacobowitz]

On Mon, Nov 07, 2005 at 09:21:36PM +0100, Mark Kettenis wrote:
> > Date: Mon, 7 Nov 2005 14:07:24 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > 
> > > If I read the code correctly, there is one rather serious limitation
> > > though: restoring mmapped area's will fail if the same area isn't
> > > mapped in the target process.  Especially on systems that randomize
> > > the location of mmapped memory this will make the usefullness of this
> > > feature pretty limited :(.
> > 
> > Why should it?  The expected use is to restore these dumps into the
> > same running session - just after stepping a bit.  So unless you step
> > across a very large free(), it should be fine.
> 
> Ah, somehow I forgar about the "same running session" part.  Guess
> that's one of the things that needs to be clearly documented then ;-).

Yeah - a general purpose "restore core file" command would be neat, but
I think, not practical enough to be useful.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [RFC] a prototype checkpoint-restart using core files

[Eli Zaretskii]

> Date: Mon, 7 Nov 2005 19:57:25 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sources.redhat.com, eliz@gnu.org, gdb-patches@sources.redhat.com
> 
> Heh, I'd expected Eli to ask for documentation ;-)

There was no sense asking for documentation for code that wasn't
officially suggested as a patch.

Anyway, I trust Michael to add the documentation when he decides to go
for RFA ;-)

> Anyway, in this cause I think that's important since I expect a lot of
> users won't understand its limitations.

Yes.

Re: [RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

Daniel Jacobowitz wrote:
> On Mon, Nov 07, 2005 at 09:21:36PM +0100, Mark Kettenis wrote:
> 
>>>Date: Mon, 7 Nov 2005 14:07:24 -0500
>>>From: Daniel Jacobowitz <drow@false.org>
>>>
>>>>If I read the code correctly, there is one rather serious limitation
>>>>though: restoring mmapped area's will fail if the same area isn't
>>>>mapped in the target process.  Especially on systems that randomize
>>>>the location of mmapped memory this will make the usefullness of this
>>>>feature pretty limited :(.
>>>
>>>Why should it?  The expected use is to restore these dumps into the
>>>same running session - just after stepping a bit.  So unless you step
>>>across a very large free(), it should be fine.
>>
>>Ah, somehow I forgar about the "same running session" part.  Guess
>>that's one of the things that needs to be clearly documented then ;-).
> 
> 
> Yeah - a general purpose "restore core file" command would be neat, but
> I think, not practical enough to be useful.

Going forward in time is actually possible -- sometimes.
I can debug for a while, then save a corefile, then quit,
start a new debug session, open the same program, run to
main -- and then load the old corefile.

This bit isn't in my posted patch, but if the sbrk boundary
has been advanced, I can detect it, and do a target function
call to sbrk to bring it up to match.

Where I run into a problem is with shared libraries.
Sometimes, as Mark says, a segment hasn't been mmapped
in yet.  I don't yet know what to do about that.

Re: [RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

Eli Zaretskii wrote:
>>Date: Mon, 7 Nov 2005 19:57:25 +0100 (CET)
>>From: Mark Kettenis <mark.kettenis@xs4all.nl>
>>CC: gdb@sources.redhat.com, eliz@gnu.org, gdb-patches@sources.redhat.com
>>
>>Heh, I'd expected Eli to ask for documentation ;-)
> 
> 
> There was no sense asking for documentation for code that wasn't
> officially suggested as a patch.
> 
> Anyway, I trust Michael to add the documentation when he decides to go
> for RFA ;-)

And I trust you to make sure that I do.   ;-)

Re: [RFC] a prototype checkpoint-restart using core files

[Soam Vasani]

Michael Snyder wrote:
> Sometimes, as Mark says, a segment hasn't been mmapped
> in yet.  I don't yet know what to do about that.

I think the core file doesn't have enough information for
restoring mmap'd regions ?

While dumping a checkpoint you could save /proc/pid/maps, and
do the mmaps before restoring the rest of the state from core file.
But I'm not sure /proc/pid/maps has enough information, the "flags"
argument to mmap (thats the 4th one) doesn't seem to be shown there.

regards,
soam

Re: [RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

Soam Vasani wrote:
> Michael Snyder wrote:
> 
>> Sometimes, as Mark says, a segment hasn't been mmapped
>> in yet.  I don't yet know what to do about that.
> 
> 
> I think the core file doesn't have enough information for
> restoring mmap'd regions ?
> 
> While dumping a checkpoint you could save /proc/pid/maps, and
> do the mmaps before restoring the rest of the state from core file.
> But I'm not sure /proc/pid/maps has enough information, the "flags"
> argument to mmap (thats the 4th one) doesn't seem to be shown there.

Well, I already use /proc/pid/maps to generate the corefile.
That's how I know which memory segments need to be saved.

And I know which ones are writeable (I don't have to save
the ones that aren't).

Re: [RFC] a prototype checkpoint-restart using core files

[Stan Shebs]

Michael Snyder wrote:

> Folks, this isn't for commit, just for discussion.
>
> Attached is an experimental patch that adds a command
> "restore-core-file" or "rcore", which is the inverse of
> "generate-core-file" (gcore).  Instead of copying the
> memory and register state of a process into a file,
> it takes an existing corefile, and copies its memory
> and register state into the child process.

My prototype is even lamer :-) I use target read/write
operations to collect state - but it can step backwards.
An improved version in the works uses vfork() to make
core images more cheaply.

>
> The idea was to experiment with the concept of doing
> checkpoint and restore, by using a corefile as the
> checkpoint file.  Obviously it has limitations --
> it doesn't save any kernel state, I/O state etc.
> Just user state.
>
> But it turns out that if you avoid those limitations,
> it works!  As a conservative rule of thumb, you can
> go back to an earlier state so long as you don't cross
> a system call.  And in practice there are lots of
> system calls that can be regarded as "stateless",
> or that change only user state -- so you can cross
> those.

One idea I've considered is getting the OS to set up
some kind of notification at system calls, and then use
it to warn the user who tries to resume the inferior after
rolling back. In addition to obvious corruption issues,
you can also get some funky Heisenbugs, for instance
if the code of interest is inside "if (!file_exists()) {",
but a forewarned user can then decide whether to press
on or just rerun.

On shared memory, there's an old Mark Linton paper (1988
debug workshop I think) where they deal with shared memory
and replay by using the compiler to instrument all memory
refs that might be to shmem, basically adding a test to see
if the address is in shmem and if so, updating the shared
memory bits from a saved version. A hairy solution to a
hairy problem...

Stan

Re: [RFC] a prototype checkpoint-restart using core files

[Michael Snyder]

Stan Shebs wrote:
 > Michael Snyder wrote:

 >> Folks, this isn't for commit, just for discussion.
 >>
 >> Attached is an experimental patch that adds a command
 >> "restore-core-file" or "rcore", which is the inverse of
 >> "generate-core-file" (gcore).  Instead of copying the
 >> memory and register state of a process into a file,
 >> it takes an existing corefile, and copies its memory
 >> and register state into the child process.
 >
 > My prototype is even lamer :-) I use target read/write
 > operations to collect state - but it can step backwards.

Don't beat up on yourself -- gcore works by using
target read and write.  It's just a convenient
file format for saving target state.

 > An improved version in the works uses vfork() to make
 > core images more cheaply.

Aha!  Linux fork is similar to vfork, in that it does
not immediately make a copy of user memory.  Did you
have a chance to look at my later patch based on fork?

 > One idea I've considered is getting the OS to set up
 > some kind of notification at system calls, and then use
 > it to warn the user who tries to resume the inferior
 > after rolling back.

Yeah, ptrace on linux provides such a notification.
I know you don't have ptrace on your OS, but you might
think about mimicing the ptrace user interface, if only
so that you don't have to design one of your own.  ;-)

Stan, do you remember the trace/replay utility that
Michael Chastain invented?  I was thinking that would
come in pretty handy in this context.

Michael