Howdy from Apple; Fix and Continue implemented Yet Again

Howdy from Apple; Fix and Continue implemented Yet Again

[Jason Molenda]

Hi all,

Sorry for the total lack of communication from the Apple gdb group
these past couple months.  We were working at a full-on pace for
the Apple World Wide Developer's Conference preview release and
became even more lax in our net.community.communications than
normal.

The two biggest features we've added to gdb for this WWDC release are
cached symfiles and fix and continue.

Klee has been working on cached symfiles on-and-off for the last
year (as those looking at our sources already know), and this is
the release where we've enabled it by default for our users.  Klee
can speak far more intelligently on the topic than me, but briefly
this is akin to a precompiled header file in gcc; gdb builds up an
objfile structure for a shared library and then dumps that objfile
structure to disk.  When that shared library is loaded by an inferior
process, gdb can map in the frozen objfile structure from disk
instead of re-parsing/allocating all of the debug info from the
shared library.

Cached symfiles shows its greatest benefits when you've got a lot
of shared libraries that don't change often.  We are shipping the
feature with a script that builds cached symfiles for all of the
system shared libraries; it would be entirely possible to do the
same for a user's project if the user had several shared libraries
as a part of his project and they were most often unchanged.


Fix and continue is a feature implemented by many other debuggers,
which we added to our gdb for this release.  Sun Workshop, SGI
ProDev WorkShop, Microsoft's Visual Studio, HP's wdb, and Sun's
Hotspot Java VM all provide this feature in one way or another.
I based our implementation on the HP wdb Fix and Continue feature,
which they added a few years back.  Although my final implementation
follows the general outlines of the approach they took, there is
almost no shared code between them.  Some of this is because of
the architectual differences (both the processor and the ABI),
but even more of it is due to implementation design differences.

The wdb approach is to have the debugger as the center of the world;
you edit your file from within wdb, wdb recompiles your files, and
wdb applies the fix; when debugging ends, wdb removes your executable.

Instead, I moved much of this up to our IDE, Xcode, and left gdb
with the back-end work, with a single "fix in this file" command.
Our IDE has a perfectly good editor and build system -- no reason to
duplicate that in gdb.  It was a good approach for us, but it's
hard to say which is better if we were talking about a FSF sources
implementation of the same.

I'm happy to outline everything involved in implementing F&C in
gdb if people are curious, but I don't expect the code to be
especially contributable -- even if I moved all of the processor/ABI
details into their appropriate arch/target vectors, there would
still remain the fact that the separation of work I chose makes no
sense without an IDE on top of gdb.  (and IMHO the approach that
HP chose doesn't fit well with the common behavior of gdb.)

We've still got a lot of work to do for our next OS release, but
I think we'll all be aspiring to be more involved with the net
lists.  (well, we'll all fall asleep for a week or so after WWDC
finishes on Friday, but after THAT :-)

Jason

PS-  Another interesting new feature added by the devtools group
at Apple was "Zero Link", where the link phase of compilation is
elided.  Instead the app turns into dozens -- hundreds -- of little
shared libraries that are pulled in as they're referenced by a stub
program.  Pretty cool stuff, but it's relevant to gdb because we've
found gdb gets pretty pokey when a program consists of hundreds of
shared libraries.  I don't know when our focus will turn this way
exactly, but I expect we're going to be turning our attention to
optimizing symbol lookups in gdb.  I know lots of other folks are
working in this code, so we'll be extra mindful about talking about
these things with the community.  (I shouldn't speak for others --
*I'll* be extra mindful, and I'll encourage my cohorts to do the
same :)

Re: Howdy from Apple; Fix and Continue implemented Yet Again

[Jason Molenda]

On Fri, Jun 27, 2003 at 12:40:04AM -0700, Jason Molenda wrote:
`
> Fix and continue is a feature implemented by many other debuggers,
> which we added to our gdb for this release.  

It was pointed out that I never said what F&C _is_.  It's a feature
for debugging your program, where you can modify your sources
mid-debug-session, and the debugger will "patch" in the new code.
At a very high level, the source file you're modifying is built as
a shared library, gdb executes inferior function calls to load that
shared library into the program, gdb finds all old versions of all
functions contained in that shared library and writes trampolines
in the prologues to jump to the new versions of those functions,
and gdb redirects all file-static (including global) variables in
that shared library to point to the statics/globals in the original
executable.

From an end user's perspective, they're running their program, they
see it misbehave or they know it's about to misbehave, but it took
them fifteen minutes to get to this point -- they don't want to
restart from the beginning and navigate all the way back here.
With F&C, they make the source change, patch it in, and continue
on with their work.

I don't have anything interesting planned this weekend so maybe I'll
write up a short ditty about how it is implemented in gdb and some
of the more technical aspects of the feature.  I've been doing little
else since winter, so I've got a pretty good idea of how it all fits
together. :-)


Jason

Re: Howdy from Apple; Fix and Continue implemented Yet Again

[Kris Warkentin]

That's pretty interesting.  Does it require any special os support in the
debug interface or is it all done with dlopen/dlclose, etc.?  Other than
compiling your app PIC, does it require any other special compiler features?

cheers,

Kris

----- Original Message -----
From: "Jason Molenda" <jason-swarelist@molenda.com>
To: <gdb@sources.redhat.com>
Sent: Friday, June 27, 2003 11:29 PM
Subject: Re: Howdy from Apple; Fix and Continue implemented Yet Again


> On Fri, Jun 27, 2003 at 12:40:04AM -0700, Jason Molenda wrote:
> `
> > Fix and continue is a feature implemented by many other debuggers,
> > which we added to our gdb for this release.
>
> It was pointed out that I never said what F&C _is_.  It's a feature
> for debugging your program, where you can modify your sources
> mid-debug-session, and the debugger will "patch" in the new code.
> At a very high level, the source file you're modifying is built as
> a shared library, gdb executes inferior function calls to load that
> shared library into the program, gdb finds all old versions of all
> functions contained in that shared library and writes trampolines
> in the prologues to jump to the new versions of those functions,
> and gdb redirects all file-static (including global) variables in
> that shared library to point to the statics/globals in the original
> executable.
>
> >From an end user's perspective, they're running their program, they
> see it misbehave or they know it's about to misbehave, but it took
> them fifteen minutes to get to this point -- they don't want to
> restart from the beginning and navigate all the way back here.
> With F&C, they make the source change, patch it in, and continue
> on with their work.
>
> I don't have anything interesting planned this weekend so maybe I'll
> write up a short ditty about how it is implemented in gdb and some
> of the more technical aspects of the feature.  I've been doing little
> else since winter, so I've got a pretty good idea of how it all fits
> together. :-)
>
>
> Jason
>

Re: Howdy from Apple; Fix and Continue implemented Yet Again

[Jason Molenda]

On Sat, Jun 28, 2003 at 09:54:22AM -0400, Kris Warkentin wrote:
> That's pretty interesting.  Does it require any special os support in the
> debug interface or is it all done with dlopen/dlclose, etc.?  Other than
> compiling your app PIC, does it require any other special compiler features?

There aren't a lot of additional requirements.  The main app doesn't
have to be PIC, but the Fixed file does.  Mike Stump added two gcc
options to support Fix and Continue:  the first pads the prologues
of every function with a few NOP instructions, the second creates
indirect references for all file-static data.  We also had to modify
the Objective C runtime to not register the duplicate classes in
the Fixed file, and a new option to our dlopen equivalent,
NSLinkModule.

The NOPs are added so that we're guaranteed to have room for the
5-instruction long trampoline sequence (four insns plus an illegal
op code so I can reliably detect the sequence).  F&C is only used
on unoptimized code[1] but it's possible for a function to be too
small on a PowerPC if it has no body or arguemnts. 

[1] F&C could be used on optimized code as long as you never try to
update the PC from the middle of an old function to the middle of a
new function.  If you keep running the old version of a function,
or you don't ever fix while a function is on the stack, optimized
code should work correctly.


The file-static indirection is a bit trickier.  Normally references
to global data are made indirectly, with the static link editor or
dynamic linker updating the references at link/runtime.  But when
you refer to static data in your file, or global data that is
allocated in that file, the indirection is not necessary.  So a
simplistic approach to F&C (the one H-P took) will have a separate
copy of each file static/global variable for each time you Fix a
file.  (H-P does copy the values from the globals at Fix time)
Instead, we changed our compiler and assembler to force all of
these file-static references though a table of indirections.  The
table is an array of addresses, one for each static, containing the
offset of the variable in the DATA section.  gdb tries to match
these up to identify which (struct symbol) each entry is referring to,
finds the original structure, and changes the indirection table to
point to the original one.

In the dynamic linker, we added a feature to suppress static
initializers in Fixed solibs.  We don't export the function names
in the Fixed solib (aka don't add RTLD_GLOBAL to the dlopen() call
in Linux-speek :), and we added a
NSLINKMODULE_OPTION_DONT_CALL_MOD_INIT_ROUTINES option to NSLinkModule()
for this.  The file static data will be redirected to the
already-initialized data so there's no need to run the initializers,
and doing so could have side effects to the program, so we just
avoid the whole issue by not running those.


The important thing for our customers is that F&C is ready to be
used at all times when they're doing "development style" builds.
We build -g -O0 and add in the compiler flag to pad prologues 
automatically.  Then, half-way through debugging, they realize they
want to use F&C and it's ready to go.

The H-P implementation does use one other feature from the compiler.
They rebuild the source file from within gdb, so they require the
compiler to put the entire compiler command line option in the .o file.

J

PPC; Was: Howdy from Apple; Fix and Continue implemented Yet Again

[Andrew Cagney]

Jason,

Just curious.  What are the odds of someone from Apple finding the time 
to upgrade the PPC so that it uses the new frame unwind code?

Andrew

Re: PPC; Was: Howdy from Apple; Fix and Continue implemented Yet Again

[Klee Dienes]

I'm very much looking forward to doing that; I'd say the odds are 
excellent.  It's been about six months since our last merge with the 
Cygnus source tree, which is much longer than I like; fixing this is 
first on my list of things to do after Mac OS X 10.3 goes GM.

On Wednesday, July 30, 2003, at 10:57AM, Andrew Cagney wrote:

> Just curious.  What are the odds of someone from Apple finding the 
> time to upgrade the PPC so that it uses the new frame unwind code?