Re: [ECOS] Debugging multi-threaded eCos application using GDB

[Bart Veer <bartv@redhat.com>]

>>>>> "Stefan" == Stefan Syberichs <Stefan.Syberichs@ascom.ch> writes:

    Stefan> Hello,
    Stefan> We have a similar problem like Nadine: Debugging an eCos
    Stefan> application for the linux target is hard, because gdb is
    Stefan> not aware of the eCos threads.

    >> We do have some ideas on how to fix this properly, basically
    >> making the thread-aware debugging inside gdb much more
    >> flexible, but so far nobody has been willing to contribute or
    >> fund any of the work involved.

    >> Bart

    Stefan> Bart, what exactly are the ideas for gdb ? Or does anybody
    Stefan> have a solution meanwhile ?

First a caution: I am not an expert on gdb internals; I am quite
likely to be wrong on some of the details, and there may be a better
solution than what I am proposing here.

gdb has the concept of a target vector: basically a table of C
functions to perform debug-related operations like reading an
area of memory from the target, or inserting a breakpoint.
Implementing thread-aware debugging for a target currently involves
filling in a couple more slots in this vector, for example to set
the current thread for the purposes of e.g. getting register
information. 

When you build gdb for a given configuration there will be a small
number (possibly one) of target vectors. For a typical embedded system
there will be a "remote" target, where the various functions in the
target vector work by exchanging messages with gdb stubs over a serial
line or some other transport layer. There may also be a sim target to
run an instruction-set or architectural simulator. For a native Linux
gdb there will be a different target vector which I believe operates
primarily via the ptrace() system call.

If gdb is using the "remote" target vector then most of the hard work
is done by gdb stubs running on the target. Those stubs can be
eCos-aware, so they do the right thing when debugging multi-threaded
eCos applications. On the other hand, if you used the same gdb stubs
to debug multi-threaded code developed using some other RTOS then
thread-aware debugging would not work.

If gdb is using the "sim" target vector then that simulator is not
necessarily configured to run just eCos executables. Therefore the
thread-related entries in the target vector may not be filled in: it
would be possible to fill them in with entries that were
eCos-specific, but that could cause problems for people using the
simulator for other code. Although many gdb maintainers are employed
by Red Hat, the code is owned by the FSF and belongs to the community
as a whole: Red Hat cannot just make eCos-specific changes if those
could cause problems for other people.

If gdb is using the "native linux" target vector then the
thread-related entries in the target vector know about Linux threads,
not eCos threads.

To provide thread-aware eCos debugging in an environment which does
not involve gdb stubs, e.g. the simulator or the synthetic target, you
can implement the necessary functions in the target vector on top of
existing functionality. For example, the target vector entry for
getting hold of the current thread could resolve the symbol
Cyg_Scheduler::current_thread and read the corresponding memory
location.

So it would be possible to define additional eCos-specific target
vectors. You could have a target sim-eCos which is just like the
normal sim target, but replaces certain entries with eCos-specific
ones. Similarly for the native Linux gdb you could have a target
vector for synthetic-eCos, which behaves mostly like the normal Linux
target vector except when it comes to thread-aware debugging. This
would all have to be done at the C level - there is no C++ inside gdb
so you cannot do these things through derived classes and virtual
vectors. A major problem is that if something changes inside the RTOS,
e.g. the variable identifying the current thread is renamed for some
reason, then the gdb sources would have to be edited and gdb would
have to be renamed. This could be especially problematical for highly
configurable systems such as eCos.

A cleaner approach would be to make the target vector scriptable.
Some of the entries in the target vector would no longer be direct C
functions, instead they would invoke a scripting language interpreter
and run an appropriate command. Those commands could in turn access
other entries in the target vector, e.g. to perform low-level
operations like reading a location in memory. If you want to debug an
eCos application, you load in a script provided with eCos. If you are
using some other RTOS, that would provide its own script. If something
changed in the internals of the RTOS there would be no need to rebuild
gdb, instead you just use an updated script.

Adding scripting functionality would give a great deal of flexibility:
things like thread-aware debugging could now be made to work in the
synthetic target, or in an architectural simulator, or via an ICE, or
whatever. However the effort involved would be considerable, probably
including a lengthy debate within the gdb community on which scripting
language should be used (my guess is that they would end up with
guile, althoug I would prefer Tcl :-).

>>>>> "Jifl" == Jonathan Larmour <jlarmour@redhat.com> writes:

    Jifl> Actually I would have thought the easiest route would be to
    Jifl> allow included stubs in the synth target to be available at
    Jifl> a socket, and then using the remote protocol. Making it
    Jifl> available as a socket should "just" be a case of adding a
    Jifl> virtual vectored comm interface. It's probably a bit too
    Jifl> much work for us to just do for fun, particularly since the
    Jifl> synth target doesn't use virtual vectors at all right now.

I don't how this could sensibly work for the synthetic target. We are
not using the remote protocol, we are using native debugging. 

Yes, theoretically it would be possible to run something like Redboot
inside the synthetic target, connect to it via a socket and the remote
protocol, and then boot a RAM-startup synthetic target application
into it. When debugging the target would run in polled mode, which is
now what you want for a synthetic application. Interrupting a running
program would involve detecting SIGIO on the socket from gdb and doing
the right thing, not impossible but messy. I would much prefer to
avoid this route. Fixing the problem in gdb instead would give a clean
and general-purpose solution.

Bart