[ECOS] Debugging multi-threaded eCos application using GDB

[ECOS] Debugging multi-threaded eCos application using GDB

[Nadine.Albiez-extern]

Hello,
I have a question about debugging an eCos  multi-threaded application using gdb.
When I want to see the list of my threads , the command "info threads" returns nothing

     (gdb) info threads
          (gdb) thread 1
           Thread ID 1 not known.  Use the "info threads" command to
           see the IDs of currently known threads.

What can I do for enabling the multi-threaded ? When I use a linux test application I have no problem, "info threads" returns the list of threads.....
Is there any option when generating the eCos kernel to allow Gdb to see the eCos's Thread (cyg_thread_create)?
I observe the same behavior with the example program "two_threaded".
Is there a package to add to ecos.ecc? I use eCos version 1.4.8. , gdb version 4.18 (I try gdb 5.0 with the same result) and work on a linux target .

Thanks for your help

     Nadine Albiez

Re: [ECOS] Debugging multi-threaded eCos application using GDB

[Bart Veer]

>>>>> "Nadine" == Nadine Albiez-extern <Nadine.Albiez-extern@ascom.fr> writes:

    Nadine> Hello,
    Nadine> I have a question about debugging an eCos multi-threaded
    Nadine> application using gdb. When I want to see the list of my
    Nadine> threads , the command "info threads" returns nothing

    Nadine>      (gdb) info threads
    Nadine>           (gdb) thread 1
    Nadine>            Thread ID 1 not known.  Use the "info threads" command to
    Nadine>            see the IDs of currently known threads.

    Nadine> What can I do for enabling the multi-threaded ? When I use
    Nadine> a linux test application I have no problem, "info threads"
    Nadine> returns the list of threads..... Is there any option when
    Nadine> generating the eCos kernel to allow Gdb to see the eCos's
    Nadine> Thread (cyg_thread_create)? I observe the same behavior
    Nadine> with the example program "two_threaded". Is there a
    Nadine> package to add to ecos.ecc? I use eCos version 1.4.8. ,
    Nadine> gdb version 4.18 (I try gdb 5.0 with the same result) and
    Nadine> work on a linux target .

Thread-aware debugging is not available on all targets, and
unfortunately that includes the synthetic target which I assume you
are referring to.

The problem is with the current implementation of thread-aware
debugging. It expects to be able to interact with some code inside the
embedded target, for example RedBoot or gdb stubs, that knows about
the specific thread implementation being used. Debugging the synthetic
target does not involve interacting with remote stubs, instead it uses
facilities provided by the system such as ptrace().

To confuse things further, gdb might think that you are trying to
debug a multi-threaded Linux application where the threading is
provided by Linux. This does not work either: synthetic target threads
are not built on top of Linux threads, you want the eCos semantics
rather than the Linux ones.

We do have some ideas on how to fix this properly, basically making
the thread-aware debugging inside gdb much more flexible, but so far
nobody has been willing to contribute or fund any of the work
involved.

Bart

[ECOS] Debugging multi-threaded eCos application using GDB

[Stefan Syberichs]

Hello, 

We have a similar problem like Nadine:
Debugging an eCos application for the linux target is hard, because gdb is 
not aware of
the eCos threads.

> We do have some ideas on how to fix this properly, basically making
> the thread-aware debugging inside gdb much more flexible, but so far
> nobody has been willing to contribute or fund any of the work
> involved.

> Bart


Bart, what exactly are the ideas for gdb ? Or does anybody have a solution 
meanwhile ?

regards, Stefan

----------------------------------------------------------------------------------------------------------

To: Nadine dot Albiez-extern at ascom dot fr 
Subject: Re: [ECOS] Debugging multi-threaded eCos application using GDB 
From: Bart Veer <bartv at redhat dot com> 
Date: Mon, 23 Apr 2001 13:52:09 +0100 
Cc: ecos-discuss at sources dot redhat dot com 
References: <OF8761DF62.DD359E81-ONC1256A37.00272947@ascom.fr> 
Reply-To: bartv at redhat dot com 

>>>>> "Nadine" == Nadine Albiez-extern <Nadine.Albiez-extern@ascom.fr> 
writes:

    Nadine> Hello,
    Nadine> I have a question about debugging an eCos multi-threaded
    Nadine> application using gdb. When I want to see the list of my
    Nadine> threads , the command "info threads" returns nothing

    Nadine>      (gdb) info threads
    Nadine>           (gdb) thread 1
    Nadine>            Thread ID 1 not known.  Use the "info threads" 
command to
    Nadine>            see the IDs of currently known threads.

    Nadine> What can I do for enabling the multi-threaded ? When I use
    Nadine> a linux test application I have no problem, "info threads"
    Nadine> returns the list of threads..... Is there any option when
    Nadine> generating the eCos kernel to allow Gdb to see the eCos's
    Nadine> Thread (cyg_thread_create)? I observe the same behavior
    Nadine> with the example program "two_threaded". Is there a
    Nadine> package to add to ecos.ecc? I use eCos version 1.4.8. ,
    Nadine> gdb version 4.18 (I try gdb 5.0 with the same result) and
    Nadine> work on a linux target .

Thread-aware debugging is not available on all targets, and
unfortunately that includes the synthetic target which I assume you
are referring to.

The problem is with the current implementation of thread-aware
debugging. It expects to be able to interact with some code inside the
embedded target, for example RedBoot or gdb stubs, that knows about
the specific thread implementation being used. Debugging the synthetic
target does not involve interacting with remote stubs, instead it uses
facilities provided by the system such as ptrace().

To confuse things further, gdb might think that you are trying to
debug a multi-threaded Linux application where the threading is
provided by Linux. This does not work either: synthetic target threads
are not built on top of Linux threads, you want the eCos semantics
rather than the Linux ones.

We do have some ideas on how to fix this properly, basically making
the thread-aware debugging inside gdb much more flexible, but so far
nobody has been willing to contribute or fund any of the work
involved.

Bart

Re: [ECOS] Debugging multi-threaded eCos application using GDB

[Jonathan Larmour]

Stefan Syberichs wrote:
> 
> Hello,
> 
> We have a similar problem like Nadine:
> Debugging an eCos application for the linux target is hard, because gdb is
> not aware of
> the eCos threads.
> 
> > We do have some ideas on how to fix this properly, basically making
> > the thread-aware debugging inside gdb much more flexible, but so far
> > nobody has been willing to contribute or fund any of the work
> > involved.
> 
> > Bart
> 
> Bart, what exactly are the ideas for gdb ? Or does anybody have a solution
> meanwhile ?

Actually I would have thought the easiest route would be to allow included
stubs in the synth target to be available at a socket, and then using the
remote protocol. Making it available as a socket should "just" be a case of
adding a virtual vectored comm interface. It's probably a bit too much work
for us to just do for fun, particularly since the synth target doesn't use
virtual vectors at all right now.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Debugging multi-threaded eCos application using GDB

[Bart Veer]

>>>>> "Stefan" == Stefan Syberichs <Stefan.Syberichs@ascom.ch> writes:

    Stefan> Hello,
    Stefan> We have a similar problem like Nadine: Debugging an eCos
    Stefan> application for the linux target is hard, because gdb is
    Stefan> not aware of the eCos threads.

    >> We do have some ideas on how to fix this properly, basically
    >> making the thread-aware debugging inside gdb much more
    >> flexible, but so far nobody has been willing to contribute or
    >> fund any of the work involved.

    >> Bart

    Stefan> Bart, what exactly are the ideas for gdb ? Or does anybody
    Stefan> have a solution meanwhile ?

First a caution: I am not an expert on gdb internals; I am quite
likely to be wrong on some of the details, and there may be a better
solution than what I am proposing here.

gdb has the concept of a target vector: basically a table of C
functions to perform debug-related operations like reading an
area of memory from the target, or inserting a breakpoint.
Implementing thread-aware debugging for a target currently involves
filling in a couple more slots in this vector, for example to set
the current thread for the purposes of e.g. getting register
information. 

When you build gdb for a given configuration there will be a small
number (possibly one) of target vectors. For a typical embedded system
there will be a "remote" target, where the various functions in the
target vector work by exchanging messages with gdb stubs over a serial
line or some other transport layer. There may also be a sim target to
run an instruction-set or architectural simulator. For a native Linux
gdb there will be a different target vector which I believe operates
primarily via the ptrace() system call.

If gdb is using the "remote" target vector then most of the hard work
is done by gdb stubs running on the target. Those stubs can be
eCos-aware, so they do the right thing when debugging multi-threaded
eCos applications. On the other hand, if you used the same gdb stubs
to debug multi-threaded code developed using some other RTOS then
thread-aware debugging would not work.

If gdb is using the "sim" target vector then that simulator is not
necessarily configured to run just eCos executables. Therefore the
thread-related entries in the target vector may not be filled in: it
would be possible to fill them in with entries that were
eCos-specific, but that could cause problems for people using the
simulator for other code. Although many gdb maintainers are employed
by Red Hat, the code is owned by the FSF and belongs to the community
as a whole: Red Hat cannot just make eCos-specific changes if those
could cause problems for other people.

If gdb is using the "native linux" target vector then the
thread-related entries in the target vector know about Linux threads,
not eCos threads.

To provide thread-aware eCos debugging in an environment which does
not involve gdb stubs, e.g. the simulator or the synthetic target, you
can implement the necessary functions in the target vector on top of
existing functionality. For example, the target vector entry for
getting hold of the current thread could resolve the symbol
Cyg_Scheduler::current_thread and read the corresponding memory
location.

So it would be possible to define additional eCos-specific target
vectors. You could have a target sim-eCos which is just like the
normal sim target, but replaces certain entries with eCos-specific
ones. Similarly for the native Linux gdb you could have a target
vector for synthetic-eCos, which behaves mostly like the normal Linux
target vector except when it comes to thread-aware debugging. This
would all have to be done at the C level - there is no C++ inside gdb
so you cannot do these things through derived classes and virtual
vectors. A major problem is that if something changes inside the RTOS,
e.g. the variable identifying the current thread is renamed for some
reason, then the gdb sources would have to be edited and gdb would
have to be renamed. This could be especially problematical for highly
configurable systems such as eCos.

A cleaner approach would be to make the target vector scriptable.
Some of the entries in the target vector would no longer be direct C
functions, instead they would invoke a scripting language interpreter
and run an appropriate command. Those commands could in turn access
other entries in the target vector, e.g. to perform low-level
operations like reading a location in memory. If you want to debug an
eCos application, you load in a script provided with eCos. If you are
using some other RTOS, that would provide its own script. If something
changed in the internals of the RTOS there would be no need to rebuild
gdb, instead you just use an updated script.

Adding scripting functionality would give a great deal of flexibility:
things like thread-aware debugging could now be made to work in the
synthetic target, or in an architectural simulator, or via an ICE, or
whatever. However the effort involved would be considerable, probably
including a lengthy debate within the gdb community on which scripting
language should be used (my guess is that they would end up with
guile, althoug I would prefer Tcl :-).

>>>>> "Jifl" == Jonathan Larmour <jlarmour@redhat.com> writes:

    Jifl> Actually I would have thought the easiest route would be to
    Jifl> allow included stubs in the synth target to be available at
    Jifl> a socket, and then using the remote protocol. Making it
    Jifl> available as a socket should "just" be a case of adding a
    Jifl> virtual vectored comm interface. It's probably a bit too
    Jifl> much work for us to just do for fun, particularly since the
    Jifl> synth target doesn't use virtual vectors at all right now.

I don't how this could sensibly work for the synthetic target. We are
not using the remote protocol, we are using native debugging. 

Yes, theoretically it would be possible to run something like Redboot
inside the synthetic target, connect to it via a socket and the remote
protocol, and then boot a RAM-startup synthetic target application
into it. When debugging the target would run in polled mode, which is
now what you want for a synthetic application. Interrupting a running
program would involve detecting SIGIO on the socket from gdb and doing
the right thing, not impossible but messy. I would much prefer to
avoid this route. Fixing the problem in gdb instead would give a clean
and general-purpose solution.

Bart

Re: [ECOS] Debugging multi-threaded eCos application using GDB

[Jonathan Larmour]

Bart Veer wrote:
> 
> >>>>> "Jifl" == Jonathan Larmour <jlarmour@redhat.com> writes:
> 
>     Jifl> Actually I would have thought the easiest route would be to
>     Jifl> allow included stubs in the synth target to be available at
>     Jifl> a socket, and then using the remote protocol. Making it
>     Jifl> available as a socket should "just" be a case of adding a
>     Jifl> virtual vectored comm interface. It's probably a bit too
>     Jifl> much work for us to just do for fun, particularly since the
>     Jifl> synth target doesn't use virtual vectors at all right now.
> 
> I don't how this could sensibly work for the synthetic target. We are
> not using the remote protocol, we are using native debugging.
>
> Yes, theoretically it would be possible to run something like Redboot
> inside the synthetic target, connect to it via a socket and the remote
> protocol, and then boot a RAM-startup synthetic target application
> into it.

That's what I was thinking sort of - except you could just build your
application for "ROM" startup and include stubs instead. After all, this
only applies if you explicitly want thread debugging.

> When debugging the target would run in polled mode, which is
> now what you want for a synthetic application. Interrupting a running
> program would involve detecting SIGIO on the socket from gdb and doing
> the right thing, not impossible but messy.

A call to sigaction - I don't think that's a big deal. And polling isn't a
prerequisite - when waiting for input from GDB you can just do a blocking
read.

> I would much prefer to
> avoid this route. Fixing the problem in gdb instead would give a clean
> and general-purpose solution.

I'm not sure about clean for *GDB*, and I think it would be very difficult
to do right: The biggest problem is that even the layout of the kernel
structures depend on the configuration - just look at the definition of
Cyg_Thread - and that's not the only structure to analyse.

Perhaps a hybrid approach to yours may be to make sure that inferior calls
worked with the synthetic target, and write a series of GDB macros that did
the equivalent of info threads, but by calling into eCos. But there can
definitely be problems with this approach wrt interrupts and effectively
asynchronous state.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] Debugging multi-threaded eCos application using GDB

[Bart Veer]

>>>>> "Jifl" == Jonathan Larmour <jlarmour@redhat.com> writes:

    >> Yes, theoretically it would be possible to run something like
    >> Redboot inside the synthetic target, connect to it via a socket
    >> and the remote protocol, and then boot a RAM-startup synthetic
    >> target application into it.

    Jifl> That's what I was thinking sort of - except you could just
    Jifl> build your application for "ROM" startup and include stubs
    Jifl> instead. After all, this only applies if you explicitly want
    Jifl> thread debugging.

There would be various problems:

1) you would be using the native Linux target vector for starting up
   the application, then switching to the remote target vector. That
   is not exactly normal behaviour for a Linux gdb session. There
   could be all kinds of confusion if e.g. during initialization you
   set a few Linux native breakpoints.

2) in the synthetic target only ROM startup is currently supported,
   and the code region really is made read-only. Therefore attempting
   to insert a breakpoint instruction from inside gdb stubs is not
   going to work very well. Although RAM startup support could be
   added, it would be a shame to lose functionality such as
   applications being unable to overwrite their own code by accident.

3) Linux ptrace debugging probably uses more advanced functionality
   than typical gdb stubs, e.g. hardware breakpoints and watchpoints.
   It would be a shame to lose those.

Yes, something along your lines is possible. However I would much
prefer to see it fixed generally, by making the gdb target vector
scriptable, and thus solving the problem for simulators etc. at the
same time. All this assumes that the gdb folks do not have their own
plans in this area, of course.

Bart