libGDB and gdbserver questions

libGDB and gdbserver questions

[Donny Kurniawan]

Hello,

I am trying to write a remote debugger for multiple processes. As far
as I know, most of the parallel debuggers (that use GDB as the
backend), use multiple instances of GDB, where each instance controls
one process.

The problem is: I want to be able to handle a lot of processes (1000
processes is the minimum), clearly having 1000 instances of GDB
controlling 1000 processes is not very scalable, and it will choke the
"master UI controller" (that controls GDBs).

I've been thinking about 2 solutions:

(1) gdbserver
Since one gdbserver requires one instance of GDB, my plan is to use
gdbserver as the low-level debugging engine and extend GDB to be able
to handle multiple connections from gdbserver (correct me if I'm being
too simplistic). Is there a support for it in GDB?

(2) libGDB
Actually, what's the state of libGDB (v. 2)? Not much docs that I can
find, only http://sources.redhat.com/gdb/papers/libgdb2/ and the
source code. Is it going to be more suitable to address the problem
mentioned above.

Any pointers/directions/opinions are appreciated...

I tried googling and looking at the mailing list, but the minimum
requirement to be able to handle 1000 procs makes the problem unusual.

Thank you,
Donny

Re: libGDB and gdbserver questions

[Daniel Jacobowitz]

On Tue, Nov 15, 2005 at 11:48:03PM +1100, Donny Kurniawan wrote:
> (1) gdbserver
> Since one gdbserver requires one instance of GDB, my plan is to use
> gdbserver as the low-level debugging engine and extend GDB to be able
> to handle multiple connections from gdbserver (correct me if I'm being
> too simplistic). Is there a support for it in GDB?

This isn't the problem.  Extending the GDB native backend to handle
multiple processes wouldn't be hard; it's extending the user interface
and process control and breakpoint management and shared library
support and so on to handle multiple processes that is difficult.

GDB simply doesn't support what you want to do.  Yet.

> (2) libGDB
> Actually, what's the state of libGDB (v. 2)? Not much docs that I can

Nothing has happened on this front in a long long time.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: libGDB and gdbserver questions

[Vladimir Prus]

Donny Kurniawan wrote:

> Hello,
> 
> I am trying to write a remote debugger for multiple processes. As far
> as I know, most of the parallel debuggers (that use GDB as the
> backend), use multiple instances of GDB, where each instance controls
> one process.
> 
> The problem is: I want to be able to handle a lot of processes (1000
> processes is the minimum), clearly having 1000 instances of GDB
> controlling 1000 processes is not very scalable, and it will choke the
> "master UI controller" (that controls GDBs).

I'm not sure what you mean about "master UI controller". The biggest
scalability problem is that each gdb will load symbol table for your
program, and for real program that might take 100M of memory. Multiply that
by 1000 and you need 64-bit box.

However, it's not likely that you have 1000 different programs. If you have
two programs, and each one is run on 500 machines, you can start two copies
of gdb.

Then each copy of gdb would connect to a "redirector" you can write, that
will basically forward all packets to invididual instances of gdbserver.
But it will present those 500 instances as 500 threads, and gdb can work
with threads more or less fine.

As least, that what I plan to do for multi-program debugging.

- Volodya

Re: libGDB and gdbserver questions

[Donny Kurniawan]

> > "master UI controller" (that controls GDBs).
>
> I'm not sure what you mean about "master UI controller". The biggest

What I mean is the UI that is presented to the user, which manages
breakpoints, execution, stepping etc. for multiple processes.

> However, it's not likely that you have 1000 different programs. If you have

It is only one single program being run for 1000 times (1000 processes).

> two programs, and each one is run on 500 machines, you can start two copies
> of gdb.
>
> Then each copy of gdb would connect to a "redirector" you can write, that
> will basically forward all packets to invididual instances of gdbserver.
> But it will present those 500 instances as 500 threads, and gdb can work
> with threads more or less fine.

Hmmm.... I'm curious with this approach. I did write a "proxy"
redirector between gdbserver and gdb. But it only connects one
gdbserver to one gdb.

So basically with this approach (many gdbservers to one gdb), we
present processes as threads to gdb? How about debugging multi-thread
(!) program with many processes then? How do we present threads to
gdb?

It seems to me, this approach is a clean hack, but not the way it's
supposed to be.

Donny

>
> As least, that what I plan to do for multi-program debugging.
>
> - Volodya
>
>

Re: libGDB and gdbserver questions

[Donny Kurniawan]

> This isn't the problem.  Extending the GDB native backend to handle
> multiple processes wouldn't be hard; it's extending the user interface
> and process control and breakpoint management and shared library
> support and so on to handle multiple processes that is difficult.
>
> GDB simply doesn't support what you want to do.  Yet.
>

I do suspect that there are some "managers" (breakpoint manager,
stepping manager etc.) somewhere in GDB. Well, it can be clearly seen
that the UI (text UI, curses, etc.) doesn't support multiple
processes. But, from your reply (just to reconfirm), you are saying
that the managers themselves are not easily extended to support
multiple processes?

Donny

Re: libGDB and gdbserver questions

[Daniel Jacobowitz]

On Wed, Nov 16, 2005 at 10:45:06AM +1100, Donny Kurniawan wrote:
> > This isn't the problem.  Extending the GDB native backend to handle
> > multiple processes wouldn't be hard; it's extending the user interface
> > and process control and breakpoint management and shared library
> > support and so on to handle multiple processes that is difficult.
> >
> > GDB simply doesn't support what you want to do.  Yet.
> >
> 
> I do suspect that there are some "managers" (breakpoint manager,
> stepping manager etc.) somewhere in GDB. Well, it can be clearly seen
> that the UI (text UI, curses, etc.) doesn't support multiple
> processes. But, from your reply (just to reconfirm), you are saying
> that the managers themselves are not easily extended to support
> multiple processes?

None of GDB supports multiple processes.

Presenting them as threads would work with a sufficiently clever stub,
but be very inefficient.  For instance, a breakpoint would have to be
manually inserted into each one of them.  GDB also does not have any
notion of "thread-specific" memory - if your processes don't share
memory, then getting GDB to read the correct thread's memory will be
hard.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: libGDB and gdbserver questions

[Vladimir Prus]

Donny Kurniawan wrote:

>> > "master UI controller" (that controls GDBs).
>>
>> I'm not sure what you mean about "master UI controller". The biggest
> 
> What I mean is the UI that is presented to the user, which manages
> breakpoints, execution, stepping etc. for multiple processes.

Well, UI has to deal with 1000 threads of executions anyway, be it
processes, threads, or anything. It's not trivial, but independent of the
number of gdb instances, IMO.

>> However, it's not likely that you have 1000 different programs. If you
>> have
> 
> It is only one single program being run for 1000 times (1000 processes).

Not MPI, per chance? 
Actually, I always though it's a good idea to add multi-process debugging
(including MPI) to KDevelop (http://kdevelop.org). Maybe, we can team up
for that?

>> two programs, and each one is run on 500 machines, you can start two
>> copies of gdb.
>>
>> Then each copy of gdb would connect to a "redirector" you can write, that
>> will basically forward all packets to invididual instances of gdbserver.
>> But it will present those 500 instances as 500 threads, and gdb can work
>> with threads more or less fine.
> 
> Hmmm.... I'm curious with this approach. I did write a "proxy"
> redirector between gdbserver and gdb. But it only connects one
> gdbserver to one gdb.
> 
> So basically with this approach (many gdbservers to one gdb), we
> present processes as threads to gdb? How about debugging multi-thread
> (!) program with many processes then? How do we present threads to
> gdb?

Heh, that's problem of your "master UI controller". I'd extend the remote
protocol to allow UI to query which real process and real thread each
gdb-visible thread corresponds to.

> It seems to me, this approach is a clean hack, but not the way it's
> supposed to be.

Unfortunately, I don't know better way. Refactoring gdb to support several
connections to remote targets and to add an explicit notion of "process"
might be very hard.

- Volodya

Re: libGDB and gdbserver questions

[Konstantin Karganov]

> Then each copy of gdb would connect to a "redirector" you can write, that
> will basically forward all packets to invididual instances of gdbserver.
MPICH already has such wrapper (I was sure it was, but couldn't find
it yesterday...):
It has a "parallel gdb" version called mpigdb:
http://www-unix.mcs.anl.gov/mpi/mpich/docs/userguide/node26.htm#Node29

It wraps the stdio and allows to send gdb commands either to all nodes
or to one specified node. The worst thing is that it works
synchronously - when you issue "step" to one node you can not issue
other commands (even switching to another node) until the step
completes.

As for me, such solution is not suitable even for small MPI programs,
because of process interaction and blocking of the individual steps on
MPI_Recv() and similar.
Anyhow, I don't know the interactive debugger that is capable of
debugging 1000 and more processes.

--
Best regards,
Konstantin.

Re: libGDB and gdbserver questions

[Daniel Jacobowitz]

On Wed, Nov 16, 2005 at 10:58:25AM +0300, Vladimir Prus wrote:
> Unfortunately, I don't know better way. Refactoring gdb to support several
> connections to remote targets and to add an explicit notion of "process"
> might be very hard.

Yes, I think it will be - however, it needs to get done eventually, I
think.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: libGDB and gdbserver questions

[Stan Shebs]

Daniel Jacobowitz wrote:

>On Wed, Nov 16, 2005 at 10:58:25AM +0300, Vladimir Prus wrote:
>
>>Unfortunately, I don't know better way. Refactoring gdb to support several
>>connections to remote targets and to add an explicit notion of "process"
>>might be very hard.
>>
>
>Yes, I think it will be - however, it needs to get done eventually, I
>think.
>
It was on the original GDB 5 plan I believe - I guess now it would
have to be part of a GDB 7 plan... :-)

In the meantime, we could encourage the direction by accepting
refactoring-type patches that gather random inferior-related
globals into objects.

Stan

Re: libGDB and gdbserver questions

[Donny Kurniawan]

> > It is only one single program being run for 1000 times (1000 processes).
>
> Not MPI, per chance?

Well, currently I'm putting scalability as the main objective (not the
parallel/MPI functionality).


Donny

Re: libGDB and gdbserver questions

[Donny Kurniawan]

> It has a "parallel gdb" version called mpigdb:
> http://www-unix.mcs.anl.gov/mpi/mpich/docs/userguide/node26.htm#Node29
>
> It wraps the stdio and allows to send gdb commands either to all nodes
> or to one specified node. The worst thing is that it works
> synchronously - when you issue "step" to one node you can not issue
> other commands (even switching to another node) until the step
> completes.
>

Hi Konstantin,

We used mpigdb in the past. But, clearly it will choke if we use it to
handle 1000 procs. Thus, we're looking for a very lightweight debug
engine.

Donny