Tools to debug multiple cores/processes at the same time?

Tools to debug multiple cores/processes at the same time?

[Paul Smith]

In my environment I'm working on a distributed system where the same
process is running on a number of different systems.  The way these
processes work is that they should all have (approximately) the same
set of in-memory content at the same time.

Our test environment is set up so that if something "bad" happens, we
SIGABRT all the processes on all the systems, so I often end up with 4,
5, 6, or more core files, all from the same process, with approximately
the same in-memory content (obviously not identical addresses but the
same structure) taken at about the same time.

The host systems can be assumed, at least initially, to be identical so
no issue with different system libraries etc.

What would be really useful is for me to be able to open all these
cores at the same time, controlled with a single UI (CLI prompt is
fine), and be able to both (a) run commands that manipulate a single
core, and also (b) run commands which manipulate all core files at
once.

That would include, for example, setting convenience variables locally
for each core, but then running the same command that might use those
variables on every core and getting back the results.

I'm not really sure how this might work best: maybe a separate terminal
per core file and one "controlling" terminal that I would enter
commands into, or even commands run one after the other in the same
terminal with some kind of header.

Don't know.  I can imagine multiple ways it might work.

What I'm really wondering is, has anyone heard of something like this,
or is there any support (even partial support) for something like this
in GDB today?  I get that it seems like a somewhat specialized request.

Debugging multiple live processes at the same time would also be great,
of course.

Re: Tools to debug multiple cores/processes at the same time?

[John Baldwin]

On 8/18/21 12:55 PM, Paul Smith via Gdb wrote:
> What I'm really wondering is, has anyone heard of something like this,
> or is there any support (even partial support) for something like this
> in GDB today?  I get that it seems like a somewhat specialized request.
> 
> Debugging multiple live processes at the same time would also be great,
> of course.

I haven't tried this myself, but you might be able to do this by adding
multiple inferiors (one inferior per process).  You can use 'add-inferior'
to add new inferiors and 'info inferiors' to list them.  'inferior N'
switches between inferiors, though 'info threads' also shows threads
from all inferiors and you can switch by threads as well.  When
debugging a process that forks and have detach-on-fork disabled, each
process ends up in a separate inferior.  I haven't tried, but you can
perhaps use 'core' or 'attach' in each inferior to attach to multiple
cores or processes.  There is a knob that affects scheduling (does
a continue all processes or just the current one?) for the live case.
I'm not sure if you can have convenience variables be per-inferior,
and I'm not sure if we currently have a convenience variable that
exposes the current inferior.  I suspect you probably would have access
to that if you were to write your custom commands in python rather
than as user commands.

-- 
John Baldwin

Re: Tools to debug multiple cores/processes at the same time?

[Paul Koning]

> On Aug 18, 2021, at 3:55 PM, Paul Smith via Gdb <gdb@sourceware.org> wrote:
> 
> In my environment I'm working on a distributed system where the same
> process is running on a number of different systems.  The way these
> processes work is that they should all have (approximately) the same
> set of in-memory content at the same time.
> ...
> What I'm really wondering is, has anyone heard of something like this,
> or is there any support (even partial support) for something like this
> in GDB today?  I get that it seems like a somewhat specialized request.
> 
> Debugging multiple live processes at the same time would also be great,
> of course.

There is multiple target support, which I haven't had a chance to try yet but from what I remember reading is a recently added GDB feature.  It should handle your case plus even stranger ones, such as multiple processes running different code, or even running on different instruction sets.  It's just what you need for debugging distributed applications.

	paul

Re: Tools to debug multiple cores/processes at the same time?

[Paul Smith]

On Wed, 2021-08-18 at 16:53 -0400, Paul Koning wrote:
> There is multiple target support, which I haven't had a chance to try
> yet but from what I remember reading is a recently added GDB
> feature.  It should handle your case plus even stranger ones, such as
> multiple processes running different code, or even running on
> different instruction sets.  It's just what you need for debugging
> distributed applications.

Yeah, I saw that, and it might be the kernel of something that could
support what I'm talking about, except I don't think it manages
different "convenience variable namespaces" per inferior.

But the main thing is it doesn't seem like it supports running the same
command in all inferiors, other than by manually setting each one and
running the command.  Maybe some kind of embedded Python facility could
be created to do this, such as John mentions... I'm not sure what the
python support is currently for multiple inferiors.

Let me give an example of a debugging session:

* start a session with 3 core files

* run a command that shows info about each core (assume a python
"showinfo" command that show all threads with some details about each
one).  Maybe a command like:
  (gdb) all: showinfo
  i1: <info>
  i2: <info>
  i3: <info>

* For each core, go to a certain thread (of course the thread numbers
will be different in each core):
  (gdb) i1: thr 7
  (gdb) i2: thr 9
  (gdb) i3: thr 23

* Run a command that returns some value and stores it in a convenience
variable, on all cores:
  (gdb) all: set $v = $findstuff()

* Run a command that shows some value, on all cores:
  (gdb) all: p $v->someDetail
  i1: $1 = 7
  i2: $1 = 44
  i3: $1 = 165

I dunno, something like that.  Alternatively maybe it would be simpler
for output for each inferior to go to a separate terminal or whatever.

Re: Tools to debug multiple cores/processes at the same time?

[Philippe Waroquiers]

On Wed, 2021-08-18 at 17:09 -0400, Paul Smith via Gdb wrote:
> On Wed, 2021-08-18 at 16:53 -0400, Paul Koning wrote:
> > There is multiple target support, which I haven't had a chance to try
> > yet but from what I remember reading is a recently added GDB
> > feature.  It should handle your case plus even stranger ones, such as
> > multiple processes running different code, or even running on
> > different instruction sets.  It's just what you need for debugging
> > distributed applications.
> 
> Yeah, I saw that, and it might be the kernel of something that could
> support what I'm talking about, except I don't think it manages
> different "convenience variable namespaces" per inferior.
> 
> But the main thing is it doesn't seem like it supports running the same
> command in all inferiors, other than by manually setting each one and
> running the command.  Maybe some kind of embedded Python facility could
> be created to do this, such as John mentions... I'm not sure what the
> python support is currently for multiple inferiors.
> 
> Let me give an example of a debugging session:
> 
> * start a session with 3 core files
> 
> * run a command that shows info about each core (assume a python
> "showinfo" command that show all threads with some details about each
> one).  Maybe a command like:
>   (gdb) all: showinfo
>   i1: <info>
>   i2: <info>
>   i3: <info>
info threads should show the threads of all inferiors.

> 
> * For each core, go to a certain thread (of course the thread numbers
> will be different in each core):
>   (gdb) i1: thr 7
>   (gdb) i2: thr 9
>   (gdb) i3: thr 23
thread INFERIORNR.THREADNR
will switch to the listed thread.

> 
> * Run a command that returns some value and stores it in a convenience
> variable, on all cores:
>   (gdb) all: set $v = $findstuff()
> 
> * Run a command that shows some value, on all cores:
>   (gdb) all: p $v->someDetail
>   i1: $1 = 7
>   i2: $1 = 44
>   i3: $1 = 165
> 
> I dunno, something like that.  Alternatively maybe it would be simpler
> for output for each inferior to go to a separate terminal or whatever.
The command  'thread apply' can apply a command to all threads
or to a selected list of threads.
If you have multiple inferiors, the command can then be run for all threads
of all inferiors.
I do not see an (easy) way to run a command only once per inferior,
when an inferior has multiple threads.
For this, likely, the best would be to have a new gdb command such as
   inferior apply all | INFERIOR-LIST

Note that the command 'show convenience' lists a convenience variable $_inferior.

Philippe

RE: Tools to debug multiple cores/processes at the same time?

[Aktemur, Tankut Baris]

> Let me give an example of a debugging session:
> 
> * start a session with 3 core files
> 
> * run a command that shows info about each core (assume a python
> "showinfo" command that show all threads with some details about each
> one).  Maybe a command like:
>   (gdb) all: showinfo
>   i1: <info>
>   i2: <info>
>   i3: <info>
> 
> * For each core, go to a certain thread (of course the thread numbers
> will be different in each core):
>   (gdb) i1: thr 7
>   (gdb) i2: thr 9
>   (gdb) i3: thr 23
> 
> * Run a command that returns some value and stores it in a convenience
> variable, on all cores:
>   (gdb) all: set $v = $findstuff()
> 
> * Run a command that shows some value, on all cores:
>   (gdb) all: p $v->someDetail
>   i1: $1 = 7
>   i2: $1 = 44
>   i3: $1 = 165
> 
> I dunno, something like that.  Alternatively maybe it would be simpler
> for output for each inferior to go to a separate terminal or whatever.

Hi,

Have you considered using the "thread apply" command?

-Baris


Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Re: Tools to debug multiple cores/processes at the same time?

[Andrew Burgess]

* Paul Smith via Gdb <gdb@sourceware.org> [2021-08-18 15:55:57 -0400]:

> In my environment I'm working on a distributed system where the same
> process is running on a number of different systems.  The way these
> processes work is that they should all have (approximately) the same
> set of in-memory content at the same time.
> 
> Our test environment is set up so that if something "bad" happens, we
> SIGABRT all the processes on all the systems, so I often end up with 4,
> 5, 6, or more core files, all from the same process, with approximately
> the same in-memory content (obviously not identical addresses but the
> same structure) taken at about the same time.
> 
> The host systems can be assumed, at least initially, to be identical so
> no issue with different system libraries etc.
> 
> What would be really useful is for me to be able to open all these
> cores at the same time, controlled with a single UI (CLI prompt is
> fine), and be able to both (a) run commands that manipulate a single
> core, and also (b) run commands which manipulate all core files at
> once.
> 
> That would include, for example, setting convenience variables locally
> for each core, but then running the same command that might use those
> variables on every core and getting back the results.
> 
> I'm not really sure how this might work best: maybe a separate terminal
> per core file and one "controlling" terminal that I would enter
> commands into, or even commands run one after the other in the same
> terminal with some kind of header.
> 
> Don't know.  I can imagine multiple ways it might work.
> 
> What I'm really wondering is, has anyone heard of something like this,
> or is there any support (even partial support) for something like this
> in GDB today?  I get that it seems like a somewhat specialized request.
> 
> Debugging multiple live processes at the same time would also be great,
> of course.

I'm going to suggest something a little different.

Sometimes I need to debug the same thing running in two "almost"
identical setups, where in one case something will go wrong, and in
another everything's fine.  I start up two GDB sessions, and want to
send the same commands to each GDB until I find where the two systems
start to diverge.

I make use of a terminal call Terminator[1], this terminal allows me
to place multiple terminals into a single "group" and broadcast
keystrokes to every terminal in that group.  I can always pull a
terminal out of the group and interact with one session alone, then
place it back into the group, so long as I leave it in a consistent
state with the other group members I can continue to debug them all in
sync.

I'm sure there are other terminals that must offer this functionality.

The suggestions from others for multi-target and/or custom python
commands are all good too, but that above approach has always been
good enough for my needs, and might be of use to you.

Thanks,
Andrew

[1] https://terminator-gtk3.readthedocs.io/en/latest/