Tools to debug multiple cores/processes at the same time?

[Paul Smith <psmith@gnu.org>]

In my environment I'm working on a distributed system where the same
process is running on a number of different systems.  The way these
processes work is that they should all have (approximately) the same
set of in-memory content at the same time.

Our test environment is set up so that if something "bad" happens, we
SIGABRT all the processes on all the systems, so I often end up with 4,
5, 6, or more core files, all from the same process, with approximately
the same in-memory content (obviously not identical addresses but the
same structure) taken at about the same time.

The host systems can be assumed, at least initially, to be identical so
no issue with different system libraries etc.

What would be really useful is for me to be able to open all these
cores at the same time, controlled with a single UI (CLI prompt is
fine), and be able to both (a) run commands that manipulate a single
core, and also (b) run commands which manipulate all core files at
once.

That would include, for example, setting convenience variables locally
for each core, but then running the same command that might use those
variables on every core and getting back the results.

I'm not really sure how this might work best: maybe a separate terminal
per core file and one "controlling" terminal that I would enter
commands into, or even commands run one after the other in the same
terminal with some kind of header.

Don't know.  I can imagine multiple ways it might work.

What I'm really wondering is, has anyone heard of something like this,
or is there any support (even partial support) for something like this
in GDB today?  I get that it seems like a somewhat specialized request.

Debugging multiple live processes at the same time would also be great,
of course.