From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from eggs.gnu.org (eggs.gnu.org [IPv6:2001:470:142:3::10]) by sourceware.org (Postfix) with ESMTPS id 525053985C54 for ; Wed, 18 Aug 2021 19:55:58 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 525053985C54 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36120) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mGRfK-0006cn-12 for gdb@sourceware.org; Wed, 18 Aug 2021 15:55:58 -0400 Received: from [160.231.0.90] (port=29911 helo=pdslaptop) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mGRfJ-0000pF-SR for gdb@sourceware.org; Wed, 18 Aug 2021 15:55:57 -0400 Message-ID: <3ba28620390947de2a84cabbd1958852aad97d5e.camel@gnu.org> Subject: Tools to debug multiple cores/processes at the same time? From: Paul Smith Reply-To: psmith@gnu.org To: gdb@sourceware.org Date: Wed, 18 Aug 2021 15:55:57 -0400 Organization: GNU's Not UNIX! Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5-0ubuntu1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=BAYES_20, KAM_DMARC_STATUS, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Aug 2021 19:55:59 -0000 In my environment I'm working on a distributed system where the same process is running on a number of different systems. The way these processes work is that they should all have (approximately) the same set of in-memory content at the same time. Our test environment is set up so that if something "bad" happens, we SIGABRT all the processes on all the systems, so I often end up with 4, 5, 6, or more core files, all from the same process, with approximately the same in-memory content (obviously not identical addresses but the same structure) taken at about the same time. The host systems can be assumed, at least initially, to be identical so no issue with different system libraries etc. What would be really useful is for me to be able to open all these cores at the same time, controlled with a single UI (CLI prompt is fine), and be able to both (a) run commands that manipulate a single core, and also (b) run commands which manipulate all core files at once. That would include, for example, setting convenience variables locally for each core, but then running the same command that might use those variables on every core and getting back the results. I'm not really sure how this might work best: maybe a separate terminal per core file and one "controlling" terminal that I would enter commands into, or even commands run one after the other in the same terminal with some kind of header. Don't know. I can imagine multiple ways it might work. What I'm really wondering is, has anyone heard of something like this, or is there any support (even partial support) for something like this in GDB today? I get that it seems like a somewhat specialized request. Debugging multiple live processes at the same time would also be great, of course.