From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by sourceware.org (Postfix) with ESMTPS id 8F71E3858400 for ; Thu, 19 Aug 2021 11:04:27 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8F71E3858400 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=embecosm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=embecosm.com Received: by mail-wr1-x429.google.com with SMTP id u16so8473774wrn.5 for ; Thu, 19 Aug 2021 04:04:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=embecosm.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=3E7j3MUOFL9dW5TME7kBM5tQ1107nGeMAexLJWVhMhw=; b=MgoYej+BDnTGPx+tYpG03zHTrPJgTu/7URq9TDiJm7SO4lj22szGNR9v4jc1Nn4Te5 jLapTFlCdw4kAn9p5mDei8WkevW9rPiufLwWilUwOp0F0S6cWHc4w7yyFX+4LfueaY9H Uu4k72ft6kOp6NTso5ekZQa09VdsRXRKr07h/moXxFxIfWMCoKpjcQhrDovgrPariSnY HZXcKpXdagSMNDgSbI//U3aMSo5kkFvC6uH/R1/gs9nAhqiXKJXzT8CNoyNlE4G+yCIU VkGUNO337ubHfmde7/PZR8N7hbyZSLwluZiTAGcv4fYKYJCQmF8km/tiAcSGdRb/sjOz 4N8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=3E7j3MUOFL9dW5TME7kBM5tQ1107nGeMAexLJWVhMhw=; b=IkoFCsJBGnZIRIKbyJ/siSuarFF9w7Wbp3DGt8s4S5oyvo6dn/WWlcCiFnkqu7FfYP AZqj8n09+FC7XHYMxc9R/kd/HJ4sOp/297x6b1pxOxiepzqbYuZFcLyW7tGvsSlmFMlm bXfp0OTxfvvmQ9kr4lZz++yKvVW5H+dQN9Ey2Xajwuxys+LnPi4VEu3gezST3Cv0VS6V EpCkQP4oRVB1ms+xoD/cPY++uNm+3gFUQwPQhOXat0WPRo9a7kTVugk1UGdyC0XT1EY1 XV74FpMhw0VuIBkP2DZtWgPlUlEWdzwoA4byqklePfbsUpzRDUPZKaV6+qHyWSyZGbEW wCaA== X-Gm-Message-State: AOAM532NpqevroiWiHUQlDKhtq+Lr6HnG3MCv8A2s72mFjMyk4w2k4h7 vzk9OKvc9NiTA4Ul5eKExUDqakVpLXQ78w== X-Google-Smtp-Source: ABdhPJwebl3BwGCtTdxR2056gQDOQjuGzecKvmF436FMf7w5sXsO76c3baxB/5JFhZcW7UH2CQ/FzQ== X-Received: by 2002:adf:f00d:: with SMTP id j13mr3181121wro.14.1629371066612; Thu, 19 Aug 2021 04:04:26 -0700 (PDT) Received: from localhost (host86-188-49-44.range86-188.btcentralplus.com. [86.188.49.44]) by smtp.gmail.com with ESMTPSA id c6sm8457353wme.45.2021.08.19.04.04.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Aug 2021 04:04:26 -0700 (PDT) Date: Thu, 19 Aug 2021 12:04:24 +0100 From: Andrew Burgess To: psmith@gnu.org Cc: gdb@sourceware.org Subject: Re: Tools to debug multiple cores/processes at the same time? Message-ID: <20210819110424.GB4124615@embecosm.com> References: <3ba28620390947de2a84cabbd1958852aad97d5e.camel@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3ba28620390947de2a84cabbd1958852aad97d5e.camel@gnu.org> X-Operating-System: Linux/5.8.18-100.fc31.x86_64 (x86_64) X-Uptime: 11:35:41 up 1 day, 23:31, X-Editor: GNU Emacs [ http://www.gnu.org/software/emacs ] X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Aug 2021 11:04:38 -0000 * Paul Smith via Gdb [2021-08-18 15:55:57 -0400]: > In my environment I'm working on a distributed system where the same > process is running on a number of different systems. The way these > processes work is that they should all have (approximately) the same > set of in-memory content at the same time. > > Our test environment is set up so that if something "bad" happens, we > SIGABRT all the processes on all the systems, so I often end up with 4, > 5, 6, or more core files, all from the same process, with approximately > the same in-memory content (obviously not identical addresses but the > same structure) taken at about the same time. > > The host systems can be assumed, at least initially, to be identical so > no issue with different system libraries etc. > > What would be really useful is for me to be able to open all these > cores at the same time, controlled with a single UI (CLI prompt is > fine), and be able to both (a) run commands that manipulate a single > core, and also (b) run commands which manipulate all core files at > once. > > That would include, for example, setting convenience variables locally > for each core, but then running the same command that might use those > variables on every core and getting back the results. > > I'm not really sure how this might work best: maybe a separate terminal > per core file and one "controlling" terminal that I would enter > commands into, or even commands run one after the other in the same > terminal with some kind of header. > > Don't know. I can imagine multiple ways it might work. > > What I'm really wondering is, has anyone heard of something like this, > or is there any support (even partial support) for something like this > in GDB today? I get that it seems like a somewhat specialized request. > > Debugging multiple live processes at the same time would also be great, > of course. I'm going to suggest something a little different. Sometimes I need to debug the same thing running in two "almost" identical setups, where in one case something will go wrong, and in another everything's fine. I start up two GDB sessions, and want to send the same commands to each GDB until I find where the two systems start to diverge. I make use of a terminal call Terminator[1], this terminal allows me to place multiple terminals into a single "group" and broadcast keystrokes to every terminal in that group. I can always pull a terminal out of the group and interact with one session alone, then place it back into the group, so long as I leave it in a consistent state with the other group members I can continue to debug them all in sync. I'm sure there are other terminals that must offer this functionality. The suggestions from others for multi-target and/or custom python commands are all good too, but that above approach has always been good enough for my needs, and might be of use to you. Thanks, Andrew [1] https://terminator-gtk3.readthedocs.io/en/latest/