From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by sourceware.org (Postfix) with ESMTPS id D364B3858D3C for ; Tue, 19 Apr 2022 20:33:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D364B3858D3C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=serhei.io Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=serhei.io Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 8E0965C017F; Tue, 19 Apr 2022 16:33:59 -0400 (EDT) Received: from imap50 ([10.202.2.100]) by compute5.internal (MEProxy); Tue, 19 Apr 2022 16:33:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=serhei.io; h=cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1650400439; x=1650486839; bh=nqI60821kY Snej2lbrnBgc/Kd1S/3/2HfVcZyQCSOMI=; b=2r0OyT+UqPrF1vcQfHvCGS+obS plDuEorqnXL+gqSdiLu5T6OYb0kYlXmrxReyMdBbUqvO1A2gqikswifwFnAiOGGW XLwryuwWKW1j+hCtGyon3zd0gZjy5/YxZ01F5RZUprKVfDMJfhY78iFMLoP/IBjy cJdkyIEGBRm4qOA2YRreo8FxCdtRliZPpBj1ggNcebi3X1QafJPMp530hp7Ti02b vouGpZKiapQrZ9MdiYKTAsWGa9YmyLt5Hd6kPznimJOqUReudKFqFY5IN6kYfIDn 3jTZKDggbaSr5LpSWpF8/lAUMpqKbJp00ytqyvkQ7qTeQf3dW8ZtCSATYZ1Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1650400439; x= 1650486839; bh=nqI60821kYSnej2lbrnBgc/Kd1S/3/2HfVcZyQCSOMI=; b=G //loLpDL34kl1asdvo4czm8jqueggWXsAeMxTlnZZENNIcmowlBFl4ZTdCD1uVqK QsdIkdgnpYaHlgpyhomUPbUSQE5+CKpINAMUXfo/lcQCkvjrrC+gc6kPi31zb1je 2a/o4YNEc4Am0mXE8PfueQP88O35mmmtC9HkBiJL+KQ6bqtMtAXDfx4OxaVkRgqw 2iPeiSunV13I6+rBYHhxAVgHiu8mD/GgAsAE1HdMunhzX8VPN0PLZrgSslA41v9j 65+d0nSeYF+gzvRB3UfaeCqN8cvtPe61vqX1ERQCHvQzSxQcSd5+U7fK29lBVcyE XxPokwF+am3eehlZIJoqg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrvddtfedgudehudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfufgv rhhhvghiucforghkrghrohhvfdcuoehmvgesshgvrhhhvghirdhioheqnecuggftrfgrth htvghrnhepgfdvledvkeevueegueegffeufedvhfduveevhfegledtvdevfeeuheekveet uefhnecuffhomhgrihhnpeigiidrihhtnecuvehluhhsthgvrhfuihiivgeptdenucfrrg hrrghmpehmrghilhhfrhhomhepmhgvsehsvghrhhgvihdrihho X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 273F7192008C; Tue, 19 Apr 2022 16:33:59 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-568-g521196dd5d-fm-20220414.001-g521196dd Mime-Version: 1.0 Message-Id: <21b838a8-1744-4c3b-8f41-f2c44ffcc9e3@www.fastmail.com> In-Reply-To: <20220419190117.903678-1-keiths@redhat.com> References: <20220419190117.903678-1-keiths@redhat.com> Date: Tue, 19 Apr 2022 16:33:38 -0400 From: "Serhei Makarov" To: "Keith Seitz" , Bunsen Subject: Re: [RFC] GDB sanity test Content-Type: text/plain X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, JMQ_SPF_NEUTRAL, RCVD_IN_DNSWL_LOW, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: bunsen@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bunsen mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2022 20:34:02 -0000 On Tue, Apr 19, 2022, at 3:01 PM, Keith Seitz via Bunsen wrote: > This RFC is meant to start a discussion about testing. As bunsen > starts to mature and more people rely on it, it is imperative that > contributions don't randomly break other projects. > > In that vein, this is a basic sanity test that I've written for GDB > which imports a single gdb.{log,sum} from a test fixture (.tar.xz). > It double-checks all results and test counts. Correct, & I have other ideas for operations to test (e.g. duplicate-add, list testruns, update, delete) based on your example. Any other operations that come to mind as a priority to test? > To do this, I've written a custom gdb test script that outputs every > valid test outcome and run this on a fedora 33 VM. > > I'm only including the actual test file, not the glue code > (tests/core.py) necessary to make it work. It looks good to me. I'm fine if you commit it and I add some SystemTap variants subject to the following caveats/questions: - Perhaps the test data is better to commit as plaintext rather than binary .tar.xz? When needed, a tar or tar.xz can be prepared with tar cJf /path/to/sample/data/* - We'll want to run the same sequence of test operations on different data sets, so I would need to rework your fixture to allow that (e.g. perhaps expected result counts / etc. could be specified in the fixture and the checking procedure generalized). Similar fixtures could be created for more realistic SystemTap and GDB data (to test the parser's edge cases), and perhaps for other completely-synthetic data (where I can encode some regressions that we would want the downstream analysis to capture). - Since there can be Bunsen setups operating on private data, ideally I'd like to make sure they can set up their own fixtures to run this testsuite against and check for breakage, the same way that Bunsen itself can invoke local scripts stored in .bunsen/scripts-whatever/. Pretty sure a way to do that can be concocted with these standard Python testing tools. Couple of comments on the code: > diff --git a/tests/gdb/test_gdb_sanity.py b/tests/gdb/test_gdb_sanity.py > new file mode 100644 > index 0000000..1ce65be > --- /dev/null > +++ b/tests/gdb/test_gdb_sanity.py > @@ -0,0 +1,82 @@ > +# A quick sanity test that the ensures that bunsen can import a really > +# basic log/sum test run which contains all test outcomes. > + > +import os.path > +from tests.core import setup_bunsen_repo, import_results Out of curiousity, is that glue code calling out to shell operations, or is it invoking methods in class Bunsen directly? > +TEST_FIXTURE = 'tests/gdb/fixtures/test_gdb_sanity/gdb-sanity.tar.xz' > +PATH_STRING = '/home/fedora33/work/gdb/fsf/virgin/linux/gdb/testsuite' If the data is synthetic (and even if it isn't), we could do a search and replace to change it to something more generic and less revealing of your work setup :)