From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 964203858D1E for ; Wed, 20 Apr 2022 15:32:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 964203858D1E Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654-YhDRQ-7_MnyRQoBs8K-wJg-1; Wed, 20 Apr 2022 11:32:32 -0400 X-MC-Unique: YhDRQ-7_MnyRQoBs8K-wJg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C3DEB3810D26; Wed, 20 Apr 2022 15:32:31 +0000 (UTC) Received: from [10.2.17.182] (unknown [10.2.17.182]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6C9DB40D0161; Wed, 20 Apr 2022 15:32:31 +0000 (UTC) Message-ID: <41037236-1123-9d34-178e-65a840ff3617@redhat.com> Date: Wed, 20 Apr 2022 08:32:29 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: [RFC] GDB sanity test To: Serhei Makarov , Bunsen References: <20220419190117.903678-1-keiths@redhat.com> <21b838a8-1744-4c3b-8f41-f2c44ffcc9e3@www.fastmail.com> From: Keith Seitz In-Reply-To: <21b838a8-1744-4c3b-8f41-f2c44ffcc9e3@www.fastmail.com> X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-13.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: bunsen@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bunsen mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2022 15:32:35 -0000 On 4/19/22 13:33, Serhei Makarov wrote: > On Tue, Apr 19, 2022, at 3:01 PM, Keith Seitz via Bunsen wrote: >> This RFC is meant to start a discussion about testing. As bunsen >> starts to mature and more people rely on it, it is imperative that >> contributions don't randomly break other projects. >> >> In that vein, this is a basic sanity test that I've written for GDB >> which imports a single gdb.{log,sum} from a test fixture (.tar.xz). >> It double-checks all results and test counts. > Correct, & I have other ideas for operations to test > (e.g. duplicate-add, list testruns, update, delete) based on your > example. > > Any other operations that come to mind as a priority to test? My priority right now is correctness testing, starting with, e.g, the summarize script. After that my next step is to test the log annotation and see if that is working as expected. >> To do this, I've written a custom gdb test script that outputs every >> valid test outcome and run this on a fedora 33 VM. >> >> I'm only including the actual test file, not the glue code >> (tests/core.py) necessary to make it work. > It looks good to me. I'm fine if you commit it and I add some SystemTap variants > subject to the following caveats/questions: > > - Perhaps the test data is better to commit as plaintext rather than binary .tar.xz? > When needed, a tar or tar.xz can be prepared with tar cJf /path/to/sample/data/* I can do that. > - We'll want to run the same sequence of test operations on different data sets, > so I would need to rework your fixture to allow that (e.g. perhaps > expected result counts / etc. could be specified in the fixture and the > checking procedure generalized). Similar fixtures could be created > for more realistic SystemTap and GDB data (to test the parser's edge cases), > and perhaps for other completely-synthetic data (where I can encode > some regressions that we would want the downstream analysis to capture). Yeah, that is something that I've only cursorily thought about as a generalized solution. This sanity test (which really is sort of a basic gdb parser test) is, IMO, okay to hardcode the expected results. I agree we will want some infrastructure to facilitate writing tests. I guess if we rely on DejaGNU .log/.sum files as test data, we could write up a function to grok the results at the end of the .sum file to compare with what the actual bunsen parsers read. That is essentially how/why I wrote summarize.py. Any other ideas how we might want to do this? > - Since there can be Bunsen setups operating on private data, ideally > I'd like to make sure they can set up their own fixtures to run this > testsuite against and check for breakage, the same way that Bunsen > itself can invoke local scripts stored in .bunsen/scripts-whatever/. > Pretty sure a way to do that can be concocted with these standard Python > testing tools. The one obvious missing bit is being able to easily invoke tests. Right now, I just rely on "python3 -m pytest tests" to run all tests. Maybe a makefile or shell script to do this would be appropriate. > Couple of comments on the code: > >> diff --git a/tests/gdb/test_gdb_sanity.py b/tests/gdb/test_gdb_sanity.py >> new file mode 100644 >> index 0000000..1ce65be >> --- /dev/null >> +++ b/tests/gdb/test_gdb_sanity.py >> @@ -0,0 +1,82 @@ >> +# A quick sanity test that the ensures that bunsen can import a really >> +# basic log/sum test run which contains all test outcomes. >> + >> +import os.path >> +from tests.core import setup_bunsen_repo, import_results > Out of curiousity, is that glue code calling out to shell operations, > or is it invoking methods in class Bunsen directly? It is 100% python. Between wrapping python API calls in a shell script for CLI usage and wrapping shell scripts as API calls for python, I prefer the former, and that's what I've implemented. Writing python libraries as shell script wrappers is beyond my pain threshold. I am writing python-based utilities/web services, not a cron job or ancient (and insecure) CGI script. >> +TEST_FIXTURE = 'tests/gdb/fixtures/test_gdb_sanity/gdb-sanity.tar.xz' >> +PATH_STRING = '/home/fedora33/work/gdb/fsf/virgin/linux/gdb/testsuite' > If the data is synthetic (and even if it isn't), we could do a search and replace > to change it to something more generic and less revealing of your work setup :) This reveals nothing other than how the VM I used to generate the data is setup. I'll continue hacking at this a bit until I'm satisfied that things are genuinely useful. I've discovered a few minor issues with the gdb parser that I'd like to correct, and that will affect this test. Keith