From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id 94CCE3856946 for ; Tue, 21 Jun 2022 15:32:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 94CCE3856946 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 65B64165C; Tue, 21 Jun 2022 08:32:17 -0700 (PDT) Received: from [10.57.41.51] (unknown [10.57.41.51]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9D3A23F66F; Tue, 21 Jun 2022 08:32:16 -0700 (PDT) Message-ID: <12c9bf5c-e3b5-1260-16fe-1916ece7d904@foss.arm.com> Date: Tue, 21 Jun 2022 16:32:15 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: Are big tests allowed in binutils? Content-Language: en-GB To: Dmitry Selyutin , binutils@sourceware.org Cc: Luke Leighton References: From: Richard Earnshaw In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-3490.8 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: binutils@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Binutils mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jun 2022 15:32:19 -0000 On 17/06/2022 06:11, Dmitry Selyutin via Binutils wrote: > Hi folks, > > I'm going to submit a series of patches which add a couple of new > instructions available as SVP64 extensions (under -mlibresoc switch). > Since I want to be a good citizen, I also want to introduce tests for > each instruction, basically comparing the expected results for dumps > (aka run_dump_test). However, I'd like to cover various possible > cases, so I generated a test to cover many different operands; this > makes tests quite big. For example, the assembly listing for a new > svremap instruction takes 10240 lines, and etalon dump takes a bit > more lines. So, questions: > 1. Is it OK at all to have tests that large? Time-wise they're fast, > I'm mostly concerned about code base size and readers' patience. > 2. Will you tolerate it if I post these tests in the mailing list? > These will be a major headache for any reader if integrated into the > patches. > 3. If it's OK to post these tests into the mailing lists, is it > allowed to post them as separate patches from the ones which add the > instructions? > > My preferred option would be to split each patch into two parts: > 1. The patch which adds the instruction itself. > 2. The patch which adds the test for this instruction. > > However, looking at patches in PowerPC, I see that the tests generally > come along with the implementation in scope of one patch. I'm afraid > this might make patches barely readable. > If needed, I can cut the size of tests; time-wise it's fast, 10420 > lines of asm take 0,04s of real time, I'm only concerned about > readability. > > I'd like to know your opinion on this. Thank you! > Good testing is about being smart. Tests get rerun many times and although each run is not too expensive, over time it adds up. Furthermore, most instructions in an architecture often follow standard 'shapes' in order to make decoding fast for the hardware. So registers numbers are often encoded in consecutive bits at well-known places within the instruction. You can use this knowledge to significantly reduce the number of tests that need to be written. For example, if you have an instruction FOO that takes two 4-bit register numbers (a 16-register architecture), and register r0 is all bits zero and r15 is all bits one, you might have a test FOO r0, r0 FOO r0, r15 FOO r15, r0 These three tests would first check that the base opcode for FOO is correct, the second that the register bits for the second register are all encoded into the right spot and the third that the register bits for the first register are placed correctly. That's three tests rather than 256 that would be needed to test all register permutations. For a select number of instructions you might check all register numbers are correctly encoded, but it shouldn't be necessary to do that for every instruction - the code that inserts the bits is probably common to all related instructions and you'd just be repeatedly testing that with no added value. The same is also true for literal values encoded in instructions: only a few tests are likely to stress the boundaries of your parsing and encoding, so focus on testing those boundaries. This obviously won't work for every instruction. Sometimes there are encoding restrictions, for example you can't both read and write a register in the same instruction. But most of the time careful choice of what you test can significantly reduce the size of the testsuite and that in turn can make tracking down a problem much easier when you do get a regression. R.