From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 24373 invoked by alias); 29 Oct 2015 19:32:46 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Received: (qmail 24351 invoked by uid 89); 29 Oct 2015 19:32:45 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Thu, 29 Oct 2015 19:32:44 +0000 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id 29063394D3D for ; Thu, 29 Oct 2015 19:32:43 +0000 (UTC) Received: from localhost.localdomain (ovpn-113-196.phx2.redhat.com [10.3.113.196]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t9TJWgRL004490; Thu, 29 Oct 2015 15:32:42 -0400 Subject: Re: [PATCH 00/16] Unit tests framework (v3) To: David Malcolm , Bernd Schmidt References: <5589B2FB.8010500@redhat.com> <1445975355-37660-1-git-send-email-dmalcolm@redhat.com> <5630B3C9.10409@redhat.com> <1446135023.26362.47.camel@surprise> Cc: gcc-patches@gcc.gnu.org From: Jeff Law Message-ID: <5632745A.6040004@redhat.com> Date: Thu, 29 Oct 2015 19:38:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <1446135023.26362.47.camel@surprise> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2015-10/txt/msg03275.txt.bz2 On 10/29/2015 10:10 AM, David Malcolm wrote: > > There may be useful things in gtest for us that I'm not using yet. For > example, the support for death tests may be useful for testing that e.g. > our checking macros kill the program sanely in the presence of malformed > internal data. Please check on this. I want to build a set of tests around the SSA_NAME manager, and that would include setting up can't happen situations and seeing if they're properly diagnosed by the checking code. So obviously I have an interest in how the framework handles this kind of scenario. > > I wanted to reuse the gtest code on the grounds that: > > * the code exists > > * it's maintained (https://github.com/google/googletest > shows the last commit was 10 days ago) > > * it has documentation: > https://github.com/google/googletest/blob/master/googletest/docs/Primer.md > (there's much more that just that) > > * it will be familiar to some people from other projects > > > That said, if it's a blocker, I can attempt a minimal reimplementation > of the subset of the API that I'm using. I'd probably lean towards continuing to use gtest. I don't currently see a compelling reason to build another framework. > >>> v1 of the kit was structured as a frontend, v2 of the kit as >>> a plugin that was built as if it were a frontend. Both of >>> these approaches were problematic, so this version >>> of the patch kit simply builds as a test case within the >>> plugin.exp suites. >> >> This is the part I'm least certain about, for two reasons: >> * They are more like source files than tests, and I think I'd prefer >> to have them alongside the source, in gcc/ rather than in the >> testsuite. This way people are invariable going to fail to notice >> them when they grep for something. > > So something like: > > gcc/test-vec.c > > or even: > > gcc/vec-tests.c > > so that it sorts next to gcc/vec.h in a listing? > > i.e. gcc/FOO-tests.c for gcc/FOO.c > > Maybe the unittests-plugin.c should live within the "gcc" directory? It's a bit of bike-shedding :-) I'd be happy with gcc/test-whatever or gcc/whatever-test. Both have pluses/minuses and in the end I doubt it'd matter much. My only worry here is doubling the number of files in gcc/. But I'd claim that we want to have more subdirectories for major logical areas. gimple/ rtl/ whatever, which would significantly mitigate the problem of too many files in gcc/ > >> * This uses a plugin into whatever compiler was built, but sometimes >> you can't really set up unit tests that way because what you want >> to test depends on target specifics. What I've often wanted is a >> special test target that gets built with a special machine >> description that has whatever patterns are needed to replicate >> tricky situations in reload or other optimization passes. >> >> The tests you have so far are focused mostly on high-level gimple/tree >> tests where this limitation is probably not showing up very much, but I >> think it would be better to have something that allows us to have more >> in-depth tests. > > I think our long-term goal should be at least 5 styles of test: > > (A) end-to-end tests of the compiler: running it on some source and > verifying properties of the diagnostics and/or generated code > (potentially running it). This is what we have today. > > (B) unit tests of individual subsystems: tests that exercise specific > internal APIs e.g. containers like vec<>, hash_map<> etc, also things > like gengtype. This is a gap in our current test coverage, and what > this patch kit aims to start filling. Right. It's essentially data structure and other very low level testing (gengtype). That's even true if I think about the SSA_NAME manager -- in an ideal world, it'd just be a container for objects that we want to release & recycle, whether they're SSA_NAMEs, pseudos, whatever we decide has the right properties to benefit from recycling. The fact that it's recycling SSA_NAMEs isn't for the most part all that important. > > (C) and (D): tests of specific passes: tests that accept IR (either > gimple or RTL), and run just one pass, and verify properties of the IR > emitted by the pass, or messages emitted by the pass. LLVM has a "-R" > remark option, so that a pass can issue remarks about what it's doing (a > new kind of diagnostic): > > http://llvm.org/releases/3.5.0/tools/clang/docs/UsersManual.html#opt-rpass > > We could implement this kind of testing by implementing gimple and RTL > frontends, and a -R option, which could integrate nicely with DejaGnu, > with > /* { dg-remark "inlined here" } */ > and the like. > > This would require finishing the gimple FE, and writing an RTL FE > (independent goals). Hopefully that would give us test coverage for > dealing with e.g. tricky reload situations. Insert note from Richi (was it?) where he wants the gimple FE made real and integrated. I think you'll have technical support if you want to tackle that next year. Something similar for RTL would be cool, but probably even harder given the amount of state that's traditionally been kept out of the IL stream. I'm sure some things are better today than in the past, but it's probably a very tanged mess to get to where we want to go with RTL. jeff