Re: [PATCH 00/16] Unit tests framework (v3)

[Jeff Law <law@redhat.com>]

On 10/29/2015 10:10 AM, David Malcolm wrote:
>
> There may be useful things in gtest for us that I'm not using yet.  For
> example, the support for death tests may be useful for testing that e.g.
> our checking macros kill the program sanely in the presence of malformed
> internal data.
Please check on this.  I want to build a set of tests around the 
SSA_NAME manager, and that would include setting up can't happen 
situations and seeing if they're properly diagnosed by the checking 
code.  So obviously I have an interest in how the framework handles this 
kind of scenario.


>
> I wanted to reuse the gtest code on the grounds that:
>
>    * the code exists
>
>    * it's maintained (https://github.com/google/googletest
>      shows the last commit was 10 days ago)
>
>    * it has documentation:
> https://github.com/google/googletest/blob/master/googletest/docs/Primer.md
> (there's much more that just that)
>
>    * it will be familiar to some people from other projects
>
>
> That said, if it's a blocker, I can attempt a minimal reimplementation
> of the subset of the API that I'm using.
I'd probably lean towards continuing to use gtest.  I don't currently 
see a compelling reason to build another framework.

>
>>> v1 of the kit was structured as a frontend, v2 of the kit as
>>> a plugin that was built as if it were a frontend.  Both of
>>> these approaches were problematic, so this version
>>> of the patch kit simply builds as a test case within the
>>> plugin.exp suites.
>>
>> This is the part I'm least certain about, for two reasons:
>>    * They are more like source files than tests, and I think I'd prefer
>>      to have them alongside the source, in gcc/ rather than in the
>>      testsuite. This way people are invariable going to fail to notice
>>      them when they grep for something.
>
> So something like:
>
>    gcc/test-vec.c
>
> or even:
>
>    gcc/vec-tests.c
>
> so that it sorts next to gcc/vec.h in a listing?
>
> i.e.   gcc/FOO-tests.c  for gcc/FOO.c
>
> Maybe the unittests-plugin.c should live within the "gcc" directory?
It's a bit of bike-shedding :-)  I'd be happy with gcc/test-whatever or 
gcc/whatever-test.  Both have pluses/minuses and in the end I doubt it'd 
matter much.

My only worry here is doubling the number of files in gcc/.  But I'd 
claim that we want to have more subdirectories for major logical areas. 
  gimple/ rtl/ whatever, which would significantly mitigate the problem 
of too many files in gcc/


>
>>    * This uses a plugin into whatever compiler was built, but sometimes
>>      you can't really set up unit tests that way because what you want
>>      to test depends on target specifics. What I've often wanted is a
>>      special test target that gets built with a special machine
>>      description that has whatever patterns are needed to replicate
>>      tricky situations in reload or other optimization passes.
>>
>> The tests you have so far are focused mostly on high-level gimple/tree
>> tests where this limitation is probably not showing up very much, but I
>> think it would be better to have something that allows us to have more
>> in-depth tests.
>
> I think our long-term goal should be at least 5 styles of test:
>
> (A) end-to-end tests of the compiler: running it on some source and
> verifying properties of the diagnostics and/or generated code
> (potentially running it).  This is what we have today.
>
> (B) unit tests of individual subsystems: tests that exercise specific
> internal APIs e.g. containers like vec<>, hash_map<> etc, also things
> like gengtype.  This is a gap in our current test coverage, and what
> this patch kit aims to start filling.
Right.  It's essentially data structure and other very low level testing 
(gengtype).

That's even true if I think about the SSA_NAME manager -- in an ideal 
world, it'd just be a container for objects that we want to release & 
recycle, whether they're SSA_NAMEs, pseudos, whatever we decide has the 
right properties to benefit from recycling.  The fact that it's 
recycling SSA_NAMEs isn't for the most part all that important.



>
> (C) and (D): tests of specific passes: tests that accept IR (either
> gimple or RTL), and run just one pass, and verify properties of the IR
> emitted by the pass, or messages emitted by the pass.  LLVM has a "-R"
> remark option, so that a pass can issue remarks about what it's doing (a
> new kind of diagnostic):
>
> http://llvm.org/releases/3.5.0/tools/clang/docs/UsersManual.html#opt-rpass
>
> We could implement this kind of testing by implementing gimple and RTL
> frontends, and a -R option, which could integrate nicely with DejaGnu,
> with
>    /* { dg-remark "inlined here" } */
> and the like.
>
> This would require finishing the gimple FE, and writing an RTL FE
> (independent goals).  Hopefully that would give us test coverage for
> dealing with e.g. tricky reload situations.
Insert note from Richi (was it?) where he wants the gimple FE made real 
and integrated.  I think you'll have technical support if you want to 
tackle that next year.

Something similar for RTL would be cool, but probably even harder given 
the amount of state that's traditionally been kept out of the IL stream. 
  I'm sure some things are better today than in the past, but it's 
probably a very tanged mess to get to where we want to go with RTL.

jeff