Re: [PATCH 00/17] RFC: Addding a unit testing framework to gcc

[Jakub Jelinek <jakub@redhat.com>]

On Wed, Jun 10, 2015 at 01:16:20PM -0400, David Malcolm wrote:
> On Wed, 2015-06-10 at 17:34 +0200, Jakub Jelinek wrote:
> > On Wed, Jun 10, 2015 at 11:24:41AM -0400, David Malcolm wrote:
> > > I picked the Google Test framework:
> > >   http://code.google.com/p/googletest/
> > 
> > I must say I'm not very excited about using this, it won't integrate
> > very well with dejagnu
> 
> Why is that a goal?  I've been using DejaGnu's unittesting API for
> testing the jit, and it is... suboptimal, to put it mildly.

Primarily consistency, people want consistent output of the testresults from
the compiler, not to have to pass one set of magic options to dejagnu to do
something and then mirror them to something completely different to make
gtest happy.  Similarly, there are all kinds of scripts that analyze gcc
testresults, having to parse a completely different format because a tiny
percentage of tests runs something different isn't a very good idea.
Plus, by using googletest, you add another build requirement, we already
have quite a lot of them.

> > whether talking about results (will it provide
> > some *.log/*.sum file with FAIL/XFAIL/PASS/XPASS etc. lines?),
> 
> It doesn't have an output formatter for the DejaGnu format, but I guess
> I could write one.  The gtest standard output format is IMHO superior to
> DejaGnu's since it tells you start-of-test/end-of-test on separate
> lines, so you can see which test killed things in the event of total

I find the dejagnu log files quite readable, unlike the gtest stuff, so
supposedly this is quite subjective.

> > etc.
> > E.g. for asan.exp testing, I just wrote a gtest emulation using
> > dejagnu, see testsuite/g++.dg/asan/dejagnu-gtest.h and
> > testsuite/lib/asan-dg.exp
> 
> ...which doesn't have things like EXPECT_STREQ, or custom comparators,
> doesn't appear to support fixtures, type-parameterized tests,
> value-parameterized tests, etc, my point being that a unit-testing
> framework is a non-trivial amount of code that could do useful things
> for us.

The question is why you really need it, or if it is more than a few lines of
macros.

> > but that was mainly meant for cases where
> > many routines are expected to crash the process.
> 
> FWIW, if I'm reading testsuite/g++.dg/asan/dejagnu-gtest.h and
> testsuite/lib/asan-dg.exp correctly, it looks like you're spawning the
> tool once per EXPECT_DEATH instance in the testsuite.  That seems
> suboptimal, gtest uses fork without exec
> to do it all directly at the point of the test without lots of extra
> work, where the parent verifies the death of the child.

That was a design choice, makes it much easier to debug the individual
tests, especially for asan where for gtest there are just way too many forks
and way too many (expected) crashes.
For the unittests, I see no problem running numerous tests from within one
process, it can emit multiple PASS/FAIL/XFAIL etc.

	Jakub