Re: [PATCH] Selftest framework (v7)

[Trevor Saunders <tbsaunde@tbsaunde.org>]

On Tue, Jun 07, 2016 at 10:18:32AM -0400, David Malcolm wrote:
> On Mon, 2016-06-06 at 22:14 -0400, Trevor Saunders wrote:
> > On Mon, Jun 06, 2016 at 11:57:49PM +0200, Jakub Jelinek wrote:
> > > On Mon, Jun 06, 2016 at 05:53:50PM -0400, Trevor Saunders wrote:
> > > > > > As far as I can 
> > > > > > tell this just involves moving the start of namespace
> > > > > > selftest
> > > > > > upwards a 
> > > > > > bit in the files where we have tests.
> > > > > 
> > > > > Yes, and it does seem cleaner to have all of the selftest code
> > > > > start
> > > > > like this:
> > > > > 
> > > > >   #if CHECKING_P
> > > > 
> > > > What are we gaining by ifdefing this? I would think on reasonable
> > > > systems the compiler would optimize out the call to the selftests
> > > > in
> > > > release builds and then the linker would gc all the unused
> > > > functions.
> > > > Do we really care about code size in places that doesn't happen
> > > > enough
> > > > to go through this?
> > > 
> > > Not everyone is building the compiler with LTO, and if you don't,
> > > then
> > > how would you optimize that away?
> > 
> > -ffunction-sections -Wl,--gc-sections should be enough I think.  I
> > guess
> > we don't use those at the moment though.
> > 
> > > And yes, not having the self-tests, especially if they are going to
> > > grow
> > > further, in release compilers is desirable, especially if it would
> > > be
> > > intermixed with hot code.
> > 
> > That's fair, though turning on --gc-sections where we can should
> > further
> > help with that, and that should be more effective with
> > -ffunction-sections -fdata-sections, so its seems to me like the
> > right
> > thing to do is add configure tests to enable those?  And then its
> > more
> > of a non issue?
> 
> I appreciate that you'd done a lot of work on eliminating preprocessor
> use in gcc, and that we'd prefer to minimize the amount of #if code we
> have - though it's relatively easy to test the with/without #if
> CHECKING_P case (compared to all of the various target-specific
> macros).

yeah, I certainly agree CHECKING_P is one of the more defensable macros.

> Historically gcc testing has largely been "black box" testing: run the
> built programs with specific inputs and look for specific outputs. My
> hope with -fself-tests is that we can build up our "white box" test
> coverage to complement the above, with unit tests.  My favorite example
> is the testing for gengtype that I posted as a followup, which gives us
> some immediate test coverage that gengtype is working as expected,
> which is hard to do using our traditional approach to testing.

I'd probably say testing hash tables / vec is more useful, but whatever
;)

> I hope that we can add a lot more tests to -fself-test, in particular,
> I want us to have unit tests for hot code, including code that's hidden
> deep inside the implementation and that might normally get inlined
> away.  To play Devil's Advocate: if we find we're able to do a release
> build with the selftests enabled and that it isn't slowing down the
> release build, does that imply we need more unit tests for our hottest
> code?  (the counterargument being that the checked build still needs to
> run in a bearable amount of time).

by enabled do you mean we build the code or actually run the tests on
every compile?  I would think basically any tests are two slow for the
latter.  For the former I would say building the self test code should
have very little effect on performance since its all separate functions.
There's some of course because of call sites effecting inlining etc, but
we can probably minimize that with -ffunction-sections --gc-sections and
attribute cold?

> That said, I want -fself-test to always run quickly (e.g. less than a
> second); let's not put anything slow in there.  Also, any unit tests
> involving analyzing several gimple or RTL statements at once seem to be
> easier to do via the gimple and RTL frontends that Prasad and I are
> working on respectively.  (I think we can use -fself-test for unit
> -testing implementation details of passes that involve one statement at
> a time, but as soon as we start dealing with multiple statements and
> control flow, that it's probably better to express it using a
> gimple/RTL dump in DejaGnu form).

That makes sense.

Trev

> 
> 
> Hope the above sounds sane
> Dave