Re: [PATCH 00/17] RFC: Addding a unit testing framework to gcc

["Martin Liška" <mliska@suse.cz>]

On 06/10/2015 05:24 PM, David Malcolm wrote:
> Our current test suite is rather coarse-grained, acting at the level
> of running gcc subprocesses, and verifying externally-visible
> properties:
>    - Did the compile succeed?
>    - Was a particular dumpfile emitted, containing something matching
>      some regex?
>    - Does the generated code run and exit gracefully?
> etc
>
> The strength of this approach is that we have good "integration"
> coverage: we know that the compiler runs and generates good code.
> However it's slow, and doesn't allow very precise control over what
> properties we can assert.
>
> The following patch kit adds a suite of unit tests to gcc, aimed at
> giving test coverage for properties that are difficult to test for
> in the current system.  For example, there are tests of ggc to
> verify that gengtype is doing sane things, of various container
> classes (vec, hash_map, hash_set) and of wide-int.  Some of the
> tests are rather "placeholdery" e.g. the tests of folding trees,
> where there's plenty of room for adding new testcases.
>
> I've split them up into multiple patches for ease of review, but
> they all stand together.
>
> I picked the Google Test framework:
>    http://code.google.com/p/googletest/
>
> I didn't do a very thorough survey of C++ test frameworks; I picked
> this one as it's used by several very large projects (including
> LLVM [1]), and is actively maintained.  Working with it has largely been
> a pleasant experience: it generates good error messages when tests
> fail (e.g. with enough information so that I can click on failures
> in Emacs and have it go to the failing test), and is sane to work
> with in gdb.  The log is easy to read; I've added an example to
> the end of this mail.  It supports parameterizing testcases across
> multiple types (I use this for testing wide-int).
>
> The only other "framework" I've used has been the DejaGnu unittest
> header, which I use for the jit testsuite; I can't recommend it (it's
> a pain to use, and I've had to extend it repeatedly to get basic
> functionality like string equality assertions).
>
> The testsuite is intended to be very fast, all in one process, and it
> takes less than a second to run; it fact, the time is dominated by a
> single very slow test, which takes 300-400ms to run, but which could
> be sped up [2]; other than that, it takes about 10ms to run.
>
> Structurally, the patches add a "unittests" frontend: this is rather
> analogous to libgccjit.so: it's a dummy frontend.  The
> unittests/Make-lang.in builds a libgccunittests.so, which does
> nothing, stubbing out the frontend hooks, but provides a DSO holding
> the code to be tested.
>
> libgccunittests.so is then linked into a "unittests.exe" binary, which
> holds the actual testcases.  An advantage of this separation is that
> although linking libgccunittests.so is rather slow (like linking "cc1"
> etc), unittests.exe is very fast to link, so that it's very fast to
> hack on individual tests without needing to relink "everything" for
> each edit.
>
> Of course, this means that we can't unittest the real frontends this
> way (but we couldn't before, and maybe we can find a route to doing
> this).
>
> I have the Make-lang.in implementing the "unittests" target so that
> it builds and *runs* the unit tests i.e. if you do a "make", the
> unittests are run (rather than just on a "make check").  Given how
> fast they are (especially relative to "make check", the only issue
> I can see with this is the output log spew).  One nice thing about
> doing it there is that it can be run at each stage of a bootstrap,
> so hopefully we fail earlier when we're going to fail.
>
> I marked it in config-lang.in as
>    build_by_default="no"
> so you have to opt-in to building it by adding "unittests" to the
>    --enable-languages=
> configure options.
>
> The split of the bulk of the link into a libgccjitunittests.so
> means that it also requires the
>    --enable-host-shared
> configure-time option.
>
> It doesn't yet bootstrap; the link fails with:
> test-folding.o: In function `testing::AssertionResult testing::internal::CmpHelperNE<tree_node*, tree_node*>(char const*, char const*, tree_node* const&, tree_node* const&)':
> test-folding.c:(.text._ZN7testing8internal11CmpHelperNEIP9tree_nodeS3_EENS_15AssertionResultEPKcS6_RKT_RKT0_[_ZN7testing8internal11CmpHelperNEIP9tree_nodeS3_EENS_15AssertionResultEPKcS6_RKT_RKT0_]+0x26e): undefined reference to `testing::internal::StringStreamToString(std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)'
>
> though this was using a system copy of gtest
> (is this due to the C++ stdlib ABI change?  it seems to only affect
> tests using EXPECT_STREQ, and if I hack them out, it seems to work).
> (perhaps we'd need to bundle our own copy of gtest?)
>
> Here's a sample log (and it's sanely color-coded if you run it at
> a tty):
>
> $ make unittests
> [==========] Running 56 tests from 16 test cases.
> [----------] Global test environment set-up.
> [----------] 6 tests from ggc_test
> [ RUN      ] ggc_test.tree_marking
> [       OK ] ggc_test.tree_marking (0 ms)
> [ RUN      ] ggc_test.custom_struct
> [       OK ] ggc_test.custom_struct (0 ms)
> [ RUN      ] ggc_test.finalization
> [       OK ] ggc_test.finalization (0 ms)
> [ RUN      ] ggc_test.deletable_global
> [       OK ] ggc_test.deletable_global (1 ms)
> [ RUN      ] ggc_test.inheritance
> [       OK ] ggc_test.inheritance (0 ms)
> [ RUN      ] ggc_test.chain_next
> [       OK ] ggc_test.chain_next (326 ms)
> [----------] 6 tests from ggc_test (327 ms total)
>
> [----------] 5 tests from bitmap_test
> [ RUN      ] bitmap_test.gc_alloc
> [       OK ] bitmap_test.gc_alloc (0 ms)
> [ RUN      ] bitmap_test.set_range
> [       OK ] bitmap_test.set_range (0 ms)
> [ RUN      ] bitmap_test.clear_bit_in_middle
> [       OK ] bitmap_test.clear_bit_in_middle (0 ms)
> [ RUN      ] bitmap_test.copying
> [       OK ] bitmap_test.copying (0 ms)
> [ RUN      ] bitmap_test.bitmap_single_bit_set_p
> [       OK ] bitmap_test.bitmap_single_bit_set_p (0 ms)
> [----------] 5 tests from bitmap_test (0 ms total)
>
> [----------] 3 tests from cfg_test
> [ RUN      ] cfg_test.linear_chain
> [       OK ] cfg_test.linear_chain (1 ms)
> [ RUN      ] cfg_test.diamond
> [       OK ] cfg_test.diamond (0 ms)
> [ RUN      ] cfg_test.fully_connected
> [       OK ] cfg_test.fully_connected (0 ms)
> [----------] 3 tests from cfg_test (1 ms total)
>
> [----------] 1 test from tree_folding_test
> [ RUN      ] tree_folding_test.arithmetic_folding
> [       OK ] tree_folding_test.arithmetic_folding (0 ms)
> [----------] 1 test from tree_folding_test (0 ms total)
>
> [----------] 2 tests from function_test
> [ RUN      ] function_test.fndecl_int_void
> [       OK ] function_test.fndecl_int_void (0 ms)
> [ RUN      ] function_test.fndecl_float_intchar
> [       OK ] function_test.fndecl_float_intchar (0 ms)
> [----------] 2 tests from function_test (0 ms total)
>
> [----------] 4 tests from representation_test
> [ RUN      ] representation_test.gimplification
> [       OK ] representation_test.gimplification (0 ms)
> [ RUN      ] representation_test.building_cfg
> [       OK ] representation_test.building_cfg (0 ms)
> [ RUN      ] representation_test.conversion_to_ssa
> [       OK ] representation_test.conversion_to_ssa (0 ms)
> [ RUN      ] representation_test.expansion_to_rtl
> [       OK ] representation_test.expansion_to_rtl (6 ms)
> [----------] 4 tests from representation_test (6 ms total)
>
> [----------] 5 tests from gimple_test
> [ RUN      ] gimple_test.assign_single
> [       OK ] gimple_test.assign_single (0 ms)
> [ RUN      ] gimple_test.assign_binop
> [       OK ] gimple_test.assign_binop (0 ms)
> [ RUN      ] gimple_test.nop_stmt
> [       OK ] gimple_test.nop_stmt (0 ms)
> [ RUN      ] gimple_test.return_stmt
> [       OK ] gimple_test.return_stmt (0 ms)
> [ RUN      ] gimple_test.return_without_value
> [       OK ] gimple_test.return_without_value (0 ms)
> [----------] 5 tests from gimple_test (0 ms total)
>
> [----------] 1 test from hash_map_test
> [ RUN      ] hash_map_test.map_of_strings_to_int
> [       OK ] hash_map_test.map_of_strings_to_int (0 ms)
> [----------] 1 test from hash_map_test (0 ms total)
>
> [----------] 1 test from hash_set_test
> [ RUN      ] hash_set_test.set_of_strings
> [       OK ] hash_set_test.set_of_strings (0 ms)
> [----------] 1 test from hash_set_test (0 ms total)
>
> [----------] 4 tests from location_test
> [ RUN      ] location_test.accessing_ordinary_linemaps
> [       OK ] location_test.accessing_ordinary_linemaps (0 ms)
> [ RUN      ] location_test.unknown_location
> [       OK ] location_test.unknown_location (0 ms)
> [ RUN      ] location_test.builtins
> [       OK ] location_test.builtins (0 ms)
> [ RUN      ] location_test.reading_source_line
> [       OK ] location_test.reading_source_line (0 ms)
> [----------] 4 tests from location_test (0 ms total)
>
> [----------] 2 tests from rtl_test
> [ RUN      ] rtl_test.test_single_set
> [       OK ] rtl_test.test_single_set (0 ms)
> [ RUN      ] rtl_test.uncond_jump
> [       OK ] rtl_test.uncond_jump (0 ms)
> [----------] 2 tests from rtl_test (0 ms total)
>
> [----------] 3 tests from tree_test
> [ RUN      ] tree_test.integer_constants
> [       OK ] tree_test.integer_constants (0 ms)
> [ RUN      ] tree_test.identifiers
> [       OK ] tree_test.identifiers (0 ms)
> [ RUN      ] tree_test.labels
> [       OK ] tree_test.labels (0 ms)
> [----------] 3 tests from tree_test (0 ms total)
>
> [----------] 10 tests from vec_test
> [ RUN      ] vec_test.quick_push
> [       OK ] vec_test.quick_push (0 ms)
> [ RUN      ] vec_test.safe_push
> [       OK ] vec_test.safe_push (0 ms)
> [ RUN      ] vec_test.truncate
> [       OK ] vec_test.truncate (0 ms)
> [ RUN      ] vec_test.safe_grow_cleared
> [       OK ] vec_test.safe_grow_cleared (0 ms)
> [ RUN      ] vec_test.pop
> [       OK ] vec_test.pop (0 ms)
> [ RUN      ] vec_test.safe_insert
> [       OK ] vec_test.safe_insert (0 ms)
> [ RUN      ] vec_test.ordered_remove
> [       OK ] vec_test.ordered_remove (0 ms)
> [ RUN      ] vec_test.unordered_remove
> [       OK ] vec_test.unordered_remove (0 ms)
> [ RUN      ] vec_test.block_remove
> [       OK ] vec_test.block_remove (0 ms)
> [ RUN      ] vec_test.qsort
> [       OK ] vec_test.qsort (0 ms)
> [----------] 10 tests from vec_test (1 ms total)
>
> [----------] 3 tests from wide_int_test/0, where TypeParam = <type>
> [ RUN      ] wide_int_test/0.test_printing
> [       OK ] wide_int_test/0.test_printing (0 ms)
> [ RUN      ] wide_int_test/0.test_ops
> [       OK ] wide_int_test/0.test_ops (0 ms)
> [ RUN      ] wide_int_test/0.test_comparisons
> [       OK ] wide_int_test/0.test_comparisons (0 ms)
> [----------] 3 tests from wide_int_test/0 (0 ms total)
>
> [----------] 3 tests from wide_int_test/1, where TypeParam = <type>
> [ RUN      ] wide_int_test/1.test_printing
> [       OK ] wide_int_test/1.test_printing (0 ms)
> [ RUN      ] wide_int_test/1.test_ops
> [       OK ] wide_int_test/1.test_ops (0 ms)
> [ RUN      ] wide_int_test/1.test_comparisons
> [       OK ] wide_int_test/1.test_comparisons (0 ms)
> [----------] 3 tests from wide_int_test/1 (0 ms total)
>
> [----------] 3 tests from wide_int_test/2, where TypeParam = <type>
> [ RUN      ] wide_int_test/2.test_printing
> [       OK ] wide_int_test/2.test_printing (0 ms)
> [ RUN      ] wide_int_test/2.test_ops
> [       OK ] wide_int_test/2.test_ops (0 ms)
> [ RUN      ] wide_int_test/2.test_comparisons
> [       OK ] wide_int_test/2.test_comparisons (0 ms)
> [----------] 3 tests from wide_int_test/2 (0 ms total)
>
> [----------] Global test environment tear-down
> [==========] 56 tests from 16 test cases ran. (338 ms total)
> [  PASSED  ] 56 tests.
>
> Thoughts?
> Dave

Hi David.

I really like the idea to introduce a unit testing. Few months ago, when I was
working on adding support for negative number of sreal class, I needed a place
to write unit tests. In the end I wrote a plugin for GCC:

gcc/testsuite/gcc.dg/plugin/sreal_plugin.c

After you will merge the patch set, I will rewrite it to be part of unit tests.

Thanks,
Martin


>
> [1] though it's not clear to me if LLVM is actually still using it;
> see e.g. http://blog.llvm.org/2009/12/lit-it.html
> [2] the test of chain_next/chain_prev in test-ggc.c, which needed
> a very large linked list in order to reliably overflow the stack on
> my box; perhaps this could be eliminated by adding something to
> libiberty to shrink the stack size?
>
> David Malcolm (17):
>    Add Make-lang.in and config-lang.in to gcc/unittests
>    Add test-bitmap.c to gcc/unittests
>    Add test-cfg.c to gcc/unittests
>    Add test-folding.c to gcc/unittests
>    Add test-functions.c to gcc/unittests
>    Add test-ggc.c to gcc/unittests
>    Add test-gimple.c to gcc/unittests
>    Add test-hash-map.c to gcc/unittests
>    Add test-hash-set.c to gcc/unittests
>    Add test-locations.c to gcc/unittests
>    Add test-rtl.c to gcc/unittests
>    Add test-tree.c to gcc/unittests
>    Add test-vec.c to gcc/unittests
>    Add test-wide-int.c to gcc/unittests
>    Add unittests-frontend.c to gcc/unittests
>    Add unittests-main.c to gcc/unittests
>    toplev.c: move location_adhoc_data_fini call
>
>   gcc/toplev.c                       |   3 +-
>   gcc/unittests/Make-lang.in         | 200 ++++++++++++
>   gcc/unittests/config-lang.in       |  34 ++
>   gcc/unittests/test-bitmap.c        | 117 +++++++
>   gcc/unittests/test-cfg.c           | 319 +++++++++++++++++++
>   gcc/unittests/test-folding.c       | 121 +++++++
>   gcc/unittests/test-functions.c     | 634 +++++++++++++++++++++++++++++++++++++
>   gcc/unittests/test-ggc.c           | 292 +++++++++++++++++
>   gcc/unittests/test-gimple.c        | 179 +++++++++++
>   gcc/unittests/test-hash-map.c      |  78 +++++
>   gcc/unittests/test-hash-set.c      |  54 ++++
>   gcc/unittests/test-locations.c     | 148 +++++++++
>   gcc/unittests/test-rtl.c           |  94 ++++++
>   gcc/unittests/test-tree.c          | 102 ++++++
>   gcc/unittests/test-vec.c           | 162 ++++++++++
>   gcc/unittests/test-wide-int.c      | 186 +++++++++++
>   gcc/unittests/unittests-frontend.c | 250 +++++++++++++++
>   gcc/unittests/unittests-main.c     | 108 +++++++
>   18 files changed, 3080 insertions(+), 1 deletion(-)
>   create mode 100644 gcc/unittests/Make-lang.in
>   create mode 100644 gcc/unittests/config-lang.in
>   create mode 100644 gcc/unittests/test-bitmap.c
>   create mode 100644 gcc/unittests/test-cfg.c
>   create mode 100644 gcc/unittests/test-folding.c
>   create mode 100644 gcc/unittests/test-functions.c
>   create mode 100644 gcc/unittests/test-ggc.c
>   create mode 100644 gcc/unittests/test-gimple.c
>   create mode 100644 gcc/unittests/test-hash-map.c
>   create mode 100644 gcc/unittests/test-hash-set.c
>   create mode 100644 gcc/unittests/test-locations.c
>   create mode 100644 gcc/unittests/test-rtl.c
>   create mode 100644 gcc/unittests/test-tree.c
>   create mode 100644 gcc/unittests/test-vec.c
>   create mode 100644 gcc/unittests/test-wide-int.c
>   create mode 100644 gcc/unittests/unittests-frontend.c
>   create mode 100644 gcc/unittests/unittests-main.c
>