From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 97881 invoked by alias); 10 Jun 2015 17:23:46 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Received: (qmail 97871 invoked by uid 89); 10 Jun 2015 17:23:46 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.4 required=5.0 tests=AWL,BAYES_00,KAM_LAZY_DOMAIN_SECURITY,SPF_HELO_PASS,T_RP_MATCHES_RCVD autolearn=no version=3.3.2 X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Wed, 10 Jun 2015 17:23:45 +0000 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id E87913173F1 for ; Wed, 10 Jun 2015 17:23:43 +0000 (UTC) Received: from [10.3.230.67] (vpn-230-67.phx2.redhat.com [10.3.230.67]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t5AHNhxm016402; Wed, 10 Jun 2015 13:23:43 -0400 Message-ID: <1433956580.12727.194.camel@surprise> Subject: Re: [PATCH 00/17] RFC: Addding a unit testing framework to gcc From: David Malcolm To: Jakub Jelinek Cc: gcc-patches@gcc.gnu.org Date: Wed, 10 Jun 2015 17:56:00 -0000 In-Reply-To: <20150610153447.GM10247@tucnak.redhat.com> References: <1433949898-22033-1-git-send-email-dmalcolm@redhat.com> <20150610153447.GM10247@tucnak.redhat.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2015-06/txt/msg00789.txt.bz2 On Wed, 2015-06-10 at 17:34 +0200, Jakub Jelinek wrote: > On Wed, Jun 10, 2015 at 11:24:41AM -0400, David Malcolm wrote: > > I picked the Google Test framework: > > http://code.google.com/p/googletest/ > > I must say I'm not very excited about using this, it won't integrate > very well with dejagnu Why is that a goal? I've been using DejaGnu's unittesting API for testing the jit, and it is... suboptimal, to put it mildly. > whether talking about results (will it provide > some *.log/*.sum file with FAIL/XFAIL/PASS/XPASS etc. lines?), It doesn't have an output formatter for the DejaGnu format, but I guess I could write one. The gtest standard output format is IMHO superior to DejaGnu's since it tells you start-of-test/end-of-test on separate lines, so you can see which test killed things in the event of total failure, and the per-test timings (which can be disabled if you want to do diffs). > choosing > what options to use, e.g. global > RUNTESTFLAGS='--target_board=unix/\{-m32,-m64\}' > to test everything twice for 32-bit and 64-bit, will that run just > all unittests twice the same?, Fair enough; yes, as written, RUNTESTFLAGS is ignored; it will run everything once. However we could express such a loop directly in the "main" of the > or possibility to run a subset of tests $ LD_LIBRARY_PATH=. ./unittests.exe --gtest_filter="*push" Note: Google Test filter = *push [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from vec_test [ RUN ] vec_test.quick_push [ OK ] vec_test.quick_push (0 ms) [ RUN ] vec_test.safe_push [ OK ] vec_test.safe_push (0 ms) [----------] 2 tests from vec_test (1 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (4 ms total) [ PASSED ] 2 tests. > etc. > E.g. for asan.exp testing, I just wrote a gtest emulation using > dejagnu, see testsuite/g++.dg/asan/dejagnu-gtest.h and > testsuite/lib/asan-dg.exp ...which doesn't have things like EXPECT_STREQ, or custom comparators, doesn't appear to support fixtures, type-parameterized tests, value-parameterized tests, etc, my point being that a unit-testing framework is a non-trivial amount of code that could do useful things for us. > but that was mainly meant for cases where > many routines are expected to crash the process. FWIW, if I'm reading testsuite/g++.dg/asan/dejagnu-gtest.h and testsuite/lib/asan-dg.exp correctly, it looks like you're spawning the tool once per EXPECT_DEATH instance in the testsuite. That seems suboptimal, gtest uses fork without exec to do it all directly at the point of the test without lots of extra work, where the parent verifies the death of the child. > If in unittests > you are doing only operations that aren't meant to take everything down or > if they crash, it is ok if it breaks the whole unit testing, > then perhaps it can be run as a single process and thus a single dejagnu > job, and just let the wrapper parse the output and transform it. I guess, but having used google test lately, DejaGnu feels like a big step backwards. > Also, no matter what testsuite framework is used, including any > headers before #include "config.h" line is a big no-no. OK. Can't remember why I did that. Dave