From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 10573 invoked by alias); 24 Oct 2019 20:50:17 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 10563 invoked by uid 89); 24 Oct 2019 20:50:16 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-1.8 required=5.0 tests=BAYES_00,FORGED_SPF_HELO,GIT_PATCH_2,KAM_ASCII_DIVIDERS,KAM_SHORT,RCVD_IN_DNSWL_NONE,RCVD_IN_SBL_CSS,SPF_HELO_PASS autolearn=no version=3.3.1 spammy=revised, abnormal, HTo:D*comcast.net, termination X-HELO: NAM06-BL2-obe.outbound.protection.outlook.com Received: from mail-eopbgr650097.outbound.protection.outlook.com (HELO NAM06-BL2-obe.outbound.protection.outlook.com) (40.107.65.97) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 24 Oct 2019 20:50:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XwxaTh/YtFEhQXJI3MNT51A99MPaSeJdvGvcnN6JMxRjwmm3BxzKX069H8L31UBHwLDM8/gOvauCgfsqXw6gwmLNubEvpSF4FR/k0Yg0WjDKuVqFFqxeD1f3cC1oop5NtkeL1Ti5cmYqnY9H1ETPhEwf8qi5lWHmUo8Fh9oXz/XfYVxxB+XjJAmnawkl3zb9wYLw679ra/zvyTi0N9JJNyMU0pGbwjBvDh9u68AYm6hb7t513iWlNc6cc2d1p+CATsZttTCWKeasDDYmnKTl5AboW9tvNybnd2J22zYAnyTTPY67lVjGYrDJWPdVp3MDcSh+kLttmP6X0HmY2NTnXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cLHC72KSg6IGnSx7SDI4azaNlC0ai63UmPTIjYSFz+g=; b=ZFtzw4AWGfa0QGnO9R9VBnLO9GTpXMVrmitUBoq6umU8hFlyNxbGUE20Y6wzg3Qv+hG9jjNDLFEq0Poym1vxDQv4amyMZy2IkHVENlxiiyhE0F7JGkAMr/g62nINz0/HMLSdmc6jzKsBcOBU3ybVTXMw0AowohBqvxR+byaHaJA0E5XVSIicHJ9aSk9oBzFVqajS7Zd/tU0hujCiGzcpUrv/F3m8iO4iq2iw5rnsVQlCSKeugJEZYYR4Ciekk8HuTPRe4RQnnJEtYMNVqfIgXzmsB4y0Y9VX52dNBwA4xWOIw6Q6yEhUiLdYgFpTI6sYVnTbgZwMKtfBRLMsgYxZIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cLHC72KSg6IGnSx7SDI4azaNlC0ai63UmPTIjYSFz+g=; b=eV6SQwLdF3zp8//fwmfoH7VuVrU8NHaHDc94klXayTVzy8JuP6SY6g55D6ZUFud9dZzNqmyDh1KbtKFNvGSUYmclT6LlV/A3PCagJOOMPvnGoVkM0PPo2yclRljUA1YDDlMqILeI5DKgWdDDKjn0VldK1B/GfixSoQHP+/N2SPg= Received: from CO2PR00MB0119.namprd00.prod.outlook.com (10.166.215.150) by CO2PR00MB0118.namprd00.prod.outlook.com (10.166.215.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2399.0; Thu, 24 Oct 2019 20:50:09 +0000 Received: from CO2PR00MB0119.namprd00.prod.outlook.com ([fe80::8de8:e0c1:679:fe37]) by CO2PR00MB0119.namprd00.prod.outlook.com ([fe80::8de8:e0c1:679:fe37%10]) with mapi id 15.20.2404.000; Thu, 24 Oct 2019 20:50:09 +0000 From: "Andrew Dean via gcc" Reply-To: Andrew Dean To: "gcc@gcc.gnu.org" , "ro@CeBiTec.Uni-Bielefeld.DE" , "mikestump@comcast.net" , "law@redhat.com" , "jason@redhat.com" CC: Gabriel Dos Reis Subject: GCC selftest improvements Date: Thu, 24 Oct 2019 20:50:00 -0000 Message-ID: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Andrew.Dean@microsoft.com; x-ms-exchange-transport-forked: True x-ms-oob-tlc-oobclassifiers: OLM:10000; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: tlLdQL7JLGD2R+Rx+Fr+mVFz3CFZnZCOSXbTwlUIzKm2Rlrsj12FwfNTPs4WxLLHV7etPK5BVUHmJ0kyBj1MTg== X-IsSubscribed: yes X-SW-Source: 2019-10/txt/msg00150.txt.bz2 TLDR: I'd like to propose adding a dependency on a modern unit testing fram= ework to make it easier to write unit tests within GCC. Before I spend much= more time on it, what sort of buy-in should I get? Are there any people in= particular I should work more closely with as I make this change? =20 Terminology: Within GCC, there are two types of tests in place: unit tests = and regression tests. The unit tests have been written with a home-grown se= lftest framework and run as part of the build process. Any failures to a un= it test results in no compiler being produced. The regression tests, on the= other hand, run after build, and use the separate DejaGnu framework. In th= is email, I am only concerning myself with the unit tests, and throughout t= he remainder of the email, any mention of tests refers to these. =20 Working on GCC, I wanted to add some new unit tests to my feature as I went= , but I noticed that there is a good deal of friction involved. Right now, = adding new unit tests requires writing the test method, then modifying a se= cond place in the code to call said test method, repeating as necessary unt= il getting all the way to either the selftest.c file or the target hook. Th= ere is also no way to do test setup/teardown automatically. Everything is m= anual. =20 I'd like to propose adding a dependency on a modern open-source unit testin= g framework as an enhancement to the current self test system. I have used = Catch2 (https://github.com/catchorg/Catch2, Boost Software License 1.0) wit= h great success in the past. I experimented with adding it to GCC and conve= rting a handful of tests to use Catch2. Although I only converted a small n= umber of tests, I didn't see any performance impact during selftest. As a b= onus, while doing so, I actually found that one test that I had written pre= viously wasn't actually being run, because I had failed to manually call it. =20 Some nice things that Catch2 provides are better error reporting (see below= for a comparison), ease of adding new tests (just include the header and w= rite a TEST_CASE(), as opposed to the manual plumbing required right now), = extension points for adding custom comparisons (I could see this being very= useful to expand on the current rtl test macros), and the ability to run a= subset of the tests without recompiling. It is also easy to integrate Catc= h2 with the existing self-test framework. =20 If this path seems useful to others, I'm happy to pursue it further. A list= of work items I see are: =20 1. Convert more tests to verify the claim that build performance is not deg= raded 2. Update the docs to list Catch2 as the new recommended way to write unit = tests 3. If all of the target self-tests are converted, then we can remove the ta= rget test hook. Similar for the lang test hook. =20 One thing that would make Catch2 an even more slam-dunk case was if we were= able to enable exceptions for the check builds. Then, running the unit tes= ts could report multiple failures at the same time instead of just aborting= at the first one. That said, even without enabling exceptions, Catch2 is o= n par with the current selftest in terms of terminating at the first failur= e. =20 Another option is to use a test framework that doesn't use exceptions, such= as Google Test (https://github.com/google/googletest, BSD 3-Clause "New" o= r "Revised" License). I personally think Catch2 is more flexible, or I woul= d lead with Google Test. For example, in Catch2, shared setup is done in pl= ace with the tests itself, having each subtest be a nested SECTION, where-a= s in GTest, you have to write a test class that derives from ::test and ove= rrides SetUp(). In addition, the sections in Catch2 can be nested further, = allowing several related tests to build on each other.=20 =20 Here is some sample output for the case where all the tests are passing: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D All tests passed (25 assertions in 5 test cases) =20 And here is the output when a test fails: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~= ~~~~ is a Catch v2.9.2 host application. Run with -? for options =20 ---------------------------------------------------------------------------= ---- test_set_range ---------------------------------------------------------------------------= ---- ../../gcc/bitmap.c:2661 ...........................................................................= .... ../../gcc/bitmap.c:2668: FAILED: REQUIRE( 6 =3D=3D bitmap_count_bits (b) ) with expansion: 6 =3D=3D 5 =20 Catch will terminate because it needed to throw an exception. The message was: Test failure requires aborting test! terminate called without an active exception ../../gcc/bitmap.c:2668: FAILED: {Unknown expression after the reported line} due to a fatal error condition: SIGABRT - Abort (abnormal termination) signal =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D test cases: 2 | 1 passed | 1 failed assertions: 5 | 3 passed | 2 failed cc1: internal compiler error: Aborted Please submit a full bug report, with preprocessed source if appropriate. Please include the complete backtrace with any bug report. See for instructions. =20 (Note that at the moment it doesn't know the name of our application or it = would have prefixed "is a Catch..." with our app name). =20 Compare that to the output of the current test framework: ../../gcc/bitmap.c:2669: test_set_range: FAIL: ASSERT_EQ ((6), (bitmap_coun= t_bits (b))) cc1: internal compiler error: in fail, at selftest.c:47 /bin/bash ../../gcc/../move-if-change tmp-macro_list macro_list echo timestamp > s-macro_list Please submit a full bug report, with preprocessed source if appropriate. Please include the complete backtrace with any bug report. See for instructions. =20 Thanks, =20 Andrew