From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 73873 invoked by alias); 15 Nov 2019 11:48:38 -0000 Mailing-List: contact libffi-discuss-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libffi-discuss-owner@sourceware.org Received: (qmail 73860 invoked by uid 89); 15 Nov 2019 11:48:37 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00,HTML_MESSAGE,KAM_SHORT,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 spammy=environmental, wholesale, H*F:U*green, Light X-HELO: mail-lf1-f50.google.com Received: from mail-lf1-f50.google.com (HELO mail-lf1-f50.google.com) (209.85.167.50) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 15 Nov 2019 11:48:36 +0000 Received: by mail-lf1-f50.google.com with SMTP id o65so7770305lff.13 for ; Fri, 15 Nov 2019 03:48:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=moxielogic-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=Er0PKDame5Qn3BjUIRIxfc24hW/PBfkAJ7jDs7CfF7s=; b=XDd8SfS64j9NUUUeHyg9Mkn0UA/N1wzSOIk7ykglZkgALD9NvpWqpQn9fvK7PhXlWQ uqgnNSCxth0QPEoUCOSWIL6Oy35VB4h9ZUxFxkivIYy97a1IGWLTDKqRiBP4rIlv78wt OD75w7Z2pFUc5IygVygd9N26v1iGdboS1lK3GRyQpje7BZL2aics+AyXFywEFzgNz1J6 rQCQ1DFDfvhXkL10JQDi+SXkChpFRgk2TFXBjpme72rFlu6bEnHpTEowupBTH0OwcjrT Cvlc79q/is4dDADD06qhPDxuped699Qv5p3wtSnn3VloA1vA5XZ2eMPUsmX3j4s9ZCcN rghw== MIME-Version: 1.0 From: Anthony Green Date: Fri, 15 Nov 2019 11:48:00 -0000 Message-ID: Subject: Evaluating libffi test results with Red Light Green Light To: libffi-discuss Content-Type: text/plain; charset="UTF-8" X-IsSubscribed: yes X-SW-Source: 2019/txt/msg00081.txt.bz2 In preparation for the 3.3 release, I've been spending a lot of time shoring up the travis-ci build/test infrastructure to include as much testing as possible. The goal, of course, is to have travis-ci give the green light when no regressions are introduced. Unfortunately, it's too much to expect perfect test results on travis-ci. Sometimes tests fail for environmental reasons. For example, qemu is failing certain tests that would pass on real hardware, wine is emitting messages that are confusing dejagnu, or wholesale execution test failures for certain emulated targets where we still want to test the build process. Last year I wrote a tool called Red Light Green Light targeted at different use cases, but I recently realized that it could be adapted to help evaluate the results of dejagnu test results, to decide if they are 'good enough' based on some policy. While dejagnu has the ability to mark tests as XFAIL (expected fail), I'm talking about failures that aren't what I would normally XFAIL. They are failings in the execution platform -- not the software being tested. If you look at any of the travis test logs now, you'll see something like this.. ./rlgl e --id=4de4d47 --policy=https://github.com/libffi/rlgl-policy.git m68k-unknown-linux-gnu/testsuite/libffi.log 1452GREEN: https://rl.gl/doc?id=RLGL-TNLN67PM AG This shows how we send libffi.log to rl.gl for evaluation against the given policy. The results are "GREEN" (good) and the link points at the analysis, including the original libffi.log. If you dig into the report, you'll see that all of the qemu execution tests fail for this target - but the policy accounts for this and still gives us a green light - because build tests are better than no tests at all. All this to say, I welcome you to have a look at Red Light Green Light, here: https://github.com/atgreen/red-light-green-light Check out the hosted version here: https://rl.gl And have a look at the libffi rlgl policy files here: https://github.com/libffi/rlgl-policy Thanks, AG