From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 7104 invoked by alias); 15 Apr 2002 21:52:53 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 7085 invoked from network); 15 Apr 2002 21:52:51 -0000 Received: from unknown (HELO gandalf.codesourcery.com) (66.60.148.227) by sources.redhat.com with SMTP; 15 Apr 2002 21:52:51 -0000 Received: from gandalf.codesourcery.com (localhost.localdomain [127.0.0.1]) by gandalf.codesourcery.com (8.11.6/8.11.6) with ESMTP id g3FLoJL10067; Mon, 15 Apr 2002 14:50:19 -0700 Date: Mon, 15 Apr 2002 15:01:00 -0000 From: Mark Mitchell To: Geoff Keating cc: "gcc@gcc.gnu.org" Subject: Re: GCC 3.1 Release Message-ID: <74040000.1018907355@gandalf.codesourcery.com> In-Reply-To: References: <46690000.1018660657@gandalf.codesourcery.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-SW-Source: 2002-04/txt/msg00642.txt.bz2 > Can you say roughly what proportion of the problems (in your > subjective judgement, 'percentage of reason why I delayed the > release') were: Sure -- but do note that this really is subjective. I have no numbers. > 1 Problems visible on x86-linux but not visible to the regression > testsuite? 25%. Most front end bugs are in this category. > 2 Problems visible only on non-free platforms (eg. solaris, > AIX, cygwin)? 25%. These aren't really too common; there are now free OS's for most chips and I'm seeing more chip bugs than OS bugs. > 3 Problems visible on non-x86 free platforms (eg. alpha)? 40%. >4 Problems introduced by previous fixes made to the branch? Small, less than 10%. > The solutions that would work for these are different. (1) means we > need a better regression testsuite. (2) and (3) mean we need > more/better testing, perhaps an expanded regression tester, or to drop > some platforms---it may be that, say, Irix is just too hard to do > unless more volunteers can be found who are willing to maintain it. > (4) means we need better moderation and regression testing of the > branch, perhaps requiring that a patch be tested for regressions on > all the important platforms before it is committed; we do something > like that at Red Hat for our general-use native releases. Thanks for thinking about this and trying so hard to improve the testing system. I'd say the majority of the problems were not things covered by any regression test; they were bugs filed in GNATS. Obviously, we keep improving the regression testsuite over time, but unfortunately users are infinitely clever and keep trying to compile new programs. (It would be much easier if they would politely only compile a small finite set of programs...) I see two major ways to make progress: more testing (which you are thankfully working particularly hard on) and better internal infrastructure -- clearer interfaces, better documentation, higher-level abstractions -- which we are all working on. A lot of what's going on is that a change that looks just fine for C++ on x86 turns out to break Fortran on alpha -- and that almost shouldn't be possible if we were to do our job right. I do think that regression testers like your across a wide variety of platforms might eliminate as many as half of all bugs. There's also the problem that some people don't seem to want to invest very much effort to fix problems on platforms they don't use, even when it's clear they created the problem. That's a social issue. Thanks, -- Mark Mitchell mark@codesourcery.com CodeSourcery, LLC http://www.codesourcery.com