From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 13365 invoked by alias); 23 Feb 2003 23:54:52 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 13351 invoked from network); 23 Feb 2003 23:54:52 -0000 Received: from unknown (HELO caip.rutgers.edu) (128.6.236.10) by 172.16.49.205 with SMTP; 23 Feb 2003 23:54:52 -0000 Received: (from ghazi@localhost) by caip.rutgers.edu (8.9.3/8.9.3) id SAA19348; Sun, 23 Feb 2003 18:54:52 -0500 (EST) Date: Mon, 24 Feb 2003 00:30:00 -0000 From: "Kaveh R. Ghazi" Message-Id: <200302232354.SAA19348@caip.rutgers.edu> To: gcc-patches@gcc.gnu.org, gcc@gcc.gnu.org Subject: GCC 3.3 release criteria Cc: mark@codesourcery.com X-SW-Source: 2003-02/txt/msg01564.txt.bz2 I think it's well past time we had written release criteria for GCC 3.3. I propose starting with the 3.1 criteria and working from there. I've copied criteria.html from the gcc-3.1 to the gcc-3.3 directory and changed "3.1" -> "3.3" everywhere. I also removed a paragraph relating to the C++ ABI change in 3.2. Then we can make changes from there in patch form. The entire file is included below. Ok to install? Where should we link to it from? Thanks, --Kaveh GCC 3.3 Release Criteria

GCC 3.3 Release Criteria

This page provides the release criteria for GCC 3.3. GCC 3.3 will not be released until these criteria have been met. This page enumerates the release criteria and provides a rationale for some of the choices made in determining these criteria.

In all cases, these criteria represent the minimum functionality required in order to make the release. If this level of minimum functionality is not provided by a release candidate, then that candidate will not become the eventual release. However, a release candidate that does meet these criteria may not necessarily become the official release; there may be other unforseen issues that prevent release. For example, if support for the Intel Pentium II is required by the release criteria, it is nevertheless unlikely that GCC 3.3 would be released even though it did not support the Intel Pentium.

Because the development of GCC is largely dependent on volunteers, the Steering Committee may eventually have to decide whether to make a release, even if the criteria here are not met. For example, if no volunteer can be found to verify correct operation of a particular application program on a particular system, then that criterion may be abandoned. However, that eventuality should be avoided if at all possible.

Bug Fixes

We strive to fix all open problem reports (PRs) in our bug tracking system that indicate a regression with respect to previous releases.

Such PRs are marked as high-priority items, and every GCC maintainer with CVS and GNATS write access may set PRs indicating a regression to high-priority.

Platform Support

GCC is available on a vast number of platforms. However, it is not possible to effectively test GCC in all possible configurations. Therefore, a smaller number of platforms have been selected as targets. The targets chosen represent both the most popular operating systems and the most popular microprocessors. Of course, where possible, the release will support other targets as well.

Primary Evaluation Platforms
Chip	OS	Triplet	Tester
Alpha	RedHat Linux 7.1	alpha-unknown-linux-gnu	Richard Henderson
HPPA	HPUX 11.0	hppa2.0w-hp-hpux11.00	John David Anglin
Intel x86	Debian GNU/Linux 2.2	i386-pc-linux-gnu
Intel x86	RedHat Linux 6.2	i686-pc-linux-gnu
Intel x86	FreeBSD 4.5	i386-unknown-freebsd4.5	Loren James Rittle
MIPS	IRIX 6.5	mips-sgi-irix6.5
PowerPC	AIX 4.3.3	powerpc-ibm-aix4.3.3.0	David Edelsohn
SPARC	Solaris 2.7	sparc-sun-solaris2.7

GCC's performance on the following platforms will not be required to meet all the criteria mentioned in this document before GCC 3.3 ships, but the performance on these systems will be of considerable interest, and it is likely that serious problems on these platforms will delay the release.

Among the secondary evaluation platforms, we are are especially concerned about free systems (i.e., GNU/Linux and the BSDs) where GCC also serves as the system compiler.

Volunteers will be required, both to test and to fix bugs, for all secondary platforms. (These volunteers may be the same person, but volunteers should be careful not to sign up for more work than they can actually do.) If volunteers cannot be found for these platforms, then the secondary platforms will be dropped from this list.

The bug-fixing volunteer will commit to ensuring that GCC 3.3 will at least bootstrap itself on each of these secondary platforms. That commitment doesn't necessarily mean fixing bugs personally; for example, if you are a manager for a company with GCC expertise you could be the volunteer if you'll commit to donating your employee's efforts as necessary. The release manager, and the GCC development team, will make reasonable efforts to assist these volunteers by answering questions and reviewing patches as time permits.

Secondary Evaluation Platforms
Chip	OS	Triplet	Tester
PowerPC	GNU/Linux	powerpc-linux-gnu	Franz Sirl
SPARC	Debian GNU/Linux 2.2	sparc-sun-linux-gnu	Ben Collins
ARM	GNU/Linux	armv4l-unknown-linux-gnu
Intel x86	Cygwin	i686-pc-cygwin

Language Support

There are GCC front-ends for many different languages. However, in this release, only the behavior of front-ends for the following languages will be considered part of the release criteria:

C
C++
Java
Fortran

The following languages will be supported by the release, but their behavior will not be a primary consideration in determining whether or not to ship a particular release candidate:

Ada
Objective-C

In particular, no application testing, code quality, or compile-time performance testing will be required for these languages. However, the regression testing criteria documented below will apply to these languages.

Regression Tests

The GCC testsuite contains extensive C and C++ regression tests, as well as some Fortran, and Objective-C tests. GCC 3.3 will not fail any of these tests which the previous release GCC passed on any of the supported platforms. In particular, the current regression testsuite will be run using GCC 3.0.4 and GCC 2.95.3 on each of the supported platforms; those results can then be compared with the output from a release candidate. Because there have often been issues with generating PIC code, we will test with -fPIC as well.

In addition, on all supported platforms, there will be no --enable-checking failures when running any of the regression test-suites.

Additional Tests

Ideally, strict compliance with the following criteria would be required for all releases of GCC, however this is simply not practical for GCC 3.3. Still, testers are encouraged to perform these tests and report possible problems.

Applications

It is important that the compiler is verified on real-world applications. The following applications represent a mix of low-level and high-level code, of numerical and logical programs, and of different programming languages.

Integration Tests
Name	Language	Version	Source URL	Build and test guide
ACE	C++	5.2	ACE (download)
Blitz	C++	20001213	blitz-20001213.tar.gz	build and test guide
Boost	C++	1.22.0	boost_all.tar.gz	build and test guide
FTensor	C++	1.1 patch 16	FTensor_gcc_integration_test.tar.gz	build and test guide
GNU Emacs	C	20.6
LAPACK	Fortran	3.0	LAPACK (testing programs)	build and test guide
Linux kernel	C	2.4.18	linux-2.4.18.tar.gz
MTL	C++	2.12.2.-20	MTL (Download), with patch
POOMA	C++	2.3.0	pooma-gcc.tar.gz	build and test guide
Qt	C++	2.3.0	qt-x11-2.3.0.tar.gz	build and test guide
root	C++	3.01.00	root-3.01

These selections were made for a variety of reasons. The Linux kernel is one of the most important pieces of free software, and kernel developers pay careful attention to GCC performance. It would be an embarrassment if GCC did not compile the kernel correctly, out of the box. The kernel taxes many of the low-level aspects of GCC, as well as many GCC extensions, including the extended assembly syntax, addresses of labels, and so forth. (Historically, there have been kernel bugs, found only by more aggressive optimization in new releases of GCC. If such bugs are encountered, then appropriate patches should be applied to the kernel before testing.)

GNU Emacs is portable to almost every system available, and is a complex application-level C program, known to have very few bugs.

POOMA is a complex expression-template library that will tax the ability of G++ to deal with templates, an area that has historically been buggy. In addition, templates have historically taken inordinately much time and memory at compile-time. With the widespread prevalence of templates in C++ programs, including the standard library, testing this area heavily is vitally important. Pooma-gcc is pooma-2.3.0 plus some scripts which simplify testing.

LAPACK is a well known linear algebra package that contains code typical for large scale Fortran programs.

Code Quality

Historically, there has been no formal release criterion that took into account performance of code generated by the compiler. It is important that the generated code performs approximately as well as previous releases. Therefore, we will use the following benchmarks for measuring code quality:

Name	Language	Source URL
gzip 1.2.4a	C
Stepanov	C++	stepanov_v1p2.C
LAPACK	Fortran	LAPACK 3.0 (timing programs)

A Java benchmark is not required for this release since there is little precedent for the behavior of the Java compiler. For Java, functional completeness and correctness are still more important than optimization.

In addition to the above benchmarks, the behavior of real programs should be considered as well. For that reason, the behavior of the elliptic curve integer factorization program ecm4c.c, which uses GNU mp, will be considered part of the release criteria.

The performance of the generated code of a release candidate should be at least as good as that of past releases of GCC since 2.95.3 on the benchmarks, and within at least 5% on the application tests.

Compile-Time Performance

There is a perception that current versions of GCC take longer to compile programs than their 2.95.3 counterparts, and that they often use more memory as well. Compile-time performance is an important part of compiler quality. It is not enough simply to provide additional optimizations; the compiler must also continue to compile programs relatively quickly. However, it is to be expected that additional optimizations and additional features will have a non-zero cost.

In order to measure compile-time performance, we will use the following unit tests:

Name	Language	Flags	Comments
insn-attrtab.c	C	-O2	This file contains a large machine-generated switch statement; it is a reasonable benchmark for testing flow optimizations and for handling large functions.
	C++
	Fortran

In addition to these unit tests, time and peak memory usage used when building the entire GNU Emacs distribution should be measured.

A release candidate's compile-time should not exceed GCC 2.95.3 by more than 15%, and peak memory usage should not exceed that of GCC 2.95.3 by more than 25%.