announce: egcs benchmark suite, alpha release

announce: egcs benchmark suite, alpha release

[Marc Lehmann]

I'm pleased to announce the first public release of the egcs benchmark suite.

WHAT IS IT?

   The egcs benchmark suite is a collection of individual tests used to
   determine relative performance changes of specific optimizations. There are
   tests that check performance of different datatypes on the same problem,
   tests that measure the C++ abstraction penalty, tests measuring function
   inlining/call overhead and more...

   Regular benchmarking will also allow to detect optimization problems.

WHAT IS IT _NOT_?

   It's not a general purpose benchmark library. It's unsuitable for comparing
   performance of one machine/os to another, and it's also unsuitable to compare
   performance between different compilers. Its sole purpose is to measure speed
   enhancements (or deficiencies) between egcs compiler revisions.

   In short, what the testsuite is for compiler correctness, is the benchmark
   suite for optimizer quality.

WHAT WE NEED

   at the moment are people that alpha test the release, give compatibility
   hints, detect problems with the setup.

   if the suite turns out to be useful, we need regular testers so we can fill
   the benchmark database and write cute perl programs that draw even cuter
   graphs ;)

   Also we need input on how to handle compiler switches, i.e. should
   we run a single test with -O2, or multiple tests with -O2, -O3, -O3
   -funroll-loops etc. (and how do we keep the results comparable in the
   latter case)

WHERE TO GET IT?

   At the moment its "only" available via cvs. Instead of checking out the
   "egcs" directory you can check out the "benchmarks" directory. (Just follow
   the instructions for checking out "egcs" but replace it by "benchmarks")

HOW TO USE IT?

   Yes, the README is not quite up-to-date, but everything you should have to do is to type:

   make

   which will compile & run all benchmarks and automatically mail me the
   results, testing everything in the package.

THANKS

   OH YES, thanks a lot to Jan Hubicka, who not only had the original idea to
   create the benchmark suite, but also delivered all the initial tests and
   programmed the initial testing framework.


      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: announce: egcs benchmark suite, alpha release

[Arvind Sankar]

hi! I tried checking out the benchmarks a while back.
When I do make check, the first test (c++/oopack) works properly, but
then everything else bums with LINK ERROR.
If I try to run make check again, make bums with this queer error:
gcc   bench.o bench.sum   -o bench
/usr/bin/ld:bench.sum: file format not recognized; treating as linker script
/usr/bin/ld:bench.sum:2: parse error
collect2: ld returned 1 exit status
make: *** [bench] Error 1

Apparently, make finds an implicit rule bench: bench.o, and that causes
it do stuff like this. Shouldn't a lot of the targets be made .PHONY?
Right now, bench.sum has no dependents, so it is always up-to-date.


On Sat, Jan 16, 1999 at 06:52:08AM +0100, Marc Lehmann wrote:
> I'm pleased to announce the first public release of the egcs benchmark suite.
> 
> WHAT IS IT?
> 
>    The egcs benchmark suite is a collection of individual tests used to
>    determine relative performance changes of specific optimizations. There are
>    tests that check performance of different datatypes on the same problem,
>    tests that measure the C++ abstraction penalty, tests measuring function
>    inlining/call overhead and more...
> 
>    Regular benchmarking will also allow to detect optimization problems.
> 
> WHAT IS IT _NOT_?
> 
>    It's not a general purpose benchmark library. It's unsuitable for comparing
>    performance of one machine/os to another, and it's also unsuitable to compare
>    performance between different compilers. Its sole purpose is to measure speed
>    enhancements (or deficiencies) between egcs compiler revisions.
> 
>    In short, what the testsuite is for compiler correctness, is the benchmark
>    suite for optimizer quality.
> 
> WHAT WE NEED
> 
>    at the moment are people that alpha test the release, give compatibility
>    hints, detect problems with the setup.
> 
>    if the suite turns out to be useful, we need regular testers so we can fill
>    the benchmark database and write cute perl programs that draw even cuter
>    graphs ;)
> 
>    Also we need input on how to handle compiler switches, i.e. should
>    we run a single test with -O2, or multiple tests with -O2, -O3, -O3
>    -funroll-loops etc. (and how do we keep the results comparable in the
>    latter case)
> 
> WHERE TO GET IT?
> 
>    At the moment its "only" available via cvs. Instead of checking out the
>    "egcs" directory you can check out the "benchmarks" directory. (Just follow
>    the instructions for checking out "egcs" but replace it by "benchmarks")
> 
> HOW TO USE IT?
> 
>    Yes, the README is not quite up-to-date, but everything you should have to do is to type:
> 
>    make
> 
>    which will compile & run all benchmarks and automatically mail me the
>    results, testing everything in the package.
> 
> THANKS
> 
>    OH YES, thanks a lot to Jan Hubicka, who not only had the original idea to
>    create the benchmark suite, but also delivered all the initial tests and
>    programmed the initial testing framework.
> 
> 
>       -----==-                                             |
>       ----==-- _                                           |
>       ---==---(_)__  __ ____  __       Marc Lehmann      +--
>       --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
>       -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
>     The choice of a GNU generation                       |
>                                                          |

Re: announce: egcs benchmark suite, alpha release

[Arvind Sankar]

On Wed, Jan 20, 1999 at 09:54:31PM -0500, Arvind Sankar wrote:
> hi! I tried checking out the benchmarks a while back.
> When I do make check, the first test (c++/oopack) works properly, but
> then everything else bums with LINK ERROR.
> If I try to run make check again, make bums with this queer error:
> gcc   bench.o bench.sum   -o bench
> /usr/bin/ld:bench.sum: file format not recognized; treating as linker script
> /usr/bin/ld:bench.sum:2: parse error
> collect2: ld returned 1 exit status
> make: *** [bench] Error 1
> 
> Apparently, make finds an implicit rule bench: bench.o, and that causes
> it do stuff like this. Shouldn't a lot of the targets be made .PHONY?
> Right now, bench.sum has no dependents, so it is always up-to-date.

fixed that, and also the local decls in benchlib.sh:runtest() shouldn't
be commented out. If they aren't portable or something, then ofiles
needs to be initialized. Currently, the tmp files keep accumulating as
the tests are processed.

-- arvind

Re: announce: egcs benchmark suite, alpha release

[Arvind Sankar]

On Wed, Jan 20, 1999 at 09:54:31PM -0500, Arvind Sankar wrote:
> hi! I tried checking out the benchmarks a while back.
> When I do make check, the first test (c++/oopack) works properly, but
> then everything else bums with LINK ERROR.
> If I try to run make check again, make bums with this queer error:
> gcc   bench.o bench.sum   -o bench
> /usr/bin/ld:bench.sum: file format not recognized; treating as linker script
> /usr/bin/ld:bench.sum:2: parse error
> collect2: ld returned 1 exit status
> make: *** [bench] Error 1
> 
> Apparently, make finds an implicit rule bench: bench.o, and that causes
> it do stuff like this. Shouldn't a lot of the targets be made .PHONY?
> Right now, bench.sum has no dependents, so it is always up-to-date.

fixed that, and also the local decls in benchlib.sh:runtest() shouldn't
be commented out. If they aren't portable or something, then ofiles
needs to be initialized. Currently, the tmp files keep accumulating as
the tests are processed.

-- arvind

Re: announce: egcs benchmark suite, alpha release

[Arvind Sankar]

hi! I tried checking out the benchmarks a while back.
When I do make check, the first test (c++/oopack) works properly, but
then everything else bums with LINK ERROR.
If I try to run make check again, make bums with this queer error:
gcc   bench.o bench.sum   -o bench
/usr/bin/ld:bench.sum: file format not recognized; treating as linker script
/usr/bin/ld:bench.sum:2: parse error
collect2: ld returned 1 exit status
make: *** [bench] Error 1

Apparently, make finds an implicit rule bench: bench.o, and that causes
it do stuff like this. Shouldn't a lot of the targets be made .PHONY?
Right now, bench.sum has no dependents, so it is always up-to-date.


On Sat, Jan 16, 1999 at 06:52:08AM +0100, Marc Lehmann wrote:
> I'm pleased to announce the first public release of the egcs benchmark suite.
> 
> WHAT IS IT?
> 
>    The egcs benchmark suite is a collection of individual tests used to
>    determine relative performance changes of specific optimizations. There are
>    tests that check performance of different datatypes on the same problem,
>    tests that measure the C++ abstraction penalty, tests measuring function
>    inlining/call overhead and more...
> 
>    Regular benchmarking will also allow to detect optimization problems.
> 
> WHAT IS IT _NOT_?
> 
>    It's not a general purpose benchmark library. It's unsuitable for comparing
>    performance of one machine/os to another, and it's also unsuitable to compare
>    performance between different compilers. Its sole purpose is to measure speed
>    enhancements (or deficiencies) between egcs compiler revisions.
> 
>    In short, what the testsuite is for compiler correctness, is the benchmark
>    suite for optimizer quality.
> 
> WHAT WE NEED
> 
>    at the moment are people that alpha test the release, give compatibility
>    hints, detect problems with the setup.
> 
>    if the suite turns out to be useful, we need regular testers so we can fill
>    the benchmark database and write cute perl programs that draw even cuter
>    graphs ;)
> 
>    Also we need input on how to handle compiler switches, i.e. should
>    we run a single test with -O2, or multiple tests with -O2, -O3, -O3
>    -funroll-loops etc. (and how do we keep the results comparable in the
>    latter case)
> 
> WHERE TO GET IT?
> 
>    At the moment its "only" available via cvs. Instead of checking out the
>    "egcs" directory you can check out the "benchmarks" directory. (Just follow
>    the instructions for checking out "egcs" but replace it by "benchmarks")
> 
> HOW TO USE IT?
> 
>    Yes, the README is not quite up-to-date, but everything you should have to do is to type:
> 
>    make
> 
>    which will compile & run all benchmarks and automatically mail me the
>    results, testing everything in the package.
> 
> THANKS
> 
>    OH YES, thanks a lot to Jan Hubicka, who not only had the original idea to
>    create the benchmark suite, but also delivered all the initial tests and
>    programmed the initial testing framework.
> 
> 
>       -----==-                                             |
>       ----==-- _                                           |
>       ---==---(_)__  __ ____  __       Marc Lehmann      +--
>       --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
>       -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
>     The choice of a GNU generation                       |
>                                                          |

egcs benchmark suite, nits

[Marc Espie]

This small patch:
- removes the dependency on gnu-make features of the Makefile,
- makes it available to your average paranoid sys-admin who doesn't have
. in his PATH.

There may be some other problems, I have been too busy with other funny
egcs things to run the benchmark to completion yet, but thanks !

Index: benchmarks/Makefile
===================================================================
RCS file: /egcs/carton/cvsfiles/benchmarks/Makefile,v
retrieving revision 1.12
diff -u -r1.12 Makefile
--- Makefile	1999/01/27 12:06:50	1.12
+++ Makefile	1999/01/28 13:48:02
@@ -40,8 +40,9 @@
 mhz: lib/mhz
 	lib/mhz >mhz
 	
-lib/mhz: lib/mhz.c lib/lib_timing.c lib/lib_stats.c
-	$(CC) -O -o $@ $^ -lm
+MHZ=lib/mhz.c lib/lib_timing.c lib/lib_stats.c
+lib/mhz: ${MHZ}
+	$(CC) -O -o $@  ${MHZ} -lm
 
 bench.a: bench.o
 	$(AR) r $@ $?
Index: benchmarks/runbench
===================================================================
RCS file: /egcs/carton/cvsfiles/benchmarks/runbench,v
retrieving revision 1.9
diff -u -r1.9 runbench
--- runbench	1999/01/27 12:06:55	1.9
+++ runbench	1999/01/28 13:48:02
@@ -12,7 +12,7 @@
 
 VERSION=0.9.2
 
-. benchlib.sh
+. ./benchlib.sh
 
 # make this default the same as in the Makefile
 TIME=5
@@ -64,7 +64,7 @@
 done
 
 init_bench
-. bench.def
+. ./bench.def
 end_bench
 
 

egcs benchmark suite, nits

[Marc Espie]

This small patch:
- removes the dependency on gnu-make features of the Makefile,
- makes it available to your average paranoid sys-admin who doesn't have
. in his PATH.

There may be some other problems, I have been too busy with other funny
egcs things to run the benchmark to completion yet, but thanks !

Index: benchmarks/Makefile
===================================================================
RCS file: /egcs/carton/cvsfiles/benchmarks/Makefile,v
retrieving revision 1.12
diff -u -r1.12 Makefile
--- Makefile	1999/01/27 12:06:50	1.12
+++ Makefile	1999/01/28 13:48:02
@@ -40,8 +40,9 @@
 mhz: lib/mhz
 	lib/mhz >mhz
 	
-lib/mhz: lib/mhz.c lib/lib_timing.c lib/lib_stats.c
-	$(CC) -O -o $@ $^ -lm
+MHZ=lib/mhz.c lib/lib_timing.c lib/lib_stats.c
+lib/mhz: ${MHZ}
+	$(CC) -O -o $@  ${MHZ} -lm
 
 bench.a: bench.o
 	$(AR) r $@ $?
Index: benchmarks/runbench
===================================================================
RCS file: /egcs/carton/cvsfiles/benchmarks/runbench,v
retrieving revision 1.9
diff -u -r1.9 runbench
--- runbench	1999/01/27 12:06:55	1.9
+++ runbench	1999/01/28 13:48:02
@@ -12,7 +12,7 @@
 
 VERSION=0.9.2
 
-. benchlib.sh
+. ./benchlib.sh
 
 # make this default the same as in the Makefile
 TIME=5
@@ -64,7 +64,7 @@
 done
 
 init_bench
-. bench.def
+. ./bench.def
 end_bench
 
 

Re: announce: egcs benchmark suite, alpha release

[Dave Love]

Do the copyright problems for the testsuite apply to this too?  There
are some Fortranesque things we could add, but at least one of was a
candidate of mine the testsuite.

linpack.f in benchmark is in that category.  Note also that Toon has
pointed out that the hand loop unrolling in its daxpy routine loses
with g77 (or did at one time); it should probably use the current BLAS
daxpy for a more realistic Fortran loop test.  Toon?

Re: announce: egcs benchmark suite, alpha release

[Jeffrey A Law]

  In message < rzqd84ert0w.fsf@djlvig.dl.ac.uk >you write:
  > Do the copyright problems for the testsuite apply to this too?  There
  > are some Fortranesque things we could add, but at least one of was a
  > candidate of mine the testsuite.
Not at the current time since we are not planning on distributing the
benchmarks as part of the compiler.

However, it would probably be wise to be more careful with the benchmark
suite than we were with the testsuites.  ie, let's keep track of where the
code came from at the least so that we'll know its status if we ever want to
consider including the benchmarks.

It'll also force us to think a little before including something -- where did
it come from, what are the redistribution terms, are we in compliance with 
the redistribution terms.

jeff

Re: announce: egcs benchmark suite, alpha release

[Marc Lehmann]

On Sun, Jan 17, 1999 at 03:31:54PM -0700, Jeffrey A Law wrote:
> 
>   In message <rzqd84ert0w.fsf@djlvig.dl.ac.uk>you write:
>   > Do the copyright problems for the testsuite apply to this too?  There
>   > are some Fortranesque things we could add, but at least one of was a

At the moment, we have only linpack.f. Contributions are welcome ;) I had to
write a c-wrapper for linpack since I'm fortran illiterate and didn't know
how to interface the c functions properly ;)

>   > candidate of mine the testsuite.
> Not at the current time since we are not planning on distributing the
> benchmarks as part of the compiler.
> 
> However, it would probably be wise to be more careful with the benchmark
> suite than we were with the testsuites.  ie, let's keep track of where the
> code came from at the least so that we'll know its status if we ever want to
> consider including the benchmarks.

At the moment, it looks like this:

all tests in the "tests" subdir are GPL'd, everything else which is free but
not gpl'ed is in the "nongpl" dir.

Every benchmark added to the suite is required to have a SOURCE comment (here
is an example from linpack.f):

c     TEST linpack_f
c     SOURCE http://www.netlib.org/benchmarks/
c     DESC 100x100 linpack, initialization time is also measured
c     CAT fortran
c     END

all of which, btw, should be machine parseable. The worst that could happen
would be to get rid of the nongpl parts.

> It'll also force us to think a little before including something -- where did
> it come from, what are the redistribution terms, are we in compliance with 
> the redistribution terms.

the SOURCE comment can't answer this. I thought about adding a
machine-parsable LICENSE comment or similar, but would this be of any help?
I'd rather create sgi and bsd subdirectories...

(Comments welcome ;)

How about an ASSIGNED keyword indicating wether the tests are assigned,
should the need arise. And an AUTHOR comment?

IF we ever would distribute the benchmark suite (I don't yet see the need to
anyway), we will have to think about the home-grown licenses in the nongpl
dir.

      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: announce: egcs benchmark suite, alpha release

[Dave Love]

>>>>> "Marc" == Marc Lehmann <pcg@goof.com> writes:

 Marc> At the moment, we have only linpack.f. Contributions are
 Marc> welcome ;) I had to write a c-wrapper for linpack since I'm
 Marc> fortran illiterate and didn't know how to interface the c
 Marc> functions properly ;)

The normal linpack benchmark is complete.  I obviously didn't look
closely enough at what's there.

 Marc> At the moment, it looks like this:

 Marc> all tests in the "tests" subdir are GPL'd, everything else
 Marc> which is free but not gpl'ed is in the "nongpl" dir.

It sounds to me as though cataloguing according to
assigment/disclaimer status would be more useful.  Is (properly free)
non-GPL stuff less useful than GPLed in ths context?

 Marc> How about an ASSIGNED keyword indicating wether the tests are
 Marc> assigned, should the need arise. And an AUTHOR comment?

You surely need complete author info if assigment/disclaimer questions
arise.  [Says a muggins chasing such things post-hoc for Emacs...]

Re: announce: egcs benchmark suite, alpha release

[Marc Lehmann]

sidenote: I've just created http://egcs.cygnus.com/benchresults/ not too
useful, but a start.

On Sun, Jan 24, 1999 at 04:28:16PM +0000, Dave Love wrote:
>  Marc> fortran illiterate and didn't know how to interface the c
>  Marc> functions properly ;)
> 
> The normal linpack benchmark is complete.  I obviously didn't look
> closely enough at what's there.

We need a standardized format, though. The timing functions from linpack
are useless for us. It should be trivial, though, to use our functions from
fortran (I've just no idea ;->)

>  Marc> all tests in the "tests" subdir are GPL'd, everything else
>  Marc> which is free but not gpl'ed is in the "nongpl" dir.
> 
> It sounds to me as though cataloguing according to
> assigment/disclaimer status would be more useful.  Is (properly free)
> non-GPL stuff less useful than GPLed in ths context?

Well, nothing is yet assigned, does "benchmarks" count as being part of gcc?
;-> But indeed, all tests/* except oopack are contributed by Jan and me (both
of us have, I think, an assignment for gcc)

oopack is gpl'ed, but of course not assigned.

I think adding a marker to each file that was being assigned would be ok
("SOURCE ASSIGNED").

Also, seperating gpl tests from nongpl tests seems useful: can somebody
assign non-gpl code to the fsf??

>  Marc> How about an ASSIGNED keyword indicating wether the tests are
>  Marc> assigned, should the need arise. And an AUTHOR comment?
> 
> You surely need complete author info if assigment/disclaimer questions
> arise.  [Says a muggins chasing such things post-hoc for Emacs...]

No problem. At the moment this is hypothetical, as nothing is assigned
yet, unless the benchmark suite becomes part of gcc, or people start
sending assignments for this program to the fsf (or to jeff, who will be
"happy" to collect yet-useless assignments ;)

Is there anything we have to do _now_, except possibly documenting future
options (AUTHOR, STATUS...)

      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: announce: egcs benchmark suite, alpha release

[Marc Lehmann]

sidenote: I've just created http://egcs.cygnus.com/benchresults/ not too
useful, but a start.

On Sun, Jan 24, 1999 at 04:28:16PM +0000, Dave Love wrote:
>  Marc> fortran illiterate and didn't know how to interface the c
>  Marc> functions properly ;)
> 
> The normal linpack benchmark is complete.  I obviously didn't look
> closely enough at what's there.

We need a standardized format, though. The timing functions from linpack
are useless for us. It should be trivial, though, to use our functions from
fortran (I've just no idea ;->)

>  Marc> all tests in the "tests" subdir are GPL'd, everything else
>  Marc> which is free but not gpl'ed is in the "nongpl" dir.
> 
> It sounds to me as though cataloguing according to
> assigment/disclaimer status would be more useful.  Is (properly free)
> non-GPL stuff less useful than GPLed in ths context?

Well, nothing is yet assigned, does "benchmarks" count as being part of gcc?
;-> But indeed, all tests/* except oopack are contributed by Jan and me (both
of us have, I think, an assignment for gcc)

oopack is gpl'ed, but of course not assigned.

I think adding a marker to each file that was being assigned would be ok
("SOURCE ASSIGNED").

Also, seperating gpl tests from nongpl tests seems useful: can somebody
assign non-gpl code to the fsf??

>  Marc> How about an ASSIGNED keyword indicating wether the tests are
>  Marc> assigned, should the need arise. And an AUTHOR comment?
> 
> You surely need complete author info if assigment/disclaimer questions
> arise.  [Says a muggins chasing such things post-hoc for Emacs...]

No problem. At the moment this is hypothetical, as nothing is assigned
yet, unless the benchmark suite becomes part of gcc, or people start
sending assignments for this program to the fsf (or to jeff, who will be
"happy" to collect yet-useless assignments ;)

Is there anything we have to do _now_, except possibly documenting future
options (AUTHOR, STATUS...)

      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: announce: egcs benchmark suite, alpha release

[Dave Love]

>>>>> "Marc" == Marc Lehmann <pcg@goof.com> writes:

 Marc> At the moment, we have only linpack.f. Contributions are
 Marc> welcome ;) I had to write a c-wrapper for linpack since I'm
 Marc> fortran illiterate and didn't know how to interface the c
 Marc> functions properly ;)

The normal linpack benchmark is complete.  I obviously didn't look
closely enough at what's there.

 Marc> At the moment, it looks like this:

 Marc> all tests in the "tests" subdir are GPL'd, everything else
 Marc> which is free but not gpl'ed is in the "nongpl" dir.

It sounds to me as though cataloguing according to
assigment/disclaimer status would be more useful.  Is (properly free)
non-GPL stuff less useful than GPLed in ths context?

 Marc> How about an ASSIGNED keyword indicating wether the tests are
 Marc> assigned, should the need arise. And an AUTHOR comment?

You surely need complete author info if assigment/disclaimer questions
arise.  [Says a muggins chasing such things post-hoc for Emacs...]

Re: announce: egcs benchmark suite, alpha release

[Marc Lehmann]

On Sun, Jan 17, 1999 at 03:31:54PM -0700, Jeffrey A Law wrote:
> 
>   In message < rzqd84ert0w.fsf@djlvig.dl.ac.uk >you write:
>   > Do the copyright problems for the testsuite apply to this too?  There
>   > are some Fortranesque things we could add, but at least one of was a

At the moment, we have only linpack.f. Contributions are welcome ;) I had to
write a c-wrapper for linpack since I'm fortran illiterate and didn't know
how to interface the c functions properly ;)

>   > candidate of mine the testsuite.
> Not at the current time since we are not planning on distributing the
> benchmarks as part of the compiler.
> 
> However, it would probably be wise to be more careful with the benchmark
> suite than we were with the testsuites.  ie, let's keep track of where the
> code came from at the least so that we'll know its status if we ever want to
> consider including the benchmarks.

At the moment, it looks like this:

all tests in the "tests" subdir are GPL'd, everything else which is free but
not gpl'ed is in the "nongpl" dir.

Every benchmark added to the suite is required to have a SOURCE comment (here
is an example from linpack.f):

c     TEST linpack_f
c     SOURCE http://www.netlib.org/benchmarks/
c     DESC 100x100 linpack, initialization time is also measured
c     CAT fortran
c     END

all of which, btw, should be machine parseable. The worst that could happen
would be to get rid of the nongpl parts.

> It'll also force us to think a little before including something -- where did
> it come from, what are the redistribution terms, are we in compliance with 
> the redistribution terms.

the SOURCE comment can't answer this. I thought about adding a
machine-parsable LICENSE comment or similar, but would this be of any help?
I'd rather create sgi and bsd subdirectories...

(Comments welcome ;)

How about an ASSIGNED keyword indicating wether the tests are assigned,
should the need arise. And an AUTHOR comment?

IF we ever would distribute the benchmark suite (I don't yet see the need to
anyway), we will have to think about the home-grown licenses in the nongpl
dir.

      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |